# Grafana 文档

Grafana 文档。内容保持上游原样。

# Administration

# Administer Grafana with the Grafana server CLI

You can administer your Grafana instance with the Grafana server CLI, a small executable bundled with Grafana server.

The Grafana server CLI has `plugins` and `admin` commands, as well as global options. To list them, run:

```
grafana cli -h
```

For more details read on.

## Run the Grafana server CLI

You can run the Grafana server CLI on the same machine Grafana server is running on.

To run the CLI you have the following options:

- Add the path to the Grafana binaries in your `PATH` environment variable.
- If your current directory is the `bin` directory, run `./grafana cli`.
- Otherwise, you can specify full path to the binary. For example, `/usr/share/grafana/bin/grafana` on Linux and `C:\Program Files\GrafanaLabs\grafana\bin\grafana.exe` on Windows.

The general syntax for commands in Grafana server CLI is:

```bash
grafana cli [global options] command [command options] [arguments...]
```

{{< admonition type="note" >}}
Some commands, such as installing or removing plugins, require `sudo` on Linux.
If you're on Windows, run Windows PowerShell as Administrator.
{{< /admonition >}}

## Global options

Grafana server CLI allows you to temporarily override certain Grafana default settings. Except for `--help` and `--version`, most global options are only used by developers.

Each global option applies only to the command in which it is used. For example, `--pluginsDir value` does not permanently change where Grafana saves plugins. It only changes it for command in which you apply the option.

### Display Grafana server CLI help

`--help` or `-h` displays the help, including default paths and Docker configuration information.

**Example:**

```bash
grafana cli -h
```

### Display Grafana server CLI version

`--version` or `-v` prints the version of Grafana server CLI currently running.

**Example:**

```bash
grafana cli -v
```

### Override default plugin directory

`--pluginsDir value` overrides the path to where your local Grafana instance stores plugins. Use this option if you want to install, update, or remove a plugin somewhere other than the default directory ("/var/lib/grafana/plugins") [$GF_PLUGIN_DIR].

**Example:**

```bash
grafana cli --pluginsDir "/var/lib/grafana/devplugins" plugins install <plugin-id>
```

### Override default plugin repo URL

`--repo value` allows you to download and install or update plugins from a repository other than the default Grafana repo.

**Example:**

```bash
grafana cli --repo "https://example.com/plugins" plugins install <plugin-id>
```

### Override default plugin .zip URL

`--pluginUrl value` allows you to download a .zip file containing a plugin from a local URL instead of downloading it from the default Grafana source.

**Example:**

```bash
grafana cli --pluginUrl https://company.com/grafana/plugins/<plugin-id>-<plugin-version>.zip plugins install <plugin-id>
```

### Override Transport Layer Security

**Warning:** Turning off TLS is a significant security risk. We do not recommend using this option.

`--insecure` allows you to turn off Transport Layer Security (TLS) verification (insecure). You might want to do this if you are downloading a plugin from a non-default source.

**Example:**

```bash
grafana cli --insecure --pluginUrl https://company.com/grafana/plugins/<plugin-id>-<plugin-version>.zip plugins install <plugin-id>
```

### Enable debug logging

`--debug` or `-d` enables debug logging. Debug output is returned and shown in the terminal.

**Example:**

```bash
grafana cli --debug plugins install <plugin-id>
```

### Override a configuration setting

`--configOverrides` is a command line argument that acts like an environmental variable override.

For example, you can use it to redirect logging to another file (maybe to log plugin installations in Grafana Cloud) or when resetting the admin password and you have non-default values for some important configuration value (like where the database is located).

**Example:**

```bash
grafana cli --configOverrides cfg:default.paths.log=/dev/null plugins install <plugin-id>
```

### Override homepath value

Sets the path for the Grafana install/home path, defaults to working directory. You do not need to use this if you are in the Grafana installation directory when using the CLI.

**Example:**

```bash
grafana cli --homepath "/usr/share/grafana" admin reset-admin-password <new password>
```

### Override config file

`--config value` overrides the default location where Grafana expects the configuration file. Refer to [Configuration](../setup-grafana/configure-grafana/) for more information about configuring Grafana and default configuration file locations.

**Example:**

```bash
grafana cli --config "/etc/configuration/" admin reset-admin-password mynewpassword
```

## Plugins commands

Grafana CLI allows you to install, upgrade, and manage your Grafana plugins. For more information about installing plugins, refer to [plugins page](../administration/plugin-management/).

All listed commands apply to the Grafana default repositories and directories. You can override the defaults with [Global Options](#global-options).

### List available plugins

```bash
grafana cli plugins list-remote
```

### Install the latest version of a plugin

```bash
grafana cli plugins install <plugin-id>
```

### Install a specific version of a plugin

```bash
grafana cli plugins install <plugin-id> <version>
```

### List installed plugins

```bash
grafana cli plugins ls
```

### Update all installed plugins

```bash
grafana cli plugins update-all
```

### Update one plugin

```bash
grafana cli plugins update <plugin-id>
```

### Remove one plugin

```bash
grafana cli plugins remove <plugin-id>
```

## Admin commands

Admin commands are only available in Grafana 4.1 and later.

### Show all admin commands

```bash
grafana cli admin
```

### Reset admin password

`grafana cli admin reset-admin-password <new password>` resets the password for the admin user using the CLI. You might need to do this if you lose the admin password. By default, this command uses the default ID of the admin user, which is 1. If you know the ID of the admin user, you can use the `--user-id` flag to specify the user ID. If the `--user-id` flag is not specified and the command cannot find the admin user, it returns the list of current admin users' username and ID. From that list you can determine the ID of the admin user and run the command again with the `--user-id` flag.

If there are two flags being used to set the homepath and the config file path, then running the command returns this error:

> Could not find config defaults, make sure homepath command line parameter is set or working directory is homepath

To correct this, use the `--homepath` global option to specify the Grafana default homepath for this command:

```bash
grafana cli --homepath "/usr/share/grafana" admin reset-admin-password <new password>
```

If you have not lost the admin password, we recommend that you change the user password either in the User Preferences or in the **Server Admin > User** tab.

If you need to set the password in a script, then you can use the [Grafana User API](../developers/http_api/user/#change-password).

#### Reset admin password

If you installed Grafana using Homebrew, you can reset the admin password using the following command:

```bash
/opt/homebrew/opt/grafana/bin/grafana cli --config /opt/homebrew/etc/grafana/grafana.ini --homepath /opt/homebrew/opt/grafana/share/grafana --configOverrides cfg:default.paths.data=/opt/homebrew/var/lib/grafana admin reset-admin-password <new password>
```

#### Reset admin password for Grafana deployed with Grafana Operator and using an external database

If you deploy Grafana with Grafana Operator and configure Grafana to use an external PostgreSQL or MySQL database, specify both the homepath and configuration file in your command:

```bash
grafana cli  --homepath /usr/share/grafana --config /etc/grafana/grafana.ini admin reset-admin-password <new password>
```

If you don't do this, the CLI:

- Won't see the database connection information since it's in a directory outside of the Grafana homepath
- Will configure the default SQLite database in `/var/lib/grafana` and reset that password instead of for your external database

### Migrate data and encrypt passwords

`data-migration` runs a script that migrates or cleans up data in your database.

`encrypt-datasource-passwords` migrates passwords from unsecured fields to secure_json_data field. Returns `ok` unless there is an error. Safe to execute multiple times.

**Example:**

```bash
grafana cli admin data-migration encrypt-datasource-passwords
```

# Back up Grafana

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA VERSION>" >}}

# Add permissions to create new correlations

## Before you begin

Adding access to create correlations for [Viewers and Editors](../../roles-and-permissions/) is available with [Role-based access control](../../roles-and-permissions/access-control/).

## Add permissions to create correlations

1. Go to the **Administration** section in Grafana.
1. Under **Users and access**, open the **Users** page.
1. Select the user to be granted access to create correlations.
1. Add _Data sources writer_ role.

# Correlation

Each correlation is configured with the following options:

**Label**
: Link label, shown in the visualization

**Description**
: Optional description

**Source data source**
: The source of results that have links displayed

**Results field**
: Defines where the link is shown in a visualization

**Target query**
: The target query run when a link is clicked

**Transformations**
: Optional manipulations to the source data included passed to the target query

Learn how to create correlations using the [Administration page](../create-a-new-correlation/#create-a-correlation-in-administration-page) or with [provisioning](../create-a-new-correlation/#create-a-correlation-with-provisioning).

## Source data source and result field

Links are shown in Explore visualizations for the results from the correlation’s source data source.
A link is assigned to one of the fields from the result provided in the correlation configuration (the results field).
Each visualization displays fields with links in a different way ([Correlations in Logs Panel](../use-correlations-in-visualizations/#correlations-in-logs-panel) and see [Correlations in Table](../use-correlations-in-visualizations/#correlations-in-table)).

## Target query

The target query is run when a link is clicked in the visualization. You can use the query editor of the selected target data source to specify the target query. Source data results can be accessed inside the target query with variables.

### Correlation Variables

You can use variables inside the target query to access the source data related to the query.
Correlations use [Grafana variable syntax](../../../dashboards/variables/variable-syntax/).
Variables are filled with values from the source results when the link is clicked. There are two types of variables you can use:

- [field variables](../../../panels-visualizations/configure-data-links/#field-variables) (allows to access field values and labels)
- correlation variables (allows to access field values and transformations)

Example: If source results contain a field called “employee”, the value of the field can be accessed with:

- A field variable ${\_\_data.fields.employee}
- A correlation variable that maps the field value above to ${employee}

In addition to mapping field values to shorter variable names, more correlation variables can be created by applying transformations to existing fields

For more details, please see the example in [Use variables and transformations in a correlation](../use-variables-and-transformations/).

Correlation creates a data link only if all variables have values in the selected data row. [Built-in variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/built-in-variables) are exception to this rule and are not required to be filled in from the returned data. These variables are interpolated automatically by data sources.

### Correlation Transformations

Correlations provide a way to extract more variables out of field values. The output of transformations is a set of new variables that can be accessed as any other variable.

There are two types of transformations: logfmt and regular expression.

Each transformation uses a selected field value as the input. The output of a transformation is a set of new variables based on the type and options of the transformation.

For more details, please see the example in [Use variables and transformations in a correlation](../use-variables-and-transformations/) for more details.

### Logfmt transformation

The logfmt transformation deconstructs a field value containing text formatted with [logfmt key/value pairs](https://brandur.org/logfmt). Each pair becomes a variable with the key being the name of the variable.

The logfmt transformation only requires specifying the input field name if you would like the transformation to apply to a different field than the results field.
Example output variables for field = “host=srv001 endpoint=/test app=foo”:

| name     | value  |
| :------- | :----- |
| host     | srv001 |
| endpoint | /test  |
| app      | foo    |

### Regular expression transformation

The regular expression transformation deconstructs a field value based on the provided regular expression.

Regular expression transformation options:

**field**
: Input field name

**expression**
: Regular expression. Named capture groups are mapped to variables matching the group name. If non-named matching groups are used a variable is created out of the first match. The value overrides the variable matching the input field or a new variable is created if mapValue is provided (check the example below)

**mapValue**
: Used with simple regex groups without named matching groups. By default, the first match overrides the variable with the name of the field that is used as the input. To change that default behavior you can specify the mapValue property. The provided name is used to create a new variable. This can be useful if your target query requires both the exact value and a part of the value extracted with the transformation.

Example: Assuming the selected field name is “employee” and the field value is “John Doe”.

Various output variables based on expression and mapValue options:

| expression        | mapValue | output variables             | comment                                                                                           |
| :---------------- | :------- | :--------------------------- | :------------------------------------------------------------------------------------------------ |
| /\\w+ (\\w+)/     | -        | employee=Doe                 | No mapValue provided. The first matching is mapped to the existing field name variable (employee) |
| /(\\w+) (\\w+)/   | name     | name=John                    | The first matching is mapped to a new variable called “name”                                      |
| /(?\\w+) (?\\w+)/ | -        | firstName=John, lastName=Doe | When named groups are used they are the names of the output variables and mapValue is ignored.    |
| /(?\\w+) (?\\w+)/ | name     | firstName=John, lastName=Doe | Same as above                                                                                     |

# Permissions

Users with [Viewer base role](../../roles-and-permissions/) or with [datasources:query RBAC role](../../roles-and-permissions/access-control/) can:

- Use correlations in Explore’s visualizations.
- List all available correlations in read-only mode.

Users with [Admin base role](../../roles-and-permissions/) or with [datasources:write RBAC role](../../roles-and-permissions/access-control/) can:

- Add, edit and delete correlations.

# Create a new correlation

## Before you begin

Make sure you have permission to add new correlations. Only users with write permissions to data sources can define new correlations.

## Create a correlation in Explore's correlations editor

To learn more, refer to Explore's [documentation](../../../explore/correlations-editor-in-explore/) about building correlations in Explore.

## Create a correlation in Administration page

1. Go to the **Administration** section in Grafana.
1. Under **Plugins and data**, open the **Correlations** page.
1. Click the “Add” button in the top right corner.
1. Provide a **label** for the correlation.
1. Provide an optional **description**.
1. Go to the next page.
1. Provide **target data source**.
1. Provide **target query** using variables.
1. Go to the next page.
1. Provide **source data source**.
1. Provide **results field**.
1. Add transformations if you need variables that are not fields in the source data source.
1. Click “Add” to add a new transformation.
1. Select the type of a transformation.
1. Configure transformation depending on the selected type.
1. Save correlation.

You can edit correlation in the same way, but you cannot change the selected data sources.

## Create a correlation with provisioning

Provision correlations by extending provisioned data sources. Correlations are defined as a subsection of the source data source configuration:

```yaml
datasources:
  - name: Data source name # source data source
    ...
    jsonData:
    ...
    correlations:
      - targetUID: uid
        label: "test"
        description: "..."
        type: query
        config:
          target:
            expr: "..."
          field: "name"
          transformations:
            - type: regex
              field: "test"
              expression: /\w+/
              mapValue: "other"
            - type: logfmt
              field: "test"
      - targetUID: uid2
        label: "test 2"
        description: "..."
        type: external
        config:
          target:
            url: "http://${example}"
          field: "name"
```

Description of provisioning properties:

**targetUID**
: Target data source UID

**label**
: Link label

**description**
: Optional description

**type**
: Correlation type. Valid values are "query" for linking to a data source query and "external" for linking to an external URL.

**config**
: Config object

**config.target**
: [Target query model](#determine-target-query-model-structure)

**config.field**
: Name of the field where link is shown

**config.transformations (list)**
: List of transformation objects

**transformation.type**
: regex, or logfmt

**transformation.field**
: The field that will be transformed. If this is not defined, it will apply the transformation to the data from the correlation's config.field.

**transformation.expression**
: Regex expression (regex transformation only)

**transformation.mapValue**
: New name of the variable from the first regex match (regex transformation only)

### Determine target query model structure

When you set up a correlation with admin page you can use the target query editor. When you use provisioning you may need to know the structure of the target query which may not be well documented depending on the plugin. Here is a quick step-by-step guide on how to determine the target query model:

1. Open Explore.
1. Select the data source you want to use as the target of the correlation.
1. Open the inspector tab and select “Query”.
1. Run a sample query.
1. Inspect results.
1. Look for the “queries” list object. Each object is created using the query model structure defined by the data source. You can use the same structure in your provisioning file.

   {{< figure src="/static/img/docs/correlations/determine-target-query-structure-inspector-10-0.png" max-width="600px" caption="Query inspector with target query structure" >}}

   The query model in this example is represented by the first entry in the queries list. Properties “refId” and “datasource” are added to all queries in runtime and can be omitted:

   ```json
   {
     "scenario_id": "random_walk",
     "alias": "app",
     "seriesCount: 2
   }
   ```

# Use correlations in visualizations

## Correlations in Logs Panel

1. Setup a correlation.
1. Open Explore.
1. Select a data source that you chose as the source data source of the correlation.
1. Run a query that results in data containing fields required to build variables in the target query.
1. Expand log row details.
1. If the selected row contains all the information required to build the target query a link appears in the “Links” section at the bottom.
1. Additional information about used variables and their values is shown next to each link.

   {{< figure src="/static/img/docs/correlations/correlations-in-logs-panel-10-0.png" max-width="600px" caption="Correlation links in Logs panel" >}}

## Correlations in Table

1. Setup a correlation.
1. Open Explore.
1. Select a data source that you chose as the source data source of the correlation.
1. Run a query that results in data containing fields required to build variables in the target query.
1. Links are added to cell rows in the column representing the field with the assigned link ([the results field](../correlation-configuration/#source-data-source-and-result-field).
1. Cells containing multiple links accessible with a context menu.

   {{< figure src="/static/img/docs/correlations/correlations-in-table-10-0.png" max-width="600px" caption="Correlations links in table" >}}

# Use variables and transformations in a correlation

## Before you begin

This example walks through creating a link in a test data source but the same principles apply to any data source.

The example emulates a scenario with two data sources:

- Logs containing lines in the format: “2020-01-01 10:00 level=error message=error service=app1.loginService” stored in a field named “msg”
- Metrics for application included in the service name of a log line (e.g. app1)

Instructions below show how to set up a link that can run metrics query for the host included in each log line with provisioning and regex transformation. Additionally, a link with a query containing the full name of the service is set up to demonstrate the logfmt transformation.

## Use variables and transformations in provisioning

1. Add the following provisioning configuration to your Grafana:

   ```yaml
   datasources:
     - name: Target
       uid: test-target
       type: testdata

     - name: Source
       uid: test-source
       type: testdata
     - name: Source
       uid: test-source
       type: testdata
       correlations:
         - targetUID: test-target
           label: App metrics
           description: Application HTTP request metrics
           config:
             type: query
             target:
               scenario_id: random_walk
               alias: $${application}
             field: msg
             transformations:
               - type: regex
                 field: msg
                 expression: service=(\w+)\.\w+
                 mapValue: application
         - targetUID: test-target
           label: Service metrics
           description: Service metrics
           config:
             type: query
             target:
               scenario_id: random_walk
               alias: $${service}
             field: msg
             transformations:
               - type: logfmt
                 field: msg
   ```

   Two data sources are created: Source (emulating logs data source) and Target (emulating metrics data source):
   - A correlation called “App metrics” is created targeting the Target data source with its UID.
     - The label and description are provided as text
     - Each correlation contains the following configuration:
       - Required correlation type (query)
       - Target query matching test data source model
     - “App metrics” correlation contains the following configuration:
       - Alias is set to ${application} variable (note that in provisioning files $ is used to access environment variables so it has to be [escaped](../../provisioning/#using-environment-variables)).
       - Regular expression transformation is created to extract values from “msg” field
         - Regular expression transformation is used to capture the application name from the full name of the service stored in the log line.
         - The output of the transformation is mapped to a variable called “application”.
     - “Service metrics” correlation is created in a similar way but with logfmt transformation to break down log message and access full name of the service (e.g. “app1.loginService”).
       - For example, when a logline “2020-01-01 10:00 level=error message=error service=app1.loginService” is provided as the input, the transformation produces new variables: level, message, and service.
       - “service” variable is used as the alias in the target query.

1. Navigate to Explore and open “Source” data source.
1. Select the “Raw Frames” scenario and provide the following data frames to emulate returning log lines:
   ```json
   [
     {
       "meta": {
         "preferredVisualisationType": "logs"
       },
       "fields": [
         {
           "name": "time",
           "values": [1, 2]
         },
         {
           "name": "msg",
           "values": [
             "level=error msg=error service=app1.loginService",
             "level=debug msg=info service=app2.userProfileService"
           ]
         }
       ]
     }
   ]
   ```
1. Run the query and open log details by clicking on the log line.

   {{< figure src="/static/img/docs/correlations/correlations-in-logs-panel-10-0.png" max-width="600px" caption="Correlation links in Logs panel" >}}

   A link “App metrics” and “Service metrics” show next to variables extracted out of the log line with transformations

1. Click on the “App metrics” link.
1. A split view is opened and the target query is run.
1. Notice how the application name from the log line is filled in as the alias property in the target query.

   {{< figure src="/static/img/docs/correlations/interpolated-target-query-10-0.png" max-width="600px" caption="Interpolated target query" >}}

   This allows you to run a specific query based on the source results:

   {{< figure src="/static/img/docs/correlations/target-query-results-10-0.png" max-width="600px" caption="Interpolated target query results" >}}

1. Go back to the source query and change raw frames’ preferred visualization type to “table” to see how links are displayed in a Table visualization.

   ```json
   [
     {
       "meta": {
         "preferredVisualisationType": "table"
       },
       "fields": [...]
     }
   ]
   ```

1. Run the query and notice how links are created in the Table cell:

   {{< figure src="/static/img/docs/correlations/correlations-in-table-10-0.png" max-width="600px" caption="Correlations links in table" >}}

# Configure LBAC for a Loki data source

This document explains how to configure Label-Based Access Control (LBAC) for data sources for a Loki data source. LBAC for data sources filters the logs that a team can query based on labels. You can configure it on both Grafana Cloud and Grafana Enterprise.

## Grafana Cloud

LBAC for data sources is generally available on Grafana Cloud for Loki data sources created with basic authentication. You create a new data source as described in [Configure a new Loki data source](#configure-a-new-loki-data-source-on-grafana-cloud). Provisioning is currently not available.

You can't configure LBAC rules for Grafana-provisioned data sources from the UI. We recommend that you replicate the settings of the provisioned data source in a new data source, and then add the LBAC configuration to the new data source.

### Before you begin

Before you configure LBAC for data sources, ensure you have the following:

- Permission to create a Loki tenant in Grafana Cloud.
- Admin data source permissions for Grafana.

Grafana recommends that you remove all permissions for roles and teams that don't require access to the data source. This ensures that only the required teams access the data source. The recommended permissions are `Admin` permission for administrators, and `Query` permission only for the teams that you want to add LBAC for data sources rules for.

### Configure a new Loki data source on Grafana Cloud

1. Access Loki data source details for your stack in Grafana Cloud.
1. Copy Loki details and create a CAP.
   - Copy the details of your Loki setup.
   - Create a Cloud Access Policy (CAP) for the Loki data source in Grafana Cloud.
   - Ensure the CAP includes `logs:read` permissions.
   - Ensure the CAP doesn't include `labels` rules.
1. Create a new Loki data source.
   - In Grafana, add a new data source and select Loki as the type.
1. Navigate back to the Loki data source.
   - Set up the Loki data source using basic authentication. Use the `userID` as the username. Use the generated CAP `token` as the password.
   - Select **Save & test**.
1. Navigate to data source permissions.
   - Go to the **Permissions** tab of the newly created Loki data source. Here, you find the LBAC for data sources rules section.

For more information on how to set up LBAC for data sources rules for a Loki data source, refer to [Create LBAC for data sources rules for the Loki data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/create-teamlbac-rules/).

## Grafana Enterprise

LBAC for data sources is available in Grafana Enterprise for Loki connected to Grafana Enterprise Logs (GEL), created with basic authentication.

You can't configure LBAC rules for Grafana-provisioned data sources from the UI. Alternatively, you can replicate the settings of the provisioned data source in a new data source, and then add the LBAC configuration to the new data source.

### Before you begin

Before you configure LBAC for data sources, ensure you have the following:

- Permission to create a cluster in your Grafana.
- Admin plugins permissions for Grafana.
- Admin data source permissions for Grafana.

Grafana recommends that you remove all permissions for roles and teams that don't require access to the data source. This ensures that only the required teams access the data source. The recommended permissions are `Admin` permission for administrators, and `Query` permission only for the teams that you want to add LBAC for data sources rules for.

### Set up a Grafana Enterprise Logs tenant and access policies

1. Access the plugins page and install the Grafana Enterprise Logs plugin.
1. Connect your plugin and use the app as the cluster.
1. Access the Grafana Enterprise Logs app and configure a tenant.
1. Store the `uid` of the tenant to use as the username for basic authentication.
1. Access the policies page inside the app and create an access policy.
   - Create an access policy for the Loki data source.
   - Ensure the access policy includes `logs:read` permissions.
   - Ensure the access policy doesn't include `labels` rules.
   - Store the `token` to use as the password for authentication.

### Configure a new Loki data source on Grafana Enterprise

1. Create a new Loki data source.
   - In Grafana, add a new data source and select Loki as the type.
1. Navigate back to the Loki data source.
   - Set up the Loki data source using basic authentication. Use the `uid` as the username. Use the generated `token` as the password.
   - Select **Save & test**.
1. Navigate to data source permissions.
   - Go to the **Permissions** tab of the newly created Loki data source. Here, you find the LBAC for data sources rules section.

For more information on how to set up LBAC for data sources rules for a Loki data source, refer to [Create LBAC for data sources rules for the Loki data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/create-teamlbac-rules/).

## Examples of LBAC rules

An LBAC rule is a `logql` query that filters logs based on labels. Each rule operates independently as its own filter, separate from other rules within a team. The following examples show common patterns. For more detail, refer to [Create LBAC for data sources rules for the Loki data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/create-teamlbac-rules/).

### Single rule

Multiple label selectors in a single rule are combined with `AND`. The following rule grants access to log lines that match both `namespace="dev"` and `cluster="us-west-0"`:

```logql
{namespace="dev", cluster="us-west-0"}
```

### Multiple rules

Multiple rules assigned to the same team are combined with `OR`. The following two rules grant access to log lines that match `namespace="dev"` or `cluster="us-west-0"`:

```logql
{namespace="dev"}
{cluster="us-west-0"}
```

### Match multiple values with a regular expression

Use the `=~` operator to match a label against a regular expression. The following rule grants access to log lines in the `dev` or `prod` namespace:

```logql
{namespace=~"dev|prod"}
```

### Exclude a label value

Use the `!=` operator to exclude log lines that carry a specific label value. The following rule grants access to all log lines except those labeled `secret="true"`:

```logql
{secret!="true"}
```

### User on multiple teams

Users on multiple teams receive access based on all the combined rules assigned to each team, combined with `OR`. For example, if Team A has `{namespace="dev"}` and Team B has `{namespace="prod"}`, a user on both teams can access log lines that match `namespace="dev"` or `namespace="prod"`.

## Related resources

- [LBAC for data sources overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/) explains how LBAC works, which data sources are supported, and current limitations.
- [Create LBAC for data sources rules for a supported data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/create-teamlbac-rules/) explains how to define and manage rules.

# Configure LBAC for a Prometheus data source

This document explains how to configure Label-Based Access Control (LBAC) for data sources for a Prometheus data source. LBAC for data sources filters the metrics that a team can query based on labels. You can configure it on both Grafana Cloud and Grafana Enterprise.

## Grafana Cloud

LBAC for data sources is generally available on Grafana Cloud for Prometheus data sources created with basic authentication. You create a new data source as described in [Configure a new Prometheus data source](#configure-a-new-prometheus-data-source-on-grafana-cloud). Provisioning is currently not available.

You can't configure LBAC rules for Grafana-provisioned data sources from the UI. We recommend that you replicate the settings of the provisioned data source in a new data source, and then add the LBAC configuration to the new data source.

### Before you begin

Before you configure LBAC for data sources, ensure you have the following:

- Permission to create a Prometheus tenant in Grafana Cloud.
- Admin data source permissions for Grafana.

Grafana recommends that you remove all permissions for roles and teams that don't require access to the data source. This ensures that only the required teams access the data source. The recommended permissions are `Admin` permission for administrators, and `Query` permission only for the teams that you want to add LBAC for data sources rules for.

### Configure a new Prometheus data source on Grafana Cloud

1. Access Prometheus data source details for your stack in Grafana Cloud.
1. Copy Prometheus details and create a CAP.
   - Copy the details of your Prometheus setup.
   - Create a Cloud Access Policy (CAP) for the Prometheus data source in Grafana Cloud.
   - Ensure the CAP includes `metrics:read` permissions.
   - Ensure the CAP doesn't include `labels` rules.
1. Create a new Prometheus data source.
   - In Grafana, add a new data source and select Prometheus as the type.
1. Navigate back to the Prometheus data source.
   - Set up the Prometheus data source using basic authentication. Use the `userID` as the username. Use the generated CAP `token` as the password.
   - Select **Save & test**.
1. Navigate to data source permissions.
   - Go to the **Permissions** tab of the newly created Prometheus data source. Here, you find the LBAC for data sources rules section.

For more information on how to set up LBAC for data sources rules for a Prometheus data source, refer to [Create LBAC for data sources rules for the Prometheus data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/create-teamlbac-rules/).

## Grafana Enterprise

LBAC for data sources is available in Grafana Enterprise for Prometheus connected to Grafana Enterprise Metrics (GEM), created with basic authentication.

You can't configure LBAC rules for Grafana-provisioned data sources from the UI. Alternatively, you can replicate the settings of the provisioned data source in a new data source, and then add the LBAC configuration to the new data source.

### Before you begin

Before you configure LBAC for data sources, ensure you have the following:

- Permission to create a cluster in your Grafana.
- Admin plugins permissions for Grafana.
- Admin data source permissions for Grafana.

Grafana recommends that you remove all permissions for roles and teams that don't require access to the data source. This ensures that only the required teams access the data source. The recommended permissions are `Admin` permission for administrators, and `Query` permission only for the teams that you want to add LBAC for data sources rules for.

### Set up a Grafana Enterprise Metrics tenant and access policies

1. Access the plugins page and install the Grafana Enterprise Metrics plugin.
1. Connect your plugin and use the app as the cluster.
1. Access the Grafana Enterprise Metrics app and configure a tenant.
1. Store the `uid` of the tenant to use as the username for basic authentication.
1. Access the policies page inside the app and create an access policy.
   - Create an access policy for the Prometheus data source.
   - Ensure the access policy includes `metrics:read` permissions.
   - Ensure the access policy doesn't include `labels` rules.
   - Store the `token` to use as the password for authentication.

### Configure a new Prometheus data source on Grafana Enterprise

1. Create a new Prometheus data source.
   - In Grafana, add a new data source and select Prometheus as the type.
1. Navigate back to the Prometheus data source.
   - Set up the Prometheus data source using basic authentication. Use the `uid` as the username. Use the generated `token` as the password.
   - Select **Save & test**.
1. Navigate to data source permissions.
   - Go to the **Permissions** tab of the newly created Prometheus data source. Here, you find the LBAC for data sources rules section.

For more information on how to set up LBAC for data sources rules for a Prometheus data source, refer to [Create LBAC for data sources rules for the Prometheus data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/create-teamlbac-rules/).

## Examples of LBAC rules

An LBAC rule is a `logql` query that filters metrics based on labels. Each rule operates independently as its own filter, separate from other rules within a team. The following examples show common patterns. For more detail, refer to [Create LBAC for data sources rules for the Prometheus data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/create-teamlbac-rules/).

### Single rule

Multiple label selectors in a single rule are combined with `AND`. The following rule grants access to metrics that match both `job="api-server"` and `region="europe"`:

```logql
{job="api-server", region="europe"}
```

### Multiple rules

Multiple rules assigned to the same team are combined with `OR`. The following two rules grant access to metrics that match `job="api-server"` or `region="europe"`:

```logql
{job="api-server"}
{region="europe"}
```

### Match multiple values with a regular expression

Use the `=~` operator to match a label against a regular expression. The following rule grants access to metrics in the `europe` or `us-east` region:

```logql
{region=~"europe|us-east"}
```

### Exclude a label value

Use the `!=` operator to exclude metrics that carry a specific label value. The following rule grants access to all metrics except those labeled `cluster="restricted"`:

```logql
{cluster!="restricted"}
```

### User on multiple teams

Users on multiple teams receive access based on all the combined rules assigned to each team, combined with `OR`. For example, if Team A has `{region="europe"}` and Team B has `{region="us-east"}`, a user on both teams can access metrics that match `region="europe"` or `region="us-east"`.

## Related resources

- [LBAC for data sources overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/) explains how LBAC works, which data sources are supported, and current limitations.
- [Create LBAC for data sources rules for a supported data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/create-teamlbac-rules/) explains how to define and manage rules.

# Configure LBAC for Tempo or Cloud Traces

{{< docs/public-preview product="Label-based access control for traces" >}}

This document explains how to configure Label-Based Access Control (LBAC) for data sources for a Tempo or Cloud Traces data source. LBAC for data sources lets you restrict which spans a team can access by defining rules based on trace attributes, providing fine-grained, team-based access control within a single tenant. It mirrors the experience used for logs and metrics LBAC in Grafana Cloud.

Grafana uses LBAC as an umbrella term for label-based access control across all data sources. Traces use **attributes**, not labels, for access control, but the Grafana UI surfaces this functionality as LBAC for consistency.

{{< admonition type="note" >}}
LBAC for traces is available only on Grafana Cloud. It isn't available in Grafana Enterprise. It applies only to the Cloud-provisioned Tempo or Cloud Traces data source that Grafana hosts, not to self-managed data sources.

Unlike logs and metrics, which also support data source-level LBAC through cloud access policies, traces LBAC is available only at the team level.
{{< /admonition >}}

## How LBAC works

When a user queries a tracing data source, Grafana evaluates the user's team memberships and the LBAC rules assigned to those teams. Grafana adds these rules to the request, so the Cloud Traces data source returns only permitted spans or attributes.

LBAC rules use attribute selectors, such as:

```traceql
{ resource.service.name="checkout", resource.env="prod" }
```

Multiple conditions in the same rule are combined with `AND` (`,`), while multiple rules across teams are combined with `OR`.

If none of a user's teams have LBAC rules for the data source, that user can query all traces, subject to their data source permissions.

## Before you begin

Before you configure LBAC for data sources, ensure you have the following:

- Permission to create a Tempo or Cloud Traces tenant in Grafana Cloud.
- Administrator permissions for Grafana.
- A team set up in Grafana.

## Known limitations

Be aware of the following limitations for traces LBAC:

- LBAC for traces is available only at the team level. Data source-level LBAC rules, configured through cloud access policies, aren't currently supported for traces. This differs from logs and metrics LBAC, which supports both team-level and data source-level rules.
- Autocomplete in search is still under development.
- Rules are restricted to resource scope attributes.
- Users with multiple rules may experience a slight performance degradation.

## Configure a new Tempo or Cloud Traces data source

Follow this workflow when adding a new data source. The data source must be hosted by Grafana and not self-managed.

1. Start your Grafana Cloud instance.
1. Access Tempo or Cloud Traces data source details for your stack.
1. Copy the [details of your Tempo or Cloud Traces setup](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/locate-url-user-password/).
1. In Grafana Cloud, navigate to **Administration** > **Users and access** > **Cloud Access Policies**.
   - Create an access policy for the Tempo or Cloud Traces data source.
   - Ensure the access policy includes `traces:read` permissions.
   - Ensure the access policy doesn't include `labels` rules. Data source-level LBAC isn't supported for traces, so any `labels` rules in the access policy are ignored.
1. In Grafana, select the Tempo or Cloud Traces data source, or create a new one.
1. Navigate back to the Tempo or Cloud Traces data source.
   - Set up the data source using basic authentication. Use the [userID or tenantID](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/locate-url-user-password/) as the username. Use the token from your access policy as the password.
   - Select **Save & test**.
1. Go to the **Permissions** tab of the newly created data source. Here, you find the LBAC for data sources rules section.
1. Choose a team from the **Team** drop-down.
1. Enter attribute selectors, such as:

   ```traceql
   { resource.service.name="checkout", resource.env="prod" }
   ```

1. Select **Save**.

## Examples of LBAC rules

An LBAC rule for traces is an attribute selector that filters spans based on trace attributes. Each rule operates independently as its own filter, separate from other rules within a team.

Follow these guidelines when you write rules:

- Use only resource scope attributes, for example `{ resource.env="prod" }`.
- Use double quotes for string values. Only string values are supported.
- Use the `=~` operator for regular expression matching, for example `{ resource.team =~ "team-a|team-b" }`.
- If you use negation (`!=` or `!~`), refer to [Troubleshoot missing service traces](#troubleshoot-missing-service-traces). TraceQL negation behaves differently from PromQL when an attribute is missing.
- Use up to two conditions in a single rule, separated by a comma (`,`), which acts as an `AND` operator.

For more detail, refer to [Create LBAC for data sources rules for a supported data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/create-teamlbac-rules/).

### Single rule

Multiple conditions in a single rule are combined with `AND`. The following rule grants access to spans that match both `resource.service.name="payments-api"` and `resource.env="prod"`:

```traceql
{ resource.service.name="payments-api", resource.env="prod" }
```

### Match multiple values with a regular expression

Use the `=~` operator to match an attribute against a regular expression. The following rule grants access to spans from `team-a` or `team-b`:

```traceql
{ resource.team =~ "team-a|team-b" }
```

### Multiple rules

Multiple rules assigned to the same team are combined with `OR`. The following two rules grant access to spans in the `prod` environment or spans from the `billing` team:

```traceql
{ resource.env="prod" }
{ resource.team="billing" }
```

### User on multiple teams

Users on multiple teams receive access based on all the combined rules assigned to each team, combined with `OR`. For example, if the teams have the following rules:

- Team A: `{ resource.cluster="us-east-1" }`
- Team B: `{ resource.service.name="frontend" }`

A user on both teams can access spans that match `resource.cluster="us-east-1"` or `resource.service.name="frontend"`.

## How LBAC affects returned data

How LBAC filters data depends on the endpoint:

- **Trace-by-ID lookups** use one of three redaction modes, described in the following sections, that control how non-matching spans are handled.
- **Search, metrics, and autocomplete endpoints** always return only the spans that match the LBAC rules, regardless of the configured redaction mode.

Cloud Traces supports three redaction modes for trace-by-ID lookups. The active mode is configured per tenant. To change the mode for your organization, contact Grafana Support.

| Mode                 | Non-matching spans                           | Best for                                         |
| -------------------- | -------------------------------------------- | ------------------------------------------------ |
| Attributes (default) | Kept, but attributes and intrinsics redacted | Preserving trace structure                       |
| Spans                | Removed entirely                             | Hiding the existence of spans (may break traces) |
| Error                | Entire request returns a `404`               | Strict, all-or-nothing visibility                |

### Attributes mode (default)

Non-matching spans are included in the response, but their attributes and intrinsics are redacted. Only the following minimal fields remain visible on redacted spans:

- `traceId`, `spanId`, `parentId`
- `name`, `kind`
- `timestamps`
- `status`

You can extend the set of always-visible fields by configuring `allowed_attributes`, which sets a list of scoped attributes and intrinsics that are never redacted. For example, you can set `span:name`, `resource.service.name`, `event:name`, or `scope.version` so they're never redacted, even on non-matching spans. Contact Grafana Support to configure `allowed_attributes` for your organization.

### Spans mode

Non-matching spans are removed from the response entirely. This can result in broken traces where a returned span has no matching parent, creating gaps in the span tree.

### Error mode

If any span in a requested trace doesn't match the LBAC policy, the entire request returns a `404` error. This is the strictest mode and suits environments where partial trace visibility isn't acceptable.

## Troubleshoot missing service traces

If a team can't see traces from a service you expect, a negation rule combined with a missing attribute may be excluding those spans.

If you're familiar with PromQL label matching, TraceQL attribute selectors behave differently for negation.

In PromQL, `labelFoo != "abc"` matches series that don't have `labelFoo` or have a value other than `"abc"`.
In TraceQL, `resource.attr != "abc"` and `resource.attr !~ "abc"` match only spans that **have** `resource.attr` and whose value matches the condition.
Spans that don't include that attribute aren't returned.

This difference matters when you combine attributes in one rule with `,` (AND).
For example, this rule matches only spans from `checkout-api` that also have the `resource.k8s.namespace.name` attribute.
Services such as `checkout-api` that don't set `resource.k8s.namespace.name` won't match, even if you want to include them.

```
{ resource.service.name="checkout-api", resource.k8s.namespace.name !~ "prod-main" }
```

To include services that don't set the attribute, add a separate rule for each service.
Multiple rules for a team combine with **OR**:

```
{ resource.service.name="checkout-api", resource.k8s.namespace.name !~ "prod-main" }
{ resource.service.name="checkout-api" }
```

Add similar OR rules for other services that don't set the namespace attribute.

To find services that don't set an attribute, run this query in Grafana Explore:

```
{ resource.k8s.namespace.name = nil } | rate() by (resource.service.name)
```

This query returns services that don't have `resource.k8s.namespace.name` set.

## Manage LBAC rules

You can edit or delete existing LBAC rules from the data source's **Permissions** tab.

### Edit an LBAC rule

1. Open your stack and select your **Tempo** or **Cloud Traces** data source.
1. Select **Permissions**.
1. Scroll to **Data access**.
1. Select the rule you want to edit and select the pencil (edit) icon.
1. Modify the **Attribute filters**.
1. Select **Save**.

### Delete an LBAC rule

1. Open your stack and select your **Tempo** or **Cloud Traces** data source.
1. Select **Permissions**.
1. Scroll to **Data access**.
1. Select the rule you want to delete.
1. Select the **X** (delete) icon.
1. Confirm the deletion.

## Related resources

- [LBAC for data sources overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/) explains how LBAC works, which data sources are supported, and current limitations.
- [Create LBAC for data sources rules for a supported data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/create-teamlbac-rules/) explains how to define and manage rules.
- [Construct a TraceQL query](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/construct-traceql-queries/) explains attribute selector syntax, including `!=`, `!~`, and `= nil`.
- [Search traces using the query builder](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-search/) explains how to run the diagnostic query in Explore.

# Create LBAC for data sources rules

LBAC for data sources works with supported data sources that use basic authentication. You can't add LBAC rules to a file-provisioned data source, and provisioned data sources are read-only in the UI. Create the data source first, then add its LBAC rules on the **Permissions** tab or provision them with the Grafana Terraform provider.

## Before you begin

Before you start, ensure you have:

- **Tenant permissions:** Permission to create the tenant for your data source, such as a Loki tenant, in Grafana Cloud or in Grafana Enterprise Metrics or Grafana Enterprise Logs.
- **Data source permissions:** `Admin` permission on the data source in Grafana.
- **A team:** At least one team set up in Grafana.

## Create an LBAC for data sources rule for a team

To add an LBAC rule to a team:

1. Navigate to your data source.
1. Select the **Permissions** tab. The LBAC for data sources rules section appears on this tab.
1. In the LBAC for data sources rules section, add a new rule for the team.
1. Define a label selector for the rule. For the selector syntax, refer to [LBAC rules](#lbac-rules). For data source-specific guidance, refer to the configuration guides in [Next steps](#next-steps).

## LBAC rules

An LBAC rule is a LogQL query that filters logs or metrics based on labels. Each rule operates independently as its own filter, separate from other rules within a team.

For example:

- **Logs:** `{namespace="dev", cluster="us-west-0"}` returns log lines that match both `namespace="dev"` and `cluster="us-west-0"`.
- **Metrics:** `{job="api-server", region="europe"}` returns metric data points that match both `job="api-server"` and `region="europe"`.

Within a single rule, comma-separated selectors combine with **AND**. For example, the rule `{namespace="dev", cluster="us-west-0"}` matches data where `namespace="dev"` **AND** `cluster="us-west-0"`.

Across a team, multiple rules combine with **OR**. For example, the two rules `{namespace="dev"}` and `{cluster="us-west-0"}` match data where `namespace="dev"` **OR** `cluster="us-west-0"`.

{{< admonition type="note" >}}
An `Admin` user who's a member of one or more teams with LBAC rules can access only the logs or metrics allowed by those teams' rules. An `Admin` user who isn't a member of any team with LBAC rules can access all logs or metrics.
{{< /admonition >}}

### Traces rules

For Tempo and Cloud Traces, rules use attribute selectors rather than LogQL labels. This functionality is in public preview, and traces LBAC is available only at the team level. Use resource-scope attributes, for example:

```
{ resource.service.name="checkout", resource.env="prod" }
```

The same logic applies as for logs and metrics: comma-separated conditions in one rule combine with **AND**, and multiple rules across a team combine with **OR**. For more information, refer to [Configure team LBAC for Tempo or Cloud Traces](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/configure-teamlbac-for-tempo/).

### Best practices

Follow these recommendations when you set up rules:

- Grant `Query` permission only to teams that should use the data source, and grant `Admin` permission only to administrators.
- Give every team that should be restricted its own rule. A team that has `Query` permission but no rule can query all logs or metrics.
- Remove label selectors from the Cloud Access Policy for the data source. Cloud Access Policy rules override LBAC for data sources rules.
- For a first setup, create as few rules as possible for each team and prefer rules that grant access over rules that exclude data, because access is the union of all rules.
- Use a dedicated data source for LBAC, and keep a separate data source without LBAC for full access, so the separation is clear.
- Manage rules as code with the Grafana Terraform provider for repeatable, reviewable setups.
- To validate rules, test them in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/). Explore shows the logs or metrics that a rule returns.

## Examples

The following examples show how different rule and team configurations affect access. Unless stated otherwise, each team has `Query` permission on the data source and administrators have `Admin` permission only.

### One rule for each team

A common use case is to grant a team access to data that has a specific label. In this example, Team A and Team B each have one rule:

- Team A has the rule `namespace="dev"`.
- Team B has the rule `namespace="prod"`.

A user in Team A can access logs or metrics that match `namespace="dev"`. A user in both Team A and Team B can access data that matches `namespace="dev"` **OR** `namespace="prod"`.

### Multiple rules for one team

A team can have more than one rule, and its members can access data that matches any of them. In this example, Team A has two rules:

- Rule 1: `namespace="dev"`
- Rule 2: `namespace="prod"`

A user in Team A can access logs or metrics that match `namespace="dev"` **OR** `namespace="prod"`.

### Exclude a label for a team

You can exclude data that has a specific label. For example, to exclude all log lines labeled `secret="true"`, add the selector `secret!="true"`. In this example, Team A has one rule:

- Team A has the rule `secret!="true"`.

A user in Team A can access logs or metrics that match `secret!="true"`, which is all data except data labeled `secret="true"`.

### Rules with multiple conditions

A single rule can combine multiple conditions with **AND**. In this example, Team A and Team B each have one rule:

- Team A has the rule `cluster="us-west-0", namespace=~"dev|prod"`.
- Team B has the rule `cluster="us-west-0", namespace="staging"`.

The `=~` operator matches a regular expression, so `namespace=~"dev|prod"` matches either `dev` or `prod`.

A user in only Team A can access logs or metrics that match `cluster="us-west-0"` **AND** (`namespace="dev"` **OR** `namespace="prod"`).

A user in only Team B can access logs or metrics that match `cluster="us-west-0"` **AND** `namespace="staging"`.

A user in both Team A and Team B can access logs or metrics that match `cluster="us-west-0"` **AND** (`namespace="dev"`, `namespace="prod"`, or `namespace="staging"`), because rules across teams combine with **OR**.

A user with an `Editor` or `Viewer` role who isn't a member of any team can't query logs or metrics, because in this setup only teams have `Query` permission on the data source.

### Rules that overlap

Two teams can have rules that overlap. In this example:

- Team A has the rule `namespace="dev"`.
- Team B has the rule `namespace!="dev"`.

A user in Team A can access logs or metrics that match `namespace="dev"`.

A user in Team B can access logs or metrics that match `namespace!="dev"`.

{{< admonition type="note" >}}
A user in both Team A and Team B can access all data, because the rules combine as `namespace="dev"` **OR** `namespace!="dev"`.
{{< /admonition >}}

### Team without a rule

When a team has no LBAC rule, its members can access all logs or metrics, subject to their data source permission. In this example, the `Editor` and `Viewer` roles have `Query` permission:

- Team A has the rule `namespace="dev"`.
- Team B has no rule configured.

A user in only Team A can access logs or metrics that match `namespace="dev"`.

A user in both Team A and Team B can access logs or metrics that match `namespace="dev"`.

A user in only Team B with an `Editor` or `Viewer` role can access all logs or metrics, because Team B has no rule and the user has `Query` permission.

### Administrator on a team with a rule

An administrator's basic role doesn't bypass LBAC rules when the administrator is a member of a team that has rules. In this example, User A has an `Admin` basic role and is a member of Team B:

- Team B has no roles assigned.
- Team B has `Query` permission on the data source.
- Team B has the rule `{ project_id="project-dev" }`.

User A can access only logs or metrics that match `{ project_id="project-dev" }`.

## Next steps

Set up LBAC for data sources for your data source:

- [Configure LBAC for data sources for Loki](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/configure-teamlbac-for-loki/)
- [Configure LBAC for data sources for Prometheus](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/configure-teamlbac-for-prometheus/)
- [Configure team LBAC for Tempo or Cloud Traces](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/configure-teamlbac-for-tempo/)

# Activate a Grafana Enterprise license from AWS Marketplace on ECS

If you have purchased a Grafana Enterprise subscription through AWS Marketplace, you must activate it in order to use Grafana Enterprise data source plugins and features in Grafana.

## Before you begin

- Purchase a subscription to [Grafana Enterprise from AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-dlncd4kzt5kx6).
- Be sure that the IAM user that was used to purchase Grafana Enterprise has permission to manage subscriptions, create new IAM users and roles, and create access policies.

To activate your license, complete the following tasks.

## Task 1: Deploy Grafana Enterprise on Amazon ECS

1. Deploy Grafana Enterprise on Amazon ECS.

   For more information about deploying an application on Amazon ECS, refer to [Creating an Amazon ECS service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service.html).

1. As you create the Amazon ECS service, use the intended Grafana Enterprise container image version.

   For example, enter `grafana/grafana-enterprise:11.3.0`.

   > Only Grafana Enterprise versions 8.3.0 and later support licenses granted through AWS Marketplace.

## Task 2: Configure Grafana for high availability

Grafana requires that you configure a database to hold dashboards, users, and other persistent data.

### Before you begin

- Ensure that you have a supported Grafana database available.
  - For a list of supported databases, refer to [Supported databases](../../../../setup-grafana/installation/#supported-databases).
  - For information about creating a database, refer to [Creating an Amazon RDS DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateDBInstance.html).
- Review the information required to connect to the RDS DB instance. For more information, refer to [Connecting to an Amazon RDS DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_CommonTasks.Connect.html).

To configure Grafana for high availability:

1. In AWS ECS, use environment variables to update the `database` parameters.

   For a list of database parameters, refer to [Configuration](../../../../setup-grafana/configure-grafana/#database).

1. Create a revision of the task definition for the ECS Task that runs Grafana Enterprise.

   For more information about creating a task, refer to [Updating a task definition using the classic console](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-task-definition.html).

1. Within the new revision, edit the Grafana Enterprise container for this task, and add the following environment variables to the container:

   ```
   GF_DATABASE_TYPE=[database type]
   GF_DATABASE_HOST=[database address and port]
   GF_DATABASE_NAME=[database name]
   GF_DATABASE_USER=[database username]
   GF_DATABASE_PASSWORD=[database password]
   ```

> For more information about how to update your ECS service with an environment variable, refer to [Updating a service using the new console](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service-console-v2.html).

## Task 3: Configure Grafana Enterprise to validate its license with AWS

In this task you configure Grafana Enterprise to validate the license with AWS instead of Grafana Labs.

1. In AWS IAM, create an access policy with the following permissions:
   - `"license-manager:CheckoutLicense"`
   - `"license-manager:ListReceivedLicenses"`
   - `"license-manager:GetLicenseUsage"`
   - `"license-manager:CheckInLicense"`

   For more information about creating an access policy, refer to [Creating IAM policies (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html).

   For more information about AWS license permissions, refer to [Actions, resources, and condition keys for AWS License Manager](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awslicensemanager.html).

1. Create an Elastic Container Service task role and attach the policy you created in the previous step.

   For more information about creating a task role, refer to [IAM Roles for Tasks](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html).

1. Create a revision of the task definition for the ECS Task that runs Grafana Enterprise.

   For more information about creating a revision of the task definition, refer to [Updating a task definition using the classic console](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-task-definition.html).

1. Within the new revision, perform the following steps:

   a. Update the Task Role of your ECS Task to the role that you created, with permission to access license information.

   b. Edit the Grafana Enterprise container for this task, and add the following environment variable to the container:

   ```
   GF_ENTERPRISE_LICENSE_VALIDATION_TYPE=aws
   ```

   For more information about how to update your ECS service with an environment variable, refer to [Updating a service using the new console](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service-console-v2.html).

### Task 4: Start or restart Grafana

1. To restart Grafana and activate your license, update the service running Grafana to use the latest revision of the task definition that you created.
1. After you update the service, navigate to your Grafana instance, sign in with Grafana Admin credentials, and navigate to **Administration > General > Stats and license** to validate that your license is active.

For more information about validating that your license is active, refer to [Grafana Enterprise license restrictions](../../#grafana-enterprise-license-restrictions).

# Activate a Grafana Enterprise license from AWS Marketplace on EKS

If you have purchased a Grafana Enterprise subscription through AWS Marketplace, you must activate it to use Grafana Enterprise data source plugins and features in Grafana.

## Before you begin:

- Purchase a subscription to [Grafana Enterprise from AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-dlncd4kzt5kx6).
- Be sure that the IAM user that was used to purchase Grafana Enterprise has permission to manage subscriptions, create new IAM users and roles, and create access policies.

To activate your license, complete the following tasks:

## Task 1: Deploy Grafana Enterprise on Amazon EKS

1. Deploy Grafana Enterprise on Amazon EKS.

   For more information about deploying an application on Amazon EKS, refer to [Getting started with Amazon EKS – AWS Management Console and AWS CLI](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html).

   For more information about installing Grafana on Kubernetes using the Helm Chart, refer to the [Grafana Helm Chart](https://github.com/grafana/helm-charts/tree/main/charts/grafana#readme).

1. Use `kubectl set image deployment/my-release grafana=grafana/grafana-enterprise:<version>` to update the container image.

   For example, enter `grafana/grafana-enterprise:11.3.0`.

> Only Grafana Enterprise versions 8.3.0 and later support licenses granted through AWS Marketplace.

## Task 2: Configure Grafana for high availability

Grafana requires that you configure a database to hold dashboards, users, and other persistent data.

### Before you begin

- Ensure that you have a supported Grafana database available.
  - For a list of supported databases, refer to [Supported databases](../../../../setup-grafana/installation/#supported-databases).
  - For information about creating a database, refer to [Creating an Amazon RDS DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateDBInstance.html).
- Review the information required to connect to the RDS DB instance. For more information, refer to [Connecting to an Amazon RDS DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_CommonTasks.Connect.html).

To configure Grafana for high availability, choose **one** of the following options:

- **Option 1:** Use `kubectl edit configmap grafana` to edit `grafana.ini` add the following section to the configuration:

  ```
  [database]
  type = [database type]
  host = [database address and port]
  name = [database name]
  user = [database username]
  password = [database password]
  ```

- **Option 2:** use `kubectl edit deployment my-release` to edit the pod `env` variables and add the following database variables:

  ```
  - name: GF_DATABASE_TYPE
    value: [database type]
  - name: GF_DATABASE_HOST
    value: [database address and port]
  - name: GF_DATABASE_NAME
    value: [database name]
  - name: GF_DATABASE_USER
    value: [database username]
  - name: GF_DATABASE_PASSWORD
    value: [database password]
  ```

For more information on Grafana High Availability setup, refer to [Set up Grafana for high availability](../../../../setup-grafana/set-up-for-high-availability/).

## Task 3: Configure Grafana Enterprise to validate its license with AWS

In this task, you configure Grafana Enterprise to validate the license with AWS instead of Grafana Labs.

1. In AWS IAM, assign the following permissions to the Node IAM role (if you are using a Node Group), or the Pod Execution role (if you are using a Fargate profile):
   - `"license-manager:CheckoutLicense"`
   - `"license-manager:ListReceivedLicenses"`
   - `"license-manager:GetLicenseUsage"`
   - `"license-manager:CheckInLicense"`

   For more information about creating an access policy, refer to [Creating IAM policies (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html).

   For more information about AWS license permissions, refer to [Actions, resources, and condition keys for AWS License Manager](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awslicensemanager.html).

1. Choose **one** of the following options to update the [license_validation_type](../../../../setup-grafana/configure-grafana/enterprise-configuration/#license_validation_type) configuration to `aws`:
   - **Option 1:** Use `kubectl edit configmap grafana` to edit `grafana.ini` add the following section to the configuration:

     ```
     [enterprise]
     license_validation_type=aws
     ```

   - **Option 2:** Use `kubectl edit deployment my-release` to edit the pod `env` variables and add the following variable:

     ```
     name: GF_ENTERPRISE_LICENSE_VALIDATION_TYPE
     value: aws
     ```

### Task 4: Start or restart Grafana

To activate Grafana Enterprise features, you must start (or restart) Grafana.

To restart Grafana on a Kubernetes cluster,

1. Run the command `kubectl rollout restart deployment my-release`.

1. After you update the service, navigate to your Grafana instance, sign in with Grafana Admin credentials, and navigate to **Administration > General > Stats and license** to validate that your license is active.

For more information about restarting Grafana, refer to [Restart Grafana](../../../../setup-grafana/start-restart-grafana/).

> If you experience issues when you update the EKS cluster, refer to [Amazon EKS troubleshooting](https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html).

# Activate a Grafana Enterprise license from AWS on an instance deployed outside of AWS

While AWS Marketplace lists ECS and EKS as the supported environments for Grafana Enterprise, you can apply a Grafana Enterprise license from AWS Marketplace to any Grafana instance with network access to the AWS licensing service.

## Before you begin

- Purchase a subscription to [Grafana Enterprise from AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-dlncd4kzt5kx6).
- Be sure that the IAM user that was used to purchase Grafana Enterprise has permission to manage subscriptions, create new IAM users, and create access policies.
- Be sure there is network access between AWS and the environment where you intend to run Grafana. Network access is required because your Grafana instance communicates with the [AWS License Manager endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/licensemanager.html) to retrieve license and subscription information. Grafana instances with access to the public internet will have access to AWS license manager.

To activate a Grafana Enterprise license from AWS on a Grafana Enterprise instance deployed outside of AWS, complete the following tasks.

## Task 1: Install Grafana Enterprise

To install Grafana, refer to the documentation specific to your implementation.

- [Install Grafana](../../../../setup-grafana/installation/).
- [Run Grafana Docker image](../../../../setup-grafana/installation/docker/).
- [Deploy Grafana on Kubernetes](../../../../setup-grafana/installation/kubernetes/#deploy-grafana-enterprise-on-kubernetes).

## Task 2: Create an AWS IAM user with access to your Grafana Enterprise license

To retrieve your license, Grafana Enterprise requires access to your AWS account and license information. To grant access, create an IAM user in AWS with access to the license, and pass its credentials as environment variables on the host or container where Grafana is running. These environment variables allow Grafana to retrieve license details from AWS.

1. In the AWS License Manager service, create an IAM policy with the following permissions:
   - `"license-manager:CheckoutLicense"`
   - `"license-manager:ListReceivedLicenses"`
   - `"license-manager:GetLicenseUsage"`
   - `"license-manager:CheckInLicense"`

   For more information about creating a policy in AWS, refer to [Creating IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html).

   For more information about AWS Identity and Access Management, refer to [IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html).

1. To limit the policy to obtain usage data just for Grafana Enterprise, in the **Resources** section of the policy, specify your license ARN.

   You can find your license ID in the **Granted Licenses** section of [AWS License Manager](https://console.aws.amazon.com/license-manager/home).

   The policy JSON should look similar to the following example:

   ```
   {
      "Version": "2012-10-17",
      "Statement": [
         {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": "license-manager:GetLicenseUsage",
               "Resource": "arn:aws:license-manager::[YOUR_ACCOUNT]:license:[YOUR_LICENSE_ID]"
         },
         {
               "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": [
                  "license-manager:CheckoutLicense",
                  "license-manager:ListReceivedLicenses",
                  "license-manager:CheckInLicense"
               ],
               "Resource": "*"
         }
      ]
   }
   ```

1. Create an IAM user and choose access key credentials as its authentication method.

   For more information about creating an IAM user, refer to [IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html).

   For more information about access key credentials, refer to [Managing access keys for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).

1. Attach the policy you created to the IAM user.

1. Add the following values as environment variables to the host or container running Grafana:
   - AWS region
   - IAM user's access key ID
   - IAM user's secret access key

   The environment variables should look similar to the following example:

   ```
   AWS_ACCESS_KEY_ID=ABCD5E75FGHIJKTM7
   AWS_SECRET_ACCESS_KEY=k8fhYAQVy+5NhCejhe6HeSjSphjRuy+12C06
   AWS_REGION=us-east-1
   ```

## Task 3: Configure Grafana Enterprise to validate its license with AWS

In this task you configure Grafana Enterprise to validate the license with AWS instead of Grafana Labs.

Choose one of the following options to update the [license_validation_type](../../../../setup-grafana/configure-grafana/enterprise-configuration/#license_validation_type) configuration to `aws`:

- **Option 1:** In the `[enterprise]` section of the grafana.ini configuration file, add `license_validation_type=aws`.

  For example:

  ```
  [enterprise]
  license_validation_type=aws
  ```

- **Option 2:** Add the following environment variable to the container or host:

  ```
  GF_ENTERPRISE_LICENSE_VALIDATION_TYPE=aws
  ```

## Task 4: Start or restart Grafana

To activate Grafana Enterprise features, start (or restart) Grafana.

For information about restarting Grafana, refer to [Restart Grafana](../../../../setup-grafana/start-restart-grafana/).

# Manage your Grafana Enterprise license in AWS Marketplace

You can use AWS Marketplace to make the following modifications to your Grafana Enterprise license:

- Add active users
- Remove active users
- Cancel your subscription to Grafana Enterprise.

**To modify your Grafana Enterprise subscription in AWS Marketplace:**

1. Open the AWS Console and navigate to [Subscription Management](https://console.aws.amazon.com/marketplace/home/subscriptions#/subscriptions).

1. Update your license.

1. Sign in to Grafana as a Server Administrator.

1. Click **Administration** in the side navigation menu, **General**, and then **Stats and license**.

1. In the **Token** section under **Enterprise License**, click **Renew token**.

   This action retrieves updated license information from AWS.

> To learn more about licensing and active users, refer to [Activate a Grafana Enterprise license purchased through AWS Marketplace](../).

# Transfer your AWS Marketplace Grafana Enterprise license

You can transfer your AWS Marketplace Grafana Enterprise license to another Grafana Enterprise instance. The transfer process requires that you first remove your license from one instance, and then apply the license to another instance.

> When you remove an Enterprise license, the system immediately disables all Grafana Enterprise features.

To remove an Enterprise license from a Grafana Enterprise instance, perform one of the following steps:

- If you are using Amazon ECS or Amazon EKS, remove the `GF_ENTERPRISE_LICENSE_VALIDATION_TYPE` environment variable from the container.
- If you have deployed Grafana Enterprise outside of AWS, remove the `aws` license_validation_type value from the grafana.ini configuration file.

It can take the system up to one hour to clear the license. After the system clears the license, you can apply the same license to another Grafana Enterprise instance.

To determine that the system has returned your license, check the license details in AWS License Manager.

# Grafana Cloud Migration Assistant

The Grafana Cloud Migration Assistant automatically migrates resources from your Grafana OSS/Enterprise instance to Grafana Cloud. It provides the following functionality:

- Securely connect your self-managed instance to a Grafana Cloud instance.
- Migrate resources such as dashboards, data sources, and folders to your cloud instance in a few easy steps.
- View the migration status of your resources in real-time.

Some benefits of the migration assistant are:

Ease of use
: Follow the steps provided by the UI to easily migrate all your resources to Grafana Cloud without using Grafana APIs or scripts.

Security
: Encrypt and securely migrate your resources to your connected Grafana Cloud instance.

Speed
: Migrate all of your resources in minutes and accelerate your transition to Grafana Cloud.

## Supported resources

The following resources are supported by the migration assistant:

- Dashboards
- Folders
- Data sources
- App Plugins
- Panel Plugins
- Library Panels
- Grafana Alerting resources

## Before you begin

To use the Grafana migration assistant, you need:

- A self-managed Grafana instance.
- A [Grafana Cloud Stack](https://grafana.com/docs/grafana-cloud/get-started/) you intend to migrate your resources to.
- [`Admin`](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/cloud-roles/) access to the Grafana Cloud Stack. To check your access level, go to `https://grafana.com/orgs/<YOUR-ORG-NAME>/members`.
- [Grafana server administrator](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/#grafana-server-administrators) access to your existing self-managed Grafana instance. To check your access level, go to `https://<GRAFANA-ONPREM-URL>/admin/users`.
- Internet access from your existing self-managed Grafana instance.
- If you're running Grafana in a [highly-available setup](https://grafana.com/docs/grafana/latest/setup-grafana/set-up-for-high-availability/), we recommend scaling Grafana down to one replica to avoid a [known bug](https://github.com/grafana/grafana/issues/107264).
- If your network requires external services to be on an allowlist to allow access, add the following IPs and URLs to your allowlist:
  - [Hosted Grafana](https://grafana.com/docs/grafana-cloud/security-and-account-management/allow-list/#hosted-grafana)
  - [Hosted Alerts](https://grafana.com/docs/grafana-cloud/security-and-account-management/allow-list/#hosted-alerts)
  - [AWS IP address ranges](https://docs.aws.amazon.com/en_us/vpc/latest/userguide/aws-ip-ranges.html) for the S3 service
  - `*.grafana.net`

## Access the migration assistant

In Grafana OSS, access to the migration assistant is limited to the server administrator.

In Grafana Enterprise, the server administrator has access to the migration assistant by default. You can also grant access to other Admins using a role-based access control (RBAC) role that enables other admins on the Grafana instance to view, build snapshots, and upload resources to Grafana Cloud.

### Grant access in Grafana Enterprise

{{< admonition type="note" >}}
You must [configure RBAC](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/configure-rbac/) before you can grant other administrators access to the Grafana Migration Assistant.
{{< /admonition >}}

To grant other Admins access to the migration assistant in Grafana Enterprise:

1. Sign in to Grafana as a server administrator.
1. Navigate to **Home** > **Administration** > **Users and access** > **Users** in the Grafana sidebar.
1. Click an Admin.
1. In the **Organizations** section, click **Change role**.
1. Select **Organization resource migrator** from the role selector menu under **Migration Assistant**.

   ![The Organization resource migrator role in the role picker](/media/docs/grafana-cloud/account-management/screenshot-grant-migration-assistant-access.png)

1. Click **Apply**.

## Use the migration assistant

You can use the migration assistant to generate a migration token on your Grafana Cloud instance, use that token to connect your self-managed Grafana instance to your Grafana Cloud instance, build snapshots of your self-managed Grafana instance, and upload these snapshots to Grafana Cloud.

### Generate a migration token on the destination cloud instance

1. Navigate to **Home** > **Administration** > **General** > **Migrate to Grafana Cloud** in the cloud instance where you intend to migrate your resources.
1. Click the **Generate a migration token** button.
1. Make a copy of the migration token by copying to clipboard. The token is required to authenticate your self-managed instance with the Grafana Cloud Stack.

### Connect your self-managed Grafana instance to the Grafana Cloud Stack

1. On your self-managed Grafana instance, navigate to **Home** > **Administration** > **General** > **Migrate to Grafana Cloud**.

1. Click the **Migrate this instance to Cloud** button.

1. Enter your token in the **Migration token** field and click **Connect to this Stack**.

### Build a snapshot

After connecting to the cloud stack, this is the empty state of the migration assistant. You need to create a snapshot of the self-managed Grafana instance to upload it to the cloud stack.

1. Select the checkbox next to each resource you want to migrate to your cloud stack.

   {{< admonition type="note" >}}
   Some resources can't be uploaded to your cloud stack alone because they rely on other resources:
   | Desired resource | Requires |
   | :---- | :---- |
   | Dashboards | <ul><li>Library Elements</li> <li>Data Sources</li> <li>Plugins</li> <li>Folders</li></ul> |
   | Library Elements | Folders |
   | Data Sources | Plugins |
   | Plugins | Nothing else |
   | Folders | Nothing else |
   | All Alert rule groups | All other resources |
   | Alert Rules | <ul><li>Dashboards</li> <li>Library Elements</li> <li>Data Sources</li> <li>Plugins</li> <li>Folders</li> <li>Notification Policies</li> <li>Notification Templates</li> <li>Contact Points</li> <li>Mute Timings</li></ul> |
   | Notification Policies | <ul><li>Notification Templates</li> <li>Contact Points</li> <li>Mute Timings</li></ul> |
   | Notification Templates | Nothing else |
   | Contact Points | Notification Templates |
   | Mute Timings | Nothing else |
   {{< /admonition >}}

1. Click **Build snapshot**

   ![A list of resources selected for migration and the Build snapshot button](/media/docs/grafana/screenshot-grafana-12-select-resources.png)

### Upload resources to the cloud

After a snapshot is created, a list of resources appears with resource Type and Status populated with **Not yet uploaded**.

1. Click **Upload snapshot** to copy the resources to the Grafana Cloud instance.

1. Use the assistant's real-time progress tracking to monitor the migration. The status changes to 'Uploaded to cloud' for resources successfully copied to the cloud.

   You can group and sort resources during and after the migration:
   - Click **Name** to sort resources alphabetically.
   - Click **Type** to group and sort by resource type.
   - Click **Status** to group and sort by upload status (pending upload, uploaded successfully, or experienced errors).

   The Snapshot information also updates to inform the user of total resources, errors, and total number of successfully migrated resources.

   ![An updates list of resources with snapshots built after attempting to upload them to Grafana Cloud](/media/docs/grafana/screenshot-grafana-12-updated-snapshot-page.png)

1. Review error details for any issues that need manual resolution.

## Snapshots created by the migration assistant

The migration assistant currently supports a subset of all resources available in Grafana. Refer to [Supported Resources](#supported-resources) for more details.

When you create a snapshot, the migration assistant makes a copy of all the resources you select and saves them in the snapshot. The snapshot reflects the current state of the resources when the snapshot is built and is stored locally on your instance, ready to be uploaded in the last stage.

Resources saved in the snapshot are strictly limited to the resources stored within an organization. This is important to note if there are multiple organizations used in your Grafana instance. If you want to migrate multiple organizations, refer to [Migrate multiple organizations](#migrate-multiple-organizations) for more information and guidance.

## Resource migration details

During a migration, resource UIDs are preserved, allowing you to correlate your local and cloud resources. If you perform the same migration multiple times, resources in your Grafana Cloud stack that were previously migrated are updated. The assistant never modifies your self-managed resources or cloud resources that didn't come from a snapshot.

### Dashboards and folders

Dashboard names and UIDs are preserved along with references to data sources. Folder hierarchy is also preserved, so you can find your dashboards and other resources saved in identical folder locations.

### Data sources

Your data sources, including credentials, are migrated securely and seamlessly to your Grafana Cloud instance, so you don't need to find and enter all your data source credentials again.

### Plugins

The migration assistant supports any plugins found in the plugins catalog. As long as the plugin is signed or is a core plugin built into Grafana, you can migrate it. Due to security reasons, Grafana Cloud doesn't support unsigned plugins. If you're using any unsigned private plugins, seek an alternative plugin from the catalog or work on a strategy to deprecate certain functionality from your self-managed instance.

Upgrade any plugins you intend to migrate before using the migration assistant as any migrated plugins will be configured on the Grafana Cloud instance as the latest version of that plugin.

{{< admonition type="caution">}}
If you want to migrate Enterprise plugins, check what type of plan your Grafana Cloud instance is on and whether this plan requires an Enterprise plugin add-on.
{{< /admonition >}}

### Grafana Alerting resources

The migration assistant can migrate the majority of Grafana Alerting resources to your Grafana Cloud instance. These include:

- Alert rules
- Notifications
- Contact points
- Mute timings
- Notification policy tree
- Notification templates

This is sufficient to have your Alerting configuration up and running in Grafana Cloud with minimal effort.

#### Migration assistant limitations on Grafana Alerting resources

The migration assistant doesn't support migration of Silences, so you need to configure them manually. Alert History also isn't available for migration.

Attempting to migrate a large number of alert rules might result in the following error:

```
Maximum number of alert rule groups reached: Delete some alert rule groups or upgrade your plan and try again.
```

To avoid this, refer to the [Alert rule limits in Grafana Cloud](https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-grafana-managed-rule/#alert-rule-limits-in-grafana-cloud) when migrating alert rules.

#### Prevent duplicated alert notifications

Successfully migrating Alerting resources to your Grafana Cloud instance can result in two sets of notifications:

1. From your OSS/Enterprise instance

1. From the newly migrated alerts in your Grafana Cloud instance

To avoid double notifications, a new `alert_rules_state` configuration option in the `custom.ini` or `grafana.ini` file controls how Alert Rules are migrated to the Grafana Cloud instance and is set to `paused` by default so you can review and test your Alerting resources in your Grafana Cloud instance without duplicate notifications.

The available options for `alert_rule_state` are:

`paused`
: Creates all Alert rules in paused state on the Cloud instance. This is helpful to avoid double notifications.

`unchanged`
: The Alert rules maintain their original state coming from the source instance.

When you are ready to start using your alert rules and notifications from your Grafana Cloud instance, run the migration again with `alert_rules_state = unchanged`.

### Resource permissions

Because the migration assistant doesn't yet migrate teams or RBAC permissions, your resources migrate with default permissions. Reconfigure permissions in your cloud stack as needed after a migration. For more information, refer to [Grafana Cloud user roles and permissions](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/cloud-roles/).

## Migrate multiple organizations

If you're using the [organizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/organization-management/#about-organizations) feature on your Grafana Instance and intend to migrate to Grafana Cloud, you need to plan this aspect of the migration carefully.

Grafana Cloud doesn't support the organizations feature, but you can use folders and RBAC to protect and grant permissions to resources instead. The recommended path is to migrate multiple organizations to a single cloud stack. This is the simplest option and provides the best user experience.

The migration assistant creates and uploads snapshots based on the resources within a specific organization. There's no option to migrate an entire Grafana instance with multiple organizations at once. You need to run the migration process for each organization you want to migrate.

The Grafana server administrator has access to the migration assistant by default. The server administrator can perform the migration by switching organizations and running the migration assistant each time. The Grafana server administrator can also grant access to the migration assistant to organization administrators who are members using the RBAC **Migration Assistant:Organization resource migrator** role. This allows those organization administrators to run the migration process for their respective organizations.

### Access Control and managing resources in the Cloud Instance

The main driver for setting up organizations in the first place is resource isolation. In order to achieve this in Grafana Cloud, you can organize resources into folders and set up teams and permissions that correspond to your organizations.

For more information about configuring teams and permissions, refer to [Configure Grafana Teams](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/team-management/configure-grafana-teams/).

# Migrate from Grafana OSS/Enterprise to Grafana Cloud manually

This migration guide is designed to assist Grafana OSS/Enterprise users in seamlessly transitioning manually to Grafana Cloud.

{{< admonition type="note" >}}
There isn't yet a standard method for importing existing data into Grafana Cloud from self-managed databases.
{{< /admonition >}}

{{< admonition type="tip" >}}
You can use the [Grafana Cloud Migration Assistant](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/migration-guide/cloud-migration-assistant/), generally available in Grafana v12, to automatically migrate your resources to Grafana Cloud.
{{< /admonition >}}

## Plan and perform a manual migration

If you need to migrate resources beyond what is supported by the Grafana Cloud Migration Assistant, you can migrate them manually with this guide. Moving your team from Grafana OSS/Enterprise to Grafana Cloud manually involves some coordination and communication in addition to the technical migration in the following documentation.

If you are an existing Grafana OSS/Enterprise customer, contact your account team at Grafana Labs to plan a transition period, arrange licenses, and learn how much your Grafana Cloud subscription costs in comparison to Grafana OSS/Enterprise. The account team can also offer specific guidance and arrange professional services to assist with your migration if needed.

Evaluate Grafana Cloud's security and compliance policies at the [Grafana Labs Trust Center](https://trust.grafana.com/).

You may choose to test Grafana Cloud for some time before migrating your entire organization. To do so, set up a “test” stack in Cloud and migrate resources there first. If you use Grafana Alerting, make sure to set up a different contact point so that alerts do not fire twice.

When you decide to migrate, set aside a day of cutover during which users should not create new dashboards or alerts. Migrate any newly-created resources, turn on your production Alerting contact points and notification policies in Cloud and turn them off in Grafana OSS/Enterprise, and notify your users. You may also choose to redirect the Grafana OSS/Enterprise URL to your Grafana Cloud URL.

| Component    | Migration Effort | Notes                                                                     |
| ------------ | ---------------- | ------------------------------------------------------------------------- |
| Folders      | Low              |                                                                           |
| Dashboards   | Low              | Data source references might need to be renamed                           |
| Alerts       | Medium           | Data source based alerts might need to be adapted                         |
| Plugins      | Medium           | Depends on the feature set of the plugin                                  |
| Data sources | High             | If the data sources reference any secrets, you need to provide them again |

## Before you begin

Ensure you have the following:

- A [Grafana Cloud Stack](https://grafana.com/docs/grafana-cloud/get-started/) and access to a Linux Machine (or a working WSL2 installation) to run the code snippets in this guide.
- Administrator access to a Grafana Cloud stack. To check you access level, Go to `https://grafana.com/orgs/<your-org-name>/members`
- Administrator access to your existing Grafana OSS/Enterprise instance. To check your access level, Go to `https://<grafana-onprem-url>/admin/users`
- Access to the credentials used to connect to your data sources. For example, API keys or usernames and passwords. Since this information is encrypted, it cannot be copied from one instance to the other.
- If some of your data sources are only available from inside your network, refer to the requirements for [Private Data Source Connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/)
- For Plugins, Reports and Playlists only: The [curl](https://github.com/curl/curl) and [jq](https://jqlang.github.io/jq/download/) command line tools

## Upgrade Grafana OSS/Enterprise to the latest version

Grafana Cloud stacks generally run the latest version of Grafana. In order to avoid issues during migration, upgrade Grafana by following our guides [here](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/).

## Migrate Grafana resources

In this guide, the term **"resources"** refers to things you create in Grafana, like dashboards, folders, alerts, data sources, and permissions.

The process of migration works by pulling the existing resources (like dashboards and folders) from the old Grafana instance, modifying them if necessary, and then pushing them to the new Grafana Cloud instance.

In the provided code snippets throughout this migration guide, you need to substitute specific placeholders with your actual credentials and instance URLs. Make the following replacements before executing the scripts:

- `$GRAFANA_SOURCE_TOKEN` with the access token from your Grafana OSS/Enterprise instance.
- `$GRAFANA_DEST_TOKEN` with the access token from your Grafana Cloud instance.
- `$GRAFANA_ONPREM_INSTANCE_URL` with the URL of your Grafana OSS/Enterprise instance. For example: `https://grafana.mydomain.com`
- `$GRAFANA_CLOUD_INSTANCE_URL` with the URL of your Grafana Cloud instance. For example: `https://myorganization.grafana.net`

### Migrate Grafana Plugins

Migration of plugins is the first step when transitioning from Grafana OSS/Enterprise to Grafana Cloud, given that plugins are integral components that influence the functionality and display of other Grafana resources, such as dashboards.

1. To retrieve the Plugins installed in your Grafana OSS/Enterprise instance, issue an HTTP GET request to the `/api/plugins` endpoint. Use the following shell command:

   ```shell
   response=$(curl -s -H "Accept: application/json" -H "Authorization: Bearer $GRAFANA_SOURCE_TOKEN" "${GRAFANA_ONPREM_INSTANCE_URL}/api/plugins")

   plugins=$(echo $response | jq '[.[] | select(.signatureType == "community" or (.signatureType != "internal" and .signatureType != "")) | {name: .id, version: .info.version}]')

   echo "$plugins" > plugins.json
   ```

   The command provided above will carry out an HTTP request to this endpoint and accomplish several tasks:
   - It issues a GET request to the `/api/plugins` endpoint of your Grafana OSS/Enterprise instance to retrieve a list of installed plugins.
   - It filters out the list to only include community plugins and those signed by external parties.
   - It extracts the plugin ID and version before storing them in a `plugins.json` file.

1. To import the plugins in your Grafana Cloud Instance, execute the following command. This command constructs an HTTP POST request to `https://grafana.com/api/instances/<stack_slug>/plugins`

   ```shell
   CLOUD_INSTANCE=$GRAFANA_CLOUD_INSTANCE_URL

   stack_slug="${CLOUD_INSTANCE#*//}"
   stack_slug="${stack_slug%%.*}"
   jq -c '.[]' plugins.json | while IFS= read -r plugin; do
     name=$(echo "$plugin" | jq -r '.name')
     version=$(echo "$plugin" | jq -r '.version')
     echo "Adding plugin $name with version $version to stack $stack_slug"
     response=$(curl -s -X POST "https://grafana.com/api/instances/$stack_slug/plugins" \
               -H "Authorization: Bearer <GRAFANA_CLOUD_ACCESS_TOKEN>" \
               -H "Content-Type: application/json" \
               -d "{\"plugin\": \"$name\", \"version\": \"$version\"}")
     echo "POST response for plugin $name version $version: $response"
   done
   ```

   Replace `<GRAFANA_CLOUD_ACCESS_TOKEN>` with your Grafana Cloud Access Policy Token. To create a new one, refer to Grafana Cloud [access policies documentation](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/)

   This script iterates through each plugin listed in the `plugins.json` file:
   - It constructs a POST request for each plugin to add it to the specified Grafana Cloud instance.
   - It reports back the response for each POST request to give you confirmation or information about any issues that occurred.

### Migrate resources that are already provisioned as-code

If you already use tools like [Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/), [Ansible](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/ansible/), or [Grafana’s HTTP API](https://grafana.com/docs/grafana-cloud/developer-resources/api-reference/http-api/) to provision resources to Grafana, redirect those to the new Grafana Cloud instance by replacing the Grafana URL and credentials.

### Migrate dashboards, folders, data sources, library panels, and alert rules using Grizzly

Grizzly is a command line tool that streamlines working with Grafana resources. Use it to migrate most of the content in your Grafana instance. Follow these steps in your terminal to install Grizzly. If you need to change the os or the architecture, Refer to the Grizzly [releases](https://github.com/grafana/grizzly/releases) and use the binary according to your needs.

```shell
# download the binary (adapt os and arch as needed)
$ curl -fSL -o "/usr/local/bin/grr" "https://github.com/grafana/grizzly/releases/download/v0.3.1/grr-linux-amd64"

# make it executable
$ chmod a+x "/usr/local/bin/grr"

# have fun :)
$ grr --help
```

First, create a new folder on your computer and navigate to it to keep your work organized.

```shell
mkdir grafana-migration
cd grafana-migration
```

To give grizzly access to your Grafana OSS/Enterprise instance and the Grafana Cloud Instance, you need to create a [service account](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/service-accounts/) and a corresponding [access token](https://www.grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/service-accounts/#service-account-tokens) on each instance. You can use these tokens to authenticate requests to pull and push resources. Follow these steps on your Grafana OSS/Enterprise instance:

- Navigate to the **Administration -> Users and access -> Service Accounts** Page within the Grafana OSS/Enterprise instance.
- Click on **Add Service Account**
- Give the Service account a descriptive name like “grizzly-migration” and apply the **Admin** role.
- After creating the account, click on **Add Service Account Token**
- Enter a name for the token
- Select **Set expiration date** and enter an expiry date for the token
- Click **Generate Token** and save this token in a password manager or other secure place.

Complete the service account creation and token generation process for your Grafana Cloud Instance by following the same steps outlined above for your Grafana OSS/Enterprise instance. This ensures that Grizzly has the necessary access token for both platforms.

Next, to tell grizzly which instances you’re going to work on, use the following commands:

```shell
grr config create-context grafana-onprem
grr config use-context grafana-onprem
grr config set output-format json
grr config set grafana.url $GRAFANA_ENT_INSTANCE_URL
grr config set grafana.token $GRAFANA_SOURCE_TOKEN

grr config create-context grafana-cloud
grr config use-context grafana-cloud
grr config set output-format json
grr config set grafana.url $GRAFANA_CLOUD_INSTANCE_URL
grr config set grafana.token $GRAFANA_DEST_TOKEN
```

Afterward, you will have two contexts set up; one for your local Grafana OSS/Enterprise installation and one for Grafana Cloud. The `grr config use-context` command allows you to switch between the two instances while using Grizzly.

#### Export existing resources

Switch to the `grafana-onprem` context and use the pull command to fetch the resources you want to migrate:

```shell
grr config use-context grafana-onprem
grr pull . \
  -t 'Dashboard/*' \
  -t 'Datasource/*' \
  -t 'DashboardFolder/*' \
  -t 'LibraryElement/*' \
  -t 'AlertRuleGroup/*' \
  -t 'AlertContactPoint/*' \
  -t 'AlertNotificationPolicy/*'
```

This will fetch the specified resources from Grafana and store them in the current directory.

#### Push the resources to your Grafana Cloud stack

With everything in place, switch to the Grafana cloud context and use the following commands to apply the resources to the configured instance:

```shell
grr config use-context grafana-cloud

grr apply . -t 'DashboardFolder/*'
grr apply . -t 'LibraryElement/*'
grr apply . -t 'Datasource/*'
grr apply . -t 'Dashboard/*'
grr apply . -t 'AlertRuleGroup/*'
grr apply . -t 'AlertContactPoint/*'
grr apply . -t 'AlertNotificationPolicy/*'
```

#### Fill in data source credentials

After migrating your data sources, you must fill in their credentials, like tokens, usernames, or passwords. For security reasons, grizzly cannot read encrypted data source credentials from the existing Grafana instance.

To fill in the missing authentication information, go to the **Connections -> Datasources** page in your new Grafana Cloud instance and verify that credentials for all data sources are set. You can skip data sources starting with `grafanacloud` - These are managed by Grafana Cloud directly and provide access to Grafana Cloud databases.

If one of your data sources can only be accessed from your internal network, take a look at the [Private Data Source Connect documentation](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/).

After you have configured the data sources, all your dashboards should be available as they were before.

##### (Optional) Configure Private Data Source Connect (PDC)

This step only applies if you use Grafana OSS/Enterprise to access network-secured data sources.

Some data sources, like Prometheus or SQL databases, live on private networks or behind fire wall rules that are not accessible by Grafana Cloud. If your Grafana OSS/Enterprise instance was hosted on the same network as your data source, you might find that Grafana Cloud cannot connect to all of the same data sources that Grafana OSS/Enterprise could access.

To access these data sources from Grafana Cloud, follow our guide to [configure PDC in your network](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/), and then configure the applicable Grafana data sources to [connect using PDC](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-a-data-source-to-use-private-data-source-connect-pdc). Note that PDC is only needed for your network-secured data sources, not for data sources like Splunk or CloudWatch that are accessible over the public internet.

For more information on how PDC works, see our [overview document](/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/).

### Migrate reports and playlists using Grafana’s HTTP API

Grizzly does not currently support Reports and Playlists as a resource, so you can perform this migration using Grafana’s HTTP API using the `curl` command.

#### Reports (For Grafana Enterprise only)

1. To export your Reports, you will need to invoke the `api/reports` endpoint of your Grafana OSS/Enterprise instance. The below shell command accomplishes this by using `curl` to send a request to the endpoint and then stores the retrieved report configuration into a file named `reports.json`.

   ```shell
   curl ${GRAFANA_ONPREM_INSTANCE_URL}/api/reports -H "Authorization: Bearer $GRAFANA_SOURCE_TOKEN" > reports.json
   ```

2. To upload the configuration data you have saved in the `reports.json` file to your new Grafana Cloud instance, run the below command. The command will take the local file `reports.json` and push its contents to the `api/reports` endpoint of your Grafana Cloud instance.

   ```shell
   jq -M -r -c '.[]' < reports.json | while read -r json; do curl -XPOST ${GRAFANA_CLOUD_INSTANCE_URL}/api/reports -H"Authorization: Bearer $GRAFANA_DEST_TOKEN" -d"$json" -H 'Content-Type: application/json'; done
   ```

#### Playlists

1. To retrieve the Playlists from your Grafana OSS/Enterprise instance, issue an HTTP GET request to the `/api/playlists` endpoint. Use the following shell command:

   ```shell
   mkdir playlists
   curl "${GRAFANA_ONPREM_INSTANCE_URL}/api/playlists" \
   -H "Authorization: Bearer $GRAFANA_SOURCE_TOKEN" \
   | jq -M -r -c '.[] | .uid' \
   | while read -r uid; do \
   curl "${GRAFANA_ONPREM_INSTANCE_URL}/api/playlists/$uid" \
       -H "Authorization: Bearer $GRAFANA_SOURCE_TOKEN" \
       > playlists/$uid.json; \
   done
   ```

   The command provided above will carry out an HTTP request to this endpoint and accomplish several tasks:
   - It fetches an array of all the playlists available in the Grafana OSS/Enterprise instance.
   - It then iterates through each playlist to obtain the complete set of details.
   - Finally, it stores each playlist's specification as separate JSON files within a directory named `playlists`

2. To import the playlists, execute the following command. This command constructs an HTTP POST request targeting the `/api/playlists` endpoint of your Grafana Cloud Instance.

   ```shell
   for playlist in playlists/*; do
     curl -XPOST "${GRAFANA_CLOUD_INSTANCE_URL}/api/playlists" \
       -H "Authorization: Bearer $GRAFANA_DEST_TOKEN" \
       -H "Content-Type: application/json" \
       -d $playlist > /dev/null;
   done
   ```

### Migrate single sign-on configuration

Grafana Cloud stacks support all of the same authentication and authorization options as Grafana OSS/Enterprise, except for [anonymous authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/anonymous-auth/) and use of the [Auth proxy](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/auth-proxy/). However, single sign-on settings cannot be exported and imported like dashboards, alerts, and other resources.

To set up SAML authentication from scratch using Grafana’s UI or API, follow [these instructions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/saml/saml-ui/) to Configure SAML authentication in Grafana.

LDAP and OIDC/OAuth2 can only be configured in Grafana Cloud by the Grafana Labs support team. Follow [these instructions](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/) to request SSO configuration from the support team.

### Migrate custom Grafana configuration

You may have customized the [configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/) of your Grafana OSS/Enterprise instance, for example with feature toggles, custom auth, or embedding options. Since Grafana configuration is stored in environment variables or the filesystem where Grafana runs, Grafana Cloud users do not have access to it. However, you can open a support ticket to ask a Grafana Labs support engineer for customizations.

The following customizations are available via support:

- Enabling [feature toggles](http://www.grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles).
- [Single sign-on and team sync using SAML, LDAP, or OAuth](http://www.grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication).
- Enable [embedding Grafana dashboards in other applications](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#allow_embedding) for Grafana Cloud contracted customers.
- [Audit logging](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/audit-grafana/) ([Usage insights logs and dashboards](https://grafana.com/docs/grafana-cloud/account-management/usage-insights/) are available in select Grafana Cloud paid accounts).

Note that the following custom configurations are not supported in Grafana Cloud:

- [Anonymous user access](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/anonymous-auth/).
- [Auth proxy](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/auth-proxy/).
- [Third-party database encryption](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/configure-database-encryption/) and the [Hashicorp Vault](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/configure-database-encryption/encrypt-secrets-using-hashicorp-key-vault/) integration.
- Running self-signed plugins, like custom-built data sources or visualizations. For more information on plugin signing, refer to our [developer documentation](https://grafana.com/developers/plugin-tools/publish-a-plugin/sign-a-plugin).

If you have a custom configuration in Grafana OSS/Enterprise that is not listed here, reach out to our support team to find out whether they can help you set it up.

## Next steps

After you have successfully migrated resources and configuration from Grafana OSS/Enterprise, consider the following steps to enhance your monitoring experience:

- **Get started with Grafana Cloud**: learn more about the functionality available in Grafana Cloud, which is not available in the open source or Enterprise editions. Read more in [Get started with Grafana Cloud](https://grafana.com/docs/grafana-cloud/get-started/)
- **AWS PrivateLink for Grafana Cloud**: securely transmit telemetry data from your AWS Virtual Private Cloud (VPC) to Grafana Cloud, entirely within the AWS network.
  Learn how to set this up with [AWS PrivateLink Integration](https://grafana.com/docs/grafana-cloud/send-data/aws-privatelink/).
- **Azure PrivateLink for Grafana Cloud**, securely transmit telemetry from your Azure Virtual Network to Grafana Cloud while staying on the Azure network, and avoid exposing your traffic to the public internet.
  Learn how to set this up with [AWS PrivateLink Integration](https://grafana.com/docs/grafana-cloud/send-data/azure-privatelink/).
- **[Grafana Integrations](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/)**: ready-made integrations to make monitoring your infrastructure and applications more straightforward.

# Manage organizations

This topic describes what organizations are, and how to create, edit, and delete them.

## About organizations

An organization is an entity that helps you isolate users and resources such as dashboards, annotations, and data sources from each other. Their purpose is to provide completely separate experiences, which look like multiple instances of Grafana, within a single instance. Multiple organizations are easier and cheaper to manage than multiple instances of Grafana.

Users, configuration settings, and Grafana Enterprise licenses are shared between organizations. Other resources, like dashboards, data sources, annotations, folders, Teams, and Alerts, are isolated within each organization and cannot be easily shared with another organization.

The following table summarizes the resources you can share and/or isolate using organizations.

| Resource                 | Mode             |
| ------------------------ | ---------------- |
| Users                    | Share or isolate |
| Folders                  | Isolate only     |
| Dashboards               | Isolate only     |
| Data sources             | Isolate only     |
| Alerts                   | Isolate only     |
| Notification channels    | Isolate only     |
| Annotations              | Isolate only     |
| Reports                  | Isolate only     |
| Service accounts         | Isolate only     |
| Authentication providers | Share only       |
| Configuration settings   | Share only       |
| Licenses                 | Share            |

The member of one organization cannot view dashboards assigned to another organization. However, a user can belong to multiple organizations.

Grafana Server Administrators are responsible for creating organizations. For more information about the Grafana Server Administrator role, refer to [Grafana server administrators](../roles-and-permissions/#grafana-server-administrators).

## View a list of organizations

Complete this task when you want to view a list of existing organizations.

### Before you begin

- Ensure that you have Grafana Server Administrator permissions

**To view a list of organizations:**

1. Sign in to Grafana as a server administrator.
1. Click **Administration** in the left-side menu, **General**, and then **Organizations**.

## Create an organization

Create an organization when you want to isolate dashboards and other resources from each other.

### Before you begin

- Ensure that you have Grafana Server Administrator permissions

**To create an organization:**

1. Sign in to Grafana as a server administrator.
1. Click **Administration** in the left-side menu, **General**, and then **Organizations**.
1. Click **+ New org**.
1. Enter the name of the new organization and click **Create**.

   Grafana creates the organization, adds you as the organization administrator, and opens the Default preferences page.

1. In the Preferences section, select a home dashboard, time zone, and week start.

   For more information about preferences, refer to [Preferences](../organization-preferences/).

For more information about adding users to an organization, refer to [Add a user to an organization](../user-management/server-user-management/add-remove-user-to-org/).

## Delete an organization

This action permanently removes an organization from your Grafana server.

{{< admonition type="warning" >}}
Deleting the organization also deletes all teams and dashboards associated the organization.
{{< /admonition >}}

### Before you begin

- Ensure that you have Grafana Server Administrator permissions
- Because the delete action removes all teams and dashboards associated with the organization, ensure that the users who rely on the organization dashboards are aware of the change

**To delete an organization:**

1. Sign in to Grafana as a server administrator.
1. Click **Administration** in the left-side menu, **General**, and then **Organizations**.
1. Click the red **X** next to the organization that you want to delete.
1. Click **Delete**.

## Edit an organization

Edit an organization when you want to change its name.

### Before you begin

- Ensure that you have Grafana Server Administrator permissions

**To edit an organization:**

1. Sign in to Grafana as a server administrator.
1. Click **Administration** in the left-side menu, **General**, and then **Organizations**.
1. Click the organization you want to edit.
1. Update the organization name and click **Update**.

# Organization preferences

Grafana preferences are basic settings. They control the Grafana UI theme, home dashboard, time zone, and so on.

Preferences are sometimes confusing because they can be set at four different levels, listed from highest level to lowest:

- **Server -** Affects all users on the Grafana server. Set by a [Grafana server admin](../roles-and-permissions/#grafana-server-administrators).
- **Organization -** Affects all users in an organization. Set by an [Organization admin](../roles-and-permissions/#organization-roles).
- **Team -** Affects all users assigned to a team. Set by an Organization Admin or Team Admin. To learn more about these roles, refer to [Teams and permissions](../roles-and-permissions/#teams-and-permissions).
- **User account -** Affects the individual user. Set by the user on their own account.

The lowest level always takes precedence. For example, if a user sets their theme to **Light**, then their visualization of Grafana displays the light theme. Nothing at any higher level can override that.

If the user is aware of the change and intended it, then that's great! But if the user is a Server Admin who made the change to their user preferences a long time ago, they might have forgotten they did that. Then, if that Server Admin is trying to change the theme at the server level, they'll get frustrated as none of their changes have any effect that they can see. (Also, the users on the server might be confused, because _they_ can see the server-level changes!)

## Change Grafana name and email

In Grafana, you can change your names and emails associated with groups or accounts in the Settings or Preferences. This topic provides instructions for each task.

Some tasks require certain permissions. For more information about roles, refer to [Roles and permissions](../roles-and-permissions/).

### Change organization name

Grafana server administrators and organization administrators can change organization names.

#### Grafana Server Admin change organization name

Follow these instructions if you are a Grafana Server Admin.

1. Click **Administration** in the left-side menu.
1. Click **General**.
1. Click **Organizations**.
1. In the organization list, click the name of the organization that you want to change.
1. In **Name**, enter the new organization name.
1. Click **Update**.

#### Organization Admin change organization name

If you are an Organization Admin, follow these steps:

1. Click **Administration** in the left-side menu.
1. Click **General**.
1. Click **Default preferences**.
1. In **Organization name**, enter the new name.
1. Click **Update organization name**.

### Change team name or email

Organization administrators and team administrators can change team names and email addresses.
To change the team name or email, follow these steps:

1. Click **Administration** in the left-side menu, **Users and access**, and select **Team**.
1. In the team list, click the name of the team that you want to change.
1. Click the **Settings** tab.
1. In the Team details section, you can edit the following:
   - **Name -** Edit this field to change the display name associated with the team.
   - **Email -** Edit this field to change the email address associated with the team.
1. Click **Update**.

### Change user name or email

To learn how to edit your user information, refer to [Edit your profile](../user-management/user-preferences/#edit-your-profile).

## Change Grafana UI theme

In Grafana, you can modify the UI theme configured in the Settings or Preferences. Set the UI theme for the server, an organization, a team, or your personal user account using the instructions in this topic.

Some tasks require certain permissions. For more information about roles, refer to [Roles and permissions](../roles-and-permissions/).

### Theme options

The theme affects how Grafana displays graphs, menus, other UI elements.

#### Default

**Default** is either the dark theme or the theme selected in a higher level. For example, if an Organization administrator set the **Light** theme, then that is the default for all the teams in that organization.

#### Dark

Here is an example of the dark theme.

![Dark theme example](/static/img/docs/preferences/dark-theme-7-4.png)

#### Light

Here is an example of the light theme.

![Light theme example](/static/img/docs/preferences/light-theme-7-4.png)

#### Experimental

There are several experimental themes that you can also choose from.
To see the color scheme of a theme before selecting it for your organization, click the user icon in the top-right corner of the page and select **Change theme**.
This opens the **Change theme** drawer, where you can view all themes.

### Change server UI theme

As a Grafana server administrator, you can change the default Grafana UI theme for all users who are on the server by setting the [default_theme](../../setup-grafana/configure-grafana/#default_theme) option in the Grafana configuration file.

To see what the current settings are, refer to [View server settings](../stats-and-license/#view-server-settings).

### Change organization UI theme

Organization administrators can change the UI theme for all users in an organization.

1. Click **Administration** in the left-side menu.
1. Click **General**.
1. Click **Default preferences**.
1. In the Preferences section, select the UI theme.
1. Click **Save**.

### Change team UI theme

Organization and team administrators can change the UI theme for all users on a team.

1. Click **Administration** in the left-side menu, **Users and access**, and select **Teams**.
1. Click the team for which you want to change the UI theme.
1. Click the **Settings** tab.
1. In the Preferences section, select the UI theme.
1. Click **Save**.

### Change your personal UI theme

You can change the UI theme for your user account. This setting overrides UI theme settings at higher levels.

1. Click the user icon in the top right corner of the page and select **Profile**.
1. In the Preferences section, select the UI theme.
1. Click **Save**.

## Change the Grafana default timezone

By default, Grafana uses the timezone in your web browser. However, you can override this setting at the server, organization, team, or individual user level. This topic provides instructions for each task.

Some tasks require certain permissions. For more information about roles, refer to [Roles and permissions](../roles-and-permissions/).

### Set server timezone

Grafana server administrators can choose a default timezone for all users on the server by setting the [default_timezone](../../setup-grafana/configure-grafana/#default_timezone) option in the Grafana configuration file.

### Set organization timezone

Organization administrators can choose a default timezone for their organization.

1. Click **Administration** in the left-side menu.
1. Click **General**.
1. Click **Default preferences**.
1. Click to select an option in the **Timezone** list. **Default** is either the browser local timezone or the timezone selected at a higher level.
1. Click **Save**.

### Set team timezone

Organization administrators and team administrators can choose a default timezone for all users on a team.

1. Click **Administration** in the left-side menu, **Users and access**, and select **Teams**.
1. Click the team for which you want to change the timezone.
1. Click the **Settings** tab.
1. Click to select an option in the **Timezone** list. **Default** is either the browser local timezone or the timezone selected at a higher level.
1. Click **Save**.

### Set your personal timezone

You can change the timezone for your user account. This setting overrides timezone settings at higher levels.

1. Click the user icon in the top right corner of the page and select **Profile**.
1. Click to select an option in the **Timezone** list. **Default** is either the browser local timezone or the timezone selected at a higher level.
1. Click **Save**.

## Change the default home page

When you sign in to Grafana, you see the Grafana home page. Click the Grafana icon or **Home** in the breadcrumb to return to it. You can replace it with a dashboard for the server, an organization, a team, or your personal user account. This topic explains how Grafana picks the home page and provides instructions for each task.

Some tasks require certain permissions. For more information about roles, refer to [Roles and permissions](../roles-and-permissions/).

### How Grafana picks the home page

Grafana checks these sources in order and uses the first one that's set:

1. The **Home Dashboard** preference. Grafana checks the user level first, then the team level, then the organization level, and uses the first level that has a dashboard set.
1. The [`home_page`](../../setup-grafana/configure-grafana/#home_page) configuration option. Grafana redirects to this path.
1. The [`default_home_dashboard_path`](../../setup-grafana/configure-grafana/#default_home_dashboard_path) configuration option. Grafana shows the dashboard from this JSON file.
1. The Grafana home page.

The **Home Dashboard** field at every level also offers **Grafana home**. Select it to ignore the dashboards set at higher levels and show the instance default: `home_page`, `default_home_dashboard_path`, or the Grafana home page. For example, if your organization sets a dashboard and you select **Grafana home** in your profile, you see the instance default instead of the organization's dashboard. At the organization level, **Grafana home** and an empty field have the same effect, because the next level up is the server configuration.

Clear the field to inherit the home page from the next level. For example, a user with an empty field gets their team's dashboard, or the organization's dashboard if the team hasn't set one.

### Set the home page for the server

Users with the Grafana Server Admin flag on their account or access to the configuration file can set the home page for all users on the server in two ways:

- [`home_page`](../../setup-grafana/configure-grafana/#home_page) in the `[users]` section redirects users to a path in Grafana, such as an app plugin page.
- [`default_home_dashboard_path`](../../setup-grafana/configure-grafana/#default_home_dashboard_path) in the `[dashboards]` section shows a dashboard from a JSON file.

If you set both, `home_page` takes precedence. Both settings apply to users who haven't set a home dashboard preference and to users who select **Grafana home**. A dashboard set in a user, team, or organization preference overrides both.

#### Redirect users to a page

Set `home_page` to a path that matches a frontend route. The path must start with a slash.

```ini
[users]
home_page = /a/my-app/home
```

#### [Optional] Convert an existing dashboard into a JSON file

1. Navigate to the page of the dashboard you want to use as the home dashboard.
1. Click the **Share** button at the top right of the screen.
1. In the Export tab, click **Save to file**. Grafana converts the dashboard to a JSON file and saves it locally.

#### Use a JSON file as the home dashboard

1. Save your JSON file somewhere that Grafana can access it. For example, in the Grafana `data` folder of Grafana.
1. Update your configuration file to set the path to the JSON file. Refer to [default_home_dashboard_path](../../setup-grafana/configure-grafana/#default_home_dashboard_path) for more information about modifying the Grafana configuration files.

```ini
[dashboards]
# Path to a custom default home dashboard. If empty, Grafana shows the Grafana home page.
default_home_dashboard_path = data/main-dashboard.json
```

Replacing `public/dashboards/home.json` on disk is not supported. If you previously customized home that way, move the file and set `default_home_dashboard_path` to its path.

### Set the home dashboard for your organization

Organization administrators can choose a default home dashboard for their organization. Teams and users inherit it unless they set their own dashboard or select **Grafana home**.

1. Click **Administration** in the left-side menu.
1. Click **General**.
1. Click **Default preferences**.
1. In the **Home Dashboard** field, search for and select the dashboard that you want to use, or select **Grafana home** to show the instance default.
1. Click **Save**.

### Set the home dashboard for your team

Organization administrators and Team Admins can set a default home dashboard for all users on a team. The team's dashboard overrides the organization's dashboard for team members who haven't set their own.

1. Click **Administration** in the left-side menu, **Users and access**, and select **Teams**.
1. Click the team for which you want to change the home dashboard.
1. Click the **Settings** tab.
1. In the **Home Dashboard** field, search for and select the dashboard that you want to use. Select **Grafana home** to opt the team out of the organization's dashboard and show the instance default instead.
1. Click **Save**.

### Set your personal home dashboard

You can choose your own personal home dashboard. This setting overrides the home dashboards set at the team and organization levels.

1. Click the user icon in the top right corner of the page and select **Profile**.
1. In the **Home Dashboard** field, search for and select the dashboard that you want to use. Select **Grafana home** to opt out of your team's or organization's dashboard and show the instance default instead.
1. Click **Save**.

## Change Grafana language

### Change server language

Grafana server administrators can change the default Grafana UI language for all users on the server by setting the [default_language](../../setup-grafana/configure-grafana/#default_language) option in the Grafana configuration file.

### Change organization language

Organization administrators can change the language for all users in an organization.

1. Click **Administration** in the left-side menu.
1. Click **General**.
1. Click **Default preferences**.
1. In the Preferences section, select an option in the **Language** dropdown.
1. Click **Save**.

### Change team language

Organization and team administrators can set a default language for all users on a team.

1. Click **Administration** in the left-side menu, **Users and access**, and select **Teams**.
1. Click the team for which you want to change the language.
1. Click the **Settings** tab.
1. In the Preferences section, select an option in the **Language** dropdown.
1. Click **Save**.

### Change your personal language

You can change the language for your user account. This setting overrides language settings at higher levels.

1. Click the user icon in the top right corner of the page and select **Profile**.
1. In the Preferences section, select an option in the **Language** dropdown.
1. Click **Save**.

# Customize navigation placement of app plugin pages

By default, Grafana app plugins and their pages appear under the "More apps" section in the navigation menu. However, as a Grafana administrator, you might want to improve user experience by relocating specific pages or entire app plugins to more relevant sections of the navigation hierarchy. This guide shows you how to customize the placement of app plugin pages across different parts of your Grafana navigation menu.

## Customize app and page navigation placement

You can change the location of your app plugin pages in two ways:

1. Move the entire app plugin (with all its pages) to a different section
2. Move specific pages from your app plugin to different sections

### 1. Move an entire app plugin to a different section

To relocate an entire app plugin to a different navigation section, use the `navigation.app_sections` configuration in your Grafana configuration file:

```ini
[navigation.app_sections]
org-example-app = explore 100
```

This configuration:

- Moves the app plugin with ID `org-example-app`
- Places it in the `explore` section
- Assigns it a sort weight of `100` (determining its position within that section)

### 2. Move individual app pages to different sections

To move specific pages from an app plugin to different navigation sections, use the `navigation.app_standalone_pages` configuration:

```ini
[navigation.app_standalone_pages]
/a/org-example-app/dashboard-page = dashboards 200
/a/org-example-app/monitoring-page = alerting 50
```

This configuration:

- Moves the page with path `/a/org-example-app/dashboard-page` to the `dashboards` section with sort weight `200`
- Moves the page with path `/a/org-example-app/monitoring-page` to the `alerting` section with sort weight `50`

## Complete example

Here's a complete example that configures both the app placement and individual page placement in your Grafana configuration:

```ini
# Move the entire app to the Explore section
[navigation.app_sections]
org-example-app = explore 50

# Move specific pages to their own sections
[navigation.app_standalone_pages]
/a/org-example-app/metrics = dashboards 100
/a/org-example-app/logs = alerting 75
```

## Find page paths

To move individual pages you need to know their paths. To identify a plugin page path:

- Visit the page in the browser and check the URL in the address bar.
- Page paths in app plugins follow the format `/a/PLUGIN_ID/PAGE_PATH`

## Troubleshooting

If your navigation changes don't appear:

1. Verify your configuration syntax is correct
2. Ensure you've restarted Grafana after making changes
3. Check that the plugin IDs and page paths exactly match what's defined in your plugin

# Improve security with the Plugins Frontend Sandbox

{{< admonition type="caution" >}}
The Plugin Frontend Sandbox [public preview](https://grafana.com/docs/release-life-cycle/) has been deprecated and will
be removed in Grafana 13.4.
{{< /admonition >}}

The Plugin Frontend Sandbox helps keep a plugin's frontend code focused on the plugin's own area of the Grafana interface.

Plugins running inside the Frontend Sandbox should continue to work normally without any noticeable changes in their intended functionality.

## Intended use

The Frontend Sandbox affects how a plugin interacts with the Grafana interface by adding extra controls to reduce the browser capabilities available to plugin code running within Grafana.

{{< admonition type="warning" >}}
Never install plugins from sources you do not trust.
{{< /admonition >}}

## Compatibility and requirements

The following applies:

- The Frontend Sandbox is available in public preview in Grafana >=11.5. It's compatible with all types of plugins including app plugins, panel plugins, and data source plugins.
- Angular-based plugins are not supported.
- Plugins developed and signed by Grafana Labs are excluded and cannot be sandboxed.

## Enable the Frontend Sandbox

For self-hosted Grafana installations, add the plugin IDs you want to sandbox in the `security` section using the `enable_frontend_sandbox_for_plugins` configuration option.

For Grafana Cloud users, you can simply use the toggle switch in the plugin catalog page to enable or disable the sandbox for each plugin. By default, the sandbox is disabled for all plugins.

{{< admonition type="note" >}}
Enabling the Frontend Sandbox might impact the performance of certain plugins. Only disable the sandbox if you fully trust the plugin and understand the security implications.
{{< /admonition >}}

## Troubleshooting

If a plugin isn't functioning correctly with the Frontend Sandbox enabled:

1. Temporarily disable the sandbox for that specific plugin
1. Test if the plugin works correctly without the sandbox
1. If the plugin only works with the sandbox disabled, confirm the plugin comes from a source you trust before continuing to use it
1. Report any sandbox-related issues to the plugin developer

# Install a plugin

{{< admonition type="note" >}}

If you're a Grafana Cloud user, follow [Install a plugin through the Grafana UI](#install-a-plugin-through-the-grafana-ui).

{{< /admonition >}}

## Install a plugin through the Grafana UI

The default way to install a plugin is through the Grafana UI.

1. In Grafana, go to **Administration > Plugins and data > Plugins** in the side navigation menu to view all plugins.
1. Browse and find the plugin you want to install.
1. Click on the plugin's logo.
1. Click **Install**.

Depending on your environment or setup, you can use use the following alternative methods to install a plugin.

## Install a plugin using Grafana CLI

The Grafana CLI allows you to install, upgrade, and manage your Grafana plugins using a command line tool. For more information about Grafana CLI plugin commands, refer to [Plugin commands](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/cli/#plugins-commands).

## Install a plugin from a ZIP file

This method is typically used for plugins not available in the Plugin Catalog or in environments without internet access.

Download the archive containing the plugin assets, and install it by extracting the archive into the plugin directory. For example:

```bash
unzip my-plugin-0.2.0.zip -d YOUR_PLUGIN_DIR/my-plugin
```

The path to the plugin directory is defined in the configuration file. For more information, refer to [Configuration](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#plugins).

## Install a plugin using Grafana configuration

{{< admonition type="note" >}}
This feature requires Grafana 11.5.0 or later.
{{< /admonition >}}

You can install plugins by adding the plugin ID to the `plugins.preinstall` section in the Grafana configuration file. This prevents the plugin from being accidentally uninstalled and can be auto-updated. For more information, refer to [Configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#preinstall).

## Install a plugin in air-gapped environment

Plugin installation usually requires an Internet connection. You can check which endpoints are used during the installation on your instance and add them to your instance’s allow list.

If this is not possible try installing a plugin using the [Grafana CLI](#install-a-plugin-using-grafana-cli) or as a [ZIP file](#install-a-plugin-from-a-zip-file).

You can fetch any plugin from Grafana.com API following the download link referenced in the API.
Here's an example based on `grafana-lokiexplore-app` plugins.

1. Open `https://grafana.com/api/plugins/grafana-lokiexplore-app` and look for `links` section
1. Find a `download` url which looks something like `https://grafana.com/api/plugins/grafana-lokiexplore-app/versions/1.0.2/download`
1. Use this URL to download the plugin ZIP file, which you can then install as described above.

## Install plugins using the Grafana Helm chart

With the Grafana Helm chart, you can install plugins using one of the methods described in this section. All the YAML snippets install v1.9.0 of the Grafana OnCall App plugin and the Redis data source plugin. When installation is complete you'll get a confirmation message indicating that the plugins were successfully installed.

### Method 1: Use the `plugins` field

Add the plugins you want to install as a list in your values file. For more information about the configuration, refer to [the Helm chart configuration reference](https://github.com/grafana/helm-charts/tree/main/charts/grafana#configuration).

```yaml
plugins:
  - https://grafana.com/api/plugins/grafana-oncall-app/versions/v1.9.0/download;grafana-oncall-app
  - redis-datasource
```

### Method 2: Use `GF_PLUGINS_PREINSTALL_SYNC`

Add the following to your `values.yaml` file:

```yaml
env:
  # Format: <plugin ID>@[<plugin version>]@<url to plugin zip>
  GF_PLUGINS_PREINSTALL_SYNC: grafana-oncall-app@1.9.0@https://grafana.com/api/plugins/grafana-oncall-app/versions/v1.9.0/download

  # Or without version and URL (latest version will be used)
  # GF_PLUGINS_PREINSTALL_SYNC: grafana-oncall-app

  # Multiple plugins (comma-separated)
  # GF_PLUGINS_PREINSTALL_SYNC: grafana-oncall-app,redis-datasource
```

### Method 3: Use `GF_PLUGINS_INSTALL` (Deprecated since v12.1.0)

Add the following to your `values.yaml` file:

```yaml
env:
  # Comma-separated list of plugin IDs
  GF_PLUGINS_INSTALL: grafana-oncall-app,redis-datasource
```

# Allow plugin backend communication

By default, you can only communicate with plugin backends remotely.

However, you can configure your Grafana instance to let the frontends of installed plugins to directly communicate with the backends of other plugins installed locally. You can use this configuration to, for example, enable a [canvas panel](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/canvas/) to call an application resource API that is permitted by the `actions_allow_post_url` option.

## Integrate your plugins

To enable backend communication between plugins, set the plugins you want to communicate with. In your configuration file (`grafana.ini` or `custom.ini` depending on your operating system), remove the semicolon to enable and then set the following configuration option:

```
  actions_allow_post_url=
```

This is a comma-separated list that uses glob matching.

- To allow access to all plugins that have a backend, use:

```
actions_allow_post_url=/api/plugins/*
```

- To access the backend of only one plugin, use:

```
actions_allow_post_url=/api/plugins/<GRAFANA_SPECIAL_APP>
```

# Plugin signatures

Plugin signature verification, also known as _signing_, is a security measure to make sure plugins haven't been tampered with. Upon loading, Grafana checks to see if a plugin is signed or unsigned when inspecting and verifying its digital signature.

Learn more at [plugin policies](https://grafana.com/legal/plugins/).

## How does verification work?

At startup, Grafana verifies the signatures of every plugin in the plugin directory.

To see the result of this verification for each plugin, navigate to **Configuration** -> **Plugins**. A signature can have any of the following signature status:

| Signature status   | Description                                                                     |
| ------------------ | ------------------------------------------------------------------------------- |
| Core               | Core plugin built into Grafana.                                                 |
| Invalid signature  | The plugin has an invalid signature.                                            |
| Modified signature | The plugin has changed since it was signed. This may indicate malicious intent. |
| Unsigned           | The plugin is not signed.                                                       |
| Signed             | The plugin signature was successfully verified.                                 |

### What happens if a plugin is not signed?

If a plugin is unsigned, then Grafana neither loads nor starts it. Grafana also writes an error message to the server log:

```bash
WARN[05-26|12:00:00] Some plugin scanning errors were found   errors="plugin '<plugin id>' is unsigned, plugin '<plugin id>' has an invalid signature"
```

## Plugin signature levels

All plugins are signed under a _signature level_. The signature level determines how the plugin can be distributed.

| **Plugin Level** | **Description**                                                                                                                                                                                                          |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Private          | <p>Private plugins are for use on your own Grafana. They may not be distributed to the Grafana community, and are not published in the Grafana catalog.</p>                                                              |
| Community        | <p>Community plugins have dependent technologies that are open source and not for profit.</p><p>Community plugins are published in the official Grafana catalog, and are available to the Grafana community.</p>         |
| Commercial       | <p>Commercial plugins have dependent technologies that are closed source or commercially backed.</p><p>Commercial plugins are published on the official Grafana catalog, and are available to the Grafana community.</p> |

## Allow unsigned plugins

{{< admonition type="note" >}}
Unsigned plugins are not supported in Grafana Cloud.
{{< /admonition >}}

We strongly recommend that you don't run unsigned plugins in your Grafana instance. However, if you're aware of the risks and you still want to load an unsigned plugin, refer to [Configuration](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#allow_loading_unsigned_plugins).

If you've allowed loading of an unsigned plugin, then Grafana writes a warning message to the server log:

```bash
WARN[06-01|16:45:59] Running an unsigned plugin   pluginID=<plugin id>
```

{{< admonition type="note" >}}
If you're developing a plugin, then you can enable development mode to allow all unsigned plugins.
{{< /admonition >}}

## Sign a plugin you've developed

If you are a plugin developer and want to know how to sign your plugin, refer to [Sign a plugin](https://grafana.com/developers/plugin-tools/publish-a-plugin/sign-a-plugin).

# Types of plugins

Grafana supports three types of plugins:

- [Panels](/grafana/plugins/panel-plugins) - These plugins make it easy to create and add any kind of panel, to show your data, or improve your favorite dashboards.
- [Data sources](/grafana/plugins/data-source-plugins) - These plugins allow you to pull data from various data sources such as databases, APIs, log files, and so on, and display it in the form of graphs, charts, and dashboards in Grafana.
- [Apps](/grafana/plugins/app-plugins) - These plugins enable the bundling of data sources, panels, dashboards, and Grafana pages into a cohesive experience.

## Panel plugins

Add new visualizations to your dashboard with panel plugins, such as the [Clock](/grafana/plugins/grafana-clock-panel), [Mosaic](/grafana/plugins/boazreicher-mosaicplot-panel) and [Variable](/grafana/plugins/volkovlabs-variable-panel) panels.

Use panel plugins when you want to:

- Visualize data returned by data source queries.
- Navigate between dashboards.
- Control external systems, such as smart home devices.

## Data source plugins

Data source plugins add support for new databases, such as [Google BigQuery](/grafana/plugins/grafana-bigquery-datasource).

Data source plugins communicate with external sources of data and return the data in a format that Grafana understands. By adding a data source plugin, you can immediately use the data in any of your existing dashboards.

Use data source plugins when you want to query data from external or third-party systems.

### Managing access for data source plugins

Administrators can grant access to data source plugins with [LBAC](/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/).

Label Based Access Control (LBAC) customizes access rights based on team memberships, ensuring that users only query data relevant to their assigned permissions.

## App plugins

Applications, or app plugins, bundle data sources and panels to provide a cohesive experience, such as the [Zabbix](/grafana/plugins/alexanderzobnin-zabbix-app) app.

Apps can also add custom pages for things like control panels.

Use app plugins when you want an out-of-the-box monitoring experience.

### Managing access for app plugins

Customize access to app plugins with [RBAC](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/rbac-for-app-plugins/).

By default, the Viewer, Editor and Admin roles have access to all app plugins that their Organization role allows them to access.

Access is granted by the `fixed:plugins.app:reader` role.

{{< admonition type="note" >}}
To prevent users from seeing an app plugin, refer to [these permissions scenarios](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/plan-rbac-rollout-strategy/#prevent-viewers-from-accessing-an-app-plugin).
{{< /admonition >}}

# Provision Grafana

Grafana has an active provisioning system that uses configuration files. You can define data sources and dashboards using files that can be version controlled, making GitOps more natural.

{{< admonition type="note" >}}

For advanced as code options such as Infrastructure as code, Git Sync or on-prem file provisioning refer to [Deploy, configure and provision Grafana with as-code workflows](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/).

{{< /admonition >}}

## Configuration file

Refer to [Configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/) for more information on what you can configure in `grafana.ini`.

### Configuration file locations

Grafana reads its default configuration from `<WORKING DIRECTORY>/conf/defaults.ini`.

Grafana reads custom configuration from `<WORKING DIRECTORY>/conf/custom.ini`. You can override the custom configuration path with the `--config` option.

{{< admonition type="note" >}}
The Deb and RPM packages install the configuration file at `/etc/grafana/grafana.ini`.
The Grafana init.d script sets the `--config` option to that path.
{{< /admonition >}}

## Use environment variables

You can use environment variable lookups in all provisioning configuration. The syntax for an environment variable is `$ENV_VAR_NAME` or `${ENV_VAR_NAME}`.

The following applies:

- Only use environment variables for configuration values. Do not use it for keys or bigger parts of the configuration file structure.
- Use environment variables in dashboard provisioning configuration, but not in the dashboard definition files themselves.

The following example looks up the data source URL port, user, and password using environment variables:

```yaml
datasources:
  - name: Graphite
    url: http://localhost:$PORT
    user: $USER
    secureJsonData:
      password: $PASSWORD
```

### Use of the special character `$`

Grafana's provisioning system considers any set of characters after an `$` a variable name.

During the replacement process, Grafana:

1. Replaces the variables that use the syntax `${ENV_VAR_NAME}`.
1. Next, it replaces the variables that use the syntax `$ENV_VAR_NAME`.

If your data contains the character `$`, for example `Pa$sw0rd`, and you're using an environment variable, use the `$ENV_VAR_NAME` syntax to avoid double expansion. If you use the `${ENV_VAR_NAME}` syntax, the value will be first replaced as `Pa$sw0rd` and then again as `Pa` since `$sw0rd` will be considered another variable.

If you want to use the literal value `Pa$sw0rd`, you need to escape the character `$` using a double `$$`: `Pa$$sw0rd`.

The following example shows how variables are replaced, assuming `PASSWORD=Pa$sw0rd`:

```yaml
datasources:
  - name: Graphite
    secureJsonData:
      password1: $PASSWORD # Resolved as Pa$sw0rd
      password2: ${PASSWORD} # Resolved as Pa
      password3: 'Pa$$sw0rd' # Resolved as Pa$sw0rd
      password4: 'Pa$sw0rd' # Resolved as Pa
```

## Data sources

You can manage data sources in Grafana by adding YAML configuration files in the [`provisioning/datasources`](../../setup-grafana/configure-grafana/#provisioning) directory.
Each configuration file contains a list of data sources, under the `datasources` key, to add or update during startup.
If the data source already exists, Grafana reconfigures it to match the provisioned configuration file.

You can also list data sources to automatically delete, using the key `deleteDatasources`.
Grafana deletes the data sources listed in `deleteDatasources` _before_ adding or updating those in the `datasources` list.

You can configure Grafana to automatically delete provisioned data sources when they're removed from the provisioning file.
To do so, add `prune: true` to the root of your data source provisioning file.
With this configuration, Grafana also removes the provisioned data sources if you remove the provisioning file entirely.

### Run multiple Grafana instances

If you run multiple instances of Grafana, add a version number to each data source in the configuration and increase it when you update the configuration.
Grafana only updates data sources with the same or lower version number than the one set in the configuration file.
This prevents old configurations from overwriting newer ones if you have different versions of the `datasource.yaml` file that don't define version numbers, and then restart instances at the same time.

### Example data source configuration file

This example provisions a [Graphite data source](../../datasources/graphite/):

```yaml
# Configuration file version
apiVersion: 1

# List of data sources to delete from the database.
deleteDatasources:
  - name: Graphite
    orgId: 1

# Mark provisioned data sources for deletion if they are no longer in a provisioning file.
# It takes no effect if data sources are already listed in the deleteDatasources section.
prune: true

# List of data sources to insert/update depending on what's
# available in the database.
datasources:
  # <string, required> Sets the name you use to refer to
  # the data source in panels and queries.
  - name: Graphite
    # <string, required> Sets the data source type.
    type: graphite
    # <string, required> Sets the access mode, either
    # proxy or direct (Server or Browser in the UI).
    # Some data sources are incompatible with any setting
    # but proxy (Server).
    access: proxy
    # <int> Sets the organization id. Defaults to orgId 1.
    orgId: 1
    # <string> Sets a custom UID to reference this
    # data source in other parts of the configuration.
    # If not specified, Grafana generates one.
    uid: my_unique_uid
    # <string> Sets the data source's URL, including the
    # port.
    url: http://localhost:8080
    # <string> Sets the database user, if necessary.
    user:
    # <string> Sets the database name, if necessary.
    database:
    # <bool> Enables basic authorization.
    basicAuth:
    # <string> Sets the basic authorization username.
    basicAuthUser:
    # <bool> Enables credential headers.
    withCredentials:
    # <bool> Toggles whether the data source is pre-selected
    # for new panels. You can set only one default
    # data source per organization.
    isDefault:
    # <map> Fields to convert to JSON and store in jsonData.
    jsonData:
      # <string> Defines the Graphite service's version.
      graphiteVersion: '1.1'
      # <bool> Enables TLS authentication using a client
      # certificate configured in secureJsonData.
      tlsAuth: true
      # <bool> Enables TLS authentication using a CA
      # certificate.
      tlsAuthWithCACert: true
    # <map> Fields to encrypt before storing in jsonData.
    secureJsonData:
      # <string> Defines the CA cert, client cert, and
      # client key for encrypted authentication.
      tlsCACert: '...'
      tlsClientCert: '...'
      tlsClientKey: '...'
      # <string> Sets the database password, if necessary.
      password:
      # <string> Sets the basic authorization password.
      basicAuthPassword:
    # <int> Sets the version. Used to compare versions when
    # updating. Ignored when creating a new data source.
    version: 1
    # <bool> Allows users to edit data sources from the
    # Grafana UI.
    editable: false
```

For provisioning examples of specific data sources, refer to that [data source's documentation](../../datasources/).

#### JSON data

Not all data sources have the same configuration settings, only the most common ones are fields in the data source provisioning file.
To provision the rest of a data source's settings, include them as JSON in the `jsonData` field.

Common settings in the [built-in core data sources](../../datasources/#built-in-core-data-sources) include:

| Name                            | Type    | Data source                                                      | Description                                                                                                                                                                                                                                                                                   |
| ------------------------------- | ------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tlsAuth`                       | boolean | _HTTP\*_, MySQL                                                  | Enable TLS authentication using client cert configured in secure JSON data                                                                                                                                                                                                                    |
| `tlsAuthWithCACert`             | boolean | _HTTP\*_, MySQL, PostgreSQL                                      | Enable TLS authentication using CA cert                                                                                                                                                                                                                                                       |
| `tlsSkipVerify`                 | boolean | _HTTP\*_, MySQL, PostgreSQL, MSSQL                               | Controls whether a client verifies the server's certificate chain and host name.                                                                                                                                                                                                              |
| `serverName`                    | string  | _HTTP\*_, MSSQL                                                  | Optional. Controls the server name used for certificate common name/subject alternative name verification. Defaults to using the data source URL.                                                                                                                                             |
| `timeout`                       | string  | _HTTP\*_                                                         | Request timeout in seconds. Overrides `dataproxy.timeout` option                                                                                                                                                                                                                              |
| `graphiteVersion`               | string  | Graphite                                                         | Graphite version                                                                                                                                                                                                                                                                              |
| `timeInterval`                  | string  | Prometheus, Elasticsearch, InfluxDB, MySQL, PostgreSQL and MSSQL | Lowest interval/step value that should be used for this data source.                                                                                                                                                                                                                          |
| `httpMode`                      | string  | InfluxDB                                                         | HTTP Method. 'GET', 'POST', defaults to GET                                                                                                                                                                                                                                                   |
| `maxSeries`                     | number  | InfluxDB                                                         | Max number of series/tables that Grafana processes                                                                                                                                                                                                                                            |
| `httpMethod`                    | string  | Prometheus                                                       | HTTP Method. 'GET', 'POST', defaults to POST                                                                                                                                                                                                                                                  |
| `customQueryParameters`         | string  | Prometheus                                                       | Query parameters to add, as a URL-encoded string.                                                                                                                                                                                                                                             |
| `manageAlerts`                  | boolean | Prometheus and Loki                                              | Manage alerts via Alerting UI                                                                                                                                                                                                                                                                 |
| `alertmanagerUid`               | string  | Prometheus and Loki                                              | UID of Alertmanager that manages Alert for this data source.                                                                                                                                                                                                                                  |
| `timeField`                     | string  | Elasticsearch                                                    | Which field that should be used as timestamp                                                                                                                                                                                                                                                  |
| `interval`                      | string  | Elasticsearch                                                    | Index date time format. nil(No Pattern), 'Hourly', 'Daily', 'Weekly', 'Monthly' or 'Yearly'                                                                                                                                                                                                   |
| `logMessageField`               | string  | Elasticsearch                                                    | Which field should be used as the log message                                                                                                                                                                                                                                                 |
| `logLevelField`                 | string  | Elasticsearch                                                    | Which field should be used to indicate the priority of the log message                                                                                                                                                                                                                        |
| `maxConcurrentShardRequests`    | number  | Elasticsearch                                                    | Maximum number of concurrent shard requests that each sub-search request executes per node                                                                                                                                                                                                    |
| `sigV4Auth`                     | boolean | Elasticsearch and Prometheus                                     | Enable usage of SigV4                                                                                                                                                                                                                                                                         |
| `sigV4AuthType`                 | string  | Elasticsearch and Prometheus                                     | SigV4 auth provider. default/credentials/keys                                                                                                                                                                                                                                                 |
| `sigV4ExternalId`               | string  | Elasticsearch and Prometheus                                     | Optional SigV4 External ID                                                                                                                                                                                                                                                                    |
| `sigV4AssumeRoleArn`            | string  | Elasticsearch and Prometheus                                     | Optional SigV4 ARN role to assume                                                                                                                                                                                                                                                             |
| `sigV4Region`                   | string  | Elasticsearch and Prometheus                                     | SigV4 AWS region                                                                                                                                                                                                                                                                              |
| `sigV4Profile`                  | string  | Elasticsearch and Prometheus                                     | Optional SigV4 credentials profile                                                                                                                                                                                                                                                            |
| `authType`                      | string  | Amazon CloudWatch                                                | Auth provider. default/credentials/keys                                                                                                                                                                                                                                                       |
| `externalId`                    | string  | Amazon CloudWatch                                                | Optional External ID                                                                                                                                                                                                                                                                          |
| `assumeRoleArn`                 | string  | Amazon CloudWatch                                                | Optional ARN role to assume                                                                                                                                                                                                                                                                   |
| `defaultRegion`                 | string  | Amazon CloudWatch                                                | Optional default AWS region                                                                                                                                                                                                                                                                   |
| `customMetricsNamespaces`       | string  | Amazon CloudWatch                                                | Namespaces of Custom Metrics                                                                                                                                                                                                                                                                  |
| `profile`                       | string  | Amazon CloudWatch                                                | Optional credentials profile                                                                                                                                                                                                                                                                  |
| `tsdbVersion`                   | string  | OpenTSDB                                                         | Version                                                                                                                                                                                                                                                                                       |
| `tsdbResolution`                | string  | OpenTSDB                                                         | Resolution                                                                                                                                                                                                                                                                                    |
| `sslmode`                       | string  | PostgreSQL                                                       | SSL mode. 'disable', 'require', 'verify-ca' or 'verify-full'                                                                                                                                                                                                                                  |
| `tlsConfigurationMethod`        | string  | PostgreSQL                                                       | SSL certificate configuration, either by 'file-path' or 'file-content'                                                                                                                                                                                                                        |
| `sslRootCertFile`               | string  | PostgreSQL, MSSQL                                                | SSL server root certificate file, must be readable by the Grafana user                                                                                                                                                                                                                        |
| `sslCertFile`                   | string  | PostgreSQL                                                       | SSL client certificate file, must be readable by the Grafana user                                                                                                                                                                                                                             |
| `sslKeyFile`                    | string  | PostgreSQL                                                       | SSL client key file, must be readable by _only_ the Grafana user                                                                                                                                                                                                                              |
| `encrypt`                       | string  | MSSQL                                                            | Determines SSL encryption handling. Options include: `disable` - data sent between client and server is not encrypted; `false` - data sent between client and server is not encrypted beyond the login packet; `true` - data sent between client and server is encrypted. Default is `false`. |
| `postgresVersion`               | number  | PostgreSQL                                                       | Postgres version as a number (903/904/905/906/1000) meaning v9.3, v9.4, ..., v10                                                                                                                                                                                                              |
| `timescaledb`                   | boolean | PostgreSQL                                                       | Enable usage of TimescaleDB extension                                                                                                                                                                                                                                                         |
| `maxOpenConns`                  | number  | MySQL, PostgreSQL and MSSQL                                      | Maximum number of open connections to the database (Grafana v5.4+)                                                                                                                                                                                                                            |
| `maxIdleConns`                  | number  | MySQL, PostgreSQL and MSSQL                                      | Maximum number of connections in the idle connection pool (Grafana v5.4+)                                                                                                                                                                                                                     |
| `connMaxLifetime`               | number  | MySQL, PostgreSQL and MSSQL                                      | Maximum amount of time in seconds a connection may be reused (Grafana v5.4+)                                                                                                                                                                                                                  |
| `keepCookies`                   | array   | _HTTP\*_                                                         | Cookies that needs to be passed along while communicating with data sources                                                                                                                                                                                                                   |
| `prometheusVersion`             | string  | Prometheus                                                       | The version of the Prometheus data source, such as `2.37.0`, `2.24.0`                                                                                                                                                                                                                         |
| `prometheusType`                | string  | Prometheus                                                       | Prometheus database type. Options are `Prometheus`, `Cortex`, `Mimir` or`Thanos`.                                                                                                                                                                                                             |
| `cacheLevel`                    | string  | Prometheus                                                       | Determines the duration of the browser cache. Valid values include: `Low`, `Medium`, `High`, and `None`.                                                                                                                                                                                      |
| `incrementalQuerying`           | string  | Prometheus                                                       | Experimental: Turn on incremental querying to enhance dashboard reload performance with slow data sources                                                                                                                                                                                     |
| `incrementalQueryOverlapWindow` | string  | Prometheus                                                       | Experimental: Configure incremental query overlap window. Requires a valid duration string, for example, `180s` or `15m` Default value is `10m` (10 minutes).                                                                                                                                 |
| `disableRecordingRules`         | boolean | Prometheus                                                       | Experimental: Turn off Prometheus recording rules                                                                                                                                                                                                                                             |
| `seriesEndpoint`                | boolean | Prometheus                                                       | Using Prometheus `/api/v1/series` endpoint                                                                                                                                                                                                                                                    |
| `implementation`                | string  | Alertmanager                                                     | The implementation of the Alertmanager data source, such as `prometheus`, `cortex` or `mimir`                                                                                                                                                                                                 |
| `handleGrafanaManagedAlerts`    | boolean | Alertmanager                                                     | When enabled, Grafana-managed alerts are sent to this Alertmanager                                                                                                                                                                                                                            |

{{< admonition type="note" >}}
Data sources tagged with _HTTP\*_ communicate using the HTTP protocol, which includes all core data source plugins except MySQL, PostgreSQL, and MSSQL.
{{< /admonition >}}

For examples of specific data sources' JSON data, refer to that [data source's documentation](../../datasources/).

#### Secure JSON data

Secure JSON data is a map of settings encrypted with a [secret key](../../setup-grafana/configure-grafana/#secret_key).
The encryption hides the JSON data from the users of Grafana.
You should use secure JSON data to store TLS certificates and passwords for data source HTTP requests.
All of these settings are optional.

| Name                | Type   | Data source                        | Description                                              |
| ------------------- | ------ | ---------------------------------- | -------------------------------------------------------- |
| `tlsCACert`         | string | _HTTP\*_, MySQL, PostgreSQL        | CA cert for out going requests                           |
| `tlsClientCert`     | string | _HTTP\*_, MySQL, PostgreSQL        | TLS Client cert for outgoing requests                    |
| `tlsClientKey`      | string | _HTTP\*_, MySQL, PostgreSQL        | TLS Client key for outgoing requests                     |
| `password`          | string | _HTTP\*_, MySQL, PostgreSQL, MSSQL | password                                                 |
| `basicAuthPassword` | string | _HTTP\*_                           | password for basic authentication                        |
| `accessKey`         | string | Amazon CloudWatch                  | Access key for connecting to Amazon CloudWatch           |
| `secretKey`         | string | Amazon CloudWatch                  | Secret key for connecting to Amazon CloudWatch           |
| `sigV4AccessKey`    | string | Elasticsearch and Prometheus       | SigV4 access key. Required when using keys auth provider |
| `sigV4SecretKey`    | string | Elasticsearch and Prometheus       | SigV4 secret key. Required when using keys auth provider |

{{< admonition type="note" >}}
The _HTTP\*_ tag denotes data sources that communicate using the HTTP protocol, including all core data source plugins except MySQL, PostgreSQL, and MSSQL.
{{< /admonition >}}

#### Custom HTTP headers for data sources

You can add HTTP headers to all requests sent to data sources managed by Grafana provisioning
Configure the header name in the `jsonData` field and the header value in `secureJsonData`.
The following example sets the `HeaderName` header to have the value `HeaderValue` and the `Authorization` header to have the value `Bearer XXXXXXXXX`:

```yaml
apiVersion: 1

datasources:
  - name: Graphite
    jsonData:
      httpHeaderName1: 'HeaderName'
      httpHeaderName2: 'Authorization'
    secureJsonData:
      httpHeaderValue1: 'HeaderValue'
      httpHeaderValue2: 'Bearer XXXXXXXXX'
```

## Plugins

You can manage plugin applications in Grafana by adding one or more YAML configuration files in the [`provisioning/plugins`](../../setup-grafana/configure-grafana/#provisioning) directory.
Each configuration file contains a list of `apps` that Grafana configures during start up.
Grafana configures each app to use the configuration in the file.

{{< admonition type="note" >}}
This feature enables you to provision plugin configurations, not the plugins themselves.
You must have already installed the plugin to use plugin configuration provisioning.
{{< /admonition >}}

### Example plugin configuration file

```yaml
apiVersion: 1

apps:
  # <string> the type of app, plugin identifier. Required
  - type: raintank-worldping-app
    # <int> Org ID. Default to 1, unless org_name is specified
    org_id: 1
    # <string> Org name. Overrides org_id unless org_id not specified
    org_name: Main Org.
    # <bool> disable the app. Default to false.
    disabled: false
    # <map> fields that will be converted to json and stored in jsonData. Custom per app.
    jsonData:
      # key/value pairs of string to object
      key: value
    # <map> fields that will be converted to json, encrypted and stored in secureJsonData. Custom per app.
    secureJsonData:
      # key/value pairs of string to string
      key: value
```

## Dashboards

You can manage dashboards in Grafana by adding one or more YAML configuration files in the [`provisioning/dashboards`](../../setup-grafana/configure-grafana/#dashboards) directory.
Each configuration file contains a list of `providers` that Grafana uses to load dashboards from the local filesystem.

### Example dashboard configuration file

```yaml
apiVersion: 1

providers:
  # <string> an unique provider name. Required
  - name: 'a unique provider name'
    # <int> Org id. Default to 1
    orgId: 1
    # <string> name of the dashboard folder.
    folder: ''
    # <string> folder UID. will be automatically generated if not specified
    folderUid: ''
    # <string> provider type. Default to 'file'
    type: file
    # <bool> disable dashboard deletion
    disableDeletion: false
    # <int> how often Grafana will scan for changed dashboards
    updateIntervalSeconds: 10
    # <bool> allow updating provisioned dashboards from the UI
    allowUiUpdates: false
    options:
      # <string, required> path to dashboard files on disk. Required when using the 'file' type
      path: /var/lib/grafana/dashboards
      # <bool> use folder names from filesystem to create folders in Grafana
      foldersFromFilesStructure: true
```

When Grafana starts, it updates or creates all dashboards found in the configured path.
These files can define the dashboard using the dashboard JSON:

```json
{
  "dashboard": {
    "id": null,
    "uid": "example-dashboard",
    "title": "Production Overview",
    "tags": ["production", "monitoring"],
    "timezone": "browser",
    "schemaVersion": 16,
    "version": 0,
    "refresh": "30s"
  },
  "folderUid": "monitoring-folder",
  "overwrite": true
}
```

Or using a Kubernetes format, for example `kubernetes-dashboard.json`:

```json
{
  "kind": "Dashboard",
  "apiVersion": "dashboard.grafana.app/v1",
  "metadata": {
    "name": "dashboard-uid"
  },
  "spec": {
    "title": "Dashboard title",
    "panels": [
      {
        "gridPos": {
          "h": 13,
          "w": 24,
          "x": 0,
          "y": 0
        },
        "options": {
          "content": "<div><h1>Example panel</h1></div>",
          "mode": "html"
        },
        "transparent": true,
        "type": "text"
      }
    ]
  }
}
```

You _must_ use the Kubernetes resource format to provision dashboards v2 / dynamic dashboards.

{{< admonition type="note" >}}
Grafana installs dashboards at the root level if you don't set the `folder` field.
{{< /admonition >}}

#### Detect updates to provisioned dashboards files

After Grafana provisions your dashboards, it checks the filesystem for changes and updates dashboards as needed.

The mechanism Grafana uses to do this depends on your `updateIntervalSeconds` value:

- **More than 10 seconds**: Grafana polls the path at that interval.
- **10 seconds or less**: Grafana watches the filesystem for changes and updates dashboards when it detects them.

{{< admonition type="note" >}}
When `updateIntervalSeconds` is 10 or less, Grafana relies on filesystem watch events to detect changes.
Depending on your filesystem and how you mount or sync dashboard files (for example, Docker bind mounts or some network filesystems), those events might not reach Grafana.
To work around this, set `updateIntervalSeconds` to more than 10 to force polling, or update your setup so filesystem watch events are propagated.
{{< /admonition >}}

#### Make changes to a provisioned dashboard

You can make changes to a provisioned dashboard in the Grafana UI but its not possible to automatically save the changes back to the provisioning source.
If `allowUiUpdates` is set to `true` and you make changes to a provisioned dashboard, when you save the dashboard, Grafana persists the changes to its database.

{{< admonition type="caution" >}}
If you save a provisioned dashboard in the UI and then later update the provisioning source, Grafana always overwrites the database dashboard with the one from the provisioning file.
Grafana ignores the `version` property in the JSON file, even if it's lower than the dashboard in the database.
{{< /admonition >}}

{{< admonition type="caution" >}}
If you save a provisioned dashboard in the UI and remove the provisioning source, Grafana deletes the dashboard in the database unless you have set the option `disableDeletion` to `true`.
{{< /admonition >}}

If you set `allowUiUpdates` to `false`, you can't save changes to a provisioned dashboard.
When you try to save changes to a provisioned dashboard, Grafana brings up a _Cannot save provisioned dashboard_ dialog box.

Grafana offers options to export the JSON definition of a dashboard.
To export the dashboard JSON definition, follow these steps:

1. Upon save, click **Advanced options** to expand the section, and then make the following selections:
   - **Model**: Choose from **Classic** or **V2 Resource**. Choose **Classic** if you plan to use the dashboard in Grafana v12.4 or older.
   - **Format**: For the V2 Resource only, choose from **JSON** or **YAML**

1. Click either **Copy JSON to Clipboard** or **Save JSON to file**.

The following screenshot illustrates this behavior.

{{< figure src="/static/img/docs/v51/provisioning_cannot_save_dashboard.png" max-width="500px" class="docs-image--no-shadow" >}}

Grafana removes the `id` field from the dashboard JSON to help the provisioning workflow.
Now you sync your dashboard changes back to the provisioning source.

### Reusable dashboard URLs

If the dashboard in the JSON file contains an [UID](../../dashboards/build-dashboards/view-dashboard-json-model/), Grafana updates that the dashboard with that UID in the database.
This lets you migrate dashboards between Grafana instances and keep consistent dashboard URLs.
When Grafana starts, it creates or updates all dashboards available in the configured folders.

{{< admonition type="caution" >}}
You can overwrite existing dashboards with provisioning.

Be careful not to reuse the same `title` multiple times within a folder or `uid` within the same Grafana instance to avoid inconsistent behavior.
{{< /admonition >}}

### Provision folders structure from filesystem to Grafana

If you already store your dashboards using folders in a Git repository or on a filesystem, and want to have the same folder names in the Grafana menu, use `foldersFromFilesStructure` option.

For example, to replicate the following dashboards structure from the filesystem to Grafana:

```
/etc/dashboards
├── /server
│   ├── /common_dashboard.json
│   └── /network_dashboard.json
└── /application
    ├── /requests_dashboard.json
    └── /resources_dashboard.json
```

use the following provisioning configuration file:

```yaml
apiVersion: 1

providers:
  - name: dashboards
    type: file
    updateIntervalSeconds: 30
    options:
      path: /etc/dashboards
      foldersFromFilesStructure: true
```

Grafana creates the `server` and `application` folders in the UI.

To use `foldersFromFilesStructure`, you must unset the `folder` and `folderUid` options.

To provision dashboards to the root level, store them in the root of your `path`.

{{< admonition type="note" >}}
Nested folder structures are supported: the folder hierarchy on disk is recreated in Grafana.

For example, `folderTwo/folderThree/dashboard3.json` creates a folder `folderTwo` containing a folder `folderThree` that contains the dashboard.

The folder depth is limited to `4` levels.
{{< /admonition >}}

## Alerting

For information on provisioning Grafana Alerting, refer to [Provision Grafana Alerting resources](../../alerting/set-up/provision-alerting-resources/).

### Supported settings

The following sections detail the supported settings and secure settings for each alert notification type.
In the provisioning YAML use `settings` for settings and `secure_settings` for secure settings.
Grafana encrypts secure settings in the database.

#### Alert notification `pushover`

| Name         | Secure setting |
| ------------ | -------------- |
| `apiToken`   | yes            |
| `userKey`    | yes            |
| `device`     |                |
| `priority`   |                |
| `okPriority` |                |
| `retry`      |                |
| `expire`     |                |
| `sound`      |                |
| `okSound`    |                |

#### Alert notification `discord`

| Name                   | Secure setting |
| ---------------------- | -------------- |
| `url`                  | yes            |
| `avatar_url`           |                |
| `content`              |                |
| `use_discord_username` |                |

#### Alert notification `slack`

| Name             | Secure setting |
| ---------------- | -------------- |
| `url`            | yes            |
| `recipient`      |                |
| `username`       |                |
| `icon_emoji`     |                |
| `icon_url`       |                |
| `uploadImage`    |                |
| `mentionUsers`   |                |
| `mentionGroups`  |                |
| `mentionChannel` |                |
| `token`          | yes            |
| `color`          |                |

#### Alert notification `victorops`

| Name          |
| ------------- |
| `url`         |
| `autoResolve` |

#### Alert notification `kafka`

| Name             |
| ---------------- |
| `kafkaRestProxy` |
| `kafkaTopic`     |

#### Alert notification `LINE`

| Name    | Secure setting |
| ------- | -------------- |
| `token` | yes            |

#### Alert notification `MQTT`

| Name            | Secure setting |
| --------------- | -------------- |
| `brokerUrl`     |                |
| `clientId`      |                |
| `topic`         |                |
| `messageFormat` |                |
| `username`      |                |
| `password`      | yes            |
| `retain`        |                |
| `qos`           |                |
| `tlsConfig`     |                |

##### TLS configuration

| Name                 | Secure setting |
| -------------------- | -------------- |
| `insecureSkipVerify` |                |
| `clientCertificate`  | yes            |
| `clientKey`          | yes            |
| `caCertificate`      | yes            |

#### Alert notification `pagerduty`

| Name             | Secure setting |
| ---------------- | -------------- |
| `integrationKey` | yes            |
| `autoResolve`    |                |

#### Alert notification `sensu`

| Name       | Secure setting |
| ---------- | -------------- |
| `url`      |                |
| `source`   |                |
| `handler`  |                |
| `username` |                |
| `password` | yes            |

#### Alert notification `sensugo`

| Name        | Secure setting |
| ----------- | -------------- |
| `url`       |                |
| `apikey`    | yes            |
| `entity`    |                |
| `check`     |                |
| `handler`   |                |
| `namespace` |                |

#### Alert notification `prometheus-alertmanager`

| Name                | Secure setting |
| ------------------- | -------------- |
| `url`               |                |
| `basicAuthUser`     |                |
| `basicAuthPassword` | yes            |

#### Alert notification `teams`

| Name  | Secure setting |
| ----- | -------------- |
| `url` |                |

#### Alert notification `dingding`

| Name  | Secure setting |
| ----- | -------------- |
| `url` |                |

#### Alert notification `email`

| Name          | Secure setting |
| ------------- | -------------- |
| `singleEmail` |                |
| `addresses`   |                |

#### Alert notification `opsgenie`

| Name               | Secure setting |
| ------------------ | -------------- |
| `apiKey`           | yes            |
| `apiUrl`           |                |
| `autoClose`        |                |
| `overridePriority` |                |
| `sendTagsAs`       |                |

#### Alert notification `telegram`

| Name          | Secure setting |
| ------------- | -------------- |
| `bottoken`    | yes            |
| `chatid`      |                |
| `uploadImage` |                |

#### Alert notification `threema`

| Name           | Secure setting |
| -------------- | -------------- |
| `gateway_id`   |                |
| `recipient_id` |                |
| `api_secret`   | yes            |

#### Alert notification `webhook`

| Name          | Secure setting |
| ------------- | -------------- |
| `url`         |                |
| `http_method` |                |
| `username`    |                |
| `password`    | yes            |
| `tls_config`  |                |
| `hmac_config` |                |

##### TLS configuration

| Name                 | Secure setting |
| -------------------- | -------------- |
| `insecureSkipVerify` |                |
| `clientCertificate`  | yes            |
| `clientKey`          | yes            |
| `caCertificate`      | yes            |

##### HMAC signature configuration

| Name              | Secure setting |
| ----------------- | -------------- |
| `secret`          | yes            |
| `header`          |                |
| `timestampHeader` |                |

#### Alert notification `googlechat`

| Name  | Secure setting |
| ----- | -------------- |
| `url` |                |

#### Alert notification `Cisco Webex Teams`

| Name        | Secure setting |
| ----------- | -------------- |
| `message`   |                |
| `room_id`   |                |
| `api_url`   |                |
| `bot_token` | yes            |

## Grafana Enterprise

Grafana Enterprise supports:

- [Provisioning role-based access control with Grafana](../roles-and-permissions/access-control/rbac-grafana-provisioning/)
- [Provisioning role-based access control with Terraform](../roles-and-permissions/access-control/rbac-terraform-provisioning/)

## Configuration management tools

The Grafana community maintains libraries for many popular configuration management tools.

| Tool    | Project                                                                                                                           |
| ------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Puppet  | <https://forge.puppet.com/puppet/grafana>                                                                                         |
| Ansible | <https://github.com/grafana/grafana-ansible-collection>                                                                           |
| Chef    | <https://github.com/sous-chefs/chef-grafana>                                                                                      |
| NixOS   | [`services.grafana.provision` module](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/monitoring/grafana.nix) |

# Recorded queries (deprecated)

{{< admonition type="warning" >}}
Recorded queries are deprecated. Please use the new [Grafana-managed recording rules](/docs/grafana/latest/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules) instead.

To learn how to migrate your recorded queries to Grafana-managed recording rules, refer to the migration documentation [here.](/docs/grafana/latest/alerting/alerting-rules/create-recording-rules/migrate-recorded-queries)
{{< /admonition >}}

Recorded queries allow you to see trends over time by taking a snapshot of a data point on a set interval. This can give you insight into historic trends.

For our plugins that do not return time series, it might be useful to plot historical data. For example, you might want to query ServiceNow to see a history of request response times but it can only return current point-in-time metrics.

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/docs/grafana-cloud/).
{{< /admonition >}}

## How recorded queries work

{{< admonition type="note" >}}
An administrator must configure a Prometheus data source and associate it with a [Remote write target](#remote-write-target) before recorded queries can be used.
{{< /admonition >}}

Recorded queries only work with backend data source plugins. Refer to [Backend data source plugin](/tutorials/build-a-data-source-backend-plugin/) for more information about backend data source plugins. You can recorded four types of queries:

- single row and column - A query that returns a single row and column.
- row count - A query that returns meaningful rows to be counted.
- expression - Any expression. To learn more about creating and using expressions, see [Write expression queries](../../panels-visualizations/query-transform-data/expression-queries/).
- dataplane numeric - A query that returns [dataplane numeric kind](/developers/dataplane/numeric) data.

After a recorded query is created or enabled, it immediately creates a snapshot and continues to create snapshots at the set interval. The recorded query stops taking snapshots when it is disabled, deleted, or when Grafana is not running. Data points are gathered in the backend by running the recorded query and forwarding each result to a remote-write enabled Prometheus instance.

## Using a recorded query

To use a recorded query, create one and add it to a dashboard. After that, it can be managed in **Preferences** from the **Recorded queries** tab.

### Create a recorded query

1.  Find/create a query you want to record on a dashboard in an edit panel. The query must only return one row and column. If it returns more, you can still record the number of results returned using the “count” option.
    - The query's data source must be a backend data source.
    - Expressions can be used to aggregate data from a time series query. Refer to [Write expression queries](../../panels-visualizations/query-transform-data/expression-queries/) to learn more about creating and using expressions.
1.  Click the record query button located in the top right of the query editor.
1.  Enter recorded query information. All fields are required unless otherwise indicated.
    - Name - Name of the recorded query.
    - Description - (optional) Describe the recorded query as you want it to appear in the recorded query list.
    - Interval - The interval on which the snapshot will be taken. The interval starts when you create the recorded query and will stop if you pause or delete the recorded query. Refer to [Managing recorded queries](#manage-recorded-queries) for more information on pausing and deleting recorded queries.
    - Range - The relative time range of the query. If you select a range of `30m` and an interval of `1h` the query will take a snapshot every hour of the past 30 minutes.
    - Count query results - If you want to count the rows returned from your query toggle this option on. If this option is off, your query must return one row with one value.
1.  Test your recorded query by clicking the test recorded query button.
1.  Click `Start recording query` to start recording your query.

### Add a recorded query

You can add existing recorded queries to panels in a dashboard. For each recorded query that you add, a Prometheus query is created: `generated_recorded_query_name{id="generated_id", name="recorded query name"}`. The created query from Prometheus returns all the recorded query’s gathered snapshots.

1. Navigate to a panel in a dashboard where you wish to add a recorded query.
1. Click the `+ Recorded query` button located below the queries.
1. If you want to filter recorded queries by data source, select a data source from the filter by data source drop down menu.
1. Click the `Add` button on your recorded query to add it to the panel.

After adding your recorded query to the panel, the panel data source will become `-- Mixed --`. Your recorded query is represented by a `Prometheus` query with a name label matching your recorded query name. Refer to [Prometheus](../../datasources/prometheus/) to learn more about the `Prometheus` data source.

If after adding a recorded query, a query with a `-- Mixed --` data source instead of `Prometheus` data source appears, this could mean that a Prometheus remote write target was not set up for recorded queries. Refer to [Remote write target](#remote-write-target) to set up a remote write point.

### Manage recorded queries

Recorded queries can be paused/activated and deleted from the Recorded queries tab in Preferences. Deleting a recorded query will remove it from Grafana, but the information that was gathered in Prometheus will still be there. Pausing a recorded query will no longer gather new data points until it is resumed.

### Remote write target

The remote write target is the Prometheus data source that recorded query data points are written to. You will need a Prometheus with remote write enabled and you will need to create a data source for this Prometheus.

The remote write target can be edited by clicking `Edit Remote Write Target` in the upper right on the Recorded Queries tab in Preferences. Select the Prometheus data source that has remote write enabled and enter the remote write path.

# Assign RBAC roles

{{< admonition type="note" >}}
Available in [Grafana Enterprise](/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](/docs/grafana-cloud).
{{< /admonition >}}

In this topic you'll learn how to use the role picker, provisioning, and the HTTP API to assign fixed and custom roles to users and teams.

## Assign fixed roles in the UI using the role picker

This section describes how to:

- Assign a fixed role to a user, team or service account as an organization administrator.
- Assign a fixed role to a user as a server administrator. This approach enables you to assign a fixed role to a user in multiple organizations, without needing to switch organizations.

In both cases, the assignment applies only to the user, team or service account within the affected organization, and no other organizations. For example, if you grant the user the **Data source editor** role in the **Main** organization, then the user can edit data sources in the **Main** organization, but not in other organizations.

**Before you begin:**

- [Plan your RBAC rollout strategy](ref:plan-rbac-rollout-strategy).
- Identify the fixed roles that you want to assign to the user, team or service account.

  For more information about available fixed roles, refer to [RBAC role definitions](ref:rbac-role-definitions).

- Ensure that your own user account has the correct permissions:
  - If you are assigning permissions to a user, team or service account within an organization, you must have organization administrator or server administrator permissions.
  - If you are assigning permissions to a user who belongs to multiple organizations, you must have server administrator permissions.
  - Your Grafana user can also assign fixed role if it has either the `fixed:roles:writer` fixed role assigned to the same organization to which you are assigning RBAC to a user, or a custom role with `users.roles:add` and `users.roles:remove` permissions.
  - Your own user account must have the roles you are granting. For example, if you would like to grant the `fixed:users:writer` role to a team, you must have that role yourself.

**To assign a fixed role to a user, team or service account:**

1. Sign in to Grafana.
2. Switch to the organization that contains the user, team or service account.

   For more information about switching organizations, refer to [Switch organizations](/docs/grafana/<GRAFANA_VERSION>/administration/user-management/user-preferences/#switch-organizations).

3. In the left-side menu, click **Administration**, **Users and access**, and then **Users**, **Teams**, or **Service accounts**.
4. In the **Role** column, select the fixed role that you want to assign to the user, team, or service account.
5. Click **Update**.

**To assign a fixed role as a server administrator:**

1. Sign in to Grafana as a server administrator.
1. Click **Administration** in the left-side menu, **Users and access**, and then **Users**.
1. Click a user.
1. In the Organizations section, click **Change role**.
1. Select a role within an organization that you want to assign to the user.
1. Click **Save**.

## Assign fixed or custom roles using the HTTP API

This section describes how to:

- Assign a fixed role to a user, team or service account as an organization administrator.

**Before you begin:**

- [Plan your RBAC rollout strategy](ref:plan-rbac-rollout-strategy).
- Identify the fixed roles that you want to assign to the user, team or service account.

  For more information about available fixed roles, refer to [RBAC role definitions](ref:rbac-role-definitions).

- Ensure that your own user account has the correct permissions:
  - If you are assigning permissions to a user, team or service account within an organization, you must have organization administrator or server administrator permissions.
  - If you are assigning permissions to a user who belongs to multiple organizations, you must have server administrator permissions.
  - Your Grafana user can also assign fixed role if it has either the `fixed:roles:writer` fixed role assigned to the same organization to which you are assigning RBAC to a user, or a custom role with `users.roles:add` and `users.roles:remove` permissions.
  - Your own user account must have the roles you are granting. For example, if you would like to grant the `fixed:users:writer` role to a team, you must have that role yourself.

**To assign a fixed role to a user or service account:**

Refer to the [RBAC HTTP API for user role assignment](/docs/grafana/<GRAFANA_VERSION>/developers/http_api/access_control/#add-a-user-role-assignment).

The service account ID can be used in place of the user ID.

**To assign a fixed role to a team:**

Refer to the [RBAC HTTP API for team role assignment](/docs/grafana/<GRAFANA_VERSION>/developers/http_api/access_control/#add-a-team-role-assignment).

## Assign fixed or custom roles to a team using provisioning

Instead of using the Grafana role picker, you can use file-based provisioning to assign fixed roles to teams. If you have a large number of teams, provisioning can provide an easier approach to assigning and managing role assignments.

**Before you begin:**

- Refer to [Role provisioning](ref:rbac-grafana-provisioning)
- Ensure that the team to which you are adding the fixed role exists. For more information about creating teams, refer to [Manage teams](/docs/grafana/<GRAFANA_VERSION>/administration/team-management/)

**To assign a role to a team:**

1. Open the YAML configuration file.

1. Refer to the following table to add attributes and values.

   | Attribute                | Description                                                                                                                                                                                    |
   | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | `roles`                  | Enter the custom role or custom roles you want to create/update.                                                                                                                               |
   | `roles > name`           | Enter the name of the custom role.                                                                                                                                                             |
   | `roles > version`        | Enter the custom role version number. Role assignments are independent of the role version number.                                                                                             |
   | `roles > global`         | Enter `true`. You can specify the `orgId` otherwise.                                                                                                                                           |
   | `roles > permissions`    | Enter the permissions `action` and `scope` values. For more information about permissions actions and scopes, refer to [RBAC permissions, actions, and scopes](ref:custom-role-actions-scopes) |
   | `teams`                  | Enter the team or teams to which you are adding the custom role.                                                                                                                               |
   | `teams > orgId`          | Because teams belong to organizations, you must add the `orgId` value.                                                                                                                         |
   | `teams > name`           | Enter the name of the team.                                                                                                                                                                    |
   | `teams > roles`          | Enter the custom or fixed role or roles that you want to grant to the team.                                                                                                                    |
   | `teams > roles > name`   | Enter the name of the role.                                                                                                                                                                    |
   | `teams > roles > global` | Enter `true`, or specify `orgId` of the role you want to assign to the team. Fixed roles are global.                                                                                           |

   For more information about managing custom roles, refer to [Create custom roles using provisioning](ref:manage-rbac-roles-create-custom-roles-using-provisioning).

1. Reload the provisioning configuration file.

   For more information about reloading the provisioning configuration at runtime, refer to [Reload provisioning configurations](/docs/grafana/<GRAFANA_VERSION>/developers/http_api/admin/#reload-provisioning-configurations).

The following example creates the `custom:users:writer` role and assigns it to the `user writers` and `user admins` teams along with the `fixed:users:writer` role:

The following example:

- Creates the `custom:users:writer` role.
- Assigns the `custom:users:writer` role and the `fixed:users:writer` role to the `user admins` and `user writers` teams.

```yaml
# config file version
apiVersion: 2

# Roles to insert/update in the database
roles:
  - name: 'custom:users:writer'
    description: 'List/update other users in the organization'
    version: 1
    global: true
    permissions:
      - action: 'org.users:read'
        scope: 'users:*'
      - action: 'org.users:write'
        scope: 'users:*'

# Assignments to teams
teams:
  - name: 'user writers'
    orgId: 1
    roles:
      # Custom role assignment
      - name: 'custom:users:writer'
        global: true
      # Fixed role assignment
      - name: 'fixed:users:writer'
        global: true
  - name: 'user admins'
    orgId: 1
    roles:
      - name: 'custom:users:writer'
        global: true
      - name: 'fixed:users:writer'
        global: true
```

> **Note**: The roles don't have to be defined in the provisioning configuration files to be assigned. If roles exist in the database, they can be assigned.

**Remove a role assignment from a team:**

If you want to remove an assignment from a team, add `state: absent` to the `teams > roles` section, and reload the configuration file.

The following example:

- Creates the `custom:users:writer` role
- Assigns the `custom:users:writer` role and the `fixed:users:writer` role to the `user admins` team
- Removes the `custom:users:writer` and the `fixed:users:writer` assignments from the `user writers` team, if those assignments exist.

```yaml
# config file version
apiVersion: 2

# Roles to insert/update in the database
roles:
  - name: 'custom:users:writer'
    description: 'List/update other users in the organization'
    version: 1
    global: true
    permissions:
      - action: 'org.users:read'
        scope: 'users:*'
      - action: 'org.users:write'
        scope: 'users:*'

# Assignments to teams
teams:
  - name: 'user writers'
    orgId: 1
    roles:
      - name: 'fixed:users:writer'
        global: true
        state: 'absent' # Remove assignment
      - name: 'custom:users:writer'
        global: true
        state: 'absent' # Remove assignment
  - name: 'user admins'
    orgId: 1
    roles:
      - name: 'fixed:users:writer'
        global: true
      - name: 'custom:users:writer'
        global: true
```

> **Note**: The roles don't have to be defined in the provisioning configuration files to be revoked. If roles exist in the database, they can be revoked.

# Configure RBAC options in Grafana

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/docs/grafana-cloud).
{{< /admonition >}}

The table below describes the available RBAC configuration options for your Grafana stack. Like any other Grafana configuration, you can apply these options as [environment variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#override-configuration-with-environment-variables).

| Setting                         | Required | Description                                                                                                                                                                                                                                                                                                                     | Default |
| ------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `permission_cache`              | No       | Enable to use in memory cache for loading and evaluating users' permissions.                                                                                                                                                                                                                                                    | `true`  |
| `permission_validation_enabled` | No       | Grafana enforces validation for permissions when a user creates or updates a role. The system checks the internal list of scopes and actions for each permission to determine they are valid. By default, if a scope or action is not recognized, Grafana logs a warning message. When set to `true`, Grafana returns an error. | `true`  |
| `reset_basic_roles`             | No       | Reset Grafana's basic roles' (Viewer, Editor, Admin, Grafana Admin) permissions to their default. Warning, if this configuration option is left to `true` this will be done on every reboot.                                                                                                                                    | `true`  |

## Example RBAC configuration

```bash
[rbac]

permission_cache = true
```

# Create custom RBAC roles

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/docs/grafana-cloud/).
{{< /admonition >}}

Create a custom RBAC role when basic roles and fixed roles do not meet your permissions requirements. For a list of the actions and scopes you can customize a role with, refer to [Grafana RBAC permission actions and scopes](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/custom-role-actions-scopes/).

{{< admonition type="caution" >}}
**Before creating custom roles**, consider whether you can meet your access requirements using:

- **[Folder permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/folder-access-control/)**: Control access to dashboards, alert rules, and other resources by folder
- **[Fixed roles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/rbac-fixed-basic-role-definitions/)**: Pre-built roles for common access patterns

Use custom roles only when you need fine-grained control that these options don't provide.
{{< /admonition >}}

Creating and editing custom roles is not currently possible in the Grafana UI. Instead, use one of the following methods:

- The [RBAC HTTP API](#create-custom-roles-using-the-http-api)
- [Terraform](#create-custom-roles-using-terraform)
- [Grafana provisioning](#create-custom-roles-using-file-based-provisioning) using a YAML file

## Before you begin

Before you begin, keep in mind the following:

- [Plan your RBAC rollout strategy](ref:plan-rbac-rollout-strategy).
- Determine which permissions you want to add to the custom role. To see a list of actions and scope, refer to [RBAC permissions, actions, and scopes](ref:custom-role-actions-scopes).
- Ensure that you have permissions to create a custom role.
  - By default, the Grafana Admin role has permission to create custom roles.
  - A Grafana Admin can delegate the custom role privilege to another user by creating a custom role with the relevant permissions and adding the `permissions:type:delegate` scope.

## Create custom roles using the HTTP API

The following examples show you how to create a custom role using the Grafana HTTP API. For more information about the HTTP API, refer to [Create a new custom role](ref:api-rbac-create-a-new-custom-role).

{{< admonition type="note" >}}
When you create a custom role you can only give it the same permissions you already have. For example, if you only have `users:create` permissions, then you can't create a role that includes other permissions.
{{< /admonition >}}

The following example creates a `custom:users:admin` role and assigns the `users:create` action to it.

**Example request**

```
curl --location --request POST '<grafana_url>/api/access-control/roles/' \
--header 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \
--header 'Content-Type: application/json' \
--data-raw '{
    "version": 1,
    "uid": "jZrmlLCkGksdka",
    "name": "custom:users:admin",
    "displayName": "custom users admin",
    "description": "My custom role which gives users permissions to create users",
    "global": true,
    "permissions": [
        {
            "action": "users:create"
        }
    ]
}'
```

**Example response**

```
{
    "version": 1,
    "uid": "jZrmlLCkGksdka",
    "name": "custom:users:admin",
    "displayName": "custom users admin",
    "description": "My custom role which gives users permissions to create users",
    "global": true,
    "permissions": [
        {
            "action": "users:create",
            "updated": "2021-05-17T22:07:31.569936+02:00",
            "created": "2021-05-17T22:07:31.569935+02:00"
        }
    ],
    "updated": "2021-05-17T22:07:31.564403+02:00",
    "created": "2021-05-17T22:07:31.564403+02:00"
}
```

Refer to the [RBAC HTTP API](ref:api-rbac-create-a-new-custom-role) for more details.

## Create custom roles using Terraform

You can use the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) to manage custom roles and their assignments. This is the recommended method for Grafana Cloud users who want to manage RBAC as code. For more information, refer to [Provisioning RBAC with Terraform](ref:rbac-terraform-provisioning).

The following example creates a custom role and assigns it to a team:

```terraform
resource "grafana_role" "custom_folder_manager" {
  name        = "custom:folders:manager"
  description = "Custom role for reading and creating folders"
  uid         = "custom-folders-manager"
  version     = 1
  global      = true

  permissions {
    action = "folders:read"
    scope  = "folders:*"
  }

  permissions {
    action = "folders:create"
    scope  = "folders:uid:general" # Allows creating folders at the root level
  }
}

resource "grafana_role_assignment" "custom_folder_manager_assignment" {
  role_uid = grafana_role.custom_folder_manager.uid
  teams    = ["<TEAM_UID>"]
}
```

For more information, refer to the [`grafana_role`](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/role) and [`grafana_role_assignment`](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/role_assignment) documentation in the Terraform Registry.

## Create custom roles using file-based provisioning

You can use [file-based provisioning](ref:rbac-grafana-provisioning) to create custom roles for self-managed instances.

1. Open the YAML configuration file and locate the `roles` section.

1. Refer to the following table to add attributes and values.

| Attribute     | Description                                                                                                                                                                                                                                                                                                                                                                                                          |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`        | A human-friendly identifier for the role that helps administrators understand the purpose of a role. `name` is required and cannot be longer than 190 characters. We recommend that you use ASCII characters. Role names must be unique within an organization.                                                                                                                                                      |
| `uid`         | A unique identifier associated with the role. The UID enables you to change or delete the role. You can either generate a UID yourself, or let Grafana generate one for you. You cannot use the same UID within the same Grafana instance.                                                                                                                                                                           |
| `orgId`       | Identifies the organization to which the role belongs. The [default org ID](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#auto_assign_org_id) is used if you do not specify `orgId`.                                                                                                                                                                                                              |
| `global`      | Global roles are not associated with any specific organization, which means that you can reuse them across all organizations. This setting overrides `orgId`.                                                                                                                                                                                                                                                        |
| `displayName` | Human-friendly text that is displayed in the UI. Role display name cannot be longer than 190 ASCII-based characters. For fixed roles, the display name is shown as specified. If you do not set a display name the display name replaces `':'` (a colon) with `' '` (a space).                                                                                                                                       |
| `description` | Human-friendly text that describes the permissions a role provides.                                                                                                                                                                                                                                                                                                                                                  |
| `group`       | Organizes roles in the role picker.                                                                                                                                                                                                                                                                                                                                                                                  |
| `version`     | A positive integer that defines the current version of the role, which prevents overwriting newer changes.                                                                                                                                                                                                                                                                                                           |
| `hidden`      | Hidden roles do not appear in the role picker.                                                                                                                                                                                                                                                                                                                                                                       |
| `state`       | State of the role. Defaults to `present`, but if set to `absent` the role will be removed.                                                                                                                                                                                                                                                                                                                           |
| `force`       | Can be used in addition to state `absent`, to force the removal of a role and all its assignments.                                                                                                                                                                                                                                                                                                                   |
| `from`        | An optional list of roles from which you want to copy permissions.                                                                                                                                                                                                                                                                                                                                                   |
| `permissions` | Provides users access to Grafana resources. For a list of permissions, refer to [RBAC permissions actions and scopes](ref:rbac-role-definitions). If you do not know which permissions to assign, you can create and assign roles without any permissions as a placeholder. Using the `from` attribute, you can specify additional permissions or permissions to remove by adding a `state` to your permission list. |

1. Reload the provisioning configuration file.

   For more information about reloading the provisioning configuration at runtime, refer to [Reload provisioning configurations](/docs/grafana/<GRAFANA_VERSION>/developers/http_api/admin/#reload-provisioning-configurations).

The following example creates a local role:

```yaml
# config file version
apiVersion: 2

roles:
  - name: custom:users:writer
    description: 'List, create, or update other users.'
    version: 1
    orgId: 1
    permissions:
      - action: 'users:read'
        scope: 'global.users:*'
      - action: 'users:write'
        scope: 'global.users:*'
      - action: 'users:create'
```

The following example creates a hidden global role. The `global: true` option creates a global role, and the `hidden: true` option hides the role from the role picker.

```yaml
# config file version
apiVersion: 2

roles:
  - name: custom:users:writer
    description: 'List, create, or update other users.'
    version: 1
    global: true
    hidden: true
    permissions:
      - action: 'users:read'
        scope: 'global.users:*'
      - action: 'users:write'
        scope: 'global.users:*'
      - action: 'users:create'
```

The following example creates a global role based on other fixed roles. The `from` option contains the roles from which we want to
copy permissions. The permission `state: absent` option can be used to specify permissions to exclude from the copy.

```yaml
# config file version
apiVersion: 2

roles:
  - name: custom:org.users:writer
    description: 'List and remove other users from the organization.'
    version: 1
    global: true
    from:
      - name: 'fixed:org.users:reader'
        global: true
      - name: 'fixed:org.users:writer'
        global: true
    permissions:
      - action: 'org.users:write'
        scope: 'users:*'
        state: 'absent'
      - action: 'org.users:add'
        scope: 'users:*'
        state: 'absent'
```

## Delete a custom role using Grafana provisioning

Delete a custom role when you no longer need it. When you delete a custom role, the custom role is removed from users and teams to which it is assigned.

**Before you begin:**

- Identify the role or roles that you want to delete.
- Ensure that you have access to the YAML configuration file.

**To delete a custom role:**

1. Open the YAML configuration file and locate the `roles` section.

1. Refer to the following table to add attributes and values.

   | Attribute | Description                                                                                                                                |
   | --------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
   | `name`    | The name of the custom role you want to delete. You can specify a `uid` instead of a role name. The role `name` or the `uid` are required. |
   | `orgId`   | Identifies the organization to which the role belongs.                                                                                     |
   | `state`   | The state of the role set to `absent` to trigger its removal.                                                                              |
   | `force`   | When set to `true`, the roles are removed even if there are existing assignments.                                                          |

1. Reload the provisioning configuration file.

   For more information about reloading the provisioning configuration at runtime, refer to [Reload provisioning configurations](/docs/grafana/<GRAFANA_VERSION>/developers/http_api/admin/#reload-provisioning-configurations).

The following example deletes a custom role:

```yaml
# config file version
apiVersion: 2

roles:
  - name: 'custom:reports:editor'
    orgId: 1
    state: 'absent'
    force: true
```

You can also delete a custom role using the API. Refer to the [RBAC HTTP API](ref:api-rbac-delete-a-custom-role) for more details.

# Grafana RBAC permission actions and scopes

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/docs/grafana-cloud/).
{{< /admonition >}}

An RBAC permission comprises an action and a scope:

- **Action:** An action describes what tasks a user can perform on a resource.
- **Scope:** A scope describes where an action can be performed, such as reading a specific user profile. In this example, a permission is associated with the scope `users:<userId>` to the relevant role.

To learn more about the Grafana resources to which you can apply RBAC, refer to [Resources with RBAC permissions](ref:rbac-fixed-roles).

If you're using Grafana Enterprise or Grafana Cloud, you can create **custom roles** with specific sets of permissions. Refer to [Create custom roles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/create-custom-roles) to learn how.

## Action definitions

The following list contains role-based access control actions.

<!-- prettier-ignore-start -->
| Action                                | Applicable scopes                                                                                                   | Description                                                                                                                                                                                                               |
|---------------------------------------| ------------------------------------------------------------------------------------------------------------------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `alert.instances.external:read`       | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Read alerts and silences in data sources that support alerting.                                                                                                                                                           |
| `alert.instances.external:write`      | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Manage alerts and silences in data sources that support alerting.                                                                                                                                                         |
| `alert.instances:create`              | None                                                                                                                | Create silences in the current organization.                                                                                                                                                                              |
| `alert.instances:read`                | None                                                                                                                | Read alerts and silences in the current organization.                                                                                                                                                                     |
| `alert.instances:write`               | None                                                                                                                | Update and expire silences in the current organization.                                                                                                                                                                   |
| `alert.notifications.external:read`   | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Read templates, contact points, notification policies, and mute timings in data sources that support alerting.                                                                                                            |
| `alert.notifications.external:write`  | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Manage templates, contact points, notification policies, and mute timings in data sources that support alerting.                                                                                                          |
| `alert.notifications:write`           | None                                                                                                                | **Deprecated.** Manage templates, contact points, notification policies, and mute timings in the current organization. Use the granular actions (`alert.notifications.templates:write`, `alert.notifications.receivers:write`, etc.) instead. |
| `alert.notifications:read`            | None                                                                                                                | **Deprecated.** Read all templates, contact points, notification policies, and mute timings in the current organization. Use the granular actions (`alert.notifications.templates:read`, `alert.notifications.receivers:read`, etc.) instead. |
| `alert.rules.external:read`           | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Read alert rules in data sources that support alerting (Prometheus, Mimir, and Loki)                                                                                                                                      |
| `alert.rules.external:write`          | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Create, update, and delete alert rules in data sources that support alerting (Mimir and Loki).                                                                                                                            |
| `alert.rules:create`                  | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Create Grafana alert rules in a folder and its subfolders. Combine this permission with `folders:read` in a scope that includes the folder and `datasources:query` in the scope of data sources the user can query.       |
| `alert.rules:delete`                  | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Delete Grafana alert rules in a folder and its subfolders. Combine this permission with `folders:read` in a scope that includes the folder.       |
| `alert.rules:read`                    | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Read Grafana alert rules in a folder and its subfolders. Combine this permission with `folders:read` in a scope that includes the folder and `datasources:query` in the scope of data sources the user can query.         |
| `alert.rules:write`                   | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Update Grafana alert rules in a folder and its subfolders. Combine this permission with `folders:read` in a scope that includes the folder. To allow query modifications add `datasources:query` in the scope of data sources the user can query.       |
| `alert.silences:create`               | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Create rule-specific silences in a folder and its subfolders.                                                                                                                                                             |
| `alert.silences:read`                 | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Read all general silences and rule-specific silences in a folder and its subfolders.                                                                                                                                      |
| `alert.silences:write`                | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Update and expire rule-specific silences in a folder and its subfolders.                                                                                                                                                  |
| `alert.provisioning:read`             | None                                                                                                                | Read all Grafana alert rules, notification policies, etc via provisioning API. Permissions to folders and datasource are not required.                                                                                    |
| `alert.provisioning.secrets:read`     | None                                                                                                                | Same as `alert.provisioning:read` plus ability to export resources with decrypted secrets.                                                                                                                                |
| `alert.provisioning:write`            | None                                                                                                                | Update all Grafana alert rules, notification policies, etc via provisioning API. Permissions to folders and datasource are not required.                                                                                  |
| `alert.provisioning.provenance:write` | None                                                                                                                | Set provisioning status for alerting resources. Cannot be used alone. Requires user to have permissions to access resources                                                                                               |
| `annotations:create`                  | <ul><li>`annotations:*`</li><li>`annotations:type:*`</li><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                      | Create annotations.                                                                                                                                                                                                       |
| `annotations:delete`                  | <ul><li>`annotations:*`</li><li>`annotations:type:*`</li><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                      | Delete annotations.                                                                                                                                                                                                       |
| `annotations:read`                    | <ul><li>`annotations:*`</li><li>`annotations:type:*`</li><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                      | Read annotations and annotation tags.                                                                                                                                                                                     |
| `annotations:write`                   | <ul><li>`annotations:*`</li><li>`annotations:type:*`</li><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                      | Update annotations.                                                                                                                                                                                                       |
| `banners:write`                       | None                                                                                                                | Create [announcement banners](/docs/grafana-cloud/whats-new/2024-09-10-announcement-banner/).                                                                                                                             |
| `dashboards:create`                   | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Create dashboards in one or more folders and their subfolders.                                                                                                                                                            |
| `dashboards:delete`                   | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul>             | Delete one or more dashboards.                                                                                                                                                                                            |
| `dashboards.insights:read`            | None                                                                                                                | Read dashboard insights data and see presence indicators. To view insights, `dashboards:read` on the dashboard is also needed.                                                                                            |
| `dashboards.permissions:read`         | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul>             | Read permissions for one or more dashboards.                                                                                                                                                                              |
| `dashboards.permissions:write`        | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul>             | Update permissions for one or more dashboards.                                                                                                                                                                            |
| `dashboards:read`                     | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul>             | Read one or more dashboards.                                                                                                                                                                                              |
| `dashboards:write`                    | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul>             | Update one or more dashboards.                                                                                                                                                                                            |
| `dashboards.public:write`             | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li></ul>                                                         | Write shared dashboard configuration.                                                                                                                                                                                     |
| `datasources.caching:read`            | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Read data source query caching settings.                                                                                                                                                                                  |
| `datasources.caching:write`           | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Update data source query caching settings.                                                                                                                                                                                |
| `datasources:create`                  | None                                                                                                                | Create data sources.                                                                                                                                                                                                      |
| `datasources:delete`                  | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Delete data sources.                                                                                                                                                                                                      |
| `datasources:explore`                 | None                                                                                                                | Enable access to the **Explore** tab.                                                                                                                                                                                     |
| `datasources.id:read`                 | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Read data source IDs.                                                                                                                                                                                                     |
| `datasources.insights:read`           | None                                                                                                                | Read data sources insights data. To view insights, `datasources:read` on the datasource is also needed.                                                                                                                   |
| `datasources.permissions:read`        | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | List data source permissions.                                                                                                                                                                                             |
| `datasources.permissions:write`       | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Update data source permissions.                                                                                                                                                                                           |
| `datasources:query`                   | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Query data sources.                                                                                                                                                                                                       |
| `datasources:read`                    | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | List data sources.                                                                                                                                                                                                        |
| `datasources:write`                   | <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>                                                       | Update data sources.                                                                                                                                                                                                      |
| `featuremgmt.read`                    | None                                                                                                                | Read feature toggles.                                                                                                                                                                                                     |
| `featuremgmt.write`                   | None                                                                                                                | Write feature toggles.                                                                                                                                                                                                    |
| `folders.permissions:read`            | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Read permissions for one or more folders and their subfolders.                                                                                                                                                            |
| `folders.permissions:write`           | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Update permissions for one or more folders and their subfolders.                                                                                                                                                          |
| `folders:create`                      | <ul><li>`folders:*`</li><li>`folders:uid:*`</li><li>`folders:uid:general`</li></ul>                                 | Create folders or subfolders. If granted with scope `folders:uid:general`, it allows to create root level folders. Otherwise, it allows creating subfolders under the specified folders.                                  |
| `folders:delete`                      | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Delete one or more folders and their subfolders.                                                                                                                                                                          |
| `folders:read`                        | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Read one or more folders and their subfolders.                                                                                                                                                                            |
| `folders:write`                       | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Update one or more folders and their subfolders.                                                                                                                                                                          |
| `ldap.config:reload`                  | None                                                                                                                | Reload the LDAP configuration.                                                                                                                                                                                            |
| `ldap.status:read`                    | None                                                                                                                | Verify the availability of the LDAP server or servers.                                                                                                                                                                    |
| `ldap.user:read`                      | None                                                                                                                | Read users via LDAP.                                                                                                                                                                                                      |
| `ldap.user:sync`                      | None                                                                                                                | Sync users via LDAP.                                                                                                                                                                                                      |
| `library.panels:create`               | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                               | Create a library panel in one or more folders and their subfolders.                                                                                                                                                       |
| `library.panels:read`                 | <ul><li>`folders:*`</li><li>`folders:uid:*`</li><li>`library.panels:*`</li><li>`library.panels:uid:*`</li></ul>     | Read one or more library panels.                                                                                                                                                                                          |
| `library.panels:write`                | <ul><li>`folders:*`</li><li>`folders:uid:*`</li><li>`library.panels:*`</li><li>`library.panels:uid:*`</li></ul>     | Update one or more library panels.                                                                                                                                                                                        |
| `library.panels:delete`               | <ul><li>`folders:*`</li><li>`folders:uid:*`</li><li>`library.panels:*`</li><li>`library.panels:uid:*`</li></ul>     | Delete one or more library panels.                                                                                                                                                                                        |
| `licensing.reports:read`              | None                                                                                                                | Get custom permission reports.                                                                                                                                                                                            |
| `licensing:delete`                    | None                                                                                                                | Delete the license token.                                                                                                                                                                                                 |
| `licensing:read`                      | None                                                                                                                | Read licensing information.                                                                                                                                                                                               |
| `licensing:write`                     | None                                                                                                                | Update the license token.                                                                                                                                                                                                 |
| `migrationassistant:migrate`          | None                                                                                                                | Execute on-prem to cloud migrations through the Migration Assistant.   |
| `org.users:write`                     | <ul><li>`users:*`</li><li>`users:id:*`</li></ul>                                                                    | Update the organization role (`None`, `Viewer`, `Editor`, or `Admin`) of a user.                                                                                                                                          |
| `org.users:add`                       | <ul><li>`users:*`</li><li>`users:id:*`</li></ul>                                                                    | Add a user to an organization or invite a new user to an organization.                                                                                                                                                    |
| `org.users:read`                      | <ul><li>`users:*`</li><li>`users:id:*`</li></ul>                                                                    | Get user profiles within an organization.                                                                                                                                                                                 |
| `org.users:remove`                    | <ul><li>`users:*`</li><li>`users:id:*`</li></ul>                                                                    | Remove a user from an organization.                                                                                                                                                                                       |
| `orgs.preferences:read`               | None                                                                                                                | Read organization preferences.                                                                                                                                                                                            |
| `orgs.preferences:write`              | None                                                                                                                | Update organization preferences.                                                                                                                                                                                          |
| `orgs.quotas:read`                    | None                                                                                                                | Read organization quotas.                                                                                                                                                                                                 |
| `orgs.quotas:write`                   | None                                                                                                                | Update organization quotas.                                                                                                                                                                                               |
| `orgs:create`                         | None                                                                                                                | Create an organization.                                                                                                                                                                                                   |
| `orgs:delete`                         | None                                                                                                                | Delete one or more organizations.                                                                                                                                                                                         |
| `orgs:read`                           | None                                                                                                                | Read one or more organizations.                                                                                                                                                                                           |
| `orgs:write`                          | None                                                                                                                | Update one or more organizations.                                                                                                                                                                                         |
| `plugins.app:access`                  | <ul><li>`plugins:*`</li><li>`plugins:id:*`</li></ul>                                                                | Access one or more application plugins (still enforcing the organization role)                                                                                                                                            |
| `plugins:install`                     | None                                                                                                                | Install and uninstall plugins.                                                                                                                                                                                            |
| `plugins:write`                       | <ul><li>`plugins:*`</li><li>`plugins:id:*`</li></ul>                                                                | Edit settings for one or more plugins.                                                                                                                                                                                    |
| `provisioning:reload`                 | `provisioners:*`                                                                                                    | Reload provisioning files. To find the exact scope for specific provisioner, refer to [Scope definitions](#scope-definitions).                                                                                            |
| `reports:create`                      | None                                                                                                                | Create reports.                                                                                                                                                                                                           |
| `reports:write`                       | <ul><li>`reports:*`</li><li>`reports:id:*`</li></ul>                                                                | Update reports.                                                                                                                                                                                                           |
| `reports.settings:read`               | None                                                                                                                | Read report settings.                                                                                                                                                                                                     |
| `reports.settings:write`              | None                                                                                                                | Update report settings.                                                                                                                                                                                                   |
| `reports:delete`                      | <ul><li>`reports:*`</li><li>`reports:id:*`</li></ul>                                                                | Delete reports.                                                                                                                                                                                                           |
| `reports:read`                        | <ul><li>`reports:*`</li><li>`reports:id:*`</li></ul>                                                                | List all available reports or get a specific report.                                                                                                                                                                      |
| `reports:send`                        | <ul><li>`reports:*`</li><li>`reports:id:*`</li></ul>                                                                | Send a report email.                                                                                                                                                                                                      |
| `roles:delete`                        | <ul><li>`permissions:type:delegate`</li><ul>                                                                        | Delete a custom role.                                                                                                                                                                                                     |
| `roles:read`                          | <ul><li>`roles:*`</li><li>`roles:uid:*`</li></ul>                                                                   | List roles and read a specific role with its permissions.                                                                                                                                                                 |
| `roles:write`                         | <ul><li>`permissions:type:delegate`</li><ul>                                                                        | Create or update a custom role.                                                                                                                                                                                           |
| `roles:write`                         | <ul><li>`permissions:type:escalate`</li><ul>                                                                        | Reset basic roles to their default permissions.                                                                                                                                                                           |
| `secret.securevalues:create`          | <ul><li>`secret.securevalues:*`</li><li>                                                                            | Create secure values.                                                                                                                                                                                                     |
| `secret.securevalues:read`            | <ul><li>`secret.securevalues:*`</li><li>                                                                            | Read and list secure values.                                                                                                                                                                                              |
| `secret.securevalues:write`           | <ul><li>`secret.securevalues:*`</li><li>                                                                            | Update secure values.                                                                                                                                                                                                     |
| `secret.securevalues:delete`          | <ul><li>`secret.securevalues:*`</li><li>                                                                            | Delete secure values.                                                                                                                                                                                                     |
| `server.stats:read`                   | None                                                                                                                | Read Grafana instance statistics.                                                                                                                                                                                         |
| `server.usagestats.report:read`       | None                                                                                                                | View usage statistics report.                                                                                                                                                                                             |
| `serviceaccounts:write`               | <ul><li>`serviceaccounts:*`</li><ul>                                                                                | Create Grafana service accounts.                                                                                                                                                                                          |
| `serviceaccounts:create`              | None                                                                                                                | Update Grafana service accounts.                                                                                                                                                                                          |
| `serviceaccounts:delete`              | <ul><li>`serviceaccounts:*`</li><li>`serviceaccounts:id:*`</li></ul>                                                | Delete Grafana service accounts.                                                                                                                                                                                          |
| `serviceaccounts:read`                | <ul><li>`serviceaccounts:*`</li><li>`serviceaccounts:id:*`</li></ul>                                                | Read Grafana service accounts.                                                                                                                                                                                            |
| `serviceaccounts.permissions:write`   | <ul><li>`serviceaccounts:*`</li><li>`serviceaccounts:id:*`</li></ul>                                                | Update Grafana service account permissions to control who can do what with the service account.                                                                                                                           |
| `serviceaccounts.permissions:read`    | <ul><li>`serviceaccounts:*`</li><li>`serviceaccounts:id:*`</li></ul>                                                | Read Grafana service account permissions to see who can do what with the service account.                                                                                                                                 |
| `settings:read`                       | <ul><li>`settings:*`</li><li>`settings:auth.saml:*`</li><li>`settings:auth.saml:enabled`</li></ul> (property level) | Read the [Grafana configuration settings](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/)                                                                                                               |
| `settings:write`                      | <ul><li>`settings:*`</li><li>`settings:auth.saml:*`</li><li>`settings:auth.saml:enabled`</li></ul> (property level) | Update any Grafana configuration settings that can be [updated at runtime](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/settings-updates-at-runtime/).                                                 |
| `support.bundles:create`              | None                                                                                                                | Create support bundles.                                                                                                                                                                                                   |
| `support.bundles:delete`              | None                                                                                                                | Delete support bundles.                                                                                                                                                                                                   |
| `support.bundles:read`                | None                                                                                                                | List and download support bundles.                                                                                                                                                                                        |
| `snapshots:create`                    | None                                                                                                                | Create snapshots.                                                                                                                                                                                                         |
| `snapshots:delete`                    | None                                                                                                                | Delete snapshots.                                                                                                                                                                                                         |
| `snapshots:read`                      | None                                                                                                                | List snapshots.                                                                                                                                                                                                           |
| `status:accesscontrol`                | <ul><li>`services:accesscontrol`</li><ul>                                                                           | Get access-control enabled status.                                                                                                                                                                                        |
| `teams.permissions:read`              | <ul><li>`teams:*`</li><li>`teams:id:*`</li></ul>                                                                    | Read members and Team Sync setup for teams.                                                                                                                                                                               |
| `teams.permissions:write`             | <ul><li>`teams:*`</li><li>`teams:id:*`</li></ul>                                                                    | Add, remove and update members and manage Team Sync setup for teams.                                                                                                                                                      |
| `teams.roles:add`                     | <ul><li>`permissions:type:delegate`</li><ul>                                                                        | Assign a role to a team.                                                                                                                                                                                                  |
| `teams.roles:read`                    | <ul><li>`teams:*`</li><li>`teams:id:*`</li></ul>                                                                    | List roles assigned directly to a team.                                                                                                                                                                                   |
| `teams.roles:remove`                  | <ul><li>`permissions:type:delegate`</li><ul>                                                                        | Unassign a role from a team.                                                                                                                                                                                              |
| `teams:create`                        | None                                                                                                                | Create teams.                                                                                                                                                                                                             |
| `teams:delete`                        | <ul><li>`teams:*`</li><li>`teams:id:*`</li></ul>                                                                    | Delete one or more teams.                                                                                                                                                                                                 |
| `teams:read`                          | <ul><li>`teams:*`</li><li>`teams:id:*`</li></ul>                                                                    | Read one or more teams and team preferences. To list teams through the UI one of the following permissions is required in addition to `teams:read`: `teams:write`, `teams.permissions:read` or `teams.permissions:write`. |
| `teams:write`                         | <ul><li>`teams:*`</li><li>`teams:id:*`</li></ul>                                                                    | Update one or more teams and team preferences.                                                                                                                                                                            |
| `users.authtoken:read`                | <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>                                                      | List authentication tokens that are assigned to a user.                                                                                                                                                                   |
| `users.authtoken:write`               | <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>                                                      | Update authentication tokens that are assigned to a user.                                                                                                                                                                 |
| `users.password:write`                | <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>                                                      | Update a user’s password.                                                                                                                                                                                                 |
| `users.permissions:read`              | <ul><li>`users:*`</li><ul>                                                                                          | List permissions of a user.                                                                                                                                                                                               |
| `users.permissions:write`             | <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>                                                      | Update a user’s organization-level permissions.                                                                                                                                                                           |
| `users.quotas:read`                   | <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>                                                      | List a user’s quotas.                                                                                                                                                                                                     |
| `users.quotas:write`                  | <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>                                                      | Update a user’s quotas.                                                                                                                                                                                                   |
| `users.roles:add`                     | <ul><li>`permissions:type:delegate`</li><ul>                                                                        | Assign a role to a user or a service account.                                                                                                                                                                             |
| `users.roles:read`                    | <ul><li>`users:*`</li><ul>                                                                                          | List roles assigned directly to a user or a service account.                                                                                                                                                              |
| `users.roles:remove`                  | <ul><li>`permissions:type:delegate`</li><ul>                                                                        | Unassign a role from a user or a service account.                                                                                                                                                                         |
| `users:create`                        | None                                                                                                                | Create a user.                                                                                                                                                                                                            |
| `users:delete`                        | <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>                                                      | Delete a user.                                                                                                                                                                                                            |
| `users:disable`                       | <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>                                                      | Disable a user.                                                                                                                                                                                                           |
| `users:enable`                        | <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>                                                      | Enable a user.                                                                                                                                                                                                            |
| `users:logout`                        | <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>                                                      | Sign out a user.                                                                                                                                                                                                          |
| `users:read`                          | <ul><li>`global.users:*`</li><ul>                                                                                   | Read or search user profiles.                                                                                                                                                                                             |
| `users:write`                         | <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>                                                      | Update a user’s profile.                                                                                                                                                                                                  |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

### Grafana Alerting Notification action definitions

| Action                                          | Applicable scopes                                       | Description                                                                                                                                                                                                                                          |
| ----------------------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `alert.notifications.receivers:read`            | `receivers:*`<br>`receivers:uid:*`                      | Read contact points.                                                                                                                                                                                                                                 |
| `alert.notifications.receivers.secrets:read`    | `receivers:*`<br>`receivers:uid:*`                      | Export contact points with decrypted secrets.                                                                                                                                                                                                        |
| `alert.notifications.receivers:create`          | None                                                    | Create a new contact points. The creator is automatically granted full access to the created contact point.                                                                                                                                          |
| `alert.notifications.receivers:write`           | `receivers:*`<br>`receivers:uid:*`                      | Update existing contact points.                                                                                                                                                                                                                      |
| `alert.notifications.receivers.protected:write` | `receivers:*`<br>`receivers:uid:*`                      | Update [protected fields](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points#grafana-cloud-protected-fields) in contact points (such as target URLs for integrations). This scope only applies to Grafana Cloud. |
| `alert.notifications.receivers:delete`          | `receivers:*`<br>`receivers:uid:*`                      | Update and delete existing contact points.                                                                                                                                                                                                           |
| `alert.notifications.receivers:test`            | None                                                    | Test contact point notification. Deprecated. Use "alert.notifications.receivers.test:create"                                                                                                                                                         |
| `alert.notifications.receivers.test:create`     | `receivers:*`<br>`receivers:uid:*`<br>`receivers:uid:-` | Test contact points to verify their configuration. Use scope `receivers:uid:-` to grant permission to test new integrations                                                                                                                          |
| `receivers.permissions:read`                    | `receivers:*`<br>`receivers:uid:*`                      | Read permissions for contact points.                                                                                                                                                                                                                 |
| `receivers.permissions:write`                   | `receivers:*`<br>`receivers:uid:*`                      | Manage permissions for contact points.                                                                                                                                                                                                               |
| `alert.notifications.time-intervals:read`       | None                                                    | Read mute time intervals.                                                                                                                                                                                                                            |
| `alert.notifications.time-intervals:write`      | None                                                    | Create new or update existing mute time intervals.                                                                                                                                                                                                   |
| `alert.notifications.time-intervals:delete`     | None                                                    | Delete existing time intervals.                                                                                                                                                                                                                      |
| `alert.notifications.templates:read`            | None                                                    | Read templates.                                                                                                                                                                                                                                      |
| `alert.notifications.templates:write`           | None                                                    | Create new or update existing templates.                                                                                                                                                                                                             |
| `alert.notifications.templates:delete`          | None                                                    | Delete existing templates.                                                                                                                                                                                                                           |
| `alert.notifications.templates.test:write`      | None                                                    | Test templates with custom payloads (preview and payload editor functionality).                                                                                                                                                                      |
| `alert.notifications.routes:read`               | None                                                    | **Deprecated.** Read notification policies. Use `notifications.alerting.grafana.app/routingtrees:get` with an appropriate scope instead.                                                                                                             |
| `alert.notifications.routes:write`              | None                                                    | **Deprecated.** Create, update, or delete notification policies. Use `notifications.alerting.grafana.app/routingtrees:update` and `notifications.alerting.grafana.app/routingtrees:delete` with an appropriate scope instead.                        |

## Scope definitions

The following list contains role-based access control scopes.

<!-- prettier-ignore-start -->
| Scopes                                                               | Descriptions                                                                                                                                                                                                                                       |
| -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <ul><li>`annotations:*`</li><li>`annotations:type:*`</li></ul>       | Restrict an action to a set of annotations. For example, `annotations:*` matches any annotation, and `annotations:type:organization` matches organization annotations. |
| <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li></ul>          | Restrict an action to a set of dashboards. For example, `dashboards:*` matches any dashboard, and `dashboards:uid:1` matches the dashboard whose UID is `1`.                                                                                       |
| <ul><li>`datasources:*`</li><li>`datasources:uid:*`</li></ul>        | Restrict an action to a set of data sources. For example, `datasources:*` matches any data source, and `datasources:uid:1` matches the data source whose UID is `1`.                                                                               |
| <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                | Restrict an action to a set of folders. For example, `folders:*` matches any folder, and `folders:uid:1` matches the folder whose UID is `1`. Note that permissions granted to a folder cascade down to subfolders located under it.               |
| <ul><li>`global.users:*`</li><li>`global.users:id:*`</li></ul>       | Restrict an action to a set of global users. For example, `global.users:*` matches any user and `global.users:id:1` matches the user whose ID is `1`.                                                                                              |
| <ul><li>`library.panels:*`</li><li>`library.panels:uid:*`</li></ul>  | Restrict an action to a set of library panels. For example, `library.panels:*` matches any library panel, and `library.panel:uid:1` matches the library panel whose UID is `1`.                                                                    |
| <ul><li>`orgs:*`</li><li>`orgs:id:*`</li></ul>                       | Restrict an action to a set of organizations. For example, `orgs:*` matches any organization and `orgs:id:1` matches the organization whose ID is `1`.                                                                                             |
| <ul><li>`permissions:type:delegate`</li><ul>                         | The scope is only applicable for roles associated with the Access Control itself and indicates that you can delegate your permissions only, or a subset of it, by creating a new role or making an assignment.                                     |
| <ul><li>`permissions:type:escalate`</li><ul>                         | The scope is required to trigger the reset of basic roles permissions. It indicates that users might acquire additional permissions they did not previously have.                                                                                  |
| <ul><li>`plugins:*`</li><li>`plugins:id:*`</li></ul>                 | Restrict an action to a set of plugins. For example, `plugins:id:grafana-oncall-app` matches Grafana OnCall plugin, and `plugins:*` matches all plugins.                                                                                           |
| <ul><li>`provisioners:*`</li><ul>                                    | Restrict an action to a set of provisioners. For example, `provisioners:*` matches any provisioner, and `provisioners:accesscontrol` matches the role-based access control [provisioner](ref:rbac-grafana-provisioning).           |
| <ul><li>`reports:*`</li><li>`reports:id:*`</li></ul>                 | Restrict an action to a set of reports. For example, `reports:*` matches any report and `reports:id:1` matches the report whose ID is `1`.                                                                                                         |
| <ul><li>`roles:*`</li><li>`roles:uid:*`</li></ul>                    | Restrict an action to a set of roles. For example, `roles:*` matches any role and `roles:uid:randomuid` matches only the role whose UID is `randomuid`.                                                                                            |
| <ul><li>`services:accesscontrol`</li><ul>                            | Restrict an action to target only the role-based access control service. You can use this in conjunction with the `status:accesscontrol` actions.                                                                                                  |
| <ul><li>`serviceaccounts:*`</li><li>`serviceaccounts:id:*`</li></ul> | Restrict an action to a set of service account from an organization. For example, `serviceaccounts:*` matches any service account and `serviceaccount:id:1` matches the service account whose ID is `1`.                                           |
| <ul><li>`settings:*`</li><ul>                                        | Restrict an action to a subset of settings. For example, `settings:*` matches all settings, `settings:auth.saml:*` matches all SAML settings, and `settings:auth.saml:enabled` matches the enable property on the SAML settings.                   |
| <ul><li>`teams:*`</li><li>`teams:id:*`</li></ul>                     | Restrict an action to a set of teams from an organization. For example, `teams:*` matches any team and `teams:id:1` matches the team whose ID is `1`.                                                                                              |
| <ul><li>`users:*`</li><li>`users:id:*`</li></ul>                     | Restrict an action to a set of users from an organization. For example, `users:*` matches any user and `users:id:1` matches the user whose ID is `1`.                                                                                              |
| <ul><li>None</li><ul>                                                | If an action has "None" specified for the scope, then the action doesn't require a scope. For example, the `teams:create` action doesn't require a scope and allows users to create teams.                                                         |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

## Discover plugin actions

The action definitions table above lists actions for core Grafana features. App plugins can define their own actions, which follow the pattern `<plugin-id>.<resource>:<operation>`.

To discover which actions a plugin supports, query an existing role that has plugin permissions. For example, to see what actions are available for a plugin, you can query the basic Admin role:

```bash
curl -X GET "https://your-grafana-instance/api/access-control/roles/basic_admin" \
  -H "Authorization: Bearer <your-service-account-token>"
```

The response includes all permissions granted to that role, including plugin-specific actions. Plugin actions typically use `None` for their scope because they operate at the organization level.

For a centralized reference of plugin roles and their default permissions, refer to [Grafana Cloud app plugin role definitions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/plugin-role-definitions/).

# Manage RBAC roles

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/docs/grafana-cloud/).
{{< /admonition >}}

This section includes instructions for how to view permissions associated with roles, create custom roles, and update and delete roles.

## View basic role definitions

You can retrieve the full definition of a basic role, including all associated permissions, using the API or by navigating directly to the endpoint URL in your browser while logged in as an Admin.

### Using the API

To get the definition of a basic role:

```bash
GET /api/access-control/roles/basic_<role>
```

Where `<role>` is one of: `viewer`, `editor`, `admin`, or `grafana_admin`.

For example, to get the Viewer role definition:

```bash
curl --location 'https://<your-stack-name>.grafana.net/api/access-control/roles/basic_viewer' \
  --header 'Authorization: Bearer <service-account-token>'
```

### Using the browser

You can also view the role definition directly in your browser by navigating to:

```
https://<your-stack-name>.grafana.net/api/access-control/roles/basic_viewer
```

This works when logged in as an Admin user.

For more information, refer to [Get a role](ref:api-rbac-get-a-role).

For a reference of basic and fixed role assignments, refer to [RBAC role definitions](ref:rbac-role-definitions).

## Update role permissions

If the default basic role permissions don't meet your requirements you can change them.

Basic roles are mutable. You can add or remove individual permissions on the `Viewer`, `Editor`, `Admin`, and `Grafana Admin` roles without creating a custom role. Each permission is a combination of an `action` and a `scope`. For example, you can remove access to a specific app plugin from all viewers by removing the relevant permissions from the `basic:viewer` role.

Before you change basic role permissions, decide which roles to modify and how the change affects your users and teams. For planning guidance, refer to [Plan your RBAC rollout strategy](ref:plan-rbac-rollout-strategy).

{{< admonition type="caution" >}}
Changes that you make to a basic role apply to every [organization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/organization-management/) in the Grafana instance. For example, if you add the `fixed:users:writer` role's permissions to the `Viewer` basic role, all viewers in every organization in the instance can create users.

Basic role changes are scoped to a single Grafana instance. On Grafana Cloud, each stack is a separate instance, so you must apply the change to each stack individually.
{{< /admonition >}}

{{< admonition type="note" >}}
If you only need to change access for specific users or teams rather than for every viewer, editor, or admin, leave the basic roles unchanged. Instead, assign the `No Basic Role` organization role and grant fixed or custom roles to those users or teams. For more information, refer to [Assign RBAC roles](ref:assign-rbac-roles).
{{< /admonition >}}

You can change basic roles' permissions [via the configuration file](#update-basic-role-permissions-in-the-configuration-file) or [using the RBAC API](#update-basic-role-permissions-using-the-rbac-api).

### Update basic role permissions in the configuration file

Before you begin, determine the permissions you want to add or remove from a basic role. For more information about the permissions associated with basic roles, refer to [RBAC role definitions](ref:rbac-fixed-basic-role-definitions-basic-role-assignments).

{{< admonition type="note" >}}
You cannot modify the `None` permissions.
{{< /admonition >}}

To change permissions for a basic role:

1. Open the YAML configuration file and locate the `roles` section.

1. Refer to the following table to add attributes and values.

   | Attribute             | Description                                                                                                                                               |
   | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | `name`                | The name of the basic role you want to update. You can specify a `uid` instead of a role name. The role `name` or the `uid` are required.                 |
   | `orgId`               | Identifies the organization to which the role belongs. `global` can be used instead to specify it's a global role.                                        |
   | `version`             | Identifies the version of the role, which prevents overwriting newer changes.                                                                             |
   | `overrideRole`        | If set to true, role will be updated regardless of its version in the database. There is no need to specify `version` if `overrideRole` is set to `true`. |
   | `from`                | List of roles from which to copy permissions.                                                                                                             |
   | `permissions > state` | The state of the permission. You can set it to `absent` to ensure it exclusion from the copy list.                                                        |

1. Reload the provisioning configuration file. For more information about reloading the provisioning configuration at runtime, refer to [Reload provisioning configurations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/admin/#reload-provisioning-configurations).

#### Example: Modify the Grafana Admin role

The following example modifies the `Grafana Admin` basic role permissions.

In the new configuration:

- Permissions to list, grant, and revoke roles to teams are removed.
- Permission to read and write Grafana folders is added.

```yaml
# config file version
apiVersion: 2

roles:
  - name: 'basic:grafana_admin'
    global: true
    version: 3
    from:
      - name: 'basic:grafana_admin'
        global: true
    permissions:
      # Permissions to remove
      - action: 'teams.roles:read'
        scope: 'teams:*'
        state: 'absent'
      - action: 'teams.roles:remove'
        scope: 'permissions:type:delegate'
        state: 'absent'
      - action: 'teams.roles:add'
        scope: 'permissions:type:delegate'
        state: 'absent'
      # Permissions to add
      - action: 'folders:read'
        scope: 'folder:*'
      - action: 'folders:write'
        scope: 'folder:*'
```

{{< admonition type="note" >}}
You can add multiple `fixed`, `basic` or `custom` roles to the `from` section. Their permissions will be copied and added to the basic role.
Make sure to **increment** the role version for the changes to be accounted for.
{{< /admonition >}}

### Update basic role permissions using the RBAC API

Use the RBAC HTTP API when you want to change basic role permissions programmatically, for example from a script or a CI pipeline. The API uses two endpoints:

- `GET /api/access-control/roles/{roleUID}` returns a basic role and all of its permissions. Refer to [Get a role](ref:api-rbac-get-a-role).
- `PUT /api/access-control/roles/{roleUID}` replaces the role's permissions with the set that you send. Refer to [Update a role](ref:api-rbac-update-a-role).

Basic role UIDs follow the pattern `basic_<role>`, where `<role>` is `viewer`, `editor`, `admin`, or `grafana_admin`.

Because `PUT` replaces the entire permission set, fetch the current role first, modify its permissions, increment its `version`, and then send the result back. Each request authenticates with a [service account token](ref:service-accounts) that has the `Role writer` role, or with basic authentication as a `Grafana Admin`.

{{< admonition type="note" >}}
Send `global: true` in the request body when you update a global basic role, and set `version` to a value higher than the current version. Grafana rejects updates that don't increase the version.
{{< /admonition >}}

#### Example: Modify the Grafana Admin role using the API

The following example makes the same change as the [configuration file example](#example-modify-the-grafana-admin-role) using the API instead. It removes the permissions to list, grant, and revoke roles to teams, and adds permissions to read and write Grafana folders.

The script fetches the `basic:grafana_admin` role, removes and adds permissions with `jq`, increments the version, and updates the role:

```bash
# Fetch the role, modify its permissions, and increment its version
curl -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  -X GET '<GRAFANA_URL>/api/access-control/roles/basic_grafana_admin' | \
  jq 'del(.created) | del(.updated) | del(.permissions[].created) | del(.permissions[].updated) | .version += 1' | \
  jq 'del(.permissions[] | select(.action == "teams.roles:read")) | del(.permissions[] | select(.action == "teams.roles:add")) | del(.permissions[] | select(.action == "teams.roles:remove"))' | \
  jq '.permissions += [{"action": "folders:read", "scope": "folders:*"}, {"action": "folders:write", "scope": "folders:*"}]' > basic_grafana_admin.json

# Update the role with the modified permissions
curl -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" -H "Content-Type: application/json" \
  -X PUT -d @basic_grafana_admin.json \
  '<GRAFANA_URL>/api/access-control/roles/basic_grafana_admin'
```

Replace the following placeholders:

- _`<SERVICE_ACCOUNT_TOKEN>`_: A service account token with permission to read and write roles, for example `glsa_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`. To use basic authentication instead, replace the `Authorization` header with `-u admin:<PASSWORD>`.
- _`<GRAFANA_URL>`_: The base URL of your Grafana instance, for example `https://<YOUR_STACK_NAME>.grafana.net` on Grafana Cloud or `http://localhost:3000` for a local instance.

The `jq` filters first strip the read-only `created` and `updated` timestamps and increment the role `version`, then remove the unwanted permissions, and finally append the new ones. The `PUT` request sends the resulting role definition back to Grafana.

#### Example: Remove Grafana Assistant access from Viewers

By default, viewers can access all app plugins that their organization role allows, including Grafana Assistant. To prevent viewers from accessing Grafana Assistant, remove the following permissions from the `basic:viewer` role:

| Action                               | Scope                              |
| ------------------------------------ | ---------------------------------- |
| `grafana-assistant-app.chats:access` |                                    |
| `plugins.app:access`                 | `plugins:id:grafana-assistant-app` |

You can remove these permissions with the configuration file or with the API.

Use the `role > from` list and `permission > state` option of your provisioning file to remove the permissions:

```yaml
apiVersion: 2

roles:
  - name: 'basic:viewer'
    global: true
    version: 9
    from:
      - name: 'basic:viewer'
        global: true
    permissions:
      - action: 'grafana-assistant-app.chats:access'
        state: 'absent'
      - action: 'plugins.app:access'
        scope: 'plugins:id:grafana-assistant-app'
        state: 'absent'
```

Alternatively, use the RBAC HTTP API. The following script fetches the `basic:viewer` role, removes the two Grafana Assistant permissions, increments the version, and updates the role:

```bash
# Fetch the role, remove the Grafana Assistant permissions, and increment its version
curl -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  -X GET '<GRAFANA_URL>/api/access-control/roles/basic_viewer' | \
  jq 'del(.created) | del(.updated) | del(.permissions[].created) | del(.permissions[].updated) | .version += 1' | \
  jq 'del(.permissions[] | select(.action == "grafana-assistant-app.chats:access")) | del(.permissions[] | select(.action == "plugins.app:access" and .scope == "plugins:id:grafana-assistant-app"))' > basic_viewer.json

# Update the role with the modified permissions
curl -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" -H "Content-Type: application/json" \
  -X PUT -d @basic_viewer.json \
  '<GRAFANA_URL>/api/access-control/roles/basic_viewer'
```

Replace the following placeholders:

- _`<SERVICE_ACCOUNT_TOKEN>`_: A service account token with permission to read and write roles.
- _`<GRAFANA_URL>`_: The base URL of your Grafana instance.

To find the IDs of other app plugins, refer to [RBAC for app plugins](ref:rbac-for-app-plugins). That page also explains how to grant fine-grained app plugin access to specific users and teams without changing the basic roles.

## Reset basic roles to their default

You have two options to reset the basic roles permissions to their default.

### Use the configuration option

> **Note**: Available as of Grafana Enterprise 9.4.

> Warning: If this option is left to true, permissions will be reset on every boot.

Use the [`reset_basic_roles`](ref:configure-rbac-configure-rbac-in-grafana) option to reset basic roles permissions to their default on Grafana instance boot up.

1. Open you configuration file and update the rbac section as follow:

```bash
[rbac]
reset_basic_roles = true
```

### Use the HTTP endpoint

An alternative to the configuration option is to use the HTTP endpoint.

1. Open the YAML configuration file and locate the `roles` section.

1. Grant the `action: "roles:write", scope: "permissions:type:escalate` permission to `Grafana Admin`. Note that this permission has not been granted to any basic roles by default, because users could acquire more permissions than they previously had through the basic role permissions reset.

   ```yaml
   apiVersion: 2
   roles:
     - name: 'basic:grafana_admin'
       global: true
       version: 3
       from:
         - name: 'basic:grafana_admin'
           global: true
       permissions:
         # Permission allowing to reset basic roles
         - action: 'roles:write'
           scope: 'permissions:type:escalate'
   ```

1. As a `Grafana Admin`, call the API endpoint to reset the basic roles to their default. Refer to the [RBAC HTTP API](ref:api-rbac-reset-basic-roles-to-their-default) for more details.

# Plan your RBAC rollout strategy

{{< admonition type="note" >}}
Available in [Grafana Enterprise](/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](/docs/grafana-cloud).
{{< /admonition >}}

An RBAC rollout strategy helps you determine _how_ you want to implement RBAC prior to assigning RBAC roles to users and teams.

Your rollout strategy should help you answer the following questions:

- Should I assign basic roles to users, or should I assign fixed roles or custom roles to users?
- When should I create custom roles?
- To which entities should I apply fixed and custom roles? Should I apply them to users, teams? Should I modify the basic roles permissions instead?
- How do I roll out permissions in a way that makes them easy to manage?
- Which approach should I use when assigning roles? Should I use the Grafana UI, provisioning, or the API?

## Review basic role and fixed role definitions

As a first step in determining your permissions rollout strategy, we recommend that you become familiar with basic role and fixed role definitions. In addition to assigning fixed roles to any user and team, you can also modify basic roles permissions, which changes what a Viewer, Editor, or Admin can do. This flexibility means that there are many combinations of role assignments for you to consider. If you have a large number of Grafana users and teams, we recommend that you make a list of which fixed roles you might want to use. Keep in mind that the `None` basic role, which is a role without permissions, cannot be modified or updated.

To learn more about basic roles and fixed roles, refer to the following documentation:

- [Basic role definitions](ref:rbac-fixed-basic-role-definitions-basic-role-assignments)
- [Fixed role definitions](ref:rbac-fixed-basic-role-definitions-fixed-role-definitions)

## User and team considerations

RBAC is a flexible and powerful feature with many possible permissions assignment combinations available. Consider the follow guidelines when assigning permissions to users and teams.

- **Assign roles to users** when you have a one-off scenario where a small number of users require access to a resource or when you want to assign temporary access. If you have a large number of users, this approach can be difficult to manage as you scale your use of Grafana. For example, a member of your IT department might need the `fixed:licensing:reader` and `fixed:licensing:writer` roles so that they can manage your Grafana Enterprise license.

- **Assign roles to teams** when you have a subset of users that align to your organizational structure, and you want all members of the team to have the same level of access. For example, all members of a particular engineering team might need the `fixed:reports:reader` and `fixed:reports:writer` roles to be able to manage reports.

  When you assign additional users to a team, the system automatically assigns permissions to those users.

### Authentication provider considerations

You can take advantage of your current authentication provider to manage user and team permissions in Grafana. When you map users and teams to SAML and LDAP groups, you can synchronize those assignments with Grafana.

For example:

1. Map SAML, LDAP, or Oauth roles to Grafana basic roles (viewer, editor, or admin).

2. Use the Grafana Enterprise team sync feature to synchronize teams from your SAML, LDAP, or OAuth provider to Grafana. For more information about team sync, refer to [Team sync](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/).

3. Within Grafana, assign RBAC permissions to users and teams.

## When to modify basic roles or create custom roles

Consider the following guidelines when you determine if you should modify basic roles or create custom roles.

- **Modify basic roles** when Grafana's definitions of what viewers, editors, and admins can do does not match your definition of these roles. You can add or remove permissions from any basic role.

  {{< admonition type="note" >}}
  Changes that you make to basic roles impact the role definition for all [organizations](/docs/grafana/<GRAFANA_VERSION>/administration/organization-management/) in the Grafana instance. For example, when you add the `fixed:users:writer` role's permissions to the viewer basic role, all viewers in any org in the Grafana instance can create users within that org.
  {{< /admonition >}}

  {{< admonition type="note" >}}
  You cannot modify the `No Basic Role` permissions.
  {{< /admonition >}}

- **Create custom roles** when fixed role definitions don't meet you permissions requirements. For example, the `fixed:dashboards:writer` role allows users to delete dashboards. If you want some users or teams to be able to create and update but not delete dashboards, you can create a custom role with a name like `custom:dashboards:creator` that lacks the `dashboards:delete` permission.

## How to assign RBAC roles

Use any of the following methods to assign RBAC roles to users and teams.

- **Grafana UI:** Use the Grafana UI when you want to assign a limited number of RBAC roles to users and teams. The UI contains a role picker that you can use to select roles.
- **Grafana HTTP API:** Use the Grafana HTTP API if you would like to automate role assignment.
- **Terraform:** Use Terraform to assign and manage user and team role assignments if you use Terraform for provisioning.
- **Grafana provisioning:** Grafana provisioning provides a robust approach to assigning, removing, and deleting roles. Within a single YAML file you can include multiple role assignment and removal entries.

## Permissions scenarios

We've compiled the following permissions rollout scenarios based on current Grafana implementations.

{{< admonition type="note" >}}
If you have a use case that you'd like to share, feel free to contribute to this docs page. We'd love to hear from you!
{{< /admonition >}}

### Provide internal viewer employees with the ability to use Explore, but prevent external viewer contractors from using Explore

1. In Grafana, create a team with the name `Internal employees`.
1. Assign the `fixed:datasources:explorer` role to the `Internal employees` team.
1. Add internal employees to the `Internal employees` team, or map them from a SAML, LDAP, or OAuth team using [Team Sync](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/).
1. Assign the viewer role to both internal employees and contractors.

### Limit viewer, editor, or admin permissions

1. Review the list of permissions associated with the basic role.
1. [Change the permissions of the basic role](ref:manage-rbac-roles-update-basic-role-permissions).

### Allow only members of one team to manage Alerts

1. Create an `Alert Managers` team, and assign that team all applicable Alerting fixed roles.
1. Add users to the `Alert Managers` team.
1. Remove all permissions with actions prefixed with `alert.` from the Viewer, Editor, and Admin basic roles.

### Provide dashboards to users in two or more geographies

1. Create a folder for each geography, for example, create a `US` folder and an `EU` folder.
1. Add dashboards to each folder.
1. Use folder permissions to add US-based users as Editors to the `US` folder and assign EU-based users as Editors to the `EU` folder.

### Assign a user specific set of roles

1. Create a user with the `No Basic Role` selected under organization roles.
1. Assign the user a set of fixed roles that meet your requirements.

### Create a custom role to access alerts in a specific folder

To see an alert rule in Grafana, the user must have read access to the folder that stores the alert rule, permission to read alerts in the folder, and permission to query all data sources that the rule uses.

The API command in this example is based on the following:

- A `Test-Folder` with ID `92`
- Two data sources: `DS1` with UID `_oAfGYUnk`, and `DS2` with UID `YYcBGYUnk`
- An alert rule that is stored in `Test-Folder` and queries the two data sources.

The following request creates a custom role that includes permissions to access the alert rule:

```
curl --location --request POST '<grafana_url>/api/access-control/roles/' \
--header 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697' \
--header 'Content-Type: application/json' \
--data-raw '{
    "version": 1,
    "name": "custom:alerts.reader.in.folder.123",
    "displayName": "Read-only access to alerts in folder Test-Folder",
    "description": "Let user query DS1 and DS2, and read alerts in folder Test-Folders",
    "group":"Custom",
    "global": false,
    "permissions": [
        {
            "action": "folders:read",
            "scope": "folders:uid:YEcBGYU22"
        },
        {
            "action": "alert.rules:read",
            "scope": "folders:uid:YEcBGYU22"
        },
        {
            "action": "datasources:query",
            "scope": "datasources:uid:_oAfGYUnk"
        },
        {
            "action": "datasources:query",
            "scope": "datasources:uid:YYcBGYUnk"
        }
    ]
}'
```

### Enable an editor to create custom roles

By default, only a Grafana Server Admin can create and manage custom roles. If you want your `Editors` to do the same, [update the `Editor` basic role permissions](ref:manage-rbac-roles-update-basic-role-permissions). There are two ways to achieve this:

- Add the following permissions to the `basic:editor` role, using provisioning or the [RBAC HTTP API](ref:api-rbac-update-a-role):

  | action         | scope                       |
  | -------------- | --------------------------- |
  | `roles:read`   | `roles:*`                   |
  | `roles:write`  | `permissions:type:delegate` |
  | `roles:delete` | `permissions:type:delegate` |

  As an example, here is a small bash script that fetches the role, modifies it using `jq` and updates it:

  ```bash
  # Fetch the role, modify it to add the desired permissions and increment its version
  curl -H 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697' \
    -X GET '<grafana_url>/api/access-control/roles/basic_editor' | \
    jq 'del(.created)| del(.updated) | del(.permissions[].created) | del(.permissions[].updated) | .version += 1' | \
    jq '.permissions += [{"action": "roles:read", "scope": "roles:*"}, {"action": "roles:write", "scope": "permissions:type:delegate"}, {"action": "roles:delete", "scope": "permissions:type:delegate"}]' > /tmp/basic_editor.json

  # Update the role
  curl -H 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697' -H 'Content-Type: application/json' \
    -X PUT-d @/tmp/basic_editor.json '<grafana_url>/api/access-control/roles/basic_editor'
  ```

- Or add the `fixed:roles:writer` role permissions to the `basic:editor` role using the `role > from` list of your provisioning file:

  ```yaml
  apiVersion: 2

  roles:
    - name: 'basic:editor'
      global: true
      version: 3
      from:
        - name: 'basic:editor'
          global: true
        - name: 'fixed:roles:writer'
          global: true
  ```

> **Note:** Any user or service account with the ability to modify roles can only create, update, or delete roles with permissions they have been granted. For example, a user with the `Editor` role would be able to create and manage roles only with the permissions they have or with a subset of them.

### Enable viewers to create reports

If you want your `Viewers` to create reports, [update the `Viewer` basic role permissions](ref:manage-rbac-roles-update-basic-role-permissions). There are two ways to achieve this:

- Add the following permissions to the `basic:viewer` role, using provisioning or the [RBAC HTTP API](ref:api-rbac-update-a-role):

  | Action           | Scope                           |
  | ---------------- | ------------------------------- |
  | `reports:create` | n/a                             |
  | `reports:write`  | `reports:*` <br> `reports:id:*` |
  | `reports:read`   | `reports:*`                     |
  | `reports:send`   | `reports:*`                     |

  As an example, here is a small bash script that fetches the role, modifies it using `jq` and updates it:

  ```bash
  # Fetch the role, modify it to add the desired permissions and increment its version
  curl -H 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697' \
    -X GET '<grafana_url>/api/access-control/roles/basic_viewer' | \
    jq 'del(.created)| del(.updated) | del(.permissions[].created) | del(.permissions[].updated) | .version += 1' | \
    jq '.permissions += [{"action": "reports:create"}, {"action": "reports:read", "scope": "reports:*"}, {"action": "reports:write", "scope": "reports:*"}, {"action": "reports:send", "scope": "reports:*"}]' > /tmp/basic_viewer.json

  # Update the role
  curl -H 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697' -H 'Content-Type: application/json' \
    -X PUT-d @/tmp/basic_viewer.json '<grafana_url>/api/access-control/roles/basic_viewer'
  ```

- Or add the `fixed:reports:writer` role permissions to the `basic:viewer` role using the `role > from` list of your provisioning file:

  ```yaml
  apiVersion: 2

  roles:
    - name: 'basic:viewer'
      global: true
      version: 3
      from:
        - name: 'basic:viewer'
          global: true
        - name: 'fixed:reports:writer'
          global: true
  ```

> **Note:** The `fixed:reports:writer` role assigns more permissions than just creating reports. For more information about fixed role permission assignments, refer to [Fixed role definitions](ref:rbac-fixed-basic-role-definitions-fixed-role-definitions).

### Prevent a Grafana Admin from creating and inviting users

To prevent a Grafana Admin from creating users and inviting them to join an organization, you must [update a basic role permission](ref:manage-rbac-roles-update-basic-role-permissions).
The permissions to remove are:

| Action          | Scope     |
| --------------- | --------- |
| `users:create`  |           |
| `org.users:add` | `users:*` |

There are two ways to achieve this:

- Use [RBAC HTTP API](ref:api-rbac-update-a-role).

  As an example, here is a small bash script that fetches the role, modifies it using `jq` and updates it:

  ```bash
  # Fetch the role, modify it to remove the undesired permissions and increment its version
  curl -H 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697' \
    -X GET '<grafana_url>/api/access-control/roles/basic_grafana_admin' | \
    jq 'del(.created)| del(.updated) | del(.permissions[].created) | del(.permissions[].updated) | .version += 1' | \
    jq 'del(.permissions[] | select (.action == "users:create")) | del(.permissions[] | select (.action == "org.users:add" and .scope == "users:*"))' > /tmp/basic_grafana_admin.json

  # Update the role
  curl -H 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697' -H 'Content-Type: application/json' \
    -X PUT-d @/tmp/basic_grafana_admin.json '<grafana_url>/api/access-control/roles/basic_grafana_admin'
  ```

- Or use the `role > from` list and `permission > state` option of your provisioning file:

  ```yaml
  apiVersion: 2

  roles:
    - name: 'basic:grafana_admin'
      global: true
      version: 3
      from:
        - name: 'basic:grafana_admin'
          global: true
      permissions:
        - action: 'users:create'
          state: 'absent'
        - action: 'org.users:add'
          scope: 'users:*'
          state: 'absent'
  ```

### Prevent Viewers from accessing an App Plugin

By default, Viewers, Editors and Admins have access to all App Plugins that their organization role allows them to access.
To change this default behavior and prevent Viewers from accessing an App plugin, you must [update a basic role's permissions](ref:manage-rbac-roles-update-basic-role-permissions).

In this example, three App plugins have been installed and enabled:
| Name | ID | Required Org role |
|--------------------|-----------------------------|-------------------|
| On Call | grafana-oncall-app | Viewer |
| Kentik Connect Pro | kentik-connect-app | Viewer |
| Enterprise logs | grafana-enterprise-logs-app | Admin |

By default, Viewers will hence be able to see both, On Call and Kentik Connect Pro App plugins.
If you want to revoke their access to the On Call App plugin, you need to:

1. Remove the permission to access all application plugins:
   | Action | Scope |
   |----------------------|-------------|
   | `plugins.app:access` | `plugins:*` |
1. Grant the permission to access the Kentik Connect Pro App plugin only:
   | Action | Scope |
   |----------------------|---------------------------------|
   | `plugins.app:access` | `plugins:id:kentik-connect-app` |

Here are two ways to achieve this:

- Use [RBAC HTTP API](ref:api-rbac-update-a-role).

  As an example, here is a small bash script that fetches the role, modifies it using `jq` and updates it:

  ```bash
  # Fetch the role, modify it to remove the undesired permissions, add the new permission and increment its version
  curl -H 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697' \
    -X GET '<grafana_url>/api/access-control/roles/basic_viewer' | \
    jq 'del(.created)| del(.updated) | del(.permissions[].created) | del(.permissions[].updated) | .version += 1' | \
    jq 'del(.permissions[] | select (.action == "plugins.app:access" and .scope == "plugins:*"))' | \
    jq '.permissions += [{"action": "plugins.app:access", "scope": "plugins:id:kentik-connect-app"}]' > /tmp/basic_viewer.json

  # Update the role
  curl -H 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697' -H 'Content-Type: application/json' \
    -X PUT -d @/tmp/basic_viewer.json '<grafana_url>/api/access-control/roles/basic_viewer'
  ```

  The token that is used in this request is the [service account token](ref:service-accounts).

- Or use the `role > from` list and `permission > state` option of your provisioning file:

  ```yaml
  ---
  apiVersion: 2

  roles:
    - name: 'basic:viewer'
      version: 8
      global: true
      from:
        - name: 'basic:viewer'
          global: true
      permissions:
        - action: 'plugins.app:access'
          scope: 'plugins:*'
          state: 'absent'
        - action: 'plugins.app:access'
          scope: 'plugins:id:kentik-connect-app'
          state: 'present'
  ```

  If your goal is to remove an access to an app you should remove it from the role and update it. For example:

  ```bash
  # Fetch the role, modify it to remove permissions to kentik-connect-app and increment role version
  curl -H 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697' \
    -X GET '<grafana_url>/api/access-control/roles/basic_viewer' | \
    jq 'del(.created)| del(.updated) | del(.permissions[].created) | del(.permissions[].updated) | .version += 1' | \
    jq 'del(.permissions[] | select (.action == "plugins.app:access" and .scope == "plugins:id:kentik-connect-app"))'

  # Update the role
  curl -H 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697' -H 'Content-Type: application/json' \
    -X PUT -d @/tmp/basic_viewer.json '<grafana_url>/api/access-control/roles/basic_viewer'
  ```

### Manage user permissions through teams

In the scenario where you want users to grant access by the team they belong to, we recommend to set users role to `No Basic Role` and let the team assignment assign the role instead.

1. In Grafana, ensure the following configuration settings are enabled.

   ```
   [users]
   # Set to true to automatically assign new users to the default organization (id 1)
   auto_assign_org = true

   # Set this value to automatically add new users to the provided organization (if auto_assign_org above is set to true)
   auto_assign_org_id = <org_id>

   # Default role new users will be automatically assigned (if auto_assign_org above is set to true)
   auto_assign_org_role = None
   ```

1. Restart the Grafana instance.
1. Create a team with the desired name.
1. Assign fixed roles to the team.
1. Add users to the team.

A user will be added to the default organization automatically but won't have any permissions until assigned to a team.

### Reduce scope of service accounts

Using Service Accounts is an efficient way to facilitate M2M communications. However, they can pose a security threat if not scoped appropriately. To limit the scope of a service account, you can begin by creating a Service Account with `None` basic role and then assign the necessary permissions for the account.

1. Refer to [Service Accounts](ref:service-accounts) and add a new Service Account.
1. Set the basic role to `None`.
1. Set the fixed roles needed for the Service Account.

This will reduce the required permissions for the Service Account and minimize the risk of compromise.

# Grafana Cloud app plugin role definitions

{{< admonition type="note" >}}
Available in [Grafana Cloud](/docs/grafana-cloud).
{{< /admonition >}}

This page lists the RBAC roles available for Grafana Cloud app plugins. Plugin roles control access to specific plugin features and can be assigned to users, teams, or basic roles.

For general information about how RBAC works with app plugins, refer to [RBAC for app plugins](ref:rbac-for-app-plugins).

{{< admonition type="note" >}}
Third-party plugins can define their own RBAC roles. This page documents roles for Grafana Cloud app plugins only. Refer to the documentation for third-party plugins to learn about their available roles.
{{< /admonition >}}

## Default plugin permissions by basic role

When you assign a user a basic organization role (Viewer, Editor, or Admin), they automatically receive default plugin permissions. The following table summarizes the default access level for each Grafana Cloud plugin.

| Plugin                           | Viewer                                  | Editor                                                      | Admin                                         |
| -------------------------------- | --------------------------------------- | ----------------------------------------------------------- | --------------------------------------------- |
| **Adaptive Logs**                | Read exemptions                         | Read exemptions                                             | Admin access                                  |
| **Adaptive Metrics**             | Read recommendations, exemptions        | Read recommendations, exemptions                            | Admin access                                  |
| **Adaptive Traces**              | Read recommendations                    | Read recommendations                                        | Admin access                                  |
| **Application Observability**    | View access                             | View access                                                 | Admin access                                  |
| **Assistant**                    | Chat access, user rules/quickstarts     | + MCP servers, investigations                               | + Tenant-wide settings                        |
| **Cloud Provider**               | Read access                             | Read access                                                 | Provider-specific write access                |
| **Cost Attributions**            | Read attributions                       | Read attributions                                           | Read attributions                             |
| **Cost Management and Billing**  | —                                       | —                                                           | Full access                                   |
| **Database Observability**       | Read access                             | Read access                                                 | Admin access                                  |
| **Fleet Management (Collector)** | Read access                             | Read access                                                 | Full access                                   |
| **Frontend Observability**       | Read apps, source maps                  | + Write apps, source maps                                   | + Delete apps                                 |
| **Grafana Auth**                 | —                                       | —                                                           | Write access policies                         |
| **IRM**                          | Read all                                | + Write alert groups, schedules, maintenance, user settings | + Write integrations, escalation chains, etc. |
| **k6**                           | Read settings                           | + Write settings                                            | Admin access                                  |
| **Knowledge Graph**              | Read assertions                         | + Write configuration and rules                             | + Full write access                           |
| **Kubernetes Monitoring**        | Read all                                | Read all                                                    | Admin access                                  |
| **Labels**                       | Read labels                             | + Create, edit, delete labels                               | + Full write access                           |
| **Machine Learning**             | Read forecasting, outliers, sift        | + Write forecasting, outliers, sift                         | + Full write access                           |
| **OnCall**                       | Read all                                | + Write alert groups, schedules, maintenance, user settings | + Write integrations, escalation chains, etc. |
| **Private Data Connect**         | —                                       | —                                                           | Full access                                   |
| **Session Replay**               | View session recordings                 | View session recordings                                     | View session recordings                       |
| **SLO**                          | Read SLOs                               | Create, edit, delete SLOs                                   | + Modify org preferences                      |
| **Synthetic Monitoring**         | Read checks, probes, alerts, thresholds | + Create, edit, delete checks, probes, alerts, thresholds   | + Manage access tokens                        |

{{< admonition type="note" >}}
The permissions above are automatically granted based on the user's organization role. You can assign additional plugin-specific roles (listed below) to grant more granular access.
{{< /admonition >}}

## Adaptive Logs plugin

Plugin ID: `grafana-adaptivelogs-app`

| Plugin role                                                 | Description                                       |
| ----------------------------------------------------------- | ------------------------------------------------- |
| `plugins:grafana-adaptivelogs-app:admin`                    | Read/write access to everything in Adaptive Logs  |
| `plugins:grafana-adaptivelogs-app:patterns-editor`          | Read/write access to recommendations and patterns |
| `plugins:grafana-adaptivelogs-app:patterns-reader`          | Read access to recommendations and patterns       |
| `plugins:grafana-adaptivelogs-app:segments-admin`           | Create and manipulate segments                    |
| `plugins:grafana-adaptivelogs-app:expiring-exemptions-user` | Use the expiring exemptions button                |
| `plugins:grafana-adaptivelogs-app:plugin-access`            | Access to the Adaptive Logs plugin                |

## Adaptive Metrics plugin

Plugin ID: `grafana-adaptive-metrics-app`

| Plugin role                                              | Description                                         |
| -------------------------------------------------------- | --------------------------------------------------- |
| `plugins:grafana-adaptive-metrics-app:admin`             | Read/write access to everything in Adaptive Metrics |
| `plugins:grafana-adaptive-metrics-app:rules-editor`      | Read/write access to recommendations and rules      |
| `plugins:grafana-adaptive-metrics-app:rules-reader`      | Read access to recommendations and rules            |
| `plugins:grafana-adaptive-metrics-app:exemptions-editor` | Edit access to recommendation exemptions            |
| `plugins:grafana-adaptive-metrics-app:exemptions-reader` | Read access to recommendation exemptions            |
| `plugins:grafana-adaptive-metrics-app:segments-editor`   | Edit access to segments                             |
| `plugins:grafana-adaptive-metrics-app:segments-reader`   | Read access to segments                             |
| `plugins:grafana-adaptive-metrics-app:config-editor`     | Edit access to plugin configuration                 |
| `plugins:grafana-adaptive-metrics-app:config-reader`     | Read access to plugin configuration                 |
| `plugins:grafana-adaptive-metrics-app:plugin-access`     | Access to the Adaptive Metrics plugin               |

## Adaptive Traces plugin

Plugin ID: `grafana-adaptivetraces-app`

| Plugin role                                | Description                                        |
| ------------------------------------------ | -------------------------------------------------- |
| `plugins:grafana-adaptivetraces-app:admin` | Read/write access to everything in Adaptive Traces |

## Application Observability plugin

Plugin ID: `grafana-app-observability-app`

| Plugin role                                    | Description                                                         |
| ---------------------------------------------- | ------------------------------------------------------------------- |
| `plugins:grafana-app-observability-app:admin`  | Read/write access to everything in Application Observability plugin |
| `plugins:grafana-app-observability-app:viewer` | View access in Application Observability plugin                     |

## Cloud Provider plugin

Plugin ID: `grafana-csp-app`

| Plugin role                            | Description                                         |
| -------------------------------------- | --------------------------------------------------- |
| `plugins:grafana-csp-app:aws-writer`   | Read/Write access to AWS in Cloud provider plugin   |
| `plugins:grafana-csp-app:azure-writer` | Read/Write access to Azure in Cloud provider plugin |
| `plugins:grafana-csp-app:gcp-writer`   | Read/Write access to GCP in Cloud provider plugin   |
| `plugins:grafana-csp-app:reader`       | Read access in Cloud provider plugin                |

## Cost Attributions plugin

Plugin ID: `grafana-attributions-app`

| Plugin role                                                 | Description                                         |
| ----------------------------------------------------------- | --------------------------------------------------- |
| `plugins:grafana-attributions-app:cost-attributions-viewer` | View the Cost Attributions application and its data |

## Cost Management and Billing plugin

Plugin ID: `grafana-cmab-app`

| Plugin role                                         | Description                                      |
| --------------------------------------------------- | ------------------------------------------------ |
| `plugins:grafana-cmab-app:full-admin`               | Full access to all features                      |
| `plugins:grafana-cmab-app:billing-and-usage-reader` | Read-only access to billing and usage data       |
| `plugins:grafana-cmab-app:invoice-reader`           | Read-only access to invoices, FOCUS & usage data |
| `plugins:grafana-cmab-app:cost-attribution-admin`   | Full access to cost attributions                 |
| `plugins:grafana-cmab-app:cost-attribution-reader`  | Read-only access to cost attributions            |
| `plugins:grafana-cmab-app:usage-alerts-admin`       | Full access to usage alerts                      |
| `plugins:grafana-cmab-app:usage-alerts-reader`      | Read-only access to usage alerts                 |

## Database Observability plugin

Plugin ID: `grafana-dbo11y-app`

| Plugin role                         | Description                                                      |
| ----------------------------------- | ---------------------------------------------------------------- |
| `plugins:grafana-dbo11y-app:admin`  | Read/write access to everything in Database Observability plugin |
| `plugins:grafana-dbo11y-app:reader` | Read access to everything in Database Observability plugin       |

## Easystart / Integrations plugin

Plugin ID: `grafana-easystart-app`

| Plugin role                                         | Description             |
| --------------------------------------------------- | ----------------------- |
| `plugins:grafana-easystart-app:integrations-writer` | Administer integrations |

## Fleet Management (Collector plugin)

Plugin ID: `grafana-collector-app`

| Plugin role                                          | Description                          |
| ---------------------------------------------------- | ------------------------------------ |
| `plugins:grafana-collector-app:collector-app-admin`  | Full access to Fleet Management      |
| `plugins:grafana-collector-app:collector-app-reader` | Read-only access to Fleet Management |

## Frontend Observability plugin

Plugin ID: `grafana-kowalski-app`

| Plugin role                                                              | Description                                                         |
| ------------------------------------------------------------------------ | ------------------------------------------------------------------- |
| `plugins:grafana-kowalski-app:frontend-observability-admin`              | Read/write access to everything in Frontend Observability plugin    |
| `plugins:grafana-kowalski-app:frontend-observability-editor`             | Read/write access to everything but app deletion                    |
| `plugins:grafana-kowalski-app:frontend-observability-viewer`             | View access only                                                    |
| `plugins:grafana-kowalski-app:frontend-observability-sourcemap-uploader` | View access with the ability to read settings and upload sourcemaps |

## Grafana Assistant plugin

Plugin ID: `grafana-assistant-app`

| Plugin role                                                  | Description                                                       |
| ------------------------------------------------------------ | ----------------------------------------------------------------- |
| `plugins:grafana-assistant-app:assistant-admin`              | Manage both user and tenant-wide Assistant resources and settings |
| `plugins:grafana-assistant-app:assistant-mcp-user`           | Use Grafana Assistant and add personal MCP servers                |
| `plugins:grafana-assistant-app:assistant-user`               | Basic access to Grafana Assistant with read-only capabilities     |
| `plugins:grafana-assistant-app:assistant-investigation-user` | Use Assistant Backend Investigations                              |

## Grafana Auth plugin

Plugin ID: `grafana-auth-app`

| Plugin role                       | Description                                        |
| --------------------------------- | -------------------------------------------------- |
| `plugins:grafana-auth-app:writer` | Write and manage access policies for Grafana Cloud |

## IRM plugin

Plugin ID: `grafana-irm-app`

### Core roles

| Plugin role                                      | Description                                                                                                                                                                                           |
| ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `plugins:grafana-irm-app:admin`                  | Read/write access to everything in IRM                                                                                                                                                                |
| `plugins:grafana-irm-app:editor`                 | Similar to Admin, minus abilities to: create Integrations, create Escalation Chains, create Outgoing Webhooks, update ChatOps settings, update other user's settings, and update general IRM settings |
| `plugins:grafana-irm-app:reader`                 | Read-only access to everything in IRM                                                                                                                                                                 |
| `plugins:grafana-irm-app:oncaller`               | Read access to everything in IRM, plus edit access to Alert Groups, Schedules, and own settings                                                                                                       |
| `plugins:grafana-irm-app:notifications-receiver` | Receive alert notifications, plus edit own IRM settings                                                                                                                                               |
| `plugins:grafana-irm-app:incident-access`        | Access to Grafana IRM incidents                                                                                                                                                                       |

### Alert groups

| Plugin role                                          | Description                                                                              |
| ---------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `plugins:grafana-irm-app:alert-groups-reader`        | Read-only access to Alert Groups                                                         |
| `plugins:grafana-irm-app:alert-groups-editor`        | Read access to Alert Groups + ability to act on Alert Groups (acknowledge, resolve, etc) |
| `plugins:grafana-irm-app:alert-groups-direct-paging` | Manually create new Alert Groups (Direct Paging)                                         |

### Integrations

| Plugin role                                   | Description                       |
| --------------------------------------------- | --------------------------------- |
| `plugins:grafana-irm-app:integrations-reader` | Read-only access to Integrations  |
| `plugins:grafana-irm-app:integrations-editor` | Read/write access to Integrations |

### Escalation chains

| Plugin role                                        | Description                            |
| -------------------------------------------------- | -------------------------------------- |
| `plugins:grafana-irm-app:escalation-chains-reader` | Read-only access to Escalation Chains  |
| `plugins:grafana-irm-app:escalation-chains-editor` | Read/write access to Escalation Chains |

### Schedules

| Plugin role                                | Description                    |
| ------------------------------------------ | ------------------------------ |
| `plugins:grafana-irm-app:schedules-reader` | Read-only access to Schedules  |
| `plugins:grafana-irm-app:schedules-editor` | Read/write access to Schedules |

### ChatOps

| Plugin role                              | Description                           |
| ---------------------------------------- | ------------------------------------- |
| `plugins:grafana-irm-app:chatops-reader` | Read-only access to ChatOps settings  |
| `plugins:grafana-irm-app:chatops-editor` | Read/write access to ChatOps settings |

### Outgoing webhooks

| Plugin role                                        | Description                            |
| -------------------------------------------------- | -------------------------------------- |
| `plugins:grafana-irm-app:outgoing-webhooks-reader` | Read-only access to Outgoing Webhooks  |
| `plugins:grafana-irm-app:outgoing-webhooks-editor` | Read/write access to Outgoing Webhooks |

### Maintenance

| Plugin role                                  | Description                                  |
| -------------------------------------------- | -------------------------------------------- |
| `plugins:grafana-irm-app:maintenance-reader` | Read-only access to Integration Maintenance  |
| `plugins:grafana-irm-app:maintenance-editor` | Read/write access to Integration Maintenance |

### API keys

| Plugin role                               | Description                                                       |
| ----------------------------------------- | ----------------------------------------------------------------- |
| `plugins:grafana-irm-app:api-keys-reader` | Read-only access to OnCall API Keys                               |
| `plugins:grafana-irm-app:api-keys-editor` | Read/write access to OnCall API Keys + ability to consume the API |

### User settings

| Plugin role                                    | Description                                                                        |
| ---------------------------------------------- | ---------------------------------------------------------------------------------- |
| `plugins:grafana-irm-app:user-settings-reader` | Read-only access to own IRM User Settings                                          |
| `plugins:grafana-irm-app:user-settings-editor` | Read/write access to own IRM User Settings + view basic info about other IRM users |
| `plugins:grafana-irm-app:user-settings-admin`  | Read/write access to your own + other's IRM User Settings                          |

### Notification and general settings

| Plugin role                                            | Description                                    |
| ------------------------------------------------------ | ---------------------------------------------- |
| `plugins:grafana-irm-app:notification-settings-reader` | Read-only access to IRM Notification Settings  |
| `plugins:grafana-irm-app:notification-settings-editor` | Read/write access to IRM Notification Settings |
| `plugins:grafana-irm-app:settings-reader`              | Read-only access to IRM Settings               |
| `plugins:grafana-irm-app:settings-editor`              | Read/write access to IRM Settings              |

## k6 Cloud plugin

Plugin ID: `k6-app`

| Plugin role             | Description                                 |
| ----------------------- | ------------------------------------------- |
| `plugins:k6-app:admin`  | Admin access to everything in k6            |
| `plugins:k6-app:editor` | Read/write access to k6 with limited scopes |
| `plugins:k6-app:reader` | Read-only access to k6                      |

## Knowledge Graph plugin

Plugin ID: `grafana-asserts-app`

| Plugin role                                          | Description                                       |
| ---------------------------------------------------- | ------------------------------------------------- |
| `plugins:grafana-asserts-app:knowledge-graph-writer` | Read/write/create in Knowledge Graph              |
| `plugins:grafana-asserts-app:knowledge-graph-reader` | Read-only access to everything in Knowledge Graph |
| `plugins:grafana-asserts-app:knowledge-graph-access` | Access to Knowledge Graph                         |

## Kubernetes Monitoring plugin

Plugin ID: `grafana-k8s-app`

| Plugin role                      | Description                              |
| -------------------------------- | ---------------------------------------- |
| `plugins:grafana-k8s-app:admin`  | Admin access to everything in k8s plugin |
| `plugins:grafana-k8s-app:reader` | Read-only access to k8s plugin           |

## Labels plugin

Plugin ID: `grafana-labels-app`

| Plugin role                                | Description                     |
| ------------------------------------------ | ------------------------------- |
| `plugins:grafana-labels-app:labels-writer` | Read/write/create/delete Labels |
| `plugins:grafana-labels-app:labels-reader` | Read-only access to Labels      |

## Machine Learning plugin

Plugin ID: `grafana-ml-app`

| Plugin role                          | Description                            |
| ------------------------------------ | -------------------------------------- |
| `plugins:grafana-ml-app:ml-editor`   | Read and write access to ML features   |
| `plugins:grafana-ml-app:ml-viewer`   | Read access to ML features             |
| `plugins:grafana-ml-app:sift-editor` | Read and write access to Sift features |
| `plugins:grafana-ml-app:sift-viewer` | Read access to Sift features           |

## Private Data Connect plugin

Plugin ID: `grafana-pdc-app`

| Plugin role                                      | Description           |
| ------------------------------------------------ | --------------------- |
| `plugins:grafana-pdc-app:private-networks-read`  | Read Private Networks |
| `plugins:grafana-pdc-app:private-networks-write` | Edit Private Networks |

## Session Replay plugin

Plugin ID: `grafana-sessionreplay-app`

| Plugin role                                           | Description                                       |
| ----------------------------------------------------- | ------------------------------------------------- |
| `plugins:grafana-sessionreplay-app:recordings-viewer` | View session recordings in Frontend Observability |

## SLO plugin

Plugin ID: `grafana-slo-app`

| Plugin role                          | Description                                                  |
| ------------------------------------ | ------------------------------------------------------------ |
| `plugins:grafana-slo-app:slo-reader` | View SLOs in folders where you have folder read permission   |
| `plugins:grafana-slo-app:slo-writer` | Manage SLOs in folders where you have folder edit permission |
| `plugins:grafana-slo-app:slo-admin`  | SLO Writer, plus the ability to modify org preferences       |

## Synthetic Monitoring plugin

Plugin ID: `grafana-synthetic-monitoring-app`

### Core roles

| Plugin role                                       | Description                                                                                                                    |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `plugins:grafana-synthetic-monitoring-app:admin`  | Full access to write and manage checks, probes, alerts, thresholds, and access tokens as well as enabling/disabling the plugin |
| `plugins:grafana-synthetic-monitoring-app:editor` | Add, update and delete checks, probes, alerts, and thresholds                                                                  |
| `plugins:grafana-synthetic-monitoring-app:reader` | Read checks, probes, alerts, thresholds, and access tokens                                                                     |

### Granular roles

| Plugin role                                                     | Description                     |
| --------------------------------------------------------------- | ------------------------------- |
| `plugins:grafana-synthetic-monitoring-app:checks-reader`        | Read checks                     |
| `plugins:grafana-synthetic-monitoring-app:checks-writer`        | Create, edit and delete checks  |
| `plugins:grafana-synthetic-monitoring-app:probes-reader`        | Read probes                     |
| `plugins:grafana-synthetic-monitoring-app:probes-writer`        | Create, edit and delete probes  |
| `plugins:grafana-synthetic-monitoring-app:alerts-reader`        | Read alerts                     |
| `plugins:grafana-synthetic-monitoring-app:alerts-writer`        | Create, edit and delete alerts  |
| `plugins:grafana-synthetic-monitoring-app:thresholds-reader`    | Read thresholds                 |
| `plugins:grafana-synthetic-monitoring-app:thresholds-writer`    | Read and edit thresholds        |
| `plugins:grafana-synthetic-monitoring-app:access-tokens-writer` | Create and delete access tokens |

## Role assignment

Plugin roles can be assigned to:

- **Users**: Individual user accounts
- **Teams**: All members of a team inherit the role
- **Basic Roles**: Can be added to Viewer, Editor, or Admin base roles

To assign roles, use:

- **UI**: Administration > Users/Teams > Select user/team > Roles tab
- **API**: `PUT /api/access-control/users/{userId}/roles` or `PUT /api/access-control/teams/{teamId}/roles`

For more information about managing RBAC roles, refer to [Manage RBAC roles](ref:manage-rbac-roles).

## Query plugin roles

You can query your Grafana Cloud stack's available plugin roles using the API:

```bash
curl -s -H "Authorization: Bearer YOUR_SERVICE_ACCOUNT_TOKEN" \
  "https://YOUR_STACK.grafana.net/api/access-control/roles?includeHidden=true" | \
  jq '[.[] | select(.name | startswith("plugins:"))]'
```

## Related documentation

- [RBAC for app plugins](ref:rbac-for-app-plugins): Overview of how RBAC works with app plugins
- [Folder access control](ref:folder-access-control): How folders interact with plugin roles
- [RBAC fixed role definitions](ref:rbac-fixed-basic-role-definitions): Fixed roles for Grafana features
- [Manage RBAC roles](ref:manage-rbac-roles): How to manage role assignments

# Basic and fixed roles permissions

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/docs/grafana-cloud).
{{< /admonition >}}

{{< admonition type="caution" >}}
**All Grafana users must have a basic role assigned**. For an overview of each type of role refer to [Basic roles in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/#basic-roles).
{{< /admonition >}}

The following tables list permissions associated with basic and fixed roles. This does not include basic role assignments added by plugins or apps.

## Permissions associated to basic roles

| Basic role                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | UID                                                                                                                                                      | Associated fixed roles                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Description                                                                     |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| Grafana Admin                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `basic_grafana_admin`                                                                                                                                    |
| `fixed:authentication.config:writer`<br>`fixed:general.auth.config:writer`<br>`fixed:ldap:writer`<br>`fixed:licensing:writer`<br>`fixed:migrationassistant:migrator`<br>`fixed:org.users:writer`<br>`fixed:organization:maintainer`<br>`fixed:plugins:maintainer`<br>`fixed:provisioning:writer`<br>`fixed:roles:writer`<br>`fixed:settings:reader`<br>`fixed:settings:writer`<br>`fixed:stats:reader`<br>`fixed:support.bundles:writer`<br>`fixed:usagestats:reader`<br>`fixed:users:writer` | Default [Grafana server administrator](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/#grafana-server-administrators) assignments. |
| Admin                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | `basic_admin`                                                                                                                                            | All roles assigned to Editor and `fixed:reports:writer` <br>`fixed:datasources:writer`<br>`fixed:organization:writer`<br>`fixed:datasources.permissions:writer`<br>`fixed:teams:writer`<br>`fixed:dashboards:writer`<br>`fixed:dashboards.permissions:writer`<br>`fixed:dashboards.public:writer`<br>`fixed:folders:writer`<br>`fixed:folders.permissions:writer`<br>`fixed:alerting:writer`<br>`fixed:alerting.provisioning.secrets:reader`<br>`fixed:alerting.provisioning:writer`<br>`fixed:datasources.caching:writer`<br>`fixed:plugins:writer`<br>`fixed:library.panels:writer` | Default [Grafana organization administrator](ref:rbac-basic-roles) assignments. |
| Editor                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | `basic_editor`                                                                                                                                           | All roles assigned to Viewer and `fixed:datasources:explorer` <br>`fixed:dashboards:creator`<br>`fixed:folders:creator`<br>`fixed:annotations:writer`<br>`fixed:alerting:writer`<br>`fixed:library.panels:creator`<br>`fixed:library.panels:general.writer`<br>`fixed:alerting.provisioning.provenance:writer`<br>`fixed:queries:writer`                                                                                                                                                                                                                                              | Default [Editor](ref:rbac-basic-roles) assignments.                             |
| Viewer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | `basic_viewer`                                                                                                                                           | `fixed:datasources.id:reader`<br>`fixed:organization:reader`<br>`fixed:annotations:reader`<br>`fixed:alerting:reader`<br>`fixed:plugins.app:reader`<br>`fixed:dashboards.insights:reader`<br>`fixed:datasources.insights:reader`<br>`fixed:library.panels:general.reader`<br>`fixed:folders.general:reader`<br>`fixed:datasources.builtin:reader`<br>`fixed:queries:reader`                                                                                                                                                                                                           | Default [Viewer](ref:rbac-basic-roles) assignments.                             |
| None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | n/a                                                                                                                                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Default [None](ref:rbac-basic-roles)                                            |

{{< admonition type="note" >}}
Grafana Cloud stacks include additional plugin role assignments not shown here. Refer to [Grafana Cloud app plugin role definitions](ref:plugin-role-definitions) for a complete list.
{{< /admonition >}}

## Permissions associated to fixed roles

The following table has the existing built-in fixed role definitions. Other fixed roles might be added by plugins installed in Grafana.
The UUID presented here can be used as an identifier for [Terraform provisioning](ref:rbac-terraform-provisioning).

{{< admonition type="caution" >}}
These UUIDs won't be available if your instance was created before Grafana v10.2.0.

To learn how to use the roles API to determine the role UUIDs, refer to [Manage RBAC roles](ref:rbac-manage-rbac-roles).
{{< /admonition >}}

| Fixed role                                      | UUID                                | Permissions                                                                                                                                                                                                                                                                                                                          | Description                                                                                                                                                                                                                                                                           |
| ----------------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `fixed:alerting:reader`                         | `fixed_O2oP1_uBFozI2i93klAkcvEWR30` | All permissions from `fixed:alerting.rules:reader` <br>`fixed:alerting.instances:reader`<br>`fixed:alerting.notifications:reader`                                                                                                                                                                                                    | Read-only permissions for all Grafana, Mimir, Loki and Alertmanager alert rules\*, alerts, contact points, and notification policies.[\*](#alerting-roles)                                                                                                                            |
| `fixed:alerting:writer`                         | `fixed_-PAZgSJsDlRD8NUg-PFSeH_BkJY` | All permissions from `fixed:alerting.rules:writer` <br>`fixed:alerting.instances:writer`<br>`fixed:alerting.notifications:writer`                                                                                                                                                                                                    | Create, update, and delete Grafana, Mimir, Loki and Alertmanager alert rules\*, silences, contact points, templates, mute timings, and notification policies.[\*](#alerting-roles)                                                                                                    |
| `fixed:alerting.instances:reader`               | `fixed_ut5fVS-Ulh_ejFoskFhJT_rYg0Y` | `alert.instances:read` for organization scope <br> `alert.instances.external:read` for scope `datasources:*`                                                                                                                                                                                                                         | Read all alerts and silences in the organization produced by Grafana Alerts and Mimir and Loki alerts and silences.[\*](#alerting-roles)                                                                                                                                              |
| `fixed:alerting.instances:writer`               | `fixed_pKOBJE346uyqMLdgWbk1NsQfEl0` | All permissions from `fixed:alerting.instances:reader` and<br> `alert.instances:create`<br>`alert.instances:write` for organization scope <br> `alert.instances.external:write` for scope `datasources:*`                                                                                                                            | Create, update and expire all silences in the organization produced by Grafana, Mimir, and Loki.[\*](#alerting-roles)                                                                                                                                                                 |
| `fixed:alerting.notifications:reader`           | `fixed_hmBn0lX5h1RZXB9Vaot420EEdA0` | All permissions from `fixed:alerting.receivers:reader`, `fixed:alerting.templates:reader`, `fixed:alerting.time-intervals:reader`, `fixed:alerting.routes:reader`, `fixed:alerting.inhibition-rules:reader`, and `fixed:alerting.notifications.external:reader`                                                                      | Read all Grafana and Alertmanager contact points, templates, and notification policies.[\*](#alerting-roles)                                                                                                                                                                          |
| `fixed:alerting.notifications:writer`           | `fixed_XplK6HPNxf9AP5IGTdB5Iun4tJc` | All permissions from `fixed:alerting.notifications:reader` and<br>`fixed:alerting.receivers:writer`, `fixed:alerting.templates:writer`, `fixed:alerting.time-intervals:writer`, `fixed:alerting.routes:writer`, `fixed:alerting.inhibition-rules:writer`, and `fixed:alerting.notifications.external:writer`                         | Create, update, and delete contact points, templates, mute timings and notification policies for Grafana and external Alertmanager.[\*](#alerting-roles)                                                                                                                              |
| `fixed:alerting.provisioning:writer`            | `fixed_y7pFjdEkxpx5ETdcxPvp0AgRuUo` | `alert.provisioning:read` and `alert.provisioning:write`                                                                                                                                                                                                                                                                             | Create, update and delete Grafana alert rules, notification policies, contact points, templates, etc via provisioning API. [\*](#alerting-roles)                                                                                                                                      |
| `fixed:alerting.provisioning.secrets:reader`    | `fixed_9fmzXXZZG-Od0Amy2ofEG8Uk--c` | `alert.provisioning:read` and `alert.provisioning.secrets:read`                                                                                                                                                                                                                                                                      | Read-only permissions for Provisioning API and let export resources with decrypted secrets [\*](#alerting-roles)                                                                                                                                                                      |
| `fixed:alerting.provisioning.provenance:writer` | `fixed_eAxlzfkTuobvKEgXHveFMBZrOj8` | `alert.provisioning.provenance:write`                                                                                                                                                                                                                                                                                                | Set provenance status to alert rules, notification policies, contact points, etc. Should be used together with regular writer roles. [\*](#alerting-roles)                                                                                                                            |
| `fixed:alerting.rules:reader`                   | `fixed_fRGKL_vAqUsmUWq5EYKnOha9DcA` | `alert.rule:read`, `alert.silences:read` for scope `folders:*` <br> `alert.rules.external:read` for scope `datasources:*` <br> `alert.notifications.time-intervals:read` <br> `alert.notifications.receivers:list`                                                                                                                   | Read all\* Grafana, Mimir, and Loki alert rules.[\*](#alerting-roles) and read rule-specific silences                                                                                                                                                                                 |
| `fixed:alerting.rules:writer`                   | `fixed_YJJGwAalUwDZPrXSyFH8GfYBXAc` | All permissions from `fixed:alerting.rules:reader` and <br> `alert.rule:create` <br> `alert.rule:write` <br> `alert.rule:delete` <br> `alert.silences:create` <br> `alert.silences:write` for scope `folders:*` <br> `alert.rules.external:write` for scope `datasources:*`                                                          | Create, update, and delete all\* Grafana, Mimir, and Loki alert rules.[\*](#alerting-roles) and manage rule-specific silences                                                                                                                                                         |
| `fixed:alerting.receivers:reader`               | `fixed_e7HL8sF-9P8aZwmKqhN45Ob0WOI` | `alert.notifications.receivers:read` for scope `receivers:*`                                                                                                                                                                                                                                                                         | Read all contact points in Grafana.[\*](#alerting-roles)                                                                                                                                                                                                                              |
| `fixed:alerting.receivers:creator`              | `fixed_h-ZX5WBw4bfcmzFStUF3_IWfngk` | `alert.notifications.receivers:create`<br>`alert.notifications.receivers.test:create` for scope `receivers:uid:-`                                                                                                                                                                                                                    | Create new contact points in Grafana. The creator is automatically granted full access to the created contact point.[\*](#alerting-roles)                                                                                                                                             |
| `fixed:alerting.receivers:writer`               | `fixed_XWX0KuQZNhgVXmzmQPFpdX5cArU` | All permissions from `fixed:alerting.receivers:creator` and<br>`alert.notifications.receivers:read`, `alert.notifications.receivers:write`, `alert.notifications.receivers:delete`, and `alert.notifications.receivers.test:create` for scope `receivers:*`                                                                          | Create, update, and delete all contact points in Grafana.[\*](#alerting-roles)                                                                                                                                                                                                        |
| `fixed:alerting.templates:reader`               | `fixed_irwxMD5Lx5dbHqU7ahIgGc2ssfg` | `alert.notifications.templates:read`                                                                                                                                                                                                                                                                                                 | Read all templates in Grafana alerting.[\*](#alerting-roles)                                                                                                                                                                                                                          |
| `fixed:alerting.templates:writer`               | `fixed_Ntb782dlLhaBJMLBo2ohdcB3mas` | `alert.notifications.templates:read`, `alert.notifications.templates:write`, `alert.notifications.templates:delete`, `alert.notifications.templates.test:write`                                                                                                                                                                      | Create, update, and delete all templates in Grafana alerting. Test templates with custom payloads.[\*](#alerting-roles)                                                                                                                                                               |
| `fixed:alerting.time-intervals:reader`          | `fixed_ENpv0QyUkQYV8WG_c8RHefVMwCE` | `alert.notifications.time-intervals:read`                                                                                                                                                                                                                                                                                            | Read all time intervals in Grafana alerting.[\*](#alerting-roles)                                                                                                                                                                                                                     |
| `fixed:alerting.time-intervals:writer`          | `fixed_J-L93TBpJby55gkKdGdT9F17a0o` | `alert.notifications.time-intervals:read`, `alert.notifications.time-intervals:write`, `alert.notifications.time-intervals:delete`                                                                                                                                                                                                   | Create, update, and delete all time intervals in Grafana alerting.[\*](#alerting-roles)                                                                                                                                                                                               |
| `fixed:alerting.routes:reader`                  | `fixed_ByG4cCa4s9Jc_VpRwq80y_jypcQ` | `notifications.alerting.grafana.app/routingtrees:get` for scope `notifications.alerting.grafana.app/routingtrees:*`                                                                                                                                                                                                                  | Read all notification policies in Grafana alerting.[\*](#alerting-roles)                                                                                                                                                                                                              |
| `fixed:alerting.routes:writer`                  | `fixed_uSfE2ZCFozB2VvCU9B7R-adgkg4` | All permissions from `fixed:alerting.routes:reader` and<br>`notifications.alerting.grafana.app/routingtrees:create`, and<br>`notifications.alerting.grafana.app/routingtrees:update`, `notifications.alerting.grafana.app/routingtrees:delete` for scope `notifications.alerting.grafana.app/routingtrees:*`                         | Create, update, and delete notification policies in Grafana alerting.[\*](#alerting-roles)                                                                                                                                                                                            |
| `fixed:alerting.managed-routes:creator`         | `fixed_bw49IoZC6KEicSfgNxNEuukxXa4` | `notifications.alerting.grafana.app/routingtrees:create`                                                                                                                                                                                                                                                                             | Create notification policies in Grafana alerting.[\*](#alerting-roles)                                                                                                                                                                                                                |
| `fixed:alerting.notifications.config:reader`    | `fixed_81ewa_t30G_KtKCYXLOlhYoixbg` | `notifications.alerting.grafana.app/configs:get` for scope `configs:*`                                                                                                                                                                                                                                                               | Read the alerting notifications configuration, including the external Alertmanager sync state.[\*](#alerting-roles)                                                                                                                                                                   |
| `fixed:alerting.notifications.config:writer`    | `fixed_JUJgqtKuWByW6o3ggE73E7yioas` | All permissions from `fixed:alerting.notifications.config:reader` and<br>`notifications.alerting.grafana.app/configs:update` for scope `configs:*`                                                                                                                                                                                   | Update the alerting notifications configuration, including the external Alertmanager sync target.[\*](#alerting-roles)                                                                                                                                                                |
| `fixed:alerting.alertmanager-imports:writer`    | `fixed_tKsnflM69PHBFvDKdlsuXVvdwG0` | `notifications.alerting.grafana.app/alertmanagerimports:create`, `notifications.alerting.grafana.app/alertmanagerimports:get`, `notifications.alerting.grafana.app/alertmanagerimports:update`, `notifications.alerting.grafana.app/alertmanagerimports:delete` for scope `notifications.alerting.grafana.app/alertmanagerimports:*` | Read, write, and delete Prometheus/Mimir-compatible Alertmanager configurations imported via the convert API.[\*](#alerting-roles)                                                                                                                                                    |
| `fixed:alerting.notifications.external:reader`  | `fixed_rWNhdHUNdZkN8S6e5LAmKKtWt2w` | `alert.notifications.external:read` for scope `datasources:*`                                                                                                                                                                                                                                                                        | Read notification policies and contact points in external providers.[\*](#alerting-roles)                                                                                                                                                                                             |
| `fixed:alerting.notifications.external:writer`  | `fixed_TSFB5oWYpd8nZgEUeQ0h4FyGmH0` | All permissions from `fixed:alerting.notifications.external:reader` and<br>`alert.notifications.external:write` for scope `datasources:*`                                                                                                                                                                                            | Add, update, and delete contact points and notification policies in external providers.[\*](#alerting-roles)                                                                                                                                                                          |
| `fixed:alerting.inhibition-rules:reader`        | `fixed_pC-Ofc7pvmj7Yyfc7oliu67RxEQ` | `alert.notifications.inhibition-rules:read` for scope `inhibition-rules:*`                                                                                                                                                                                                                                                           | Read all inhibition rules in Grafana alerting.[\*](#alerting-roles)                                                                                                                                                                                                                   |
| `fixed:alerting.inhibition-rules:writer`        | `fixed_0uRAEVdAXFyxEuqoVhspH6DjvKc` | All permissions from `fixed:alerting.inhibition-rules:reader` and<br>`alert.notifications.inhibition-rules:write`, `alert.notifications.inhibition-rules:delete` for scope `inhibition-rules:*`                                                                                                                                      | Create, update, and delete all inhibition rules in Grafana alerting.[\*](#alerting-roles)                                                                                                                                                                                             |
| `fixed:alerting:admin`                          | `fixed_ceFufpsSIv8LgBB81M9_NSHydxg` | All permissions from `fixed:alerting:writer` and<br>`alert.notifications.receivers.permissions:read`, `alert.notifications.receivers.permissions:write`, `alert.notifications.receivers:readSecrets`, `alert.notifications.receivers:updateProtected` for scope `receivers:*`                                                        | Full write access in Grafana and all external providers, including their permissions, protected fields and secrets.[\*](#alerting-roles)                                                                                                                                              |
| `fixed:annotations:reader`                      | `fixed_hpZnoizrfAJsrceNcNQqWYV-xNU` | `annotations:read` for scope `annotations:*`                                                                                                                                                                                                                                                                                         | Read all annotations and annotation tags.                                                                                                                                                                                                                                             |
| `fixed:annotations:writer`                      | `fixed_ZVW-Aa9Tzle6J4s2aUFcq1StKWE` | All permissions from `fixed:annotations:reader` <br>`annotations:write` <br>`annotations.create`<br> `annotations:delete` for scope `annotations:*`                                                                                                                                                                                  | Read, create, update and delete all annotations and annotation tags.                                                                                                                                                                                                                  |
| `fixed:authentication.config:writer`            | `fixed_0rYhZ2Qnzs8AdB1nX7gexk3fHDw` | `settings:read` for scope `settings:auth.saml:*` <br> `settings:write` for scope `settings:auth.saml:*`                                                                                                                                                                                                                              | Read and update authentication and SAML settings.                                                                                                                                                                                                                                     |
| `fixed:general.auth.config:writer`              | `fixed_QFxIT_FGtBqbIVJIwx1bLgI5z6c` | `settings:read` for scope `settings:auth:oauth_allow_insecure_email_lookup` <br> `settings:write` for scope `settings:auth:oauth_allow_insecure_email_lookup`                                                                                                                                                                        | Read and update the Grafana instance's general authentication configuration settings.                                                                                                                                                                                                 |
| `fixed:dashboards:creator`                      | `fixed_ZorKUcEPCM01A1fPakEzGBUyU64` | `dashboards:create`<br>`folders:read`                                                                                                                                                                                                                                                                                                | Create dashboards.                                                                                                                                                                                                                                                                    |
| `fixed:dashboards:reader`                       | `fixed_Sgr67JTOhjQGFlzYRahOe45TdWM` | `dashboards:read`                                                                                                                                                                                                                                                                                                                    | Read all dashboards.                                                                                                                                                                                                                                                                  |
| `fixed:dashboards:writer`                       | `fixed_OK2YOQGIoI1G031hVzJB6rAJQAs` | All permissions from `fixed:dashboards:reader` and <br>`dashboards:write`<br>`dashboards:delete`<br>`dashboards:create`<br>`dashboards.permissions:read`<br>`dashboards.permissions:write`                                                                                                                                           | Read, create, update, and delete all dashboards.                                                                                                                                                                                                                                      |
| `fixed:dashboards.insights:reader`              | `fixed_JlBJ2_gizP8zhgaeGE2rjyZe2Rs` | `dashboards.insights:read`                                                                                                                                                                                                                                                                                                           | Read dashboard insights data and see presence indicators.                                                                                                                                                                                                                             |
| `fixed:dashboards.permissions:reader`           | `fixed_f17oxuXW_58LL8mYJsm4T_mCeIw` | `dashboards.permissions:read`                                                                                                                                                                                                                                                                                                        | Read all dashboard permissions.                                                                                                                                                                                                                                                       |
| `fixed:dashboards.permissions:writer`           | `fixed_CcznxhWX_Yqn8uWMXMQ-b5iFW9k` | All permissions from `fixed:dashboards.permissions:reader` and <br>`dashboards.permissions:write`                                                                                                                                                                                                                                    | Read and update all dashboard permissions.                                                                                                                                                                                                                                            |
| `fixed:dashboards.public:writer`                | `fixed_f_GHHRBciaqESXfGz2oCcooqHxs` | `dashboards.public:write`                                                                                                                                                                                                                                                                                                            | Create, update, delete or pause a shared dashboard.                                                                                                                                                                                                                                   |
| `fixed:datasources:creator`                     | `fixed_XX8jHREgUt-wo1A-rPXIiFlX6Zw` | `datasources:create`                                                                                                                                                                                                                                                                                                                 | Create data sources.                                                                                                                                                                                                                                                                  |
| `fixed:datasources:explorer`                    | `fixed_qDzW9mzx9yM91T5Bi8dHUM2muTw` | `datasources:explore`                                                                                                                                                                                                                                                                                                                | Enable the Explore feature. Data source permissions still apply, you can only query data sources for which you have query permissions.                                                                                                                                                |
| `fixed:datasources:reader`                      | `fixed_C2x8IxkiBc1KZVjyYH775T9jNMQ` | `datasources:read`<br>`datasources:query`                                                                                                                                                                                                                                                                                            | Read and query data sources.                                                                                                                                                                                                                                                          |
| `fixed:datasources:writer`                      | `fixed_q8HXq8kjjA5IlHHgBJlKlUyaNik` | All permissions from `fixed:datasources:reader` and <br>`datasources:create`<br>`datasources:write`<br>`datasources:delete`                                                                                                                                                                                                          | Read, query, create, delete, or update a data source.                                                                                                                                                                                                                                 |
| `fixed:datasources.builtin:reader`              | `fixed_q8HXq8kjjA5IlHHgBJlKlUyaNik` | `datasources:read` and `datasources:query` scoped to `datasources:uid:grafana`                                                                                                                                                                                                                                                       | An internal role used to grant Viewers access to the builtin example data source in Grafana.                                                                                                                                                                                          |
| `fixed:datasources.caching:reader`              | `fixed_D2ddpGxJYlw0mbsTS1ek9fj0kj4` | `datasources.caching:read`                                                                                                                                                                                                                                                                                                           | Read data source query caching settings.                                                                                                                                                                                                                                              |
| `fixed:datasources.caching:writer`              | `fixed_JtFjHr7jd7hSqUYcktKvRvIOGRE` | `datasources.caching:read`<br>`datasources.caching:write`                                                                                                                                                                                                                                                                            | Enable, disable, or update query caching settings.                                                                                                                                                                                                                                    |
| `fixed:datasources.id:reader`                   | `fixed_entg--fHmDqWY2-69N0ocawK0Os` | `datasources.id:read`                                                                                                                                                                                                                                                                                                                | Read the ID of a data source based on its name.                                                                                                                                                                                                                                       |
| `fixed:datasources.insights:reader`             | `fixed_EBZ3NwlfecNPp2p0XcZRC1nfEYk` | `datasources.insights:read`                                                                                                                                                                                                                                                                                                          | Read data source insights data.                                                                                                                                                                                                                                                       |
| `fixed:datasources.permissions:reader`          | `fixed_ErYA-cTN3yn4h4GxaVPcawRhiOY` | `datasources.permissions:read`                                                                                                                                                                                                                                                                                                       | Read data source permissions.                                                                                                                                                                                                                                                         |
| `fixed:datasources.permissions:writer`          | `fixed_aiQh9YDfLOKjQhYasF9_SFUjQiw` | All permissions from `fixed:datasources.permissions:reader` and <br>`datasources.permissions:write`                                                                                                                                                                                                                                  | Create, read, or delete permissions of a data source.                                                                                                                                                                                                                                 |
| `fixed:folders:creator`                         | `fixed_gGLRbZGAGB6n9uECqSh_W382RlQ` | `folders:create`                                                                                                                                                                                                                                                                                                                     | Create folders in the root level.                                                                                                                                                                                                                                                     |
| `fixed:folders:reader`                          | `fixed_yeW-5QPeo-i5PZUIUXMlAA97GnQ` | `folders:read`<br>`dashboards:read`                                                                                                                                                                                                                                                                                                  | Read all folders and dashboards.                                                                                                                                                                                                                                                      |
| `fixed:folders:writer`                          | `fixed_wJXLoTzgE7jVuz90dryYoiogL0o` | All permissions from `fixed:dashboards:writer` and <br>`folders:read`<br>`folders:write`<br>`folders:create`<br>`folders:delete`<br>`folders.permissions:read`<br>`folders.permissions:write`                                                                                                                                        | Read, update, and delete all folders and dashboards. Create folders and subfolders.                                                                                                                                                                                                   |
| `fixed:folders.general:reader`                  | `fixed_rSASbkg8DvpG_gTX5s41d7uxRvI` | `folders:read` scoped to `folders:uid:general`                                                                                                                                                                                                                                                                                       | An internal role used to correctly display access to the folder tree for Viewer role.                                                                                                                                                                                                 |
| `fixed:folders.permissions:reader`              | `fixed_E06l4cx0JFm47EeLBE4nmv3pnSo` | `folders.permissions:read`                                                                                                                                                                                                                                                                                                           | Read all folder permissions.                                                                                                                                                                                                                                                          |
| `fixed:folders.permissions:writer`              | `fixed_3GAgpQ_hWG8o7-lwNb86_VB37eI` | All permissions from `fixed:folders.permissions:reader` and <br>`folders.permissions:write`                                                                                                                                                                                                                                          | Read and update all folder permissions.                                                                                                                                                                                                                                               |
| `fixed:ldap:reader`                             | `fixed_lMcOPwSkxKY-qCK8NMJc5k6izLE` | `ldap.user:read`<br>`ldap.status:read`                                                                                                                                                                                                                                                                                               | Read the LDAP configuration and LDAP status information.                                                                                                                                                                                                                              |
| `fixed:ldap:writer`                             | `fixed_p6AvnU4GCQyIh7-hbwI-bk3GYnU` | All permissions from `fixed:ldap:reader` and <br>`ldap.user:sync`<br>`ldap.config:reload`                                                                                                                                                                                                                                            | Read and update the LDAP configuration, and read LDAP status information.                                                                                                                                                                                                             |
| `fixed:library.panels:creator`                  | `fixed_6eX6ItfegCIY5zLmPqTDW8ZV7KY` | `library.panels:create`<br>`folders:read`                                                                                                                                                                                                                                                                                            | Create library panel at the root level.                                                                                                                                                                                                                                               |
| `fixed:library.panels:general.reader`           | `fixed_ct0DghiBWR_2BiQm3EvNPDVmpio` | `library.panels:read`                                                                                                                                                                                                                                                                                                                | Read all library panels at the root level.                                                                                                                                                                                                                                            |
| `fixed:library.panels:general.writer`           | `fixed_DgprkmqfN_1EhZ2v1_d1fYG8LzI` | All permissions from `fixed:library.panels:general.reader` plus<br>`library.panels:create`<br>`library.panels:delete`<br>`library.panels:write`                                                                                                                                                                                      | Create, read, write or delete all library panels and their permissions at the root level.                                                                                                                                                                                             |
| `fixed:library.panels:reader`                   | `fixed_tvTr9CnZ6La5vvUO_U_X1LPnhUs` | `library.panels:read`                                                                                                                                                                                                                                                                                                                | Read all library panels.                                                                                                                                                                                                                                                              |
| `fixed:library.panels:writer`                   | `fixed_JTljAr21LWLTXCkgfBC4H0lhBC8` | All permissions from `fixed:library.panels:reader` plus<br>`library.panels:create`<br>`library.panels:delete`<br>`library.panels:write`                                                                                                                                                                                              | Create, read, write or delete all library panels and their permissions.                                                                                                                                                                                                               |
| `fixed:licensing:reader`                        | `fixed_OADpuXvNEylO2Kelu3GIuBXEAYE` | `licensing:read`<br>`licensing.reports:read`                                                                                                                                                                                                                                                                                         | Read licensing information and licensing reports.                                                                                                                                                                                                                                     |
| `fixed:licensing:writer`                        | `fixed_gzbz3rJpQMdaKHt-E4q0PVaKMoE` | All permissions from `fixed:licensing:reader` and <br>`licensing:write`<br>`licensing:delete`                                                                                                                                                                                                                                        | Read licensing information and licensing reports, update and delete the license token.                                                                                                                                                                                                |
| `fixed:migrationassistant:migrator`             | `fixed_LLk2p7TRuBztOAksTQb1Klc8YTk` | `migrationassistant:migrate`                                                                                                                                                                                                                                                                                                         | Execute on-prem to cloud migrations through the Migration Assistant.                                                                                                                                                                                                                  |
| `fixed:org.users:reader`                        | `fixed_oCqNwlVHLOpw7-jAlwp4HzYqwGY` | `org.users:read`                                                                                                                                                                                                                                                                                                                     | Read users within a single organization.                                                                                                                                                                                                                                              |
| `fixed:org.users:writer`                        | `fixed_VERj5nayasjgf_Yh0sWqqCkxWlw` | All permissions from `fixed:org.users:reader` and <br>`org.users:add`<br>`org.users:remove`<br>`org.users:write`                                                                                                                                                                                                                     | Within a single organization, add a user, invite a new user, read information about a user and their role, remove a user from that organization, or change the role of a user.                                                                                                        |
| `fixed:organization:maintainer`                 | `fixed_CMm-uuBaPUBf4r8XG3jIvxo55bg` | All permissions from `fixed:organization:reader` and <br> `orgs:write`<br>`orgs:create`<br>`orgs:delete`<br>`orgs.quotas:write`                                                                                                                                                                                                      | Create, read, write, or delete an organization. Read or write its quotas. This role needs to be assigned globally.                                                                                                                                                                    |
| `fixed:organization:reader`                     | `fixed_0SZPJlTHdNEe8zO91zv7Zwiwa2w` | `orgs:read`<br>`orgs.quotas:read`                                                                                                                                                                                                                                                                                                    | Read an organization and its quotas.                                                                                                                                                                                                                                                  |
| `fixed:organization:writer`                     | `fixed_Y4jGqDd8w1yCrPwlik8z5Iu8-3M` | All permissions from `fixed:organization:reader` and <br> `orgs:write`<br>`orgs.preferences:read`<br>`orgs.preferences:write`                                                                                                                                                                                                        | Read an organization, its quotas, or its preferences. Update organization properties, or its preferences.                                                                                                                                                                             |
| `fixed:playlists:reader`                        |                                     | `playlists:read`                                                                                                                                                                                                                                                                                                                     | Read and list playlists.                                                                                                                                                                                                                                                              |
| `fixed:playlists:writer`                        |                                     | All permissions from `fixed:playlists:reader` and<br>`playlists:write`                                                                                                                                                                                                                                                               | Read, create, update, and delete playlists.                                                                                                                                                                                                                                           |
| `fixed:plugins:maintainer`                      | `fixed_yEOKidBcWgbm74x-nTa3lW5lOyY` | `plugins:install`                                                                                                                                                                                                                                                                                                                    | Install and uninstall plugins. Needs to be assigned globally.                                                                                                                                                                                                                         |
| `fixed:plugins:writer`                          | `fixed_MRYpGk7kpNNwt2VoVOXFiPnQziE` | `plugins:write`                                                                                                                                                                                                                                                                                                                      | Enable and disable plugins and edit plugins' settings.                                                                                                                                                                                                                                |
| `fixed:plugins.app:reader`                      | `fixed_AcZRiNYx7NueYkUqzw1o2OGGUAA` | `plugins.app:access`                                                                                                                                                                                                                                                                                                                 | Access application plugins (still enforcing the organization role).                                                                                                                                                                                                                   |
| `fixed:provisioning:writer`                     | `fixed_bgk1FCyR6OEDwhgirZlQgu5LlCA` | `provisioning:reload`                                                                                                                                                                                                                                                                                                                | Reload provisioning.                                                                                                                                                                                                                                                                  |
| `fixed:queries:reader`                          | `fixed_-rEsxR-pZg1-fQL-XDQaouEpK98` | `queries:read`                                                                                                                                                                                                                                                                                                                       | Read saved queries.                                                                                                                                                                                                                                                                   |
| `fixed:queries:writer`                          | `fixed_7TMOlueweuPX_PhffesRB2FTQn4` | All permissions from `fixed:queries:reader` and<br>`queries:write`                                                                                                                                                                                                                                                                   | Create, update, and delete saved queries.                                                                                                                                                                                                                                             |
| `fixed:reports:reader`                          | `fixed_72_8LU_0ukfm6BdblOw8Z9q-GQ8` | `reports:read`<br>`reports:send`<br>`reports.settings:read`                                                                                                                                                                                                                                                                          | Read all reports and shared report settings.                                                                                                                                                                                                                                          |
| `fixed:reports:writer`                          | `fixed_jBW3_7g1EWOjGVBYeVRwtFxhUNw` | All permissions from `fixed:reports:reader` and <br>`reports:create`<br>`reports:write`<br>`reports:delete`<br>`reports.settings:write`                                                                                                                                                                                              | Create, read, update, or delete all reports and shared report settings.                                                                                                                                                                                                               |
| `fixed:roles:reader`                            | `fixed_GkfG-1NSwEGb4hpK3-E3qHyNltc` | `roles:read`<br>`teams.roles:read`<br>`users.roles:read`<br>`users.permissions:read`                                                                                                                                                                                                                                                 | Read all access control roles, roles and permissions assigned to users, teams.                                                                                                                                                                                                        |
| `fixed:roles:resetter`                          | `fixed_WgPpC3qJRmVpVTJavFNwfS5RuzQ` | `roles:write` with scope `permissions:type:escalate`                                                                                                                                                                                                                                                                                 | Reset basic roles to their default.                                                                                                                                                                                                                                                   |
| `fixed:roles:writer`                            | `fixed_W5aFaw8isAM27x_eWfElBhZ0iOc` | All permissions from `fixed:roles:reader` and <br>`roles:write`<br>`roles:delete`<br>`teams.roles:add`<br>`teams.roles:remove`<br>`users.roles:add`<br>`users.roles:remove`                                                                                                                                                          | Create, read, update, or delete all roles, assign or remove roles to users, teams.                                                                                                                                                                                                    |
| `fixed:serviceaccounts:creator`                 | `fixed_Ikw60fckA0MyiiZ73BawSfOULy4` | `serviceaccounts:create`                                                                                                                                                                                                                                                                                                             | Create Grafana service accounts.                                                                                                                                                                                                                                                      |
| `fixed:serviceaccounts:reader`                  | `fixed_QFjJAZ88iawMLInYOxPA1DB1w6I` | `serviceaccounts:read`                                                                                                                                                                                                                                                                                                               | Read Grafana service accounts.                                                                                                                                                                                                                                                        |
| `fixed:serviceaccounts:writer`                  | `fixed_iBvUNUEZBZ7PUW0vdkN5iojc2sk` | `serviceaccounts:read`<br>`serviceaccounts:create`<br>`serviceaccounts:write`<br>`serviceaccounts:delete`<br>`serviceaccounts.permissions:read`<br>`serviceaccounts.permissions:write`                                                                                                                                               | Create, update, read and delete all Grafana service accounts and manage service account permissions.                                                                                                                                                                                  |
| `fixed:settings:reader`                         | `fixed_0LaUt1x6PP8hsZzEBhqPQZFUd8Q` | `settings:read`                                                                                                                                                                                                                                                                                                                      | Read Grafana instance settings.                                                                                                                                                                                                                                                       |
| `fixed:settings:writer`                         | `fixed_joIHDgMrGg790hMhUufVzcU4j44` | All permissions from `fixed:settings:reader` and<br>`settings:write`                                                                                                                                                                                                                                                                 | Read and update Grafana instance settings.                                                                                                                                                                                                                                            |
| `fixed:stats:reader`                            | `fixed_OnRCXxZVINWpcKvTF5A1gecJ7pA` | `server.stats:read`                                                                                                                                                                                                                                                                                                                  | Read Grafana instance statistics.                                                                                                                                                                                                                                                     |
| `fixed:support.bundles:reader`                  | `fixed_gcPjI3PTUJwRx-GJZwDhNa7zbos` | `support.bundles:read`                                                                                                                                                                                                                                                                                                               | List and download support bundles.                                                                                                                                                                                                                                                    |
| `fixed:support.bundles:writer`                  | `fixed_dTgCv9Wxrp_WHAhwHYIgeboxKpE` | `support.bundles:read`<br>`support.bundles:create`<br>`support.bundles:delete`                                                                                                                                                                                                                                                       | Create, delete, list and download support bundles.                                                                                                                                                                                                                                    |
| `fixed:teams:creator`                           | `fixed_nzVQoNSDSn0fg1MDgO6XnZX2RZI` | `teams:create`<br>`org.users:read`                                                                                                                                                                                                                                                                                                   | Create a team and list organization users (required to manage the created team).                                                                                                                                                                                                      |
| `fixed:teams:read`                              | `fixed_Z8pB0GQlrqRt8IZBCJQxPWvJPgQ` | `teams:read`                                                                                                                                                                                                                                                                                                                         | List all teams.                                                                                                                                                                                                                                                                       |
| `fixed:teams:writer`                            | `fixed_xw1T0579h620MOYi4L96GUs7fZY` | `teams:create`<br>`teams:delete`<br>`teams:read`<br>`teams:write`<br>`teams.permissions:read`<br>`teams.permissions:write`                                                                                                                                                                                                           | Create, read, update and delete teams and manage team memberships.                                                                                                                                                                                                                    |
| `fixed:usagestats:reader`                       | `fixed_eAM0azEvnWFCJAjNkUKnGL_1-bU` | `server.usagestats.report:read`                                                                                                                                                                                                                                                                                                      | View usage statistics report.                                                                                                                                                                                                                                                         |
| `fixed:users:reader`                            | `fixed_buZastUG3reWyQpPemcWjGqPAd0` | `users:read`<br>`users.quotas:read`<br>`users.authtoken:read`                                                                                                                                                                                                                                                                        | Read all users and their information, such as team memberships, authentication tokens, and quotas.                                                                                                                                                                                    |
| `fixed:users:writer`                            | `fixed_wjzgHHo_Ux25DJuELn_oiAdB_yM` | All permissions from `fixed:users:reader` and <br>`users:write`<br>`users:create`<br>`users:delete`<br>`users:enable`<br>`users:disable`<br>`users.password:write`<br>`users.permissions:write`<br>`users:logout`<br>`users.authtoken:write`<br>`users.quotas:write`                                                                 | Read and update all attributes and settings for all users in Grafana: update user information, read user information, create or enable or disable a user, make a user a Grafana administrator, sign out a user, update a user’s authentication token, or update quotas for all users. |

### Alerting roles

You can use predefined roles to manage user access to alert rules, alert instances, and alert notification settings and create custom roles to limit user access to alert rules in a folder.

Access to Grafana alert rules is an intersection of many permissions:

- Permission to read a folder. For example, the fixed role `fixed:folders:reader` includes the action `folders:read` and a folder scope `folders:id:`.
- Permission to query **all** data sources that a given alert rule uses. If a user cannot query a given data source, they cannot see any alert rules that query that data source.

There is only one exclusion. Role `fixed:alerting.provisioning:writer` does not require user to have any additional permissions and provides access to all aspects of the alerting configuration via special provisioning API.

For more information about the permissions required to access alert rules, refer to [Create a custom role to access alerts in a folder](ref:plan-rbac-rollout-strategy-create-a-custom-role-to-access-alerts-in-a-folder).

#### Alerting basic roles

The following table lists the default RBAC alerting role assignments to the basic roles:

| Basic role | Associated fixed roles                                                                                                                                                                                                                                                                                                                                                    | Description                                                                     |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| Admin      | `fixed:alerting:admin`<br>`fixed:alerting.provisioning.secrets:reader`<br>`fixed:alerting.provisioning:writer`                                                                                                                                                                                                                                                            | Default [Grafana organization administrator](ref:rbac-basic-roles) assignments. |
| Editor     | `fixed:alerting.rules:writer`<br>`fixed:alerting.instances:writer`<br>`fixed:alerting.receivers:writer`<br>`fixed:alerting.templates:writer`<br>`fixed:alerting.time-intervals:writer`<br>`fixed:alerting.routes:writer`<br>`fixed:alerting.inhibition-rules:writer`<br>`fixed:alerting.notifications.external:writer`<br>`fixed:alerting.provisioning.provenance:writer` | Default [Editor](ref:rbac-basic-roles) assignments.                             |
| Viewer     | `fixed:alerting.rules:reader`<br>`fixed:alerting.instances:reader`<br>`fixed:alerting.receivers:reader`<br>`fixed:alerting.templates:reader`<br>`fixed:alerting.time-intervals:reader`<br>`fixed:alerting.routes:reader`<br>`fixed:alerting.inhibition-rules:reader`<br>`fixed:alerting.notifications.external:reader`                                                    | Default [Viewer](ref:rbac-basic-roles) assignments.                             |

# RBAC for app plugins

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](/docs/grafana-cloud).
{{< /admonition >}}

RBAC can be used to manage access to [app plugins](https://grafana.com/docs/grafana/latest/administration/plugin-management/#app-plugins).
Each app plugin grants the basic Viewer, Editor and Admin organization roles a default set of plugin permissions.
You can use RBAC to restrict which app plugins a basic organization role has access to.
Some app plugins have fine-grained RBAC support, which allows you to grant additional access to these app plugins to teams and users regardless of their basic organization roles.

## Restricting access to app plugins

By default, Viewers, Editors and Admins have access to all App Plugins that their organization role allows them to access.
To change this default behavior and prevent a basic organization role from accessing an App plugin, you must [update the basic role's permissions](ref:manage-rbac-roles-update-basic-role-permissions).
See an example of [preventing Viewers from accessing an app plugin](ref:restrict-access-to-app-plugin-example) to learn more.
To grant access to a limited set of app plugins, you will need plugin IDs. You can find them in `plugin.json` files or in the URL when you open the app plugin in the Grafana Cloud UI.

Note that unless an app plugin has fine-grained RBAC support, it is not possible to grant access to this app plugin for a user whose organization role does not have access to that app plugin.

## Fine-grained access to app plugins

Plugins with fine-grained RBAC support allow you to manage access to plugin features at a more granular level.
For instance, you can grant admin access to an app plugin to a user with Viewer organization role. Or restrict the Editor organization role from being able to edit plugin resources.

For a centralized reference of all Grafana Cloud app plugin roles, default permissions by basic role, and available plugin roles, refer to [Grafana Cloud app plugin role definitions](ref:plugin-role-definitions). Some plugins also have dedicated RBAC documentation with additional context and use cases, linked in the table below.

The following table lists app plugins that have fine-grained RBAC support:

| App plugin                                                                                                                                                                            | App plugin ID                      | App plugin permission documentation                                                                                                                                                              |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [Access policies](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/)                                                          | `grafana-auth-app`                 | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Adaptive Logs](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/reduce-costs/logs-costs/adaptive-logs/)                                                            | `grafana-adaptivelogs-app`         | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Adaptive Metrics](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/reduce-costs/metrics-costs/control-metrics-usage-via-adaptive-metrics/adaptive-metrics-plugin/) | `grafana-adaptive-metrics-app`     | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Adaptive Traces](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/reduce-costs/traces-costs/)                                                                      | `grafana-adaptivetraces-app`       | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Agent Observability](https://grafana.com/docs/grafana-cloud/observe-and-act/agent-observability/)                                                                                    | `grafana-agento11y-app`            | [Agent Observability security and access controls](https://grafana.com/docs/grafana-cloud/observe-and-act/agent-observability/privacy-and-security/security/)                                    |
| [Application Observability](https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/)                                                                   | `grafana-app-observability-app`    | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Attributions](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/cost-attributions/)                                                                                 | `grafana-attributions-app`         | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Cloud Provider](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/)                                                                               | `grafana-csp-app`                  | [Cloud Provider Observability role-based access control](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/rbac/)                                             |
| [Cost Management and Billing](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/)                                                                                    | `grafana-cmab-app`                 | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Database Observability](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/database-observability/)                                                                       | `grafana-dbo11y-app`               | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Fleet Management](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/)                                                                                                | `grafana-collector-app`            | [Role-based access control for Fleet Management](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/set-up/role-based-access-control/)                                            |
| [Frontend Observability](https://grafana.com/docs/grafana-cloud/monitor-applications/frontend-observability/)                                                                         | `grafana-kowalski-app`             | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Grafana Assistant](https://grafana.com/docs/grafana-cloud/alerting-and-irm/machine-learning/llm-plugin/)                                                                             | `grafana-assistant-app`            | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Incident](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/incident/)                                                                                                     | `grafana-incident-app`             | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Integrations and Connections](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/)                                                                                     | `grafana-easystart-app`            | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [IRM](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/)                                                                                                                   | `grafana-irm-app`                  | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [IRM Labels](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/)                                                                                                            | `grafana-labels-app`               | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Knowledge Graph](https://grafana.com/docs/grafana-cloud/monitor-applications/asserts/)                                                                                               | `grafana-asserts-app`              | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Kubernetes Monitoring](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/)                                                                                            | `grafana-k8s-app`                  | [Kubernetes Monitoring role-based access control](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/control-access/#precision-access-with-rbac-custom-plugin-roles) |
| [Machine Learning](https://grafana.com/docs/grafana-cloud/alerting-and-irm/machine-learning/)                                                                                         | `grafana-ml-app`                   | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [OnCall](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/oncall/)                                                                                                         | `grafana-oncall-app`               | [Configure RBAC for OnCall](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/oncall/manage/user-and-team-management/#manage-users-and-teams-for-grafana-oncall)                       |
| [Performance Testing (k6)](https://grafana.com/docs/grafana-cloud/testing/k6/)                                                                                                        | `k6-app`                           | [Configure RBAC for k6](https://grafana.com/docs/grafana-cloud/testing/k6/projects-and-users/configure-rbac/)                                                                                    |
| [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/)                                                    | `grafana-pdc-app`                  | [Plugin role definitions](ref:plugin-role-definitions)                                                                                                                                           |
| [Service Level Objective (SLO)](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/)                                                                                         | `grafana-slo-app`                  | [Configure RBAC for SLO](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/set-up/rbac/)                                                                                               |
| [Session Replay](https://grafana.com/docs/grafana-cloud/monitor-applications/frontend-observability/session-replay/)                                                                  | `grafana-sessionreplay-app`        | [Session Replay role-based access control](https://grafana.com/docs/grafana-cloud/monitor-applications/frontend-observability/session-replay/rbac/)                                              |
| [Synthetic Monitoring](https://grafana.com/docs/grafana-cloud/testing/synthetic-monitoring/)                                                                                          | `grafana-synthetic-monitoring-app` | [Configure RBAC for Synthetic Monitoring](https://grafana.com/docs/grafana-cloud/testing/synthetic-monitoring/user-and-team-management/)                                                         |

### Revoke fine-grained access from app plugins

To list all the permissions granted to a basic role, use the [HTTP API endpoint to query for the role](https://grafana.com/docs/grafana/latest/developers/http_api/access_control/#get-a-role). Basic role UIDs are listed in [RBAC role definitions list](ref:rbac-role-definitions).

To remove any undesired plugin permissions from a basic role, [update the basic role's permissions](ref:manage-rbac-roles-update-basic-role-permissions).

## Grant additional access to app plugins

To grant access to app plugins, you can use the predefined [fixed plugin roles](https://grafana.com/docs/grafana/latest/administration/roles-and-permissions/access-control/#fixed-roles) or create [custom roles](https://grafana.com/docs/grafana/latest/administration/roles-and-permissions/access-control/#custom-roles) with specific plugin permissions.
To learn about how to assign an RBAC role, refer to [the documentation on assigning RBAC roles](https://grafana.com/docs/grafana/latest/administration/roles-and-permissions/access-control/assign-rbac-roles/#assign-rbac-roles).

# Provisioning RBAC with Grafana

{{< admonition type="note" >}}
Available in [Grafana Enterprise](/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) for self-managed instances. This feature is not available in Grafana Cloud.
{{< /admonition >}}

You can create, change or remove [Custom roles](ref:manage-rbac-roles-create-custom-roles-using-provisioning) and create or remove [basic role assignments](ref:assign-rbac-roles-assign-a-fixed-role-to-a-basic-role-using-provisioning), by adding one or more YAML configuration files in the `provisioning/access-control/` directory.

Because this method requires access to the file system where Grafana is running, it's only available for self-managed Grafana instances. To provision RBAC in Grafana Cloud, use [Terraform](ref:rbac-terraform-provisioning) or the [HTTP API](ref:api-rbac-create-and-manage-custom-roles).

Grafana performs provisioning during startup. After you make a change to the configuration file, you can reload it during runtime. You do not need to restart the Grafana server for your changes to take effect.

**Before you begin:**

- Ensure that you have access to files on the server where Grafana is running.

**To manage and assign RBAC roles using provisioning:**

1. Sign in to the Grafana server.

1. Locate the Grafana provisioning folder.

1. Create a new YAML in the following folder: **provisioning/access-control**. For example, `provisioning/access-control/custom-roles.yml`

1. Add RBAC provisioning details to the configuration file.

   Refer to [Manage RBAC roles](ref:rbac-manage-rbac-roles) and [Assign RBAC roles](ref:rbac-assign-rbac-roles) for instructions.

   Refer to [example role provisioning file](#example-role-configuration-file-using-grafana-provisioning) for a complete example of a provisioning file.

1. Reload the provisioning configuration file.

   For more information about reloading the provisioning configuration at runtime, refer to [Reload provisioning configurations](/docs/grafana/<GRAFANA_VERSION>/developers/http_api/admin/#reload-provisioning-configurations).

## Example role configuration file using Grafana provisioning

The following example shows a complete YAML configuration file that:

- Create custom roles
- Delete custom roles
- Update basic roles permissions
- Assign roles to teams
- Revoke assignments of roles to teams

### Example

```yaml
---
# config file version
apiVersion: 2

# <list> list of roles to insert/update/delete
roles:
  # <string, required> name of the role you want to create or update. Required.
  - name: 'custom:users:writer'
    # <string> uid of the role. Has to be unique for all orgs.
    uid: customuserswriter1
    # <string> description of the role, informative purpose only.
    description: 'Create, read, write users'
    # <int> version of the role. Has to be greater than the stored role version to apply updates. Increase by 1 when you change the role.
    version: 2
    # <int> org id. Defaults to Grafana's default if not specified.
    orgId: 1
    # <list> list of the permissions granted by this role.
    permissions:
      # <string, required> action allowed.
      - action: 'users:read'
        #<string> scope it applies to.
        scope: 'users:*'
      - action: 'users:write'
        scope: 'users:*'
      - action: 'users:create'
      # NOTE: the `datasources:query` action implies `datasources:read`
      # Optional `datasourceType` for scopes `datasources:uid:<DATASOURCE_UID>`.
      # If you omit it, Grafana resolves the plugin type from the data source when this file is provisioned.
      # It is required if there are two datasources with the same uid.
      - action: 'datasources:query'
        scope: 'datasources:uid:loki-uid-here'
        datasourceType: loki
  - name: 'custom:global:users:reader'
    # <bool> overwrite org id and creates a global role.
    global: true
    # <string> state of the role. Defaults to 'present'. If 'absent', role will be deleted.
    state: 'absent'
    # <bool> force deletion revoking all grants of the role.
    force: true
  - uid: 'basic_editor'
    # <bool> always apply the specified changes to the role, regardless of the role version in the database
    overrideRole: true
    global: true
    # <list> list of roles to copy permissions from.
    from:
      - uid: 'basic_editor'
        global: true
      - name: 'fixed:users:writer'
        global: true
    # <list> list of the permissions to add/remove on top of the copied ones.
    permissions:
      - action: 'users:read'
        scope: 'users:*'
      - action: 'users:write'
        scope: 'users:*'
        # <string> state of the permission. Defaults to 'present'. If 'absent', the permission will be removed.
        state: absent

# <list> list role assignments to teams to create or remove.
teams:
  # <string, required> name of the team you want to assign roles to. Required.
  - name: 'Users writers'
    # <int> org id. Will default to Grafana's default if not specified.
    orgId: 1
    # <list> list of roles to assign to the team
    roles:
      # <string> uid of the role you want to assign to the team.
      - uid: 'customuserswriter1'
        # <int> org id. Will default to Grafana's default if not specified.
        orgId: 1
      # <string> name of the role you want to assign to the team.
      - name: 'fixed:users:writer'
        # <bool> overwrite org id to specify the role is global.
        global: true
        # <string> state of the assignment. Defaults to 'present'. If 'absent', the assignment will be revoked.
        state: absent
```

## Useful Links

[Provisioning RBAC setup with Terraform](ref:rbac-terraform-provisioning)

[Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/)

# Provisioning RBAC with Terraform

{{< admonition type="note" >}}
Available in [Grafana Enterprise](/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](/docs/grafana-cloud).
{{< /admonition >}}

You can create, change or remove [Custom roles](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/role) and create or remove [role assignments](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/role_assignment), by using [Terraform's Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

Before you provision roles and assignments, decide which roles to create and how to assign them to users and teams. For planning guidance, refer to [Plan your RBAC rollout strategy](ref:plan-rbac-rollout-strategy).

## Before you begin

- Ensure you have the grafana/grafana [Terraform provider](https://registry.terraform.io/providers/grafana/grafana/) 1.29.0 or higher.

- Ensure you are using Grafana 9.2 or higher.

- Decide how to authenticate the provider. We recommend a [service account token](ref:service-accounts) for self-managed Grafana and Grafana Cloud. For more information, refer to [Create a service account token for provisioning](#create-a-service-account-token-for-provisioning).

## Create a service account token for provisioning

We recommend using service account tokens for provisioning. [Service accounts](ref:service-accounts) support fine grained permissions, which allows you to easily authenticate and use the minimum set of permissions needed to provision your RBAC infrastructure.

To create a service account token for provisioning, complete the following steps.

1. [Create a new service account](ref:service-accounts-create-a-service-account-in-grafana) for your CI pipeline.
1. [Assign permissions to service account](ref:service-accounts-assign-roles-to-a-service-account-in-grafana):
   - You will need roles “Role reader”, "Role writer" and roles including any permissions that will be provisioned. For example, to create or assign a role that allows creating users, a service account needs permissions to create users.
   - Alternatively, you can assign "Admin" basic role to the service account.
1. [Create a new service account token](ref:service-accounts-to-add-a-token-to-a-service-account) for use in Terraform.

Alternatively, you can use basic authentication. To view all the supported authentication formats, refer to the [provider authentication documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs#authentication).

## Configure the Terraform provider

RBAC support is included as part of the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

The following example configures the provider. To keep your service account token out of source control, set the credentials with the `GRAFANA_URL` and `GRAFANA_AUTH` environment variables instead of hardcoding them in the configuration:

```terraform
terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 1.29.0"
    }
  }
}

# Reads credentials from the GRAFANA_URL and GRAFANA_AUTH environment variables
provider "grafana" {}
```

Export the environment variables before you run Terraform:

```bash
export GRAFANA_URL="<GRAFANA_URL>"
export GRAFANA_AUTH="<SERVICE_ACCOUNT_TOKEN>"
```

Replace the following placeholders:

- _`<GRAFANA_URL>`_: The base URL of your Grafana instance, for example `https://<YOUR_STACK_NAME>.grafana.net` on Grafana Cloud or `http://localhost:3000` for a local instance.
- _`<SERVICE_ACCOUNT_TOKEN>`_: A service account token with permission to manage the roles and assignments that you provision, for example `glsa_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`.

Alternatively, set the `url` and `auth` attributes directly in the `provider` block. To view all the supported authentication formats, refer to the [provider authentication documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs#authentication).

## Provision basic roles

Basic roles (`None`, `Viewer`, `Editor`, `Admin`, and `Grafana Admin`) correspond to a user's or service account's organization role. A basic role's permissions are derived from the organization role, so you manage basic roles by setting the organization role rather than by creating an RBAC role assignment. The `grafana_role_assignment` resource only assigns fixed and custom roles.

To change which permissions a basic role grants, rather than who has the role, edit the basic role's permissions instead. For more information, refer to [Manage RBAC roles](ref:manage-rbac-roles).

{{< admonition type="note" >}}
Assigning a basic role such as `basic_admin` with `grafana_role_assignment` fails with the error `this endpoint cannot be used to assign basic, managed or external services roles`.
{{< /admonition >}}

### Set the organization role for a service account

Set the `role` attribute on the [`grafana_service_account`](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/service_account) resource:

```terraform
resource "grafana_service_account" "admin_sa" {
  name = "terraform_admin_sa"
  role = "Admin"
}
```

### Set the organization role for users

How you set a user's organization role depends on your Grafana deployment.

**Self-managed Grafana:** use the [`grafana_organization`](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/organization) resource to manage members by organization role. This resource uses Grafana's admin API, so it requires basic authentication and manages organization membership authoritatively.

```terraform
resource "grafana_organization" "org" {
  name    = "my_org"
  admins  = ["admin@example.com"]
  editors = ["editor@example.com"]
  viewers = ["viewer@example.com"]
}
```

**Grafana Cloud:** the `grafana_organization` resource isn't supported, and no Terraform resource currently sets an individual user's organization role. Manage organization roles with the `PATCH /api/org/users/{user_id}` [Organization HTTP API](ref:org-http-api-update-user) endpoint, or through SCIM provisioning or SAML or OIDC role mapping.

### Assign a fixed or custom role to a team

Use fixed or custom roles to grant permissions to teams:

```terraform
resource "grafana_team" "writers_team" {
  name = "terraform_writers_team"
}

# Assign a fixed role to a team
resource "grafana_role_assignment" "writers_team_fixed_role" {
  role_uid = "fixed:dashboards:writer"
  teams    = [grafana_team.writers_team.id]
}
```

## Provision custom roles

The following example shows how to provision a custom role with some permissions.

1. Copy this code block into a .tf file on your local machine.

```terraform
resource "grafana_role" "my_new_role" {
  name        = "custom:users:manager"
  description = "Manage organization users and teams"
  uid         = "customusersmanager"
  version     = 1
  global      = false

  # Manage organization users
  permissions {
    action = "org.users:read"
    scope  = "users:*"
  }
  permissions {
    action = "org.users:add"
    scope  = "users:*"
  }
  permissions {
    action = "org.users:write"
    scope  = "users:*"
  }

  # Manage teams
  permissions {
    action = "teams:create"
  }
  permissions {
    action = "teams:read"
    scope  = "teams:*"
  }
  permissions {
    action = "teams:write"
    scope  = "teams:*"
  }
}
```

For a reference of the available actions and scopes that you can grant in a custom role, refer to [RBAC actions and scopes](ref:custom-role-actions-scopes).

2. Run the command `terraform apply`.
3. Go to Grafana's UI and check that the new role appears in the role picker:
   ![Role Picker](/static/img/docs/enterprise/tf_custom_role.png)

## Provision role assignments

The following example shows how to provision role assignments.
In this example a team, user and service account are provisioned, and the custom role from the previous example is assigned to them.

1. Extend the configuration file from the [previous example](#provision-custom-roles) with the following:

```terraform
resource "grafana_team" "test_team" {
	name = "terraform_test_team"
}

resource "grafana_user" "test_user" {
	email = "terraform_user@test.com"
	login    = "terraform_test_user"
	password = <TEST_PASSWORD>
}

resource "grafana_service_account" "test_sa" {
  name = "terraform_test_sa"
  role = "Viewer"
}

resource "grafana_role_assignment" "my_new_role_assignment" {
  role_uid = grafana_role.my_new_role.uid
  users = [grafana_user.test_user.id]
  teams = [grafana_team.test_team.id]
  service_accounts = [grafana_service_account.test_sa.id]
}
```

1. Substitute `<TEST_PASSWORD>` with a test password for your test user.

1. Run the command `terraform apply`.

1. Go to Grafana's UI and check that a user, team and service account have been created, and that the role has been assigned to them:
   ![User Role Assignment](/static/img/docs/enterprise/tf_user_role_assignment.png)
   ![Team Role Assignment](/static/img/docs/enterprise/tf_team_role_assignment.png)
   ![Service Account Role Assignment](/static/img/docs/enterprise/tf_service_account_role_assignment.png)

Note that instead of using a provisioned role, you can also look up the `uid` of an already existing fixed or custom role and use that instead.
You can use the [API endpoint for listing roles](ref:api-rbac-create-and-manage-custom-roles) to look up role `uid`s.
Similarly, you can look up and use `id`s of users, teams and service accounts that have not been provisioned to assign roles to them.

## Useful links

- [Plan your RBAC rollout strategy](ref:plan-rbac-rollout-strategy)
- [Manage RBAC roles](ref:manage-rbac-roles)
- [RBAC setup with Grafana provisioning](ref:rbac-grafana-provisioning)
- [Grafana Cloud Terraform provisioning](/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/)

# Troubleshooting RBAC

In this section, you’ll learn about logs that are available for RBAC and you’ll find the most common RBAC issues.

## Enable debug logging

You can enable debug log messages for RBAC in the Grafana configuration file. Debug logs are added to the Grafana server logs.

```bash
[log]
filters = accesscontrol:debug accesscontrol.evaluator:debug dashboard.permissions:debug
```

## Enable audit logging

{{< admonition type="note" >}}
Available in [Grafana Enterprise](/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](/docs/grafana-cloud).
{{< /admonition >}}

You can enable auditing in the Grafana configuration file.

```bash
[auditing]
enabled = true
```

All permission and role updates, and role assignments are added to audit logs.
Learn more about [access control audit logs](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/audit-grafana/#access-control).

## Missing dashboard, folder or data source permissions

[Dashboard and folder permissions](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/#dashboard-permissions) and [data source permissions](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/#data-source-permissions) can go out of sync if a Grafana instance version is upgraded, downgraded and then upgraded again.
This happens when an instance is downgraded from a version that uses RBAC to a version that uses the legacy access control, and dashboard, folder or data source permissions are updated.
These permission updates will not be applied to RBAC, so permissions will be out of sync when the instance is next upgraded to a version with RBAC.

{{< admonition type="note" >}}
the steps provided below will set all dashboard, folder and data source permissions to what they are set to with the legacy access control.
If you have made dashboard, folder or data source permission updates with RBAC enabled, these updates will be wiped.
{{< /admonition >}}

To resynchronize the permissions:

1. make a backup of your database
1. run the following SQL queries
   ```sql
   DELETE
   FROM builtin_role
   where role_id IN (SELECT id
                     FROM role
                     WHERE name LIKE 'managed:%');
   DELETE
   FROM team_role
   where role_id IN (SELECT id
                     FROM role
                     WHERE name LIKE 'managed:%');
   DELETE
   FROM user_role
   where role_id IN (SELECT id
                     FROM role
                     WHERE name LIKE 'managed:%');
   DELETE
   FROM permission
   where role_id IN (SELECT id
                     FROM role
                     WHERE name LIKE 'managed:%');
   DELETE
   FROM role
   WHERE name LIKE 'managed:%';
   DELETE
   FROM migration_log
   WHERE migration_id IN ('teams permissions migration',
                          'dashboard permissions',
                          'dashboard permissions uid scopes',
                          'data source permissions',
                          'data source uid permissions',
                          'managed permissions migration',
                          'managed folder permissions alert actions repeated migration',
                          'managed permissions migration enterprise');
   ```
1. restart your Grafana instance

# Folder access control

Folders are the primary way to organize and control access to resources in Grafana. When you assign permissions to a folder, those permissions apply to all resources within that folder, including dashboards, alert rules, SLOs, and more.

This page explains how folder permissions work and how to use them effectively to manage access for teams and users.

## Before you begin

- You need Admin permission on a folder to manage its permissions
- To create folders, you need the Folder Creator role or appropriate organization permissions
- Folder permissions are available in all Grafana editions (OSS, Enterprise, and Cloud)

{{< admonition type="caution" >}}
**Folder limitations:**

- Folders can be nested up to **4 levels deep**
- Folder names cannot contain underscores (`_`) or percent signs (`%`)
  {{< /admonition >}}

## Default permissions for new folders

How a folder is created determines its initial permissions:

- **Folders created in the UI** are automatically granted: Admin role gets Admin permission, Editor role gets Edit permission, Viewer role gets View permission
- **Folders created via as-code** (Terraform, API, provisioning) only have permissions explicitly defined in the configuration, plus Admin role gets Admin permission by default

## How folder permissions work

Folder permissions follow a simple principle: **a user's effective access to a resource is determined by their folder permission level**.

When you grant a user or team permission on a folder:

- The permission applies to the folder itself
- The permission cascades to all subfolders
- The permission applies to all resources in those folders (dashboards, alert rules, etc.)

### Permission levels

Folders support three permission levels. Each level includes all capabilities of the levels below it.

| Permission | Folder capabilities                                        | Resource capabilities                                                                                                                               |
| ---------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| **View**   | View folder and navigate its contents                      | View dashboards, read alert rules, read silences, read annotations, read library panels                                                             |
| **Edit**   | Create subfolders, modify folder properties, delete folder | Create and edit dashboards, create and write alert rules, create and write silences, create and write annotations, create and manage library panels |
| **Admin**  | Manage folder permissions                                  | All Edit capabilities plus manage dashboard permissions                                                                                             |

#### Detailed action breakdown

**View permission grants:**

- `folders:read` - View the folder in navigation
- `dashboards:read` - View dashboards in the folder
- `alert.rules:read` - View alert rules in the folder
- `alert.silences:read` - View alert silences
- `annotations:read` - View annotations
- `library.panels:read` - View library panels

**Edit permission grants (in addition to View):**

- `folders:write` - Modify folder name and properties
- `folders:delete` - Delete the folder
- `folders:create` - Create subfolders
- `dashboards:create` - Create new dashboards
- `dashboards:write` - Edit existing dashboards
- `dashboards:delete` - Delete dashboards
- `alert.rules:create` - Create new alert rules
- `alert.rules:write` - Edit alert rules
- `alert.rules:delete` - Delete alert rules
- `alert.silences:create` - Create silences
- `alert.silences:write` - Edit silences
- `annotations:create` - Create annotations
- `annotations:write` - Edit annotations
- `annotations:delete` - Delete annotations
- `library.panels:create` - Create library panels
- `library.panels:write` - Edit library panels
- `library.panels:delete` - Delete library panels

**Admin permission grants (in addition to Edit):**

- `folders.permissions:read` - View folder permissions
- `folders.permissions:write` - Modify folder permissions
- `dashboards.permissions:read` - View dashboard permissions
- `dashboards.permissions:write` - Modify dashboard permissions

### Permission inheritance

Permissions cascade from parent folders to child folders. A user with Edit permission on a parent folder automatically has Edit permission on all subfolders.

```
Production/                    # User has Edit permission here
├── Team A Dashboards/         # User inherits Edit permission
│   └── Service Metrics/       # User inherits Edit permission
└── Team B Dashboards/         # User inherits Edit permission
```

{{< admonition type="note" >}}
You cannot grant a user lower permissions on a subfolder than they have on a parent folder. Permission inheritance always flows downward.
{{< /admonition >}}

## Resources that use folder permissions

The following resources are stored in folders and respect folder permissions:

| Resource           | How folder permissions apply                                                                                  |
| ------------------ | ------------------------------------------------------------------------------------------------------------- |
| **Dashboards**     | View/Edit/Admin directly controlled by folder permission                                                      |
| **Subfolders**     | Inherit parent folder permissions, can add additional permissions                                             |
| **Alert rules**    | View/Edit controlled by folder permission                                                                     |
| **Alert silences** | View/Create/Edit controlled by folder permission                                                              |
| **Library panels** | View/Create/Edit/Delete controlled by folder permission                                                       |
| **SLOs**           | Folder permission + SLO role determines effective access (refer to [Plugin permissions](#plugin-permissions)) |
| **Annotations**    | Stored on dashboards; inherit permissions from the dashboard's parent folder                                  |

### Plugin permissions

Some Grafana Cloud plugins use a two-layer permission model where effective access is determined by both folder permissions and a plugin-specific role:

```
Effective plugin access = minimum(folder permission, plugin role)
```

For example, with SLOs:

- A user with folder **Edit** permission and **SLO Writer** role can create and edit SLOs in that folder
- A user with folder **View** permission and **SLO Writer** role can only view SLOs (limited by folder)
- A user with folder **Edit** permission and **SLO Reader** role can only view SLOs (limited by plugin role)

For more information, refer to [SLO access control](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/set-up/rbac/).

## Manage folder permissions

Folders are accessed through the **Dashboards** section in Grafana. You can manage permissions from the folder's context menu or from within the folder view.

### View current permissions

1. In the left navigation, click **Dashboards**.
1. Navigate to the folder you want to inspect.
1. Click the folder name to open it, or click the **⋮** menu next to the folder.
1. Select **Manage permissions**.

The permissions dialog shows all users, teams, and roles with access to this folder.

### Add a permission

1. Open the folder's **Manage permissions** dialog.
1. Click **Add a permission**.
1. Select who to grant access to:
   - **User** - A specific user account
   - **Team** - All members of a team
   - **Service Account** - A service account for API or automation access
   - **Role** - Users with a specific organization role (Viewer, Editor, Admin)
1. Select the permission level (View, Edit, or Admin).
1. Click **Save**.

### Change a permission

1. Open the folder's **Manage permissions** dialog.
1. Find the user, team, or role in the list.
1. Use the permission dropdown to select a new level.
1. The change saves automatically.

### Remove a permission

1. Open the folder's **Manage permissions** dialog.
1. Find the user, team, or role in the list.
1. Click the **×** button to remove their access.
1. Confirm the removal.

{{< admonition type="warning" >}}
Removing a permission removes access to the folder and all its contents. The user or team will no longer see dashboards, alert rules, or other resources in that folder.
{{< /admonition >}}

## Design your folder structure to manage permissions

A well-designed folder structure makes permission management simpler. The recommended pattern is to create top-level folders for each team, with subfolders for organizing content:

```
SRE Team/                      # SRE team has Admin
├── Production Monitoring/     # Inherited Admin
├── On-Call Dashboards/        # Inherited Admin
└── Runbooks/                  # Inherited Admin

Platform Team/                 # Platform team has Admin
├── Infrastructure/            # Inherited Admin
└── Cost Tracking/             # Inherited Admin

Shared/                        # Everyone has View, specific teams have Edit
├── Company KPIs/              # Marketing team has Edit
└── Executive Dashboards/      # Leadership has View
```

This pattern provides:

- **Team ownership**: Each team manages their own folder and contents
- **Clear boundaries**: Resources are organized by responsibility
- **Collaboration**: Shared folders enable cross-team visibility

For detailed guidance on folder organization patterns, including automation with Terraform and SCIM provisioning, refer to:

- [Configure multi-team access]({{< relref "../../../setup-grafana/configure-access/multi-team-access" >}})
- [Managing access in Grafana: A single-stack journey with teams, roles, and real-world patterns](https://grafana.com/blog/managing-access-in-grafana-a-single-stack-journey-with-teams-roles-and-real-world-patterns/)

## Related documentation

- [Manage dashboard permissions]({{< relref "../../user-management/manage-dashboard-permissions" >}}) - Dashboard-level permission management
- [Configure multi-team access]({{< relref "../../../setup-grafana/configure-access/multi-team-access" >}}) - Recommended folder structures for organizations
- [RBAC fixed roles]({{< relref "./access-control/rbac-fixed-basic-role-definitions" >}}) - Enterprise folder-related RBAC roles
- [Alerting folder permissions]({{< relref "../../../alerting/set-up/configure-rbac/access-folders" >}}) - How folders control alert rule access
- [Team management]({{< relref "../../team-management" >}}) - Create and manage teams for folder access

# Migrate API keys to service account tokens

{{< admonition type="note" >}}
API keys are deprecated. [Service accounts](ref:service-accounts) now replace API keys for authenticating with the **HTTP APIs** and interacting with Grafana.
{{< /admonition >}}

API keys specify a role—either **Admin**, **Editor**, or **Viewer**—that determine the permissions associated with interacting with Grafana.

Compared to API keys, service accounts have limited scopes that provide more security. For more information on the benefits of service accounts, refer to [service account benefits](ref:service-accounts-benefits).

When you migrate an API key to a service account, a service account is created with a service account token. Your existing API key—now migrated to a service account token—will continue working as before.

To find the migrated API keys, click **Administration** in the left-side menu, then **Users and access -> Service Accounts**, select the service account, and locate the **Token**.

## Migrate API keys using the Grafana user interface

This section shows you how to migrate API keys to Grafana service accounts using the Grafana user interface. You can choose to migrate a single API key or all API keys. When you migrate all API keys, you can no longer create API keys and must use service accounts instead.

#### Before you begin

To follow these instructions, you need at least one of the following:

- Administrator permissions
- Editor permissions
- Service account writer

For more information about permissions, refer to [Roles and permissions](ref:roles-and-permissions).

#### Steps

To migrate all API keys to service accounts, complete the following steps:

1. Sign in to Grafana, point to **Administration**, **Users and access**, and click **API Keys**.
1. In the top of the page, find the section which says **Switch from API keys to service accounts**
1. Click **Migrate to service accounts now**.
1. A confirmation window will appear, asking to confirm the migration. Click **Yes, migrate now** if you are willing to continue.
1. Once migration is successful, you can choose to forever hide the API keys page. Click **Hide API keys page forever** if you want to do that.

To migrate a single API key to a service account, complete the following steps:

1. Sign in to Grafana.
1. Click **Administration** in the left-side menu, **Users and access**, and select **API Keys**.
1. Find the API Key you want to migrate.
1. Click **Migrate to service account**.

## Migrate API keys using the HTTP API

This section shows you how to programmatically migrate API keys to Grafana service accounts using the HTTP API. For API additional information, refer to [Service account HTTP APIs](ref:api-service-account).

#### Before you begin

To follow these instructions, you need one of the following:

- Administrator permissions
- Editor permissions
- Service account writer

#### Steps

Complete the following steps to migrate from API keys to service accounts for API:

1. Call the `POST /api/serviceaccounts` endpoint and the `POST /api/serviceaccounts/<id>/tokens`.

   This action generates a service account token.

1. Store the ID and secret that the system returns to you.
1. Pass the token in the `Authorization` header, prefixed with `Bearer`.

   This action authenticates API requests.

1. SATs used for authentication
1. Remove code that handles the old `/api/auth/keys` endpoint.
1. Track the [API keys](http://localhost:3000/org/apikeys) in use and migrate them to SATs.

#### Example

Your current setup

```sh
curl -X POST -H "Content-Type: application/json" -d '{"name": "my-api-key", "role": "Viewer"}' http://admin:admin@localhost:3000/api/auth/keys

# response from the api
{"id":2,"name":"my-api-key","key":"eyJrIjoiTFRSN1RBOVc3SGhjblc0bWZodXZ3MnNDcU92Um5VZUIiLKJuIjoibXktYXBpLWtleSIsImlkIjoxfQ=="}%
```

New setup

```sh
# create a service account
curl -X POST -H "Content-Type: application/json" -d '{"name": "my-service-account", "role": "Viewer"}' http://admin:admin@localhost:3000/api/serviceaccounts

# response with the created service account id,name, login
{"id":1,"name":"my-service-account","login":"sa-my-service-account","orgId":1,"isDisabled":false,"role":"Viewer","tokens":0,"avatarUrl":""}%

# create the service account token with the service account id 1 - /serviceaccounts/{id} returned from the previous step
curl -X POST -H "Content-Type: application/json" -d '{"name": "my-service-account-token"}' http://admin:admin@localhost:3000/api/serviceaccounts/1/tokens

# response with the created SAT id,name and key.
{"id":2,"name":"my-service-account-token","key":"glsa_iNValIdinValiDinvalidinvalidinva_5b582697"}%

# now you can authenticate the same way as you did with the API key
curl --request GET --url http://localhost:3000/api/folders --header 'Authorization: Bearer glsa_iNValIdinValiDinvalidinvalidinva_5b582697'

# response
[{"id":1,"uid":"a5261a84-eebc-4733-83a9-61f4713561d1","title":"gdev dashboards"}]%
```

## Migrate API keys using Terraform

{{< admonition type="note" >}}
The terraform resource `api_key` is removed from the Grafana Terraform Provider in v3.0.0.
Before you migrate and remove the use of the resource, you should pin your terraform version to a version less-than or equal-to v2.19.0.
For more information, refer to the [Grafana Terraform Provider release notes](https://github.com/grafana/terraform-provider-grafana/releases/tag/v3.0.0).
{{< /admonition >}}

To pin the Grafana Terraform Provider to v2.19.0:

```hcl
terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = "2.19.0"
    }
  }
}
```

This section shows you how to migrate your Terraform configuration for API keys to Grafana service accounts. For additional information, refer to [Grafana Service Accounts in Terraform](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/service_account_token).

#### Steps

Complete the following steps to migrate from API keys to service accounts using Terraform:

1. Generate `grafana_service_account` and `grafana_service_account_token` resources.
1. Specify the desired scopes and expiration date when creating the service account.
1. Use the token returned from `grafana_service_account_token` to authenticate the API requests.
1. Remove the terraform configuration for creating your `grafana_api_key` resources.

**Example: your current Terraform configuration**

```tf
terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
    }
  }
}

# configure the provider with basic auth
provider "grafana" {
  url  = "http://localhost:3000"
  auth = "admin:admin"
}

resource "grafana_api_key" "foo" {
  name = "key_foo"
  role = "Viewer"
}

resource "grafana_api_key" "bar" {
  name            = "key_bar"
  role            = "Admin"
  seconds_to_live = 30
}
```

**Your new Terraform configuration**

_Note:_ you can create multiple tokens using one service account.

```tf
terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
    }
  }
}

# configure the provider with basic auth
provider "grafana" {
  url  = "http://localhost:3000"
  auth = "admin:admin"
}

# Creating a service account in Grafana instance to be used as auth and attach tokens
# notice we can attach multiple tokens to one service account
resource "grafana_service_account" "sa-admin" {
  name             = "sa-admin"
  role             = "Admin"
}

# Creating a service account token in Grafana instance to be used for creating resources in Grafana instance
resource "grafana_service_account_token" "sat-bar" {
  name           = "sat-bar"
  service_account_id = grafana_service_account.sa-admin.id
}

# Creating a service account token in Grafana instance to be used for creating resources in Grafana instance
resource "grafana_service_account_token" "sat-foo" {
  name           = "sat-foo"
  service_account_id = grafana_service_account.sa-admin.id
  seconds_to_live    = 30
}
```

## Migrate Cloud Stack API keys using Terraform

This section shows you how to migrate your Terraform configuration for Grafana cloud stack API keys to Grafana cloud stack service accounts.

For migration your cloud stack api keys, use the `grafana_cloud_stack_service_account` and `gafana_cloud_stack_service_account_token` resources. For additional information, refer to [Grafana Cloud Stack Service Accounts in Terraform](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_stack_service_account).

{{< admonition type="note" >}}
This is only relevant for Grafana Cloud **Stack** API keys `grafana_cloud_stack_api_key`. Grafana Cloud API keys resource `grafana_cloud_api_key` are not deprecated and should be used for authentication for managing your Grafana cloud.
{{< /admonition >}}

#### Steps

Complete the following steps to migrate from cloud stack API keys to cloud stack service accounts using Terraform:

1. Generate `grafana_cloud_stack_service_account` and `grafana_cloud_stack_service_account_token` resources.
1. Specify the desired scopes and expiration date when creating the service account.
1. Use the token returned from `grafana_cloud_stack_service_account_token` to authenticate the API requests.
1. Remove the Terraform configuration for creating your `grafana_cloud_stack_api_key` resources.

**Example: Your current Terraform configuration**

```tf
terraform {
  required_providers {
    grafana = {
      source = "grafana/grafana"
    }
  }
}

# Declaring the first provider to be only used for creating the cloud-stack
provider "grafana" {
  alias = "cloud"

  cloud_api_key = "<API-Key>"
}

resource "grafana_cloud_stack" "my_stack" {
  provider = grafana.cloud

  name        = "my_stack"
  slug        = "my_stack"
  region_slug = "eu" # Example “us”,”eu” etc
}

# Creating a Grafana API key to be used as auth
resource "grafana_cloud_stack_api_key" "management" {
  provider = grafana.cloud

  stack_slug = grafana_cloud_stack.my_stack.slug
  name       = "management-key"
  role       = "Admin"
}
```

**Your new Terraform configuration**

```tf
terraform {
  required_providers {
    grafana = {
      source = "grafana/grafana"
    }
  }
}

# Declaring the first provider to be only used for creating the cloud-stack
provider "grafana" {
  alias = "cloud"

  cloud_api_key = "<API-Key>"
}

resource "grafana_cloud_stack" "my_stack" {
  provider = grafana.cloud

  name        = "my_stack"
  slug        = "my_stack"
  region_slug = "eu" # Example “us”,”eu” etc
}

# Creating a grafana cloud stack service account
resource "grafana_cloud_stack_service_account" "mystack_cloud-stack_service_account" {
  provider   = grafana.cloud
  stack_slug = grafana_cloud_stack.my_stack.slug

  name = "mystack-cloud-stack-sa"
  role = "Admin"
}

# Creating a grafana cloud stack service account token
resource "grafana_cloud_stack_service_account_token" "mystack_cloud-stack_service-account_token" {
  provider   = grafana.cloud
  stack_slug = grafana_cloud_stack.my_stack.slug

  name               = "mystack-cloud-stack-sa-token"
  service_account_id = grafana_cloud_stack_service_account.mystack_cloud-stack_service_account.id
}
```

# View server statistics and license

This setting contains information about tools that Grafana Server Admins can use to learn more about their Grafana servers.

## View Grafana server settings

> Refer to [Role-based access control](../roles-and-permissions/access-control/) in Grafana Enterprise to understand how you can control access with RBAC permissions.

If you are a Grafana server administrator, use the Settings tab to view the settings that are applied to your Grafana server via the [Configuration](../../setup-grafana/configure-grafana/#configuration-file-location) file and any environmental variables.

> **Note:** Only Grafana server administrators can access the **Server Admin** menu. For more information about about administrative permissions, refer to [Roles and permissions](../roles-and-permissions/#grafana-server-administrators).

### View server settings

1. Log in to your Grafana server with an account that has the Grafana Admin flag set.
1. Click **Administration** in the left-side menu, **General**, and then **Settings**.

### Available settings

For a full list of server settings, refer to [Configuration](../../setup-grafana/configure-grafana/#server).

## View Grafana server stats

> Refer to [Role-based access control](../roles-and-permissions/access-control/) in Grafana Enterprise to understand how you can control access with RBAC permissions.

If you are a Grafana server admin, then you can view useful statistics about your Grafana server in the Stats & Licensing tab.

> **Note:** Only Grafana server administrators can access the **Server Admin** menu. For more information about about administrative permissions, refer to [Roles and permissions](../roles-and-permissions/#grafana-server-administrators).

### View server stats

1. Log in to your Grafana server with an account that has the Grafana Admin flag set.
1. Click **Administration** in the left-side menu, **General**, and then **Stats and license**.

### Available stats

The following statistics are displayed in the Stats tab:

- Total users
  **Note:** Total users = Total admins + Total editors + Total viewers
- Total admins
- Total editors
- Total viewers
- Active users (seen last 30 days)
  **Note:** Active users = Active admins + Active editors + Active viewers
- Active admins (seen last 30 days)
- Active editors (seen last 30 days)
- Active viewers (seen last 30 days)
- Active sessions
- Total dashboards
- Total orgs
- Total playlists
- Total snapshots
- Total dashboard tags
- Total starred dashboards
- Total alerts

### Counting users

If a user belongs to several organizations, then that user is counted once as a user in the highest organization role they are assigned, regardless of how many organizations the user belongs to.

For example, if Sofia is a Viewer in two organizations, an Editor in two organizations, and Admin in three organizations, then she would be reflected in the stats as:

- Total users 1
- Total admins 1

# Administer Grafana Teams

This topic describes how to administer Grafana Teams.

## View a list of Teams

See the complete list of teams in your Grafana organization.

To view a list of teams:

1. Sign in to Grafana as an organization administrator or a team administrator.
1. Click the arrow next to **Administration** in the left-side menu, click **Users and access**, and select **Teams**.

The role you use to sign in to Grafana determines how you see Teams lists.

### Organization administrator view

The following example shows a list as it appears to an `organization` administrator.

![Team list view for org admin](/media/docs/grafana/screenshot-org-admin-team-list.png)

### Team administrator view

The following example shows a list as it appears to a `team` administrator.

![Team list view for team admin](/media/docs/grafana/screenshot-team-admin-team-list.png)

## Teams best practices

Grafana recommends you use Teams to organize and manage access to Grafana’s core resources, such as dashboards and alerts. Teams is an easy organizational tool to manage, and allows flexible sharing between teams.

Grafana recommends that you use Instances or Stacks to separate Teams if you want true isolation, to ensure that no information leaks between Teams. You can synchronize some resources between instances using provisioning.

## Secure Grafana Teams

The most important thing to consider for securing Teams is to only grant team administrator rights to the users you trust to administer the Team.

# Configure Grafana Teams

This topic describes how to set up and configure Grafana Teams.

For a tutorial on working with Teams, refer to [Create users and teams](https://grafana.com/tutorials/create-users-and-teams/).

## Before you begin

Before you begin creating and working with Grafana Teams:

- Ensure that you have either the `Organization Administrator` role or `team administrator` permissions.
  Refer to [Organization roles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/#organization-roles) and [RBAC permissions, actions, and scopes](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/custom-role-actions-scopes/#rbac-permissions-actions-and-scopes) for a list of Grafana roles and role-based access control actions.
- Decide which users belong to which teams and what permissions team members receive.
- Configure the default basic role for users to join Grafana. This role applies to users where no role is set by the identity provider (IDP).
  - The `None` basic role can't view any resources by default. Recommended for `isolated` teams.
  - the `Viewer` role can view all resources by default. Recommended for `collaborative` teams.
- Ensure team sync is turned on if you plan to manage team members through team sync.
  Refer to [Configure Team Sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/) for a list of providers and instructions on how to turn on team sync for each provider.

{{< admonition type="note" >}}
[Grafana Organizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/organization-management/) do not exist in Grafana Cloud.
Grafana Cloud uses the term _organization_ to refer to accounts in grafana.com, or GCOM.
In Grafana Enterprise and OSS, Teams belong to Grafana Organizations.
Refer to [About organizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/organization-management/#about-organizations) for more information.
{{< /admonition >}}

## Create a Grafana Team

A team is a group of users within a Grafana instance that have common permissions needs. Teams help make user-permission management more efficient. A user can belong to multiple Teams.
Grafana Teams includes common access to the following:

- dashboards
- data sources
- folders
- alerts
- reports
- cloud access policies
- annotations
- playlists

To create a Team, complete the following steps:

1. Sign in to Grafana as an `org administrator` or `team administrator`.
1. Click the arrow next to **Administration** in the left-side menu, click **Users and access**, and select **Teams**.
1. Click **New Team**.
1. Fill in each field and click **Create**.
1. Click **Save**. You can now add a Team member.

The **New team** page also includes a **Auto-create a team folder** option. Select **Auto-create a team folder** if you want Grafana to create a folder owned by that team during setup. For more information, refer to [Manage team folders](/docs/grafana/<GRAFANA_VERSION>/administration/team-management/team-folders/).

## Add a Team member

**Note**

> In Grafana Enterprise or Grafana Cloud with RBAC enforcement enabled, adding or viewing team members requires the `org.users:read` permission, which is typically granted only to `Organization Administrators`.
>
> In Grafana OSS (or unlicensed), `team administrators` can search for and add any user in the organization without requiring additional permission.

Add a member to a new Team or add a team member to an existing Team when you want to provide access to team dashboards and folders to another user.

To add a team member, complete the following steps:

1. Sign in to Grafana as an `org administrator` or `team administrator`.
1. Click the arrow next to **Administration** in the left-side menu, click **Users and access**, and select **Teams**.
1. Click the name of the Team to which you want to add members, and click **+ Add member**.
1. Search for and select a user.
1. Choose whether to add the user as a Team **Member** or **Admin**.
1. Click **Save**.

## Grant or change Team member permissions

Complete this task when you want to add or modify team member permissions.

To grant team member permissions:

1. Sign in to Grafana as an `org administrator` or `team administrator`.
1. Click the arrow next to **Administration** in the left-side menu, click **Users and access**, and select **Teams**.
1. Click the pencil next to the name of the Team for which you want to add or modify team member permissions.
1. In the team member list, locate the user that you want to change. You can use the search field to filter the list if necessary.
1. Under the **Permission** column, select the new permission level.

## Add roles to a Grafana Team

You can add or delete roles from a specified team.

To add a role, complete the following steps:

1. Sign in to Grafana as an `org administrator` or `team administrator`.
1. Click the arrow next to **Administration** in the left-side menu, click **Users and access**, and select **Teams**.
1. Select the Team and click under the **Role** column.
   Select from a list of current fixed or plugin roles or clear all roles and start over.
   As you hover over each role a list of permissions appears to the right.
   You can clear any permission for additional fine-grained control.
1. Click **Update** to add the new role or roles.

To delete a role, remove the check next to the role name and click **Update**.

## Delete a team

Delete a team when you no longer need it. This action permanently deletes the team and removes all team permissions from dashboards and folders. This task requires that you have `Organization Administrator` permissions.

1. Sign in to Grafana as an `org administrator` or `team administrator`.
1. Click the arrow next to **Administration** in the left-side menu, click **Users and access**, and select **Teams**.
1. Click the **red X** on the right side of the name of the team.
1. Click **Delete**.

<!-- ## Create isolated or collaborative teams

Grafana Teams can either be `isolated` or `collaborative`. Isolated teams can only see their own resources. They can't see other team’s dashboards, data, or alerts. Collaborative teams have access to other team’s resources. Grafana Cloud users must contact Support.

To create an isolated team add the following to the Grafana configuration file:

```ini
auto_assign_org_role = None
role_attribute_path = contains(groups[*], 'admin') && 'Admin' || 'None'
```

To create a collaborative team add the following to the Grafana configuration file:

```ini
auto_assign_org_role = Viewer
role_attribute_path = contains(groups[*], 'admin') && 'Admin' || 'None'
```
You can also use a terraform script as shown in the following example:

```hcl
terraform {
  required_providers {
    grafana = {
      source = "grafana/grafana"
    }
  }
}

provider "grafana" {
  url  = "http://localhost:3000/"
  auth = "admin:admin"
}

resource "grafana_folder" "awesome_folder" {
  title = "Awesome Team Folder"
}


resource "grafana_team" "awesome-team" {
  name  = "Awesome Team"
}

resource "grafana_team_external_group" "awesome-team-group" {
  team_id = grafana_team.awesome-team.id
  groups = [
    "Awesome_group"
  ]
}

resource "grafana_role" "team_role" {
  name  = "team_role"
  uid = "team_role"
  version = 4
  global = true

  permissions {
    action = "datasources:create"
  }

  permissions {
    action = "serviceaccounts:create"
  }

  # below should be deleted after bug fix to view service accounts
  permissions {
    action = "users.roles:read"
    scope = "users:id:*"
  }

}

resource "grafana_role_assignment" "assign_role" {
  role_uid         = grafana_role.team_role.uid
  teams            = [grafana_team.awesome-team.id]
}

resource "grafana_service_account" "awesome_service_account" {
  name        = "awesome_service_account"
  role        = "None"
  is_disabled = false
}

resource "grafana_service_account_permission" "awesome_service_account_permissions" {
  service_account_id = grafana_service_account.awesome_service_account.id

  permissions {
    team_id    = grafana_team.awesome-team.id
    permission = "Admin"
  }
}

resource "grafana_folder_permission" "awesome_folder_permission" {
  folder_uid = grafana_folder.awesome_folder.uid
  permissions {
    team_id    = grafana_team.awesome-team.id
    permission = "Admin"
  }

  permissions {
    user_id = grafana_service_account.awesome_service_account.id
    permission = "Admin"
  }
}
``` -->

# Manage team folders

![Folder detail page showing a team folder.](/media/whats-new/team-folders/0-Team-Folders-Overview.png)

Team folders help you associate folders with teams so admins and team members can find the right dashboards, library panels, and alert rules faster. They streamline setup by letting admins create and assign a dedicated folder when creating a team, establishing clear ownership from the start. A team folder is still a regular folder—the team association adds ownership metadata and team-aware navigation, but it doesn’t change or replace folder permission.

Team folders is in public preview.

Before you begin, ensure you have the following:

- Permission to create root-level folders, or permission to create subfolders in the parent folder you want to use.
- Permission to manage folder permissions and read teams if you want to assign or change a folder owner in the UI.

## Create a team folder when you create a team

When team folders are enabled, the **New team** page includes an option to create a team folder automatically.

To create a team and its folder, complete the following steps:

1. Open **Administration** and go to **Users and access** > **Teams**.
1. Click **New team**.
1. Enter the team details.
1. Under **Team folder**, select **Auto-create a team folder**.
1. Click **Create**.

Grafana creates a folder with the same name as the team and assigns that team as the folder owner. After the folder is created, review the folder permissions and update them if needed.

![New team page with the Team folder section visible and Auto-create a team folder selected.](/media/whats-new/team-folders/2-new-team-detail.png)

For more information about creating teams, refer to [Configure Grafana Teams](/docs/grafana/<GRAFANA_VERSION>/administration/team-management/configure-grafana-teams/).

## Create or assign a team folder from Dashboards

You can create a new folder for a team or assign ownership to an existing folder.

### Create a new team folder

To create a new team folder, complete the following steps:

1. Click **Dashboards** in the primary menu.
1. Click **New** and select **New folder**.
1. Enter a unique folder name.
1. Select **Assign an owner to the folder**.
1. In **Team** dropdown, select the team that should own the folder.
1. Click **Create**.

Grafana creates a regular folder and stores the selected team as the folder owner.

![New folder drawer with Assign an owner to the folder enabled and a team selected in the Team field.](/media/whats-new/team-folders/3-New-Folder-for-Team.png)

### Assign or change the owner of an existing folder

To assign or change the owner of an existing folder, complete the following steps:

1. Click **Dashboards** in the primary menu.
1. Open the folder that you want to update.
1. Click **Folder actions** and select **Manage folder owner**.
1. Select a team, or clear the current value if you want to remove the owner.
1. Click **Save owner**.

If a folder has a team owner, Grafana displays an **Owned by** label in the folder header.

![Folder page header showing the Owned by label and the Manage folder owner modal open from Folder actions.](/media/whats-new/team-folders/4-Manage-Owner-Folder.png)

## Find team folders

When team folders are enabled, Grafana surfaces them in the browse dashboards page and in team details.

### Browse dashboards

The **Dashboards** page displays a **My team folders** section near the top when you belong to teams that own folders. Expand that section to open the folders owned by your teams.

![Dashboards page with the My team folders section expanded and two team-owned folders listed.](/media/whats-new/team-folders/5-Find-you-team-folders.png)

### Save or move dashboards

When you are saving or moving dashboard the folder picker shows a **Team folders** section at the top of the folder tree. Your first teams folder will be preselected by default.

![Folder picker in a save or move flow with Team folders shown at the top of the tree and one team folder selected.](/media/whats-new/team-folders/6-Save-dashboard.png)

### Review folders owned by a team

Each team page includes a **Folders** tab that lists all the folders owned by that team.

![Team details page on the Folders tab showing the owned folders table with Name and Full path columns.](/media/whats-new/team-folders/8-Team-Folders---list-view-in-Team.png)

## Manage access for team folders

Folder ownership does not replace folder permissions. Team folders use the same permission model as any other folder.

If team members need access to the folder, configure folder permissions for the team or for the users, roles, or service accounts that should use it. For more information, refer to [Manage dashboard permissions](/docs/grafana/<GRAFANA_VERSION>/administration/user-management/manage-dashboard-permissions/).

## Limitations

Team folders currently have the following limitations:

- You can select only one team owner per folder.
- You can't assign or change a team owner for repository-managed folders.

## Next steps

- Refer to [Manage dashboards](/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/manage-dashboards/) for the rest of the dashboards management workflow.
- Refer to [Manage dashboard permissions](/docs/grafana/<GRAFANA_VERSION>/administration/user-management/manage-dashboard-permissions/) to control access to team folders.

# Manage dashboard permissions

Dashboard and folder permissions enable you to grant a viewer the ability to edit and save dashboard changes, or limit an editor's permission to modify a dashboard.

For more information about dashboard permissions, refer to [Dashboard permissions](../../roles-and-permissions/#dashboard-permissions).

## Grant folder permissions

When you grant user permissions for folders, that setting applies to all dashboards and subfolders contained in the folder. Consider using this approach to assigning dashboard and folder permissions when you have users, service accounts or teams who require access to groups of related dashboards or folders.

### Before you begin

- Ensure you have organization administrator privileges
- Identify the dashboard folder permissions you want to modify and the users, service accounts or teams to which you want to grant access. For more information about dashboard permissions, refer to [Dashboard permissions](../../roles-and-permissions/#dashboard-permissions).

**To grant dashboard folder permissions**:

1. Sign in to Grafana as an organization administrator.
1. In the left-side menu, click **Dashboards**.
1. Hover your mouse cursor over a folder and click **Go to folder**.
1. Click the **Permissions** tab, and then click **Add a permission**.
1. In the **Add Permission For** dropdown menu, select **User**, **Service Account**, **Team**, or **Role**.
1. Select the user, service account, team, or role.
1. Select the permission and click **Save**.

## Grant dashboard permissions

When you grant folder permissions, that setting applies to all dashboards and subfolders in the folder. For a more granular approach to assigning permissions, you can also assign user permissions to individual dashboards.

For example, if a user with the viewer organization role requires editor (or admin) access to a dashboard, you can assign those elevated permissions on an individual basis.

{{< admonition type="note" >}}
If you have assigned a user dashboard folder permissions, you cannot also assign the user permission to dashboards contained in the folder.
{{< /admonition >}}

Grant dashboard permissions when you want to restrict or enhance dashboard access for users who do not have permissions defined in the associated folder.

### Before you begin

- Ensure you have organization administrator privileges
- Identify the dashboard permissions you want to modify and the users, service accounts, or teams to which you want to grant access

**To grant dashboard permissions**:

1. Sign in to Grafana as an organization administrator.
1. In the left-side menu, click **Dashboards**.
1. Open a dashboard.
1. In the top right corner of the dashboard, click **Dashboard settings** (the cog icon).
1. Click **Permissions** in left-side menu, and then **Add a permission**.
1. In the **Add Permission For** dropdown menu, select **User**, **Service Account**, **Team**, or **Role**.
1. Select the user, service account, team, or role.
1. Select the permission and click **Save**.

# Edit dashboard permissions

Edit dashboard permissions when you are want to enhance or restrict a user's access to a dashboard. For more information about dashboard permissions, refer to [Dashboard permissions](../../roles-and-permissions/#dashboard-permissions).

### Before you begin

- Identify the dashboard and user permission you want to change
- Ensure you have organization administrator privileges

**To edit dashboard permissions**:

1. Sign in to Grafana as an organization administrator.
1. In the left-side menu, click **Dashboards**.
1. Open a dashboard.
1. In the top-right corner of the dashboard, click **Dashboard settings** (the cog icon).
1. Click **Permissions** in left-side menu.
1. In the dropdown, update the permissions, and click **Save**.

## Restrict access to dashboards

Grafana applies the highest permission a given user has to access a resource like a dashboard, so if you want to prevent a user from accessing a folder or dashboard you need to consider the user's organization role, folder permissions, and dashboard permissions.

- You cannot override organization administrator permissions. Organization administrators have access to all organization resources.
- User permissions set for a folder propagate to all dashboards and subfolders contained in a folder. Permissions also cascade down to all dashboards and folders under the subfolders, and so on.
- A lower permission level does not affect access if a more general rule exists with a higher permission.

Refer to the following examples to understand how organization and dashboard permissions impact a user's access to dashboards.

### Example 1

In this example, user1 has the editor organization role.

Dashboard permissions settings:

- Everyone with Editor role can edit
- user1 is set to `view`

Result: User1 has edit permissions because the user's organization role is Editor.

### Example 2

In this example, user1 has the viewer organization role and is a member of team1.

Dashboard permissions settings:

- Everyone with Viewer role can view
- user1 is set to `edit`
- team1 is set to `admin`

Result: User1 has administrator permissions for the dashboard because user1 is a member of team1.

### Example 3

In this example, user1 has the viewer organization role.

Dashboard permissions settings:

- user1 is set to `admin`, which is inherited from the permissions set in parent folder
- user1 is set to `edit`

Result: You receive an error message that cannot override a higher permission with a lower permission in the same dashboard. User1 has administrator permissions.

### Example 4

In this example, user1 has the viewer organization role.

Folder permissions settings:

- user1 is set to `edit`, which is inherited from the permissions set in grandparent folder

Result: User1 has editor permissions for the folder because user1 has permissions on a folder that contains this folder.

> Refer to [Role-based access Control](../../roles-and-permissions/access-control/) in Grafana Enterprise to understand how to use RBAC permissions to restrict access to dashboards, folders, administrative functions, and other resources.

# Manage users in an organization

Organization administrators can invite users to join their organization. Organization users have access to organization resources based on their role, which is **Admin**, **Editor**, or **Viewer**. Permissions associated with each role determine the tasks a user can perform in the system.

For more information about organization user permissions, refer to [Organization users and permissions](../../roles-and-permissions/#organization-users-and-permissions).

{{< admonition type="note" >}}
Users added at the organization level will have access to all stacks and services by default, without the ability to be filtered by stack unless Single Sign-On (SSO) or Role-Based Access Control (RBAC) is implemented.
{{< /admonition >}}

{{< section >}}

## View a list of organization users

You can see a list of users with accounts in your Grafana organization. If necessary, you can use the search field to filter the list.

### Before you begin

- Ensure you have organization administrator privileges

**To view a list of organization users**:

1. Sign in to Grafana as an organization administrator.
1. Navigate to **Administration > Users and access > Users**.

{{< admonition type="note" >}}
If you have [server administrator](../../roles-and-permissions/#grafana-server-administrators) permissions, you can also [view a global list of users](../server-user-management/#view-a-list-of-users) in the Server Admin section of Grafana.
{{< /admonition >}}

## Change a user's organization permissions

Update user permissions when you want to enhance or restrict a user's access to organization resources. For more information about organization permissions, refer to [Organization roles](../../roles-and-permissions/#organization-roles).

{{< admonition type="note" >}}
Organization roles sync from the authentication provider on user sign-in. To prevent synchronization of organization roles from the authentication provider regardless of their role in the authentication provider, then refer to the `skip_org_role_sync` setting in your Grafana configuration. Refer to [skip org role sync](../../../setup-grafana/configure-grafana/#authgrafana_com-skip_org_role_sync) for more information.
{{< /admonition >}}

### Before you begin

- Ensure you have organization administrator privileges

**To change the organization role of a user**:

1. Sign in to Grafana as an organization administrator.
1. Navigate to **Administration > Users and access > Users**.
1. Find the user account for which you want to change the role.

   If necessary, use the search field to filter the list.

1. Locate the user on the list and in the **Role** column, click the user role.
1. Select the role that you want to assign.
1. Click **Update**.

{{< admonition type="note" >}}
If you have [server administrator](../../roles-and-permissions/#grafana-server-administrators) permissions, you can also [change a user's organization permissions](../server-user-management/change-user-org-permissions/) in the Server Admin section.
{{< /admonition >}}

## Invite a user to join an organization

When you invite users to join an organization, you assign the **Admin**, **Editor**, or **Viewer** role, or select **No basic role**. Organization roles control user access to resources, such as dashboards and data sources, owned by the organization. Users receive an email that prompts them to accept the invitation.

- If you know that the user already has access Grafana and you know their user name, then you issue an invitation by entering their user name.
- If the user is new to Grafana, then use their email address to issue an invitation. The system automatically creates the user account on first sign in.

{{< admonition type="note" >}}
If you have [server administrator](../../roles-and-permissions/#grafana-server-administrators) permissions, you can also manually [add a user to an organization](../server-user-management/add-remove-user-to-org/).
{{< /admonition >}}

### Before you begin

- Ensure you have organization administrator privileges.
- If the user already has access to Grafana, obtain their user name.
- Determine the permissions you want to assign to the user. For more information about organization permissions, refer to [Organization roles](../../roles-and-permissions/#organization-roles).

**To invite or add an existing user account to your organization**:

1. Sign in to Grafana as an organization administrator.
1. To switch to the organization to which you want to invite a user, hover your mouse over your profile and click **Switch organization** and select an organization.

   > **Note**: It might be that you are currently in the proper organization and don't need to switch organizations.

1. Navigate to **Administration > Users and access > Users**.
1. Click **Organization users**.
1. Click **Invite**.
1. Enter the following information:

   | Field             | Description                                                                                                                                                                                                                                                              |
   | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
   | Email or username | Either the email or username that the user will use to sign in to Grafana.                                                                                                                                                                                               |
   | Name              | The user's name.                                                                                                                                                                                                                                                         |
   | Role              | Click the organization role to assign this user. For more information about organization roles, refer to [Organization roles](../../roles-and-permissions/#organization-roles)..                                                                                         |
   | Send invite email | Switch to on if your organization has configured. The system sends an email to the user inviting them to sign in to Grafana and join the organization. Switch to off if you are not using email. The user can sign in to Grafana with the email or username you entered. |

1. Click **Submit**.

If the invitee is not already a user, the system adds them.

## Manage a pending invitation

Periodically review invitations you have sent so that you can see a list of users that have not yet accepted the invitation or cancel a pending invitation.

{{< admonition type="note" >}}
The **Pending Invites** button is only visible if there are unanswered invitations.
{{< /admonition >}}

### Before you begin

- Ensure you have organization administrator privileges

**To manage a pending invitation**:

1. Sign in to Grafana as an organization administrator.
1. Navigate to **Administration > Users and access > Users**.
1. Click **Pending Invites**.

   The **Pending Invites** button appears only when there are unaccepted invitations.

To cancel an invitation, click the red **X** next to the invitation.

To copy an invitation link and send it directly to a user, click Copy Invite. You can then paste the invite link into a message.

## Remove a user from an organization

You can remove a user from an organization when they no longer require access to the dashboard or data sources owned by the organization. No longer requiring access to an organization might occur when the user has left your company or has internally moved to another organization.

This action does not remove the user account from the Grafana server.

### Before you begin

- Ensure you have organization administrator privileges

**To remove a user from an organization**:

1. Sign in to Grafana as an organization administrator.
1. Navigate to **Administration > Users and access > Users**.
1. Find the user account that you want to remove from the organization.

   Use the search field to filter the list, if necessary.

1. Click the red **X** to remove the user from the organization.

> **Note:** If you have [server administrator](../../roles-and-permissions/#grafana-server-administrators) permissions, you can also [remove a user from an organization](../server-user-management/add-remove-user-to-org/#remove-a-user-from-an-organization) on the Users page of the Server Admin section.

# Add or remove a user in an organization

Server administrators can add and remove users in organizations. To do this as an organization administrator, see [Manage users in an organization](../../manage-org-users/).

## Add a user to an organization

Add a user to an organization when you want the user to have access to organization resources such as dashboards, data sources, and playlists. A user must belong to at least one organization.

You are required to specify an Admin role for each organization. The first user you add to an organization becomes the Admin by default. After you assign the Admin role to a user, you can add other users to an organization as either Admins, Editors, or Viewers.

### Before you begin

- [Create an organization](../../../organization-management/#create-an-organization)
- [Add a user](./#add-a-user) to Grafana
- Ensure you have [Grafana server administrator privileges](../assign-remove-server-admin-privileges/)

**To add a user to an organization**:

1. Sign in to Grafana as a server administrator.
1. Click **Administration** in the left-side menu, **Users and access**, and then **Users**.
1. Click a user.
1. In the Organizations section, click **Add user to organization**.
1. Select an organization and a role.

   For more information about user permissions, refer to [Organization roles](../../../roles-and-permissions/#organization-roles).

1. Click **Add to organization**.

The next time the user signs in, they will be able to navigate to their new organization using the Switch Organizations option in the user profile menu.

{{< admonition type="note" >}}
If you have [organization administrator](../../../roles-and-permissions/#organization-roles) permissions and _not_ [server administrator](../../../roles-and-permissions/#grafana-server-administrators) permissions, you can still [invite a user to join an organization](../../manage-org-users/#invite-a-user-to-join-an-organization).
{{< /admonition >}}

## Remove a user from an organization

Remove a user from an organization when they no longer require access to the dashboards, data sources, or alerts in that organization.

### Before you begin

- Ensure you have Grafana server administrator privileges

**To remove a user from an organization**:

1. Sign in to Grafana as a server administrator.
1. Click **Administration** in the left-side menu, **Users and access**, and then **Users**.
1. Click a user.
1. In the Organization section, click **Remove from organization** next to the organization from which you want to remove the user.
1. Click **Confirm removal**.

> **Note:** If you have [organization administrator](../../../roles-and-permissions/#organization-roles) permissions and _not_ [server administrator](../../../roles-and-permissions/#grafana-server-administrators) permissions, you can still [remove a user from an organization](../../manage-org-users/#remove-a-user-from-an-organization) in the Users section of organization configuration.

# Assign or remove Grafana server administrator privileges

Grafana server administrators are responsible for creating users, organizations, and managing permissions. For more information about the server administration role, refer to [Grafana server administrators](../../../roles-and-permissions/#grafana-server-administrators).

{{< admonition type="note" >}}
Server administrators are "super-admins" with full permissions to create, read, update, and delete all resources and users in all organizations, as well as update global settings such as licenses. Only grant this permission to trusted users.
{{< /admonition >}}

## Before you begin

- [Add a user](../#add-a-user)
- Ensure you have Grafana server administrator privileges

**To assign or remove Grafana administrator privileges**:

1. Sign in to Grafana as a server administrator.
1. Click **Administration** in the left-side menu, **Users and access**, and then **Users**.
1. Click a user.
1. In the Permissions section, next to Grafana Admin, click **Change**.
1. Click **Yes** or **No**, depending on whether or not you want this user to have the Grafana server administrator role.
1. Click **Change**.

The system updates the user's permission the next time they load a page in Grafana.

# Change a user's organization permissions

Update organization permissions when you want to enhance or restrict a user's access to organization resources. For more information about organization permissions, refer to [Organization roles](../../../roles-and-permissions/#organization-roles).

## Before you begin

- [Add a user to an organization](../add-remove-user-to-org/)
- Ensure you have Grafana server administrator privileges

**To change a user's organization permissions**:

1. Sign in to Grafana as a server administrator.
1. Click **Administration** in the left-side menu, **Users and access**, and then **Users**.
1. Click a user.
1. In the Organizations section, click **Change role** for the role you want to change
1. Select another role.
1. Click **Save**.

{{< admonition type="note" >}}
In order for the change to take effect and be reflected within the instance, the account where permissions were altered will need to sign out fully and back in. Role assignment is evaluated during sign in, so if a user has not signed back in after their role was adjusted the instance will continue to reflect their previous role.
{{< /admonition >}}

# Manage user preferences

Grafana allows you to manage certain aspects of your user account, including your user name, email, and password.

You can also view important information about your account, such as the organizations and roles to which you are assigned and the Grafana sessions associated with your account.

## Change your Grafana password

You can change your Grafana password at any time.

{{< admonition type="note" >}}
If your Grafana instance uses an external authentication provider, then you might not be able to change your password in Grafana. Contact your Grafana administrator for more information.
{{< /admonition >}}

**To change your password**:

1. Sign in to Grafana.
1. Click the user icon in the top right corner of the page and select **Change Password**.
1. Enter your old password and a new password.
1. Confirm your new password.
1. Click **Change Password**.

## Edit your profile

Your profile includes your name, user name, and email address, which you can update.

**To edit your profile**:

1. Sign in to Grafana.
1. Click the user icon in the top right corner of the page and select **Profile**.
1. In the **Profile** section, update your details and click **Save**.

## Edit your preferences

You can choose the way you would like data to appear in Grafana, including the user interface theme, home dashboard, timezone, and first day of the week. You can set these preferences for your own account, for a team, for an organization, or Grafana-wide using configuration settings. Your user preferences take precedence over team, organization, and Grafana default preferences. For more information, see [Grafana preferences](../../organization-preferences/).

- **Interface theme** determines whether Grafana appears in light mode or dark mode. You can also choose from several experimental modes. By default, the interface theme is set to dark mode. You can also [quickly view and change themes](#view-and-change-themes) using the **Change theme** option.
- **Home dashboard** replaces the Grafana home page with a dashboard of your choice. If you don't choose one, you see the dashboard your team or organization sets, or the instance default. Select **Grafana home** to skip team and organization dashboards and see the instance default. For the full order, refer to [How Grafana picks the home page](../../organization-preferences/#how-grafana-picks-the-home-page).
- **Timezone** is used by dashboards when you set time ranges, so that you view data in your timezone instead of UTC.
- **Week start** is the first day of the week you want to use in dashboard time ranges, for example, `This week`.
- **Language** determines the language used for parts of the Grafana interface.

**To edit your preferences**:

1. Sign in to Grafana.
1. Click the user icon in the top right corner of the page and select **Profile**.
1. Update any of the values in the Preferences section.
1. Click **Save** at the bottom of the section.

### View and change themes

{{< admonition type="note" >}}
The **Change theme** drawer is an experimental feature. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Enable the `grafanaconThemes` feature toggle in Grafana to use this feature. This feature is automatically enabled in Grafana Cloud.
{{< /admonition >}}

The **Change theme** drawer allows you to see the color scheme of a theme without having to select it first.
It also lets you quickly change a color scheme from anywhere in Grafana without first going to your profile.

To access the **Change theme** drawer, click the user icon in the top-right corner of the page and select **Change theme**.

{{< figure src="/media/docs/grafana/screenshot-themes-drawer-v11.6.png" max-width="650px" alt="The Change theme drawer opened" >}}

## Switch organizations

When you sign in to Grafana, the system signs you in to a default organization. If you are assigned to multiple organizations, then you might need to switch organizations. For example, if you need to view a dashboard not associated with your current organization, then you should switch organizations to view associated dashboards.

**To switch organizations**:

1. Sign in to Grafana.
1. Click the user icon in the top right corner of the page and select **Switch organization**.
1. Next to the organization that you want to sign in to, click **Switch to**.

## View your assigned organizations

Every user is a member of at least one organization. You can have different roles in each organization of which you are a member.

**To view your assigned organizations**:

1. Sign in to Grafana.
1. Click the user icon in the top right corner of the page and select **Profile**.
1. Scroll down to the Organizations section and review the following information:
   - **Name**: The name of the organizations of which you are a member.
   - **Role**: The role to which you are assigned in the organization. For more information about roles and permissions, refer to [Organization users and permissions](../../roles-and-permissions/#organization-users-and-permissions).
   - **Current**: Grafana indicates the organization that you are currently signed into as _Current_. If you are a member of multiple organizations, you can click **Select** to switch to that organization.

## View your Grafana sessions

Grafana logs your sessions in each Grafana instance. You can review this section if you suspect someone has misused your Grafana credentials.

**To view your Grafana sessions**:

1. Sign in to Grafana.
1. Click the user icon in the top right corner of the page and select **Profile**.
1. Scroll down to the **Sessions** section.

## Sign out a user session

You can sign out other sessions using your account in order to prevent other people from accessing Grafana using your credentials.

**To sign out one of your Grafana sessions**:

1. Sign in to Grafana.
1. Click the user icon in the top right corner of the page and select **Profile**.
1. Scroll down to the **Sessions** section.
1. Click the red "sign out" icon next to the session you would like to sign out.

# Ai Tools

# Manage resources with `grafanactl`

{{< admonition type="caution" >}}

`grafanactl` is being deprecated, and we're bringing all our learning and experience into the new, improved CLI tool [`gcx`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/ai-tools/grafana-cli/gcx). The `grafanactl` repository in GitHub is scheduled to be archived; check the [repository](https://github.com/grafana/grafanactl) for its current status.

To migrate from `grafanactl` to `gcx`, search-and-replace `grafanactl` with `gcx`. For `grafanactl resources serve`, use `gcx dev serve` instead.

{{< /admonition >}}

## Migrate resources between environments

Using the `config` and `resources` options, you can migrate Grafana resources from one environment to another, for example, from a development to production environment.
The `config` option lets you define the configuration context.
Using `resources` with `pull`, `push`, and `serve` lets you pull a defined resource from one instance, and push that resource to another instance. `Serve` allows you to preview changes locally before pushing.

Use these steps to migrate resources between environments:

{{< admonition type="note" >}}
Currently, the `serve` command only works with dashboards.
{{< /admonition >}}

Use these steps to migrate resources between environments:

{{< admonition type="note" >}}
Resources are pulled and pushed from the `./resources` directory by default.
This can be configured with the `-p, --path` flags to specify custom paths on disk.
{{< /admonition >}}

1. Make changes to dashboards and other resources using the Grafana UI in your **development instance**.
1. Pull those resources from the development environment to your local machine:

   ```bash
   grafanactl config use-context YOUR_CONTEXT
   grafanactl resources pull --path ./resources/ -o yaml
   ```

1. (Optional) Preview the resources locally before pushing:

   ```bash
   grafanactl config use-context YOUR_CONTEXT
   grafanactl resources serve ./resources/
   ```

1. Switch to the **production instance** and push the resources:

   ```bash
   grafanactl config use-context YOUR_CONTEXT
   grafanactl resources push -p ./resources/
   ```

## Back up Grafana resources

This workflow helps you back up all Grafana resources from one instance and later restore them. This is useful to replicate a configuration or perform disaster recovery.

1. Use `grafanactl` to pull all resources from your target environment:

   ```bash
   grafanactl config use-context YOUR_CONTEXT
   grafanactl resources pull --path ./resources/ -o yaml
   ```

1. Save the exported resources to version control or cloud storage.

## Restore Grafana resources

1. (Optional) Preview the backup locally:

   ```bash
   grafanactl config use-context YOUR_CONTEXT
   grafanactl resources serve ./resources/
   ```

1. To restore the resources later or restore them on another instance, push the saved resources:

   ```bash
   grafanactl config use-context YOUR_CONTEXT
   grafanactl resources push -p ./resources/
   ```

## Manage dashboards as code

With this workflow, you can define and manage dashboards as code, saving them to a version control system like Git. This is useful for teams that want to maintain a history of changes, collaborate on dashboard design, and ensure consistency across environments.

1. Use a dashboard generation script (for example, with the [Foundation SDK](https://github.com/grafana/grafana-foundation-sdk)). You can find an example implementation in the Grafana as code [hands-on lab repository](https://github.com/grafana/dashboards-as-code-workshop/tree/main/part-one-golang-starter).

1. Serve and preview the output of the dashboard generator locally:

   ```bash
   grafanactl config use-context YOUR_CONTEXT
   grafanactl resources serve --script 'go run scripts/generate-dashboard.go' --watch './scripts'
   ```

1. When the output looks correct, generate dashboard manifest files:

   ```bash
   go run scripts/generate-dashboard.go --generate-resource-manifests --output './resources'
   ```

1. Push the generated resources to your Grafana instance:

   ```bash
   grafanactl config use-context YOUR_CONTEXT
   grafanactl resources push -p ./resources/
   ```

## Explore and modify resources from the terminal

This section describes how to use the Grafana CLI to interact with Grafana resources directly from your terminal. These commands allow you to browse, inspect, update, and delete resources without using the Grafana UI. This approach is useful for advanced users who want to manage resources more efficiently or integrate Grafana operations into automated workflows.

### Find and delete dashboards using invalid data sources

Use this workflow to identify dashboards that reference incorrect or outdated data sources, and remove them if necessary.

1. Set the context to the appropriate environment:

   ```bash
   grafanactl config use-context YOUR_CONTEXT
   ```

1. Find dashboards using specific data sources:

   ```bash
   grafanactl resources get dashboards -ojson | jq '.items | map({ uid: .metadata.name, datasources: .spec.panels | map(.datasource.uid)  })'
   [
      {
         "uid": "important-production-dashboard",
         "datasources": [
            "mimir-prod"
         ]
      },
      {
         "uid": "test-dashboard-from-dev",
         "datasources": [
            "mimir-prod",
            "mimir-dev"
         ]
      },
      {
         "uid": "test-dashboard-from-stg",
         "datasources": [
            "mimir-prod",
            "mimir-stg",
            "mimir-dev"
         ]
      }
   ]
   ```

   This command lists dashboard UIDs along with the data source UIDs used in their panels. You can then identify the dashboards that are using invalid or unexpected data sources.

1. Delete the identified dashboards directly:

   ```bash
   grafanactl resources delete dashboards/test-dashboard-from-stg,test-dashboard-from-dev
   ✔ 2 resources deleted, 0 errors
   ```

### Find and deprecate dashboards using the old API version

Use this workflow to locate dashboards using a deprecated API version and mark them accordingly.

1. Set the context to the appropriate environment:

   ```bash
   grafanactl config use-context YOUR_CONTEXT
   ```

1. List all available resources types and versions:

   ```bash
   grafanactl resources list
   ```

   This command returns a list of resources, including their versions, types, and quantities:

   ```bash
   GROUP                               VERSION   KIND
   folder.grafana.app                  v1        folder
   dashboard.grafana.app               v1        dashboard
   dashboard.grafana.app               v1        librarypanel
   dashboard.grafana.app               v2        dashboard
   dashboard.grafana.app               v2        librarypanel
   playlist.grafana.app                v1        playlist
   ```

1. Find dashboards that are still using a deprecated API version:

   ```bash
   grafanactl resources get dashboards.v1.dashboard.grafana.app
   ```

   This command returns a table displaying the resource type, resource name, and associated namespace:

   ```bash
   KIND         NAME                                   NAMESPACE
   dashboards   really-old-dashboard                   default
   ```

1. Edit each of these dashboards to add a `deprecated` tag:

   ```bash
   grafanactl resources edit dashboards.v1.dashboard.grafana.app/really-old-dashboard -p '{"spec":{"tags":["deprecated"]}}'
   ```

{{< admonition type="tip" >}}
You can get help by using the `grafanactl --help` command.
{{< /admonition >}}

# Alerting

# Improve alert rule quality with Alert Advisor

Alerts are easier to act on when they carry the right context.
A runbook URL helps you investigate an issue, and a team label helps you route your notifications.

Alert Advisor checks your Grafana-managed alert rules against a quality policy of required labels and annotations, so you can find and fix rules that are missing that context.
With Alert Advisor, you can:

- Define which labels and annotations your Grafana-managed alert rules must have.
- Find rules that are missing required labels or annotations and update them.
- Optionally block supported provisioning writes that don't meet selected requirements.

## Before you begin

Before you begin, check that you have the following permissions in your Grafana organization:

- **Rule access:** Have permission to view the Grafana-managed alert rules you want to assess.
- **Policy permissions:** To configure the quality policy, have the Admin role in your organization or a role with `alert.rules.quality:write` and `alert.rules:read`.
- **Rule updates:** To fix findings, have permission to update the affected rules.
  For provisioned rules, you also need access to their source configuration and permission to reapply it.

For more information about alerting permissions and folder access, refer to [Configure RBAC](ref:configure-rbac).

## Understand policy scope

A single quality policy applies to Grafana-managed alert rules across folders in your organization.
Data source-managed alert rules and recording rules are excluded.

Alert Advisor reports a finding when a required label or annotation is missing, empty, or contains only whitespace.
The checks do not validate the content itself, they only check for the presence of a particular field.
For example, requiring a runbook URL doesn't check whether the link works.

### Detection and enforcement

You choose a mode for each requirement:

| Mode        | Behavior                                                                                                   |
| ----------- | ---------------------------------------------------------------------------------------------------------- |
| Detect-only | Missing values appear as findings but don't block rule creation or updates.                                |
| Enforced    | Missing values appear as findings and block rule creation or updates through supported provisioning paths. |

Both modes contribute to the quality score.
Enabling enforcement doesn't stop existing rules from evaluating or sending notifications.

### Supported enforcement paths

Enforcement applies to provisioned rules created or updated through Terraform and the Alerting provisioning HTTP API, with the following exceptions:

- Provisioning API requests that set the `X-Disable-Provenance` header, including Terraform `grafana_rule_group` resources with `disable_provenance = true`, aren't enforced.
- `kubectl` writes are enforced only when the manifest sets `grafana.app/managedBy` to `kubectl` or `terraform` together with a non-empty `grafana.app/managerId`, or sets `grafana.com/provenance` to `api`.

Writes from the Grafana rule editor aren't enforced, but the resulting rules can still have quality findings.

## Configure a quality policy

Start with detect-only requirements to identify missing information before enabling enforcement.
The following example requires a runbook URL and a `team` label.

The **Quality policy settings** button is visible only if you have the policy permissions listed in [Before you begin](#before-you-begin).

To configure a quality policy, follow these steps:

1. Go to **Alerting** > **Alert rules** and select **Alert quality**.
2. Click **Quality policy settings** (the cog icon) to open the **Quality policy** drawer.
3. Under **Required annotations**, turn on **Runbook URL**.

   You can also require **Summary** and **Description**.

4. Under **Required labels**, enter `team` in **Label keys** and select the matching option.

   You can select keys already used by your rules or add a new key.

5. Keep **Enforce** off for the requirements you add.

   New requirements start in detect-only mode.

6. Click **Save**.

   Grafana displays **Policy saved** to confirm the change.

The saved policy identifies rules with missing or empty `runbook_url` annotations or `team` labels.

A policy with no required annotations or labels checks nothing.

## Review alert quality

In Alert Advisor, the **Alert quality** tab lists rules that don't meet your policy and shows an overall quality score.

To review the findings, follow these steps:

1. Go to **Alerting** > **Alert rules** and select **Alert quality**.
2. Review the **Alert quality score** and the number of rules that need attention.
3. Inspect the missing annotations and labels in the **Findings** list.

Each missing field appears as a badge. To check whether a requirement is enforced or detect-only, hover over its badge.

### Understand the score

The score represents the proportion of assessed rules that meet every configured requirement, on a scale of 0 to 10.
For example, if 8 of 10 rules meet the policy, the score is 8.0.
Grafana rounds the score to one decimal place.

The score measures compliance with your policy, not alert accuracy, severity, or noise. Both detect-only and enforced requirements count toward the score.

Your permissions determine which rules you can view and assess.
Search filters change the findings' list, not the score or the total rule count.
Grafana also displays 10.0 when there are no rules to assess, so check the rule count alongside the score.

## Resolve quality findings

Add meaningful values for the missing fields in each rule.
For the example policy, set `runbook_url` to the rule's runbook URL and `team` to the owning team's name.

Changing label values can affect notification routing.
Choose values that match your team's [notification policies](ref:notification-policies).

### Update provisioned rules

For provisioned rules, make changes in the source configuration so they remain part of your provisioning workflow.

To resolve a finding on a provisioned rule, follow these steps:

1. Locate the rule in its source configuration, using the rule name, folder, and group shown in **Alert quality**.
2. Add values for the missing annotations and labels.
3. Reapply the configuration through your usual provisioning workflow.
4. Reload the **Alert quality** page and check the rule's findings.

A rule disappears from the findings list when it meets every configured requirement.
If other required fields are still missing, the rule remains in the list with those findings.

### Update rules in Grafana

For rules that you can edit in Grafana, use the **Edit** action on the quality page.

To resolve a finding in the rule editor, follow these steps:

1. Click **Edit** for the affected rule.
2. Fill in the missing annotations and labels in the rule editor.
3. Click **Save**.
4. Return to **Alert quality** and reload the page to check the findings.

## Optional: Enforce quality requirements

After reviewing and resolving findings, enable enforcement for the requirements you want to apply to supported provisioning writes.
You can enforce selected requirements while keeping others detect-only.

To enable enforcement, follow these steps:

1. On the **Alert quality** page, click **Quality policy settings** (the cog icon) to open the **Quality policy** drawer.
2. Turn on **Enforce** beside each requirement you want to enforce.

   For example, enforce `team` while keeping **Runbook URL** detect-only.
   To enforce all currently configured requirements, turn on **Enforce all requirements** instead.

3. Click **Save**.

   Grafana displays **Policy saved** to confirm the change.

### Disable enforcement or remove a requirement

To return a requirement to detect-only, turn off its **Enforce** control (or **Enforce all requirements**, if every requirement is enforced) and click **Save**.
The requirement remains in the policy and continues to affect the quality score.

To stop checking a field altogether, remove its requirement instead: turn off its switch under **Required annotations**, or remove its key from **Label keys**. Then click **Save**.

## Troubleshoot quality findings and rejected writes

Use the following guidance when you can't access the feature, resolve a finding, or apply a rule change.

| Symptom                                                                    | Action                                                                                                                                                                                    |
| -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Alert quality** or the **Quality policy settings** button isn't visible. | Ask your stack administrator to check your [permissions](#before-you-begin) and the feature's availability in your stack.                                                                 |
| A rule has no **Edit** action.                                             | If it has a **Provisioned** badge, update its source configuration. Otherwise, check your permission to update rules in that folder.                                                      |
| Grafana reports that no quality policy is configured.                      | Add at least one requirement in the **Quality policy** drawer and click **Save**. An empty policy doesn't assess any fields.                                                              |
| No findings match your search, but the score is below 10.0.                | Clear **Search** to view all findings. Search filters don't change the score.                                                                                                             |
| A finding remains after enforcement is disabled.                           | Detect-only requirements still produce findings. Fill in the missing value or remove the requirement from the policy.                                                                     |
| A provisioning write is rejected by the quality policy.                    | Use the rule and field details in the error to update the source configuration, then reapply it.                                                                                          |
| A write succeeds despite a quality finding.                                | Check that the requirement is enforced and the policy is saved. Confirm that the write uses a [supported enforcement path](#supported-enforcement-paths) and isn't one of its exceptions. |

## Next steps

Use these guides to improve rule context, manage rules as code, and configure access:

- **Rule context:** [Labels and annotations](ref:labels-and-annotations).
- **Provisioning:** [Use Terraform to provision alerting resources](ref:terraform-provisioning).
- **API management:** [Use the HTTP API to manage alerting resources](ref:http-api-provisioning).
- **Permissions:** [Configure RBAC](ref:configure-rbac).

# Import data source-managed rules to Grafana-managed rules

You can convert existing alert rules from data sources such as Mimir, Loki, and Prometheus into Grafana-managed alert rules. This enables you to operate and manage these rules using Grafana Alerting.

This guide explains two methods for importing data source-managed rules:

- Using the [Grafana Alerting user interface](#import-rules-with-grafana-alerting) to import rules from connected data sources or Prometheus rule YAML files.

- Using [command-line tools](#import-rules-with-command-line-tools) like `mimirtool` and `cortextool` to import rules from YAML files.

Importing rules is a safe operation: the original data source–managed rules remain intact in their original location. During import, the rules are converted into Grafana-managed rules while preserving their configuration and behavior.

Choose the method that best fits your workflow. As a best practice, test and validate your import process before migrating all alert rules.

## Import your notification configuration first

Rules only tell you about a problem if the notification reaches someone. Grafana-managed rules route through the Grafana Alertmanager, not through the Alertmanager your data source-managed rules used, so imported rules notify through whatever contact points and notification policies already exist in Grafana.

Consider [importing your Alertmanager configuration](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/import-alertmanager-configuration/) before you import rules. Grafana imports the configuration as it is and runs it the way your Prometheus or Mimir Alertmanager does. You can then route the rules you import through the imported policy tree.

## How it works

When you use any of the import methods, data source–managed rules are copied to another folder as Grafana-managed rules.

The original data source–managed rules remain intact in their original location.

The copied rules are converted to Grafana-managed rules, preserving their behavior by using equivalent Grafana-managed features. The following settings are applied during the conversion:

- **Unique UIDs**

  The newly created rules are assigned unique UIDs.
  If you don’t want a UID to be auto-generated, you can specify one using the `__grafana_alert_rule_uid__` label.

- **Query offset**

  A query offset is applied to each rule. For example, an offset of `1m` adjusts the query's time range to `To: now-1m`.

  The rule query offset is taken from the `query_offset` value in the rule group configuration. If empty, it defaults to the [`rule_query_offset` configuration setting](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#rule_query_offset), which is `1m` by default.

- **Rule query conversion**

  For alert rules, adds `prometheus_math` and `threshold` expressions to preserve Prometheus no data behavior, ensuring the alert stays in **Normal** state when `query` returns no data.

- **Missing series evaluations to resolve**

  The [Missing series evaluations to resolve](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/#configure-missing-series-evaluations-to-resolve) setting is set to `1` to replicate Prometheus’s alert eviction behavior.

- **Rule group labels**

  Labels defined at the rule group level are added as labels to each imported rule within the group.

- **Sequential evaluation**

  Imported rules are evaluated sequentially within each rule group, mirroring Prometheus behavior. This differs from native Grafana-managed alert rules, where the evaluation order is not enforced. For more details, refer to [evaluation strategies](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/).

- **Feature compatibility**

  The rule group `limit` option and the `query` function within alert rule templates are not currently supported in Grafana-managed rules. If the `limit` option is present, the import fails. However, rules with `query` in templates are imported.

{{< admonition type="note" >}}
Rules with the label `__grafana_origin` are not included in rule imports. These rules are typically created by apps such as **Kubernetes Monitoring**, **Synthetic Monitoring**, and other **Grafana plugins**.
{{< /admonition >}}

## Import rules with Grafana Alerting

You can use the Grafana Alerting user interface to import rules from the following sources:

- Connected Mimir and Loki data sources with the ruler API enabled
- Prometheus YAML rule files

Imported rules using this method are editable in the user interface. Like regular Grafana-managed rules, you can later export them for provisioning.

#### Before you begin

To use Grafana Alerting to migrate rules, you need the following [RBAC permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/):

- **Alerting**: `Rules Writer`, `Set provisioning status`.
- **Datasources**: `Reader`.
- **Folders**: `Creator`.

  The Folders permission is optional and only necessary if you want to create new folders for your target namespace. If your account doesn't have permissions to view a namespace, the tool creates a new one.

To convert data source-managed alert rules to Grafana managed alerts:

1. Go to **Alerting > Alert rules**.

2. Navigate to the Data source-managed alert rules section and click **Import to Grafana-managed rules**.

3. Choose the **Import source** from which you want to import rules:
   - Select **Existing data source-managed rules** to import rules from connected Mimir or Loki data sources with the ruler API enabled.
   - Select **Prometheus YAML file** to import rules by uploading a Prometheus YAML rule file.

4. In the **Data source** dropdown, select the data source that the imported alert rules will query.

5. (Optional) In Additional settings, select a target folder or designate a new folder to import the rules into.

   If you import the rules into an existing folder, don't choose a folder with existing alert rules, as they could get overwritten.

6. (Optional) Select a Namespace and/or Group to determine which rules are imported.

7. (Optional) Turn on **Pause imported alerting rules**.

   Pausing stops alert rule evaluation and doesn’t create any alert instances for the newly created Grafana-managed alert rules.

8. (Optional) Turn on **Pause imported recording rules**.

   Pausing stops alert rule evaluation behavior for the newly created Grafana-managed alert rules.

9. (Optional) In the **Target data source** of the **Recording rules** section, you can select the data source to which the imported recording rules will write metrics. By default, it is the data source selected in the **Data source** dropdown.

10. Click **Import**.

    A preview shows the rules that will be imported. If your target folder contains folders with the same name of the imported folders, a warning displays to inform you. You can explore the warning to see a list of folders that might be overwritten.

    Click **Yes, import** to import the rules.

## Import rules with command-line tools

You can also use command-line tools to import data source-managed rules as Grafana-managed rules: use `mimirtool` for Mimir and Prometheus rules, and `cortextool` for Loki rules.

Both tools provide `rules` commands to import rules groups. For example:

- `rules load` can import rule groups from files.
- `rules sync` can read rule files and applies only the differences compared to existing Grafana-managed rules. This is useful for automation workflows and pipelines.

By default, rules imported using the API or command-line tools are **Provisioned** and not editable in the user interface. To make them editable, enable the [X-Disable-Provenance](#x-disable-provenance) header.

#### Before you begin

You need to have installed [`mimirtool`](/docs/mimir/latest/manage/tools/mimirtool/) or [`cortextool`](https://github.com/grafana/cortex-tools) (version `0.11.3` or later).

You need a service account with the following [RBAC permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/):

- **Alerting**: `Rules Reader`, `Rules Writer`, `Set provisioning status`.
- **Datasources**: `Reader`.
- **Folders**: `Creator`, `Reader`, `Writer`.

You need to a service account token with your service account. For more details, refer to [service accounts and service account tokens](/docs/grafana/latest/administration/service-accounts/).

### mimirtool

To convert and import them into a Grafana instance, you can use the `mimirtool rules load` command:

```bash
MIMIR_ADDRESS=<GRAFANA_BASE_URL>/api/convert/ \
MIMIR_AUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN> \
MIMIR_TENANT_ID=1 \
mimirtool rules load rule_file.yaml \
  --extra-headers "X-Grafana-Alerting-Datasource-UID=<DATASOURCE_UID_QUERY_TARGET>"
```

This command imports Prometheus alert rules defined in `rule_file.yaml` as Grafana-managed alert rules. It's important to know that:

1. When using the `<GRAFANA_BASE_URL>/api/convert/` endpoint, `mimirtool` interacts with Grafana—not with a Mimir instance. In this case, `MIMIR_TENANT_ID` must always be set to `1`.
1. The [`X-Grafana-Alerting-Datasource-UID` header](#x-grafana-alerting-datasource-uid) configures the data source that the imported alert rules will query. Use multiple `--extra-headers` flags to include other [optional headers](#optional-headers).

Similarly, the `rules sync` command can import and update Grafana-managed alert rules.

```bash
MIMIR_ADDRESS=<GRAFANA_BASE_URL>/api/convert/ \
MIMIR_AUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN> \
MIMIR_TENANT_ID=1 \
mimirtool rules sync rule_file.yaml \
  --extra-headers "X-Grafana-Alerting-Datasource-UID=<DATASOURCE_UID_QUERY_TARGET>" \
  --concurrency 1
```

The `--concurrency` flag must be set to `1`, as the default value of `8` may cause API errors.

This `sync` command reads rules from the file, compares them with the existing Grafana-managed rules in the instance, and applies only the differences—creating, updating, or deleting rules as needed.

```output
## Sync Summary: 0 Groups Created, 1 Groups Updated, 0 Groups Deleted
```

For more information other Mimirtool commands and options, see the [Mimirtool documentation](/docs/mimir/latest/manage/tools/mimirtool/#rules) and the [Mimir HTTP Rule API documentation](/docs/mimir/latest/references/http-api/#ruler-rules:~:text=config/v1/rules-,Get%20rule%20groups%20by%20namespace,DELETE%20%3Cprometheus%2Dhttp%2Dprefix%3E/config/v1/rules/%7Bnamespace%7D,-Delete%20tenant%20configuration).

### cortextool

For Loki alert rules, use [`cortextool`](https://github.com/grafana/cortex-tools) (version `0.11.3` or later) with the `--backend=loki` flag. For example:

```bash
CORTEX_ADDRESS=<GRAFANA_BASE_URL>/api/convert/ \
CORTEX_AUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN> \
CORTEX_TENANT_ID=1 \
cortextool rules load loki_rules.yaml \
  --extra-headers "X-Grafana-Alerting-Datasource-UID=<LOKI_DATASOURCE_UID_QUERY_TARGET>" \
  --backend=loki
```

### Optional Headers

Additional configuration headers for more granular import control include the following:

#### `X-Disable-Provenance`

When this header is set to `true`:

- The imported rules are not marked as provisioned.
- They can then be edited in the Grafana UI.
- They are excluded from the `GET` and `DELETE` operations on the [`/api/convert` endpoints](#compatible-endpoints).

Do not enable this header when using the `rules sync` command, as it relies on the `GET` and `DELETE` operations to detect and update existing rules.

#### `X-Grafana-Alerting-Alert-Rules-Paused`

Set to `true` to import alert rules in paused state.

#### `X-Grafana-Alerting-Recording-Rules-Paused`

Set to `true` to import recording rules in paused state.

#### `X-Grafana-Alerting-Datasource-UID`

The UID of the data source to use for alert rule queries.

If not specified in the header, Grafana uses the configured default from `unified_alerting.prometheus_conversion.default_datasource_uid`. If neither the header nor the configuration option is provided, the request fails.

#### `X-Grafana-Alerting-Target-Datasource-UID`

The UID of the target data source for recording rules. If not specified, the value from `X-Grafana-Alerting-Datasource-UID` is used.

#### `X-Grafana-Alerting-Folder-UID`

Enter the UID of the target destination folder for imported rules.

#### `X-Grafana-Alerting-Notification-Settings`

JSON-encoded [`AlertRuleNotificationSettings` object](#alertrulenotificationsettings-object) that allows setting the contact point for the alert rules.

{{< collapse title="AlertRuleNotificationSettings object" >}}

##### AlertRuleNotificationSettings object

When you set `X-Grafana-Alerting-Notification-Settings`, the header value must be a JSON-encoded object with the following keys:

| Field                   | Type       | Required | Example                                    | Description                                                                                             |
| ----------------------- | ---------- | -------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
| `receiver`              | `string`   | Yes      | `"grafana-default"`                        | Name of the contact point (receiver) to which alerts are routed. Must exist in Grafana before import.   |
| `group_by`              | `[]string` | No       | `["alertname","grafana_folder","cluster"]` | Label set used by Alertmanager to aggregate alerts into a single notification.                          |
| `group_wait`            | `duration` | No       | `"30s"`                                    | How long Alertmanager waits before sending the first notification for a new group.                      |
| `group_interval`        | `duration` | No       | `"5m"`                                     | Time to wait before adding new alerts to an existing group's next notification.                         |
| `repeat_interval`       | `duration` | No       | `"4h"`                                     | Minimum time before a previously-sent notification is repeated. Must not be less than `group_interval`. |
| `mute_time_intervals`   | `[]string` | No       | `["maintenance"]`                          | One or more mute time interval names that silence alerts during those windows.                          |
| `active_time_intervals` | `[]string` | No       | `["maintenance"]`                          | List of active time interval names. Alerts are suppressed unless the current time matches one of them.  |

{{< /collapse >}}

### Compatible endpoints

The API endpoints listed in this section are supported in Grafana and are used by `mimirtool` and `cortextool`, as shown earlier. These endpoints are compatible with [Mimir HTTP API](/docs/mimir/latest/references/http-api/).

In these endpoints, a "namespace" corresponds to a folder title in Grafana.

The `POST` endpoints can be used to import data source–managed alert rules. They accept requests in both YAML and JSON. If no media type is specified, YAML is assumed.

| Endpoint | Method                                                | Summary                                                                                                                                                                                         | Mimir equivalent                                                         |
| -------- | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| POST     | `/convert/prometheus/config/v1/rules`                 | [Create or update multiple rule groups](#create-or-update-multiple-rule-groups) across multiple namespaces. Requires [`X-Grafana-Alerting-Datasource-UID`](#x-grafana-alerting-datasource-uid). | None                                                                     |
| POST     | `/convert/prometheus/config/v1/rules/:namespaceTitle` | Create or update a single rule group in a namespace. Requires [`X-Grafana-Alerting-Datasource-UID`](#x-grafana-alerting-datasource-uid).                                                        | [Set rule group](/docs/mimir/latest/references/http-api/#set-rule-group) |

The `GET` and `DELETE` endpoints work only with provisioned and imported alert rules. All `GET` endpoints support both JSON and YAML response formats based on the `Accept` header: use `application/json` for JSON responses, or `application/yaml` for YAML responses. YAML is the default format when no `Accept` header is specified.

| Endpoint | Method                                                       | Summary                                             | Mimir equivalent                                                                                     |
| -------- | ------------------------------------------------------------ | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| GET      | `/convert/prometheus/config/v1/rules`                        | Get all imported rule groups across all namespaces. | [List rule groups](/docs/mimir/latest/references/http-api/#list-rule-groups)                         |
| GET      | `/convert/prometheus/config/v1/rules/:namespaceTitle`        | Get imported rule groups in a specific namespace.   | [Get rule groups by namespace](/docs/mimir/latest/references/http-api/#get-rule-groups-by-namespace) |
| GET      | `/convert/prometheus/config/v1/rules/:namespaceTitle/:group` | Get imported rule group in a specific namespace.    | [Get rule group](/docs/mimir/latest/references/http-api/#get-rule-group)                             |
| DELETE   | `/convert/prometheus/config/v1/rules/:namespaceTitle`        | Delete all imported alert rules in a namespace.     | [Delete namespace](/docs/mimir/latest/references/http-api/#delete-namespace)                         |
| DELETE   | `/convert/prometheus/config/v1/rules/:namespaceTitle/:group` | Delete a specific imported rule group.              | [Delete rule group](/docs/mimir/latest/references/http-api/#delete-rule-group)                       |

#### Create or update multiple rule groups

```
POST /convert/prometheus/config/v1/rules
```

Creates or updates multiple rule groups across multiple namespaces. This endpoint expects a request with a map of namespace titles to arrays of rule groups, and returns `202` on success.

This endpoint has no Mimir equivalent and is Grafana-specific for bulk operations.

##### Example request body

```yaml
namespace1:
  - name: MyGroupName1
    rules:
      - alert: MyAlertName1
        expr: up == 0
        labels:
          severity: warning
namespace2:
  - name: MyGroupName2
    rules:
      - alert: MyAlertName2
        expr: rate(http_requests_total[5m]) > 0.1
        labels:
          severity: critical
```

# Configure data source-managed alert rules

{{< admonition type="caution" >}}
Pre-provisioned Loki and Prometheus data source-managed alerts have been deprecated in Grafana Cloud and can not be created in new stacks.
New Grafana Cloud stacks will use Grafana-managed alerting (GMA) by default. Datasource-managed alerting (DMA) is not provisioned in new stacks. Existing stacks are not affected.

This applies to the default Loki and Prometheus Grafana Cloud data sources managed by Grafana Labs and Cloud Alertmanager, which will not be available nor will Grafana provision the data source for it. If you add your own Mimir, Loki, or Alertmanager data sources, you can continue to use data source-managed alerts.

Cloud users can import DMA rules into GMA rules with the import tool. See the [import data source-managed alerts to Grafana-managed alerts documentation](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/alerting-migration/) for information on how to do this.
{{< /admonition >}}

# Configure data source-managed alert rules

Data source-managed alert rules are alert rules that are stored in the data source, such as in Prometheus or Loki, rather than in Grafana.

In Grafana Alerting, you can:

1. Create and edit data source-managed rules for Grafana Mimir and Grafana Loki data sources.
1. View rules from Prometheus data sources when [Manage alerts via Alerting UI](/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/configure/) is enabled. However, you cannot create or edit these rules in Grafana.
1. [Import data source-managed rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/alerting-migration/) from Loki, Mimir, and Prometheus into Grafana-managed rules.

{{< admonition type="note" >}}
Data source-managed rules are supported for horizontal scalability, but they can introduce more operational complexity than Grafana-managed alert rules.

We recommend using [Grafana-managed alert rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/) whenever possible, as they provide a richer feature set and better integration with the full Grafana Alerting workflow.
{{< /admonition >}}

## Comparison with Grafana-managed rules

The table below compares Grafana-managed and data source-managed alert rules.

| <div style="width:200px">Feature</div>                                                                                                                                                                                                                                                                | <div style="width:200px">Grafana-managed alert rule</div>                                                                  | <div style="width:200px">Data source-managed alert rule                           |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| Supported data sources                                                                                                                                                                                                                                                                                | All backend data sources enabling the [`alerting` option](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-rbac/) | Only supports creating rules for Mimir and Loki data sources                      |
| Mix and match data sources                                                                                                                                                                                                                                                                            | Yes                                                                                                                        | No                                                                                |
| Add [expressions](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#advanced-options-expressions) to transform<wbr /> your data and set [alert conditions](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#alert-condition) | Yes                                                                                                                        | No                                                                                |
| [No data and error states](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/)                                                                                                                                                                      | Yes                                                                                                                        | No                                                                                |
| [Stale alert instances](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/)                                                                                                                                                                           | Yes                                                                                                                        | No                                                                                |
| [Images in alert notifications](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/images-in-notifications/)                                                                                                                                                     | Yes                                                                                                                        | No                                                                                |
| [Role-based access control](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-rbac/)                                                                                                                                                                                                          | Yes                                                                                                                        | No                                                                                |
| [Alert state history](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-state-history/)                                                                                                                                                                                              | Yes                                                                                                                        | No                                                                                |
| [Alert version history](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-rules/#view-compare-and-restore-alert-rules-versions)                                                                                                                                                      | Yes                                                                                                                        | No                                                                                |
| [Terraform provisioning](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/terraform-provisioning/)                                                                                                                                                                        | Yes                                                                                                                        | No                                                                                |
| [Recording rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/)                                                                                                                                                                                                    | Yes                                                                                                                        | Yes                                                                               |
| Organization                                                                                                                                                                                                                                                                                          | Organize and manage access with folders                                                                                    | Use namespaces                                                                    |
| Alert rule evaluation                                                                                                                                                                                                                                                                                 | Alert evaluation is done in Grafana                                                                                        | Alert rule evaluation is done in the data source and allow for horizontal scaling |
| Scaling                                                                                                                                                                                                                                                                                               | Alert rules are stored in the Grafana database.                                                                            | Alert rules are stored within the data source and allow for horizontal scaling    |

The following diagram shows the architecture of a Mimir setup that uses data source-managed alert rules.

{{< figure src="/media/docs/alerting/mimir-managed-alerting-architecture-v2.png" max-width="750px" alt="Data source-managed alerting architecture based on Grafana Mimir" >}}

## Create data source-managed alert rules

To create or edit data source-managed alert rules, follow these instructions.

### Before you begin

Verify that you have write permission to the Mimir or Loki data source. Otherwise, you cannot create or update data source-managed alert rules.

#### Enable the Ruler API

For more information, refer to the [Mimir Ruler API](/docs/mimir/latest/references/http-api/#ruler) or [Loki Ruler API](/docs/loki/latest/api/#ruler).

- **Mimir** - use the `/prometheus` prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the [Query API](/docs/mimir/latest/operators-guide/reference-http-api/#querier--query-frontend) and [Ruler API](/docs/mimir/latest/operators-guide/reference-http-api/#ruler) are under the same URL. You cannot provide a separate URL for the Ruler API.

- **Loki** - The `local` rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.

#### Permissions

Alert rules for Mimir or Loki instances can be edited or deleted by users with **Editor** or **Admin** roles.

If you do not want to manage alert rules for a particular data source, go to its settings and clear the **Manage alerts via Alerting UI** checkbox.

### Set alert rule name

{{< docs/shared lookup="alerts/configure-alert-rule-name.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Define query and condition

Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.

{{< admonition type="note" >}}
By default, new alert rules are Grafana-managed. To switch to **Data source-managed**, follow these instructions.
{{< /admonition >}}

1. Select a Prometheus-based data source from the drop-down list.

   You can also click **Open advanced data source picker** to find more options.

1. Enter a PromQL or LogQL query, including the alert condition.
1. In the **Rule type** option, select **Data source-managed**.
1. Click **Preview alerts**.

### Set alert evaluation behavior

Use [alert rule evaluation](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/) to determine how frequently an alert rule should be evaluated and how quickly it should change its state.

1. Select a namespace or click **+ New namespace**.
1. Select an evaluation group or click **+ New evaluation group**.

   If you are creating a new evaluation group, specify the interval for the group.

   All rules within the same group are evaluated sequentially over the same time interval. You can reorder them from the **Alert rules** page.

1. Enter a pending period.

   The [pending period](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#pending-period) is the period in which an alert rule can be in breach of the condition until it fires.

   After a condition is met, the alert goes into the **Pending** state. If the condition remains active for the duration specified, the alert transitions to the **Firing** state, else it reverts to the **Normal** state.

### Configure labels and notifications

Add [labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#labels) to your alert rules to set which [notification policy](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/) should handle your firing alert instances.

All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.

1. Add labels if you want to change the way your notifications are routed.

   Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.

### Configure notification message

Use [annotations](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#annotations) to add information to alert messages that can help respond to the alert.

Annotations are included by default in notification messages, and can use text or [templates](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/) to display dynamic data from queries.

Grafana provides several optional annotations.

1. Optional: Add a summary.

   Short summary of what happened and why.

1. Optional: Add a description.

   Description of what the alert rule does.

1. Optional: Add a Runbook URL.

   Webpage where you keep your runbook for the alert

1. Optional: Add a custom annotation.

   Add any additional information that could help address the alert.

1. Optional: **Link dashboard and panel**.

   [Link the alert rule to a panel](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/link-alert-rules-to-panels/) to facilitate alert investigation.

1. Click **Save rule**.

[//]: <> ({{< docs/shared lookup="alerts/configure-notification-message.md" source="grafana" version="<GRAFANA_VERSION>" >}})

# Configure Grafana-managed alert rules

{{< admonition type="caution" >}}
Pre-provisioned Loki and Prometheus data source-managed alerts have been deprecated in Grafana Cloud and can not be created in new stacks.
New Grafana Cloud stacks will use Grafana-managed alerting (GMA) by default. Datasource-managed alerting (DMA) is not provisioned in new stacks. Existing stacks are not affected.

This applies to the default Loki and Prometheus Grafana Cloud data sources managed by Grafana Labs and Cloud Alertmanager, which will not be available nor will Grafana provision the data source for it. If you add your own Mimir, Loki, or Alertmanager data sources, you can continue to use data source-managed alerts.

Cloud users can import DMA rules into GMA rules with the import tool. See the [import data source-managed alerts to Grafana-managed alerts documentation](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/alerting-migration/) for information on how to do this.
{{< /admonition >}}

# Configure Grafana-managed alert rules

Grafana-managed alert rules are the default way to create alert rules in Grafana.

Grafana-managed rules inherit their model from Prometheus Alerting and extend it with greater flexibility—such as multi-data source queries, expression-based transformations, advanced alert conditions, images in notifications, custom states, and more.

To create or edit Grafana-managed alert rules, follow the instructions below.

{{< admonition type="tip" >}}
For quick-start tutorials on key alerting features, see [Getting started with Grafana Alerting tutorials](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/tutorials/).
{{< /admonition  >}}

## Before you begin

Before you create Grafana-managed alert rules, review the following requirements and options.

### Supported data sources

Grafana-managed alert rules can query backend data sources when the data source's `plugin.json` file sets `{"backend": true, "alerting": true}`.

Before you create an alert rule, verify that the data sources you plan to query are compatible and properly configured.

You can find the public data sources that support alert rules in the [Grafana Plugins directory](/grafana/plugins/data-source-plugins/?features=alerting).

### Alert rule limits in Grafana Cloud

In Grafana Cloud, the number of Grafana-managed alert rules you can create depends on your Grafana Cloud plan.

- Free Forever plan: You can create up to 500 free alert rules, with each alert rule having a maximum of 1000 alert instances.
- All paid plans: They have a soft limit of 2000 alert rules and support unlimited alert instances. To increase the limit, open a support ticket from the [Cloud portal](/docs/grafana-cloud/account-management/support/).

### Permissions

Only users with **Edit** permissions for the folder storing the rules can edit or delete Grafana-managed alert rules. Only admins can restore deleted Grafana-managed alert rules.

### Provisioning

Note that if you delete an alert resource created in the UI, you can no longer retrieve it.

To backup and manage alert rules, you can [provision alerting resources](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/) using options such as configuration files, Terraform, or the Alerting API.

[//]: <> ({{< docs/shared lookup="alerts/configure-provisioning-before-begin.md" source="grafana" version="<GRAFANA_VERSION>" >}})

### Default versus Advanced options

Grafana Alerting provides multiple configuration options to support different alerting needs.

To simplify alert rule creation, some settings display a **Default** and an **Advanced** configuration view:

- [Define query and condition](#define-query-and-condition)
- [Configure notifications](#configure-notifications)

Use **Default** to create basic query conditions and route alerts directly to a contact point. Use **Advanced** for more complex query definitions, or to route alerts using notification policies.

You can switch between options at any time. After you create an alert rule, Grafana remembers your last selection and uses it as the default for the next rule.

## Set alert rule name

{{< docs/shared lookup="alerts/configure-alert-rule-name.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Define query and condition

Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.

Depending on the data source, the query editor offers a **Builder** and a **Code** option. **Builder** helps you construct a query with a visual interface. The **Builder** option is useful if you have limited experience with the query language, while **Code** lets you write the query directly for more control. Switch between them using the **Builder** and **Code** tabs on the query editor toolbar.

The **Default** option allows you to configure one query and one alert condition. The **Advanced** option allows you to use multiple queries and expressions for more complex rule definitions.

{{< collapse title="Default options" >}}

1. Select a data source.
1. Add a [query](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/).
1. Add an [alert condition](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#alert-condition).

   The **When** input includes the reducer function and the last input is the threshold.

1. Click **Preview** to verify.
   {{< /collapse >}}

{{< collapse title="Advanced options" >}}

1. Select a data source.
1. From the **Options** drop-down menu, specify a [time range](/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/use-dashboards/#time-units-and-relative-ranges).

   Note that Grafana Alerting only supports fixed relative time ranges, for example, `now-24hr: now`.
   It doesn't support absolute time ranges: `2021-12-02 00:00:00 to 2021-12-05 23:59:592` or semi-relative time ranges: `now/d to: now`.

1. Add a query.

   To add multiple [queries](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/), click **Add query**.

   All alert rules are managed by Grafana by default. If you want to switch to a data source-managed alert rule, click **Switch to data source-managed alert rule**.

1. Add one or more [expressions](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#expression-queries).

   a. For each expression, select either **Classic condition** to create a single alert rule, or choose from the **Math**, **Reduce**, and **Resample** options to generate separate alert for each series.

   When using Prometheus, you can use an instant vector and built-in functions, so you don't need to add additional expressions.

   b. Click **Preview** to verify that the expression is successful.

1. To add a [recovery threshold](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#recovery-threshold), enable the **Custom recovery threshold** toggle and enter a value that defines when the alert should recover—transition to `Normal` state from the `Alerting` or `Pending` state.

   You can only add one recovery threshold, and it must be part of the alert condition.

1. Click **Set as alert condition** on the query or expression you want to set as your [alert condition](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#alert-condition).
   {{< /collapse >}}

You can switch between **Default** and **Advanced** options at any time.

Switching from **Advanced** to **Default** may result in queries and expressions that can't be converted. In this case, a warning message asks if you want to continue to reset to default settings.

If the [Default versus Advanced options](#default-versus-advanced-options) feature is not enabled in your Grafana instance, follow the **Advanced options** instructions.

## Set folder and labels

Organize your alert rule with a folder and set of labels.

In the **Labels** section, you can optionally choose whether to add labels to organize your alert rules and their notifications. For more details, refer to [alert rule labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#labels).

1. Select a folder or click **+ New folder**.

1. Add labels, if required.

   Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.

## Configure alert evaluation behavior

Use [alert rule evaluation](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/) to determine how frequently an alert rule should be evaluated and how quickly it should change its state.

To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.

1. Select an evaluation group or click **+ New evaluation group**.

   If you are creating a new evaluation group, specify the interval for the group.

   All rules within the same group are evaluated concurrently over the same time interval.

1. Enter a [pending period](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#pending-period).

   The pending period is the period in which an alert rule can be in breach of the condition until it fires.

   After a condition is met, the alert goes into the **Pending** state.
   If the condition remains active for the duration specified, the alert transitions to the **Firing** state, else it reverts to the **Normal** state.

1. Optionally, set the [Keep firing for](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#keep-firing-for) period.

   You can set the minimum amount of time that an alert remains firing after the breached threshold expression no longer returns any results. This sets an alert to a "Recovering" state for the duration of time set here. The Recovering state can be used to reduce noise from flapping alerts. Select "none" to stop an alert from firing immediately after the breach threshold is cleared.

1. Turn on pause alert notifications, if required.

   You can pause alert rule evaluation to prevent noisy alerting while tuning your alerts.
   Pausing stops alert rule evaluation and doesn't create any alert instances.
   This is different from [mute timings](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/), which stop notifications from being delivered, but still allow for alert rule evaluation and the creation of alert instances.

1. In **Configure no data and error handling**, you can define the alerting behavior and alerting state for two scenarios:
   - When the evaluation returns **No data** or all values are null.
   - When the evaluation returns **Error** or timeout.

   ### Configure no data and error handling

   Configure the behavior when the evaluation returns no data:

   {{< docs/shared lookup="alerts/modify-nodata-state.md" source="grafana" version="<GRAFANA_VERSION>" >}}

   You can also configure the behavior when the evaluation returns an error:

   {{< docs/shared lookup="alerts/modify-error-state.md" source="grafana" version="<GRAFANA_VERSION>" >}}

   For more details, refer to [alert instance states](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/) and [modify the no data or error state](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/#modify-the-no-data-or-error-state).

1. In **Configure no data and error handling**, you can also configure [Missing series evaluations to resolve](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/#configure-missing-series-evaluations-to-resolve): how many consecutive evaluation intervals must pass without data before an alert instance is considered stale.

## Configure notifications

Configure who receives notifications when the alert rule fires.

The **Default** option allows you to select a [contact point](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/contact-points/) to handle notifications for this alert rule. The **Advanced** option routes notifications through [notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/).

{{< collapse title="Default options" >}}

1.  Select an existing [contact point](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/contact-points/).

    All notifications for this alert rule are sent to this contact point automatically and notification policies aren't used.

1.  Optionally, configure additional notification settings like grouping or timing options.

{{< /collapse >}}

{{< collapse title="Advanced options" >}}

With this option, all notifications for this alert rule are managed by the [notification policy tree](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/), which routes alerts based on their labels.

Click **Change** to change the routing from the default policy to another policy.

You can preview which notification policy would handle notifications from this alert rule.

1. Click **Preview routing** after configuring the previous alert rule settings.

   Based on the alert labels and query labels, Grafana displays the notification policies that would handle the notifications.

1. Click **View route** to view policy and routing details.

{{< /collapse >}}

You can switch between **Default** and **Advanced** options at any time.

If the [Default versus Advanced options](#default-versus-advanced-options) feature is not enabled in your Grafana instance, follow the **Advanced options** instructions.

## Configure notification message

Use [annotations](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#annotations) to add information to alert messages that can help respond to the alert.

Annotations are included by default in notification messages, and can use text or [templates](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/) to display dynamic data from queries.

Grafana provides several optional annotations.

1. Optional: Add a summary.

   Short summary of what happened and why.

1. Optional: Add a description.

   Description of what the alert rule does.

1. Optional: Add a Runbook URL.

   Webpage where you keep your runbook for the alert

1. Optional: Add a custom annotation.

   Add any additional information that could help address the alert.

1. Optional: **Link dashboard and panel**.

   [Link the alert rule to a panel](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/link-alert-rules-to-panels/) to facilitate alert investigation.

1. Click **Save rule**.

[//]: <> ({{< docs/shared lookup="alerts/configure-notification-message.md" source="grafana" version="<GRAFANA_VERSION>" >}})

# Create and link alert rules to panels

Grafana allows you to link an alert rule to a dashboard panel. This can help you:

- Inform alert responders about where to investigate and which data to examine.
- Visualize the alert state directly from dashboards.
- Include a screenshot of the panel in notification messages.

An alert rule is linked to a panel by setting the [`__dashboardUid__` and `__panelId__` annotations](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#annotations). Both annotations must be set together.

## Link alert rules to panels

When configuring the alert rule, you can set the dashboard and panel annotations as shown in this [video](https://youtu.be/ClLp-iSoaSY?si=qKWnvSVaQuvYcuw9&t=170).

1. Configure the alert rule.
1. In the **Configure notification message** section, click **Link dashboard and panel**.
1. Select an existing dashboard, then choose a panel from the selected dashboard.
1. Complete the alert rule configuration and click **Save rule** to initiate the alert rule.

You can then [view the alert state on the panel](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-state/#view-alert-state-on-panels).

By default, notification messages include a link to the dashboard panel. Additionally, you can [enable displaying panel screenshots in notifications](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/images-in-notifications/).

{{< figure src="/media/docs/alerting/panel-displays-alert-state.png" max-width="1200px" caption="A panel displaying the alert status and state changes." >}}

## Create alert rules from panels

{{< shared id="create-alert-from-panel" >}}

To streamline alert creation, you can create an alert rule directly from a panel.

1. Navigate to a dashboard in the **Dashboards** section.
1. Hover over the top-right corner of a panel and click the panel menu icon.
1. From the dropdown menu, select **More...** > **New alert rule**.
1. This opens the **Edit rule** form and pre-fills some values:
   - Sets the annotations to the corresponding dashboard and panel.
   - Sets the alert rule query using the panel query.
1. Complete the alert rule configuration and click **Save rule** to initiate the alert rule.

{{< /shared >}}

You can then [view the alert state on the panel](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-state/#view-alert-state-on-panels).

By default, notification messages include a link to the dashboard panel. Additionally, you can [enable displaying panel screenshots in notifications](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/images-in-notifications/).

{{< admonition type="note" >}}
Changes to panel and alert rule queries aren't synchronized. If you change a query, you have to update it in both the panel and the alert rule.
{{< /admonition >}}

## Access linked alert rules from panels

This option is available only in [time series panels](/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/visualizations/time-series/). To access alert rules associated to a time series panel, complete the following steps.

1. Hover over the top-right corner of the panel and click the panel menu icon.
1. Click **Edit**.
1. Click the **Alert** tab to view existing alert rules or create a new one.

{{< admonition type="tip" >}}
For a practical example, refer to our [Getting started: Link alerts to visualizations tutorial](http://www.grafana.com/tutorials/alerting-get-started-pt6/).
{{< /admonition >}}

# Configure rule sequences

{{< docs/public-preview product="Rule sequences" >}}

# Configure rule sequences

Use rule sequences in Grafana Alerting when a recording rule or alert rule depends on the results of another recording rule. A rule sequence references existing Grafana-managed rules and defines their evaluation order and shared evaluation interval.

You can create and manage rule sequences through the Grafana Alerting API or Terraform. The API uses `v0alpha1` and is subject to change.

## Before you begin

Before you create a sequence, ensure you have the following:

- **Grafana version:** Use Grafana Cloud or Grafana OSS or Enterprise 13.1 or later.
- **Rules:** Create at least one Grafana-managed recording rule and any dependent recording rules or alert rules. Place all the rules in the same folder.
- **Recording rule storage:** Ensure your recording rules write to a Prometheus-compatible data source. In Grafana Cloud, rules without an explicit target write to your managed Prometheus data source by default. In self-managed Grafana, configure a target data source or a default target. For details, refer to [Create Grafana-managed recording rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/).
- **API access:** Use an authenticated account with permission to manage alert rules in the folder. For authentication options, refer to [Authentication options for the HTTP API](/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/authentication/).

## Understand evaluation order

For each evaluation interval, Grafana evaluates the rules in two stages:

1. Evaluate the recording rules in the order listed in `spec.recordingRules`.
1. Evaluate the alert rules in the order listed in `spec.alertingRules`, after the recording rules.

For example, you can evaluate a recording rule that calculates a request rate, then a recording rule that uses that metric to calculate an error ratio, and then an alert rule that queries the error ratio.

The sequence's `spec.trigger.interval` sets the evaluation interval for every rule in the sequence, overriding each rule's individual interval while it belongs to the sequence. You continue to manage each rule's queries, expressions, and other settings separately.

Rules outside the sequence retain their existing evaluation behavior. A sequence defines evaluation order, not a general-purpose organizational group.

Evaluation order doesn't guarantee that a preceding rule produces new data. Grafana continues to the next rule even if a recording rule fails or returns no data. A dependent rule can query older data or receive no data, depending on its query and the target data source. Configure its error and no-data handling accordingly.

Even when a recording rule succeeds, its samples might not be available to query immediately. Grafana waits for the write request to complete but doesn't check whether the target data source can return the samples before it evaluates the next rule. A sequence guarantees evaluation order, not read-after-write consistency.

## Create a sequence with the API

Create a `RuleSequence` resource that references the rules by name. For alert rules and recording rules, use each resource's `metadata.name`, which corresponds to its rule UID.

Send a `POST` request to `/apis/rules.alerting.grafana.app/v0alpha1/namespaces/{namespace}/rulesequences` with the `Content-Type: application/json` header and the following body:

```json
{
  "apiVersion": "rules.alerting.grafana.app/v0alpha1",
  "kind": "RuleSequence",
  "metadata": {
    "name": "request-error-sequence",
    "namespace": "<NAMESPACE>",
    "annotations": {
      "grafana.app/folder": "<FOLDER_UID>"
    }
  },
  "spec": {
    "trigger": {
      "interval": "1m"
    },
    "recordingRules": [{ "name": "<REQUEST_RATE_RULE_UID>" }, { "name": "<ERROR_RATIO_RULE_UID>" }],
    "alertingRules": [{ "name": "<ALERT_RULE_UID>" }]
  }
}
```

Replace the placeholders with the following values:

- **`<NAMESPACE>`:** Use your Grafana instance's namespace in both the request path and body. For namespace formats, refer to [API structure in Grafana](/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis/#namespace-namespace).
- **`<FOLDER_UID>`:** Use the UID of the folder that contains all referenced rules.
- **`<REQUEST_RATE_RULE_UID>`:** Use the UID of the recording rule that calculates the request rate.
- **`<ERROR_RATIO_RULE_UID>`:** Use the UID of the recording rule that queries the request-rate metric and calculates the error ratio.
- **`<ALERT_RULE_UID>`:** Use the UID of the alert rule that queries the error-ratio metric.

The example evaluates the three rules in that order every minute. Configure the dependent rules to query the metrics produced by the preceding recording rules. Adding a rule to a sequence doesn't change its query or automatically connect it to another rule's output.

To retrieve the sequence, send a `GET` request to `/apis/rules.alerting.grafana.app/v0alpha1/namespaces/{namespace}/rulesequences/{name}`. Use `request-error-sequence` as `{name}` for this example.

## Create a sequence with Terraform

Use the `grafana_apps_rules_rulesequence_v0alpha1` resource to manage a sequence in Terraform. Set `metadata.folder_uid` to the folder that contains the rules, and reference their UIDs in `spec.recording_rules` and `spec.alerting_rules`.

For a complete example and resource fields, refer to [grafana_apps_rules_rulesequence_v0alpha1](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_rules_rulesequence_v0alpha1).

## Update or remove a sequence

To change the order, membership, or interval through the API, retrieve the current sequence and send the updated resource with a `PUT` request to `/apis/rules.alerting.grafana.app/v0alpha1/namespaces/{namespace}/rulesequences/{name}`. Preserve its `metadata.resourceVersion` from the response.

To remove a sequence, send a `DELETE` request to `/apis/rules.alerting.grafana.app/v0alpha1/namespaces/{namespace}/rulesequences/{name}`. Deleting a sequence doesn't delete the rules it references.

After Grafana picks up the change, rules removed from a sequence return to their stored evaluation interval and group settings.

If you manage the sequence with Terraform, update or remove it in your Terraform configuration instead.

## Sequence constraints

When you create or update a sequence, follow these constraints:

- **Recording rules:** Include at least one recording rule. Alert rules are optional.
- **Folder:** Place the sequence and all referenced rules in the same folder.
- **Membership:** Reference each rule only once. A rule can belong to only one sequence.
- **Rule changes:** Remove a rule from its sequence before you delete the rule or move it to another folder.
- **Interval:** Use a positive interval that's an integer multiple of your instance's base evaluation interval. For example, with a base interval of `10s`, you can use `10s`, `30s`, or `1m`, but not `15s`. The API rejects invalid intervals instead of adjusting them.
- **Self-managed evaluation jitter:** In Grafana OSS and Enterprise, keep the `jitterAlertRulesWithinGroups` feature toggle disabled when you use rule sequences. This toggle disables sequential evaluation and is disabled by default.

## Related resources

For more information about recording rules and evaluation behavior, refer to the following resources:

- [Create Grafana-managed recording rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/)
- [How rules are evaluated within a group](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/evaluation-within-a-group/)

# Create data source-managed recording rules

[Recording rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/) allow you to periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.

Alert rules and dashboards can then query the new metric resulting from the recording rule. This is faster than querying real-time data and can help to reduce system load.

Data source-managed recording rules can query Prometheus-based data sources like Mimir or Loki. For more information on recording rules in Prometheus, refer to [Defining recording rules in Prometheus](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/).

Note that in data source-managed groups, the alert rules and recording rules within the same evaluation group are evaluated sequentially. This is useful to ensure that a recording rule is evaluated before any other alert rule queries the pre-computed metric.

## Before you begin

- Verify that you have write permission to the Prometheus or Loki data source. Otherwise, you will not be able to create or update Grafana Mimir managed alerting rules.

- For Grafana Mimir and Loki data sources, enable the ruler API by configuring their respective services.
  - **Loki** - The `local` rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.

  - **Mimir** - use the `/prometheus` prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the [Query API](/docs/mimir/latest/operators-guide/reference-http-api/#querier--query-frontend) and [Ruler API](/docs/mimir/latest/operators-guide/reference-http-api/#ruler) are under the same URL. You cannot provide a separate URL for the Ruler API.

## Add new recording rule

To create a new data source-managed recording rule:

1. Click **Alerts & IRM** -> **Alerting** -> **Alert rules**.
1. At the top of the Alert rules page, click **More** -> **New Data source recording rule**.

## Enter recording rule name

The recording rule name must be a Prometheus metric name and contain no whitespace.

## Define recording rule

Select your data source and enter a query. The queries used in data source-managed recording rules always run as instant queries.

## Add namespace and group

1. From the **Namespace** dropdown, select an existing rule namespace or add a new one.

   Namespaces can contain one or more rule groups and only have an organizational purpose.

1. From the **Group** dropdown, select an existing group within the selected namespace or add a new one.

   Rules within a group are run sequentially at a regular interval, with the same evaluation time.

   Newly created rules are appended to the end of the group, and you can reorder them from the **Alert rules** page.

## Add labels

Optionally, you can add custom labels to the resulting metric by selecting existing key-value pairs from the drop down or entering the new key or value.

## Query the new metric in dashboards or alert rules

Click **Save rule** or **Save rule and exit** to save the rule.

Once saved, the new recording metric is available for use in dashboards and alert rules.

# Create Grafana-managed recording rules

[Recording rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/) allow you to periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.

Alert rules and dashboards can then query the new metric resulting from the recording rule. This is faster than querying real-time data and can help to reduce system load.

Grafana does not contain an embedded time-series database to store recording rule results. You must bring your own Prometheus-compatible database to store the series generated by recording rules.

Grafana-managed recording rules offer the same Prometheus-like semantics but allow you to query [data sources supported by alerting](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/#supported-data-sources). Additionally, you can use recording rules to import and map data from other data sources into Prometheus.

In Grafana Cloud, recording rules support writing to Prometheus-compatible data sources configured with [Private Data Source Connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) (PDC). This allows you to write recording rule results to a Prometheus or Mimir instance in a private network without opening that network to inbound traffic from Grafana Cloud.

{{< admonition type="note" >}}

Grafana-managed recording rules are enabled by default in Grafana Cloud.

In Grafana OSS and Enterprise, you must enable them by following the [Before you begin](#before-you-begin) instructions.

{{< /admonition >}}

To configure Grafana-managed recording rules, complete the following steps.

## Before you begin

This section only applies to Grafana OSS and Grafana Enterprise.

If a rule does not explicitly specify a target data source for writing (for example, if it was created before Grafana 12.1), Grafana will fall back to the `default_datasource_uid` set in the configuration:

```
[recording_rules]
default_datasource_uid = my-uid
```

{{< admonition type="note" >}}
Grafana Cloud: If you leave **Target data source** blank when creating a recording rule, Grafana writes the results to your managed Prometheus data source named `grafanacloud-prom` by default. This may be different from the default data source you use in dashboards. To write to a different backend, explicitly select a target data source. In self-managed Grafana, you can set the default fallback with `default_datasource_uid` in the `[recording_rules]` section of the configuration.
{{< /admonition >}}

If you previously configured recording rules using the `url` and `basic_auth_*` configuration options, these are no longer supported. You must either:

- Set `default_datasource_uid` in the `[recording_rules]` section of the configuration file to point to the target data source
- Or, before upgrading to Grafana 12.1, enable the `grafanaManagedRecordingRulesDatasources` feature flag and update each recording rule individually to include a target data source

## Add new recording rule

To create a new Grafana-managed recording rule:

1. Click **Alerts & IRM** -> **Alerting** ->
   **Alert rules**.
1. At the top of the Alert rules page, click **More** -> **New Grafana recording rule**.

1. Enter the names to identify your recording rule and metric.

   The metric name must be a Prometheus metric name and contain no whitespace. For details, refer to [Prometheus metric names](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).

## Define recording rule

Define a query to get the data you want to measure and set the recording rule output.

1. Select a data source.
1. From the **Options** dropdown, specify a time range.

   {{< admonition type="note" >}}
   Grafana Alerting only supports fixed relative time ranges, for example, `now-24hr: now`.

   It does not support absolute time ranges: `2021-12-02 00:00:00 to 2021-12-05 23:59:592` or semi-relative time ranges: `now/d to: now`.
   {{< /admonition >}}

1. Add a query.

   To add multiple queries, click **Add query**.

1. Add one or more [expressions](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#expression-queries).

   a. For each expression, select either **Classic condition** to create a single recording rule, or choose from the **Math**, **Reduce**, and **Resample** options.

   When using Prometheus, you can use an instant vector and built-in functions, so you don't need to add additional expressions.

   b. Click **Preview** to verify that the expression is successful.

1. Click **Set as recording rule output** on the query or expression you want to set as your rule output.

## Set evaluation behavior

Use recording rule evaluation to determine how frequently a recording rule should be evaluated.

To do this, you need to make sure that your recording rule is in the right evaluation group with an interval that works best for your use case.

1. Select a folder or click **+ New folder**.
1. Select an evaluation group or click **+ New evaluation group**.

   If you are creating a new evaluation group, specify the interval for the group.

   All rules within the same group are evaluated concurrently over the same time interval. Every recording rule in a group uses the same evaluation time, meaning that all queries from the same group are always aligned with each other.

   If another recording rule or alert rule depends on this rule's results, you can define an explicit evaluation order through the API or Terraform. For instructions, refer to [Configure rule sequences](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/rule-sequences/).

1. Before or after creating a recording rule, you have the option to **Pause evaluation** if necessary.

### Advanced configuration

[min_interval](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#min_interval) sets the minimum interval to enforce between rule evaluations. The default value is 10s which equals the scheduler interval. Rules are adjusted if they are less than this value or if they are not multiple of the scheduler interval (10s). Higher values can help with resource management as fewer evaluations are scheduled over time.

This setting has precedence over each individual rule frequency. If a rule frequency is lower than this value, then this value is enforced.

This setting applies to both Grafana-managed alert and recording rules.

## Add labels

Optionally, you can add custom labels to the resulting metric by selecting existing key-value pairs from the drop down or entering the new key or value.

## Query the new metric in dashboards or alert rules

Click **Save rule** or **Save rule and exit** to save the rule.

Once saved, the new recording metric is available for use in dashboards and alert rules.

# Create Grafana-managed recording rules for Tempo

{{< docs/private-preview product="Grafana-managed recording rules for Tempo" >}}

Grafana-managed recording rules let you pre-compute query results and store them as new time series metrics. These metrics can then be reused in dashboards, alerts, or downstream queries, improving performance and reliability.

This guide explains how to configure recording rules for the **Tempo data source**, including product-specific considerations, such as TraceQL metrics queries, time range alignment, and evaluation delays.

For additional information, refer to [Grafana-managed recording rules](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/).

## Create a new Tempo recording rule

1. In Grafana, go to **Alerts & IRM** > **Alerting** > **Alert rules**.

1. From the **More+** menu, select **New Grafana recording rule**.

1. Enter recording rule and metric name.

   In the **New recording rule** window, configure the following:
   - **Name**: A human-readable identifier for the recording rule.
   - **Metric**: The Prometheus-compatible name of the new metric series that will be generated. For details, refer to [Prometheus metric names](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
   - **Target data source**: Must be a **Prometheus** data source with _write_ permissions (to store the rule's results). Ensure your Prometheus or Mimir metrics backend allows Grafana to write the new series.

1. Define the recording rule.
   - **Data source**: Select the **Tempo** data source with _read_ permissions.
   - In the **Options** dropdown, specify a time range.

     {{< admonition type="note" >}}
     Grafana Alerting only supports fixed relative time ranges, for example, `now-5m: now-4m`.

     It does not support absolute time ranges: `2021-12-02 00:00:00 to 2021-12-05 23:59:592` or semi-relative time ranges: `now/d to: now`.
     {{< /admonition >}}
     - TraceQL metrics queries are executed as **instant queries** over a window of time.
     - The range you select defines the aggregation window. For example use `now-5m: now-4m` for a 1-minute recording-rule interval with a 4-minute delay. The TraceQL query will run over the selected interval.
     - Always include a **delay** of a few minutes (for example, 2–5 minutes) to account for traces still in flight. This avoids missing late-arriving spans.
     - **Max data points** and **Interval** are not relevant for TraceQL instant queries.

   - **Query**: Enter a valid TraceQL metrics query.
     ```
     { resource.service.name = "checkout" }  | count_over_time()
     ```
   - **Expressions**: Typically use **Reduce** > **Last** to take the most recent computed value.

   {{< figure src="/media/docs/grafana/alerting/screenshot-recording-rule-tempo-definition-private-preview.png" alt="Defining the recording rule for Tempo." >}}

1. Organize the rule by selecting a **folder** and adding **labels**.
   - Labels can help group or filter rules later.
   - Folders provide UI organization and RBAC scoping.

1. Set evaluation behavior.

   Recording rules are executed by evaluation groups at fixed intervals. Align this interval with the query's time range.

   {{< admonition type="caution" >}}
   The **Evaluation interval** of the evaluation group should equal the size of the query window. For example, if your query covers `now-5m: now-4m` (1-minute window with 4-minute delay), use a **1-minute evaluation interval**. This ensures each evaluation produces a new, non-overlapping sample.
   {{< /admonition >}}

   {{< figure src="/media/docs/grafana/alerting/screenshot-recording-rule-tempo-evaluation-group-private-preview.png" alt="Defining the recording rule evaluation group for Tempo." max-width="400px" >}}

## Best practices

- Always include a delay of at least 2 minutes in the query range to avoid losing in-flight spans.
- Align the evaluation interval with your query window size.
- The time series generated by a recording rule will be time-shifted based on the selected query time range. To align it back with the original timeline, you can apply the `offset` modifier in your Prometheus queries.

# Migrate recorded queries

Users can transpose their [now-deprecated recorded queries](/docs/grafana/latest/administration/recorded-queries/) into Grafana-managed recording rules in a few easy steps. The query PromQL for each recorded query has been exposed on the recorded queries list along with the existing data source, time range, and interval values to simplify the migration process.

## Migrate your recorded queries to Grafana-managed alert rules

1. Navigate to **Administration** -> **Plugins and Data** -> **Recorded queries.**

1. Note the data source, query PromQL, interval, and time range, and copy them somewhere accessible.

   {{< figure alt="Example of relevant recorded query information"  src="/media/docs/alerting/rec-query-example.png" max-width="800px" >}}

1. Now navigate to **Alerting** -> **Alert rules.**

1. At the top of the Alert rules page, click **More** -> **New Grafana recording rule**.

   Add a name for your Recording Rule and a name for the new metric that the recording rule generates.

1. Select your data source and paste your ratio query PromQL into the query builder.

   Click **Options** and validate that the Time Range is the same as your recorded query.

1. Select the Folder you would like the rule to be created in.

1. Add any labels to the rule.

1. Select or create an evaluation group. Set your evaluation group’s evaluation interval to the interval of your recorded query.

1. Review your rule and click Save **rule and exit** when you are finished.

1. Update any areas where the recorded query was referenced to use the new Grafana-managed recording rule.

# Labels and annotations template examples

Templating allows you to add dynamic data from queries to alert labels and annotations. Dynamic data enhances alert context, making it easier for responders to quickly assess and address the issue.

This page provides common examples for templating labels and annotations. For more information on templating, refer to:

- [Template annotations and labels](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/)
- [Annotation and label template reference](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/)
- [Alerting template language](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/language)

## Annotation example

[Annotations](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#annotations) add extra details to alert instances and are often used to provide helpful information for identifying the issue and guiding the response.

A common use case for annotations is to display the specific query value or threshold that triggered the alert.

For example, you can display the query value from the [`$values`](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/#values) variable to inform about the CPU value that triggered the alert.

```
CPU usage has exceeded 80% ({{ $values.A.value }}) for the last 5 minutes.
```

Alternatively, you can use the [`index()`](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/language/#functions) function to retrieve the query value as follows.

```
CPU usage has exceeded 80% ({{ index $values "A" }}) for the last 5 minutes.
```

```template_output
CPU usage has exceeded 80% (81.2345) for the last 5 minutes.
```

### Include labels for extra details

To provide additional context, you can include labels from the query using the [`$labels`](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/#labels) variable.

For instance, the previous case could also include the affected `instance`.

```
CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) for the last 5 minutes.
```

```template_output
CPU usage for Instance 1 has exceeded 80% (81.2345) for the last 5 minutes.
```

You can incorporate any labels returned by the query into the template. For instance, the following template includes information about the environment and region where the alert occurred.

```
Alert triggered in {{ $labels.environment }} on {{ $labels.region }} region.
```

```template_output
Alert triggered in production on AMER region.
```

### Print a range query

To print the value of an instant query you can print its Ref ID using the `index` function or the `$values` variable:

```
{{ $values.A.Value }}
```

For range queries, reduce them from a time series to an instant vector using a reduce expression. You can then print the result by referencing its Ref ID. For example, if the reduce expression averages `A` with the Ref ID `B`, you would then print `$values.B`:

```
{{ $values.B.Value }}
```

### Humanize the value of a query

To print the humanized value of an instant query, use the [`humanize`](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/#humanize) function:

```
{{ humanize $values.A.Value }}
```

Alternatively:

```
{{ humanize (index $values "A").Value }}
```

```template_output
554.9
```

To print the value of an instant query as a percentage, use the [`humanizePercentage`](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/#humanizepercentage) function:

```
{{ humanizePercentage $values.A.Value }}
```

```template_output
10%
```

For additional functions to display or format data, refer to:

- [Annotation and label template functions](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/#functions)
- [Template language functions](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/language/#functions)

## Label example

[Labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#labels) determine how alerts are routed and managed, ensuring that notifications reach the right teams at the right time. If the labels returned by your queries don’t fully capture the necessary context, you can create a new label and sets its value based on query data.

### Based on query value

Here’s an example of creating a `severity` label based on a query value:

```go
{{- if (gt $values.A.Value 90.0) -}}critical
{{- else if (gt $values.A.Value 80.0) -}}high
{{- else if (gt $values.A.Value 60.0) -}}medium
{{- else -}}low
{{- end -}}
```

In this example, the `severity` label is determined by the query value:

- `critical` for values above 90,
- `high` for values above 80,
- `medium` for values above 60,
- and `low` for anything below.

You can then use the `severity` label to control how alerts are handled. For instance, you could send `critical` alerts immediately, while routing `low` severity alerts to a team for further investigation.

> **Note:** An alert instance is uniquely identified by its set of labels.
>
> - Avoid displaying query values in labels, as this can create numerous alert instances—one for each distinct label set. Instead, use annotations for query values.
> - If a templated label's value changes, it maps to a different alert instance, and the previous instance is considered **stale**. Learn all the details in this [example using dynamic labels](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/dynamic-labels/).

[//]: <> ({{< docs/shared lookup="alerts/note-dynamic-labels.md" source="grafana" version="<GRAFANA_VERSION>" >}})

### Based on query label

You can use labels to differentiate alerts coming from various environments (e.g., production, staging, dev). For example, you may want to add a label that sets the environment based on the instance’s label. Here’s how you can template it:

```go
{{- if eq $labels.instance "prod-server-1" -}}
production
{{- else if eq $labels.instance "staging-server-1" -}}
staging
{{- else -}}
development
{{- end -}}
```

This would print:

- For instance `prod-server-1`, the label would be `production`.
- For `staging-server-1`, the label would be `staging`.
- All other instances would be labeled `development`.

To make this template more flexible, you can use a regular expression that matches the instance name with the instance name prefix using the [`match()`](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/#match) function:

```go
{{- if match "^prod-server-.*" $labels.instance -}}
production
{{- else if match "^staging-server-.*" $labels.instance -}}
staging
{{- else -}}
development
{{- end -}}
```

{{< collapse title="Legacy Alerting templates" >}}

## Legacy Alerting templates

For users working with Grafana's legacy alerting system, templates can still be utilized to extract useful information from alert conditions. However, it's important to note that you cannot use `$labels` to print labels from the query if you are using classic conditions, and must use `$values` instead. The reason for this is classic conditions discard these labels to enforce uni-dimensional behavior (at most one alert per alert rule). If classic conditions didn't discard these labels, then queries that returned many time series would cause alerts to flap between firing and resolved constantly as the labels would change every time the alert rule was evaluated.

Instead, the `$values` variable contains the reduced values of all time series for all conditions that are firing. For example, if you have an alert rule with a query A that returns two time series, and a classic condition B with two conditions, then `$values` would contain `B0`, `B1`, `B2` and `B3`. If the classic condition B had just one condition, then `$values` would contain just `B0` and `B1`.

#### Print all labels from a classic condition

To print all labels of all firing time series use the following template (make sure to replace `B` in the regular expression with the Ref ID of the classic condition if it's different):

```go
{{ range $k, $v := $values -}}
{{ if (match "B[0-9]+" $k) -}}
{{ $k }}: {{ $v.Labels }}{{ end }}
{{ end }}
```

For example, a classic condition for two time series exceeding a single condition would print:

```
B0: instance=server1
B1: instance=server2
```

If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then its labels will be duplicated for each condition that is exceeded:

```
B0: instance=server1
B1: instance=server2
B2: instance=server1
B3: instance=server2
```

If you need to print unique labels you should consider changing your alert rules from uni-dimensional to multi-dimensional instead. You can do this by replacing your classic condition with reduce and math expressions.

#### Print all values from a classic condition

To print all values from a classic condition take the previous example and replace `$v.Labels` with `$v.Value`:

```go
{{ range $k, $v := $values -}}
{{ if (match "B[0-9]+" $k) -}}
{{ $k }}: {{ $v.Value }}{{ end }}
{{ end }}
```

For example, a classic condition for two time series exceeding a single condition would print:

```
B0: 81.2345
B1: 84.5678
```

If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then `$values` will contain the values of all conditions:

```
B0: 81.2345
B1: 92.3456
B2: 84.5678
B3: 95.6789
```

{{< /collapse >}}

# Alerting template language

Notification templates and alert rule templates, such as annotations and labels, both use the Go template language, [text/template](https://pkg.go.dev/text/template).

Both types of templates can use the same keywords, functions, and comparison operators of the Go template language, such as `range`, `if`, `and`, `index`, `eq`, and more.

However, it's important to note that because notifications and alert rules operate in distinct contexts, some additional variables and functions are only available for either notification or alert rule templates. Refer to:

- [Annotation and label template reference](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/)
- [Notification template reference](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/)

This documentation provides an overview of the functions and operators of the Go template language that are available for both notification and alert rule templates.

## Print

To print the value of something, use `{{` and `}}`. You can print the value of a [variable](#variables), a field of a variable, the result of a function, or the value of dot.

```go
{{ $values }}
{{ $values.A.Value }}
{{ humanize 1000.0 }}
{{ .Alerts }}
```

## Dot

In `text/template`, there is a special cursor called dot, written as `.`. You can think of this cursor as a variable whose value changes depending on where in the template it is used.

At the start of notification templates, dot (`.`) refers to [Notification Data](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#notification-data).

```go
{{ .Alerts }}
```

In annotation and label templates, dot (`.`) is initialized with all alert data. It’s recommended to use the [`$labels` and `$values` variables](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/#variables) instead to directly access the alert labels and query values.

{{< admonition type="note" >}}
Dot (`.`) might refer to something else when used in a [range](#range), a [with](#with), or when writing [templates](#templates) used in other templates.
{{< /admonition >}}

[//]: <> (The above section is not included in the shared file because `refs` links are not supported in shared files.)

{{< docs/shared lookup="alerts/template-language.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Annotation and label template reference

Annotations and labels in alert rules can be defined using plain text. However, you can also define templates to customize their values with dynamic data from alert rule queries.

For example, you can template the `summary` annotation to include information from query values, providing relevant alert context for responders. Refer to [Template annotations and labels](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/) for various use cases.

In templates, variables represent dynamic values from queries, while functions perform actions to transform or format this data.

## Variables

Variables represent dynamic values from alert rule queries that can be displayed or accessed in your templates.

The `$` and `.` symbols are used to reference variables and their properties. You can reference variables directly in your alert rule definitions using the `$` symbol followed by the variable name. Similarly, you can access properties of variables using the dot (`.`) notation in alert rule templates.

```
{{ $values.A.Value }}
```

Templates are based on the **Go templating system**. Refer to [Template language](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/language/) for additional information.

The following variables are available when templating annotations and labels:

| Variables          | Description                                                                                                                                                                                                                                                                      |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [$labels](#labels) | Contains all labels from the query, only query labels.                                                                                                                                                                                                                           |
| [$values](#values) | Contains the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs.                                                                                                                                                                  |
| [$value](#value)   | A string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule. When a single data source is used, it returns the value of the query. It is generally recommended to use [$values](#values). |

### $labels

The `$labels` variable contains all labels from the query. It excludes [user-configured and reserved labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#label-types), containing only query labels.

{{< figure src="/media/docs/alerting/query-labels-and-values.png" max-width="1200px" caption="An alert rule displaying labels and value from a query." >}}

For example, suppose you have a query that returns CPU usage for all of your servers, and you have an alert rule that fires when any of your servers have exceeded 80% CPU usage for the last 5 minutes. You want to add a summary annotation to the alert that tells you which server is experiencing high CPU usage. With the `$labels` variable you can write a template that prints a human-readable sentence such as:

```
CPU usage for {{ $labels.instance }} has exceeded 80% for the last 5 minutes
```

The outcome of this template would be:

```
CPU usage for server1 has exceeded 80% for the last 5 minutes
```

> If you are using a classic condition then `$labels` will not contain any labels from the query. Classic conditions discard these labels to enforce uni-dimensional behavior (at most one alert per alert rule). If you want to use labels from the query in your template then use the example [here](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/examples/#print-all-labels-from-a-classic-condition).

### $values

The `$values` variable is a table containing the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs (e.g, `A`, `B`, `C`, etc.). It does not contain the results of range queries, as they can return hundreds or thousands of rows.

Each Ref IDs, such as `$values.A`, has the following properties

| Property | Type            | Description                                                 |
| -------- | --------------- | ----------------------------------------------------------- |
| `Value`  | Float           | The value returned by the instant query or expression.      |
| `Labels` | Key/value pairs | The labels associated with the instant query or expression. |

To print labels from a specific query or expression, use its Ref ID and the `Labels` property:

```
{{ $values.A.Labels.instance }}
```

`$values` is indexed by Ref ID (`A`, `B`, `C`), not by the alert name.

For most alert messages, use [`$labels`](#labels) instead. `$labels` contains the query labels for the alert instance, so you don't need the Ref ID:

```
{{ $labels.instance }}
```

Here's the previous example printing now the value of the instant query with Ref ID `A`:

```
{{ $values.A.Value }} CPU usage for {{ $labels.instance }} over the last 5 minutes.
```

If the alert has the label `instance=server1` and the query returns `81.2345`, the template would print:

```
81.2345 CPU usage for instance1 over the last 5 minutes.
```

If the query in Ref ID `A` is a range query rather than an instant query then add a reduce expression with Ref ID `B` and replace `$values.A.Value` with `$values.B.Value`:

```
{{ $values.B.Value }} CPU usage for {{ $labels.instance }} over the last 5 minutes.
```

Alternatively, you can use the `index()` function to retrieve the query value:

```
{{ index $values "B" }} CPU usage for {{ index $labels "instance" }} over the last 5 minutes.
```

{{< admonition type="note" >}}

Variable names that start with a number (for example, `1B`) are not [valid identifiers in Go templates](https://go.dev/ref/spec#Identifiers).

To access a value or label whose key starts with a number, use the `index` function:

```
{{ index $values "1B" }} CPU usage for {{ index $labels "1instance" }} over the last 5 minutes.
```

Using `{{ $values.1B.Value }}` is invalid and causes the template code to render as plain text.

{{< /admonition  >}}

#### $value

The `$value` variable is a string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule.

When a single data source is used in the alert rule, `$value` will return the query value directly.

This example prints the `$value` variable:

```
{{ $value }}: CPU usage has exceeded 80% for the last 5 minutes.
```

When using multiple data sources, it would display something like this:

```
[ var='A' labels={instance=instance1} value=81.234, , [ var='B' labels={instance=instance2} value=1 ] ]: CPU usage has exceeded 80% for the last 5 minutes.
```

But with a single data source, it would display just the value of the query:

```
81.234: CPU usage has exceeded 80% for the last 5 minutes.
```

Instead, we recommend using [$values](#values), which contains the same information as `$value` but is structured in an easier-to-use table format.

## Functions

Functions can perform actions in templates such as transforming or formatting data.

Note that the [functions provided by Go's template language](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/language/#functions), such as `index`, `and`, `printf`, and `len`, are available, along with many others.

In addition, the following functions are also available for templating annotations and labels:

**Numbers**

| Name                                      | Arguments        | Returns | Description                                                  |
| ----------------------------------------- | ---------------- | ------- | ------------------------------------------------------------ |
| [humanize](#humanize)                     | number or string | string  | Humanizes decimal numbers.                                   |
| [humanize1024](#humanize1024)             | number or string | string  | Like `humanize`, but uses 1024 as the base rather than 1000. |
| [humanizeDuration](#humanizeduration)     | number or string | string  | Humanizes a duration in seconds.                             |
| [humanizePercentage](#humanizepercentage) | number or string | string  | Humanizes a ratio value to a percentage.                     |
| [humanizeTimestamp](#humanizetimestamp)   | number or string | string  | Humanizes a Unix timestamp.                                  |
| [toTime](#totime)                         | number or string | time    | Converts a Unix timestamp in seconds to time.                |

**Strings**

| Name                            | Arguments                  | Returns | Description                                                                                   |
| ------------------------------- | -------------------------- | ------- | --------------------------------------------------------------------------------------------- |
| [title](#title)                 | string                     | string  | Capitalizes the first character of each word.                                                 |
| [toUpper](#toupper)             | string                     | string  | Returns all text in uppercase.                                                                |
| [toLower](#tolower)             | string                     | string  | Returns all text in lowercase.                                                                |
| [stripPort](#stripport)         | string                     | string  | Returns only host.                                                                            |
| [match](#match)                 | pattern, text              | boolean | Matches the text against a regular expression pattern.                                        |
| [reReplaceAll](#rereplaceall)   | pattern, replacement, text | string  | Replaces text matching the regular expression.                                                |
| [graphLink](#graphlink)         | expr                       | string  | Returns the path to the graphical view in `Explore` for the given expression and data source. |
| [tableLink](#tablelink)         | expr                       | string  | Returns the path to the tabular view in `Explore` for the given expression and data source.   |
| [parseDuration](#parseduration) | string                     | float   | Parses a duration string such as "1h" into the number of seconds it represents.               |
| [stripDomain](#stripdomain)     | string                     | string  | Returns the result of removing the domain part of a FQDN.                                     |

**Others**

| Name                        | Arguments     | Returns                | Description                                                                      |
| --------------------------- | ------------- | ---------------------- | -------------------------------------------------------------------------------- |
| [args](#args)               | []interface{} | map[string]interface{} | Translates a list of objects to a map with keys arg0, arg1 etc.                  |
| [safeHtml](#safehtml)       | string        | string                 | Marks string as HTML not requiring auto-escaping.                                |
| [externalURL](#externalurl) | none          | string                 | Returns the external URL of the Grafana server as configured in the ini file(s). |
| [pathPrefix](#pathprefix)   | none          | string                 | Returns the path of the Grafana server as configured in the ini file(s).         |

For further context on these functions, note that templating in Grafana is based on the [Prometheus template implementation](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/), enabling the use of these functions and Prometheus-like templates for formatting alert messages within Grafana.

#### humanize

The `humanize` function humanizes decimal numbers:

```
{{ humanize 1000.0 }}
```

```
1k
```

#### humanize1024

The `humanize1024` function works similarly to `humanize` but uses 1024 as the base rather than 1000:

```
{{ humanize1024 1024.0 }}
```

```
1ki
```

#### humanizeDuration

The `humanizeDuration` function humanizes a duration in seconds:

```
{{ humanizeDuration 60.0 }}
```

```
1m 0s
```

#### humanizePercentage

The `humanizePercentage` function humanizes a ratio value between 0 and 1 to a percentage:

```
{{ humanizePercentage 0.2 }}
```

```
20%
```

#### humanizeTimestamp

The `humanizeTimestamp` function humanizes a Unix timestamp:

```
{{ humanizeTimestamp 1577836800.0 }}
```

```
2020-01-01 00:00:00 +0000 UTC
```

#### toTime

The `toTime` function converts a Unix timestamp in seconds to time.:

```
{{ toTime 1727802106 }}
```

```
2024-10-01 17:01:46 +0000 UTC
```

#### title

The `title` function capitalizes the first character of each word:

```
{{ title "hello, world!" }}
```

```
Hello, World!
```

#### toUpper

The `toUpper` function returns all text in uppercase:

```
{{ toUpper "Hello, world!" }}
```

```
HELLO, WORLD!
```

#### toLower

The `toLower` function returns all text in lowercase:

```
{{ toLower "Hello, world!" }}
```

```
hello, world!
```

#### stripPort

The `stripPort` splits string into host and port, then returns only host:

```
{{ stripPort "example.com:8080" }}
```

```
example.com
```

#### match

The `match` function matches the text against a regular expression pattern:

```
{{ match "a.*" "abc" }}
```

```
true
```

#### reReplaceAll

The `reReplaceAll` function replaces text matching the regular expression:

```
{{ reReplaceAll "localhost:(.*)" "example.com:$1" "localhost:8080" }}
```

```
example.com:8080
```

#### graphLink

The `graphLink` function returns the path to the graphical view in [Explore](/docs/grafana/<GRAFANA_VERSION>/visualizations/explore/) for the given expression and data source:

```
{{ graphLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }}
```

```
/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":false,"range":true}]
```

#### parseDuration

The `parseDuration` function parses a duration string such as "1h" into the number of seconds it represents.

```
{{ parseDuration "1h" }}
```

```
3600
```

#### stripDomain

The `stripDomain` removes the domain part of a FQDN, leaving port untouched:

```
{{ stripDomain "example.com:8080" }}
```

```
example:8080
```

#### tableLink

The `tableLink` function returns the path to the tabular view in [Explore](/docs/grafana/<GRAFANA_VERSION>/visualizations/explore/) for the given expression and data source:

```
{{ tableLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }}
```

```
/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":true,"range":false}]
```

#### args

The `args` function translates a list of objects to a map with keys arg0, arg1 etc. This is intended to allow multiple arguments to be passed to templates:

```
{{define "x"}}{{.arg0}} {{.arg1}}{{end}}{{template "x" (args 1 "2")}}
```

```
1 2
```

#### safeHtml

The `safeHtml` function marks string as HTML not requiring auto-escaping:

```
{{ safeHtml "<b>Text</b>"}}
```

```
<b>Text</b>
```

#### externalURL

The `externalURL` function returns the external URL of the Grafana server as configured in the ini file(s):

```
{{ externalURL }}
```

```
https://example.com/grafana
```

#### pathPrefix

The `pathPrefix` function returns the path of the Grafana server as configured in the ini file(s):

```
{{ pathPrefix }}
```

```
/grafana
```

## Differences with notification templates

Both notification templates and alert rule templates use the Go templating system. However, the [functions and variables available in notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/) differ from those used in annotations and labels templates, which are described in this documentation.

Annotation and label templates operate in the context of an individual alert instance, while notification templates apply to a notification that includes a group of alert(s).

For example, notification templates provide the `.Alerts` variable, which includes the list of all firing and resolved alerts in the notification. This variable is not available in alert rule templates, which operate within the context of a single alert instance.

Additionally, you cannot reuse templates for labels and annotations as you can with notification templates. Instead, you need to write each template inline within the label or annotation fields and manually copy them wherever you want to reuse the templates.

# Configure notification policies

Notification policies determine how alerts are routed to contact points.

Policies have a tree structure. Each policy can have one or more child policies and a set of label matchers.

Each alert (or alert instance) is evaluated by the default policy and subsequently by each child policy. Alerts are routed to the appropriate notification policy by matching alert labels with the policy's label matchers. For further details, refer to [label matching and routing in notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/).

{{< figure src="/media/docs/alerting/get-started-notification-policy-tree-combo.png" max-width="750px" alt="A diagram displaying how the notification policy tree routes alerts" >}}

When an alert instance is assigned to a notification policy, the notification policy is responsible for:

- [Grouping similar alerts](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/#group-notifications) to minimize the number of notifications.
- Controlling when notifications are sent using the [timing options](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/#timing-options).
- Determining the [contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/) that receive the alert notification.

{{< admonition type="note" >}}
The default notification policy and its child policies are assigned to a [specific Alertmanager](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/#alertmanager-architecture), and they cannot use contact points or mute timings from other Alertmanagers.
{{< /admonition >}}

## Edit the default notification policy

You can change the contact point, grouping, and timing options for this policy. Mute timings and active time intervals aren't available on the root policy. Add them on a [child policy](#add-a-child-policy) instead.

1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Notification configuration**, then select the **Notification policies** tab.
1. From the **Choose Alertmanager** dropdown, select an external Alertmanager. By default, the **Grafana Alertmanager** is selected.
1. In the Default policy section, click **...** -> **Edit**.
1. In **Default contact point**, update the [contact point](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/) for where to send notifications when alert rules do not match any specific policy.
1. In **Group by**, choose labels to group alerts. If multiple alerts match this policy, they are grouped by the selected labels, and notifications are sent per group. For more details on using this option, refer to [Group notifications](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/#group-notifications).
1. In **Timing options**, set the [timing options](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/#timing-options) to configure when notifications are sent.
   - **Group wait**: the time to wait before sending the first notification for a new group of alerts. Default is 30 seconds.
   - **Group interval**: the time to wait before sending a notification about changes in the alert group. Default is 5 minutes.
   - **Repeat interval**: the time to wait before sending a notification if the group has not changed since the last notification. Default is 4 hours.
1. Click **Save** to save your changes.

## Add a child policy

You can create a child policy under a default policy or under an existing child policy.

If you want to choose where to position your policy, see the section on **Add a sibling policy**.

1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Notification configuration**, then select the **Notification policies** tab.
1. From the **Choose Alertmanager** dropdown, select an Alertmanager. By default, the **Grafana Alertmanager** is selected.
1. Click **+New child policy** from the default policy or an existing child policy.
1. In the Matching labels section of the child policy, add one or more matching label rules to narrow down a specific case from the parent policy.

   For instance, a child policy of the default policy handles `team=security` alerts, or a child policy handles only the `severity=critical` alerts of a parent policy.

1. In the **Contact point** dropdown, select the [contact point](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/) to send notifications. If left empty, the contact point of the parent policy is [inherited](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/#inheritance).
1. Optionally, enable **Continue matching subsequent sibling nodes** to continue matching sibling policies even after the alert matched the current policy. If enabled, multiple policies can handle the same alert.
1. Optionally, enable **Override grouping** to set different [grouping](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/#group-notifications) than the parent policy. If disabled, the grouping of the parent policy is [inherited](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/#inheritance).
1. Optionally, enable **Override general timings** to set different [timing options](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/#timing-options) than the parent policy. If disabled, the timing options of the parent policy are [inherited](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/#inheritance).
1. Optionally, in **Mute timings** and **Active timings**, select time intervals that apply to this policy. These fields aren't available on the root policy.
1. Click **Save policy** to save your changes.

## Add a sibling policy

1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Notification configuration**, then select the **Notification policies** tab.
1. Find the child policy you want to create a sibling for.
1. Click **Add new policy** -> **New sibling above** or **New sibling below**.

   It's important to determine which policy receives the alert first and to set the correct order of sibling and child policies.

   Policies are evaluated from top to bottom. If a matching policy is found, the system continues to evaluate its child policies in the order they are displayed. For more details, refer to [notification policies routing](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/#routing).

1. Follow the instructions from step 5 onward in [adding a child policy](#add-a-child-policy).

## Search for policies

Grafana allows you to search within the tree of policies by the following:

- **Label matchers**
- **Contact Points**

To search by contact point simply select a contact point from the **Search by contact point** dropdown. The policies that use that contact point are highlighted in the user interface.

To search by label matchers simply enter a valid matcher in the **Search by matchers** input field. Multiple matchers can be combined with a comma (`,`).

It is important to note that all matched policies are **exact** matches. Grafana supports regular expressions for creating label matchers. It does not support regular expression or partial matching in the search for policies.

## Mute timings

Mute timings pause notifications during a recurring period. Active time intervals allow notifications only during a recurring period.

You can assign them only on child notification policies, not on the root policy. They aren't inherited from a parent notification policy. For instructions, refer to [Configure mute timings](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/).

## Manage multiple notification policy trees

By default, Grafana uses a single notification policy tree for all alert routing. As organizations grow in size and complexity, managing a single global tree can become difficult. Multiple notification policy trees allow you to split routing logic into separate, independently managed routing trees.

Each notification policy tree has its own name, root policy, and child policies which function similarly to the default policy tree. You can create additional routing trees to organize routing logic by team, service, or domain.

The Default Policy can't be deleted. You can, however, reset the policy to clear it.

For provisioning instructions, refer to [Enable multiple notification policy trees via Terraform](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/terraform-provisioning/#enable-multiple-notification-policy-trees).

## Example

An example of an alert configuration.

- Create a "default" contact point for slack notifications, and set it on the default policy.
- Edit the default policy grouping to group alerts by `cluster`, `namespace` and `severity` so that you get a notification per alert rule and specific Kubernetes cluster and namespace.
- Create specific route for alerts coming from the development cluster with an appropriate contact point.
- Create a specific route for alerts with "critical" severity with a more invasive contact point integration, like pager duty notification.
- Create specific routes for particular teams that handle their own on-call rotations.

# Configure silences

Silences stop notifications from being created for a specified time window but do not interrupt alert evaluation. Use them to temporarily prevent alert notifications, such as during incident response or a maintenance window.

{{< admonition type="note" >}}
Silences are assigned to a [specific Alertmanager](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alertmanager/) and only suppress notifications for alerts managed by that Alertmanager.

In Grafana Cloud, the standalone Alertmanager UI is deprecated and no longer available. Manage silences here in the Grafana Alerting UI instead. For details, refer to [Configure Alertmanagers](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alertmanager/).
{{< /admonition >}}

## Mute timings and active time intervals vs silences

[Mute timings and active time intervals](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/) and [silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/) are distinct methods to suppress notifications. They do not prevent alert rules from being evaluated or stop alert instances from appearing in the user interface; they only prevent notifications from being created.

The following table highlights the key differences between mute timings and silences.

|            | Mute timing                                                 | Silence                                                          |
| ---------- | ----------------------------------------------------------- | ---------------------------------------------------------------- |
| **Setup**  | Created and then added to notification policies             | Matches alerts using labels to determine whether to silence them |
| **Period** | Uses time interval definitions that can repeat periodically | Has a fixed start and end time                                   |

[//]: <> ({{< docs/shared lookup="alerts/mute-timings-vs-silences.md" source="grafana" version="<GRAFANA_VERSION>" >}})

## Add silences

To add a silence, complete the following steps.

1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Silences**.
1. From the **Alertmanager** dropdown, select an external Alertmanager to create and manage silences for the external data source. Otherwise, keep the default option of Grafana.
1. Click **Create silence** to open the Create silence page.
1. In **Silence start and end**, select the start and end date to indicate when the silence should go into effect and expire.
1. Optionally, in **Duration**, specify how long the silence is enforced. This automatically updates the end time in the **Silence start and end** field.
1. In the **Label** and **Value** fields, enter one or more _Matching Labels_ to determine which alerts the silence applies to.

   {{< collapse title="How label matching works" >}}

Use [labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/) and label matchers to link alert rules to [notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/) and [silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/). This allows for a flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.

A label matcher consists of 3 distinct parts, the **label**, the **value** and the **operator**.

- The **Label** field is the name of the label to match. It must exactly match the label name.

- The **Value** field matches against the corresponding value for the specified **Label** name. How it matches depends on the **Operator** value.

- The **Operator** field is the operator to match against the label value. The available operators are:

  | Operator | Description                                        |
  | -------- | -------------------------------------------------- |
  | `=`      | Select labels that are exactly equal to the value. |
  | `!=`     | Select labels that are not equal to the value.     |
  | `=~`     | Select labels that regex-match the value.          |
  | `!~`     | Select labels that do not regex-match the value.   |

{{< admonition type="note" >}}
If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match in order to link a rule to a policy.
{{< /admonition >}}

**Label matching example**

If you define the following set of labels for your alert:

`{ foo=bar, baz=qux, id=12 }`

then:

- A label matcher defined as `foo=bar` matches this alert rule.
- A label matcher defined as `foo!=bar` does _not_ match this alert rule.
- A label matcher defined as `id=~[0-9]+` matches this alert rule.
- A label matcher defined as `baz!~[0-9]+` matches this alert rule.
- Two label matchers defined as `foo=bar` and `id=~[0-9]+` match this alert rule.

**Exclude labels**

You can also write label matchers to exclude labels.

Here is an example that shows how to exclude the label `Team`. You can choose between any of the values below to exclude labels.

| Label  | Operator | Value |
| ------ | -------- | ----- |
| `team` | `=`      | `""`  |
| `team` | `!~`     | `.+`  |
| `team` | `=~`     | `^$`  |

    {{< /collapse >}}

[//]: <> ({{< docs/shared lookup="alerts/how_label_matching_works.md" source="grafana" version="<GRAFANA_VERSION>" >}})

Any matching alerts (in the firing state only) display under **Affected alert instances**. Grafana evaluates the silence against each instance's current [labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#labels), so if a label value changes, that instance is treated as a new one and might no longer match.

1. In **Comment**, add details about the silence.
1. Click **Submit**.

## Edit silences

To edit a silence, complete the following steps.

1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Silences** to view the list of existing silences.
1. Find the silence you want to edit, then click **Edit** (pen icon).
1. Make the desired changes, then click **Submit** to save your changes.

## Remove silences

To remove a silence, complete the following steps.

1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Silences** to view the list of existing silences.
1. Select the silence you want to end, then click **Unsilence**.

{{< admonition type="note" >}}
Clicking **Unsilence** ends the silence immediately, which is the only way to end a silence before its configured end time. Silences cannot be permanently deleted manually.
{{< /admonition >}}

## Silence duration and retention

**Duration limits**

There is no maximum duration for a silence. You can create a silence for any length of time. However, administrators can configure the following limits:

- `alertmanager_max_silences_count`: Maximum number of active and pending silences per tenant (default: 0, no limit), expired silences are also counted.
- `alertmanager_max_silence_size_bytes`: Maximum size of a silence in bytes (default: 0, no limit).

**Expired silence retention**

Expired silences are automatically deleted after 5 days. This retention period is not configurable. The cleanup process runs automatically every 15 minutes to remove expired silences that are older than the retention period.

## Rule-specific silences

Rule-specific silences are silences that apply only to a specific alert rule. They're created when you silence an alert rule directly using the **Silence notifications** action in the UI.

As opposed to general silences, rule-specific silence access is tied directly to the alert rule they act on. They can be created manually by including the specific label matcher: `__alert_rule_uid__=<alert rule UID>`.

## URL link to a silence form

Default notification messages often include a link to silence alerts.

In custom notification templates, you can use [`.Alert.SilenceURL`](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#alert) to redirect users to the UI where they can silence the given alert.

If [`.Alert.SilenceURL`](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#alert) doesn’t fit your specific use case, you can also create a custom silence link for your custom templates.

{{< collapse title="Create a custom silence link" >}}

When linking to a silence form, provide the default matching labels and comment via `matcher` and `comment` query parameters. The `matcher` parameter should be in the following format `[label][operator][value]` where the `operator` parameter can be one of the following: `=` (equals, not regular expression), `!=` (not equals, not regular expression), `=~` (equals, regular expression), `!~` (not equals, regular expression).
The URL can contain many query parameters with the key `matcher`.
For example, to link to silence form with matching labels `severity=critical` & `cluster!~europe-.*` and comment `Silence critical EU alerts`, create a URL `https://mygrafana/alerting/silence/new?matcher=severity%3Dcritical&matcher=cluster!~europe-*&comment=Silence%20critical%20EU%20alert`.

To link to a new silence page for an external Alertmanager, add a `alertmanager` query parameter with the Alertmanager data source name.

{{< /collapse >}}

## Inhibition rules

Inhibition rules suppress notifications for target alerts when a related source alert is already firing. For more information, refer to [Configure inhibition rules](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/inhibition-rules/).

# Import Alertmanager configuration to Grafana-managed notifications

You can import an existing Prometheus or Mimir Alertmanager configuration into Grafana Alerting. Grafana runs the imported receivers as Mimir Alertmanager v0 integrations and surfaces the configuration as Grafana notification resources: contact points, a notification policy tree, notification templates, time intervals, and inhibition rules. You operate your notification setup from Grafana.

Importing is a safe operation. The source Alertmanager keeps its configuration, and Grafana never writes back to it.

The import happens in two stages:

- **Stage**: Grafana keeps the imported configuration as a unit of its own and combines it with your Grafana resources at runtime.
- **Promote**: Grafana merges the imported configuration permanently into your Grafana configuration. Every imported resource becomes a normal, editable Grafana resource, and you can no longer revert the import in one action.

Because Grafana keeps the staged configuration separate, you can revert or re-import it all at once instead of resource by resource. Its resources are read-only, and no Grafana resource can reference them. The imported policy tree is the exception: an alert rule can route to it by name. Until a rule does, your existing notifications are unchanged.

{{< admonition type="note" >}}
Importing Alertmanager configuration is in [public preview](https://grafana.com/docs/release-life-cycle/#public-preview). The API is behind the `alertingImportAlertmanagerAPI` [feature toggle](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/) and the user interface is behind `alertingMigrationWizardUI`. Both are disabled by default. In Grafana Cloud, contact Support to enable them.
{{< /admonition >}}

## Before you begin

Before you import an Alertmanager configuration, make sure you have the following:

- **A source Grafana can read**: A configuration YAML file with its template files, or a configured Mimir Alertmanager data source. Prometheus Alertmanager data sources aren't supported.
- **Permissions**: Creating, reading, updating, or deleting an import requires the corresponding scoped Alertmanager import permissions. Promoting an import requires read and delete permissions for the import, create permissions for contact points and notification policy trees, and write permissions for any notification templates, time intervals, and inhibition rules in the configuration. The Admin role has these permissions by default. For more details, refer to [RBAC permissions](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/).

Not every Alertmanager configuration can be imported as-is. Refer to [limitations](#limitations) before you start.

## How it works

Grafana imports the configuration as it is and evaluates it the way your source Alertmanager does, for configurations from [supported versions](#limitations). Receivers keep their Alertmanager fields, and they notify with the same logic and message format, because Grafana runs them as Mimir-compatible integrations instead of rewriting them into native Grafana ones.

Mimir-compatible integrations don't offer what a native Grafana integration adds on top, such as [images in notifications](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/images-in-notifications/). To use those features, promote the import and then rebuild the contact point as a Grafana one.

### What gets imported

Grafana imports every field of the configuration: receivers, the routing tree, template files, time intervals, and inhibition rules. Fields keep their names and values, apart from the renames described in [name conflicts](#name-conflicts). Both `time_intervals` and the deprecated `mute_time_intervals` become time intervals.

The `global` section is the one part with no place of its own. Grafana resolves its values into the settings of each integration that relies on them as the configuration is parsed, so the defaults you set globally still apply.

### Routing

Grafana adds the imported routing tree as a named policy tree of its own, alongside your default notification policy and any other named policy trees. Alerts reach it only when an alert rule routes to it by name. The matchers inside the imported tree then route those alerts the way they did in your source Alertmanager. For more details on serving more than one policy tree, refer to [manage multiple notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy/#manage-multiple-notification-policy-trees).

If the imported root route leaves `group_wait`, `group_interval`, or `repeat_interval` unset, Grafana uses the [default notification timing values](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy/#edit-the-default-notification-policy).

You choose the tree name when you import. This name is also the identifier of the import, so pick something you recognize, such as `mimir-prod`. The name can't be `default` or match another named policy tree. It must be a valid DNS subdomain name and is length-limited; for more details, refer to [limitations](#limitations).

### Name conflicts

Contact points, time intervals, and notification templates are identified by name, and the imported configuration may reuse names that already exist. Rather than fail or overwrite, Grafana renames the incoming resource:

1. Grafana appends `_` and the import identifier to the name, for example `default` becomes `default_mimir-prod`.
1. If that name is also taken, Grafana appends a number, for example `default_mimir-prod_01`.

All references to a renamed resource are updated throughout the imported configuration, so routing still points at the right contact point.

Templates count conflicts differently. An imported template never conflicts with a Grafana-managed template of the same name, only with a template from another imported configuration. For more details, refer to [limitations](#limitations).

### Staged resources are read-only

While an import is staged, its resources appear with an **Imported** status in the Grafana Alerting user interface and in the notification APIs, and you can't edit or delete them individually. The deprecated provisioning API is the exception: it doesn't return them at all.

You also can't reference an imported resource from a Grafana resource. For example, you can't pick an imported contact point in a Grafana notification policy. The imported policy tree is the exception: an alert rule can route to it by name while the import is staged.

To make imported resources editable and available for reference, promote the import.

### One import at a time

Grafana stores one imported configuration per organization. Before you stage another configuration, promote or revert the existing one. You can also explicitly replace the existing configuration during import.

## Import with the Grafana Alerting user interface

The Grafana Alerting user interface imports notification resources and alert rules in one flow. It requires both the `alertingImportAlertmanagerAPI` and `alertingMigrationWizardUI` [feature toggles](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/).

1. Go to **Alerting** > **Alert rules**.
1. In the **More** menu, click **Import to Grafana Alerting**.
1. Choose how the resources are added:
   - **Stage** brings the configuration in as a read-only, reversible copy.
   - **Promote** merges the configuration into your live configuration immediately. This can't be undone. To reverse it, you have to delete each resulting resource by hand.
1. Click **Next**.
1. On the **Import notification resources** step, choose the **Import source**:
   - **Alertmanager config YAML** uploads a configuration file. Optionally, upload the template files the configuration references. Each file is imported as a template named after the file.
   - **Alertmanager data source** reads the configuration from a configured Alertmanager data source.
1. Enter a **Policy tree name**.

   Grafana validates the configuration as you fill in the form and reports any conflicts. The form lists the resources that Grafana renames before you import.

1. Click **Next**, and either configure the [alert rules import](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/alerting-migration/) or skip the step.
1. Review the summary, then click **Start import**.

## Auto-sync an Alertmanager configuration

Auto-sync keeps a staged notification configuration up to date with a Mimir or Cortex Alertmanager data source. Grafana periodically reads the source configuration and applies changes to the staged resources without writing back to the source. Auto-sync doesn't import or synchronize alert rules.

{{< admonition type="note" >}}
Auto-sync is in [private preview](https://grafana.com/docs/release-life-cycle/#private-preview). The `alerting.syncExternalAlertmanager` feature toggle is disabled by default and requires a Grafana restart when enabled. To access **Settings** > **Import**, also enable `alertingMigrationWizardUI`. In Grafana Cloud, contact Support to request access.
{{< /admonition >}}

### Enable auto-sync

Before you begin, configure a Mimir or Cortex Alertmanager data source. Auto-sync supports only these data sources. If a staged configuration with a different identifier already exists, promote or revert it first.

You need the following permissions, scoped to `configs:uid:default`:

- `notifications.alerting.grafana.app/configs:get` to read the configuration and sync status.
- `notifications.alerting.grafana.app/configs:update` to enable or disable auto-sync.

Organization administrators have these permissions by default. You can also grant them through [RBAC](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/).

1. Go to **Alerting** > **Settings**.
1. Click the **Import** tab.
1. In **Auto-sync configuration**, select the Alertmanager data source.
1. Click **Save**.

Grafana uses the data source UID as the import identifier and notification policy tree name. The imported resources remain read-only while staged. To send notifications through the imported policy tree, route your Grafana-managed alert rules to that tree. Use the data source UID as the routing tree name.

The **Auto-sync configuration** section shows the sync status and any errors from the last sync attempt. If Grafana can't fetch or validate an update, it keeps the configuration from the last successful sync.

### Stop auto-sync or promote the configuration

While auto-sync is configured, you can't manually import another notification configuration, and the user interface hides **Revert** for the sync-managed configuration. You can still import alert rules separately.

To stop synchronization, click **Disable sync** in **Auto-sync configuration** and confirm. Disabling sync doesn't delete the staged configuration. You can then [review, promote, or revert it](#review-promote-or-revert-a-staged-configuration).

[Promoting the sync-managed configuration](#promote-a-staged-configuration) stops synchronization from that data source.

If **Auto-sync configuration** shows **Managed by operator**, you can't change the setting through the user interface. In Grafana Cloud, contact Support to change or disable auto-sync.

## Import with the API

The Alertmanager import endpoints are compatible with the [Mimir Alertmanager HTTP API](/docs/mimir/latest/references/http-api/#alertmanager), so you can use `mimirtool` or plain HTTP requests.

In these endpoints, an import is addressed by its identifier, which is set with the `X-Grafana-Alerting-Config-Identifier` header and defaults to `imported`.

| Method | Endpoint                                          | Summary                                                                            |
| ------ | ------------------------------------------------- | ---------------------------------------------------------------------------------- |
| POST   | `/api/convert/api/v1/alerts`                      | Import an Alertmanager configuration, optionally promoting it in the same request. |
| GET    | `/api/convert/api/v1/alerts`                      | Get the staged configuration. Secrets are masked.                                  |
| DELETE | `/api/convert/api/v1/alerts`                      | Delete the staged configuration. The Grafana configuration is unaffected.          |
| POST   | `/api/convert/api/v1/alerts/{Identifier}/promote` | Promote a staged configuration into the Grafana configuration.                     |

The `POST` endpoint accepts YAML and JSON. If no media type is specified, YAML is assumed. The request body has the same shape as the Mimir Alertmanager configuration API:

```yaml
template_files:
  default.tmpl: '{{ define "custom" }}Custom message{{ end }}'
alertmanager_config: |
  route:
    receiver: webhook
  receivers:
    - name: webhook
      webhook_configs:
        - url: 'https://example.com/webhook'
```

A successful import returns the merge result:

```json
{
  "status": "success",
  "stats": {
    "added_route": "mimir-prod",
    "added_receivers": ["webhook"],
    "added_templates": ["default.tmpl"]
  }
}
```

### Configure auto-sync and read its status

Auto-sync uses a dedicated notification configuration API. Use credentials with the [auto-sync permissions](#enable-auto-sync).

{{< admonition type="note" >}}
Enable the `alerting.syncExternalAlertmanager` feature toggle before using this API. For availability and restart requirements, refer to [Auto-sync an Alertmanager configuration](#auto-sync-an-alertmanager-configuration).
{{< /admonition >}}

The configuration is a per-organization resource named `default`. Grafana creates it automatically.

Read the configuration and status with `GET /apis/notifications.alerting.grafana.app/v0alpha1/namespaces/{namespace}/configs/default`:

```sh
curl \
  -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  "<GRAFANA_BASE_URL>/apis/notifications.alerting.grafana.app/v0alpha1/namespaces/<NAMESPACE>/configs/default"
```

Replace the following placeholders in these examples:

- `<GRAFANA_BASE_URL>`: The base URL of your Grafana instance.
- `<SERVICE_ACCOUNT_TOKEN>`: A service account token with the required configuration permissions.
- `<NAMESPACE>`: The API namespace for your Grafana organization.
- `<DATASOURCE_UID>`: The UID of the Mimir or Cortex Alertmanager data source to synchronize.

Enable auto-sync with `PATCH /apis/notifications.alerting.grafana.app/v0alpha1/namespaces/{namespace}/configs/default`. Set `spec.externalAlertmanagerSync.datasourceUid` with a JSON patch:

```sh
curl -X PATCH \
  -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  -H "Content-Type: application/json-patch+json" \
  "<GRAFANA_BASE_URL>/apis/notifications.alerting.grafana.app/v0alpha1/namespaces/<NAMESPACE>/configs/default" \
  --data '[{"op":"add","path":"/spec/externalAlertmanagerSync","value":{"datasourceUid":"<DATASOURCE_UID>"}}]'
```

A successful patch saves the setting; it doesn't mean the first sync has completed. Read the resource again to inspect these status fields:

- `status.externalAlertmanagerSync.datasourceUid`: The data source UID used on the last sync attempt.
- `status.externalAlertmanagerSync.origin`: Whether the source is configured through the API or managed by an operator. For an operator-managed source in Grafana Cloud, contact Support to change or disable it.
- `status.conditions`: Find the condition with `type` set to `ExternalAlertmanagerSynced`. Its `status` is `True` after a successful sync, `False` after a failed attempt, or `Unknown` when no sync is configured. Read `reason` and `message` for details. The `MergeCommitted` reason means the configuration was promoted and synchronization has stopped.

Status updates are asynchronous and can lag behind configuration changes.

To disable an API-configured sync, clear the setting with the same PATCH endpoint:

```sh
curl -X PATCH \
  -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  -H "Content-Type: application/json-patch+json" \
  "<GRAFANA_BASE_URL>/apis/notifications.alerting.grafana.app/v0alpha1/namespaces/<NAMESPACE>/configs/default" \
  --data '[{"op":"add","path":"/spec/externalAlertmanagerSync","value":{}}]'
```

Clearing the setting stops future synchronization but doesn't delete the staged notification resources. It doesn't override an operator-managed source. Read the status again to confirm that synchronization is no longer configured. Don't delete the `default` configuration resource or write its status; Grafana manages both its lifecycle and status.

### Optional headers

Use these headers for more granular import control:

#### `X-Grafana-Alerting-Config-Identifier`

The identifier of the import, which is also the name of the notification policy tree it creates. It must be a valid DNS subdomain name of at most 40 characters, using only lowercase alphanumeric characters, `-`, and `.`. Defaults to `imported`.

#### `X-Grafana-Alerting-Dry-Run`

Set to `true` to validate the configuration and report the merge result without saving anything. Use this to preview renames before importing.

#### `X-Grafana-Alerting-Promote`

Set to `true` to promote the configuration in the same request instead of staging it. Combine it with `X-Grafana-Alerting-Dry-Run` to preview what a promotion merges into the live configuration.

#### `X-Grafana-Alerting-Config-Force-Replace`

Set to `true` to replace an existing staged configuration that has a different identifier. Without it, importing a second configuration fails.

### `mimirtool`

Use `mimirtool alertmanager load` to import a configuration and its template files:

```bash
MIMIR_ADDRESS=<GRAFANA_BASE_URL>/api/convert/ \
MIMIR_AUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN> \
MIMIR_TENANT_ID=1 \
mimirtool alertmanager load alertmanager.yaml default.tmpl \
  --extra-headers "X-Grafana-Alerting-Config-Identifier=mimir-prod"
```

Replace the following placeholders:

- `<GRAFANA_BASE_URL>`: The base URL of your Grafana instance.
- `<SERVICE_ACCOUNT_TOKEN>`: A service account token with permission to import notification resources.

When the address points at `<GRAFANA_BASE_URL>/api/convert/`, `mimirtool` talks to Grafana rather than to a Mimir instance, so `MIMIR_TENANT_ID` must always be `1`.

`mimirtool alertmanager get` and `mimirtool alertmanager delete` read and remove the staged configuration in the same way.

### Promote a staged configuration

To promote a configuration that's already staged, call the promote endpoint with its identifier:

```bash
curl -X POST \
  -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  <GRAFANA_BASE_URL>/api/convert/api/v1/alerts/mimir-prod/promote
```

After promotion, the staged configuration no longer exists. Its resources are part of the Grafana configuration. You can edit them through the regular notification APIs and user interface.

## Review, promote, or revert a staged configuration

Staged configurations are managed in Grafana Alerting settings.

1. Go to **Alerting** > **Settings**.
1. Click the **Import** tab.

The **Staged configuration** section lists the contact points, notification policies, templates, time intervals, and inhibition rules the import contains, and links to each resource so you can inspect it before promoting.

To discard the import and everything it added, click **Revert**. Your Grafana configuration is unaffected. For a configuration managed by [auto-sync](#auto-sync-an-alertmanager-configuration), disable auto-sync before reverting. The user interface hides **Revert** while auto-sync is active. Deleting the staged configuration through the API doesn't disable auto-sync: Grafana can import it again when the source configuration changes or Grafana restarts.

{{< admonition type="warning" >}}
Reverting deletes the imported notification policy tree. Alert rules that route to that tree lose their target, and their alerts fall back to the root of your default notification policy. Update those rules to point elsewhere before you revert.
{{< /admonition >}}

## Limitations

Consider the following when you import an Alertmanager configuration:

- **Mimir version**: Grafana supports notification configuration imports from Grafana Mimir up to 3.0.x. Later Mimir versions can use integrations and fields that the imported receivers don't support.
- **Prometheus Alertmanager version**: Grafana supports notification configuration imports from Prometheus Alertmanager up to v0.27. Later versions add integrations and fields that the imported receivers don't support.
- **Unsupported receiver fields**: Integration fields that read their values from elsewhere, including every `*_file` and `*_ref` variant, have no equivalent in Grafana. Grafana rejects a configuration that contains one, so nothing is imported. Supply the value inline in the matching field instead, and drop the `_file` or `_ref` variant.
- **Data source support**: Grafana reads a configuration from a Mimir Alertmanager data source. Upstream Prometheus Alertmanager doesn't expose a configuration API, so those data sources aren't supported. Import a configuration YAML file instead.
- **Global settings**: After you promote an import, the `global` section no longer exists as a section of its own. Grafana has resolved its values into each integration setting that relies on them, so a promoted contact point carries the resolved value rather than a reference to `global`.
- **Policy tree name**: The name of the imported policy tree, which is also the import identifier, must be a valid DNS subdomain name of at most 40 characters, using only lowercase alphanumeric characters, `-`, and `.`.
- **One staged configuration at a time**: Grafana stores one staged configuration per organization. Before you stage another configuration, promote or revert the existing one. You can also explicitly replace the existing configuration during import.
- **Template name conflicts across imports**: Grafana renames colliding template files, but the templates defined inside those files share one namespace. Say an earlier import defines a template called `default.email` that renders `X`, and a new import defines `default.email` again, in a file under a different name, rendering `Y`. Which definition wins isn't deterministic, and every contact point that uses that template gets the winner. As a result, contact points from the earlier import can start sending the wrong content. Check the names of the templates you define, not just the filenames, before you import a second configuration.
- **Deprecated provisioning API**: A staged configuration isn't visible in the deprecated provisioning API, which doesn't return its contact points, templates, or time intervals. Use the Grafana Alerting notification API (`notifications.alerting.grafana.app`) instead.
- **Inhibition rules**: Imported inhibition rules are supported through the API only. There's no user interface for creating or editing them. For more details, refer to [configure inhibition rules](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/inhibition-rules/).

## Next steps

After you import your notification configuration:

- [Import your data source-managed alert rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/alerting-migration/) and route them through the imported policy tree.
- Review the imported [contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/), [notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy/), [templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/), and [time intervals](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/).

# Configure inhibition rules

{{< admonition type="note" >}}
Available in Grafana 13 or higher.
{{< /admonition >}}

An inhibition rule suppresses notifications for target alerts when source alerts with matching label values are already firing. This lets you reduce noise when a root-cause alert makes dependent alerts redundant.

For example, if a node is down (the **source**), you can inhibit all alerts for services running on that node (the **target**). This prevents your team from receiving individual alerts for each affected service when the underlying cause is already captured in the source alert.

{{< admonition type="note" >}}
Inhibition rules are assigned to a [specific Alertmanager](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/#alertmanager-architecture) and only suppress notifications for alerts managed by that Alertmanager.
{{< /admonition >}}

{{< admonition type="caution" >}}
Inhibition rules are intended for compatibility with configurations imported from Prometheus Alertmanager or Mimir. They have no dedicated management UI in Grafana by design.

If not carefully configured, inhibition rules can silently suppress alerts and make issues harder to detect. Consider [silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/) or [mute timings](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/) for most suppression use cases.
{{< /admonition >}}

## Inhibition rules vs silences

Both inhibition rules and [silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/) suppress alert notifications. The key difference is that inhibition rules suppress alerts automatically based on the presence of another alert, while silences suppress alerts for a fixed time window regardless of other alerts.

|              | Inhibition rule                               | Silence                                |
| ------------ | --------------------------------------------- | -------------------------------------- |
| **Trigger**  | Active when a matching source alert is firing | Active during a configured time window |
| **Duration** | Lasts as long as the source alert fires       | Has a fixed start and end time         |
| **Setup**    | Defined as a persistent configuration         | Created manually per occurrence        |

## Manage inhibition rules

You can manage inhibition rules by using the Grafana App Platform API. There is no dedicated UI for creating or editing inhibition rules.

The API resource is:

- **Group:** `notifications.alerting.grafana.app`
- **Version:** `v1beta1`
- **Resource:** `inhibitionrules`

{{< admonition type="caution" >}}
The inhibition rules API is in beta (`v1beta1`) and is subject to change.
{{< /admonition >}}

Inhibition rules are also supported in the Prometheus Alertmanager. Refer to [Configure Alertmanager](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alertmanager/) to set up an external Alertmanager.

## Inhibition rule schema

Each inhibition rule has the following fields:

| Field             | Required | Description                                                                                            |
| ----------------- | -------- | ------------------------------------------------------------------------------------------------------ |
| `name`            | Yes      | Unique name for the inhibition rule. Immutable after creation.                                         |
| `source_matchers` | Yes      | One or more matchers that identify source (inhibiting) alerts.                                         |
| `target_matchers` | Yes      | One or more matchers that identify target (inhibited) alerts.                                          |
| `equal`           | No       | Labels that must have equal values in both source and target alerts for the inhibition to take effect. |

### Matcher format

Both `source_matchers` and `target_matchers` are lists of structured matcher objects. Each matcher has:

| Field   | Description                                                                  |
| ------- | ---------------------------------------------------------------------------- |
| `label` | The name of the label to match against.                                      |
| `type`  | The matching operator. One of `=`, `!=`, `=~`, `!~`.                         |
| `value` | The value to match against. For `=~` and `!~`, this is a regular expression. |

| Operator | Description                                                          |
| -------- | -------------------------------------------------------------------- |
| `=`      | Select alerts where the label equals the value.                      |
| `!=`     | Select alerts where the label does not equal the value.              |
| `=~`     | Select alerts where the label matches the regular expression.        |
| `!~`     | Select alerts where the label does not match the regular expression. |

### The `equal` field

When `equal` is specified, the inhibition only applies when the source and target alerts have the same value for each listed label. For example, setting `equal: [cluster]` ensures that an alert in one cluster only inhibits alerts in the same cluster.

If `equal` is omitted, any firing source alert matching `source_matchers` inhibits all matching target alerts regardless of their label values.

Missing labels and empty-value labels are treated as equivalent for the purpose of `equal` matching.

## Example: suppress warning alerts

The following example defines an inhibition rule that suppresses all `warning`-severity alerts in a cluster when a `critical`-severity alert is already firing for the same cluster.

```json
{
  "name": "critical-inhibits-warnings",
  "source_matchers": [{ "label": "severity", "type": "=", "value": "critical" }],
  "target_matchers": [{ "label": "severity", "type": "=", "value": "warning" }],
  "equal": ["cluster", "namespace"]
}
```

In this example:

- The _source_ matcher selects `critical` alerts. When one fires, the rule becomes active.
- The _target_ matcher selects `warning` alerts. The rule suppresses these while the source alert fires.
- The `equal` field ensures suppression only applies when source and target share the same `cluster` and `namespace` label values.

## Inhibition rules and managed routes

An Alertmanager can serve multiple routing trees: the default Grafana-managed route and any routes imported from Prometheus Alertmanager or Mimir configurations. Inhibition rules behave differently depending on how each rule is created:

- **Grafana-managed rules** are created through the API and apply across the entire Alertmanager. They can match alerts handled by any route, including imported routes.
- **Imported rules** come from an imported Alertmanager configuration and are scoped to the alerts handled by their imported route. Grafana adds an internal label matcher that limits each imported rule to its own imported route, so an imported rule can't suppress alerts that flow through other routes.

To suppress alerts across multiple imported configurations, create a Grafana-managed rule through the API instead of relying on the inhibition rules embedded in an imported configuration.

# Configure mute timings and active time intervals

Mute timings and active time intervals let you determine how your alert notifications are handled during designated periods of time. After you create a time interval, you can apply it as either a mute timing or active time interval for your notification policies.

A mute timing is a recurring interval that stops notifications for one or multiple notification policies during a specified period. It suppresses notifications but does not interrupt alert evaluation.

Use mute timings to temporarily pause notifications for a specific recurring period, such as a regular maintenance window or weekends.

Active time intervals provide the opposite functionality, where alerts handled by a notification policy are suppressed unless the notification happens at a time that matches the time interval. Use active time intervals for periods where you want to reduce alert noise. Mute timings take precedence over active time intervals when they overlap.

{{< admonition type="note" >}}
Mute timings and active time intervals are assigned to a [specific Alertmanager](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/#alertmanager-architecture) and only suppress notifications for alerts managed by that Alertmanager.
{{< /admonition >}}

## Mute timings and active time intervals vs silences

[Mute timings and active time intervals](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/) and [silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/) are distinct methods to suppress notifications. They do not prevent alert rules from being evaluated or stop alert instances from appearing in the user interface; they only prevent notifications from being created.

The following table highlights the key differences between mute timings and silences.

|            | Mute timing                                                 | Silence                                                          |
| ---------- | ----------------------------------------------------------- | ---------------------------------------------------------------- |
| **Setup**  | Created and then added to notification policies             | Matches alerts using labels to determine whether to silence them |
| **Period** | Uses time interval definitions that can repeat periodically | Has a fixed start and end time                                   |

[//]: <> ({{< docs/shared lookup="alerts/mute-timings-vs-silences.md" source="grafana" version="<GRAFANA_VERSION>" >}})

## Add time intervals

1. In the left-side menu, click **Alerts & IRM**, and then **Alerting**.
1. Click **Notification configuration**, then select the **Time intervals** tab.
1. From the **Alertmanager** dropdown, select an external Alertmanager. By default, the **Grafana Alertmanager** is selected.
1. Click **+ New time interval**.
1. Fill out the form to create a [time interval](#time-intervals) to match against for your mute timing or active time interval.
1. Save your changes.

## Assign a time interval to a notification policy

After you create a time interval, assign it as a mute timing or an active time interval on a [child notification policy](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy/#add-a-child-policy). You can't assign these settings to the root policy. They aren't inherited from a parent policy.

1. In the left-side menu, click **Alerts & IRM**, and then **Alerting**.
1. Click **Notification configuration**, then select the **Notification policies** tab.
1. Find a child notification policy and click **...** -> **Edit**.
1. From **Mute timings** or **Active timings**, choose the time intervals to add to the policy.
1. Save your changes.

## Time intervals

A time interval is a specific duration during which alerts are suppressed. The duration typically consists of a specific time range and the days of the week, month, or year.

A mute timing or active time interval can contain multiple time intervals.

Supported time interval options are:

- Time range: The time inclusive of the start and exclusive of the end time (in UTC if no location has been selected, otherwise local time).
- Location: Depending on the location you select, the time range is displayed in local time.
- Days of the week: The day or range of days of the week. Example: `monday:thursday`.
- Days of the month: The date 1-31 of a month. Negative values can also be used to represent days that begin at the end of the month. For example: `-1` for the last day of the month.
- Months: The months of the year in either numerical or the full calendar month. For example: `1, may:august`.
- Years: The year or years for the interval. For example: `2021:2024`.

All fields are lists; to match the field, at least one list element must be satisfied. Fields also support ranges using `:` (e.g., `monday:thursday`).

If a field is left blank, any moment of time matches the field. For an instant of time to match a complete time interval, all fields must match.

If you want to specify an exact duration, specify all the options.

**Example**

If you wanted to create a time interval for the first Monday of the month, for March, June, September, and December, between the hours of 12:00 and 24:00 UTC your time interval specification would be:

- Time range:
  - Start time: `12:00`
  - End time: `24:00`
- Days of the week: `monday`
- Months: `3, 6, 9, 12`
- Days of the month: `1:7`

# Configure an Alertmanager contact point

Use the Alertmanager integration in a contact point to send specific alerts to a different Alertmanager.

If you have an existing Alertmanager running in your infrastructure, you can use a contact point to forward Grafana alerts to your Alertmanager.

For example, a team might run its own Alertmanager to manage notifications from other alerting systems. That team can create alerts with Grafana and configure an Alertmanager contact point to forward only their alerts to their existing Alertmanager.

This setup avoids duplicating Alertmanager configurations for better maintenance.

{{< admonition type="note" >}}
To send all Grafana-managed alerts to an Alertmanager, add it as a data source and enable it to receive all alerts. With this setup, you can configure multiple Alertmanagers to receive all alerts.

For setup instructions, refer to [Configure Alertmanagers](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/).
{{< /admonition >}}

## Configure an Alertmanager for a contact point

To create a contact point with Alertmanager integration, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a name for the contact point.
1. From the **Integration** list, select **Alertmanager**.
1. In the **URL** field, enter the URL of the Alertmanager.
1. (Optional) Configure [optional settings](#optional-settings).
1. Click **Save contact point**.

For more details on contact points, including how to test them and enable notifications, refer to [Configure contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/).

## Alertmanager settings

| Option | Description                                                                                                                                                                  |
| ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| URL    | The Alertmanager URL. This field is [protected](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/) from modification in Grafana Cloud. |

#### Optional settings

| Option              | Description                             |
| ------------------- | --------------------------------------- |
| Basic Auth User     | Username for HTTP Basic Authentication. |
| Basic Auth Password | Password for HTTP Basic Authentication. |

#### Optional notification settings

| Option                   | Description                                                         |
| ------------------------ | ------------------------------------------------------------------- |
| Disable resolved message | Enable this option to prevent notifications when an alert resolves. |

# Configure Amazon SNS notifications

Use the Amazon SNS integration in a contact point to send alert notifications to a SNS topic. Then, configure the SNS topic to forward notifications to distinct subscriber channels used in your SNS account.

## Before you begin

Before you begin, ensure you have the following:

- **AWS SNS Topic**: An SNS topic to send notifications to.
- **AWS IAM Identity with necessary access**: An IAM identity (e.g. user, role) with the necessary permissions to publish messages to the SNS topic.

For a minimal setup, refer to [Example using an Access Key](#example-using-an-access-key).

## Configure Amazon SNS for a contact point

To create a contact point with a SNS integration, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a name for the contact point.
1. From the **Integration** list, select **AWS SNS**.
1. Set up the required [settings](#sns-settings) for your SNS configuration.
1. Click **Save contact point**.

For more details on contact points, including how to test them and enable notifications, refer to [Configure contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/).

## SNS Settings

- **The Amazon SNS API URL**: (Optional) The SNS API URL, e.g., `https://sns.us-east-2.amazonaws.com`. If not specified, the SNS API URL from the SNS SDK will be used.
- **Signature Version (sigv4)**: Configures AWS's Signature Verification 4 signing process to sign requests.
  - **Region**: (Optional) The AWS region. If blank, the region from the default credentials chain is used.
  - **Access Key** : (Optional) The AWS API access key.
  - **Secret Key**: (Optional) The AWS API secret key.
    > Both `Access Key` and `Secret Key` must be provided together or left blank together.
    >
    > If left blank, Grafana searches for credentials using the default credentials chain, including environment variables (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`), the shared credential file, and EC2 instance roles.
  - **Profile**: (Optional) Named AWS profile used to authenticate.
- **SNS topic ARN**: (Optional) If you don't specify this value, you must specify a value for the `Phone number` or `Target ARN`. If you are using a FIFO SNS topic you should set a message group interval longer than 5 minutes to prevent messages with the same group key being deduplicated by the SNS default deduplication window.
- **Phone number**: (Optional) Phone number if message is delivered via SMS in E.164 format. If you don't specify this value, you must specify a value for the `SNS topic ARN` or `Target ARN`.
- **Target ARN**: (Optional) The mobile platform endpoint ARN if message is delivered via mobile notifications. If you don't specify this value, you must specify a value for the `SNS topic ARN` or `Phone number`.
- **Subject**: (Optional) Customize the subject. This field supports [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/) and, by default, uses the default title template (`default.title`). It **cannot be an empty string**.
- **Message**: (Optional) Customize the message. This field supports [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/) and, by default, uses the default message template (`default.message`).
- **Attributes**: (Optional) Add any SNS message attributes.

## Example using an Access Key

This section outlines a minimal setup to configure Amazon SNS with Alerting.

### 1. Create an SNS Topic and Email Subscriber

1. **Navigate to SNS in AWS Console**:
   - Go to the [Amazon SNS Console](https://console.aws.amazon.com/sns/v3/home).

2. **Create a new topic**:
   - On the **Topics** page, choose **"Create topic"**.
   - Select **"Standard"** as the type.
   - Enter a **Name** for your topic, e.g., `My-Topic`.
   - **Encryption**: Leave disabled for this minimal setup.
   - Click **"Create topic"**.

3. (Optional) **Add an email subscriber to help test**:
   - Within your newly created topic, click on **"Create subscription"**.
   - **Protocol**: Choose `Email`.
   - **Endpoint**: Enter your email address to receive test notifications.
   - Click **"Create subscription"**.
   - **Confirm Subscription**: Check your email and confirm the subscription by clicking the provided link.

### 2. Create an IAM Policy, User, and Access Key

1. **Navigate to IAM in AWS Console**:
   - Go to the [IAM Console](https://console.aws.amazon.com/iam/home).

2. **Create a new policy**:
   - On the **Policies** page, choose **"Create policy"**.
   - Switch to the **"JSON"** tab and paste the following policy, replacing `Resource` with your SNS topic ARN:

     ```json
     {
       "Version": "2012-10-17",
       "Statement": [
         {
           "Effect": "Allow",
           "Action": ["sns:Publish", "sns:GetTopicAttributes"],
           "Resource": "arn:aws:sns:<region>:<account_id>:<topic_name>"
         }
       ]
     }
     ```

   - Click **"Next"**, name it (e.g., `SNSPublishPolicy`), and click **"Create policy"**.

3. **Create a new IAM user and assign the policy**
   - In the IAM Console, on the **Users** page, choose **"Create user"**.
   - Enter a **User name**, e.g., `alerting-sns-user`.
   - Click **"Next"**.
   - In **Set permissions**, select **"Attach policies directly"**.
   - Search for the policy you created earlier (`SNSPublishPolicy`) and select it.
   - Click **"Next"** , and click **"Create user"**.

4. **Create an Access Key**:
   - Within your newly created user, click on **"Create access key"**.
   - Select an appropriate use-case, e.g., `Application running outside AWS`.
   - Click **"Next"** , and click **"Create access key"**.
   - **Save Credentials**: Note the **Access key ID** and **Secret access key** that are required in the next step.

### 3. Configure the SNS Contact Point in Grafana

Follow the steps in [configure Amazon SNS for a contact point](#configure-amazon-sns-for-a-contact-point), using the settings below and replacing the placeholders with the SNS and IAM values created in the previous steps.

- **The Amazon SNS API URL**: `https://sns.<region>.amazonaws.com`
- **Signature Version (sigv4)**:
  - **Region**: `<region>`
  - **Access Key**: `<YOUR_ACCESS_KEY>`.
  - **Secret Key**: `<YOUR_SECRET_ACCESS_KEY>`
- **SNS topic ARN**: `arn:aws:sns:<region>:<account_id>:<topic_name>`

[Test the contact point](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/#test-a-contact-point) to ensure it's working, or [enable notifications](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/#enable-notifications-for-a-contact-point) for it.

## Additional Resources

- [Configure contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/)
- [Amazon SNS Documentation](https://docs.aws.amazon.com/sns/index.html)
- [Amazon IAM Documentation](https://docs.aws.amazon.com/iam/index.html)
- [Prometheus Alertmanager SNS Configuration](https://prometheus.io/docs/alerting/configuration/#sns_config)

# Configure Discord for Alerting

Use the Grafana Alerting - Discord integration to receive alert notifications in your Discord channels when your Grafana alert rules are triggered and resolved.

## Before you begin

Create a Webhook to enable Grafana to send alert notifications to Discord channels.
To create a Webhook in Discord, complete the following steps.

1. Follow the steps in the [Intro to Webhooks guide](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks).
1. Copy the Webhook URL.

## Procedure

To create your Discord integration in Grafana Alerting, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a contact point name.
1. From the Integration list, select **Discord**.
1. In the **Webhook URL** field, paste in your Webhook URL.
1. Click **Test** to check that your integration works.

   ** For Grafana Alertmanager only.**

   A test alert notification should be sent to the Discord channel that you associated with the Webhook.

1. Click **Save contact point**.

## Next steps

The Discord contact point is ready to receive alert notifications.

To add this contact point to your alert, complete the following steps.

1. In Grafana, navigate to **Alerting** > **Alert rules**.
1. Edit or create a new alert rule.
1. Scroll down to the **Configure labels and notifications** section.
1. Under **Notifications** click **Select contact point**.
1. From the drop-down menu, select the previously created contact point.
1. Click **Save rule and exit**.

# Configure email for alert notifications

Use the email integration to send alert notifications to one or more addresses.

You can customize the [subject and main section of the email body](#optional-settings-using-templates). By default, the subject and body are generated from the alert data included in the notification.

## Before you begin

In Grafana OSS, you must configure SMTP settings before you can enable email notifications.

{{<admonition type="note">}}
In Grafana Cloud, SMTP configuration is not required.
{{</admonition>}}

1. Open the [configuration file](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/). The file is typically named `grafana.ini` or `custom.ini` and located in the `conf` directory of your Grafana installation.

1. Configure the [SMTP settings](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#smtp) for your email server in the `[smtp]` section.

1. Save your changes and restart Grafana.

1. Test email notifications by creating a contact point.

## Configure Email for a contact point

To create a contact point with a email integration, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ New contact point**.
1. Enter a name for the contact point.
1. From the **Integration** list, select **Email**.
1. Set up the required [settings](#email-settings) for your Email configuration.
1. Click **Save contact point**.

For more details on contact points, including how to test them and enable notifications, refer to [Configure contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/).

## Email settings

| Option    | Description                                                                                |
| --------- | ------------------------------------------------------------------------------------------ |
| Addresses | The list of email addresses to send the notifications. Email addresses are case sensitive. |

#### Optional settings

| Option       | Description                                                               |
| ------------ | ------------------------------------------------------------------------- |
| Single email | Send a single email to all email addresses rather than individual emails. |

#### Optional settings using templates

{{<admonition type="note">}}
You can customize the email subject and main section of the email body, but you can't edit HTML or CSS for visual changes.

In Grafana OSS and Enterprise, you can edit the full email template. However, this is not officially supported because it's an internal API that may change without prior notice.
{{</admonition>}}

| Option  | Description                                                                                                                                                                                           |
| ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Subject | Sets the email subject, replacing the default template. Supports [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/).                  |
| Message | Sets the main section of the email body, replacing the default template. Supports [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/). |

{{< figure src="/media/docs/alerting/custom-email-message5.png" caption="Email notification with custom message." max-width="750px" >}}

#### Optional notification settings

| Option                   | Description                                                         |
| ------------------------ | ------------------------------------------------------------------- |
| Disable resolved message | Enable this option to prevent notifications when an alert resolves. |

# Configure Google Chat for Alerting

Use the Grafana Alerting - Google Chat integration to receive alert notifications in your Google Chat space when your Grafana alert rules are triggered and resolved.

## Before you begin

Create a Webhook to enable Grafana to send alert notifications to a Google Chat space.
To create a Webhook in Google Chat space, complete the following steps.

1. Follow the steps in [Google's Chat app guide](https://developers.google.com/workspace/chat/quickstart/webhooks#create_a_webhook).
1. Copy the Webhook URL.

## Procedure

To create your Google Chat integration in Grafana Alerting, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a contact point name.
1. From the Integration list, select **Google Chat**.
1. In the **URL** field, paste in your Webhook URL.
1. Click **Test** to check that your integration works.

   ** For Grafana Alertmanager only.**

   A test alert notification should be sent to the Google Chat space that you associated with the Webhook.

1. Click **Save contact point**.

## Next steps

The Google Chat contact point is ready to receive alert notifications.

To add this contact point to your alert, complete the following steps.

1. In Grafana, navigate to **Alerting** > **Alert rules**.
1. Edit or create a new alert rule.
1. Scroll down to the **Configure labels and notifications** section.
1. Under **Notifications** click **Select contact point**.
1. From the drop-down menu, select the previously created contact point.
1. Click **Save rule and exit**.

# Configure Grafana IRM for Alerting

[//]: <> 'The IRM instructions are different for Grafana Cloud, so this page is currently skipped from Cloud docs.'

# Configure Grafana IRM for Alerting

In Grafana OSS and Grafana Enterprise, you can use a webhook contact point to send alerts to [Grafana IRM](/docs/grafana-cloud/alerting-and-irm/irm/). Grafana IRM can then route alerts based on escalation chains for your team's workflows.

The alert notification flow is as follows:

<sup>\*</sup> **Grafana OSS/Enterprise** (send webhook alerts)->**Grafana Cloud IRM** (route via [escalation chains](/docs/grafana-cloud/alerting-and-irm/irm/configure/escalation-routing/escalation-chains/))

{{< admonition type="note" >}}

The Free Forever plan in Grafana Cloud IRM includes 3 IRM active users per month.

These instructions apply only to Grafana OSS and Grafana Enterprise. To configure IRM for Grafana Cloud Alerting, refer to the [Grafana Cloud documentation](/docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points/integrations/configure-irm).

{{< /admonition >}}

## Configure an integration in Grafana IRM

First, enable an integration in IRM to accept alerts from Grafana Alerting. You can either create a new integration or use an existing **Alertmanager** or **Webhook** integration in IRM.

To create the integration, follow the same steps as described in [Configure an OnCall integration in IRM](/docs/grafana-cloud/alerting-and-irm/irm/configure/integrations/configure-integrations/):

1. Navigate to **Alerts & IRM** -> **IRM** -> **Integrations**.
1. Click **+ New integration**.
1. Select either **Alertmanager** or **Webhook** integration from the list.
   - **Alertmanager** integration – Includes preconfigured IRM templates for processing Grafana and Prometheus alerts.

   - **Webhook** integration – Uses default IRM templates for general alert processing.

1. Provide a title, description, and assign it to a team, then click **Create Integration**.
1. On the integration details page, copy the **HTTP Endpoint** URL to use in the next section.

## Configure a webhook contact point in Grafana Alerting

Next, configure the contact point in Grafana Alerting to send alert notifications to Grafana IRM.

In **Grafana OSS** and **Grafana Enterprise**, you need to create a **Webhook** contact point using the HTTP endpoint URL you copied in the previous step.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a name for the contact point.
1. From the **Integration** list, select **Webhook**.
1. In the **URL** field, enter the HTTP endpoint URL of the IRM integration.
1. Click **Save contact point**.

After configuring the contact point in Grafana Alerting and the integration in Grafana IRM, you can start testing the Alerting-to-IRM integration.

{{< figure src="/media/docs/alerting/test-alert-group-on-irm.png" max-width="750px" caption="Example of a test alert group in Grafana IRM" >}}

For more information, see:

- **[Configure contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/)** – Learn how to test the integration and enable notifications in Alerting.
- **[Webhook contact point](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier)** – Learn the format of the webhook payload and additional settings in Alerting.
- **[Configure IRM alert templates](/docs/grafana-cloud/alerting-and-irm/irm/configure/escalation-routing/alert-templates/)** – Learn how to process the incoming webhook messages in IRM.

## Enable heartbeat monitoring in Grafana IRM (optional)

Enabling the heartbeat in the IRM integration acts as a monitoring for your Grafana Alerting setup. If Grafana Alerting stops sending alerts—due to downtime or misconfiguration—Grafana IRM creates a new alert group to notify you.

To set up heartbeat monitoring, you must enable the heartbeat in IRM and create an alert rule that continuously sends alerts to the heartbeat endpoint.

#### Enable the heartbeat for the IRM integration

1. In **IRM**, select the IRM integration you configured earlier.
1. Click the ⋮ (three dots) on top right, then select **Heartbeat Settings**.
1. Copy the **Endpoint** URL—this will be used in a new contact point in Grafana Alerting.
1. Set the **heartbeat interval**, time period after which Grafana IRM starts a new alert group if it doesn’t receive a heartbeat.

#### Create another Webhook contact point in Grafana Alerting

Follow the same steps as before to create a webhook contact point in Grafana Alerting:

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a name for the contact point.
1. From the **Integration** list, select **Webhook**.
1. In the **URL** field, enter the endpoint URL of the heartbeat.
1. Click **Save contact point**.

You can now click the **Test** button to send an alert to the heartbeat endpoint. In **IRM**, verify the heartbeat status in the **Heartbeat** column on the **Integrations** page.

{{< figure src="/media/docs/alerting/view-heartbeat-status-on-irm.png" max-width="750px" caption="Heartbeat status column in the Grafana IRM Integrations page" >}}

#### Create an alert rule in Grafana Alerting

Create a [Grafana-managed alert rule](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/) with the following settings:

- **Always firing** – Use a query and alert condition that constantly fire. For example, select a Prometheus data source and set the query to `vector(1) > 0`.
- Configure a [pending period](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#pending-period) that is shorter than the **heartbeat interval**.
- Choose the **webhook contact point** you created for the heartbeat to forward alerts.
- Adjust [timing options](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/#timing-options) in the alert rule or notification policy to ensure alerts are forwarded before the **heartbeat interval** elapses:
  - **Group wait**: `0s`
  - **Group interval**: `1s`
  - **Repeat interval**: shorter than the **heartbeat interval**.

After it's created, the alert rule acts as a heartbeat, verifying that Grafana Alerting is running and sending alerts to Grafana IRM.

# Configure Jira for Alerting

Use the Jira integration in a contact point to create issues in your Jira instance when alerts fire. The integration supports both Jira Cloud and Jira Server/Data Center installations.

## Before you begin

Before you begin, ensure you have the following:

- A Jira instance (Cloud, Server, or Data Center)
- API access credentials for Jira
- Appropriate permissions to create issues in your target Jira project

## Configure Jira for a contact point

To create a contact point with a Jira integration, complete the following steps:

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
2. Click **+ Add contact point**.
3. Enter a name for the contact point.
4. From the **Integration** list, select **Jira**.
5. Set up the required [settings](#required-settings) for your Jira configuration.
6. Click **Save contact point**.

For more details on contact points, including how to test them and enable notifications, refer to [Configure contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/).

### Required Settings

| Key                 | Description                                                                                                                                                                                                                                                                                   |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| URL                 | The URL of the REST API of your Jira instance. Supported versions: `2` and `3` (e.g., `https://your-domain.atlassian.net/rest/api/3`). This field is [protected](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/) from modification in Grafana Cloud. |
| Basic Auth User     | Username for authentication. For Jira Cloud, use your email address.                                                                                                                                                                                                                          |
| Basic Auth Password | Password or personal token. For Jira Cloud, you need to obtain a personal token [here](https://id.atlassian.com/manage-profile/security/api-tokens) and use it as the password.                                                                                                               |
| API Token           | An alternative to basic authentication, a bearer token is used to authorize the API requests. See [Jira documentation](https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html) for more information.                                                        |
| Project Key         | The project key identifying the project where issues will be created. Project keys are unique identifiers for a project.                                                                                                                                                                      |
| Issue Type          | The type of issue to create (e.g., `Task`, `Bug`, `Incident`). Make sure that you specify a type that is available in your project.                                                                                                                                                           |

### Optional Settings

| Key                     | Description                                                                                                                                                                                                                                                                                                           |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Summary                 | The issue title. Supports templating. Max length is 255 characters.                                                                                                                                                                                                                                                   |
| Description             | The description of the issue. Depending on version of the API, it can be a text, Markdown, or JSON (v3 API only). Maximum size of the field is 32kb.<br>**Note:** JSON is not limited by the client, and if it exceeds the size, the API is likely to reject the request.                                             |
| Labels                  | Custom labels can be added to organize and filter issues created in Jira. Supports templating, allowing dynamic label generation based on alert data.                                                                                                                                                                 |
| Priority                | The priority level of the issue (e.g., `Low`, `Medium`, `High`, `Critical`). <br>Ensure that the priority value matches the available options for your Jira instance. You can customize priority levels in Jira [here](https://support.atlassian.com/jira-cloud-administration/docs/configure-priorities/).           |
| Resolve Transition      | The transition name to move the issue to a resolved state when an alert is resolved. Ensure that the value matches a valid transition available in your Jira workflow for the specified issue type. If this field is empty, the issue will not be transitioned to Done.                                               |
| Reopen Transition       | The transition name to move the issue back to an open state when an alert reoccurs. Ensure that the value matches a valid transition available in your Jira workflow for the specified issue type. If this field is empty, the issue will not be reopened.                                                            |
| Reopen Duration         | The time duration (in minutes) to control whether to reopen an issue that was closed within this duration or create a new one. If not specified, the most recent issue that matches the deduplication key will be updated and reopened (if reopen transition is specified).                                           |
| "Won't fix" Transition  | Specify a resolution status that should be ignored when searching for existing issues. For example, issues with this resolution will not be reopened or updated by subsequent alerts.                                                                                                                                 |
| Deduplication Key Field | Custom field to store the deduplication key. Must be a text field. <br> If not specified, the deduplication key is added to labels in the format of `ALERT(hash sum)`. See [Jira documentation](https://support.atlassian.com/jira-cloud-administration/docs/create-a-custom-field/) for how to create custom fields. |
| Fields                  | Extra Jira issue fields, including standard and custom fields. For custom fields, use `customfield_<FIELD_ID>`, where `<FIELD_ID>` is the numeric field ID in your Jira instance. For JSON parsing and templating behavior, refer to [Custom field data](#custom-field-data).                                         |

### Custom field data

Grafana attempts to parse each string field value as JSON when it loads the contact point. Grafana sends values parsed as objects or arrays as structured JSON, such as `{"value": "High"}`, but doesn't evaluate templates inside them. Grafana evaluates values that remain strings as notification templates, but doesn't parse their rendered output as JSON. Templates therefore can't generate dynamic object or array field values, such as those required by Jira select-list fields.

To set different structured values for different alerts, create a Jira contact point for each static value and route alerts with [notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy/). Or use a [Webhook contact point with a custom payload](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier/#custom-payload) to send structured JSON that includes alert data.

# Configure MQTT notifications

Use the MQTT integration in contact points to send alert notifications to your MQTT broker.

## Configure MQTT for a contact point

To create a contact point with MQTT integration, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a name for the contact point.
1. From the **Integration** list, select **MQTT**.
1. Enter your broker URL in the **Broker URL** field. Supports `tcp`, `ssl`, `mqtt`, `mqtts`, `ws`, `wss` schemes. For example: `tcp://127.0.0.1:1883`.
1. Enter the MQTT topic name in the **Topic** field.
1. (Optional) Configure [additional settings](#optional-settings).
1. Click **Save contact point**.

For more details on contact points, including how to test them and enable notifications, refer to [Configure contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/).

### Required Settings

| Option     | Description                                                                                                                                                                        |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Broker URL | The URL of the MQTT broker. This field is [protected](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/) from modification in Grafana Cloud. |
| Topic      | The topic to which the message will be sent.                                                                                                                                       |

### Optional Settings

| Option                   | Description                                                                                                                                                                                                                                                                                                                                                                          |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Message format           | If set to `json` (default), the notification message uses the [default JSON payload](#default-json-payload). <br/> If set to `text`, the notification message is fully customizable.                                                                                                                                                                                                 |
| Message                  | Depends on the **Message format** option. <br/> In `json` format, defines only the `message` field of the [default JSON payload](#default-json-payload). <br/> In `text` format, defines the [entire custom payload](#custom-payload). <br/> This field supports [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/). |
| Client ID                | The client ID to use when connecting to the MQTT broker. If blank, a random client ID is used.                                                                                                                                                                                                                                                                                       |
| Username                 | The username to use when connecting to the MQTT broker.                                                                                                                                                                                                                                                                                                                              |
| Password                 | The password to use when connecting to the MQTT broker.                                                                                                                                                                                                                                                                                                                              |
| QoS                      | The quality of service to use when sending the message. Options are `At most once`, `At least once`, and `Exactly once`.                                                                                                                                                                                                                                                             |
| Retain                   | If set to true, the message will be retained by the broker.                                                                                                                                                                                                                                                                                                                          |
| TLS                      | TLS configuration options, including CA certificate, client certificate, and client key, and disable certificate verification.                                                                                                                                                                                                                                                       |
| Disable resolved message | Enable this option to prevent notifications when an alert resolves.                                                                                                                                                                                                                                                                                                                  |

## Default JSON payload

If the **Message format** option is `json` (the default), the payload is like this example.

```json
{
  "receiver": "My MQTT integration",
  "status": "firing",
  "orgId": 1,
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High memory usage",
        "team": "blue",
        "zone": "us-1"
      },
      "annotations": {
        "description": "The system has high memory usage",
        "runbook_url": "https://myrunbook.com/runbook/1234",
        "summary": "This alert was triggered for zone us-1"
      },
      "startsAt": "2021-10-12T09:51:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/1afz29v7z/edit",
      "fingerprint": "c6eadffa33fcdf37",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1",
      "dashboardURL": "",
      "panelURL": "",
      "values": {
        "B": 44.23943737541908,
        "C": 1
      }
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "High CPU usage",
        "team": "blue",
        "zone": "eu-1"
      },
      "annotations": {
        "description": "The system has high CPU usage",
        "runbook_url": "https://myrunbook.com/runbook/1234",
        "summary": "This alert was triggered for zone eu-1"
      },
      "startsAt": "2021-10-12T09:56:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/d1rdpdv7k/edit",
      "fingerprint": "bc97ff14869b13e3",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1",
      "dashboardURL": "",
      "panelURL": "",
      "values": {
        "B": 44.23943737541908,
        "C": 1
      }
    }
  ],
  "groupLabels": {},
  "commonLabels": {
    "team": "blue"
  },
  "commonAnnotations": {},
  "externalURL": "https://play.grafana.org/",
  "version": "1",
  "groupKey": "{}:{}",
  "message": "**Firing**\n\nLabels:\n - alertname = T2\n - team = blue\n - zone = us-1\nAnnotations:\n - description = This is the alert rule checking the second system\n - runbook_url = https://myrunbook.com\n - summary = This is my summary\nSource: https://play.grafana.org/alerting/1afz29v7z/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1\n\nLabels:\n - alertname = T1\n - team = blue\n - zone = eu-1\nAnnotations:\nSource: https://play.grafana.org/alerting/d1rdpdv7k/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1\n"
}
```

### Body

If the **Message format** option is `json` (the default), the payload contains the following fields.

| Key                 | Type                             | Description                                                                                                                                                                                                                               |
| ------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `receiver`          | string                           | Name of the contact point                                                                                                                                                                                                                 |
| `status`            | string                           | Current status of the alert, `firing` or `resolved`                                                                                                                                                                                       |
| `orgId`             | number                           | ID of the organization related to the payload                                                                                                                                                                                             |
| `alerts`            | array of [alerts](#alert-object) | Alerts that are triggering                                                                                                                                                                                                                |
| `groupLabels`       | object                           | Labels that are used for grouping, map of string keys to string values                                                                                                                                                                    |
| `commonLabels`      | object                           | Labels that all alarms have in common, map of string keys to string values                                                                                                                                                                |
| `commonAnnotations` | object                           | Annotations that all alarms have in common, map of string keys to string values                                                                                                                                                           |
| `externalURL`       | string                           | External URL to the Grafana instance sending this webhook                                                                                                                                                                                 |
| `version`           | string                           | Version of the payload                                                                                                                                                                                                                    |
| `groupKey`          | string                           | Key that is used for grouping                                                                                                                                                                                                             |
| `message`           | string                           | Custom message configured in **Message** (**Optional Settings**). <br/> Supports [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/); the output is formatted as a string. |

{{< admonition type="note" >}}

When using the `json` **Message format**, only the **message** field of the JSON payload is customizable, and its output is formatted as a string.

To customize the full payload in text or JSON format, use the `text` format and define a [custom payload](#custom-payload).

{{< /admonition >}}

### Alert object

The Alert object represents an alert included in the notification group, as provided by the [`alerts` field](#body).

{{< docs/shared lookup="alerts/table-for-json-alert-object.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Custom payload

When you set the **Message format** option to `text`, you can customize the entire payload of the MQTT message.

In this mode, the **Message** option defines the entire payload. It supports [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/) and can generate notification messages in plain text, JSON, or any custom format.

For examples of templates that produce plain text or JSON messages, refer to [notification template examples](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/examples/).

# Configure Opsgenie for Alerting

Use the Grafana Alerting - Opsgenie integration to receive alert notifications in your Opsgenie alert dashboard when your Grafana alert rules are triggered and resolved.

## Before you begin

Create an API key to enable Grafana to send alert notifications to Opsgenie alert dashboard.

To create an API key in Opsgenie, complete the following steps.

1. Follow the steps in the [API integration guide](https://support.atlassian.com/opsgenie/docs/create-a-default-api-integration/).

   Make sure you turn on the integration.

1. Copy the API key.

## Procedure

To create your Opsgenie integration in Grafana Alerting, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a contact point name.
1. From the **Integration** list, select **Opsgenie**.
1. In the **API key** field, paste in your API key.
1. Configure the **Alert API URL**.
   1. For Grafana Alertmanager, enter `https://api.opsgenie.com/v2/alerts`.
   1. For other Alertmanagers, enter the host for sending Opsgenie API requests, depending on the hosted region.
1. Click **Test** to check that your integration works.

   **For Grafana Alertmanager only.**

   A test alert notification is sent to the Alerts page in Opsgenie.

1. Click **Save contact point**.

## Next steps

The Opsgenie contact point is ready to receive alert notifications.

To add this contact point to your alert rule, complete the following steps:

1. In Grafana, navigate to **Alerting** > **Alert rules**.
1. Edit or create a new alert rule.
1. Scroll down to the **Configure labels and notifications** section.
1. Under **Notifications**, click **Select contact point**.
1. From the drop-down menu, select the previously created contact point.
1. Click **Save rule and exit**.

# Configure Slack for Alerting

Use the Grafana Alerting - Slack integration to send Slack notifications when your alerts are firing.

There are two ways of integrating Slack into Grafana Alerting.

1. Use a [Slack API token](https://api.slack.com/authentication/token-types)

   Enable your app to access the Slack API. If, for example, you are interested in more granular control over permissions, or your project is expected to regularly scale, resulting in new channels being created, this is the best option.

1. Use a [Webhook URL](https://api.slack.com/messaging/webhooks)

   Webhooks is the simpler way to post messages into Slack. Slack automatically creates a bot user with all the necessary permissions to post messages to one particular channel of your choice.

Note that you can only setup one Slack channel per contact point.

You can customize the `title` and `body` of the Slack message using [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/); however, you cannot modify its visual appearance with custom blocks. Additional optional settings are available to customize bot appearance, mentions, and message formatting. Refer to the [Optional settings](#optional-settings) section for more details.

## Before you begin

### Slack API Token

If you are using a Slack API Token, complete the following steps.

1. Follow step 1 of the [Slack API Quickstart](https://docs.slack.dev/app-management/quickstart-app-settings/#creating) to create the app.
1. Continue onto the second step of the [Slack API Quickstart](https://docs.slack.dev/app-management/quickstart-app-settings/#scopes) and add the [chat:write.public](https://api.slack.com/scopes/chat:write.public) scope as described to give your app the ability to post in all public channels without joining.
1. In OAuth Tokens for Your Workspace, copy the Bot User OAuth Token.
1. Open your Slack workplace.
1. Right click the channel you want to receive notifications in.
1. Click View channel details.
1. Scroll down and copy the Channel ID.

   Note that while going through these steps, Slack may prompt you to Reinstall your app in order for the changes to take effect.

### Webhook URL

If you are using a Webhook URL, complete the following steps.

1. Follow the [Creating an app](https://docs.slack.dev/app-management/quickstart-app-settings/#creating) section of the Slack API Quickstart to create the app.
1. Follow the [Create an incoming webhook](https://api.slack.com/messaging/webhooks#create_a_webhook) section of the Slack Incoming Webhooks guide to enable incoming webhooks for your app and create a webhook for your workspace.
1. Copy the **Webhook URL** that starts with `https://hooks.slack.com/services/`. You need this when setting up your contact point integration in Grafana Alerting.

## Procedure

To create your Slack integration in Grafana Alerting, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a contact point name.
1. From the Integration list, select Slack.
1. If you are using a Slack API token:
   - In the **Recipient** field, copy in the channel ID.
   - In the **Token** field, copy in the Bot User OAuth Token that starts with “xoxb-”.
1. If you are using a Webhook URL, in the **Webhook** field, copy in your Slack app Webhook URL.
1. Click **Test** to check that your integration works.

   ** For Grafana Alertmanager only.**

1. Click **Save contact point**.

## Optional settings

You can configure additional settings to customize your Slack notifications beyond the required fields.

### Customize bot appearance

You can customize how the bot appears in Slack channels:

| Option     | Description                                                                                                                             |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Username   | Override the default bot username that appears in Slack. Set the `username` field to customize the display name.                        |
| Icon emoji | Provide an emoji to use as the icon for the message. Set the `icon_emoji` field (for example, `:rocket:`). This overrides the icon URL. |
| Icon URL   | Provide a URL to an image to use as the icon for the message. Set the `icon_url` field with the image URL.                              |

### Configure mentions

You can automatically mention users, groups, or the entire channel when notifications are sent:

| Option          | Description                                                                                                                                                                                                                                                                               |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Mention users   | Mention one or more specific users in the notification. Set the `mentionUsers` field with a comma-separated list of user IDs (for example, `U024BE7LH,U024BE7LJ`). To find a user's ID, refer to [Slack's documentation on finding member IDs](https://api.slack.com/methods/users.list). |
| Mention groups  | Mention one or more user groups in the notification. Set the `mentionGroups` field with a comma-separated list of group IDs. You can copy group IDs from the group's Slack profile URL.                                                                                                   |
| Mention channel | Mention the entire channel or active members. Set the `mentionChannel` field to one of the following:<br>- Empty or omit the field to disable channel mentions<br>- `here` to mention every active channel member<br>- `channel` to mention every channel member                          |

### Customize message content

You can customize the notification message using [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/):

| Option    | Description                                                                                                                                        |
| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| Title     | Set a custom title for the Slack message. Use the `title` field with template variables (for example, `{{ template "slack.default.title" . }}`).   |
| Text body | Set a custom body text for the Slack message. Use the `text` field with template variables (for example, `{{ template "slack.default.text" . }}`). |
| Color     | Set the color of the vertical bar on the left side of the message attachment. Use the `color` field.                                               |

### Advanced configuration

For specialized environments, you can override the default Slack API endpoint:

| Option       | Description                                                                                                                                                                 |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Endpoint URL | Specify a custom Slack API endpoint for non-webhook requests. Set the `endpointUrl` field to your custom endpoint. The default is `https://slack.com/api/chat.postMessage`. |

## Next steps

The Slack contact point is ready to receive alert notifications.

To add this contact point to your alert, complete the following steps.

1. In Grafana, navigate to **Alerting** > **Alert rules**.
1. Edit or create a new alert rule.
1. Scroll down to the **Configure labels and notifications** section.
1. Under Notifications click **Select contact point**.
1. From the drop-down menu, select the previously created contact point.
1. **Click Save rule and exit**.

# Configure Microsoft Teams for Alerting

Use the Grafana Alerting - Microsoft Teams integration to receive notifications in your team’s channel when your alerts are firing.

Note that you can customize the `title` and `message` of the notification using [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/); however, you cannot modify its visual appearance with adaptive cards.

## Before you begin

To set up Microsoft Teams for integration with Grafana Alerting, create a new workflow that accepts Webhook requests. This allows Grafana to send alert notifications to Microsoft Teams channels.

### Create a workflow in Microsoft Teams

1. To create a new workflow, follow the steps in [Create flows in Microsoft Teams](https://learn.microsoft.com/en-us/power-automate/teams/teams-app-create).
1. Microsoft provides a template library. You can use the template **Post to a channel when a webhook request is received**.
1. At the end of workflow creation wizard, copy the URL that is provided.

**Note**
If you chose a private channel for the target of the workflow, you need to edit workflow before using it. Expand the step "Send each adaptive card", and then expand action "Post your own adaptive card as the Flow bot to a channel". Change "Post as" to User, and save the workflow.

## Procedure

To create your MS Teams integration in Grafana Alerting, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a contact point name.
1. From the Integration list, select **Microsoft Teams**.
1. In the **URL** field, copy in your Webhook URL.
1. Click **Test** to check that your integration works.

   ** For Grafana Alertmanager only.**

   A test alert notification should be sent to the MS Team channel.

1. Click **Save** contact point.

## Next steps

The Microsoft Teams contact point is ready to receive alert notifications.

To add this contact point to your alert, complete the following steps.

1. In Grafana, navigate to **Alerting** > **Alert rules**.
1. Edit or create a new alert rule.
1. Scroll down to the **Configure labels and notifications** section.
1. Under Notifications click **Select contact point**.
1. From the drop-down menu, select the previously created contact point.
1. **Click Save rule and exit**.

## Troubleshooting

- If Grafana reports that the notification was sent successfully but it was not delivered to the channel, check the workflow's run history. You can find it in the workflow details page.

# Configure Telegram for Alerting

Use the Grafana Alerting - Telegram integration to send [Telegram](https://telegram.org/) notifications when your alerts are firing.

## Before you begin

### Telegram limitation

Telegram messages are limited to 4096 UTF-8 characters. If you use a `parse_mode` other than `None`, truncation may result in an invalid message, causing the notification to fail.
For longer messages, we recommend using an alternative contact method.

### Telegram bot API token and chat ID

To integrate Grafana with Telegram, you need to get a Telegram **bot API token** and a **chat ID** (the ID of the Telegram chat where you want to receive the alert notifications). To complete the integration, use the browser version of Telegram.

### Set up your Telegram bot

Create a [Telegram bot](https://core.telegram.org/bots/api). You can associate this bot to your chats and perform different actions with it, such as receiving alerts from Grafana.

To set up the bot, complete the following steps.

1. **Open the Telegram app** on your device.
1. Find the Telegram bot named **BotFather**.
1. Type or press `/newbot`.
1. Choose a name for the bot. It must end in **bot** or **\_bot**. E.g. "my_bot".
1. **Copy the API token**.

### Chat ID

Add the bot to a group chat by following the steps below. Once the bot is added to the chat, you will be able to route your alert notifications to that group.

1. In the Telegram app, **open a group or start a new one**.
1. Search and **add the bot to the group**.
1. Copy the **chat ID** from the URL in your browser's address bar. It should look like this: `https://web.telegram.org/a/#-4266674385`.

   The chat ID is the sequence of numbers that follows the `#` symbol. For example: `-4266674385`.

## Procedure

To create your Telegram integration in Grafana Alerting, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a contact point name.
1. From the Integration list, select Telegram.
1. In the **BOT API Token** field, copy in the bot API token.
1. In the **Chat ID** field, copy in the chat ID.
1. Click **Test** to check that your integration works.

   ** For Grafana Alertmanager only.**

1. Click **Save contact point**.

## Next steps

The Telegram contact point is ready to receive alert notifications.

To add this contact point to your alert, complete the following steps.

1. In Grafana, navigate to **Alerting** > **Alert rules**.
1. Edit or create a new alert rule.
1. Scroll down to the **Configure labels and notifications** section.
1. Under Notifications click **Select contact point**.
1. From the drop-down menu, select the previously created contact point.
1. **Click Save rule and exit**.

# Configure PagerDuty for Alerting

Use the Grafana Alerting - PagerDuty integration to receive notifications in PagerDuty when your alerts are firing.

## Before you begin

To set up PagerDuty for integration with Grafana Alerting, you need to create a [PagerDuty](https://www.pagerduty.com/) account. There are several set up steps to perform within PagerDuty before you set up the integration in Grafana Alerting.

### Create a Service

In PagerDuty, a service represents a component, microservice, or infrastructure element that a team oversees, manages, and monitors.

1. Refer to [PagerDuty’s services and integrations guide](https://support.pagerduty.com/docs/services-and-integrations#create-a-service).

1. Follow steps 1 to 5 under **Create a Service**.

{{< admonition type="note" >}}
In step 5, choose **Create a service without an integration**.
{{< /admonition >}}

### Obtain a PagerDuty integration key

1. Once the service is created, click **Integrations** within the Service options.
1. Click **+ Add an integration**.
1. Select **Events API V2**.
1. Click **Add**.
1. Click the drop-down arrow to display the integration details.
1. Copy the **Integration Key**.

## Procedure

To create your PagerDuty integration in Grafana Alerting, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a contact point name.
1. From the Integration list, select **PagerDuty**.
1. In the **Integration Key** field, copy in your integration key.
1. Click **Test** to check that your integration works.

   ** For Grafana Alertmanager only.**

   An incident should display in the Service’s Activity tab in PagerDuty.

1. Click **Save contact point**.

## Next steps

The PagerDuty contact point is ready to receive alert notifications.

To add this contact point to your alert, complete the following steps.

1. In Grafana, navigate to **Alerting** > **Alert rules**.
1. Edit or create a new alert rule.
1. Scroll down to the **Configure labels and notifications** section.
1. Under Notifications click **Select contact point**.
1. From the drop-down menu, select the previously created contact point.
1. **Click Save rule and exit**.

# Configure webhook notifications

Use the webhook integration in contact points to send alert notifications to your webhook.

The webhook integration is a flexible way to integrate alerts into your system. When a notification is triggered, it sends a JSON request with alert details and additional data to the webhook endpoint.

## Configure webhook for a contact point

To create a contact point with webhook integration, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.
1. Click **+ Add contact point**.
1. Enter a name for the contact point.
1. From the **Integration** list, select **Webhook**.
1. In the **URL** field, copy in your Webhook URL.
1. (Optional) Configure [additional settings](#webhook-settings).
1. Click **Save contact point**.

For more details on contact points, including how to test them and enable notifications, refer to [Configure contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/).

## Webhook settings

| Option | Description                                                                                                                                                             |
| ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| URL    | The Webhook URL. This field is [protected](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/) from modification in Grafana Cloud. |

{{< admonition type="caution" >}}

The URL is not a secured field. It's stored in plain text and isn't masked when the contact point is read back. Include secrets such as tokens or passwords in the URL at your own risk.

To authenticate to your webhook endpoint, use the authorization fields in the [optional settings](#optional-settings) instead, such as Basic Authentication or the Authorization request header. These fields are stored securely and are masked on read.

{{< /admonition >}}

#### Optional settings

| Option                            | Description                                                                                                                                                                               |
| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| HTTP Method                       | Specifies the HTTP method to use: `POST` or `PUT`.                                                                                                                                        |
| Basic Authentication Username     | Username for HTTP Basic Authentication.                                                                                                                                                   |
| Basic Authentication Password     | Password for HTTP Basic Authentication.                                                                                                                                                   |
| Authentication Header Scheme      | Scheme for the `Authorization` Request Header. Default is `Bearer`.                                                                                                                       |
| Authentication Header Credentials | Credentials for the `Authorization` Request header.                                                                                                                                       |
| Extra Headers                     | Additional HTTP headers to include in the request. You can also override the default `Content-Type: application/json` header to specify a different content type for the request payload. |
| Max Alerts                        | Maximum number of alerts to include in a notification. Any alerts exceeding this limit are ignored. `0` means no limit.                                                                   |
| TLS                               | TLS configuration options, including CA certificate, client certificate, and client key.                                                                                                  |
| HMAC Signature                    | HMAC signature configuration options.                                                                                                                                                     |
| OAuth2                            | OAuth2 configuration options for authenticating the webhook request. Refer to [OAuth2 authentication](#oauth2-authentication) for details.                                                |

{{< admonition type="note" >}}

You can configure either HTTP Basic Authentication or the Authorization request header, but not both.

{{< /admonition >}}

#### HMAC signature

You can secure your webhook notifications using HMAC signatures to verify the authenticity and integrity of the requests. When enabled, Grafana signs the webhook payload with a shared secret using HMAC-SHA256.

| Option           | Description                                                                                                                                                                                                                    |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Secret           | The shared secret key used to generate the HMAC signature.                                                                                                                                                                     |
| Header           | The HTTP header where the signature will be set. Default is `X-Grafana-Alerting-Signature`.                                                                                                                                    |
| Timestamp Header | Optional header to include a timestamp in the signature calculation. When specified, Grafana will set a Unix timestamp in this header and include it in the HMAC calculation. This provides protection against replay attacks. |

When HMAC signing is configured, Grafana generates a signature using HMAC-SHA256 with your secret key. If a timestamp header is specified, a Unix timestamp is included in the signature calculation. The signature is calculated as:

```
HMAC(timestamp + ":" + body)
```

The timestamp is sent in the specified header. If no timestamp header is specified, the signature is calculated just from the request body. The signature is sent as a hex-encoded string in the specified signature header.

##### Validate a request

To validate incoming webhook requests from Grafana, follow these steps:

1. Extract the signature from the header (default is `X-Grafana-Alerting-Signature`).
2. If you configured a timestamp header, extract the timestamp value and verify it's recent to prevent replay attacks.
3. Calculate the expected signature:
   - Create an HMAC-SHA256 hash using your shared secret
   - If using timestamps, include the timestamp followed by a colon (`:`) before the request body
   - Hash the raw request body
   - Convert the result to a hexadecimal string
4. Compare the calculated signature with the one in the request header.

#### OAuth2 authentication

You can authenticate webhook requests using OAuth2 client credentials. When configured, Grafana requests an access token from the token URL using the client ID and secret, then includes the token as a bearer token in the `Authorization` header of the webhook request.

{{< admonition type="note" >}}

If you configure OAuth2 authentication, it takes precedence over HTTP Basic Authentication and the Authorization request header for the webhook request.

{{< /admonition >}}

| Option              | Description                                                                                                                                                                                                                         |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Token URL           | The URL of the OAuth2 token endpoint. This field is [protected](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/) from modification in Grafana Cloud.                                        |
| Client ID           | The OAuth2 client ID.                                                                                                                                                                                                               |
| Client Secret       | The OAuth2 client secret.                                                                                                                                                                                                           |
| Scopes              | Optional list of OAuth2 scopes to request when obtaining an access token.                                                                                                                                                           |
| Endpoint Parameters | Optional key-value pairs to include in the access token request.                                                                                                                                                                    |
| TLS                 | Optional TLS configuration used only for the OAuth2 token request, including CA certificate, client certificate, and client key.                                                                                                    |
| Proxy               | Optional HTTP proxy configuration used only for the OAuth2 token request. The proxy URL is [protected](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/) from modification in Grafana Cloud. |

#### Optional settings using templates

Use the following settings to include custom data within the [JSON payload](#body). Both options support using [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/).

| Option                            | Description                                                                                                                                                                                                   |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Title                             | Sends the value as a string in the `title` field of the [JSON payload](#body). Supports [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/).   |
| Message                           | Sends the value as a string in the `message` field of the [JSON payload](#body). Supports [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/). |
| [Custom Payload](#custom-payload) | Optionally override the default payload format with a custom template.                                                                                                                                        |

#### Optional notification settings

| Option                   | Description                                                         |
| ------------------------ | ------------------------------------------------------------------- |
| Disable resolved message | Enable this option to prevent notifications when an alert resolves. |

## Default JSON payload

The following example shows the payload of a webhook notification containing information about two firing alerts:

```json
{
  "receiver": "My Super Webhook",
  "status": "firing",
  "orgId": 1,
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High memory usage",
        "team": "blue",
        "zone": "us-1"
      },
      "annotations": {
        "description": "The system has high memory usage",
        "runbook_url": "https://myrunbook.com/runbook/1234",
        "summary": "This alert was triggered for zone us-1"
      },
      "startsAt": "2021-10-12T09:51:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/1afz29v7z/edit",
      "fingerprint": "c6eadffa33fcdf37",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1",
      "dashboardURL": "",
      "panelURL": "",
      "values": {
        "B": 44.23943737541908,
        "C": 1
      }
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "High CPU usage",
        "team": "blue",
        "zone": "eu-1"
      },
      "annotations": {
        "description": "The system has high CPU usage",
        "runbook_url": "https://myrunbook.com/runbook/1234",
        "summary": "This alert was triggered for zone eu-1"
      },
      "startsAt": "2021-10-12T09:56:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/d1rdpdv7k/edit",
      "fingerprint": "bc97ff14869b13e3",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1",
      "dashboardURL": "",
      "panelURL": "",
      "values": {
        "B": 44.23943737541908,
        "C": 1
      }
    }
  ],
  "groupLabels": {},
  "commonLabels": {
    "team": "blue"
  },
  "commonAnnotations": {},
  "externalURL": "https://play.grafana.org/",
  "version": "1",
  "groupKey": "{}:{}",
  "truncatedAlerts": 0,
  "title": "[FIRING:2]  (blue)",
  "state": "alerting",
  "message": "**Firing**\n\nLabels:\n - alertname = T2\n - team = blue\n - zone = us-1\nAnnotations:\n - description = This is the alert rule checking the second system\n - runbook_url = https://myrunbook.com\n - summary = This is my summary\nSource: https://play.grafana.org/alerting/1afz29v7z/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1\n\nLabels:\n - alertname = T1\n - team = blue\n - zone = eu-1\nAnnotations:\nSource: https://play.grafana.org/alerting/d1rdpdv7k/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1\n"
}
```

### Body

The JSON payload of webhook notifications includes the following key-value pairs:

| Key                 | Type                      | Description                                                                      |
| ------------------- | ------------------------- | -------------------------------------------------------------------------------- |
| `receiver`          | string                    | Name of the contact point.                                                       |
| `status`            | string                    | Current status of the alert, `firing` or `resolved`.                             |
| `orgId`             | number                    | ID of the organization related to the payload.                                   |
| `alerts`            | array of [alerts](#alert) | Alerts that are triggering.                                                      |
| `groupLabels`       | object                    | Labels that are used for grouping, map of string keys to string values.          |
| `commonLabels`      | object                    | Labels that all alarms have in common, map of string keys to string values.      |
| `commonAnnotations` | object                    | Annotations that all alarms have in common, map of string keys to string values. |
| `externalURL`       | string                    | External URL to the Grafana instance sending this webhook.                       |
| `version`           | string                    | Version of the payload structure.                                                |
| `groupKey`          | string                    | Key that is used for grouping.                                                   |
| `truncatedAlerts`   | number                    | Number of alerts that were truncated.                                            |
| `state`             | string                    | State of the alert group (either `alerting` or `ok`).                            |

The following key-value pairs are also included in the JSON payload and can be configured in the [webhook settings using notification templates](#optional-settings-using-templates).

| Key       | Type   | Description                                                                                                          |
| --------- | ------ | -------------------------------------------------------------------------------------------------------------------- |
| `title`   | string | Custom title. Configurable in [webhook settings using notification templates](#optional-settings-using-templates).   |
| `message` | string | Custom message. Configurable in [webhook settings using notification templates](#optional-settings-using-templates). |

### Alert

The Alert object represents an alert included in the notification group, as provided by the [`alerts` field](#body).

{{< docs/shared lookup="alerts/table-for-json-alert-object.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Custom Payload

The `Custom Payload` option allows you to completely customize the webhook payload using [templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/). This gives you full control over the structure and content of the webhook request.

For detailed information about how to create and manage notification templates, refer to [notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/).

{{< admonition type="note" >}}

- When using Custom Payload, the [`Title` and `Message` fields](#optional-notification-settings) are ignored as the entire payload structure is determined by your template.
- Custom Payload is not yet [generally available](https://grafana.com/docs/release-life-cycle/#general-availability) in Grafana Cloud.

{{< /admonition >}}

| Option            | Description                                                                                                                                                          |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Payload Template  | [Notification template](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/) that defines the structure of the webhook payload. |
| Payload Variables | Key-value pairs that define additional variables available in the template under `.Vars.<variable_name>`.                                                            |

Example of a custom payload template that includes variables:

```
{
  "alert_name": "{{ .CommonLabels.alertname }}",
  "status": "{{ .Status }}",
  "environment": "{{ .Vars.environment }}",
  "custom_field": "{{ .Vars.custom_field }}"
}
```

### JSON Template Functions

When creating custom payloads, several template functions are available to help generate valid JSON structures. These include functions for creating dictionaries (`coll.Dict`), arrays (`coll.Slice`, `coll.Append`), and converting between JSON strings and objects (`data.ToJSON`, `data.JSON`).

For detailed information about these and other template functions, refer to [notification template functions](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#namespaced-functions).

Example using JSON helper functions:

{{< docs/shared lookup="alerts/example-custom-json-payload.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Notification template examples

Notification templates allow you to change the default notification messages.

You can modify the content and format of notification messages. For example, you can customize the content to show only specific information or adjust the format to suit a particular contact point, such as Slack or Email.

{{< admonition type="note" >}}
Avoid adding extra information about alert instances in notification templates, as this information is only visible in the notification message.

Instead, you should [use annotations or labels](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/) to add information directly to the alert, ensuring it's also visible in the alert state and alert history within Grafana. You can then print the new alert annotation or label in notification templates.
{{< /admonition >}}

This page provides various examples illustrating how to template common notification messages. For more details about notification templates, refer to:

- [Template notifications](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/)
- [Select, create, and preview a notification template](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/manage-notification-templates/)
- [Notification template reference](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/)

## Basic examples

Notification templates can access the [notification data](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#notification-data) using the dot (`.`). The following examples demonstrate some basic uses of the [template language](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/language/).

### Text formatting examples

Text formatting in notification templates varies by contact point type. Each platform has its own formatting syntax:

#### Slack formatting

For Slack contact points, you can use:

```go
{{ define "slack_formatted_message" -}}
*This text will be bold in Slack*
_This text will be italic in Slack_
Regular text without formatting
{{ end }}
```

Execute the template:

```go
{{ template "slack_formatted_message" . }}
```

```template_output
*This text will be bold in Slack*
_This text will be italic in Slack_
Regular text without formatting
```

#### Additional contact point types

{{< admonition type="note" >}}
Each contact point type has its own formatting requirements:

- Some support Markdown-style formatting
- Others use HTML tags
- Some may not support formatting at all

Check your specific contact point's documentation for supported formatting syntax.
{{< /admonition >}}

## Template logic examples

Beyond text formatting, you can use template logic to create dynamic notification content. The following examples demonstrate basic template operations:

### Conditional logic

To check if there are common labels (`.CommonLabels`) for all alerts in the notification, use `if`:

```go
{{ define "custom_message" -}}
{{ if .CommonLabels }}
Alerts have common labels
{{ else }}
There are no common labels
{{ end }}
{{ end }}
```

### Iterating over alerts

To iterate on the alerts in the notification and print a specific label, use `range` and `index`:

```go
{{ define "custom_message" -}}
{{ range .Alerts }}
The name of the alert is {{ index .Labels "alertname" }}
{{ end }}
{{ end }}
```

Alternatively, you can use the `.` notation to print the value of the key.

```go
{{ define "custom_message" -}}
{{ range .Alerts }}
The name of the alert is {{ .Labels.alertname }}
{{ end }}
{{ end }}
```

You can then use the template by passing the [notification data (dot `.`)](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#notification-data):

```go
{{ template "custom_message" . }}
```

```template_output
The name of the alert is InstanceDown

The name of the alert is CpuOverload
```

## Print alerts with summary and description

Here's an example that displays the summary and description annotations for each alert in the notification.

```go
{{ define "custom.alerts" -}}
{{ len .Alerts }} alert(s)
{{ range .Alerts -}}
  {{ template "alert.summary_and_description" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.summary_and_description" }}
  Summary: {{.Annotations.summary}}
  Status: {{ .Status }}
  Description: {{.Annotations.description}}
{{ end -}}
```

In this example:

- A template (`alert.summary_and_description`) is defined to print the `summary`, `status`, and `description` of one [alert](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#alert).
- The main template `custom.alerts` iterates the list of alerts (`.Alerts`) in [notification data](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#notification-data), executing the `alert.summary_and_description` template to print the details of each alert.

Execute the template by passing the dot (`.`):

```go
{{ template "custom.alerts" . }}
```

```template_output
2 alert(s)

  Summary: The database server db1 has exceeded 75% of available disk space.
  Status: firing
  Description: This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption.

  Summary: The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.
  Status: resolved
  Description: This alert fires when a web server responds with more 5xx errors than expected. This could be an issue with the web server or a backend service.
```

## Print firing and resolved alerts

The following example is similar to the previous one, but it separates firing and resolved alerts.

```go
{{ define "custom.firing_and_resolved_alerts" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
  {{ template "alert.summary_and_description" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
  {{ template "alert.summary_and_description" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.summary_and_description" }}
  Summary: {{.Annotations.summary}}
  Status: {{ .Status }}
  Description: {{.Annotations.description}}
{{ end -}}
```

Instead of `.Alerts`, the template accesses `.Alerts.Firing` and `.Alerts.Resolved` separately to print details for each alert.

Run the template by passing the dot (`.`):

```go
{{ template "custom.firing_and_resolved_alerts" . }}
```

```template_output
1 resolved alert(s)

  Summary: The database server db1 has exceeded 75% of available disk space.
  Status: resolved
  Description: This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption.

1 firing alert(s)

  Summary: The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.
  Status: firing
  Description: This alert fires when a web server responds with more 5xx errors than expected. This could be an issue with the web server or a backend service.
```

## Print common labels and annotations

This example displays only the labels and annotations that are common to all alerts in the notification.

```go
{{ define "custom.common_labels_and_annotations" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ len .Alerts.Firing }} firing alert(s)
Common labels: {{ len .CommonLabels.SortedPairs }}
{{ range .CommonLabels.SortedPairs -}}
- {{ .Name }} = {{ .Value }}
{{ end }}
Common annotations: {{ len .CommonAnnotations.SortedPairs }}
{{ range .CommonAnnotations.SortedPairs }}
- {{ .Name }} = {{ .Value }}
{{ end }}
{{ end -}}
```

Note that `.CommonAnnotations` and `.CommonLabels` are part of [notification data](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#notification-data).

Execute the template by passing the dot (`.`) as argument:

```go
{{ template "custom.common_labels_and_annotations" . }}
```

```template_output
1 resolved alert(s)
1 firing alert(s)
Common labels: 2
- grafana_folder = server_alerts
- team = server_admin

Common annotations: 0
```

## Print individual labels and annotations

This example displays all labels and annotations for each [alert](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#alert) in the notification.

```go
{{ define "custom.alert_labels_and_annotations" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
  {{ template "alert.labels_and_annotations" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
  {{ template "alert.labels_and_annotations" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.labels_and_annotations" }}
Alert labels: {{ len .Labels.SortedPairs }}
{{ range .Labels.SortedPairs -}}
- {{ .Name }} = {{ .Value }}
{{ end -}}
Alert annotations: {{ len .Annotations.SortedPairs }}
{{ range .Annotations.SortedPairs -}}
- {{ .Name }} = {{ .Value }}
{{ end -}}
{{ end -}}
```

In this example:

- The `custom.alert_labels_and_annotations` template iterates over the list of resolved and firing alerts, similar to previous examples. It then executes `alert.labels_and_annotations` for each alert.
- The `alert.labels_and_annotations` template prints all the alert labels and annotations by accessing `.Labels.SortedPairs` and `.Annotations.SortedPairs`.

Run the template by passing the dot (`.`):

```go
{{ template "custom.alert_labels_and_annotations" . }}
```

```template_output
1 resolved alert(s)

Alert labels: 4
- alertname = db_server_disk_space
- grafana_folder = server_alerts
- server = db1
- team = server_admin

Alert annotations: 2
- summary = The database server db1 has exceeded 75% of available disk space.
- description = This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption.

1 firing alert(s)

Alert labels: 4
- alertname = web_server_http_errors
- grafana_folder = server_alerts
- server = web1
- team = server_admin

Alert annotations: 2
- summary = The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.
- description = This alert fires when a web server responds with more 5xx errors than expected. This could be an issue with the web server or a backend service.
```

## Print URLs for runbook and alert data in Grafana

Note that the following example works only for Grafana-managed alerts. It displays some [alert data](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#alert) such as `DashboardURL`, `PanelURL`, and `SilenceURL`, which are exclusive to Grafana-managed alerts.

```go
{{ define "custom.alert_additional_details" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
  {{ template "alert.additional_details" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
  {{ template "alert.additional_details" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.additional_details" }}
- Dashboard: {{ .DashboardURL }}
- Panel: {{ .PanelURL }}
- AlertGenerator: {{ .GeneratorURL }}
- Silence: {{ .SilenceURL }}
- RunbookURL: {{ .Annotations.runbook_url}}
{{ end -}}
```

Pass the dot (`.`) to execute the template:

```go
{{ template "custom.alert_additional_details" . }}
```

```template_output
1 resolved alert(s)

- Dashboard: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395
- Panel: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395&viewPanel=31
- AlertGenerator: ?orgId=1
- Silence: https://example.com/alerting/silence/new
- RunbookURL: https://example.com/on-call/db_server_disk_space

1 firing alert(s)

- Dashboard: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395
- Panel: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395&viewPanel=31
- AlertGenerator: ?orgId=1
- Silence: https://example.com/alerting/silence/new
- RunbookURL: https://example.com/on-call/web_server_http_errors
```

## Print a notification title or subject

A title or subject provides a one-line summary of the notification content.

Here’s a basic example that displays the number of firing and resolved alerts in the notification.

```go
{{ define "custom_title" -}}
{{ if gt (.Alerts.Firing | len) 0 }}🚨 {{ .Alerts.Firing | len }} firing alerts. {{ end }}{{ if gt (.Alerts.Resolved | len) 0 }}✅ {{ .Alerts.Resolved | len }} resolved alerts.{{ end }}
{{ end -}}
```

Execute the template by passing the dot (`.`) as argument:

```go
{{ template "custom_title" . }}
```

```template_output
🚨 1 firing alerts. ✅ 1 resolved alerts.
```

The next example is a copy of the default title/subject template used in Grafana.

```go
{{ define "copy_of_default_title" -}}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ if gt (.Alerts.Resolved | len) 0 }}, RESOLVED:{{ .Alerts.Resolved | len }}{{ end }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}
{{ end }}
```

This is a more advanced example:

- Prints the number of firing and resolved alerts in the notification.
- Outputs `.GroupLabels`, the labels used to [group multiple alerts in one notification](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/).
- Prints `CommonLabels`, excluding labels in `.GroupLabels`.

Execute the template by passing the dot (`.`):

```go
{{ template "copy_of_default_title" . }}
```

```template_output
[FIRING:1, RESOLVED:1] api warning (sql_db)
```

## Print a link to a dashboard with time range

You can include a link to a dashboard or panel in your alert notifications. This is useful when the alert rule is created from a dashboard panel or monitors a target visualized in an existing dashboard.

Including a dashboard link in the notification helps responders quickly navigate to the relevant context for investigation.

Use one of the following methods to include a dashboard link with the correct time range in the alert notification:

1. You can [link the alert rule to a panel](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/link-alert-rules-to-panels/). This includes the dashboard and panel URLs via `{{.Alert.DashboardURL}}` and `{{.Alert.PanelURL}}`.

   ```go
   {{ define "custom.link_to_dashboard" -}}
   {{ range .Alerts -}}
     Dashboard: {{.DashboardURL}}
     Panel: {{ .PanelURL }}
   {{ end -}}
   {{ end -}}
   ```

   Run the template using:

   ```go
   {{ template "custom.link_to_dashboard" . }}
   ```

   ```template_output
   Dashboard: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395
   Panel: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395&viewPanel=31
   ```

   These URLs include a time range based on the alert’s timing:
   - `from`: One hour before the alert started.
   - `to`: The current time if the alert is firing, or the alert’s end time if resolved.

1. Alternatively, you can use a custom annotation to set the dashboard URL and build the full URL using the `from` and `to` query parameters derived from `{{.Alert.StartsAt}}` and `{{.Alert.EndsAt}}`.

   ```go
   {{ define "custom.my_dashboard_url_annotation" -}}
   {{ range .Alerts -}}

     {{/* StartsAt - 1h */}}
     {{- $from := (.StartsAt.Add -3600000000000).UnixMilli }}

     {{- $to := "" }}
     {{- if eq .Status "resolved" }}
        {{- $to = (.EndsAt).UnixMilli }}
     {{- else -}}
       {{/* Use current time if alert is firing */}}
       {{- $to = (time.Now).UnixMilli }}
     {{- end -}}

     Dashboard: {{.Annotations.MyDashboardURL}}?from={{$from}}&to={{$to}}
   {{ end }}
   {{ end }}
   ```

   To use this template, define a custom annotation named `MyDashboardURL` that contains the base dashboard URL without `from` and `to` parameters. For example: `http://localhost:3000/d/uiyahbsdaubsd`.

   Run the template using:

   ```go
   {{ template "custom.my_dashboard_url_annotation" . }}
   ```

   ```template_output
   Dashboard: http://localhost:3000/d/uiyahbsdaubsd?from=1740070380000&to=1740071880000
   ```

## Custom JSON payload

The [custom payload option](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier/#custom-payload) in the webhook contact point allows you to customize the payload of webhook notifications using a custom template.

The following example generates a custom JSON payload by executing other templates with `tmpl.Exec`, and using functions like `coll.Dict` and `data.ToJSON` to process and format JSON data.

{{< docs/shared lookup="alerts/example-custom-json-payload.md" source="grafana" version="<GRAFANA_VERSION>" >}}

```template_output
{
 "alerts": [
  {
   "endsAt": "0001-01-01T00:00:00Z",
   "labels": {
    "alertname": "InstanceDown",
    "grafana_folder": "Test Folder",
    "instance": "instance1"
   },
   "startsAt": "2025-04-21T10:19:46.179Z",
   "status": "firing"
  },
  {
   "endsAt": "2025-04-22T10:19:46.179Z",
   "labels": {
    "alertname": "CpuUsage",
    "grafana_folder": "Test Folder",
    "instance": "instance1"
   },
   "startsAt": "2025-04-22T06:19:46.179Z",
   "status": "resolved"
  }
 ],
 "allVariables": {},
 "commonAnnotations": {},
 "commonLabels": {
  "grafana_folder": "Test Folder",
  "instance": "instance1"
 },
 "externalURL": "http://localhost:3000/",
 "groupKey": "",
 "groupLabels": {
  "group_label": "group_label_value"
 },
 "message": "**Firing**\n\nValue: B=22, C=1\nLabels:\n - alertname = InstanceDown\n - grafana_folder = Test Folder\n - instance = instance1\nAnnotations:\n - summary = Instance instance1 has been down for more than 5 minutes\nSource: http://grafana.com/alerting/grafana/cdeqmlhvflz40f/view?orgId=1\nSilence: http://localhost:3000/alerting/silence/new?alertmanager=grafana\u0026matcher=alertname%3DInstanceDown\u0026matcher=grafana_folder%3DTest+Folder\u0026matcher=instance%3Dinstance1\u0026orgId=1\nDashboard: http://localhost:3000/d/dashboard_uid?from=1745227186179\u0026orgId=1\u0026to=1745317189058\nPanel: http://localhost:3000/d/dashboard_uid?from=1745227186179\u0026orgId=1\u0026to=1745317189058\u0026viewPanel=1\n\n\n**Resolved**\n\nValue: B=22, C=1\nLabels:\n - alertname = CpuUsage\n - grafana_folder = Test Folder\n - instance = instance1\nAnnotations:\n - summary = CPU usage above 90%\nSource: http://grafana.com/alerting/grafana/oZSMdGj7z/view?orgId=1\nSilence: http://localhost:3000/alerting/silence/new?alertmanager=grafana\u0026matcher=alertname%3DCpuUsage\u0026matcher=grafana_folder%3DTest+Folder\u0026matcher=instance%3Dinstance1\u0026orgId=1\nDashboard: http://localhost:3000/d/dashboard_uid?from=1745299186179\u0026orgId=1\u0026to=1745317186179\nPanel: http://localhost:3000/d/dashboard_uid?from=1745299186179\u0026orgId=1\u0026to=1745317186179\u0026viewPanel=1\n",
 "orgId": 1,
 "receiver": "TestReceiver",
 "state": "alerting",
 "status": "firing",
 "title": "[FIRING:1, RESOLVED:1] group_label_value (Test Folder instance1)",
 "truncatedAlerts": null,
 "version": "1"
}
```

# Use images in notifications

{{< admonition type="note" >}}
Grafana Cloud users can request this feature by [opening a support ticket in the Cloud Portal](/profile/org#support).
{{< /admonition >}}

Images in notifications help recipients of alert notifications better understand why an alert has fired or resolved by including a screenshot of the panel associated with the alert.

{{< admonition type="note" >}}
This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager.
{{< /admonition >}}

When an alert is fired or resolved, Grafana takes a screenshot of the panel associated with the alert. This is determined via the Dashboard UID and Panel ID annotations of the rule. Grafana cannot take a screenshot for alerts that are not associated with a panel.

Grafana takes at most two screenshots for each alert: once when the alert fires and again when the alert is resolved. Screenshots are not re-taken over the lifetime of the alert. Instead, you should open the panel in Grafana to follow the data in real time. In addition, depending on how alerts are grouped in your notification policies, Grafana might send a notification with many screenshots of the same panel. This happens because Grafana does not know how your alerts are grouped at the time a screenshot is taken, and so acts conservatively by taking a screenshot for every alert.

After a screenshot has been taken, Grafana can either upload it to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; upload the screenshot to its internal web server; or upload it to the service that is receiving the notification, such as Slack. Which option you should choose depends on how your Grafana is managed and which integrations you use. More information on this can be found in Requirements.

Refer to the table at the end of this page for a list of contact points and their support for images in notifications.

## Requirements

1. To use images in notifications, Grafana OSS and Enterprise must be set up to use the [Grafana image renderer](https://github.com/grafana/grafana-image-renderer/) service.

2. When a screenshot is taken, it is saved to the [data][paths] folder, even if Grafana is configured to upload screenshots to a cloud storage service. Grafana must have write-access to this folder, otherwise screenshots cannot be saved to disk and an error is logged for each failed screenshot attempt.

3. You should use a cloud storage service unless sending alerts to Discord, email, Pushover, Slack or Telegram. These integrations support either embedding screenshots in the email or attaching screenshots to the notification, while other integrations must link screenshots uploaded to a cloud storage bucket. If a cloud storage service has been configured then integrations that support both link screenshots from the cloud storage bucket instead of embedding or attaching screenshots to the notification.

4. If uploading screenshots to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; and accessing screenshots in the bucket requires authentication, logging into a VPN or corporate network; then image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN. If this happens, we recommend using [integrations which support uploading images](#supported-contact-points) or [disabling images in notifications](#configuration) altogether.

5. When uploading screenshots to a cloud storage service Grafana uses a random 20 character (30 characters for Azure Blob Storage) filename for each image. This makes URLs hard to guess but not impossible.

6. Grafana does not delete screenshots from cloud storage. We recommend configuring a retention policy with your cloud storage service to delete screenshots older than 1 month.

7. If Grafana is configured to upload screenshots to its internal web server, and accessing Grafana requires logging into a VPN or corporate network; image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN. If this happens, we recommend using [integrations which support uploading images](#supported-contact-points) or [disabling images in notifications](#configuration) altogether.

8. Grafana does not delete screenshots uploaded to its internal web server. To delete screenshots from `static_root_path/images/attachments` after a certain amount of time, we recommend setting up a CRON job.

9. Note you cannot adjust the number and size of images or their placement in notifications.

## Configuration

{{< admonition type="note" >}}
Grafana Cloud users can request this feature by [opening a support ticket in the Cloud Portal](/profile/org#support).
{{< /admonition >}}

Having set up Grafana to use the image renderer service, set `capture` in `[unified_alerting.screenshots]` to `true`:

    # Enable screenshots in notifications. You must have either installed the Grafana image rendering
    # plugin, or set up Grafana to use a remote rendering service.
    # For more information on configuration options, refer to [rendering].
    capture = false

If screenshots should be uploaded to cloud storage then `upload_external_image_storage` should also be set to `true`:

    # Uploads screenshots to the local Grafana server or remote storage such as Azure, S3 and GCS. Please
    # see [external_image_storage] for further configuration options. If this option is false, screenshots
    # are persisted to disk for up to temp_data_lifetime.
    upload_external_image_storage = false

Restart Grafana for the changes to take effect.

## Advanced configuration

We recommend that `max_concurrent_screenshots` is less than or equal to `concurrent_render_request_limit`. The default value for both `max_concurrent_screenshots` and `concurrent_render_request_limit` is `5`:

    # The maximum number of screenshots that can be taken at the same time. This option is different from
    # concurrent_render_request_limit as max_concurrent_screenshots sets the number of concurrent screenshots
    # that can be taken at the same time for all firing alerts where as concurrent_render_request_limit sets
    # the total number of concurrent screenshots across all Grafana services.
    max_concurrent_screenshots = 5

## Supported contact points

Grafana supports a wide range of contact points with varied support for images in notifications. The table below shows the list of all contact points supported in Grafana and their support for uploading screenshots to the receiving service and referencing screenshots that have been uploaded to a cloud storage service.

| Name                    | Upload from disk                                           | Reference from cloud storage                             |
| ----------------------- | ---------------------------------------------------------- | -------------------------------------------------------- |
| DingDing                | No                                                         | No                                                       |
| Discord                 | Yes (Maximum of 10 per notification)                       | Yes (Maximum of 10 per notification)                     |
| Email                   | Yes (Embedded in the email)                                | Yes                                                      |
| Google Chat             | No                                                         | Yes                                                      |
| Kafka                   | No                                                         | No                                                       |
| Line                    | No                                                         | No                                                       |
| MQTT                    | No                                                         | No                                                       |
| Microsoft Teams         | No                                                         | Yes                                                      |
| Opsgenie                | No                                                         | Yes                                                      |
| PagerDuty               | No                                                         | Yes                                                      |
| Prometheus Alertmanager | No                                                         | No                                                       |
| Pushover                | Yes (Maximum of 1 per notification)                        | No                                                       |
| Sensu Go                | No                                                         | No                                                       |
| Slack                   | Yes (when using Bot tokens, maximum of 5 per notification) | Yes (when using webhooks, maximum of 1 per notification) |
| Telegram                | Yes                                                        | No                                                       |
| Threema                 | No                                                         | No                                                       |
| VictorOps               | No                                                         | No                                                       |
| Webhook                 | No                                                         | Yes                                                      |

## Limitations

- This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager.
- This feature is not supported when using custom templates in email notifications.
- A number of contact points support at most one image per notification. In this case, just the first image is either uploaded to the receiving service or referenced from cloud storage per notification.
- When multiple alerts are sent in a single notification a screenshot might be included for each alert. The order the images are shown is random.
- If uploading screenshots to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; and accessing screenshots in the bucket requires authentication, logging into a VPN or corporate network; image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN.

## Troubleshooting

If Grafana has been set up to send images in notifications, however notifications are still being received without them, follow the troubleshooting steps below:

1. Check that images in notifications has been set up as per the instructions.
2. Enable debug logging in Grafana and look for logs with the logger `ngalert.image`.
3. If the alert is not associated with a dashboard there are logs for `Cannot take screenshot for alert rule as it is not associated with a dashboard`.
4. If the alert is associated with a dashboard, but no panel in the dashboard, there are logs for `Cannot take screenshot for alert rule as it is not associated with a panel`.
5. If images cannot be taken because of mis-configuration or an issue with image rendering, there are logs for `Failed to take an image` including the Dashboard UID, Panel ID, and the error message.
6. Check that the contact point supports images in notifications and whether it supports uploading images to the receiving service or referencing images that have been uploaded to a cloud storage service.

## Monitor

Grafana provides the following metrics to observe the performance and failure rate of images in notifications.
For example, if a screenshot could not be taken within the expected time (10 seconds) then the counter `grafana_screenshot_failures_total` is updated.

- `grafana_alerting_image_cache_hits_total`
- `grafana_alerting_image_cache_misses_total`
- `grafana_screenshot_duration_seconds`
- `grafana_screenshot_failures_total`
- `grafana_screenshot_successes_total`
- `grafana_screenshot_upload_failures_total`
- `grafana_screenshot_upload_successes_total`

# Alerting template language

Notification templates and alert rule templates, such as annotations and labels, both use the Go template language, [text/template](https://pkg.go.dev/text/template).

Both types of templates can use the same keywords, functions, and comparison operators of the Go template language, such as `range`, `if`, `and`, `index`, `eq`, and more.

However, it's important to note that because notifications and alert rules operate in distinct contexts, some additional variables and functions are only available for either notification or alert rule templates. Refer to:

- [Annotation and label template reference](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/)
- [Notification template reference](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/)

This documentation provides an overview of the functions and operators of the Go template language that are available for both notification and alert rule templates.

## Print

To print the value of something, use `{{` and `}}`. You can print the value of a [variable](#variables), a field of a variable, the result of a function, or the value of dot.

```
{{ $values }}
{{ $values.A.Value }}
{{ humanize 1000.0 }}
{{ .Alerts }}
```

## Dot

In `text/template`, there is a special cursor called dot, written as `.`. You can think of this cursor as a variable whose value changes depending on where in the template it is used.

At the start of notification templates, dot (`.`) refers to [Notification Data](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#notification-data).

```
{{ .Alerts }}
```

In annotation and label templates, dot (`.`) is initialized with all alert data. It’s recommended to use the [`$labels` and `$values` variables](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/#variables) instead to directly access the alert labels and query values.

{{< admonition type="note" >}}
Dot (`.`) might refer to something else when used in a [range](#range), a [with](#with), or when writing [templates](#templates) used in other templates.
{{< /admonition >}}

[//]: <> (The above section is not included in the shared file because `refs` links are not supported in shared files.)

{{< docs/shared lookup="alerts/template-language.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Manage notification templates

In contact points, you can select notification templates to customize the notification messages sent.

By default, Grafana provides a template for the notification title (`{{define "default.title"}}`) and a template for the notification message (`{{define "default.message"}}`). Both default templates display common alert details.

You can also create custom templates to customize the content and format of notification messages, which can then be applied to one or more contact points. In Grafana, a custom notification template is created within a notification template group.

This documentation provides step-by-step instructions for selecting templates in contact points, previewing templates, and creating notification template groups using the Grafana UI.

## Select a notification template for a contact point

To add an existing notification template to your contact point, complete the following steps.

1. Click an existing contact point or create a new one.
1. In **Optional settings**, click any field that contains templates.

   For example, if you are creating an email contact point integration, click **Message** or **Subject**.

1. Click **Edit**.
   A dialog box opens where you can select notification templates.
1. Click **Select notification template** or **Enter custom message** to customize a template or message
   - You can select an existing notification template and [preview](#preview-a-notification-template) it using the default payload.
   - You can also copy the notification template and use it in the **Enter custom message** tab.

1. Click **Save contact point**.

You can create custom notification templates using the **Enter custom message** tab. For reusable and consistent notifications, a best practice is to custom notification templates as described in the following section.

## Create a notification template and notification template group

Create notification templates to customize notification messages and reuse them in contact points.

In Grafana, custom notification templates (`{{define "<NAME>"}}`) are created within a notification template group, allowing you to test and implement multiple templates together.

Your notification template name (`{{define "<NAME>"}}`) must be unique. You cannot have two templates with the same name in the same notification template group or in different notification template groups. Therefore, avoid using names already defined as default templates, such as: `__subject`, `__text_values_list`, `__text_alert_list`, `default.title` and `default.message`.

In Grafana Cloud, you can create up to 30 notification templates by default. Template changes can take 1 to 2 minutes to take effect while a background job synchronizes them with storage.

To create a notification template in Grafana, complete the following steps.

1. Click **Alerts & IRM** -> **Contact points**.
1. Click the **Notification Templates** tab and then **+ New notification template**.

1. Enter a name for the notification template group.

1. Write the content of the template in the content field.

1. Save your changes.

   If `{{ define }}` is not included in the content, `{{ define "<NOTIFICATION_TEMPLATE_NAME>" }}` and `{{ end }}` is automatically added to the start and end.

To create a notification template group that contains more than one notification template, complete the following steps.

1. Click **+ New notification template**.

1. Enter a name for the notification template group.

1. Write each template in the Content field, including `{{ define "<NOTIFICATION_TEMPLATE_NAME>" }}` and `{{ end }}` at the start and end of each template.

1. Save your changes.

For more details on how to write notification templates, refer to the [template reference](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/) and [examples](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/examples/).

## Preview a notification template

Preview how your notification templates should look before using them in your contact points, helping you understand the result of the template you are creating as well as enabling you to fix any errors before saving it.

{{< admonition type="note" >}}
Notification template preview is only for Grafana Alertmanager.
{{< /admonition >}}

To preview your notification templates:

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Templates** tab.
1. Click **+ New notification template** or edit an existing template group.
1. Add or update your template content.

   Default data is provided and you can add or edit alert data to it as well as alert instances. You can add alert data directly in the Payload data window itself or click **Select alert instances** or **Add custom alerts**.

1. Optional: To add alert data from existing alert instances:

   a. Click **Select alert instances**.

   b. Hover over the alert instances to view more information on each alert instance.

   c. Click **Confirm** to add the alert instance(s) to the payload.

1. Optional: To add alert data using the Alert data editor, click **Add custom data:**

   a. Add annotations, custom labels and/or set a dashboard or a panel.

   b. Toggle Firing/resolved depending on whether you want to add firing or resolved alerts to your notification.

   c. Click **Add alert data**.

   d. Click **Refresh preview** to see what your template content should look like and the corresponding payload data.

   If there are any errors in your template, they are displayed in the Preview and you can correct them before saving.

1. Save your changes.

# Notification template reference

By default, Grafana provides predefined templates to format notification messages.

You can also customize your notifications with custom templates, which are based on the [Go template language](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/language/).

This documentation lists the data available for use in notification templates.

## Notification Data

In notification templates, dot (`.`) is initialized with the following data:

| Name                | Type              | Description                                                                                             |
| ------------------- | ----------------- | ------------------------------------------------------------------------------------------------------- |
| `Receiver`          | string            | The name of the contact point sending the notification                                                  |
| `Status`            | string            | The status is `firing` if at least one alert is firing, otherwise `resolved`.                           |
| `Alerts`            | [][Alert](#alert) | List of all firing and resolved alerts in this notification.                                            |
| `Alerts.Firing`     | [][Alert](#alert) | List of all firing alerts in this notification.                                                         |
| `Alerts.Resolved`   | [][Alert](#alert) | List of all resolved alerts in this notification.                                                       |
| `GroupLabels`       | [KV](#kv)         | The labels that group these alerts in this notification based on the `Group by` option.                 |
| `CommonLabels`      | [KV](#kv)         | The labels common to all alerts in this notification.                                                   |
| `CommonAnnotations` | [KV](#kv)         | The annotations common to all alerts in this notification.                                              |
| `ExternalURL`       | string            | A link to Grafana, or the Alertmanager that sent this notification if using an external Alertmanager.   |
| `GroupKey`          | string            | The key used to identify this alert group.                                                              |
| `TruncatedAlerts`   | integer           | The number of alerts, if any, that were truncated in the notification. Supported by Webhook and OnCall. |

It's important to remember that [a single notification can group multiple alerts](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/) to reduce the number of alerts you receive. `Alerts` is an array that includes all the alerts in the notification.

Here's an example that prints all available notification data from dot (`.`):

```go
{{ define "custom_template" }}
  {{ .Receiver }}
  {{ .Status }}
  There are {{ len .Alerts }} alerts
  There are {{ len .Alerts.Firing }} firing alerts
  There are {{ len .Alerts.Resolved }} resolved alerts
  {{ .GroupLabels }}
  {{ .CommonLabels }}
  {{ .CommonAnnotations }}
  {{ .ExternalURL }}
{{ end }}
```

You can execute this template by passing the dot (`.`):

```go
{{ template "custom_template" . }}
```

## Alert

`Alert` contains data for an individual alert:

| Name           | Type          | Description                                                                                                                                                                                                                         |
| -------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Status`       | string        | Firing or resolved.                                                                                                                                                                                                                 |
| `Labels`       | [KV](#kv)     | The labels associated with this alert. <br/> It includes all [types of labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#label-types), but only query labels used in the alert condition. |
| `Annotations`  | [KV](#kv)     | The annotations for this alert.                                                                                                                                                                                                     |
| `StartsAt`     | [Time](#time) | The time the alert fired                                                                                                                                                                                                            |
| `EndsAt`       | [Time](#time) | Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received.                                                                                      |
| `GeneratorURL` | string        | A link to Grafana, or the source of the alert if using an external alert generator.                                                                                                                                                 |
| `Fingerprint`  | string        | A unique string that identifies the alert.                                                                                                                                                                                          |

Grafana-managed alerts include these additional properties:

| Name           | Type      | Description                                                                                                                                          |
| -------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DashboardURL` | string    | A link to the Grafana Dashboard if the alert has a Dashboard UID annotation, with time range from `1h` before alert start to end (or now if firing). |
| `PanelURL`     | string    | A link to the panel if the alert has a Panel ID annotation, with time range from `1h` before alert start to end (or now if firing).                  |
| `SilenceURL`   | string    | A link to silence the alert.                                                                                                                         |
| `Values`       | [KV](#kv) | The values of expressions used to evaluate the alert condition. Only relevant values are included.                                                   |
| `ValueString`  | string    | A string that contains the labels and value of each reduced expression in the alert.                                                                 |
| `OrgID`        | integer   | The ID of the organization that owns the alert.                                                                                                      |

This example iterates over the list of firing and resolved alerts (`.Alerts`) in the notification and prints the data for each alert:

```go
{{ define "custom_template" }}
{{ range .Alerts }}
  {{ .Status }}
  {{ .Labels }}
  {{ .Annotations }}
  {{ .StartsAt }}
  {{ .EndsAt }}
  {{ .GeneratorURL }}
  {{ .Fingerprint }}

  {{/* Only available for Grafana-managed alerts */}}
  {{ .DashboardURL }}
  {{ .PanelURL }}
  {{ .SilenceURL }}
  {{ .Values }}
  {{ .ValueString }}
{{ end }}
{{ end }}
```

You can run this template by passing the dot (`.`):

```go
{{ template "custom_template" . }}
```

## KV

`KV` is a set of key value pairs, where each key and value is a string.

Similarly to accessing variable properties, you can use `.` to retrieve the value of a value. For example:

```go
{{ define "custom_template" }}
  {{ .CommonLabels.grafana_folder }}
{{ end }}
```

If a KV happens to contain numbers or bools then these are string representations of the numeric or boolean value.

Additionally, KV provides methods to sort the pairs, remove keys, and iterate over just the keys or values:

| Method name | Description                                    | Arguments | Returns   |
| ----------- | ---------------------------------------------- | --------- | --------- |
| SortedPairs | Returns a sorted list of key/value pairs.      |           | Pairs     |
| Remove      | Returns a copy of the KV with the keys removed | []string  | [KV](#kv) |
| Names       | Return the names of the label names            |           | []string  |
| Values      | Return a list of the values                    |           | []string  |

Here's an example of using these methods:

```go
{{ define "custom_template" }}
  {{ .CommonLabels.SortedPairs }}
  {{ .CommonLabels.Names }}
  {{ .CommonLabels.Values }}
  {{ .CommonLabels.Remove (stringSlice "grafana_folder") }}
{{ end }}
```

## Time

Some template functions and properties return a `Time` object, which refers to the [type `Time`](https://pkg.go.dev/time#Time) in Go's time package.

When accessing a `Time` object, you can use various [`Time` functions](https://pkg.go.dev/time#Time) in your templates as follows.

```go
{{ define "custom_template" }}
  {{ range .Alerts }}
    {{ .StartsAt  }}
    {{ .StartsAt.Add 6000000000000  }}
    {{ .StartsAt.Add -6000000000000  }}
    {{ .StartsAt.AddDate 1 0 0  }}
    {{ .StartsAt.Year   }}/{{ .StartsAt.Month   }}/{{ .StartsAt.Day   }}
    {{ .StartsAt.Hour   }}:{{ .StartsAt.Minute   }}:{{ .StartsAt.Second   }}
    {{ .StartsAt.YearDay   }}-{{ .StartsAt.Weekday   }}
    {{ .StartsAt.Unix }} {{ .StartsAt.UnixMilli }}
  {{ end}}
{{ end }}
```

## Functions

Functions can perform actions in templates such as transforming or formatting data.

Note that the [functions provided by Go's template language](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/language/#functions), such as `index`, `and`, `printf`, and `len`, are available, along with many others.

In addition, the following functions are also available for templating notifications:

| Name           | Arguments                  | Returns       | Description                                                                                                                                                                                                      |
| -------------- | -------------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `title`        | string                     | string        | Capitalizes the first character of each word.                                                                                                                                                                    |
| `toUpper`      | string                     | string        | Returns all text in uppercase.                                                                                                                                                                                   |
| `toLower`      | string                     | string        | Returns all text in lowercase.                                                                                                                                                                                   |
| `trimSpace`    | string                     | string        | Removes leading and trailing white spaces.                                                                                                                                                                       |
| `match`        | pattern, text              | boolean       | Matches the text against a regular expression pattern.                                                                                                                                                           |
| `reReplaceAll` | pattern, replacement, text | string        | Replaces text matching the regular expression.                                                                                                                                                                   |
| `join`         | sep string, s []string     | string        | Concatenates the elements of s to create a single string. The separator string sep is placed between elements in the resulting string.                                                                           |
| `safeHtml`     | string                     | string        | Marks string as HTML not requiring auto-escaping.                                                                                                                                                                |
| `stringSlice`  | ...string                  | string        | Returns the passed strings as a slice of strings. auto-escaping.                                                                                                                                                 |
| `date`         | string, [Time](#time)      | string        | Format a time in the specified format. For format options, refer to [Go's time format documentation](https://pkg.go.dev/time#pkg-constants).                                                                     |
| `tz`           | string, [Time](#time)      | [Time](#time) | Returns the time in the specified timezone, such as `Europe/Paris`. For available timezone options, refer to the [list of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). |

Here's an example using some functions to format text:

```go
{{ define "custom_template" }}
  {{ title "hello, world!" }}
  {{ toUpper "Hello, world!" }}
  {{ toLower "Hello, world!" }}
  {{ trimSpace "Hello, world!" }}
  {{ match "a.*" "abc" }}
  {{ reReplaceAll "localhost:(.*)" "example.com:$1" "localhost:8080" }}
  {{ join "-" (stringSlice "a" "b" "c") }}
  {{ safeHtml "<b>Text</b>"}}
  {{ stringSlice "a" "b" "c" }}
{{ end }}
```

`date` and `tz` can format times. For example, to print the time an alert fired in the format `15:04:05 MST`:

```go
{{ define "custom_template" }}
  {{ range .Alerts }}
    {{ .StartsAt | date "15:04:05 MST" }}
  {{ end}}
{{ end }}
```

You can then use `tz` to change the timezone from UTC to local time, such as `Europe/Paris`.

```go
{{ define "custom_template" }}
  {{ range .Alerts }}
    {{ .StartsAt | tz "Europe/Paris" }}
    {{ .StartsAt | tz "Europe/Paris" | date "15:04:05 MST" }}
  {{ end}}
{{ end }}
```

```template-output
2024-10-30 21:01:45.227 +0100 CET
21:01:45 CET
```

## Namespaced Functions

{{< admonition type="note" >}}

Namespaced Functions are not yet [generally available](https://grafana.com/docs/release-life-cycle/#general-availability) in Grafana Cloud.

{{< /admonition >}}

In addition to the top-level functions, the following namespaced functions are also available:

### Collection Functions

| Name          | Arguments                  | Returns | Description                                                                                                                                                                      |
| ------------- | -------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `coll.Dict`   | key string, value any, ... | map     | Creates a map with string keys from key/value pairs. All keys are converted to strings. If an odd number of arguments is provided, the last key will have an empty string value. |
| `coll.Slice`  | ...any                     | []any   | Creates a slice (array/list) from the provided arguments. Useful for creating lists that can be iterated over with `range`.                                                      |
| `coll.Append` | value any, list []any      | []any   | Creates a new list by appending a value to the end of an existing list. Does not modify the original list.                                                                       |

Example using collection functions:

```go
{{ define "collection.example" }}
{{- /* Create a dictionary of alert metadata */ -}}
{{- $metadata := coll.Dict
    "severity" "critical"
    "team" "infrastructure"
    "environment" "production"
-}}

{{- /* Create a slice of affected services */ -}}
{{- $services := coll.Slice "database" "cache" "api" -}}

{{- /* Append a new service to the list */ -}}
{{- $services = coll.Append "web" $services -}}

{{- /* Use the collections in a template */ -}}
Affected Services: {{ range $services }}{{ . }},{{ end }}

Alert Metadata:
{{- range $k, $v := $metadata }}
  {{ $k }}: {{ $v }}
{{- end }}
{{ end }}
```

Output:

```
Affected Services: database,cache,api,web,

Alert Metadata:
  environment: production
  severity: critical
  team: infrastructure
```

### Data Functions

| Name                | Arguments              | Returns | Description                                                                                                                      |
| ------------------- | ---------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `data.JSON`         | jsonString string      | any     | Parses a JSON string into an object that can be manipulated in the template. Works with both JSON objects and arrays.            |
| `data.ToJSON`       | obj any                | string  | Serializes any object (maps, arrays, etc.) into a JSON string. Useful for creating webhook payloads.                             |
| `data.ToJSONPretty` | indent string, obj any | string  | Creates an indented JSON string representation of an object. The first argument specifies the indentation string (e.g., spaces). |

Example using data functions:

```go
{{ define "data.example" }}
{{- /* First, let's create some alert data as a JSON string */ -}}
{{ $jsonString := `{
  "service": {
    "name": "payment-api",
    "environment": "production",
    "thresholds": {
      "error_rate": 5,
      "latency_ms": 100
    }
  }
}` }}

{{- /* Parse the JSON string into an object we can work with */ -}}
{{ $config := $jsonString | data.JSON }}

{{- /* Create a new alert payload */ -}}
{{ $payload := coll.Dict
    "service" $config.service.name
    "environment" $config.service.environment
    "status" .Status
    "errorThreshold" $config.service.thresholds.error_rate
}}

{{- /* Output the payload in different JSON formats */ -}}
Compact JSON: {{ $payload | data.ToJSON }}

Pretty JSON with 2-space indent:
{{ $payload | data.ToJSONPretty "  " }}
{{ end }}
```

Output:

```
Compact JSON: {"environment":"production","errorThreshold":5,"service":"payment-api","status":"resolved"}

Pretty JSON with 2-space indent:
{
  "environment": "production",
  "errorThreshold": 5,
  "service": "payment-api",
  "status": "resolved"
}
```

### Template Functions

| Name          | Arguments                    | Returns | Description                                                                                                                                  |
| ------------- | ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `tmpl.Exec`   | name string, [context any]   | string  | Executes a named template and returns the result as a string. Similar to the `template` action but allows for post-processing of the result. |
| `tmpl.Inline` | template string, context any | string  | Renders a string as a template.                                                                                                              |

```go
{{ define "template.example" -}}
{{ coll.Dict
    "info" (tmpl.Exec `info` . | data.JSON)
    "severity" (tmpl.Inline `{{ print "critical" | toUpper }}` . )
    | data.ToJSONPretty " "}}
{{- end }}

{{- /* Define a sub-template */ -}}
{{ define "info" -}}
{{coll.Dict
    "team" "infrastructure"
    "environment" "production" | data.ToJSON }}
{{- end }}
```

Output:

```json
{
  "info": {
    "environment": "production",
    "team": "infrastructure"
  },
  "severity": "CRITICAL"
}
```

### Time Functions

| Name       | Arguments | Returns | Description                                                                                                  |
| ---------- | --------- | ------- | ------------------------------------------------------------------------------------------------------------ |
| `time.Now` |           | Time    | Returns the current local time as a time.Time object. Can be formatted using Go's time formatting functions. |

Example using time functions:

```go
{{ define "time.example" }}
{{- /* Get current time in different formats */ -}}
Current Time (UTC): {{ (time.Now).UTC.Format "2006-01-02 15:04:05 MST" }}
Current Time (Local): {{ (time.Now).Format "Monday, January 2, 2006 at 15:04:05" }}

{{- /* Compare alert time with current time */ -}}
{{ $timeAgo := (time.Now).Sub .StartsAt }}
Alert fired: {{ $timeAgo }} ago
{{ end }}
```

Output:

```
Current Time (UTC): 2025-03-08 18:14:27 UTC
Current Time (Local): Saturday, March 8, 2025 at 14:14:27
Alert fired: 25h49m32.78574723s ago
```

## Differences with annotation and label templates

In the alert rule, you can also template annotations and labels to include additional information. For example, you might add a `summary` annotation that displays the query value triggering the alert.

Annotation and label templates add relevant information to individual alert instances, while notification templates inform about a group of alert instances.

Since both types of templates operate in distinct contexts, the [functions and variables available in annotation and label templates](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/) differ from those used in notification templates.

# Example of dynamic labels in alert instances

Labels are essential for scaling your alerting setup. They define metadata like `severity`, `team`, `category`, or `environment`, which you can use for alert routing.

A label like `severity="critical"` can be set statically in the alert rule configuration, or dynamically based on a query value such as the current free disk space. Dynamic labels **adjust label values at runtime**, allowing you to reuse the same alert rule across different scenarios.

This example shows how to define dynamic labels based on query values, along with key behavior to keep in mind when using them.

First, it's important to understand how Grafana Alerting treats [labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#labels).

## Alert instances are defined by labels

Each alert rule creates a separate alert instance for every unique combination of labels.

This is called [multi-dimensional alerts](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/): one rule, many instances—**one per unique label set**.

For example, a rule that queries CPU usage per host might return multiple series (or dimensions):

- `{alertname="ServerHighCPU", instance="prod-server-1" }`
- `{alertname="ServerHighCPU", instance="prod-server-2" }`
- `{alertname="ServerHighCPU", instance="prod-server-3" }`

Each unique label combination defines a distinct alert instance, with its own evaluation state and potential notifications.

The full label set of an alert instance can include:

- Labels from the query result (e.g., `instance`)
- Auto-generated labels (e.g., `alertname`)
- User-defined labels from the rule configuration

## User-defined labels

As shown earlier, alert instances automatically include labels from the query result, such as `instance` or `job`. To add more context or control alert routing, you can define _user-defined labels_ in the alert rule configuration:

{{< figure src="/media/docs/alerting/example-dynamic-labels-edit-labels-v3.png" max-width="750px" alt="Edit labels UI in the alert rule configuration." >}}

User-defined labels can be either:

- **Fixed labels**: These have the same value for every alert instance. They are often used to include common metadata, such as team ownership.

- **Templated labels**: These calculate their values based on the query result at evaluation time.

## Templated labels

Templated labels evaluate their values dynamically, based on the query result. This allows the label value to vary per alert instance.

Use templated labels to inject additional context into alerts. To learn about syntax and use cases, refer to [Template annotations and labels](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/).

You can define templated labels that produce either:

- A fixed value per alert instance.
- A dynamic value per alert instance that changes based on the last query result.

### Fixed values per alert instance

You can use a known label value to enrich the alert with additional metadata not present in existing labels. For example, you can map the `instance` label to an `env` label that represents the deployment environment:

```go
{{- if eq $labels.instance "prod-server-1" -}}production
{{- else if eq $labels.instance "stag-server-1" -}}staging
{{- else -}}development
{{- end -}}
```

This produces alert instances like:

- `{alertname="ServerHighCPU", instance="prod-server-1", env="production"}`
- `{alertname="ServerHighCPU", instance="stag-server-1", env="staging"}`

In this example, the `env` label is fixed for each alert instance and does not change during its lifecycle.

### Dynamic values per alert instance

You can define a label whose value depends on the numeric result of a query—mapping it to a predefined set of options. This is useful for representing `severity` levels within a single alert rule.

Instead of defining three separate rules like:

- _CPU ≥ 90_ → `severity=critical`
- _CPU ≥ 80_ → `severity=warning`
- _CPU ≥ 70_ → `severity=minor`

You can define a single rule and assign `severity` dynamically using a template:

```go
{{/* $values.B.Value refers to the numeric result from query B */}}
{{- if gt $values.B.Value 90.0 -}}critical
{{- else if gt $values.B.Value 80.0 -}}warning
{{- else if gt $values.B.Value 70.0 -}}minor
{{- else -}}none
{{- end -}}
```

This pattern lets you express multiple alerting scenarios in a single rule, while still routing based on the `severity` label value.

## Example overview

In the previous severity template, you can set the alert condition to `$B > 70` to prevent firing when `severity=none`, and then use the `severity` label to route distinct alert instances to different contact points.

For example, configure a [notification policy](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/) that matches `alertname="ServerHighCPU"` with the following children policies:

- `severity=critical` → escalate to an incident response and management solution (IRM).
- `severity=warning` → send to the team's Slack channel.
- `severity=minor` → send to a non-urgent queue or log-only dashboard.

The resulting alerting flow might look like this:

| Time | $B query | Alert instance                                     | Routed to            |
| :--- | :------- | :------------------------------------------------- | :------------------- |
| t1   | 65       | `{alertname="ServerHighCPU", severity="none"}`     | `Not firing`         |
| t2   | 75       | `{alertname="ServerHighCPU", severity="minor"}`    | Non-urgent queue     |
| t3   | 85       | `{alertname="ServerHighCPU", severity="warning"}`  | Team Slack channel   |
| t4   | 95       | `{alertname="ServerHighCPU", severity="critical"}` | IRM escalation chain |

This alerting setup allows you to:

- Use a single rule for multiple severity levels.
- Route alerts dynamically using the label value.
- Simplify alert rule maintenance and avoid duplication.

However, dynamic labels can introduce unexpected behavior when label values change. The next section explains this.

## Caveat: a label change affects a distinct alert instance

Remember: **alert instances are defined by their labels**.

If a dynamic label changes between evaluations, this new value affects a separate alert instance.

Here's what happens if `severity` changes from `minor` to `warning`:

1. The instance with `severity="minor"` disappears → it becomes a missing series.
1. A new instance with `severity="warning"` appears → it starts from scratch.
1. After two evaluations without data, the `minor` instance is **resolved and evicted**.

Here’s a sequence example:

| Time | Query value | Instance `severity="none"` | Instance `severity="minor"` | Instance `severity="warning"` |
| :--- | :---------- | :------------------------- | :-------------------------- | :---------------------------- |
| t0   |             |                            |                             |                               |
| t1   | 75          |                            | 🔴 📩                       |                               |
| t2   | 85          |                            | ⚠️ MissingSeries            | 🔴 📩                         |
| t3   | 85          |                            | ⚠️ MissingSeries            | 🔴                            |
| t4   | 50          | 🟢                         | 📩 Resolved and evicted     | ⚠️ MissingSeries              |
| t5   | 50          | 🟢                         |                             | ⚠️ MissingSeries              |
| t6   | 50          | 🟢                         |                             | 📩 Resolved and evicted       |

Learn more about this behavior in [Stale alert instances](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/).

In this example, the `minor` and `warning` alerts likely represent the same underlying issue, but Grafana treats them as distinct alert instances. As a result, this scenario generates two firing notifications and two resolved notifications, one for each instance.

This behavior is important to keep in mind when dynamic label values change frequently.

It can lead to multiple notifications firing and resolving in short intervals, resulting in **noisy and confusing notifications**.

## Try it with TestData

You can replicate this scenario using the [TestData data source](/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/) to simulate an unstable signal—like monitoring a noisy sensor.

This setup reproduces label flapping and shows how dynamic label values affect alert instance behavior.

1. Add the **TestData** data source through the **Connections** menu.
1. Create an alert rule.

   Navigate to **Alerting** → **Alert rules** and click **New alert rule**.

1. Simulate a query (`$A`) that returns a noisy signal.

   Select **TestData** as the data source and configure the scenario.
   - Scenario: Random Walk
   - Series count: 1
   - Start value: 51
   - Min: 50, Max: 100
   - Spread: 100 (ensures large changes between consecutive data points)

1. Add an expression.
   - Type: Reduce
   - Input: A
   - Function: Last (to get the most recent value)
   - Name: B

1. Define the alert condition.

   Use a threshold like `$B >= 50` (it always fires).

1. Click **Edit Labels** to add a dynamic label.

   Create a new label `severity` and set its value to the following:

   ```go
   {{/* $values.B.Value refers to the numeric result from query B */}}
   {{- if gt $values.B.Value 90.0 -}}P1
   {{- else if gt $values.B.Value 80.0 -}}P2
   {{- else if gt $values.B.Value 70.0 -}}P3
   {{- else if gt $values.B.Value 60.0 -}}P4
   {{- else if gt $values.B.Value 50.0 -}}P5
   {{- else -}}none
   {{- end -}}
   ```

1. Set evaluation behavior.

   Set a short evaluation interval (e.g., `10s`) to observe quickly label flapping and alert instance transitions in the history.

1. Preview alert routing to verify the label template.

   In **Configure notifications**, toggle **Advanced options**.  
   Click **Preview routing** and check the value of the `severity` label:

   {{< figure src="/media/docs/alerting/example-dynamic-labels-preview-label.png" max-width="750px" caption="Preview routing multiple times to verify how label values change over time." >}}

1. Observe alert state changes.

   Click **Save rule and exit**, and open the [alert history view](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-state-history/) to see how changes in `severity` affect the state of distinct alert instances.

   {{< figure src="/media/docs/alerting/example-dynamic-labels-alert-history-page.png" max-width="750px" caption="You can find multiple transitions over time as the label value fluctuates." >}}

   {{< admonition type="tip" >}}
   You can explore this **[alerting example in Grafana Play](https://play.grafana.org/alerting/grafana/dynamic-label/view?tech=docs&pg=alerting-examples&plcmt=callout-tip&cta=alert-dynamic-labels)**.

   Open the example to view alert evaluation results, generated alert instances, the alert history timeline, and alert rule details.
   {{< /admonition >}}

## Considerations

Dynamic labels lets you reuse a single alert rule across multiple escalation scenarios—but it also introduces complexity. When the label value depends on a noisy metric and changes frequently, it can lead to flapping alert instances and excessive notifications.

These alerts often require tuning to stay reliable and benefit from continuous review. To get the most out of this pattern, consider the following:

- **Tune evaluation settings and queries for stability**

  Increase the [evaluation interval and pending period](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/) to reduce the frequency of state changes. Additionally, consider smoothing metrics with functions like `avg_over_time` to reduce flapping.

- **Use wider threshold bands**

  Define broader ranges in your label template logic to prevent label switching caused by small value changes.

- **Disable resolved notifications**

  When labels change frequently and alerts resolve quickly, you can reduce the number of notifications by disabling resolved notifications at the contact point.

- **Disable the Missing series evaluations setting**

  The [Missing series evaluations setting](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/) (default: 2) defines how many intervals without data are allowed before resolving an instance. Consider disabling it if it's unnecessary for your use case, as it can complicate alert troubleshooting.

- **Preserve context across related alerts**

  Ensure alert metadata includes enough information to help correlate related alerts during investigation.

- **Use separate alert rules and static labels when simpler**

  In some cases, defining separate rules with static labels may be easier to manage than one complex dynamic rule. This also allows you to customize alert queries for each specific case.

## Learn more

Here's a list of additional resources related to this example:

- [Multi-dimensional alerting example](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/) – Explore how Grafana creates separate alert instances for each unique set of labels.
- [Labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#labels) – Learn about the different types of labels and how they define alert instances.
- [Template labels in alert rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/) – Use templating to set label values dynamically based on query results.
- [Stale alert instances](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/) – Understand how Grafana resolves and removes stale alert instances.
- [Handle missing data](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/missing-data/) – Learn how Grafana distinguishes between missing series and `NoData`.
- [Notification policies and routing](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/) – Create multiple notification policies to route alerts based on label values like `severity` or `team`.
- [Dynamic label example in Grafana Play](https://play.grafana.org/alerting/grafana/dynamic-label/view?tech=docs&pg=alerting-examples&plcmt=learn-more&cta=alert-dynamic-labels) - View this example in Grafana Play to explore alert instances and state transitions with dynamic labels.

# Example of dynamic thresholds per dimension

In Grafana Alerting, each alert rule supports only one condition expression.

That's enough in many cases—most alerts use a fixed numeric threshold like `latency > 3s` or `error_rate > 5%` to determine their state.

As your alerting setup grows, you may find that different targets require different threshold values.

Instead of duplicating alert rules, you can assign a **different threshold value to each target**—while keeping the same condition. This simplifies alert maintenance.

This example shows how to do that using [multi-dimensional alerts](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/) and a [Math expression](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#math).

## Example overview

You're monitoring latency across multiple API services. Initially, you want to get alerted if the 95th percentile latency (`p95_api_latency`) exceeds 3 seconds, so your alert rule uses a single static threshold:

```
p95_api_latency > 3
```

But the team quickly finds that some services require stricter thresholds. For example, latency for payment APIs should stay under 1.5s, while background jobs can tolerate up to 5s. The team establishes different thresholds per service:

- `p95_api_latency{service="checkout-api"}`: must stay under `1.5s`.
- `p95_api_latency{service="auth-api"}`: also strict, `1.5s`.
- `p95_api_latency{service="catalog-api"}`: less critical, `3s`.
- `p95_api_latency{service="async-tasks"}`: background jobs can tolerate up to `5s`.

You want to avoid creating one alert rule per service—this is harder to maintain.

In Grafana Alerting, you can define one alert rule that monitors multiple similar components like this scenario. This is called [multi-dimensional alerts](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/): one alert rule, many alert instances—**one per unique label set**.

But there's an issue: Grafana supports only **one alert condition per rule**.

```
One alert rule
├─ One condition ( e.g., $A > 3)
│  └─ Applies to all returned series in $A
│     ├─ {service="checkout-api"}
│     ├─ {service="auth-api"}
│     ├─ {service="catalog-api"}
│     └─ {service="async-tasks"}
```

To evaluate per-service thresholds, you need a distinct threshold value for each returned series.

## Dynamic thresholds using a Math expression

You can create a dynamic alert condition by operating on two queries with a [Math expression](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#math).

- `$A` for query results (e.g., `p95_api_latency`).
- `$B` for per-service thresholds (from CSV data or another query).
- `$A > $B` is the _Math_ expression that defines the alert condition.

Grafana evaluates the _Math_ expression **per series**, by joining series from `$A` and `$B` based on their shared labels before applying the expression.

Here’s an example of an arithmetic operation:

{{< docs/shared lookup="alerts/math-example.md" source="grafana" version="<GRAFANA_VERSION>" >}}

In practice, you must align your threshold input with the label sets returned by your alert query.

The following table illustrates how a per-service threshold is evaluated in the previous example:

| $A: p95 latency query        | $B: threshold value            | $C: $A\>$B                   | State      |
| :--------------------------- | :----------------------------- | :--------------------------- | :--------- |
| `{service="checkout-api"} 3` | `{service="checkout-api"} 1.5` | `{service="checkout-api"} 1` | **Firing** |
| `{service="auth-api"} 1`     | `{service="auth-api"} 1.5`     | `{service="auth-api"} 0`     | **Normal** |
| `{service="catalog-api"} 2`  | `{service="catalog-api"} 3`    | `{service="catalog-api"} 0`  | **Normal** |
| `{service="sync-work"} 3`    | `{service="sync-work"} 5`      | `{service="sync-work"} 0`    | **Normal** |

In this example:

- `$A` comes from the `p95_api_latency` query.
- `$B` is manually defined with a threshold value for each series in `$A`.
- The alert condition compares `$A>$B` using a _Math_ relational operator (e.g., `>`, `<`, `>=`, `<=`, `==`, `!=`) that joins series by matching labels.
- Grafana evaluates the alert condition and sets the firing state where the condition is true.

The _Math_ expression works as long as each series in `$A` can be matched with exactly one series in `$B`. They must align in a way that produces a one-to-one match between series in `$A` and `$B`.

{{< admonition type="caution" >}}

If a series in one query doesn’t match any series in the other, it’s excluded from the result and a warning message is displayed:

_1 items **dropped from union(s)**: ["$A > $B": ($B: {service=payment-api})]_

{{< /admonition >}}

**Labels in both series don’t need to be identical**. If labels are a subset of the other, they can join. For example:

- `$A` returns series `{host="web01", job="event"}` 30 and `{host="web02", job="event"}` 20.
- `$B` returns series `{host="web01"}` 10 and `{host="web02"}` 0.
- `$A` + `$B` returns `{host="web01", job="event"}` 40 and `{host="web02", job="event"}` 20.

## Try it with TestData

You can use the [TestData data source](/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/) to replicate this example:

1. Add the **TestData** data source through the **Connections** menu.
1. Create an alert rule.

   Navigate to **Alerting** → **Alert rules** and click **New alert rule**.

1. Simulate a query (`$A`) that returns latencies for each service.

   Select **TestData** as the data source and configure the scenario.
   - Scenario: Random Walk
   - Alias: latency
   - Labels: service=api-$seriesIndex
   - Series count: 4
   - Start value: 1
   - Min: 1, Max: 4

     This uses `$seriesIndex` to assign unique service labels: `api-0`, `api-1`, etc.

   {{< figure src="/media/docs/alerting/example-dynamic-thresholds-latency-series-v2.png" max-width="750px" alt="TestData data source returns 4 series to simulate latencies for distinct API services." >}}

1. Define per-service thresholds with static data.

   Add a new query (`$B`) and select **TestData** as the data source.

   From **Scenario**, select **CSV Content** and paste this CSV:

   ```
    service,value
    api-0,1.5
    api-1,1.5
    api-2,3
    api-3,5
   ```

   The `service` column must match the labels from `$A`.

   The `value` column is a numeric value used for the alert comparison.

   For details on CSV format requirements, see [table data examples](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/table-data/).

1. Add a new **Reduce** expression (`$C`).
   - Type: Reduce
   - Input: A
   - Function: Mean
   - Name: C

   This calculates the average latency for each service: `api-0`, `api-1`, etc.

1. Add a new **Math** expression.
   - Type: Math
   - Expression: `$C > $B`
   - Set this expression as the **alert condition**.

   This fires if the average latency (`$C`) exceeds the threshold from `$B` for any service.

1. **Preview** the alert.

   {{< figure src="/media/docs/alerting/example-dynamic-thresholds-preview-v3.png" max-width="750px" caption="Alert preview evaluating multiple series with distinct threshold values" >}}

   {{< admonition type="tip" >}}
   You can explore this **[alerting example in Grafana Play](https://play.grafana.org/alerting/grafana/dynamic-thresholds/view?tech=docs&pg=alerting-examples&plcmt=callout-tip&cta=alert-dynamic-thresholds)**.

   Open the example to view alert evaluation results, generated alert instances, the alert history timeline, and alert rule details.
   {{< /admonition >}}

## Other use cases

This example showed how to build a single alert rule with different thresholds per series using [multi-dimensional alerts](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/) and [Math expressions](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#math).

This approach scales well when monitoring similar components with distinct reliability goals.

By aligning series from two queries, you can apply a dynamic threshold—one value per label set—without duplicating rules.

While this example uses static CSV content to define thresholds, the same technique works in other scenarios:

- **Dynamic thresholds from queries or recording rules**: Fetch threshold values from a real-time query, or from [custom recording rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/).
- **Combine multiple conditions**: Build more advanced threshold logic by combining multiple conditions—such as latency, error rate, or traffic volume.

For example, you can define a PromQL expression that sets a latency threshold which adjusts based on traffic—allowing higher response times during periods of high-load.

```
(
  // Fires when p95 latency > 2s during usual traffic (≤ 1000 req/s)
  service:latency:p95 > 2 and service:request_rate:rate1m <= 1000
)
or
(
  // Fires when p95 latency > 4s during high traffic (> 1000 req/s)
  service:latency:p95 > 4 and service:request_rate:rate1m > 1000
)
```

# Examples of high-cardinality alerts

In Prometheus and Mimir, metrics are stored as time series, where each unique set of labels defines a distinct series.

A large number of unique series (_high cardinality_) can overload your metrics backend, slow down dashboard and alert queries, and quickly increase your observability costs or exceed the limits of your Grafana Cloud plan.

These examples show how to detect and alert on early signs of high cardinality:

- **Total active series near limits**: detect when your Prometheus, Mimir, or Grafana Cloud Metrics instance approaches soft or hard limits.
- **Series increase per metric or scope:** fine-tune detection to identify growth in a particular metric or domain.
- **Sudden series growth**: detect runaway cardinality increases caused by misconfigured exporters or new deployments.
- **High ingestion rate**: detect when too many samples per second are being ingested, even if the total series count is stable.

Use these alert patterns to act on high-cardinality growth, and consider implementing [Adaptive Metrics recommendations](/docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/introduction/) to keep your observability costs under control.

## Choose metrics to monitor active series

First, identify which metric reports the number of active time series.

Prometheus, Mimir, and Grafana Cloud expose this information differently:

| Environment             | Metric                                | Description                                                                                                                                                                                   |
| :---------------------- | :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Prometheus**          | `prometheus_tsdb_head_series`         | Reports the number of active series currently stored in memory (the head block) of a single Prometheus instance. It includes series that have stopped receiving samples for up to one hour.   |
| **Grafana Cloud**       | `grafanacloud_instance_active_series` | Tracks the number of [active series in your Grafana Cloud Metrics backend (Mimir](/docs/grafana-cloud/cost-management-and-billing/manage-invoices/understand-your-invoice/metrics-invoice/)). |
| **Prometheus or Mimir** | `count({__name__!=""})`               | Counts the number of series with recent samples by scanning the TSDB index. This query is expensive and should be exposed through a recording rule.                                           |

## Detect total active series near limits

A high number of active series increases memory usage and can impact performance. Grafana Cloud enforces usage limits to prevent your instance from running into these performance issues.

In Prometheus, you can alert when the total number of active series exceeds a threshold:

```shell
prometheus_tsdb_head_series > 1.5e6
```

This fires when the instance exceeds 1.5 million active series.  
Adjust the threshold based on the capacity of your Prometheus host and observed load.

In Grafana Cloud, use the `grafanacloud_instance_active_series` metric to monitor active series across your managed Mimir backend:

```shell
grafanacloud_instance_active_series > 1.5e6
```

Grafana Cloud also provides account-level limits through the `grafanacloud_instance_metrics_limits` metric.

For more robust alerting, you can compare your current usage to the `max_global_series_per_user` limit:

```shell
(
  grafanacloud_instance_active_series
  / on (id)
    grafanacloud_instance_metrics_limits{limit_name="max_global_series_per_user"}
)
* on (id) group_left(name) grafanacloud_instance_info
> 0.9

```

- `grafanacloud_instance_active_series`  
  Returns the current number of active series per your Mimir (Prometheus) data source instance (`id`).

- `/ on (id) grafanacloud_instance_metrics_limits{limit_name="max_global_series_per_user"}`  
  Divides current usage by the account limit to calculate a utilization ratio between 0 and 1 (where `1` means the limit is reached).

- `* on (id) group_left(name) grafanacloud_instance_info`  
  Joins instance metadata to display the instance name.

- `> 0.9`  
  Defines the threshold condition to fire when usage exceeds 90% of the limit.  
  Adjust this value according to your alert goal. Alternatively, you can set the threshold as a Grafana Alerting expression in the UI.

## Detect high-cardinality per metric

Instead of monitoring the total number of active series, you can fine-tune alerts to detect high cardinality within a specific scope — for example, by filtering on certain namespaces, services, or metrics known to generate many label combinations.

[Multi-dimensional alerts](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/) let you evaluate each metric independently, so you can identify which metric is responsible for the label explosion instead of only tracking the overall total.

You can apply label filters, or use `{__name__=~"regex"}` to select specific metrics. Then, use `count by (__name__)` to group results per metric name.

Because the `__name__` selector queries the entire TSDB index, it’s recommended to query this using a **recording rule**:

```shell
# Only HTTP/RPC-style metrics
active_series_per_metric:http_rpc =
label_replace(
  count by (__name__) ({__name__=~"http_.*|rpc_.*"}),
  "metric", "$1", "__name__", "(.*)"
)
```

This recording rule stores the number of active series per metric name.

- `count by (__name__) ({__name__=~"http_.*|rpc_.*"})`  
  Counts the number of active series per metric matching the `http_.*` or `rpc_.*` regex.

- `label_replace(..., "metric", "$1", "__name__", "(.*)")`  
  Copies the metric name into a new label called `metric`.  
  This enables generating one alert instance per metric because `__name__` is not treated as a regular label.

Adjust the threshold and recording rule scope based on the label usage and normal behavior of your observed metrics.

After the recording rule is available, you can define this multi-dimensional alert rule as follows:

```shell
active_series_per_metric:http_rpc > 100
```

Grafana Alerting evaluates each row (or time series) returned by the `active_series_per_metric:http_rpc` recording rule as a separate alert instance, producing independent alert instance states:

| Alert instance                       | Value | State  |
| :----------------------------------- | :---- | :----- |
| `{metric="http_requests_total"}`     | 320   | Firing |
| `{metric="rpc_client_calls_total"}`  | 45    | Normal |
| `{metric="rpc_server_errors_total"}` | 110   | Firing |

Each metric name (`__name__`) becomes a separate alert instance, so you immediately see which metric exceeds the expected limit.

## Detect sudden cardinality growth

Even if the number of active series stays within proper limits, a sudden increase can signal a misbehaving exporter, a new deployment, or an unexpected label explosion. These peaks can help you prevent potential issues, or just inform you of deployment changes that might need adjustments.

You can use any of the metrics from the previous examples to track short-term changes in the total active series:

```shell
delta(active_series_metric[10m]) > 1000
```

This alert fires when the number of active series increases by more than **1000** within the last 10 minutes.  
Adjust the time window (for example, `[5m]` or `[30m]`) and threshold to match your environment’s normal variability.

## Detect high ingestion rate

Even if label cardinality remains under control, a high ingestion rate can affect Prometheus performance or increase observability costs.

In Prometheus, this usually happens when scrapes occur too frequently or when exporters generate large numbers of samples in short intervals.

To find an appropriate threshold to be alerted, start by monitoring normal ingestion peaks and set the threshold to a value that stays below the point where scrapes or WAL operations begin to slow down.

In Prometheus, use the `prometheus_tsdb_head_samples_appended_total` metric to measure the number of samples appended per scrape:

```shell
rate(prometheus_tsdb_head_samples_appended_total[10m]) > 1e5
```

The alert rule query returns the average ingestion rate per second over the last 10 minutes and fires when the value exceeds 100 000 samples per second.

In Grafana Cloud, use the `grafanacloud_instance_samples_per_second` metric to monitor total ingestion rate of your Mimir instances:

```shell
grafanacloud_instance_samples_per_second
  * on (id) group_left(name) grafanacloud_instance_info
> 1e5
```

Alternatively, Grafana Cloud metrics limits are based on [data points per minute (DPM)](/docs/grafana-cloud/cost-management-and-billing/manage-invoices/understand-your-invoice/metrics-invoice/): the number of samples sent per minute across all your active series.

To monitor when your actual data-point rate approaches your DPM limit, you can compare total ingestion to your plan’s DPM limit:

```shell
(
  (grafanacloud_instance_samples_per_second * 60)
  / grafanacloud_org_metrics_included_dpm_per_series
)
> 0.9

```

- `(grafanacloud_instance_samples_per_second * 60)`  
  Converts ingestion rate from data points per second to minutes.

- `/ grafanacloud_org_metrics_included_dpm_per_series`  
  Divides current DPM usage by the DPM limit to calculate a utilization ratio between 0 and 1 (where `1` means the limit is reached).

- `> 0.9`  
  Defines the threshold condition to fire when the usage exceeds 90% of the DPM limit.

This alert helps you detect when your organization is ingesting data faster and approaching its limits. Use ingestion rate alerts to detect workload spikes, exporter misconfigurations, or rapid increases in ingestion volume and cost.

## Learn more

Here’s list of additional resources related to this example:

- [Multi-dimensional alerting example](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/) – Learn how Grafana creates separate alert instances for each unique label set.
- [Understand Grafana Cloud active series and DPM](/docs/grafana-cloud/cost-management-and-billing/manage-invoices/understand-your-invoice/metrics-invoice/)– See how active series and data points per minute (DPM) are used to calculate metrics usage in Grafana Cloud.
- [Create Grafana Cloud usage alerts](/docs/grafana-cloud/cost-management-and-billing/usage-cost-alerts/) – Set up alerts when your usage or costs approach your predefined limits.
- [Plan capacity for Mimir](/docs/mimir/latest/manage/run-production-environment/planning-capacity/)– Learn how to plan ingestion rate and memory capacity for Mimir or Prometheus environments.
- [Adaptive Metrics recommendations](/docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/introduction/) – Use Adaptive Metrics to automatically reduce high-cardinality metrics and control observability costs.

# Example of multi-dimensional alerts on time series data

This example shows how a single alert rule can generate multiple alert instances — one for each label set (or time series). This is called **multi-dimensional alerting**: one alert rule, many alert instances.

In Prometheus, each unique combination of labels defines a distinct time series. Grafana Alerting uses the same model: each label set is evaluated independently, and a separate alert instance is created for each series.

This pattern is common in dynamic environments when monitoring a group of components like multiple CPUs, containers, or per-host availability. Instead of defining individual alert rules or aggregated alerts, you alert on _each dimension_ — so you can detect particular issues and include that level of detail in notifications.

For example, a query returns one series per CPU:

| `cpu` label value | CPU percent usage |
| :---------------- | :---------------- |
| cpu-0             | 95                |
| cpu-1             | 30                |
| cpu-2             | 85                |

With a threshold of `> 80`, this would trigger two alert instances for `cpu-0` and one for `cpu-2`.

## Examples overview

Imagine you want to trigger alerts when CPU usage goes above 80%, and you want to track each CPU core independently.

You can use a Prometheus query like this:

```
sum by(cpu) (
  rate(node_cpu_seconds_total{mode!="idle"}[1m])
)
```

This query returns the active CPU usage rate per CPU core, averaged over the past minute.

| CPU core | Active usage rate |
| :------- | :---------------- |
| cpu-0    | 95                |
| cpu-1    | 30                |
| cpu-2    | 85                |

This produces one series for each existing CPU.

When Grafana Alerting evaluates the query, it creates an individual alert instance for each returned series.

| Alert instance | Value |
| :------------- | :---- |
| {cpu="cpu-0"}  | 95    |
| {cpu="cpu-1"}  | 30    |
| {cpu="cpu-2"}  | 85    |

With a threshold condition like `$A > 80`, Grafana evaluates each instance separately and fires alerts only where the condition is met:

| Alert instance | Value | State  |
| :------------- | :---- | :----- |
| {cpu="cpu-0"}  | 95    | Firing |
| {cpu="cpu-1"}  | 30    | Normal |
| {cpu="cpu-2"}  | 85    | Firing |

Multi-dimensional alerts help you surface issues on individual components—problems that might be missed when alerting on aggregated data (like total CPU usage).

Each alert instance targets a specific component, identified by its unique label set. This makes alerts more specific and actionable. For example, you can set a [`summary` annotation](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#annotations) in your alert rule that identifies the affected CPU:

```
High CPU usage on {{$labels.cpu}}
```

In the previous example, the two firing alert instances would display summaries indicating the affected CPUs:

- High CPU usage on `cpu-0`
- High CPU usage on `cpu-2`

## Try it with TestData

You can quickly experiment with multi-dimensional alerts using the [**TestData** data source](/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/), which can generate multiple random time series.

1. Add the **TestData** data source through the **Connections** menu.
1. Go to **Alerting** and create an alert rule
1. Select **TestData** as the data source.
1. Configure the TestData scenario
   - Scenario: **Random Walk**
   - Labels: `cpu=cpu-$seriesIndex`
   - Series count: 3
   - Min: 70, Max: 100
   - Spread: 2

   {{< figure src="/media/docs/alerting/testdata-random-series-v2.png" max-width="750px" alt="Generating random time series data using the TestData data source" >}}

## Reduce time series data for comparison

The example returns three time series like shown above with values across the selected time range.

To alert on each series, you need to reduce the time series to a single value that the alert condition can evaluate and determine the alert instance state.

Grafana Alerting provides several ways to reduce time series data:

- **Data source query functions**. The earlier example used the Prometheus `sum` function to sum the rate results by `cpu,`producing a single value per CPU core.
- **Reduce expression**. In the query and condition section, Grafana provides the `Reduce` expression to aggregate time series data.
  - In **Default mode**, the **When** input selects a reducer (like `last`, `mean`, or `min`), and the threshold compares that reduced value.
  - In **Advanced mode**, you can add the [**Reduce** expression](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#reduce) (e.g., `last()`, `mean()`) before defining the threshold (alert condition).

For demo purposes, this example uses the **Advanced mode** with a **Reduce** expression:

1. Toggle **Advanced mode** in the top right section of the query panel to enable adding additional expressions.
1. Add the **Reduce** expression using a function like `mean()` to reduce each time series to a single value.
1. Define the alert condition using a **Threshold** like `$reducer > 80`
1. Click **Preview** to evaluate the alert rule.

   {{< figure src="/media/docs/alerting/using-expressions-with-multiple-series.png" max-width="750px" caption="The alert condition evaluates the reduced value for each alert instance and shows whether each instance is Firing or Normal." alt="Alert preview using a Reduce expression and a threshold condition" >}}

   {{< admonition type="tip" >}}
   You can explore this **[alerting example in Grafana Play](https://play.grafana.org/alerting/grafana/multi-dimensional-alerts/view?tech=docs&pg=alerting-examples&plcmt=callout-tip&cta=alert-multi-dimensional)**.

   Open the example to view alert evaluation results, generated alert instances, the alert history timeline, and alert rule details.
   {{< /admonition >}}

## Learn more

This example shows how Grafana Alerting implements a multi-dimensional alerting model: one rule, many alert instances and why reducing time series data to a single value is required for evaluation.

For additional learning resources, check out:

- [Get started tutorial – Create multi-dimensional alerts and route them](https://grafana.com/tutorials/alerting-get-started-pt2/)
- [Example of alerting on tabular data](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/table-data/)
  Update the interval of a rule group or modify the rules of the group.

# Example of alerting on tabular data

Not all data sources return time series data. SQL databases, CSV files, and some APIs often return results as rows or arrays of columns or fields — commonly referred to as tabular data.

This example shows how to create an alert rule using data in table format. Grafana treats each row as a separate alert instance, as long as the data meets the expected format.

## How Grafana Alerting evaluates tabular data

When a query returns data in table format, Grafana transforms each row into a separate alert instance.

To evaluate each row (alert instance), it expects:

1. **Only one numeric column.** This is the value used for evaluating the alert condition.
1. **Non-numeric columns.** These columns defines the label set. The column name becomes a label name; and the cell value becomes the label value.
1. **Unique label sets per row.** Each row must be uniquely identifiable by its labels. This ensures each row represents a distinct alert instance.

{{< admonition type="caution" >}}
These three conditions must be met—otherwise, Grafana can’t evaluate the table data and the rule will fail.
{{< /admonition >}}

## Example overview

Imagine you store disk usage in a `DiskSpace` table and you want to trigger alerts when the available space drops below 5%.

| Time       | Host | Disk | PercentFree |
| ---------- | ---- | ---- | ----------- |
| 2021-06-07 | web1 | /etc | 3           |
| 2021-06-07 | web2 | /var | 4           |
| 2021-06-07 | web3 | /var | 8           |

To calculate the free space per Host and Disk in this case, you can use `$__timeFilter` to filter by time but without returning the date to Grafana:

```sql
SELECT
  Host,
  Disk,
  AVG(PercentFree) AS PercentFree
FROM DiskSpace
WHERE $__timeFilter(Time)
GROUP BY Host, Disk
```

This query returns the following table response:

| Host | Disk | PercentFree |
| ---- | ---- | ----------- |
| web1 | /etc | 3           |
| web2 | /var | 4           |
| web3 | /var | 8           |

When Alerting evaluates the query response, the data is transformed into three alert instances as previously detailed:

- The numeric column becomes the value for the alert condition.
- Additional columns define the label set for each alert instance.

| Alert instance               | Value |
| ---------------------------- | ----- |
| `{Host="web1", Disk="/etc"}` | 3     |
| `{Host="web2", Disk="/var"}` | 4     |
| `{Host="web3", Disk="/var"}` | 8     |

Finally, an alert condition that checks for less than 5% of free space (`$A < 5`) would result in two alert instances firing:

| Alert instance               | Value | State  |
| ---------------------------- | ----- | ------ |
| `{Host="web1", Disk="/etc"}` | 3     | Firing |
| `{Host="web2", Disk="/var"}` | 4     | Firing |
| `{Host="web3", Disk="/var"}` | 8     | Normal |

## Try it with TestData

To test this quickly, you can simulate the table using the [**TestData** data source](/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/):

1. Add the **TestData** data source through the **Connections** menu.
1. Go to **Alerting** and create an alert rule
1. Select **TestData** as the data source.
1. From **Scenario**, select **CSV Content** and paste this CSV:

   ```bash
   host, disk, percentFree
   web1, /etc, 3
   web2, /var, 4
   web3, /var, 8
   ```

1. Set a condition like `$A < 5` and **Preview** the alert.

   Grafana evaluates the table data and fires the two first alert instances.

   {{< figure src="/media/docs/alerting/example-table-data-preview.png" max-width="750px" alt="Alert preview with tabular data using the TestData data source" >}}

   {{< admonition type="tip" >}}
   You can explore this **[alerting example in Grafana Play](https://play.grafana.org/alerting/grafana/tabular-data/view?tech=docs&pg=alerting-examples&plcmt=callout-tip&cta=alert-tabular-data)**.

   Open the example to view alert evaluation results, generated alert instances, the alert history timeline, and alert rule details.
   {{< /admonition >}}

## CSV data with Infinity

Note that when the [Infinity plugin fetches CSV data](/docs/plugins/yesoreyeram-infinity-datasource/latest/csv/), all the columns are parsed and returned as strings. By default, this causes the query expression to fail in Alerting.

To make it work, you need to format the CSV data as [expected by Grafana Alerting](#how-grafana-alerting-evaluates-tabular-data).

In the query editor, specify the column names and their types to ensure that only one column is treated as a number.

{{< figure src="/media/docs/alerting/example-table-data-infinity-csv-data.png" max-width="750px" alt="Using the Infinity data source plugin to fetch CSV data in Alerting" >}}

## Differences with time series data

Working with time series is similar—each series is treated as a separate alert instance, based on its label set.

The key difference is the data format:

- **Time series data** contains multiple values over time, each with its own timestamp.
  To evaluate the alert condition, alert rules **must reduce each series to a single number** using a function like `last()`, `avg()`, or `max()`.
- **Tabular data** doesn’t require reduction, as each row contains only a single numeric value used to evaluate the alert condition.

For comparison, see the [multi-dimensional time series data example](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/).

# Examples of trace-based alerts

Metrics are the foundation of most alerting systems. They are usually the first signal that something is wrong, but they don’t always indicate _where_ or _why_ a failure occurs.

Traces fill that gap by showing the complete path a request takes through your system. They map the workflows across services, indicating where the request slows down or fails.

{{< figure src="/media/docs/alerting/screenshot-traces-visualization-11.5.png" max-width="750px" alt="Trace view" >}}

Traces report duration and errors directly to specific services and spans, helping to find the affected component and service scope. With this additional context, alerting on tracing data can help you **identify root causes faster**.

You can create trace-based alerts in Grafana Alerting using two main approaches:

- Querying metrics generated from tracing data.
- Using TraceQL, a query language for traces available in Grafana Tempo.

This guide provides introductory examples and distinct approaches for setting up **trace-based alerts** in Grafana. Tracing data is commonly collected using **OpenTelemetry (OTel)** instrumentation. OTel allows you to integrate trace data from a wide range of applications and environments into Grafana.

## Alerting on span metrics

OpenTelemetry provides processors that convert tracing data into Prometheus-style metrics.

The **service graph** and **span metrics** processors are the standard options in Alloy and Tempo to generate Prometheus metrics from traces. They can generate the rate, error, and duration (RED) metrics from sampled spans.

You can then create alert rules that query metrics derived from traces.

{{< figure src="/media/docs/alerting/why-trace-based-metrics.png" max-width="750px" alt="Why metrics if you have traces?" >}}

[Service graph metrics](https://grafana.com/docs/tempo/latest/metrics-from-traces/service_graphs/) focus on inter-service communication and dependency health. They measure the calls between services, helping Grafana to infer the service topology. However, they measure only the interaction between two services—they don’t include the internal processing time of the client service.

You can use service graph metrics to detect infrastructure issues such as network degradation or service mesh problems.

For trace-based alerts, use [span metrics](https://grafana.com/docs/tempo/latest/metrics-from-traces/span-metrics/).

**Span metrics** measure the total processing time of a service request: capturing what happens inside the service, not just the communication between services. They include the time spent on internal processing and waiting on downstream calls, providing an **end-to-end picture of service performance**.

Depending on how you create span metrics, the following span metrics are generated:

| Span metrics generator                                                                                                                                                                                                                                                     | Metric name                            | Prometheus metric type        | Description                  |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------- | :---------------------------- | :--------------------------- |
| [Alloy](https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.connector.spanmetrics/) and [OTEL span metrics connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector)                   | `traces_span_metrics_calls_total`      | Counter                       | Total count of the span      |
|                                                                                                                                                                                                                                                                            | `traces_span_metrics_duration_seconds` | Histogram (native or classic) | Duration of the span         |
| [Tempo](https://grafana.com/docs/tempo/latest/metrics-from-traces/span-metrics/span-metrics-metrics-generator/) and [Grafana Cloud Application Observability](https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/setup/metrics-labels/) | `traces_spanmetrics_calls_total`       | Counter                       | Total count of the span      |
|                                                                                                                                                                                                                                                                            | `traces_spanmetrics_latency`           | Histogram (native or classic) | Duration of the span         |
|                                                                                                                                                                                                                                                                            | `traces_spanmetrics_size_total`        | Counter                       | Total size of spans ingested |

Each metric includes by default the following labels: `service`, `span_name`, `span_kind`, `status_code`, `status_message`, `job`, and `instance`.

In the metrics generator, you can customize how traces are converted into metrics by configuring histograms, exemplars, metric dimensions, and other options.

The following examples assume that span metrics have already been generated using one of these options or an alternative.

### Detect slow span operations

This example shows how to define an alert rule that detects when operations handled by a service become slow.

Before looking at the query, it’s useful to review a few [trace elements](https://grafana.com/docs/tempo/latest/introduction/trace-structure/) that shape how it works:

- A trace represents a single request or transaction as it flows through multiple spans and services. A span refers to a specific operation within a service.
- Each span includes the operation name (`span_name`) and its duration (the metric value), as well as additional fields like [span status](https://opentelemetry.io/docs/concepts/signals/traces/#span-status) (`status_code`) and [span kind](https://opentelemetry.io/docs/concepts/signals/traces/#span-kind) (`span_kind`).
- A server span represents work performed on the receiving side of a request, while a client span represents the outbound call (parent span) waiting for a response (client → server).

To detect slow inbound operations within a specific service, you can define an alert rule that detects when the percentile latency of server spans exceeds a threshold. For example:

_Detect when 95% of requests (excluding errors) do not complete faster than 2 seconds._

#### Using native histograms

The following PromQL query uses the `traces_span_metrics_duration_seconds` native histogram metric to define the alert rule query.

```promql
histogram_quantile(0.95,
 sum by (span_name) (
   rate(traces_span_metrics_duration_seconds{
     service_name="<SERVICE_NAME>",
     span_kind="SPAN_KIND_SERVER",
     status_code!="STATUS_CODE_ERROR"
   }[10m])
 )
) > 2
```

Here’s the query breakdown:

- `traces_span_metrics_duration_seconds`
  It’s a native histogram produced from spans using Alloy or the OTEL collector. The metric is filtered by:
  - `service_name="<SERVICE_NAME>"` targets a particular service.
  - `span_kind="SPAN_KIND_SERVER"` selects spans handling inbound requests.
  - `status_code!="STATUS_CODE_ERROR"` excludes spans that ended with errors.

  _You should query `traces_spanmetrics_latency` when using other span metric generators._

- `rate(...[10m])`
  Converts the histogram into a per-second histogram over the last 10 minutes (the distribution of spans per second during that period).
  This makes the time window explicit and ensures latencies can be calculated over the last 10 minutes using `histogram_*` functions.
- `sum by (span_name)( … )`
  Merges all series that share the same `span_name`. This creates a [multidimensional alert](https://grafana.com/docs/grafana/latest/alerting/best-practices/multi-dimensional-alerts/) that generates one alert instance per span name (operation).
- `histogram_quantile(0.95, ...)`
  Calculates p95 latency from the histogram after applying the rate.
  The query runs as an **instant Prometheus query**, returning a single value for the 10-minute window.
- `> 2`
  Defines the threshold condition. It returns only series whose p95 latency exceeds 2 seconds.
  Alternatively, you can set this threshold as a Grafana Alerting expression in the UI, as shown in the following screenshot.

  {{< figure src="/media/docs/alerting/trace-based-alertrule-screenshot.png" max-width="750px" caption="Alert rule querying span metrics and using threshold expression" >}}

#### Using classic histograms

Native histograms are stable in Prometheus since v3.8.0. Your span metric generator may therefore create classic histograms for latency span metrics, either `traces_span_metrics_duration_seconds` or `traces_spanmetrics_latency`.

When using classic histograms, the metric is the same but the metric format changes. A classic histogram represents a histogram with fixed buckets and exposes three metrics:

- `_bucket`: cumulative buckets of the observations.
- `_sum`: total sum of all observed values.
- `_count`: count of observed values.

To calculate percentiles accurately, especially exceeding a particular threshold (e.g. `` `2s` ``), you have to configure the classic histogram with the explicit bucket, such as:

```shell
["100ms", "250ms", "1s", "2s", "5s"]
```

The `otelcol.connector.spanmetrics` can configure the buckets using the [`explicit` block](https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.connector.spanmetrics/#explicit). The metric-generator in Tempo can configure the [`span_metrics.histogram_buckets` setting](https://grafana.com/docs/tempo/latest/configuration/#metrics-generator).

Here's the equivalent PromQL for classic histograms:

```promql
histogram_quantile(0.95,
 sum by (span_name, le) (
   rate(traces_span_metrics_duration_seconds_bucket{
     service_name="<SERVICE_NAME>",
     span_kind="SPAN_KIND_SERVER",
     status_code!="STATUS_CODE_ERROR"
   }[10m])
 )
) > 2
```

Key differences compared with the native histograms example:

- You must configure a histogram bucket matching the desired threshold (for example, `2s`).
- You must query the `_bucket` metric, not the base metric.
- You must include `le` in the `sum by (…)` grouping for `histogram_quantile` calculation.

Everything else remains the same.

{{< admonition type="note" >}}

The alert rules in these examples create [multi-dimensional alerts](https://grafana.com/docs/grafana/latest/alerting/best-practices/multi-dimensional-alerts/): one alert instance for each distinct span name.

Dynamic span routes such as `/product/1234` can create separate metric dimensions and alerts for each unique span, which can significantly impact metric costs and performance for large volumes.

To prevent high-cardinality data, normalize dynamic routes like `/product/{id}` using semantic attributes such as [`http.route`](https://opentelemetry.io/docs/specs/semconv/registry/attributes/http/) and [`url.template`](https://opentelemetry.io/docs/specs/semconv/registry/attributes/url/), and limit dimensions to low-cardinality fields such as `service_name`, `status_code`, or `http_method`.

{{< /admonition >}}

### Detect high error rate

This example defines an alert rule that detects when the error rate for any operation exceeds 20%. You can use this error rate alerts to identify increases in request errors, such as 5xx responses or internal failures.

The following query calculates the fraction of failed server spans for each service and operation.

```promql
(
  sum by (service, span_name) (
    rate(traces_span_metrics_calls_total{
      span_kind="SPAN_KIND_SERVER",
      status_code="STATUS_CODE_ERROR"
    }[10m])
  )
/
  sum by (service, span_name) (
    rate(traces_span_metrics_calls_total{
      span_kind="SPAN_KIND_SERVER"
    }[10m])
  )
) > 0.2
```

Here’s the query breakdown

- `traces_span_metrics_calls_total`
  A counter metric produced from spans that tracks the number of completed span operations.
  - `span_kind="SPAN_KIND_SERVER"` selects spans handling inbound requests.
  - `status_code="STATUS_CODE_ERROR"` selects only spans that ended in error.
  - Omitting the `status_code` filter in the denominator includes all spans, returning the total span count.

  _Check whether your metric generator instead creates the `traces_spanmetrics_calls_total` metric, and adjust the metric name._

- `rate(...[10m])`
  Converts the cumulative histogram into a per-second histogram over the last 10 minutes (the distribution of spans per second during that period).
  This makes the time window explicit and ensures counters can be calculated over the last 10 minutes.
- `sum by (service, span_name)( … )`
  Aggregates per service and operation, creating one alert instance for each `(service, span_name)` combination.
  This is a [multidimensional alert](https://grafana.com/docs/grafana/latest/alerting/best-practices/multi-dimensional-alerts/) that applies to all services, helping identify which service and corresponding operation is failing.
- `sum by () (...) / sum by () (...)`
  Divides failed spans by total spans to calculate the error rate per operation.
  The result is a ratio between `0` and `1,` where `1` means all operations failed.
  The query runs as an **instant Prometheus query**, returning a single value for the 10-minute window.
- `> 0.2`
  Defines the threshold condition. It returns only series whose error rate is higher than 20% of spans.
  Alternatively, you can set this threshold as a Grafana Alerting expression in the UI.

### Enable traffic guardrails

When the traffic is very low, even a single slow or failing request can trigger the alerts.

To avoid these types of false positives during low-traffic periods, you can include a **minimum traffic condition** in your alert rule queries. For example:

```promql
sum by (service, span_name)(
  increase(traces_span_metrics_calls_total{
    span_kind="SPAN_KIND_SERVER"
  }[10m])
) > 300
```

This query returns only spans that handled more than 300 requests in the 10-minute period.

This minimum level of traffic helps prevent false positives, ensuring the alert evaluates a significant number of spans before triggering.

You can combine this traffic condition with the **error-rate** query to ensure alerts fire only when both conditions are met:

```promql
((
  sum by (service, span_name) (
    rate(traces_span_metrics_calls_total{
      span_kind="SPAN_KIND_SERVER",
      status_code="STATUS_CODE_ERROR"
    }[10m])
  )
/
  sum by (service, span_name) (
    rate(traces_span_metrics_calls_total{
      span_kind="SPAN_KIND_SERVER"
    }[10m])
  )
) > 0.2)
and
(
    sum by (service, span_name)(
    increase(traces_span_metrics_calls_total{
      span_kind="SPAN_KIND_SERVER"
    }[10m])
) > 300 )

```

For a given span, the alert fires when:

- The **error rate exceeds 20%** over the last 10 minutes.
- The span **handled at least 300 requests** over the last 10 minutes.

**Alternatively**, you can split the alert into separate queries and combine them using a math expression as the threshold. In the example below, `$ErrorRateCondition` is the Grafana reference for the error-rate query, and `$TrafficCondition` is the reference for the traffic query.

{{< figure src="/media/docs/alerting/traffic-guardrail-with-separate-queries.png" max-width="500px" alt="Alert rule with threshold based on two queries" >}}

In this case, you must ensure both queries group by the same labels.

The advantage of this approach is that you can observe the results of both independent queries. You can then access the query results through the [`$values` variable](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/reference/#values) and display them in notifications or use them in custom labels.

A potential drawback of splitting queries is that each query runs separately. This increases backend load and can affect query performance, especially in environments with a large number of active alerts.

You can apply this traffic guardrail pattern to any alert rule.

### Consider sampling {#consider-sampling}

[Sampling](https://grafana.com/docs/tempo/latest/set-up-for-tracing/instrument-send/set-up-collector/tail-sampling/) is a technique used to reduce the amount of collected spans for cost-saving purposes. There are two main strategies which can be combined:

- **Head sampling**: The decision to record or drop a span is made when the trace begins. The condition can be configured probabilistically (a percentage of traces) or by filtering out certain operations.
- **Tail sampling**: The decision is made after the trace completes. This allows sampling more interesting operations, such as slow or failing requests.

With **head sampling**, alerting on span metrics should be done with caution, since span metrics will represent only a subset of all traces.

With **tail sampling**, it’s important to generate span metrics before a sampling decision is made. [Grafana Cloud Adaptive Traces](https://grafana.com/docs/grafana-cloud/adaptive-telemetry/adaptive-traces/) handle this automatically. With Alloy or the OpenTelemetry Collector, make sure the SpanMetrics connector runs before the filtering or [tail sampling processor](https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.processor.tail_sampling/).

## Using TraceQL

TraceQL is a query language for searching and filtering traces in Grafana Tempo, which uses a syntax similar to `PromQL` and `LogQL`.

With TraceQL, you can skip converting tracing data into span metrics and query raw trace data directly. It provides a more flexible filtering based on the trace structure, attributes, or resource metadata, and can detect issues faster as it doesn't wait for metric generation.

TraceQL isn't suitable for all scenarios. For example:

- **Inadequate for long-term analysis**
  Trace data has a significantly shorter retention period than metrics. For historical monitoring, it’s recommended to convert key tracing data into metrics to ensure the persistence of important data.
- **Inadequate for alerting after sampling**
  TraceQL can only query traces that are actually stored in Tempo. If sampling drops a large portion of traces, TraceQL-based alerts may miss real issues. Refer to [consider sampling](#consider-sampling) for guidance on how to generate span metrics before sampling.

  {{< admonition type="caution" >}}

  TraceQL alerting is available in Grafana v12.1 or higher, supported as a [private preview feature](https://grafana.com/docs/release-life-cycle/).
  Engineering and on-call support isn't available. No SLA is provided.

  While TraceQL can be powerful for exploring and detecting issues directly from trace data, **alerting with TraceQL shouldn't be used in production environments yet**. Use it for testing and experimentation at this moment.

  {{< /admonition >}}

The following example demonstrates how to recreate the previous **alert rule that detected slow span operations** using TraceQL.

Follow these steps to create the alert:

1. Enable TraceQL alerting
   To use TraceQL in alerts, you must enable the [**`tempoAlerting`** feature flag in your Grafana configuration](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#feature_toggles).

   If you use Grafana Cloud, contact Support to enable TraceQL alerting.

2. Configure the alert query

   In your alert rule, select the **Tempo** data source, then convert the original PromQL query into the equivalent TraceQL query:

   ```traceql
   {status != error && kind = server && .service.name = "<SERVICE_NAME>"}
   | quantile_over_time(duration, .95) by (name)
   ```

   For a given service, this query calculates the **p95 latency** for all server spans, excluding errors, and groups them by span name.

3. Configure the time range

   Currently, TraceQL alerting supports only range queries.
   To define the time window, set the query time range to **the last 10 minutes.**
   - From: `now-10m`
   - To: `now`

     {{< figure src="/media/docs/alerting/traceql-alert-configure-time-range.png" max-width="750px" alt="Time range configuration for TraceQL alert rule" >}}

4. Add a reducer expression.

   Range queries return time series data, not a single value. The alert rule must then **reduce** time series data to a single numeric value before comparing it against a threshold.

   Add a **Reduce** expression to convert the query results into a single value.

5. Set the threshold condition.

   Create a **Threshold** expression to fire when the p95 latency exceeds 2 seconds: **$B > 2**.

   {{< figure src="/media/docs/alerting/traceql-alert-configure-threshold.png" max-width="750px" alt="Alert rule configuration showing reducer and threshold expressions for TraceQL query" >}}

This final alert detects when 95% of the server spans for a particular service (excluding errors) take longer than 2 seconds to complete, using raw trace data instead of span metrics.

## Additional resources

To explore related topics and expand the examples in this guide, see the following resources:

- [Trace structure](https://grafana.com/docs/tempo/latest/introduction/trace-structure/): Learn how traces and spans are structured.

- [Grafana Tempo documentation](https://grafana.com/docs/tempo/latest/): Full reference for Grafana’s open source tracing backend.

- [Span metrics using the metrics generator in Tempo](https://grafana.com/docs/tempo/latest/metrics-from-traces/span-metrics/span-metrics-metrics-generator/): Generate span metrics directly from traces with Tempo’s built-in metrics generator.

- [Span metrics using Grafana Alloy](https://grafana.com/docs/tempo/latest/metrics-from-traces/span-metrics/span-metrics-alloy/): Configure Alloy to export span metrics from OpenTelemetry (OTel) traces.

- [Multi-dimensional alerts](https://grafana.com/docs/grafana/latest/alerting/best-practices/multi-dimensional-alerts/): Learn how to trigger multiple alert instances per alert rule like in these examples.

- [Grafana SLO documentation](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/): Use span metrics to define Service Level Objectives (SLOs) in Grafana.
- [Trace sampling](https://grafana.com/docs/tempo/latest/set-up-for-tracing/instrument-send/set-up-collector/tail-sampling/#sampling): explore strategies and configuration in Grafana Tempo.

  {{< admonition type="note" >}}

  OpenTelemetry instrumentations can record metrics independently of spans.

  These [OTEL metrics](https://opentelemetry.io/docs/specs/semconv/general/metrics/) are not derived from traces and are not affected by sampling. They can serve as an alternative to span-derived metrics.

  {{< /admonition >}}

# Grafana Alerting tutorials

This section provides step-by-step tutorials to help you learn Grafana Alerting and explore key features through practical, easy-to-follow examples.

## Get started with Grafana Alerting

- [Create and receive your first alert](https://grafana.com/tutorials/alerting-get-started/)
- [Create multi-dimensional alerts and route them](https://grafana.com/tutorials/alerting-get-started-pt2/)
- [Group alert notifications](https://grafana.com/tutorials/alerting-get-started-pt3/)
- [Template your alert notifications](https://grafana.com/tutorials/alerting-get-started-pt4/)

## Additional tutorials

- [Route alerts using dynamic labels](https://grafana.com/tutorials/alerting-get-started-pt5/)
- [Link alerts to visualizations](https://grafana.com/tutorials/alerting-get-started-pt6/)
- [Create alerts with log data](https://grafana.com/tutorials/create-alerts-with-logs/)
- [Create alerts with InfluxDB and Flux queries](https://grafana.com/tutorials/create-alerts-from-flux-queries/)

# Templates

Use templating to customize, format, and reuse alert notification messages. Create more flexible and informative alert notification messages by incorporating dynamic content, such as metric values, labels, and other contextual information.

In Grafana, you have various options to template your alert notification messages:

1. [Alert rule annotations](#template-annotations)
   - Annotations add extra information, like `summary` and `description`, to alert instances for notification messages.
   - Template annotations to display query values that are meaningful to the alert, for example, the server name or the threshold query value.

1. [Alert rule labels](#template-labels)
   - Labels are used to differentiate an alert instance from all other alert instances.
   - Template labels to add an additional label based on a query value, or when the labels from the query are incomplete or not descriptive enough.
   - Avoid displaying query values in labels as this can create numerous alert instances—use annotations instead.

1. [Notification templates](#template-notifications)
   - Notification templates are used by contact points for consistent messaging in notification titles and descriptions.
   - Template notifications when you want to customize the appearance and information of your notifications.
   - Avoid using notification templates to add extra information to alert instances—use annotations instead.

{{< admonition type="tip" >}}
For a practical example of templating, refer to our [Getting Started with Templating tutorial](https://grafana.com/tutorials/alerting-get-started-pt4/).
{{< /admonition  >}}

This diagram illustrates the entire templating process, from querying labels and templating the alert summary and notification to the final alert notification message.

{{< figure src="/media/docs/alerting/how-notification-templates-works.png" max-width="1200px" caption="How templating works" >}}

In this diagram:

1. The alert rule query returns `12345`, along with the values of the `instance` and `job` labels.
1. This query result breaches the alert rule condition, firing the alert instance.
1. The alert instance generates an annotation summary, defined by the template used in the alert rule summary. In this case, it displays the value of the `instance` label: `server1`.
1. The Alertmanager receives the firing alert instance, including the final annotation summary, and determines the contact point that will process the alert.
1. The Alertmanager uses the contact point's notification template to format the message, then sends the notification to the configured destination(s)—an email address.

## Template annotations

[Annotations](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#annotations) can be defined in the alert rule to add extra information to alert instances.

When creating an alert rule, Grafana suggests several optional annotations, such as `description`, `summary`, and `runbook_url`, which help identify and respond to alerts. You can also create custom annotations.

Annotations are key-value pairs, and their values can contain a combination of text and template code that is evaluated when the alert fires.

Annotations can contain plain text, but you should template annotations if you need to display query values that are relevant to the alert, for example:

- Show the query value that triggers the alert.
- Include labels returned by the query that identify the alert.
- Format the annotation message depending on a query value.

Here’s an example of templating an annotation, which explains where and why the alert was triggered. In this case, the alert triggers when CPU usage exceeds a threshold, and the `summary` annotation provides the relevant details.

```
CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) for the last 5 minutes.
```

The outcome of this template would be:

```
CPU usage for Instance 1 has exceeded 80% (81.2345) for the last 5 minutes.
```

Implement annotations that provide meaningful information to respond to your alerts. Annotations are displayed in the Grafana alert detail view and are included by default in notifications.

For more details on how to template annotations, refer to [Template annotations and labels](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/).

## Template labels

[Labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#labels) are used to differentiate one alert instance from all other alert instances, as the set of labels uniquely identifies an alert instance. Notification policies and silences use labels to handle alert instances.

You can also template labels based on query results. This is helpful if the labels you get from your query aren't detailed enough. For instance:

- Add a new label to change how alerts are identified and grouped into different alert groups.
- Add a new label used by notification policies or silences to manage how the alert is handled.

Here’s an example of templating a new `env` label based on the value of a query label:

```go
{{- if eq $labels.instance "prod-server-1" -}}
production
{{- else if eq $labels.instance "staging-server-1" -}}
staging
{{- else -}}
development
{{- end -}}
```

For more details on how to template labels, refer to [Template annotations and labels](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/).

## Template notifications

[Notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/) allow you to customize the content of your notifications, such as the subject of an email or the body of a Slack message.

Notification templates differ from templating annotations and labels in the following ways:

- Notification templates are assigned to the **Contact point**, rather than the alert rule.
- If not specified, the contact point uses a default template that includes relevant alert information.
- The same template can be shared across multiple contact points, making it easier to maintain and ensuring consistency.
- Notification templates should not be used to add additional information to individual alerts—use annotations for that purpose.
- While both annotation/label templates and notification templates use the same templating language, the available variables and functions differ. For more details, refer to the [notification template reference](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/) and [annotation/label template reference](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/).

Here is an example of a notification template that summarizes all firing and resolved alerts in a notification group:

```go
{{ define "alerts.message" -}}
  {{ if .Alerts.Firing -}}
    {{ len .Alerts.Firing }} firing alert(s)
    {{ template "alerts.summarize" .Alerts.Firing }}
  {{- end }}
  {{- if .Alerts.Resolved -}}
    {{ len .Alerts.Resolved }} resolved alert(s)
    {{ template "alerts.summarize" .Alerts.Resolved }}
  {{- end }}
{{- end }}

{{ define "alerts.summarize" -}}
  {{ range . -}}
  - {{ index .Annotations "summary" }}
  {{ end }}
{{ end }}
```

The notification message to the contact point would look like this:

```
1 firing alert(s)
- The database server db1 has exceeded 75% of available disk space. Disk space used is 76%, please resize the disk size within the next 24 hours.

1 resolved alert(s)
- The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.
```

For more details, refer to [Template notifications](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/).

# Alert rule state and health

Each alert rule can generate one or more alert instances—one alert instance for each series or dimension, as shown in the [multi-dimensional alert example](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/).

Each alert instance of the same alert rule represents a different target and can be in a different state; for example, one alert instance may be **Normal** while another is **Alerting**.

{{< figure src="/media/docs/alerting/alert-rule-example-multiple-alert-instances.png" max-width="750px" alt="Multi dimensional alert rule. The alert rule state and alert rule health are determined by the state of the alert instances." >}}

The alert rule state and alert rule health are determined by the [state of the alert instances](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/#alert-instance-states).

## Alert rule states

An alert rule can be in either of the following states:

| State       | Description                                                                                          |
| ----------- | ---------------------------------------------------------------------------------------------------- |
| **Normal**  | None of the alert instances returned by the evaluation engine is in a `Pending` or `Alerting` state. |
| **Pending** | At least one alert instances returned by the evaluation engine is `Pending`.                         |
| **Firing**  | At least one alert instances returned by the evaluation engine is `Alerting`.                        |

## Alert rule health

An alert rule can have one of the following health statuses:

| State                  | Description                                                                                              |
| ---------------------- | -------------------------------------------------------------------------------------------------------- |
| **Ok**                 | No error when evaluating the alert rule.                                                                 |
| **Error**              | An error occurred when evaluating the alert rule.                                                        |
| **No Data**            | The alert rule query returns no data.                                                                    |
| **{status}, KeepLast** | The rule would have received another status but was configured to keep the last state of the alert rule. |

# How rules are evaluated within a group

Each evaluation group contains an **evaluation interval** that determines how frequently the rule is evaluated. For instance, the evaluation may occur every `10s`, `30s`, `1m`, `10m`, etc.

Rules in different evaluation groups can be evaluated simultaneously.

Rules within the same evaluation group can be evaluated simultaneously or sequentially, depending on the rule type:

- **Grafana-managed** rules within the same group are evaluated concurrently—they are evaluated at different times over the same evaluation interval but display the same evaluation timestamp.

- **Data source-managed** rules within the same group are evaluated sequentially, one after the other—this is useful to ensure that recording rules are evaluated before alert rules.

- **Grafana-managed rules [imported from data source-managed rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/alerting-migration/)** are also evaluated sequentially, in the order they appear in the original rule group. Grafana runs them as an evaluation sequence: each rule starts after the previous one finishes. This preserves the evaluation order when a rule depends on a preceding recording rule.

For Grafana-managed rules that depend on recording rule results, you can define an explicit evaluation order with a rule sequence. The sequence evaluates recording rules first, followed by alert rules, at a shared interval. For instructions, refer to [Configure rule sequences](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/rule-sequences/).

# No Data and Error states

Grafana Alerting implements the **No Data** and **Error** states to handle common scenarios when evaluating alert rules, and you can modify their behavior.

An alert instance can transition to these special states:

- [No Data state](#no-data-state) occurs when the alert rule query runs successfully but returns no data points.
- [Error state](#error-state) occurs when the alert rule fails to evaluate its query or queries successfully.

{{< admonition type="note" >}}

No Data and Error states are supported only for Grafana-managed alert rules.

{{< /admonition  >}}

{{< admonition type="tip" >}}
For common examples and practical guidance on handling **Error**, **No Data**, and **stale** alert scenarios, refer to the [Handle connectivity errors](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/connectivity-errors/) and [Handle missing data](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/missing-data/) guides.
{{< /admonition  >}}

## `Error` state

The **Error** state is triggered when the alert rule fails to evaluate its query or queries successfully.

This can occur due to evaluation timeouts (default: `30s`) or repeated failures (default: `3`) when querying the data source. The [`evaluation_timeout`](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#evaluation_timeout) and [`max_attempts`](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#max_attempts) options control these settings.

The **Error** state honors the [pending period](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#pending-period): the alert instance transitions `Normal → Pending → Error`. Set the pending period to `0` to skip the **Pending** state and enter **Error** immediately.

When an alert instance enters the **Error** state, Grafana, by default, triggers a new [`DatasourceError` alert](#no-data-and-error-alerts). You can control this behavior based on the desired outcome of your alert rule in [Modify the `No Data` or `Error` state](#modify-the-no-data-or-error-state).

## `No Data` state

The **No Data** state occurs when the alert rule query runs successfully but returns no data points at all.

The **No Data** state honors the [pending period](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#pending-period): the alert instance transitions `Normal → Pending → No Data`. Set the pending period to `0` to skip the **Pending** state and enter **No Data** immediately.

When an alert instance enters the **No Data** state, Grafana, by default, triggers a new [`DatasourceNoData` alert](#no-data-and-error-alerts). You can control this behavior based on the desired outcome of your alert rule in [Modify the `No Data` or `Error` state](#modify-the-no-data-or-error-state).

## Modify the `No Data` or `Error` state

{{< figure src="/media/docs/alerting/screenshot-configure-no-and-error-handling.png" alt="A screenshot of the configure no data and error handling section in the alert rule form." max-width="750px" >}}

In [Configure no data and error handling](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/#configure-no-data-and-error-handling), you can configure the behavior when the evaluation returns no data or all values are null:

{{< docs/shared lookup="alerts/modify-nodata-state.md" source="grafana" version="<GRAFANA_VERSION>" >}}

You can also configure the behavior when the evaluation returns an error or timeout:

{{< docs/shared lookup="alerts/modify-error-state.md" source="grafana" version="<GRAFANA_VERSION>" >}}

Note that when you configure the **No Data** or **Error** behavior to set the **Alerting** or **Normal** state, Grafana attempts to keep a stable set of fields under notification `Values`. If your query returns no data or an error, Grafana re-uses the latest known set of fields in `Values`, but will use `-1` in place of the measured value.

### Keep last state

The "Keep Last State" option helps mitigate temporary data source issues, preventing alerts from unintentionally firing, resolving, and re-firing.

However, in situations where strict monitoring is critical, relying solely on the "Keep Last State" option may not be appropriate. Instead, consider using an alternative or implementing additional alert rules to ensure that issues with prolonged data source disruptions are detected.

### `No Data` and `Error` alerts

When an alert rule evaluation results in a `No Data` or `Error` state, Grafana Alerting creates a new alert instance with the following additional labels:

- `alertname`: Either `DatasourceNoData` or `DatasourceError` depending on the state.
- `datasource_uid`: The UID of the data source that caused the state.
- `rulename`: The name of the alert rule that originated the alert.

Note that `DatasourceNoData` and `DatasourceError` alert instances are independent from the original alert instance. They have different labels, which means existing silences, mute timings, and notification policies applied to the original alert may not apply to them.

You can manage these alerts like regular ones by using their labels to apply actions such as adding a silence, routing via notification policies, and more.

If the alert rule is configured to send notifications directly to a selected contact point (instead of using notification policies), the `DatasourceNoData` and `DatasourceError` alerts are also sent to that contact point. Any additional notification settings defined in the alert rule, such as muting or grouping, are preserved.

### Reduce `No Data` or `Error` alerts

To minimize the number of **No Data** or **Error** state alerts received, try the following.

1. Use the **Keep last state** option. For more information, refer to the section below. This option allows the alert to retain its last known state when there is no data available, rather than switching to a **No Data** state.
1. For **No Data** alerts, you can optimize your alert rule by expanding the time range of the query. However, if the time range is too big, it affects the performance of the query and can lead to errors due to timeout.

   To minimize timeouts resulting in the **Error** state, reduce the time range to request less data every evaluation cycle.

1. Consider increasing the [pending period](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#pending-period). Since the **No Data** and **Error** states honor the pending period, a longer period can suppress `DatasourceNoData` and `DatasourceError` notifications for transient issues.

1. To reduce multiple notifications from **Error** alerts, define a [notification policy](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/) to handle all related alerts with `alertname=DatasourceError`, and filter and group errors from the same data source using the `datasource_uid` label.

1. Change the [evaluation timeout](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#evaluation_timeout) (default: `30s`) or the [retry mechanism (`max_attempts`)](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#max_attempts) settings. This should be a last resort, as it can affect the performance of all alert rules and may cause missed evaluations if the timeout is too long. For Grafana Cloud, open a support ticket from the [Cloud Portal](https://grafana.com/docs/grafana-cloud/account-management/support/#grafana-cloud-support-options).

   {{< admonition type="tip" >}}
   For common examples and practical guidance on handling **Error**, **No Data**, and **stale** alert scenarios, refer to the [Handle connectivity errors](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/connectivity-errors/) and [Handle missing data](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/missing-data/) guides.
   {{< /admonition  >}}

## `grafana_state_reason` for troubleshooting

Occasionally, an alert instance may be in a state that isn't immediately clear to everyone. For example:

- If "no data" handling is configured to transition to a state other than `No Data`.
- If "error" handling is configured to transition to a state other than `Error`.
- If the alert rule is deleted, paused, or updated in some cases, the alert instance also transitions to the `Normal` state.
- [Stale alert instances](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/) in the `Alerting` state transition to the `Normal` state when the series disappear.

In these situations, the evaluation state may differ from the alert state, and it might be necessary to understand the reason for being in that state when receiving the notification.

The `grafana_state_reason` annotation is included in these situations, providing the reason that explains why the alert instance transitioned to its current state. For example:

- If "no data" or "error" handling transitions to the `Normal` state, the `grafana_state_reason` annotation is included with the value **No Data** or **Error**, respectively.
- If the alert rule is deleted or paused, the `grafana_state_reason` is set to **Paused** or **RuleDeleted**. For some updates, it is set to **Updated**.
- [Stale alert instances](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/) in the `Normal` state include the `grafana_state_reason` annotation with the value **MissingSeries**.

# Stale alert instances

An alert instance is considered **stale** if the alert rule query returns data but its dimension (or series) has disappeared for a number of consecutive evaluation intervals (2 by default).

This is different from the [**No Data** state](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/#no-data-state), which occurs when the alert rule query runs successfully but returns no dimensions (or series) at all.

A stale alert instance transitions to the **Normal (MissingSeries)** state as **Resolved**, and is then evicted:

| Eval. Interval   | 1   | 2               | 3                                        | 4   |
| :--------------- | :-- | :-------------- | :--------------------------------------- | :-- |
| Alert instance A | ✔  | ✔              | ✔                                       | ✔  |
| Alert instance B | ✔  | `MissingSeries` | ️`Normal(MissingSeries)` 📩<sup>\*</sup> |     |

{{< admonition type="note" >}}

Stale alert instances are supported only for Grafana-managed alert rules.

{{< /admonition  >}}

## How Grafana handles stale alert instances

The process for handling stale alert instances is as follows:

1. The alert rule runs and returns data for some label sets.

1. An alert instance that previously existed is now missing.

1. Grafana keeps the previous state of the alert instance for the number of evaluation intervals specified in [Missing series evaluations to resolve](#configure-missing-series-evaluations-to-resolve).

1. If it remains missing after the specified number of evaluation intervals (2 by default), it transitions to the **Normal** state and sets **MissingSeries** in the [`grafana_state_reason` annotation](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/#grafana_state_reason-for-troubleshooting).

   Stale alert instances in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**, and are routed for notifications like other resolved alerts.

1. The alert instance is removed from the UI.

{{< admonition type="tip" >}}

For common examples and practical guidance on handling **No Data** and **stale** alert scenarios, see [Handling missing data](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/missing-data/).

{{< /admonition  >}}

## Configure Missing series evaluations to resolve

In [Configure no data and error handling > Missing series evaluations to resolve](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/#configure-no-data-and-error-handling), you can set how many consecutive evaluation intervals must pass without data for a given dimension before the alert instance is marked as stale and resolved.

If you don't specify a value, Grafana uses the **default of 2 evaluation intervals**.

# Labels and annotations

Labels and annotations add additional information about an alert using key/value pairs:

- [Labels](#labels) are used to differentiate an alert from all other alerts and decide how to manage them.
- [Annotations](#annotations) provide extra details for alert responders to help them understand and address potential issues.

## Labels

{{< shared id="labels-basics" >}}

**Labels** are unique identifiers of an [alert instance](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals#alert-instances). You can use them for searching, silencing, and routing notifications.

Examples of labels are `server=server1` or `team=backend`. Each alert rule can have more than one label and the complete set of labels for an alert rule is called its label set. It is this label set that identifies the alert.

{{< /shared >}}

For example, an alert instance might have the label set `{alertname="High CPU usage",server="server1"}` while another alert instance might have the label set `{alertname="High CPU usage",server="server2"}`. These are two separate alert instances because although their `alertname` labels are the same, their `server` labels are different.

{{< figure alt="Image shows an example of an alert instance and the labels used on the alert instance." src="/static/img/docs/alerting/unified/multi-dimensional-alert.png" >}}

Labels are a fundamental component of alerting:

- The complete set of labels for an alert is what uniquely identifies an alert instance.
- The alerting UI shows labels for every alert instance generated during evaluation of that rule.
- [Notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/) and [silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/) use labels to match alert instances and route them to contact points or stop their notifications.
- Contact points can include information from labels in notification messages.

### Label types

An alert's label set can contain three types of labels:

**User-configured labels**

Labels that you manually configure in the alert rule to identify the generated alert instances and manage the alerts. Common custom labels, depending on the use case, are: `severity`, `priority`, `team`, and `service`.

Additionally, you can use a [template](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/templates/) to customize the label value and generate dynamic values from query data.

**Query labels**

Query labels are labels returned by the data source query.

{{< figure src="/media/docs/alerting/query-labels-and-values.png" max-width="1200px" caption="An alert rule query returning labels from the query." >}}

Query labels can generate multiple alert instances from the same alert rule, helping to distinguish alerts from different data. In this example, the `instance` label generates an alert instance for each server.

**Reserved labels**

Reserved labels are automatically added by Grafana:

- `alertname`: the name of the alert rule.
- `grafana_folder`: the title of the folder containing the alert.

Labels prefixed with `grafana_` are reserved by Grafana for special use. You can disable reserved labels via the [`unified_alerting.reserved_labels`](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana#unified_alertingreserved_labels) option.

{{<admonition type="note">}}

Two alert rules cannot produce alert instances with the same labels. If two alert rules have the same labels such as `foo=bar,bar=baz` and `foo=bar,bar=baz` then one of the generated alert instances is discarded.

Ensure the label set for an alert does not have two or more labels with the same name.

- If a configured label has the same name as a data source query label, it replaces the data source label.
- If a configured label has the same name as a reserved label, it is omitted.
  {{</admonition>}}

{{< collapse title="Label key format" >}}

Grafana has a built-in Alertmanager that supports both Unicode label keys and values. If you are using an external Prometheus Alertmanager, label keys must be compatible with their [data model](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
This means that label keys must only contain _ASCII letters_, _numbers_, and _underscores_.
Label keys must also be matched by the regular expression `[a-zA-Z_][a-zA-Z0-9_]*`.
Any invalid characters are removed or replaced by the Grafana alerting engine before being sent to the external Alertmanager according to the following rules:

- Whitespace is removed.
- ASCII characters are replaced with `_`.
- All other characters are replaced with their lower-case hex representation.
  If this is the first character, it's prefixed with `_`.

Example: A label key/value pair `Alert! 🔔="🔥"` will become `Alert_0x1f514="🔥"`.

If multiple label keys are sanitized to the same value, the duplicates have a short hash of the original label appended as a suffix.

{{< /collapse >}}

## Annotations

{{< shared id="annotations-basics" >}}

Annotations add additional information to alert instances, helping responders identify and address potential issues.

Create clear and self-explanatory annotations so that first responders can investigate without needing deeper knowledge of the alert setup.

Annotations are displayed in Grafana and are included by default in notifications. Grafana provides several optional annotations that you can edit:

- `summary`: A short summary of what the alert has detected and why.
- `description`: A detailed description of what happened and what the alert does.
- `runbook_url`: The runbook page to guide operators managing a potential incident.
- `__dashboardUid__` and `__panelId__`: [Link the alert to a dashboard and panel](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/link-alert-rules-to-panels/) to facilitate alert investigation.

{{< /shared >}}

Unlike labels, annotations do not affect [notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/) or [silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/). For example, if `team=backend` exists only in an annotation, a child policy that matches `team=backend` does not receive the alert; the [Default notification policy](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/) handles it instead.

You can edit annotations such as `summary` to explain why the alert was triggered:

```
CPU usage has exceeded 80% for the last 5 minutes.
```

Edit the `description` annotation to provide more context and how to respond:

```
The web server's CPU has exceeded 80% for more than 5 minutes.

This indicates that the system is under heavy load and may result in an outage.

Consider scaling the server's resources and investigating bottlenecks.
```

Like labels, annotations can use a [template](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/templates/) to include dynamic data from queries.

# Queries and conditions

In Grafana, queries fetch and transform data from data sources, which include databases like MySQL or PostgreSQL, time series databases like Prometheus or InfluxDB, and services like Amazon CloudWatch or Azure Monitor.

An alert rule defines the following components:

- A [query](#data-source-queries) that specifies the data to retrieve from a data source, with the syntax depending on the type of data source used.
- A [condition](#alert-condition) that must be met before the alert rule fires.
- Optional [expressions](#advanced-options-expressions) to perform transformations on the retrieved data.

Alerting periodically runs the queries and expressions, evaluating the condition. If the condition is breached, an alert instance is triggered for each time series.

## Data source queries

Alerting queries are the same as the queries used in Grafana panels, but Grafana-managed alerts are limited to querying [data sources that have Alerting enabled](/grafana/plugins/data-source-plugins/?features=alerting).

Queries in Grafana can be applied in various ways, depending on the data source and query language being used. Each data source’s query editor provides a customized user interface to help you write queries that take advantage of its unique capabilities. For details about query editors and syntax in Grafana, refer to [Query and transform data](/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/).

Alerting can work with two types of data:

1. **Time series data** — The query returns a collection of time series, where each series must be [reduced](#reduce) to a single numeric value for evaluating the alert condition.
1. **Tabular data** — The query must return data in a table format with only one numeric column. Each row must have a value in that column, used to evaluate the alert condition. See a [tabular data example](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/table-data/).

Each time series or table row is evaluated as a separate [alert instance](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/#alert-instances).

{{< figure src="/media/docs/alerting/alerting-query-conditions-default-options.png" max-width="750px" caption="Alert query using the Prometheus query editor and alert condition" >}}

## Alert condition

The alert condition is the query or expression that determines whether the alert fires or not depending whether the value satisfies the specified comparison. There can be only one condition which determines the triggering of the alert.

If the queried data meets the defined condition, Grafana fires the alert.

When using **Default options**, the `When` input [reduces the query data](#reduce), and the last input defines the threshold condition.

When using **Advanced options**, you have to choose one of your queries or expressions as the alert condition.

## Advanced options: Expressions

Expressions are only available for Grafana-managed alerts and when the **Advanced options** are enabled.

In Grafana, expressions allow you to perform calculations, transformations, or aggregations on queried data. They modify existing metrics through mathematical operations, functions, or logical expressions.

With expression queries, you can perform tasks such as calculating the percentage change between two values, applying functions like logarithmic or trigonometric functions, aggregating data over specific time ranges or dimensions, and implementing conditional logic to handle different scenarios.

{{< figure src="/media/docs/alerting/alert-rule-expressions.png" max-width="750px" caption="Alert rule expressions" >}}

The following expressions are available:

### Reduce

Aggregates time series values within the selected time range into a single number.

Reduce takes one or more time series and transforms each series into a single number, which can then be compared in the alert condition.

The following aggregations functions are included: `Min`, `Max`, `Mean`, `Median`, `Sum`, `Count`, and `Last`. For more details, refer to the [Reduce documentation](/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/expression-queries/#reduce).

### Math

Performs free-form math functions/operations on time series data and numbers. For example, `$A + 1` or `$A * 100`.

If queries being compared have **multiple series in their results**, series from different queries are matched(joined) if they have the same labels. For example:

{{< docs/shared lookup="alerts/math-example.md" source="grafana" version="<GRAFANA_VERSION>" >}}

In this case, only series with matching labels are joined, and the operation is calculated between them.

For additional scenarios on how Math handles different data types, refer to the [Math documentation](/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/expression-queries/#math).

You can also use a Math expression to define the **alert condition**. For example:

- `$B > 70` should fire if the value of B (query or expression) is more than 70.
- `$B < $C * 100` should fire if the value of B is less than the value of C multiplied by 100.
- Compare matching series from two queries, as shown in the [dynamic threshold example](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/dynamic-thresholds/).

### Resample

Realigns a time range to a new set of timestamps. This is useful when comparing time series data from different data sources where the timestamps would otherwise not align.

For more details, refer to the [Resample documentation](/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/expression-queries/#resample).

### SQL

Transforms the results of previous queries or expressions using MySQL-like syntax, which is useful for filtering rows, joining data from multiple queries, and aggregating results before evaluating the alert condition.

For more details, refer to the [SQL expressions documentation](/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/sql-expressions/).

### Threshold

Compares single numbers from previous queries or expressions (e.g., `$A`, `$B`) to a specified condition. It's often used to define the alert condition.

The threshold expression allows the comparison between two single values. Available threshold functions are:

- **Is above**: `$A > 5`
- **Is below**: `$B < 3`
- **Is equal to**: `$A == 2`
- **Is not equal to**: `$B != 4`
- **Is above or equal to**: `$A >= 8`
- **Is below or equal to**: `$B <= 16`
- **Is within range**: `$A > 0 AND $A < 10`
- **Is outside range**: `$B < 0 OR $B > 100`
- **Is within range included**: `$A >= 0 AND $A <= 10`
- **Is outside range included**: `$B <= 0 OR $B >= 100`

A threshold returns `0` when the condition is false and `1` when true.

If the threshold is set as the alert condition, the alert fires when the threshold returns `1`.

### Recovery threshold

To reduce the noise from flapping alerts, you can set a recovery threshold so that the alert returns to the `Normal` or `Recovering` state only after the recovery threshold is crossed.

Flapping alerts occur when the query value repeatedly crosses above and below the alert threshold, causing frequent state changes. This results in a series of firing-resolved-firing notifications and a noisy alert state history.

For example, if you have an alert for latency with a threshold of 1000ms and the number fluctuates around 1000 (say 980 -> 1010 -> 990 -> 1020, and so on), then each of those might trigger a notification:

- 980 -> 1010 triggers a firing alert.
- 1010 -> 990 triggers a resolving alert.
- 990 -> 1020 triggers a firing alert again.

To prevent this, you can set a recovery threshold to define two thresholds instead of one:

1. An alert transitions to the `Pending` or `Alerting` state when it crosses the alert threshold.
1. It then transitions to the `Recovering` or `Normal` state only when it crosses the recovery threshold.

In the previous example, setting the recovery threshold to 900ms means the alert only resolves when the latency falls below 900ms:

- 980 -> 1010 triggers a firing alert.
- 1010 -> 990 does not resolve the alert, keeping it in the firing state.
- 990 -> 1020 keeps the alert in the firing state.

The recovery threshold mitigates unnecessary alert state changes and reduces alert noise.

{{< collapse title="Classic condition (legacy)" >}}

#### Classic condition (legacy)

Classic conditions exist mainly for compatibility reasons and should be avoided if possible.

Classic condition checks if any time series data matches the alert condition. It always produce one alert instance only, no matter how many time series meet the condition.

| Condition operators | How it works                                                                                                                                                                                                                                                |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `and`               | Two conditions before and after must be true for the overall condition to be true.                                                                                                                                                                          |
| `or`                | If one of conditions before and after are true, the overall condition is true.                                                                                                                                                                              |
| `logic-or`          | If the condition before `logic-or` is true, the overall condition is immediately true, without evaluating subsequent conditions. For instance, `TRUE and TRUE logic-or FALSE and FALSE` evaluate to `TRUE`, because the preceding condition returns `TRUE`. |

The following aggregation functions are also available to further refine your query.

| Function           | What it does                                                                    |
| ------------------ | ------------------------------------------------------------------------------- |
| `avg`              | Displays the average of the values                                              |
| `min`              | Displays the lowest value                                                       |
| `max`              | Displays the highest value                                                      |
| `sum`              | Displays the sum of all values                                                  |
| `count`            | Counts the number of values in the result                                       |
| `last`             | Displays the last value                                                         |
| `median`           | Displays the median value                                                       |
| `diff`             | Displays the difference between the newest and oldest value                     |
| `diff_abs`         | Displays the absolute value of diff                                             |
| `percent_diff`     | Displays the percentage value of the difference between newest and oldest value |
| `percent_diff_abs` | Displays the absolute value of `percent_diff`                                   |
| `count_non_null`   | Displays a count of values in the result set that aren't `null`                 |

{{< /collapse >}}

# Contact points

Contact points contain the configuration for sending alert notifications. You can assign a contact point either in the alert rule or notification policy options.

A contact point includes one or more contact point integrations for sending alert notifications, such as:

{{< column-list >}}

- Alertmanager
- Amazon SNS
- Cisco Webex Teams
- DingDing
- Discord
- Email
- Google Chat
- Grafana IRM
- Jira
- Kafka REST Proxy
- Line
- Microsoft Teams
- MQTT
- Opsgenie
- PagerDuty
- Pushover
- Sensu Go
- Slack
- Telegram
- Threema Gateway
- VictorOps
- Webhook
- WeCom

{{< /column-list >}}

For example, a contact point could contain a PagerDuty integration; an email and Slack integration; or a PagerDuty integration, a Slack integration, and two email integrations. You can also configure a contact point with no integrations; in which case no notifications are sent.

Each contact point integration can also define the notification message to be sent, which can use the predefined message, a custom message, or notification templates.

For a complete list of supported integrations and more details about contact points, refer to [Configure contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points).

# Group alert notifications

Grouping in Grafana Alerting allows you to batch relevant alerts together into a smaller number of notifications. This is particularly important if notifications are delivered to first-responders, such as engineers on-call, where receiving lots of notifications in a short period of time can be overwhelming. In some cases, it can negatively impact a first-responder's ability to respond to an incident. For example, consider a large outage where many of your systems are down. In this case, grouping can be the difference between receiving 1 phone call and 100 phone calls.

{{< admonition type="tip" >}}
For a practical example of grouping, refer to our [Getting Started with Grouping tutorial](https://grafana.com/tutorials/alerting-get-started-pt3/).
{{< /admonition  >}}

## Group notifications

Grouping combines similar alert instances within a specific period into a single notification, reducing alert noise.

Grafana Alerting uses **alert group** for this temporary notification batch. In [Grafana IRM](/docs/grafana-cloud/alerting-and-irm/irm/use/respond-to-alerts/), the same term refers to a persistent object with a status, responders, and an escalation chain.

In the [notification policy](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/), you can configure how to group multiple alerts into a single notification:

- The `Group by` option specifies the criteria for grouping incoming alerts within the policy. The default is by alert rule.
- [Timing options](#timing-options) determine when and how often to send the notification.

{{< figure src="/media/docs/alerting/alerting-notification-policy-diagram-with-labels-v3.png" max-width="750px" alt="A diagram about the components of a notification policy, including labels and groups" >}}

Alert instances are grouped together if they have the same exact label values for the labels configured in the `Group by` option.

For example, given the `Group by` option set to the `team` label:

- `alertname:foo, team=frontend`, and `alertname:bar, team=frontend` are in one group.
- `alertname:foo, team=frontend`, and `alertname:qux, team=backend` are in another group.

### Group by alert rule or labels

By default, notification policies in Grafana group alerts by the alert rule. Specifically, they are grouped using the `alertname` and `grafana_folder` labels, as alert rule names are not unique across folders.

If you want to group alerts by other labels, something other than the alert rule, change the `Group by` option to any other combination of labels.

### A single group for all alerts

If you want to group all alerts handled by the notification policy in a single group (without grouping notifications by alert rule or other labels), leave `Group by` empty in the Default policy.

### Disable grouping

If you want to receive every alert as a separate notification, you can do so by grouping by a special label called `...`, ensuring that other labels are not present.

## Timing options

In the notification policy, you can also configure how often notifications are sent for each [group of alerts](#group-notifications). There are three distinct timers applied to groups within the notification policy:

- **[Group wait](#group-wait)**: the time to wait before sending the first notification for a new group of alerts.
- **[Group interval](#group-interval)**: the time to wait before sending a notification about changes in the alert group.
- **[Repeat interval](#repeat-interval)**: the time to wait before sending a notification if the group has not changed since the last notification.

These timers reduce the number of notifications sent. By delaying the delivery of notifications, incoming alerts can be grouped into just one notification instead of many.

{{< figure src="/media/docs/alerting/alerting-timing-options-flowchart-v2.png" max-width="750px" alt="A basic sequence diagram of the notification policy timers" caption="A basic sequence diagram of the notification policy timers" >}}

<!--
flowchart LR
    A((First alert)) -///-> B
    B[Group wait <br/>  notification] -///-> C
    B -- no changes -///-> D
    C[Group interval <br/>  notification] -- no changes -///-> D
    C -- group changes -///-> C
    D[Repeat interval <br/>  notification]
-->

### Group wait

**Default**: 30 seconds

Group wait is the duration Grafana waits before sending the first notification for a new group of alerts.

This option helps reduce the number of notifications sent for related alerts occurring within a short time frame. The longer the group wait, the more time other alerts have to be included in the initial notification of the new group. The shorter the group wait, the earlier the first notification is sent, but at the risk of not including some alerts.

If an alert is resolved before the duration elapses, no notification is sent for that alert. This reduces noise from flapping alerts.

**Example**

Consider a notification policy that:

- Matches all alert instances with the `team` label—matching labels equals to `team=~.+`.
- Groups notifications by the `team` label—one group for each distinct `team`.
- Sets the Group wait timer to `30s`.

| Time               | Incoming alert instance        | Notification policy group | Number of instances |                                                                         |
| ------------------ | ------------------------------ | ------------------------- | ------------------- | ----------------------------------------------------------------------- |
| 00:00              | `alertname=f1` `team=frontend` | `frontend`                | 1                   | Starts the group wait timer of the `frontend` group.                    |
| 00:10              | `alertname=f2` `team=frontend` | `frontend`                | 2                   |                                                                         |
| 00:20              | `alertname=b1` `team=backend`  | `backend`                 | 1                   | Starts the group wait timer of the `backend` group.                     |
| 00:30<sup>\*</sup> |                                | `frontend`                | 2                   | Group wait elapsed. <br/> Send initial notification reporting 2 alerts. |
| 00:35              | `alertname=b2` `team=backend`  | `backend`                 | 2                   |                                                                         |
| 00:40              | `alertname=b3` `team=backend`  | `backend`                 | 3                   |                                                                         |
| 00:50<sup>\*</sup> |                                | `backend`                 | 3                   | Group wait elapsed. <br/> Send initial notification reporting 3 alerts. |

### Group interval

**Default**: 5 minutes

If an alert was too late to be included in the first notification due to group wait, it is included in subsequent notifications after group interval.

Group interval is the duration to wait before sending notifications about group changes. A group change occurs when:

- A new firing alert is added to the group.
- An existing alert is resolved.

**Example**

Here are the related excerpts from the previous example:

| Time               | Incoming alert instance | Notification policy group | Number of instances |                                                                                                         |
| ------------------ | ----------------------- | ------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------- |
| 00:30<sup>\*</sup> |                         | `frontend`                | 2                   | Group wait elapsed and starts Group interval timer. <br/> Send initial notification reporting 2 alerts. |
| 00:50<sup>\*</sup> |                         | `backend`                 | 3                   | Group wait elapsed and starts Group interval timer. <br/> Send initial notification reporting 3 alerts. |

And below is the continuation of the example setting the Group interval timer to 5 minutes:

| Time               | Incoming alert instance        | Notification policy group | Number of instances |                                                                                                |
| ------------------ | ------------------------------ | ------------------------- | ------------------- | ---------------------------------------------------------------------------------------------- |
| 01:30              | `alertname=f3` `team=frontend` | `frontend`                | 3                   |                                                                                                |
| 02:30              | `alertname=f4` `team=frontend` | `frontend`                | 4                   |                                                                                                |
| 05:30<sup>\*</sup> |                                | `frontend`                | 4                   | Group interval elapsed and resets timer. <br/> Send one notification reporting 4 alerts.       |
| 05:50<sup>\*</sup> |                                | `backend`                 | 3                   | Group interval elapsed and resets timer. <br/> No group changes, and do not send notification. |
| 08:00              | `alertname=f4` `team=backend`  | `backend`                 | 4                   |                                                                                                |
| 10:30<sup>\*</sup> |                                | `frontend`                | 4                   | Group interval elapsed and resets timer. <br/> No group changes, and do not send notification. |
| 10:50<sup>\*</sup> |                                | `backend`                 | 4                   | Group interval elapsed and resets timer. <br/> Send one notification reporting 4 alerts.       |

**How it works**

Once the first notification has been sent for a new group of alerts, the group interval timer starts.

When the group interval timer elapses, the system resets the group interval timer and sends a notification only if there were group changes. This process repeats until there are no more alerts.

It's important to note that an alert instance exits the group after being resolved and notified of its state change.

When the group interval timer elapses and no alerts remain, the group is deleted. The [group wait timer](#group-wait) will then start again the next time a new alert arrives.

### Repeat interval

**Default**: 4 hours

Repeat interval acts as a reminder that alerts in the group are still firing.

The repeat interval timer decides how often notifications are sent (or repeated) if the group has not changed since the last notification.

**How it works**

Repeat interval is evaluated every time the group interval resets. If the alert group has not changed and the time since the last notification was longer than the repeat interval, then a notification is sent as a reminder that the alerts are still firing.

Repeat interval must not only be greater than or equal to group interval, but also must be a multiple of Group interval. If Repeat interval is not a multiple of group interval it is coerced into one. For example, if your Group interval is 5 minutes, and your Repeat interval is 9 minutes, the Repeat interval is rounded up to the nearest multiple of 5 which is 10 minutes.

The maximum duration of the repeat interval is 5 days, constrained by the default value of the `notification_log_retention` setting in Grafana.

**Example**

Here are the related excerpts from the previous example:

| Time               | Incoming alert instance | Notification policy group | Number of instances |                                                          |
| ------------------ | ----------------------- | ------------------------- | ------------------- | -------------------------------------------------------- |
| 05:30<sup>\*</sup> |                         | `frontend`                | 4                   | Group interval resets. <br/> Send the last notification. |
| 10:50<sup>\*</sup> |                         | `backend`                 | 4                   | Group interval resets. <br/> Send the last notification. |

And below is the continuation of the example setting the Repeat interval timer to 4 hours:

| Time     | Incoming alert instance | Notification policy group | Number of instances |                                                                                                                                                             |
| -------- | ----------------------- | ------------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 04:05:30 |                         | `frontend`                | 4                   | Group interval resets. The time since the last notification was no longer than the repeat interval.                                                         |
| 04:10:30 |                         | `frontend`                | 4                   | Group interval resets. The time since the last notification was longer than the repeat interval. <br/> Send one notification reminding the 4 firing alerts. |
| 04:10:50 |                         | `backend`                 | 4                   | Group interval resets. The time since the last notification was no longer than the repeat interval.                                                         |
| 04:15:50 |                         | `backend`                 | 4                   | Group interval resets. The time since the last notification was longer than the repeat interval. <br/> Send one notification reminding the 4 firing alerts. |

# Notification policies

Notification policies provide you with a flexible way of designing how to handle notifications and minimize alert noise.

Using label matchers, alert instances are [routed to notification policies](#routing). The notification policy can then [group multiple alert instances into a single notification](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/) and deliver it to the contact point.

{{< figure src="/media/docs/alerting/how-alerting-works.png" max-width="750px" alt="How Alerting works" >}}

Notification policies are _not_ a list, but rather are structured according to a [tree structure](https://en.wikipedia.org/wiki/Tree_structure):

- The root of the notification policy tree is the **Default notification policy**.
- Each policy can have child policies.
- Each policy can have sibling policies, sharing the same parent and hierarchical level.

Each policy consists of a set of label matchers (0 or more) that specify which alerts they are or aren't interested in handling. A matching policy refers to a notification policy with label matchers that match the alert instance’s labels.

{{< collapse title="How label matching works" >}}

Use [labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/) and label matchers to link alert rules to [notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/) and [silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/). This allows for a flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.

A label matcher consists of 3 distinct parts, the **label**, the **value** and the **operator**.

- The **Label** field is the name of the label to match. It must exactly match the label name.

- The **Value** field matches against the corresponding value for the specified **Label** name. How it matches depends on the **Operator** value.

- The **Operator** field is the operator to match against the label value. The available operators are:

  | Operator | Description                                        |
  | -------- | -------------------------------------------------- |
  | `=`      | Select labels that are exactly equal to the value. |
  | `!=`     | Select labels that are not equal to the value.     |
  | `=~`     | Select labels that regex-match the value.          |
  | `!~`     | Select labels that do not regex-match the value.   |

{{< admonition type="note" >}}
If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match to link a rule to a policy.
{{< /admonition >}}

By default, Grafana uses a single notification policy tree for all alert routing. You can also create [multiple notification policy trees](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy/#manage-multiple-notification-policy-trees) to split routing logic into separate, independently managed trees.

**Label matching example**

If you define the following set of labels for your alert:

`{ foo=bar, baz=qux, id=12 }`

then:

- A label matcher defined as `foo=bar` matches this alert rule.
- A label matcher defined as `foo!=bar` does _not_ match this alert rule.
- A label matcher defined as `id=~[0-9]+` matches this alert rule.
- A label matcher defined as `baz!~[0-9]+` matches this alert rule.
- Two label matchers defined as `foo=bar` and `id=~[0-9]+` match this alert rule.

**Exclude labels**

You can also write label matchers to exclude labels.

Here is an example that shows how to exclude the label `Team`. You can choose between any of the values below to exclude labels.

| Label  | Operator | Value |
| ------ | -------- | ----- |
| `team` | `=`      | `""`  |
| `team` | `!~`     | `.+`  |
| `team` | `=~`     | `^$`  |

{{< /collapse >}}

[//]: <> ({{< docs/shared lookup="alerts/how_label_matching_works.md" source="grafana" version="<GRAFANA_VERSION>" >}})

{{< figure src="/media/docs/alerting/notification-routing.png" max-width="750px" caption="Matching alert instances with notification policies" alt="Example of a notification policy tree" >}}

{{< admonition type="tip" >}}
For a practical example of routing with notification policies, refer to our [Getting Started with Alert Instances and Notification Routing tutorial](https://grafana.com/tutorials/alerting-get-started-pt2/).
{{< /admonition  >}}

## Routing

To determine which notification policies handle an alert instance, the system looks for matching policies starting from the top of the tree—beginning with the default notification policy.

If a matching policy is found, the system continues to evaluate its child policies in the order they are displayed. If a child policy matches the alert, the system then evaluates its child policies recursively until no more matching child policies are found. In this case, only the deepest matching child policy handles the alert instance.

By default, once a matching policy is found, the system does not continue to look for sibling policies. If you want sibling policies of one matching policy to handle the alert instance as well, then enable **Continue matching siblings** on the particular matching policy.

{{< admonition type="note" >}}

The default notification policy matches all alert instances. It always handles alert instances if there are no child policies or if none of the child policies match the alert instance's labels—this prevents any alerts from being missed.

If alerts use multiple labels, these labels must also be present in a notification policy to match and route notifications to a specific contact point.

{{< /admonition >}}

{{< collapse title="Routing example" >}}

Here's a breakdown of the previous example:

**Pod stuck in CrashLoop** does not have a `severity` label, so none of its child policies are matched. It does have a `team=operations` label, so the first policy is matched.

The `team=security` policy is not a match and **Continue matching siblings** was not configured for that policy.

**Disk Usage – 80%** has both a `team` and `severity` label, and matches a child policy of the operations team.

{{< admonition type="note" >}}
When an alert matches both a parent policy and a child policy (like it does in this case), the routing follows the child policy (`severity`) as it provides a more specific match.
{{< /admonition >}}

**Unauthorized log entry** has a `team` label but does not match the first policy (`team=operations`) since the values are not the same, so it will continue searching and match the `team=security` policy. It does not have any child policies, so the additional `severity=high` label is ignored.

{{< /collapse >}}

This routing and tree structure makes it convenient to organize and handle alerts for dedicated teams, while also narrowing down specific cases within the team by applying additional labels.

## Inheritance

In addition to child policies being a useful concept for routing alert instances, they also inherit properties from their parent policy. This also applies to child policies of the default notification policy.

By default, a child policy inherits the following notification properties from its parent:

- [Contact point](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/contact-points/)
- [Grouping options](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/)
- [Timing options](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/#timing-options)

Then, each policy can overwrite these properties if needed.

The inheritance of notification properties, together with the routing process, is an effective method for grouping related notifications and handling specific cases through child policies.

**Inheritance example**

{{< figure src="/media/docs/alerting/notification-inheritance.png" max-width="750px" alt="Simple example inhering notification settings" >}}

This example shows how the notification policy tree from the previous example allows the child policies of the `team=operations` to inherit its contact point. In this way, you can avoid specifying the same contact point multiple times for each child policy.

# Alerting on forecasts

A forecast predicts the future state of a signal, such as expected disk space utilization over the next month. Common forecast use cases include:

- **Capacity planning:** Detect when resources such as disk space, memory, CPU, or database connections are expected to approach their limits.
- **Demand and business planning:** Analyze expected demand, costs, production, or other business metrics.

Standard alert rules evaluate the current state of a signal. A forecast-based alert compares a predicted future value with a threshold condition, allowing you to act before this happens.

{{< figure src="/media/docs/alerting/forecast-vs-standard-alert2.svg" alt="Comparison of standard and forecast-based alerts." >}}

Different forecasting tools provide different forecasting approaches, which are often chosen based on the behavior of the signal.

- **Linear forecasts** are used for signals that follow a relatively steady trend.
- **Seasonal forecasts** are better suited to signals with recurring patterns, such as daily or weekly traffic.

The following examples show how to create alerts using both types of forecasts.

## Linear forecast with `predict_linear`

When a signal follows a relatively steady trend, you can use the PromQL [`predict_linear`](https://prometheus.io/docs/prometheus/latest/querying/functions/#predict_linear) function to predict its future value.

`predict_linear` performs a linear regression over the values in a range vector and extrapolates the resulting trend into the future. The function takes two arguments:

```promql
predict_linear(<range-vector>, <seconds>)
```

- `<range-vector>` provides the historical data used to calculate the trend.
- `<seconds>` specifies how far into the future to predict, in seconds.

For example:

```promql
predict_linear(<metric>[14d], 7 * 24 * 3600)
```

This uses the last `14` days of data to calculate a linear trend and predicts the value `7` days from the current evaluation time.

### How `predict_linear` works

The calculation is based on a simple linear regression. It calculates the slope from the historical data and extrapolates that slope into the future:

```text
predicted_value = value_at_evaluation_time + slope * time
```

The result is a **single predicted value at the requested future time**, rather than a forecast series covering the entire future period.

#### When to use `predict_linear`

`predict_linear` is useful when a signal follows a relatively steady trend, such as disk utilization that is consistently increasing.

{{< figure src="/media/docs/alerting/alerting_on_forecast_trend_signal2.png" alt="A stable trend signal suitable for predict_linear forecasts." >}}

It is not suitable for signals with strong recurring patterns. For example, if a signal follows daily or weekly cycles, a straight-line projection might produce values that do not reflect the signal's normal behavior.

Another consideration is that `predict_linear` assumes the current trend continues. A sudden signal change can make the extrapolated trend inaccurate and lead to an inaccurate prediction.

### Alert on a future predicted value

You can compare the predicted value with a threshold directly in an alert rule:

```promql
predict_linear(<metric>[14d], 7 * 24 * 3600) > 85
```

This evaluates whether the predicted value **seven days from now** exceeds `85`.

The following example uses `predict_linear` to forecast disk utilization for a PostgreSQL persistent volume claim:

```promql
predict_linear(
  max by (persistentvolumeclaim) (
    100 * (
      1 - kubelet_volume_stats_available_bytes{persistentvolumeclaim="<id>"}
      /
      kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="<id>"}
    )
  )[14d:1h],
  7 * 24 * 3600
)
```

The query works as follows:

1. It calculates the disk utilization percentage for the volume.
2. `predict_linear` uses the previous `14` days of disk utilization to predict the value `7` days into the future.
3. The subquery `[14d:1h]` evaluates the historical data at `1`-hour intervals. This resolution is sufficient for a forecast over a long period.
4. The alert rule uses a threshold condition that fires when the predicted utilization exceeds `85%`.
5. The alert rule evaluates every `12` hours because disk utilization changes relatively slowly.

{{< admonition type="tip" >}}
You can explore [this alert example in Grafana Play](https://play.grafana.org/alerting/grafana/qp_pvc_disk_capacity_forecast/view?tech=docs&pg=alerting-examples&plcmt=learn-more&cta=alert-forecast-with-predict-linear).

**Alert description**: The PVC is forecast to reach more than 85% used capacity within the next 7 days based on the last 14 days of usage. Review disk growth and plan capacity expansion before the volume reaches the threshold.
{{< /admonition >}}

## Seasonal forecast with Grafana Cloud Machine Learning

When a signal follows recurring patterns, such as daily or weekly traffic cycles, a linear forecast cannot accurately represent its future behavior.

{{< figure src="/media/docs/alerting/seasonal_signal_and_linear_forecast2.png" caption="A signal with seasonal patterns (green line) and its linear forecast (yellow line)." >}}

In these cases, you can use [Grafana Machine Learning forecasting](/docs/grafana-cloud/ai-tools/dynamic-alerting/forecasting/) to generate a forecast that accounts for trends and seasonality.

Unlike `predict_linear`, which returns a single predicted value at a specific point in the future, a Grafana Machine Learning forecast produces a series of predicted values over the forecast period. It also provides upper and lower bounds that represent the uncertainty of the prediction.

Each forecast produces these Prometheus metrics:

- `<forecast_metric_name>:predicted{ml_forecast="yhat"}`: The predicted value, shown as the blue line.
- `<forecast_metric_name>:predicted{ml_forecast="yhat_upper"}`: The upper bound of the prediction interval, shown as the upper edge of the blue band.
- `<forecast_metric_name>:predicted{ml_forecast="yhat_lower"}`: The lower bound of the prediction interval, shown as the lower edge of the blue band.
- `<forecast_metric_name>:actual`: The current value of the signal, shown as the green line.

{{< figure src="/media/docs/grafana-cloud/machine-learning/screenshot-traffic-latency-example.png" caption="A forecast example in Grafana Machine Learning." >}}

Because the forecast is stored in the `grafanacloud-ml-metrics` Prometheus data source, you can use the forecast metrics directly in alert rules to detect:

1. When the forecast is expected to breach a threshold at any point in the future.
2. When the current signal deviates from the expected forecast range.

### Alert on a future prediction

Like the `predict_linear` example, a forecast alert can detect whether a signal is expected to exceed a threshold within a future time window.

In this example, the alert query uses the upper forecast bound (`yhat_upper`) rather than the central prediction. This accounts for prediction uncertainty and provides a more conservative capacity-planning signal.

```promql
<forecast_metric_name>:predicted{ml_forecast="yhat_upper"} over the next 7 days > 80%
```

{{< figure src="/media/docs/grafana-cloud/machine-learning/screenshot-db-utilization-forecast-alert-example.png" alt="A screenshot of capacity planning forecast alert." >}}

In this example, the forecast provides up to 15 days of future prediction data. The maximum function value is used so that the alert fires if the forecast is expected to exceed the threshold at any point during the next 7 days.

{{< admonition type="tip" >}}
You can explore [this alert example in Grafana Play](https://play.grafana.org/alerting/grafana/qp_db_utilization_forecast/view?tab=query&tech=docs&pg=alerting-examples&plcmt=callout-tip&cta=db-utilization-forecast-alert-example).
{{< /admonition >}}

### Alert when the actual state is outside the forecast

A forecast-based alert can also detect when the current signal deviates from the prediction.

In the previous examples, forecast alerts detect potential future situations: they alert when a signal is expected to exceed a threshold at some point in the future. The key differences in this case are:

1. The alert does not look for a future threshold breach. Instead, it compares the current value with the forecast range and alerts when the actual value falls outside the forecast range.
2. The forecast acts as an **adaptive baseline**. While a standard alert rule evaluates the signal against a fixed threshold, a forecast-based alert rule evaluates the signal against the forecast, which can change over time based on seasonal patterns.

{{< figure src="/media/docs/alerting/forecast-deviation-alert-v3.svg" alt="A forecast-based alert rule can evaluate the signal against the forecast." >}}

For alerting, each forecast produces an [anomalous metric](/docs/grafana-cloud/ai-tools/machine-learning/forecasting/detect-anomalies/#the-anomalous-metric) (`<forecast_metric_name>:anomalous`) that returns `0` when the actual value is within the prediction interval. You can use this metric directly in a forecast alert:

```promql
# Fires when the actual value is outside the prediction interval
<forecast_metric_name>:anomalous
```

#### Alert only on unexpectedly high values

You can also configure an alert to fire only when the actual value exceeds the predicted upper bound.

{{< figure src="/media/docs/alerting/forecast-deviation-alert-v2.svg" alt="A forecast-based alert rule configured to fire only for unexpectedly high values." >}}

In this case, the alert rule can also query `<forecast_metric_name>:anomalous`, which returns `1` when the actual value is above the upper bound and `-1` when it is below the lower bound:

```promql
<forecast_metric_name>:anomalous == 1
```

Alternatively, you can compare the actual value directly with the predicted upper bound:

```promql
<forecast_metric_name>:actual >
ignoring (ml_forecast) <forecast_metric_name>:predicted{ml_forecast="yhat_upper"}
```

{{< admonition type="tip" >}}
You can explore a similar [alert example in Grafana Play](https://play.grafana.org/alerting/grafana/qp_rec_ratio_below_predicted/view?tab=query&tech=docs&pg=alerting-examples&plcmt=callout-tip&cta=forecast-outside-range-alert-example).
{{< /admonition >}}

{{< admonition type="note" >}}

**Adaptive alerts** are useful when a signal varies over time and a fixed threshold would miss meaningful deviations or generate alerts during normal changes in behavior.

For example, the traffic might be higher during business hours and lower overnight. Comparing the current value with the forecast range adapts to these recurring patterns.

{{< /admonition >}}

### Learn more

For more information about forecasting and alerting with Grafana Cloud Machine Learning, refer to:

- [Get started with forecasting](https://grafana.com/docs/grafana-cloud/ai-tools/dynamic-alerting/forecasting/)
- [Alerting on forecasts](https://grafana.com/docs/grafana-cloud/ai-tools/dynamic-alerting/forecasting/query-and-alerting/)
- [Forecasting examples](https://grafana.com/docs/grafana-cloud/ai-tools/dynamic-alerting/forecasting/examples)

# Alerting best practices

Designing and configuring an effective alerting system takes time. This guide focuses on building alerting systems that scale with real-world operations.

The practices described here are intentionally high-level and apply regardless of tooling. Whether you use Prometheus, Grafana Alerting, or another stack, the same constraints apply: complex systems, imperfect signals, and humans on call.

Alerting is never finished. It evolves with incidents, organizational changes, and the systems it’s meant to protect.

{{< shared id="alert-planning-fundamentals" >}}

## Prioritize symptoms, but don’t ignore infrastructure signals

Alerts should primarily detect user-facing failures, not internal component behavior. Users don't care that a pod restarted; they care when the application is slow or failing. Symptom-based alerts tie directly to user impact.

Reliability metrics that impact users—latency, errors, availability—are better paging signals than infrastructure events or internal errors.

That said, infrastructure signals still matter. They can act as early warning indicators and are often useful when alerting maturity is low. A sustained spike in CPU or memory usage might not justify a page, but it can help explain or anticipate symptom-based failures.

Infrastructure alerts tend to be noisy and are often ignored when treated like paging signals. They are usually better suited for lower-severity channels such as dashboards, alert lists, or non-paging destinations like a dedicated Slack channel, where they can be monitored without interrupting on-call.

The key is balance as your alerting matures. Use infrastructure alerts to support diagnosis and prevention, not as a replacement for symptom-based alerts.

## Escalate priority based on confidence

Alert priority is often tied to user impact and the urgency to respond, but confidence should determine when escalation is necessary.

In this context, escalation defines how responders are notified as confidence grows. This can include increasing alert priority, widening notification, paging additional responders, or opening an incident once intervention is clearly required.

Early signals are often ambiguous, and confidence in a non-transient failure is usually low. Paging too early creates noise; paging too late means users are impacted for longer before anyone acts. A small or sudden increase in latency may not justify immediate action, but it can indicate a failure in progress.

Confidence increases as signals become stronger or begin to correlate.

Escalation is justified when issues are sustained or reinforced by multiple signals. For example, high latency combined with a rising error rate, or the same event firing over a sustained period. These patterns reduce the chance of transient noise and increase the likelihood of real impact.

Use confidence in user impact to drive escalation and avoid unnecessary pages.

## Scope alerts for scalability and actionability

In distributed systems, avoid creating separate alert rules for every host, service, or endpoint. Instead, define alert rules that scale automatically using [multi-dimensional alert rules](/docs/grafana/<GRAFANA_VERSION>/alerting/examples/multi-dimensional-alerts/). This reduces rule duplication and allows alerting to scale as the system grows.

Start simple. Default to a single dimension such as `service` or `endpoint` to keep alerts manageable. Add dimensions only when they improve actionability. For example, when missing a dimension like `region` hides failures or doesn't provide enough information to act quickly.

Additional dimensions like `region` or `instance` can help identify the root cause, but more isn't always better.

## Design alerts for first responders and clear actions

Alerts should be designed for the first responder, not the person who created the alert. Anyone on call should be able to understand what's wrong and what to do next without deep knowledge of the system or alert configuration.

Avoid vague alerts that force responders to spend time figuring out context. Every alert should clearly explain why it exists, what triggered it, and how to investigate. Use [annotations](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#annotations) to link to relevant dashboards and runbooks, which are essential for faster resolution.

Alerts should indicate a real problem and be actionable, even if the impact is low. Informational alerts add noise without improving reliability.

If no action is possible, it shouldn't be an alert—consider using a dashboard instead. Over time, alerts behave like technical debt: easy to create, costly to maintain, and hard to remove.

Review alerts often and remove those that don’t lead to action.

## Alerts should have an owner and system scope

Alerts without ownership are often ignored. Every alert must have an owner: a team responsible for maintaining the alert and responding when it fires.

Alerts must also define a system scope, such as a service or infrastructure component. Scope provides organizational context and connects alerts with ownership. Defining clear scopes is easier when services are treated as first-class entities, and organizations are built around service ownership.

> [Service Center in Grafana Cloud](/docs/grafana-cloud/alerting-and-irm/service-center/) can help operate a service-oriented view of your system and align alert scope with ownership.

After scope, ownership, and alert priority are defined, routing determines where alerts go and how they escalate. **Notification routing is as important as the alerts**.

Alerts should be delivered to the right team and channel based on priority, ownership, and team workflows. Use [notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/) to define a routing tree that matches the context of your service or scope:

- Define a parent policy for default routing within the scope.
- Define nested policies for specific cases or higher-priority issues.

## Prevent notification overload with alert grouping

Without alert grouping, responders can receive many notifications for the same underlying problem.

For example, a database failure can trigger several alerts at the same time like increased latency, higher error rates, and internal errors. Paging separately for each symptom quickly turns into notification spam, even though there is a single root cause.

[Notification grouping](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/) consolidates related alerts into a single notification. Instead of receiving multiple pages for the same issue, responders get one alert that represents the incident and includes all related firing alerts.

Grouping should follow operational boundaries such as service or owner, as defined by notification policies. Downstream or cascading failures should be grouped together so they surface as one issue rather than many.

There is also a limit to how many notifications can be sent out per second. You can configure your Contact Point settings to reduce the email expenditure by setting up single email notifications rather than individual email notifications. To configure this, go to your desired email Contact Point and enable **Single email** in the [optional settings](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/integrations/configure-email/#optional-settings).

## Mitigate flapping alerts

Short-lived failure spikes often trigger alerts that auto-resolve quickly. Alerting on transient failures creates noise and leads responders to ignore them.

Require issues to persist before alerting. Set a [pending period](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#pending-period) to define how long a condition must remain true before firing. For example, instead of alerting immediately on high error rate, require it to stay above the threshold for some minutes.

Also, stabilize alerts by tuning query ranges and aggregations. Using raw data makes alerts sensitive to noise. Instead, evaluate over a time window and aggregate the data to smooth short spikes.

```promql
# Reacts to transient spikes. Avoid this.
cpu_usage > 90

# Smooth fluctuations.
avg_over_time(cpu_usage[5m]) > 90
```

For latency and error-based alerts, percentiles are often more useful than averages:

```promql
quantile_over_time(0.95, http_duration_seconds[5m]) > 3
```

Finally, avoid rapid resolve-and-fire notifications by using [`keep_firing_for`](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#keep-firing-for) or [recovery thresholds](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#recovery-threshold) to keep alerts active briefly during recovery. Both options reduce flapping and unnecessary notifications.

## Graduate symptom-based alerts into SLOs

When a symptom-based alert fires frequently, it usually indicates a reliability concern that should be measured and managed more deliberately. This is often a sign that the alert could evolve into a [service level objective (SLO)](/docs/grafana-cloud/alerting-and-irm/slo/introduction).

Traditional alerts create pressure to react immediately, while error budgets introduce a buffer of time to act, changing how urgency is handled. Alerts can then be defined in terms of error budget burn rate rather than reacting to every minor deviation.

SLOs also align distinct teams around common reliability goals by providing a shared definition of what "good" looks like. They help consolidate multiple symptom alerts into a single user-facing objective.

For example, instead of several teams alerting on high latency, a single SLO can be used across teams to capture overall API performance.

For other use cases, refer to [when alerts need an SLO](/docs/grafana/<GRAFANA_VERSION>/alerting/guides/when-slos-reduce-alert-noise/).

## Integrate alerting into incident post-mortems

Every incident is an opportunity to improve alerting. After each incident, evaluate whether alerts helped responders act quickly or added unnecessary noise.

Assess which alerts fired, and how they influenced incident response. Review whether alerts triggered too late, too early, or without enough context, and adjust thresholds, priority, or escalation based on what actually happened.

Use [silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/) during active incidents to reduce repeated notifications, but scope them carefully to avoid silencing unrelated alerts.

Post-mortems should evaluate alerts with root causes and lessons learned. If responders lacked key information during the incident, enrich alerts with additional context, dashboards, or better guidance.

## Alerts should be continuously improved

Alerting is an iterative process. Alerts that aren’t reviewed and refined lose effectiveness as systems and traffic patterns change.

Schedule regular reviews of existing alerts. Remove alerts that don’t lead to action, and tune alerts or thresholds that fire too often without providing useful signal. Reduce false positives to combat alert fatigue.

Prioritize clarity and simplicity in alert design. Simpler alerts are easier to understand, maintain, and trust under pressure. Favor fewer high-quality, actionable alerts over a large number of low-value ones.

Use dashboards and observability tools for investigation, not alerts.

{{< /shared >}}

# Handle connectivity errors in alerts

Connectivity issues are a common cause of misleading alerts or unnoticed failures.

There could be a number of reasons for these errors. Maybe your target went offline, or Prometheus couldn't scrape it. Or maybe your alert query failed because its target timed out or the network went down. These situations might look similar, but require different considerations in your alerting setup.

This guide walks through how to detect and handle these types of failures, whether you're writing alert rules in Prometheus, using Grafana Alerting, or combining both. It covers both availability monitoring and alert query failures, and outlines strategies to improve the reliability of your alerts.

## Understand connectivity issues in alerts

Typically, connectivity issues fall into a few common scenarios:

- Servers or containers crashed or were shut down.
- Service overload or timeout.
- Misconfigured authentication or incorrect permissions.
- Network issues like DNS problems or ISP outages.

When we talk about connectivity errors in alerting, we’re usually referring to one of two use cases:

1. **Your target is down or unreachable.**  
   The service crashed, the host was down, or a firewall or DNS issue blocked the connection. These are **availability problems**.

1. **Your alert query failed.**  
   The alert couldn’t evaluate its query—maybe because the data source timed out or an invalid query. These are **execution errors**.

It helps to separate these cases early, because they behave differently and require different strategies.

Keep in mind that most alert rules don’t hit the target directly. They query metrics from a monitoring system like Prometheus, which scrapes data from your actual infrastructure or application. That gives us two typical alerting setups where connectivity issues can show up:

1. **Alert rule → Target**  
   For example, an alert rule querying an external data source like a database.

2. **Alert rule → Prometheus ← Target**  
   More common in observability stacks. For instance, Prometheus scrapes a node or container, and the alert rule queries the metrics later.

   In this second setup, you can run into connectivity issues on either side. If Prometheus fails to scrape the target, your alert rule might not fire, even though something is likely wrong.

## Detect target availability with the Prometheus `up` metric

Prometheus scrapes metrics from its targets regularly, following the [`scrape_interval`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) period. The default scrape interval is 60 seconds, which is generally considered common practice.

Prometheus provides a built-in metric called `up` for every scrape target, a simple method to indicate whether scraping is successful:

- `up == 1`: Your target is reachable; Prometheus collected the target metrics as expected.

- `up == 0`: Prometheus couldn't reach your target—indicating possible downtime or network errors.

A typical PromQL expression for an alert rule to detect when a target becomes unreachable is:

`up == 0`

But this alert rule might result in noisy alerts as one single scrape failure will fire the alert. To reduce noise, you should add a delay:

`up == 0 for: 5m`

The `for` option in Prometheus (or [pending period](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/) in Grafana) delays the alert until the condition has been true for the full duration.

In this example, waiting for 5 minutes means the single scrape error won't result in a fired alert. Since Prometheus scrapes metrics every minute by default, the alert only fires after five consecutive failures.

However, this kind of `up` alert has a few potential downfalls:

- **Failures can slip between scrape intervals**: An outage that starts and ends between two evaluations go undetected. You could shorten the `for` duration, but this might lead to scrape failures that trigger false alarms.
- **Intermittent recoveries reset the `for` timer**: A single successful scrape resets the alert timer, which masks intermittent outages.

Brief connectivity drops are common in real-world environments, so expect some flakiness in `up` alerts. For example:

| Scrape result (`up`) | Alert rule evaluation                                 |
| :------------------- | :---------------------------------------------------- |
| 00:00 `up == 0`      | Timer starts                                          |
| 01:00 `up == 0`      | Timer continues                                       |
| 02:00 `up == 0`      | Timer continues                                       |
| 03:00 `up == 1`      | Successful scrape resets timer                        |
| 04:00 `up == 0`      | Timer starts again                                    |
| 05:00 `up == 0`      | No alert yet; timer hasn’t reached the `for` duration |

The longer the period, the more likely this is to happen.

A single recovery resets the alert, that’s why `up == 0 for: 5m` can sometimes be unreliable. Even if the target is down most of the time, the alert didn't fire, leaving you unaware of a potential persistent issue.

### Use `avg_over_time` to smooth signal

One way to work around these issues is to smooth the signal by averaging the `up` metric over a similar or longer period:

`avg_over_time(up[10m]) < 0.8`

This alert rule fires when the target is unreachable for more than 20% of the last 10 minutes, rather than looking for consecutive scrape failures. With a one minute scrape interval, three or more failed scrapes within the last 10 minutes now triggers the alert.

Since this query uses a threshold and time window to control accuracy, you can now lower the `for` duration (or [pending period](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/) in Grafana) to something shorter—`0m` or `1m`—so the alert fires faster.

This approach gives you more flexibility in detecting real crashes or network issues. As always, adjust the threshold and period based on your noise tolerance and how critical the target is.

### Use synthetic checks to monitor external availability

Prometheus often runs inside the same network as the target it monitors. That means Prometheus might be able to reach the target, but doesn’t ensure it’s reachable to users on the outside.

Firewalls, DNS misconfigurations, or other network issues might block public traffic while Prometheus scrapes `up` successfully.

This is where synthetic monitoring helps. Tools like the [Blackbox Exporter](https://github.com/prometheus/blackbox_exporter) let you continuously verify whether a service is available and reachable from outside your network—not just internally.

The Blackbox Exporter exposes the results of these checks as metrics, which Prometheus can scrape like any other target. For example, the `probe_success` metric reports whether the probe was able to reach the service. The setup looks like this:

**Alert rules → Prometheus ← Blackbox Exporter (external probe) → Target**

To detect when a service isn’t reachable externally, you can define an alert using the `probe_success` metric:

`probe_success == 0 for: 5m`

This alert fires when the probe has failed continuously for 5 minutes—indicating that the service couldn’t be reached from the outside.

You can then combine internal and external checks to make the detection of connectivity errors more reliable. This alert catches when the internal scrape fails or the service is externally unreachable.

`up == 0 or probe_success == 0`

As with the `up` metric, you might want to smooth this out using `avg_over_time()` for more robust detection. The smooth version might look like:

`avg_over_time(up[10m]) < 0.8 or avg_over_time(probe_success[10m]) < 0.8`

This alert fires when Prometheus couldn't scrape the target successfully for more than 20% of the past 10 minutes, or when the external probes have been failing more than 20% of the time. This smoothing technique can be applied to any binary availability signal.

## Manage offline hosts

In many setups, Prometheus scrapes multiple hosts under the same target, such as a fleet of servers or containers behind a common job label. It’s common for one host to go offline while the others continue to report metrics normally.

If your alert only checks the general `up` metric without breaking it down by labels (like `instance`, `host`, or `pod`), you might miss when a host stops reporting. For example, an alert that looks only at the aggregated status of all instances will likely fail to catch when individual instances go missing.

This isn't a connectivity error in this context — it’s not that the alert or Prometheus can't reach anything, it’s that one or more specific targets have gone silent. These kinds of problems aren’t caught by `up == 0` alerts.

For these cases, see the complementary [guide on handling missing data](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/missing-data/) — it covers common scenarios where the alert queries return no data at all, or where only some targets stop reporting. These aren't full availability failures or execution errors, but they can still lead to blind spots in alert detection.

## Handle query errors in Grafana Alerting

Not all connectivity issues come from targets going offline. Sometimes, the alert rule fails when querying its target. These aren’t availability problems—they’re query execution errors: maybe the data source timed out, the network dropped, or the query was invalid.

These errors lead to broken alerts. But they come from a different part of the stack: between the alert rule and the data source, not between the data source (for example, Prometheus) and its target.

This difference matters. Availability issues are typically handled using metrics like `up` or `probe_success` but execution errors require a different setup.

Grafana Alerting has built-in handling for execution errors, regardless of the data source. That includes Prometheus, and others like Graphite, InfluxDB, PostgreSQL, etc. By default, Grafana Alerting automatically handles query errors so you don’t miss critical failures. When an alert rule fails to execute, Grafana fires a special `DatasourceError` alert.

You can configure this behavior depending on how critical the alert is and on whether you already have other alerts detecting the issue. In [**Configure no data and error handling**](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/#modify-the-no-data-or-error-state), click **Alert state if execution error or timeout**, and choose the desired option for the alert:

- **Error (default)**: Triggers a separate `DatasourceError` alert. This default ensures alert rules always inform about query errors but can create noise.
- **Alerting**: Treats the error as if the alert condition is firing. Grafana transitions all existing instances for that rule to the `Alerting` state.
- **Normal**: Ignores the query error and transitions all alert instances to the `Normal` state. This is useful if the error isn’t critical or if you already have other alerts detecting connectivity issues.
- **Keep Last State**: Keeps the previous state until the query succeeds again. Suitable for unstable environments to avoid flapping alerts.

  {{< figure src="/media/docs/alerting/alert-rule-configure-no-data-and-error-v2.png" alt="A screenshot of the `Configure error handling` option in Grafana Alerting." max-width="500px" >}}

This applies even when alert rules query Prometheus itself—not just external data sources.

### Design alerts for connectivity errors

In practice, start by deciding if you want to create explicit alert rules — for example, using `up` or `probe_success` — to detect when a target is down or has connectivity issues.

Then, for each alert rule, choose the error-handling behavior based on whether you already have dedicated connectivity alerts, the stability of the target, and how critical the alert is. Prioritize alerts based on symptom severity rather than just infrastructure signals that might not impact users.

### Build a dedicated data source health alert

Any alert rule query can fail to reach its data source, including one built specifically to check reachability. Prometheus-style data sources expose metrics such as `up` and `probe_success` for target-level availability, but those are still queries, and they fail the same way if the data source itself becomes unreachable. In any case, detecting that the data source itself is unreachable uses a different mechanism: the execution-error handling built into Grafana Alerting.

Create a separate alert rule dedicated to detecting when the data source itself is unreachable, distinct from any application-specific rules that happen to query it:

1. Reuse a query you already know returns data reliably when the data source is healthy. Any small, cheap query works, since the result doesn't matter—only whether the query executes.
1. In [Configure no data and error handling](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/#modify-the-no-data-or-error-state), set **Alert state if execution error or timeout** to **Alerting**.
1. Add a [pending period](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/), for example `5m`, so the rule fires only once the data source has stayed unreachable for a sustained period, not for a single blip.

Set the same option to **Normal** or **Keep Last State** on your application-specific rules instead, so a transient error from the data source doesn't also fire a `DatasourceError` alert alongside the condition you're actually trying to detect. Which value to use isn't a one-time default—decide it per rule, based on whether that rule already has a dedicated connectivity alert covering it.

### Reduce redundant error notifications

A single data source error can lead to multiple alerts firing simultaneously, sometimes bombarding you with many alerts and generating too much noise.

As described previously, you can control the error-handling behavior for Grafana alerts. The **Keep Last State** or **Normal** option prevents alerts from firing and helps avoid redundant alerts, especially for services already covered by `up` or `probe_success` alerts.

When using the default behavior, a single connectivity error will likely trigger multiple `DatasourceError` alerts.

Consider not treating these alerts in the same way as the original alerts, and implement dedicated strategies for their notifications:

- Reduce duplicate notifications by grouping `DatasourceError` alerts. Use the `datasource_uid` label to group errors from the same data source.

- Route `DatasourceError` alerts separately, sending them to different teams or channels depending on their impact and urgency.

For details on how to configure grouping and routing, refer to [handling notifications](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/) and [`No Data` and `Error` alerts](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/#no-data-and-error-alerts) documentation.

## Conclusion

Connectivity issues are one of the common causes of noisy or misleading alerts. This guide covered two distinct types:

- **Availability issues**, where the target itself is down or unreachable (e.g., due to a crash or network failure).

- **Query execution errors**, where the alert rule can't reach its data source (e.g., due to timeouts, invalid queries, or data source outages).

These problems come from different parts of your stack, and require their own techniques. Prometheus and Grafana allow you to detect them, and combining distinct techniques can make your alerts more resilient.

With Prometheus, avoid relying solely on `up == 0`. Smooth queries to account for intermittent failures, and use synthetic monitoring to detect reachability issues from outside your network.

In Grafana Alerting, configure error handling explicitly. Not all alerts are equal or have the same urgency. Tune the error-handling behavior based on the reliability and severity of the alerts and whether you already have alerts dedicated to connectivity problems.

And don’t forget the third case: **missing data**. If only one host from a fleet silently disappears, you might not get alerted. If you're dealing with individual instances that stopped reporting data, see the [Guide on handling missing data](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/missing-data/) to continue exploring this topic.

# Handle missing data in Grafana Alerting

Missing data from when a target stops reporting metric data can be one of the most common issues when troubleshooting alerts. In cloud-native environments, this happens all the time. Pods or nodes scale down to match demand, or an entire job quietly disappears.

When this happens, alerts won’t fire, and you might not notice the system has stopped reporting.

Sometimes it's just a lack of data from a few instances. Other times, it's a connectivity issue where the entire target is unreachable.

This guide covers different scenarios where the underlying data is missing and shows how to design your alerts to act on those cases. If you're troubleshooting an unreachable host or a network failure, see the [Handle connectivity errors documentation](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/connectivity-errors/) as well.

## No Data vs. Missing Series

There are a few common causes when an instance stops reporting data, similar to [connectivity errors](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/connectivity-errors/):

- Host crash: The system is down, and Prometheus stops scraping the target.
- Temporary network failures: Intermittent scrape failures cause data gaps.
- Deployment changes: Decommissioning, Kubernetes pod eviction, or scaling down resources.
- Ephemeral workloads: Metrics intentionally stop reporting.
- And more.

The first thing to understand is the difference between a query failure (or connectivity error), _No Data_, and a _Missing Series_.

Alert queries often return multiple time series — one per instance, pod, region, or label combination. This is known as a **multi-dimensional alert**, meaning a single alert rule can trigger multiple alert instances (alerts).

For example, imagine a recorded metric, `http_request_latency_seconds`, that reports latency per second in the regions where the application is deployed. The query returns one series per region — for instance, `region1` and `region2` — and generates only two alert instances. In this scenario, you may experience:

- **Connectivity Error** if the alert rule query fails.
- **No Data** if the query runs successfully but returns no data at all.
- **Missing Series** if one or more specific series, which previously returned data, are missing, but other series still return data.

In both _No Data_ and _Missing Series_ cases, the query still technically "works", but the alert won’t fire unless you explicitly configure it to handle these situations.

The following tables illustrate both scenarios using the previous example, with an alert that triggers if the latency exceeds 2 seconds in any region: `avg_over_time(http_request_latency_seconds[5m]) > 2`.

**No Data Scenario:** The query returns no data for any series:

| Time  | region1    | region2    | Alert triggered              |
| :---- | :--------- | :--------- | :--------------------------- |
| 00:00 | 1.5s 🟢    | 1s 🟢      | ✅ No Alert                  |
| 01:00 | No Data ⚠️ | No Data ⚠️ | ⚠️ No Alert (Silent Failure) |
| 02:00 | 1.4s 🟢    | 1s 🟢      | ✅ No Alert                  |

**MissingSeries Scenario:** Only a specific series (`region2`) disappears:

| Time  | region1 | region2           | Alert triggered              |
| :---- | :------ | :---------------- | :--------------------------- |
| 00:00 | 1.5s 🟢 | 1s 🟢             | ✅ No Alert                  |
| 01:00 | 1.6s 🟢 | Missing Series ⚠️ | ⚠️ No Alert (Silent Failure) |
| 02:00 | 1.4s 🟢 | 1s 🟢             | ✅ No Alert                  |

In both cases, something broke silently.

## Detect missing data in Prometheus

Prometheus doesn't fire alerts when the query returns no data. It simply assumes there was nothing to report, like with query errors. Missing data won’t trigger existing alerts unless you explicitly check for it.

In Prometheus, a common way to catch missing data is by to use the `absent_over_time` function.

`absent_over_time(http_request_latency_seconds[5m]) == 1`

This triggers when all series for `http_request_latency_seconds` are absent for 5 minutes — catching the _No Data_ case when the entire metric disappears.

However, `absent_over_time()` can’t detect which specific series are missing since it doesn’t preserve labels. The alert won’t tell you which series stopped reporting, only that the query returns no data.

If you want to check for missing data per-region or label, you can specify the label in the alert query as follows:

```promQL
# Detect missing data in region1
absent_over_time(http_request_latency_seconds{region="region1"}[5m]) == 1

# Detect missing data in region2
absent_over_time(http_request_latency_seconds{region="region2"}[5m]) == 1
```

But this doesn't scale well. It is unreliable to have hard-coded queries for each label set, especially in dynamic cloud environments where instances can appear or disappear at any time.

To detect when a specific target has disappeared, see below **Evict alert instances for missing series** for details on how Grafana handles this case and how to set up detection.

## Manage No Data issues in Grafana alerts

While Prometheus provides functions like `absent_over_time()` to detect missing data, not all data sources — like Graphite, InfluxDB, PostgreSQL, and others — available to Grafana alerts support a similar function.

To handle this, Grafana Alerting implements a built-in `No Data` state logic, so you don’t need to detect missing data with `absent_*` queries. Instead, you can configure in the alert rule settings how alerts behave when no data is returned.

Similar to error handling, Grafana triggers a special _No data_ alert by default and lets you control this behavior. In [**Configure no data and error handling**](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/#modify-the-no-data-or-error-state), click **Alert state if no data or all values are null**, and choose one of the following options:

- **No Data (default):** Triggers a new `DatasourceNoData` alert, treating _No data_ as a specific problem.
- **Alerting:** Transition each existing alert instance into the `Alerting` state when data disappears.
- **Normal:** Ignores missing data and transitions all instances to the `Normal` state. Useful when receiving intermittent data, such as from experimental services, sporadic actions, or periodic reports.
- **Keep Last State:** Leaves the alert in its previous state until the data returns. This is common in environments where brief metric gaps happen regularly, like with flaky exporters or noisy environments.

  {{< figure src="/media/docs/alerting/alert-rule-configure-no-data.png" alt="A screenshot of the `Configure no data handling` option in Grafana Alerting." max-width="500px" >}}

### Manage DatasourceNoData notifications

When Grafana triggers a [NoData alert](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/#no-data-and-error-alerts), it creates a distinct alert instance, separate from the original alert instance. These alerts behave differently:

- They use a dedicated `alertname: DatasourceNoData`.
- They don’t inherit all the labels from the original alert instances.

Because of this, `DatasourceNoData` alerts might require a dedicated setup to handle their notifications. For general recommendations, see [Reduce redundant DatasourceError alerts](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/connectivity-errors/#reducing-notification-fatigue-from-datasourceerror-alerts) — similar practices can apply to _NoData_ alerts.

## Evict alert instances for missing series

_MissingSeries_ occurs when only some series disappear but not all. This case is subtle, but important.

Grafana marks missing series as [**stale**](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/) after two evaluation intervals and triggers the alert instance eviction process. Here’s what happens under the hood:

- Alert instances with missing data keep their last state for two evaluation intervals.
- If the data is still missing after that:
  - Grafana adds the annotation `grafana_state_reason: MissingSeries`.
  - The alert instance transitions to the `Normal` state.
  - A **resolved notification** is sent if the alert was previously firing.
  - The **alert instance is removed** from the Grafana UI.

If an alert instance becomes stale, you’ll find it in the [alert history](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-state-history/) as `Normal (Missing Series)` before it disappears. This table shows the eviction process from the previous example:

| Time  | region1               | region2                               | Alert triggered                                                          |
| :---- | :-------------------- | :------------------------------------ | :----------------------------------------------------------------------- |
| 00:00 | 1.5s 🟢               | 1s 🟢                                 | 🟢🟢 No Alerts                                                           |
| 01:00 | 3s 🔴 <br> `Alerting` | 3s 🔴 <br> `Alerting`                 | 🔴🔴 Alert instances triggered for both regions                          |
| 02:00 | 1.6s 🟢               | `(MissingSeries)`⚠️ <br> `Alerting` ️ | 🟢🔴 Region2 missing, state maintained.                                  |
| 03:00 | 1.4s 🟢               | `(MissingSeries)` <br> `Normal`       | 🟢🟢 `region2` was resolved, 📩 notification sent, and instance evicted. |
| 04:00 | 1.4s 🟢               | —                                     | 🟢 No Alerts. `region2` was evicted.                                     |

### Why doesn’t MissingSeries match No Data behavior?

In dynamic environments, such as autoscaling groups, ephemeral pods, spot instances, series naturally come and go. **MissingSeries** normally signals infrastructure or deployment changes.

By default, **No Data** triggers an alert to indicate a potential problem.

The eviction process for **MissingSeries** is designed to prevent alert flapping when a pod or instance disappears, reducing alert noise.

In environments with frequent scale events, prioritize symptom-based alerts over individual infrastructure signals and use aggregate alerts unless you explicitly need to track individual instances.

### Handle MissingSeries notifications

A stale alert instance triggers a **resolved notification** if it transitions from a firing state (such as `Alerting`, `No Data`, or `Error`) to `Normal`, and the [`grafana_state_reason` annotation](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/#grafana_state_reason-for-troubleshooting) is set to **MissingSeries** to indicate that the alert wasn’t resolved by recovery but evicted because the series data went missing.

Recognizing these notifications helps you handle them appropriately. For example:

- Display the `grafana_state_reason` annotation to clearly identify **MissingSeries** alerts.
- Or use the `grafana_state_reason` annotation to process these alerts differently.

Also, review these notifications to confirm whether something broke or if the alert was unnecessary. To reduce noise:

- Silence or mute alerts during planned maintenance or rollouts.
- Adjust alert rules to avoid triggering on series you expect to come and go, and use aggregated alerts instead.

### Detect missing series in Prometheus

Previously, an example showed how to detect missing data for a specific label, such as `region`:

```promQL
# Detect missing data in region1
absent_over_time(http_request_latency_seconds{region="region1"}[5m]) == 1

# Detect missing data in region2
absent_over_time(http_request_latency_seconds{region="region2"}[5m]) == 1
```

However, this approach doesn’t scale well because it requires hardcoding all possible `region` values.

As an alternative, you can create an alert rule that detects missing series dynamically using the `present_over_time` function:

```promQL
present_over_time(http_request_latency_seconds{}[24h])
unless
present_over_time(http_request_latency_seconds{}[10m])
```

Or, if you want to group by a label such as region:

```promQL
group(present_over_time(http_request_latency_seconds{}[24h])) by (region)
unless
group(present_over_time(http_request_latency_seconds{}[10m])) by (region)
```

This query finds regions (or other targets) that were present at any time in the past 24 hours but have not been present in the past 10 minutes. The alert rule then triggers an alert instance for each missing region. You can apply the same technique to any label or target dimension.

## Conclusion

Missing data isn’t always a failure. It’s a common scenario in dynamic environments when certain targets stop reporting.

Grafana Alerting handles distinct scenarios automatically. Here’s how to think about it:

- Understand `DatasourceNoData` and `MissingSeries` notifications, since they don’t behave like regular alerts.
- Use Grafana’s _No Data_ handling options to define what happens when a query returns nothing.
- When _NoData_ is not an issue, consider rewriting the query to always return data — for example, in Prometheus, use `your_metric_query OR on() vector(0)` to return `0` when `your_metric_query` returns nothing.
- Use `absent_over_time()` or `present_over_time` in Prometheus to detect when a metric or target disappears.
- If data is frequently missing due to scrape delays, use techniques to account for data delays:
  - Adjust the **Time Range** query option in Grafana to evaluate slightly behind real time (e.g., set **To** to `now-1m`) to account for late data points.
  - In Prometheus, you can use `last_over_time(metric_name[10m])` to pick the most recent sample within a given window.
- Don’t alert on every instance by default. In dynamic environments, it’s better to aggregate and alert on symptoms — unless a missing individual instance directly impacts users.
- If you’re getting too much noise from disappearing data, consider adjusting alerts, using `Keep Last State`, or routing those alerts differently.
- For connectivity issues involving alert query failures, see the sibling guide: [Handling connectivity errors in Grafana Alerting](/docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/connectivity-errors/).

# When SLOs reduce alert noise

As your alerting setup grows in production, not every alert works as expected. Sometimes they fire too often, or they resolve immediately before anyone responds. You start wondering if the problem is the alert.

When this happens, your instinct may be to tune the query or threshold, or maybe just delete it. Sometimes the alert is telling you that you're measuring reliability wrong, and implementing [service level objectives (SLOs)](/docs/grafana-cloud/alerting-and-irm/slo/introduction/) could be the right answer.

This guide covers the three most common patterns where SLOs can help reduce alert noise and improve reliability coverage, and shows how to recognize the need for SLOs in your alerting setup.

## How SLOs differ from threshold alerts

Threshold alerts evaluate a condition on its evaluation interval, for example, every `5m`, `30m`, `1h`.

If the query result crosses the threshold at that moment, the alert fires. It continues in a firing state on the next evaluation if the condition still holds, or returns to normal once resolved.

In this sense, the alert state is binary: firing or not firing.

{{< figure src="/media/docs/alerting/slog-guide-alert-state-lifecycle.png" max-width="750px" alt="States of alert instances over time" >}}

SLOs don't focus primarily on whether the service is healthy right now. They measure whether your service is **reliable over a long period**, for a week or several weeks, not at a point in time. For instance:

- An alert asks: over the past 10 minutes, did 50% of requests respond in under 3s?

- An SLO asks a similar question over a longer window: over the past 4 weeks, did 95% of requests respond in under 3 seconds?

Alerts can also use long evaluation periods, but they reset when the service recovers.

SLOs track failures against the [error budget](/grafana-cloud/alerting-and-irm/slo/introduction/#error-budget) even when the service recovers. They record how much degradation has accumulated, and report how much failure is still allowed.

{{< figure src="/media/docs/alerting/slo-guide-remaining-error-budget-panel.png" max-width="750px" alt="SLO error budget" >}}

Another key difference is how [SLO alerts](/grafana-cloud/alerting-and-irm/slo/set-up/configure-burn-rate-notifications/#burn-rate-alert-rule-conditions) work. Rather than firing when the error budget is consumed and the objective is breached, they **fire when the objective is at risk**.

SLO alerts page you when the system is degrading quickly, so you have time to act before breaching the SLO.

SLOs and threshold-based alerts are complementary. In some cases, tracking the same reliability signal in both alerts and SLOs makes sense: alerts for immediate response and SLOs for long-term visibility. In others, graduating alerts to SLOs is the right call. The following cases are the most common situations where teams start adopting SLOs from existing alerts.

## The alert has no actionable response

[Alerts should be actionable](/docs/grafana/<GRAFANA_VERSION>/alerting/guides/best-practices/#design-alerts-for-first-responders-and-clear-actions). An alert nobody acts on is operational noise. If the alert consistently triggers no response, but the metric still matters to your team, this is a reliability concern.

This reliability metric could indicate service health over time. It's worth tracking in operational dashboards, but it's not designed to trigger immediate action.

Ask the team: will we commit to an objective on the health of this service and act if performance degrades? If the answer is yes, an SLO is the right fit.

In practice, this question often surfaces whether the metric is actually measurable as a [service level indicator (SLI)](/docs/grafana-cloud/alerting-and-irm/slo/introduction/#service-level-indicators-slis).

{{< figure src="/media/docs/alerting/slo-guide-slo-performance.png" max-width="750px" alt="Service level indicator and error budget performance" >}}

Defining the SLO objective aligns the team around a shared target and determines when accumulated degradation is severe enough to require action. This is frequently the hardest part. Expect a few iterations before the objective reflects reality.

Sometimes deleting the alert is the right call if there's no commitment or SLI to anchor to a valid SLO. But if the same metric keeps surfacing in post-mortems across teams, that's a different signal that points to a shared reliability concern.

## The alert reflects a shared reliability concern

Sometimes, multiple teams maintain alerts for the same user-facing metric such as service response times, each applying their own thresholds and internal metrics for their services, but lacking an objective to evaluate the end-to-end user experience.

This sometimes results in overlapping alerts covering the same signal.

When distinct teams define their own thresholds for service performance or user-experience metrics, this reveals a shared user-experience concern that hasn't been made explicit.

In other cases, the SLO is driven by the business, such as a customer commitment or service level agreement (SLA). An SLO formalizes it as a shared objective tracked against what customers expect.

Before creating a shared SLO, teams need to agree on clear ownership.

In both cases, the shift is often from internal metrics to user-facing reliability metrics, as recommended in the [best practice to prioritize symptoms over causes](/docs/grafana/<GRAFANA_VERSION>/alerting/guides/best-practices/#prioritize-symptoms-but-dont-ignore-infrastructure-signals).

Creating a shared SLO should not eliminate existing related alerts. This is a good moment to evaluate whether existing alerts trigger immediate action. If you notice these alerts are less actionable, and that the underlying reliability metric is better measured as an SLO, then migrate the existing alert to a team-scoped SLO.

A team-scoped SLO is the safer default. Move to a shared SLO only once teams agree on ownership.

## The alert fires on transient conditions

The alert fires and resolves without human intervention, a pattern commonly referred to as flapping.

{{< figure src="/media/docs/alerting/slo-guide-flapping-alert-timeline.png" max-width="750px" caption="Alert instance states for a flapping alert" >}}

Alerting best practices recommend [tuning away flapping alerts](/docs/grafana/<GRAFANA_VERSION>/alerting/guides/best-practices/#mitigate-flapping-alerts) by tweaking alert rule settings to avoid detecting short spikes and transient issues. Tuning reduces noise, but can remove the signal entirely.

{{< figure src="/media/docs/alerting/slo-guide-non-flapping-alert-timeline.png" max-width="750px" caption="Alert instance states after mitigating flapping behavior" >}}

In this case, the flapping alert has surfaced a reliability issue. Individual firings look harmless, but the accumulated degradation over time impacts users. **A threshold-based alert can mask these errors**.

The [SLO error budget](/docs/grafana-cloud/alerting-and-irm/slo/dashboard/#view-remaining-error-budget) captures this pattern. Each transient failure consumes a fraction of the budget, making cumulative impact visible over the SLO window.

{{< figure src="/media/docs/alerting/slo-guide-remaining-error-budget-panel.png" max-width="750px" alt="Error budget burndown" >}}

This pattern isn't limited to flapping alerts. Any alert with acceptable transient failures faces the same tradeoff: tune for less noise and lose visibility, or keep the alert and learn to ignore it.

Tracking reliability signals in dashboards and alerting on long evaluation periods can be the solution. SLOs are the better fit: the error budget doesn't reset and tracks every failure within the SLO window.

## Conclusion

Not every alert needs to be tuned or silenced when it's not effective. Some are measuring something worth tracking differently.

Three common patterns signal an alert is ready for an SLO:

- **No actionable response**: the signal indicates service reliability, but doesn't trigger immediate action.
- **Shared reliability concern**: teams track similar user-facing metrics, but lack a shared objective.
- **Transient conditions**: short-lived failures are often ignored, but still impact users.

In all three cases, the shift is from evaluating a threshold over a recent period to evaluating a reliability objective tracked over a long period. This reduces operational noise and focuses alerting on whether your service meets its reliability goals. SLOs don't replace your existing alerts; they bring a distinct method for measuring service reliability.

To get started, refer to [introduction to SLOs in Grafana Cloud](/docs/grafana-cloud/alerting-and-irm/slo/introduction/).

# Alerts overview page

{{< docs/public-preview product="Alerts page" >}}

# Alert view

Grafana Alerting provides a consolidated snapshot of your firing and pending Grafana-managed alerts in a simplified view. For users with complex deployments, it can be difficult to monitor and prioritize critical incidents among a large volume of firing or pending alerts. With the Alert page, you have a view where you can quickly explore and sort your recent alert history and see what alerts require review or action.

To see your firing and pending alerts in the alert page, go to **Alerts & IRM** > **Alerting** > **Alert activity**.

{{< figure src="/media/docs/alerting/alerts-page.png" max-width="750px" alt="Filter your firing and pending alerts in the Alert view." >}}

## How it works

The Alerts page only shows alerts from Grafana-managed alert rules. Grafana uses a metric called `GRAFANA_ALERTS`, which is recorded in the default Mimir data source that is provisioned for cloud users.

{{< admonition type="note" >}}
OSS users need to manually configure this. To configure alert state history for OSS, refer to the <a href="https://grafana.com/docs/grafana/latest/alerting/set-up/configure-alert-state-history/#configure-loki-and-prometheus-for-alert-state">configure Loki and Prometheus for alert state</a> documentation.
{{< /admonition >}}

## Required permissions

To see alerts in the Alerts page, you need both of the following permissions:

- **`alert.rules:read`**: Read the alert rules in the folders that contain them.
- **`datasources:query`**: Query the data source that records alert state history. In Grafana Cloud, this is the provisioned Prometheus data source that stores the `GRAFANA_ALERTS` metric.

The page queries the alert state history data source directly, so it can't display any alerts without the `datasources:query` permission for that data source. This applies even when you can read the alert rules and view them in the **Alert rules** page.

For more information about alerting permissions, refer to [Configure role-based access control](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-rbac/). To find out which data source records alert state history in your Grafana instance, refer to [Configure alert state history](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alert-state-history/).

## Filter alerts in the Alerts page

The page displays the count of alert rules and instances that are both firing and pending. Use the filters to group your alerts and find specific alerts by label.

To group alerts by label, click the **Group by** field and select the labels from the dropdown menu. You can select multiple labels for alert grouping.

To filter by specific label values, click the **Filters** field and select a label to filter by, followed by an expression of what label value you want to use. You can enter multiple label values in your search.

You can also select a time range from the time picker to further adjust your results. Click the **time range** field and enter an absolute time range or select a period from the quick ranges list to apply a new time window to the display results.

# Declare incidents from firing alerts

Declare an incident from a firing alert to streamline your alert to incident workflow.

## Before you begin

- Ensure you have [Grafana Incident](/docs/grafana-cloud/incident/) installed.
- You must have a firing alert.

## Procedure

To declare an incident from a firing alert, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Alert rules**.
1. From the Alert rules page, click the **Firing** filter to display firing alerts. Find the firing alert that you want to declare an incident for.
1. Click **More** -> **Declare Incident**.

   Alternatively, you can declare an incident from the Alert details page.

   The **Declare Incident** pop-up opens in the Grafana Incident application.

1. In the **Declare Incident** pop-up, enter the **What's going on?** field.

   This field is pre-filled with the name of the alert rule, but you can edit it as required.

   The alert rule is also linked to the incident.

1. Select a severity.
1. Add labels, as required.
1. Click **More options** to include a channel prefix and status.
1. Click **Automated options** to enable automated actions configured by admins.
1. Click **Declare Incident**.

## Next steps

View and manage the incident in the **Grafana Incident** application.

{{< figure src="/media/docs/alerting/incident-response-management-timeline-ui.png" max-width="750px" alt="Incident timeline view in Grafana Incident" >}}

For more information, refer to the [Grafana Incident documentation](/docs/grafana-cloud/incident/configure-settings/).

# View active notifications

The Active notifications page lists groups of alerts (or alert instances) that are actively triggering notifications.

By default, Grafana Alerting [groups similar alerts into a single notification](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/).

In this view, you can:

- Find alert groups and the state of their notifications.
- Filter for alert instances that match specific criteria.

The Active notifications view is useful for debugging and verifying how notifications are grouped based on your notification policy settings.

To review notifications that were already sent, refer to [View notification history](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-notification-history/).

## View alert groups and notification state

To view alert groups, complete the following steps.

1. Click **Alerts & IRM** -> **Alerting** -> **Alert activity**.
1. Select the **Active notifications** tab to view the list of groups firing notifications.

   {{< figure src="/media/docs/alerting/active-notifications-view2.png" max-width="750px" alt="Active notifications view in Grafana Alerting" >}}

   By default, alert groups are grouped by the notification policies grouping.

   Each group displays its label set, contact point, and the number of alert instances (or alerts).

   Then, click on a group to access its alert instances. You can find alert instances by their label set and view their notification state.

### Notification states

The notification state of an alert instance can be in one of the following states:

- **Unprocessed**: The alert is received but its notification has not been processed yet.
- **Suppressed**: The alert has been silenced.
- **Active**: The alert notification has been handled. The alert is still firing and continues to be managed.

### Filter alerts

You can filter by label, state, or Alertmanager:

- **By label**: In **Search**, enter an existing label to view alerts matching the label. For example, `environment=production,region=~US|EU,severity!=warning`.

- **By state**: In **States**, select from Active, Suppressed, or Unprocessed states to view alerts matching your selected state. All other alerts are hidden.

- **By Alertmanager**: In the **Alertmanager** dropdown, select an [external Alertmanager](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alertmanager/) to view only alert groups for that specific Alertmanager. By default, the `Grafana` Alertmanager is selected.

### Custom group

From **Custom group by** dropdown, select a combination of labels to view a grouping other than the default. This helps validate the [grouping settings of your notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/).

If an alert does not contain labels specified either in the grouping of the default policy or the custom grouping, then the alert is added to a catch-all group with a header of `No grouping`.

## View notification errors

{{< admonition type="note" >}}

Notification errors are only available with [pre-configured Grafana Alertmanagers](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alertmanager/).

{{< /admonition >}}

Notification errors provide information about why they failed to be sent or were not received.

To view notification errors, navigate to **Alerts & IRM** -> **Alerting** -> **Notification configuration**, then select the **Contact points** tab.

Each contact point displays a message about the status of their latest notification deliveries.

If a contact point is failing, a red message indicates that there are errors delivering notifications. Hover over the error message to see the notification error details.

To review past delivery attempts across contact points, refer to [View notification history](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-notification-history/).

# View alert rules

The Alert rules list view page lists all existing recording and alert rules. The rules are organized in a hierarchical structure, from folder -> rule group -> rules.

To access the Alert rules page, click **Alerts & IRM** -> **Alerting** -> **Alert rules**.

Select a group to expand it and view the list of alert rules within that group.

The list view includes a number of filters to simplify managing large volumes of alerts.

## Filter and save searches

Click the **Filter** button to open the filter popup. You can filter by name, label, folder/namespace, evaluation group, data source, contact point, rule source, rule state, rule type, and the health of the alert rule from the popup menu. Click **Apply** at the bottom of the filter popup to enact the filters as you search.

Click the **Saved searches** button to open the list of previously saved searches, or click **+ Save current search** to add your current search to the saved searches list. You can also rename a saved search or set it as a default search. When you set a saved search as the default search, the Alert rules page opens with the search applied.

{{< figure src="/media/docs/alerting/alerting-saved-searches.png" max-width="750px" alt="Alert rule filter options" >}}

### Search syntax

The search input accepts a structured `key:value` syntax. The key on the left of the colon is the filter type, and the value on the right is what you want to match. Use quotation marks around values that contain spaces, for example `rule:"High CPU usage"`.

The following filter types are supported:

- `datasource`
- `namespace`
- `label`
- `group`
- `rule`
- `state`
- `type`
- `health`

For example:

- `label:severity` filters rules by the `severity` label.
- `health:error` filters rules whose health is in an error state.
- `rule:"High CPU usage"` filters rules by name.

You can combine multiple filters in a single search. Any text entered without a `key:` prefix is treated as a query that filters alert rules by name.

#### Match labels with regular expressions

The `label` filter accepts Prometheus-style matchers, so you can match label values with regular expressions instead of exact strings. The other filter types don't support regular expressions.

A label matcher uses the form `label:<KEY><OPERATOR><VALUE>`. The following operators are supported:

- `=`: matches rules whose label value is exactly equal to the value.
- `!=`: matches rules whose label value isn't equal to the value.
- `=~`: matches rules whose label value matches the regular expression.
- `!~`: matches rules whose label value doesn't match the regular expression.

For example:

- `label:severity=~crit.*` matches rules where the `severity` label matches the regular expression, such as `critical` or `crit-high`.
- `label:team!~fe.*` matches rules where the `team` label doesn't start with `fe`.

Wrap the matcher in quotation marks when the key or value contains spaces or special characters, for example `label:"team=~fe.*devs"`.

Regular expressions use [RE2 syntax](https://github.com/google/re2/wiki/Syntax) and are fully anchored, so the pattern must match the entire label value. Use `.*` or `.+` to match part of a value, for example `label:severity=~.*crit.*`. Matching is case-sensitive by default. To match without case sensitivity, add the `(?i)` inline flag, for example `label:severity=~(?i)critical`.

Label matchers apply to both the alert rule's labels and the labels on its alert instances, so a rule matches when either set of labels satisfies the matcher. For labels whose values are set by Go templates, the matcher runs against the rendered static value, not the template expression, so you can't use a template query to find a match.

The search input and the **Filter** popup are kept in sync, so changes made in either place are reflected in the other.

## Change alert rules list view

You can also change how the rule list is displayed using the **View as** option.

- **Grouped**: Displays rules grouped in a hierarchical structure, from folder/namespace, to evaluation group, to the individual rules. This is the default view.

- **List**: Displays all rules in a flat, unpaginated list.

{{< figure src="/media/docs/alerting/view-alert-rule-list-with-actions-2.png" max-width="750px" alt="View alert rule state and alert rule health in Grafana Alerting" >}}

For details on how rule states and alert instance states are displayed, refer to [View alert state](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-state/).

## View, compare and restore alert rules versions.

To view, compare, or restore previous versions for an alert rule, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Alert rules**.
1. Select an alert rule and click **View**.
1. Click the **Versions** tab.  
   The page displays a list of the previous rule versions.

On the Alert rule's Versions page you can view, compare and restore the previous rule versions.

{{< figure src="/media/docs/alerting/screenshot-grafana-alerting-version-history-v3.png" max-width="750px" alt="View alert rule history to compare and restore previous alert rules." >}}

{{< admonition type="note" >}}

- The alert rule does not guarantee sequential version increases.

- In Grafana OSS and Enterprise, the number of alert rule versions can be limited using the [`rule_version_record_limit` option](/docs/grafana/latest/setup-grafana/configure-grafana/#rule_version_record_limit). In Grafana Cloud, free users are allowed a maximum of 10 alert rule versions, while paid users have a maximum of 100 stored alert rule versions.
  {{< /admonition >}}

## Bulk pause or resume alert rules evaluations within a folder

Admin users can pause or resume all of the alert rules evaluations within a folder. To pause or resume all the alert rules evaluations in a folder, click the menu icon and select **Pause all rule evaluation** or **Resume all rule evaluation**.

## Bulk delete all alert rules within a folder

Admin users can delete all of the alert rules within a folder. To delete all the alert rules in a folder, click the menu icon and select **Delete**. Then type "Delete" into the field and click **Delete** to confirm the bulk deletion.

## Permanently delete or restore deleted alert rules

Only users with an Admin role can restore deleted Grafana-managed alert rules. After an alert rule is restored, it is restored with a new, different UID from the one it had before.

1. Go to **Alerts & IRM** -> **Alerting** -> **Alert rules**, then click the **Recently deleted** tab.
1. Click the **Restore** button to restore the alert rule or click **Delete permanently** to delete the alert rule.

{{< admonition type="note" >}}
Deleted alert rules are stored for 30 days. Grafana Enterprise and OSS users can adjust the length of time for which the rules are stored in the Grafana configuration file's `[unified_alerting].deleted_rule_retention` field. For an example of how to modify the Grafana configuration file, refer to the [documentation example here](/docs/grafana/latest/alerting/set-up/configure-alert-state-history/#configuring-grafana).  
{{< /admonition >}}

# View alert state history

View a history of all alert events generated by your Grafana-managed alert rules from one centralized page. This helps you see patterns in your alerts over time, observe trends, make predictions, and even debug alerts that might be firing too often.

An alert event is displayed each time an alert instance changes its state over a period of time. All alert events are displayed regardless of whether silences or mute timings are set, so you’ll see a complete set of your data history even if you’re not necessarily being notified.

{{< admonition type="note" >}}
Grafana OSS and Grafana Enterprise users must [configure alert state history in Loki](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alert-state-history/) to view the **History page** and **State history view**.
{{< /admonition >}}

## View from the History page

The History page shows the history and state changes of all Grafana-managed alert rules. You can filter by labels and alert states.

The **History** page also includes a **Notifications** tab for [notification history](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-notification-history/).

Users can only view the history of alert rules they have permission to access (RBAC).

To access the History page, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **History**.

   {{< figure src="/media/docs/alerting/alert-history-page.png" max-width="750px" alt="History page in Grafana Alerting" >}}

   An Alert Events chart shows your alert activity within a specific time frame. Each alert event represents an alert instance that changed its state at a particular point in time. The history of the data is displayed over a period of time.

2. Filter by labels either by entering a label or clicking a label of an instance in the list of events.
3. Filter by current state and previous state by selecting a state from the drop-down or by clicking the states from the list of events.
   Zoom in by dragging on the chart or use the time picker.

   {{< admonition type="note" >}}
   If you exceed the 5000 alerts limit, you may see data missing from the chart. To see complete results, narrow the time frame.
   {{< /admonition >}}

4. Under the chart, there is a list of events. Each event represents a state change on an alert instance. Expand a row to see the number of transitions for the alert instance, a state graph, and the value in the transition.
5. Click the alert rule name to jump to the History tab in the Alert Rule view.

   {{< figure src="/media/docs/alerting/alerting-alert-history-tab.png" max-width="750px" alt="Alert History tab in Grafana Alerting" >}}

## Use Grafana Assistant to analyze alert state history

{{< admonition type="note" >}}
This feature is available in Grafana Cloud when Grafana Assistant is enabled.
{{< /admonition >}}

The **Analyze with Assistant** button provides AI-powered analysis of your alert history to help you understand and troubleshoot alert patterns. Located in the top-right corner of the History page event list, this button uses Grafana Assistant to analyze the events displayed in your current view.

When you click the AI Triage button, the Grafana Assistant analyzes:

- Alert state transitions over the selected time range
- Alert instance patterns and frequency
- Common labels and characteristics of firing alerts
- Temporal patterns in alert behavior

The AI assistant can help you:

- Identify root causes of alert storms
- Detect patterns in alert firing behavior
- Understand correlations between different alert instances
- Get suggestions for improving alert configurations

To use the Analyze with Assistant feature:

1. Navigate to the History page as described above.
2. Filter the events to focus on the alerts you want to analyze using labels, states, or time range.
3. Click the **Analyze with Assistant** button in the top-right corner of the event list.
4. Review the AI-generated analysis and recommendations.

The AI analysis is based on the currently displayed events, so filtering your view to specific alerts or time periods will result in more focused insights.

## View from the State history view

Use the State history view to get insight into how your individual alert instances behave over time.

View information on when a state change occurred, what the previous state was, the current state, any other alert instances that changed their state at the same time as well as what the query value was that triggered the change.

To access the State history view, complete the following steps.

1. Navigate to **Alerts & IRM** -> **Alerting** -> **Alert rules**.
1. Click an alert rule.
1. Select **Show state history**.

   {{< figure src="/media/docs/alerting/alerting-state-history-view.png" max-width="750px" alt="State history view in Grafana Alerting" >}}

   The timeline view at the top displays a timeline of changes for the past hour, so you can track how your alert instances are behaving over time.

   The bottom part shows the alert instances, their previous and current state, the value of each part of the expression and a unique set of labels.

   Common labels are displayed at the top to make it easier to identify different alert instances.

1. From the timeline view, hover over a time to get an automatic display of all the changes that happened at that particular moment.

   These changes are displayed in real time in the timestamp view at the bottom of the page. The timestamp view is a list of all the alert instances that changed state at that point in time. The visualization only displays 12 instances by default.

   The value shown for each instance is for each part of the expression that was evaluated.

1. Click the labels to filter and narrow down the results.

## Explore and query the alert history

You can also use [Grafana Explore](/docs/grafana/<GRAFANA_VERSION>/visualizations/explore/) to query the data sources that store alert history.

For details on querying these data sources, refer to [Alerting Meta monitoring](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor/).

# View alert state

An alert rule and its corresponding alert instances can transition through distinct states during the alert rule evaluation.

{{< figure src="/media/docs/alerting/alert-rule-evaluation-full-statediagram_v3.svg" alt="A diagram of the distinct alert instance states and transitions." max-width="750px" >}}

There are three key components that helps us understand the behavior of our alerts:

- [Alert Instance State](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/#alert-instance-states): Refers to the state of the individual alert instances.
- [Alert Rule State](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/alert-rule-state-and-health/): Determined by the "worst state" among its alert instances.
- [Alert Rule Health](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/alert-rule-state-and-health/): Indicates the status in cases of `Error` or `NoData` events.

## View alert rule and instance states

To view the details of your alert rules and the status of alert instances:

1. Click **Alerts & IRM** -> **Alerting**.
1. Click **Alert rules** to view the list of existing alert rules.

   {{< figure src="/media/docs/alerting/view-alert-rule-list-with-actions2.png" max-width="750px" alt="View alert rule state and alert rule health in Grafana Alerting" >}}

   Each alert rule shows its state, summary, and available actions such as **Pause evaluation**, **Silence notifications**, **Export**, **Delete**, and more.

1. Click on an alert rule to view additional details and its resulting alert instances.

### View from the alert rule details page

To view more alert rule details, complete the following steps.

1. Click **Alerts & IRM** -> **Alerting** -> **Alert rules**.
1. Click to expand an alert rule.
1. Click the alert name to go to the alert details view.

   {{< figure src="/media/docs/alerting/alert-rule-view-page-with-breadcrumb.png" max-width="750px" alt="Alert rule view page in Grafana Alerting" >}}

   The namespace and group are shown in the breadcrumb navigation. They are interactive and can be used to filter rules by namespace or group.

   The rest of the alert detail content is split up into tabs:

   **Query and conditions**

   View the details of the query that is used for the alert rule, including the expressions and intermediate values for each step of the expression pipeline. A graph view is included for range queries and data sources that return time series-like data frames.

   **Instances**

   Explore each alert instance, its status, labels and various other metadata for multi-dimensional alert rules.

   Use **Search by label** to enter search criteria using label selectors. For example, `environment=production,region=~US|EU,severity!=warning`.

   **History**

   Explore the recorded history for an alert rule. You can also filter by alert state.

   **Details**

   Debug or audit using the alert rule metadata and view the alert rule annotations.

## View alert state on panels

When an [alert rule is linked to a time series panel](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/link-alert-rules-to-panels/), the time series panel displays the alert state and alert events.

A heart icon near the panel title shows the current alert state:

- A broken red heart when the alert is in `Alerting` state.
- A green heart when the alert is in `Normal` state.

Colored annotations indicate changes in alert state, such as pending, alerting, and resolving.

{{< figure src="/media/docs/alerting/panel-displays-alert-state.png" max-width="1200px" alt="A panel with a firing alert and annotations that display the pending and alerting state changes." >}}

Additionally, Grafana provides an [alert list panel](/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/visualizations/alert-list/) that you can add to a dashboard to display a list of alerts and their states.

{{< figure src="/static/img/docs/alert-list-panel/alert-list-panel.png" max-width="850px" alt="Two alert list panels displaying distinct lists of alerts." >}}

You can configure the alert list panel with various visualization options and filters to control how alerts are displayed. For more details, refer to the [Alert list documentation](/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/visualizations/alert-list/).

{{< docs/play title="this demo dashboard with alert list panels and linked alert rules" url="https://play.grafana.org/d/000000074/alerting?tech=docs&pg=alerting-demo&plcmt=callout-play&cta=alert-demo-dashboard" >}}

# View notification history

View notifications that Grafana Alerting sent for your Grafana-managed alert rules from one page. Confirm delivery, see which contact point received each notification, and debug failures.

Grafana records a notification event each time it attempts to send a notification to a contact point for a group of alerts. Both firing and resolved notifications are recorded, so you'll see a complete delivery history even when an alert has already recovered.

Notification history is different from [active notifications](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-active-notifications/). Active notifications show alerts that are currently grouped and waiting to be sent. Notification history shows notifications that were already sent.

The **History** page also includes an **Alert events** tab for [alert state history](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-state-history/).

{{< admonition type="note" >}}
Grafana OSS and Grafana Enterprise users must store notification history in Loki. Set `enabled = true` in the `[unified_alerting.notification_history]` section and configure the connection to Loki. Refer to [Before you begin](#before-you-begin).
{{< /admonition >}}

## Before you begin

Notification history stores Alertmanager notification logs in Loki. In Grafana OSS and Grafana Enterprise, configure the connection to Loki in the `[unified_alerting.notification_history]` section of your Grafana configuration file:

```ini
[unified_alerting.notification_history]
# Enable the notification history functionality in Unified Alerting.
# Alertmanager notification logs will be stored in Loki.
enabled = true

# URL of the Loki instance.
loki_remote_url = http://localhost:3100

# Optional tenant ID to attach to requests sent to Loki.
loki_tenant_id =

# Optional basic authentication for requests sent to Loki.
loki_basic_auth_username =
loki_basic_auth_password =
```

Replace `http://localhost:3100` with the URL of your Loki instance.

You can attach extra labels to every notification history record with the `[unified_alerting.notification_history.external_labels]` section.

In Grafana Cloud, notification history is managed for you.

## View from the History page

The **History** page shows notifications sent for all Grafana-managed alert rules. Filter by labels, status, contact point, and delivery outcome.

You can only view notification history for alert rules you have permission to access.

To view notification history from the **History** page, follow these steps.

1. Click **Alerts & IRM** > **Alerting** > **History**.
1. Select the **Notifications** tab.

   Each row represents a notification sent to a contact point. The list shows the following columns:
   - **Time**: When the notification was sent.
   - **Status**: Whether the notification was for a **Firing** or **Resolved** alert.
   - **Group Labels**: The labels that identify the group of alerts included in the notification.
   - **Contact point**: The contact point and integration that received the notification.

   Failed notifications are marked with a **Failed** badge. Hover over the badge to see the error details.

1. Filter the list to narrow down the results:
   - **Labels**: Enter a label or click a label on a notification in the list.
   - **Status**: Filter by **Firing** or **Resolved**.
   - **Delivery outcome**: Show only successful or failed notifications.
   - **Contact point**: Filter by contact point.
   - **Time range**: Adjust the time range with the time picker.

1. Expand a row to see the alerts included in the notification, grouped into **Firing Alerts** and **Resolved Alerts**. Each alert shows its labels, annotations, summary, and description, and links to the alert rule that generated it.

## View the details of a single notification

Select **View** on a notification row to open its detail page.

The detail page shows the full context of the notification, including the contact point and integration used, the delivery outcome, any error returned, and the complete list of firing and resolved alerts that were included.

## View notification history for an alert rule

You can review notifications sent for a specific alert rule without leaving the rule.

To view notification history for an alert rule, follow these steps.

1. Click **Alerts & IRM** > **Alerting** > **Alert rules**.
1. Click an alert rule.
1. Select the **Notifications** tab.

The tab shows the same notification list as the **History** page, pre-filtered to alerts generated by that rule.

## Explore and query notification history

Because notification history is stored in Loki, you can query it directly in Loki through [Grafana Explore](/docs/grafana/<GRAFANA_VERSION>/visualizations/explore/). Use Explore to build custom dashboards or run free-form investigations of your notification delivery.

## Next steps

- Refer to [View alert state history](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-state-history/) to review alert state changes on the **Alert events** tab of the **History** page.
- Refer to [View active notifications](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-active-notifications/) to inspect alerts that are currently grouped and waiting to be sent.
- Refer to [Manage contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/) to configure where notifications are delivered.

# Meta monitoring

Monitor your alerting metrics to ensure you identify potential issues before they become critical.

Meta monitoring is the process of monitoring your monitoring system and alerting when your monitoring is not working as it should.

In order to enable you to meta monitor, Grafana provides predefined metrics.

Identify which metrics are critical to your monitoring system (i.e. Grafana) and then set up how you want to monitor them.

You can use meta-monitoring metrics to understand the health of your alerting system in the following ways:

1. Optional: Create a dashboard in Grafana that uses this metric in a panel (just like you would for any other kind of metric).
2. Optional: Create an alert rule in Grafana that checks this metric regularly (just like you would do for any other kind of alert rule).
3. Optional: Use the Explore module in Grafana.

## Metrics for Grafana-managed alerts

To meta monitor Grafana-managed alerts, you can collect two types of metrics in a Prometheus instance:

- **State history metric (`GRAFANA_ALERTS`)** — Exported by Grafana Alerting as part of alert state history.

- **Scraped metrics** — Exported by Grafana’s `/metrics` endpoint to monitor alerting activity and performance.

You need a Prometheus-compatible server to collect and store these metrics.

### `GRAFANA_ALERTS` metric

If you have configured [Prometheus for alert state history](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alert-state-history/), Grafana writes alert state changes to the `ALERTS` metric:

```
GRAFANA_ALERTS{alertname="", alertstate="", grafana_alertstate="", grafana_rule_uid="", <additional alert labels>}
```

This `GRAFANA_ALERTS` metric is compatible with the `ALERTS` metric used by Prometheus Alerting and includes two additional labels:

1. A new `grafana_rule_uid` label for the UID of the Grafana rule.
2. A new `grafana_alertstate` label for the Grafana alert state, which differs slightly from the equivalent Prometheus state included in the `alertstate` label.

Alert labels are automatically converted before being written to Prometheus to ensure compatibility. Prometheus requires label names to start with a letter or underscore (`_`), followed only by letters, numbers, or additional underscores. Invalid characters are replaced during conversion. For example, `1my-label` becomes `_my_label`.

| Grafana state  | `alertstate`          | `grafana_alertstate`  |
| -------------- | --------------------- | --------------------- |
| **Alerting**   | `firing`              | `alerting`            |
| **Recovering** | `firing`              | `recovering`          |
| **Pending**    | `pending`             | `pending`             |
| **Error**      | `firing`              | `error`               |
| **NoData**     | `firing`              | `nodata`              |
| **Normal**     | _(no metric emitted)_ | _(no metric emitted)_ |

You can then query this metric like any other Prometheus metric:

{{< code >}}

```firing-alerts
GRAFANA_ALERTS{grafana_alertstate="alerting"}
```

```recovering-alerts
GRAFANA_ALERTS{grafana_alertstate="recovering"}
```

```critical-alerts-in-pending
GRAFANA_ALERTS{grafana_alertstate="pending", severity="critical"}
```

    {{< /code >}}

### Scraped metrics

To collect scraped Alerting metrics, configure Prometheus to scrape metrics from Grafana.

```yaml
- job_name: grafana
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  static_configs:
    - targets:
        - grafana:3000
```

The Grafana ruler, which is responsible for evaluating alert rules, and the Grafana Alertmanager, which is responsible for sending notifications of firing and resolved alerts, provide a number of metrics that let you observe them.

#### grafana_alerting_alerts

This metric is a counter that shows you the number of `normal`, `pending`, `alerting`, `nodata` and `error` alerts. For example, you might want to create an alert that fires when `grafana_alerting_alerts{state="error"}` is greater than 0.

#### grafana_alerting_schedule_alert_rules

This metric is a gauge that shows you the number of alert rules scheduled. An alert rule is scheduled unless it is paused, and the value of this metric should match the total number of non-paused alert rules in Grafana.

#### grafana_alerting_schedule_periodic_duration_seconds_bucket

This metric is a histogram that shows you the time it takes to process an individual tick in the scheduler that evaluates alert rules. If the scheduler takes longer than 10 seconds to process a tick then pending evaluations start to accumulate such that alert rules might later than expected.

#### grafana_alerting_schedule_query_alert_rules_duration_seconds_bucket

This metric is a histogram that shows you how long it takes the scheduler to fetch the latest rules from the database. If this metric is elevated, `schedule_periodic_duration_seconds` is also evaluated.

#### grafana_alerting_scheduler_behind_seconds

This metric is a gauge that shows you the number of seconds that the scheduler is behind where it should be. This number increases if `schedule_periodic_duration_seconds` is longer than 10 seconds, and decrease when it is less than 10 seconds. The smallest possible value of this metric is 0.

#### grafana_alerting_notification_latency_seconds_bucket

This metric is a histogram that shows you the number of seconds taken to send notifications for firing and resolved alerts. This metric lets you observe slow or over-utilized integrations, such as an SMTP server that is being given emails faster than it can send them.

#### grafana_alerting_state_history_writes_failed_total

This metric is a counter that shows you the number of failed writes to the configured alert state history backend. It includes a `backend` label to distinguish between different backends (such as `loki` or `prometheus`).

For example, you might want to create an alert that fires when `grafana_alerting_state_history_writes_failed_total{backend="prometheus"}` is greater than 0 to detect when Prometheus remote write is failing.

## Logs for Grafana-managed alerts

If you have configured [Loki for alert state history](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alert-state-history/), logs related to state changes in Grafana-managed alerts are stored in the Loki data source.

You can use **Grafana Explore** and the Loki query editor to search for alert state changes.

{{< code >}}

```basic-query
{from="state-history"} | json
```

```additional-filters
{from="state-history"} | json | previous=~"Normal.*" | current=~"Alerting.*"
```

```failing-rules
{from="state-history"} | json | current=~"Error.*"
```

    {{< /code >}}

In the **Logs** view, you can review details for individual alerts by selecting fields such as:

- `previous`: previous alert instance state.
- `current`: current alert instance state.
- `ruleTitle`: alert rule title.
- `ruleID` and `ruleUID`.
- `labels_alertname`, `labels_new_label`, and `labels_grafana_folder`.
- Additional available fields.

Alternatively, you can access the [History page](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-state-history/) in Grafana to visualize and filter state changes for individual alerts or all alerts.

If you have enabled notification history, you can also refer to [View notification history](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-notification-history/) to review sent notifications and delivery outcomes in the **History** page **Notifications** tab.

## Metrics for Mimir-managed alerts

To meta monitor Grafana Mimir-managed alerts, open source and on-premise users need a Prometheus/Mimir server, or another metrics database to collect and store metrics exported by the Mimir ruler.

#### rule_evaluation_failures_total

This metric is a counter that shows you the total number of rule evaluation failures.

## Metrics for Alertmanager

To meta monitor the Alertmanager, you need a Prometheus/Mimir server, or another metrics database to collect and store metrics exported by Alertmanager.

For example, if you are using Prometheus you should add a `scrape_config` to Prometheus to scrape metrics from your Alertmanager.

```yaml
- job_name: alertmanager
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  static_configs:
    - targets:
        - alertmanager:9093
```

The following is a list of available metrics for Alertmanager.

#### alertmanager_alerts

This metric is a counter that shows you the number of active, suppressed, and unprocessed alerts in Alertmanager. Suppressed alerts are silenced alerts, and unprocessed alerts are alerts that have been sent to the Alertmanager but have not been processed.

#### alertmanager_alerts_invalid_total

This metric is a counter that shows you the number of invalid alerts that were sent to Alertmanager. This counter should not exceed 0, and so in most cases, create an alert that fires if whenever this metric increases.

#### alertmanager_notifications_total

This metric is a counter that shows you how many notifications have been sent by Alertmanager. The metric uses a label "integration" to show the number of notifications sent by integration, such as email.

#### alertmanager_notifications_failed_total

This metric is a counter that shows you how many notifications have failed in total. This metric also uses a label "integration" to show the number of failed notifications by integration, such as failed emails. In most cases, use the `rate` function to understand how often notifications are failing to be sent.

#### alertmanager_notification_latency_seconds_bucket

This metric is a histogram that shows you the amount of time it takes Alertmanager to send notifications and for those notifications to be accepted by the receiving service. This metric uses a label "integration" to show the amount of time by integration. For example, you can use this metric to show the 95th percentile latency of sending emails.

## Metrics for Alertmanager in high availability mode

If you are using Alertmanager in high availability mode there are a number of additional metrics that you might want to create alerts for.

#### alertmanager_cluster_members

This metric is a gauge that shows you the current number of members in the cluster. The value of this gauge should be the same across all Alertmanagers. If different Alertmanagers are showing different numbers of members then this is indicative of an issue with your Alertmanager cluster. You should look at the metrics and logs from your Alertmanagers to better understand what might be going wrong.

#### alertmanager_cluster_failed_peers

This metric is a gauge that shows you the current number of failed peers.

#### alertmanager_cluster_health_score

This metric is a gauge showing the health score of the Alertmanager. Lower values are better, and zero means the Alertmanager is healthy.

#### alertmanager_cluster_peer_info

This metric is a gauge. It has a constant value `1`, and contains a label called "peer" containing the Peer ID of each known peer.

#### alertmanager_cluster_reconnections_failed_total

This metric is a counter that shows you the number of failed peer connection attempts. In most cases you should use the `rate` function to understand how often reconnections fail as this may be indicative of an issue or instability in your network.

# Configure alert state history

Alerting can record all alert rule state changes for your Grafana managed alert rules in a Loki or Prometheus instance, or in both.

- With Prometheus, you can query the `GRAFANA_ALERTS` metric for alert state changes in **Grafana Explore**.
- With Loki, you can query and view alert state changes in **Grafana Explore** and the [Grafana Alerting History views](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-state-history/).

## Configure Loki for alert state

The following steps describe a basic configuration:

1. **Configure Loki**

   The default Loki settings might need some tweaking as the state history view might query up to 30 days of data.

   The following change to the default configuration should work for most instances, but look at the full Loki configuration settings and adjust according to your needs.

   ```yaml
   limits_config:
     split_queries_by_interval: '24h'
     max_query_parallelism: 32
   ```

   As this might impact the performances of an existing Loki instance, use a separate Loki instance for the alert state history.

1. **Configure Grafana**

   The following Grafana configuration instructs Alerting to write alert state history to a Loki instance:

   ```toml
   [unified_alerting.state_history]
   enabled = true
   backend = loki

   # The URL of the Loki server
   loki_remote_url = http://localhost:3100
   ```

1. **Configure the Loki data source in Grafana**

   Add the [Loki data source](/docs/grafana/<GRAFANA_VERSION>/datasources/loki/) to Grafana.

If everything is set up correctly, you can access the [History view and History page](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/view-alert-state-history/) to view and filter alert state history. You can also use **Grafana Explore** to query the Loki instance, see [Alerting Meta monitoring](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor/) for details.

## Configure Prometheus for alert state (GRAFANA_ALERTS metric)

You can also configure a Prometheus instance to store alert state changes for your Grafana-managed alert rules. However, this setup does not enable the **Grafana Alerting History views**, as Loki does.

Instead, Grafana Alerting writes alert state data to the `GRAFANA_ALERTS` metric-similar to how Prometheus Alerting writes to the `ALERTS` metric.

```
GRAFANA_ALERTS{alertname="", alertstate="", grafana_alertstate="", grafana_rule_uid="", <additional alert labels>}
```

The following steps describe a basic configuration:

1. **Configure Prometheus**

   Enable the remote write receiver in your Prometheus instance by setting the `--web.enable-remote-write-receiver` command-line flag. This enables the endpoint to receive alert state data from Grafana Alerting.

1. **Configure the Prometheus data source in Grafana**

   Add the [Prometheus data source](/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/) to Grafana.

   In the [Prometheus data source configuration options](/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/configure/), set the **Prometheus type** to match your Prometheus instance type. Grafana Alerting uses this option to identify the remote write endpoint.

1. **Configure Grafana**

   The following Grafana configuration instructs Alerting to write alert state history to a Prometheus instance:

   ```toml
   [unified_alerting.state_history]
   enabled = true
   backend = prometheus
   # Target data source UID for writing alert state changes.
   prometheus_target_datasource_uid = <DATA_SOURCE_UID>

   # (Optional) Metric name for the alert state metric. Default is "GRAFANA_ALERTS".
   # prometheus_metric_name = GRAFANA_ALERTS
   # (Optional)  Timeout for writing alert state data to the target data source. Default is 10s.
   # prometheus_write_timeout = 10s
   ```

Grafana reads alert state history from the same data source, so users need the `datasources:query` permission for it to view the [Alerts page](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor-status/alerts-page/).

You can then use **Grafana Explore** to query the alert state metric. For details, refer to [Alerting Meta monitoring](/docs/grafana/<GRAFANA_VERSION>/alerting/monitor/).

```promQL
GRAFANA_ALERTS{alertstate='firing'}
```

## Configure Loki and Prometheus for alert state

You can also configure both Loki and Prometheus to record alert state changes for your Grafana-managed alert rules.

Start with the same setup steps as shown in the previous [Loki](#configure-loki-for-alert-state) and [Prometheus](#configure-prometheus-for-alert-state-alerts-metric) sections. Then, adjust your Grafana configuration as follows:

```toml
[unified_alerting.state_history]
enabled = true
backend = multiple

primary = loki
# URL of the Loki server.
loki_remote_url = http://localhost:3100

secondaries = prometheus
# Target data source UID for writing alert state changes.
prometheus_target_datasource_uid = <DATA_SOURCE_UID>

```

# Manage access using folders or data sources

You can extend the access provided by a role to alert rules and rule-specific silences by assigning permissions to individual folders or data sources.

{{< admonition type="note" >}}
Data source query permissions are required to create an alert rule or modify an alert rule's query using that data source.
{{< /admonition >}}

This allows different users, teams, or service accounts to have customized access to modify or silence alert rules in specific folders or using specific data sources.

Details on the additional access provided by folder permissions are below.

| Folder permission | Additional Access                                                                                                                           |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| View              | Read access to all general silences. Read access to alert rules and their rule-specific silences _only_ in the given folder and subfolders. |
| Edit              | View access and write access to alert rules and their rule-specific silences _only_ in the given folder and subfolders.                     |
| Admin             | Same additional access as Edit.                                                                                                             |

Some example combinations:

| Role          | Folder permission | Access                                                                                                                                                                   |
| ------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| No Basic Role | -                 | None. No access to alert rules or rule-specific silences in the folder.                                                                                                  |
| No Basic Role | View              | Read access to all general silences. Read access to alert rules and their rule-specific silences _only_ in the given folder and subfolders.                              |
| No Basic Role | Edit              | Above access and write access to alert rules (given necessary data source query permissions) and their rule-specific silences _only_ in the given folder and subfolders. |
| Viewer        | -                 | Read access to alert rules and silences in all folders.                                                                                                                  |
| Viewer        | Edit              | Read access to alert rules and silences in all folders. Write access to alert rules and their rule-specific silences _only_ in the given folder and subfolders.          |
| Editor        | View              | Read and write access to alert rules and silences in all folders. Access can only be extended.                                                                           |

{{< admonition type="note" >}}
You can't use folders to customize access to notification resources.
{{< /admonition >}}

## Folder permissions

To manage folder permissions, complete the following steps.

1. In the left-side menu, click **Dashboards**.
1. Choose the folder you want to add permissions for.

{{< admonition type="note" >}}It doesn’t matter which tab you’re on (Dashboards, Panels, or Alert rules); the folder permission you set applies to all.{{< /admonition >}}

2. Click **Manage permissions** from the Folder actions menu.
3. Update or add permissions as required.

## Data source permissions

By default, users with the basic roles Admin, Editor, and Viewer roles have query access to data sources for Grafana Alerting.

If you used fixed roles or custom roles, you need to update data source permissions to create rules or update rule queries.

Alternatively, an administrator can assign the role **Datasource Reader**, which grants the user access to all data sources.

To manage data source permissions, complete the following steps.

1. In the left-side menu, click **Connections** > **Data sources**.
1. Click the data source you want to change the permissions for.
1. Click the **Permissions** tab.
1. In the **Permission column**, update the permission, or remove it by clicking **X**.

# Manage access using roles

In Grafana Enterprise and Grafana Cloud there are Basic, Fixed, and Custom roles.

## Basic roles

There are four basic roles: `Admin`, `Editor`, `Viewer`, and `None`. Each basic role contains a number of fixed roles.

Details of the basic roles and the access they provide for Grafana Alerting are below.

| Role   | Access                                                                                                                                                                                                                                              |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Admin  | Write access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), alert enrichments, and provisioning.                                                        |
| Editor | Write access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), alert enrichments, and provisioning.                                                        |
| Viewer | Read access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), and alert enrichments.                                                                       |
| None   | A blank canvas to assign fixed or custom roles and craft permissions more precisely. For example, if you want to give a user the ability to see alert rules, but not notification settings, add No basic role and then the fixed role Rules reader. |

## Fixed roles

A fixed role is a group of multiple permissions.

Fixed roles provide users more granular access to create, view, and update Alerting resources than you would have with basic roles alone.

Details of the fixed roles and the access they provide for Grafana Alerting are below.

| Display name in UI / Fixed role                                                          | Permissions                                                                                                                                                                                                                                                                                                                | Description                                                                                                                                                                                                                                                                     |
| ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Silences Writer: `fixed:alerting.instances:writer`                                       | All permissions from `fixed:alerting.instances:reader` and<br> `alert.instances:create`<br>`alert.instances:write` for organization scope <br> `alert.instances.external:write` for scope `datasources:*`                                                                                                                  | Add and update silences in Grafana and external providers.                                                                                                                                                                                                                      |
| Instances and Silences Reader: `fixed:alerting.instances:reader`                         | `alert.instances:read` for organization scope <br> `alert.instances.external:read` for scope `datasources:*`                                                                                                                                                                                                               | Read alert instances and silences in Grafana and external providers.                                                                                                                                                                                                            |
| Notifications Writer: `fixed:alerting.notifications:writer`                              | All permissions from `fixed:alerting.notifications:reader`,<br> `fixed:alerting.receivers:writer`,<br> `fixed:alerting.templates:writer`,<br> `fixed:alerting.time-intervals:writer`,<br> `fixed:alerting.routes:writer`,<br> `fixed:alerting.inhibition-rules:writer`,<br> `fixed:alerting.notifications.external:writer` | Add, update, and delete notification policies and contact points in Grafana and external providers.                                                                                                                                                                             |
| Notifications Reader: `fixed:alerting.notifications:reader`                              | All permissions from `fixed:alerting.receivers:reader`,<br> `fixed:alerting.templates:reader`,<br> `fixed:alerting.time-intervals:reader`,<br> `fixed:alerting.routes:reader`,<br> `fixed:alerting.inhibition-rules:reader`,<br> `fixed:alerting.notifications.external:reader`                                            | Read notification policies and contact points in Grafana and external providers.                                                                                                                                                                                                |
| Rules Writer: `fixed:alerting.rules:writer`                                              | All permissions from `fixed:alerting.rules:reader` and <br> `alert.rule:create` <br> `alert.rule:write` <br> `alert.rule:delete` <br> `alert.silences:create` <br> `alert.silences:write` for scope `folders:*` <br> `alert.rules.external:write` for scope `datasources:*`                                                | Create, update, and delete all alert rules and manage rule-specific silences.                                                                                                                                                                                                   |
| Rules Reader: `fixed:alerting.rules:reader`                                              | `alert.rule:read`, `alert.silences:read` for scope `folders:*` <br> `alert.rules.external:read` for scope `datasources:*` <br> `alert.notifications.time-intervals:read` <br> `alert.notifications.receivers:list`                                                                                                         | Read all alert rules and rule-specific silences in Grafana and external providers.                                                                                                                                                                                              |
| Full access: `fixed:alerting:writer`                                                     | All permissions from `fixed:alerting.rules:writer` <br>`fixed:alerting.instances:writer`<br>`fixed:alerting.notifications:writer`                                                                                                                                                                                          | Add, update, and delete alert rules, silences, contact points, and notification policies in Grafana and external providers.                                                                                                                                                     |
| Full read-only access: `fixed:alerting:reader`                                           | All permissions from `fixed:alerting.rules:reader` <br>`fixed:alerting.instances:reader`<br>`fixed:alerting.notifications:reader`                                                                                                                                                                                          | Read alert rules, alert instances, silences, contact points, and notification policies in Grafana and external providers.                                                                                                                                                       |
| Read via Provisioning API + Export Secrets: `fixed:alerting.provisioning.secrets:reader` | `alert.provisioning:read` and `alert.provisioning.secrets:read`                                                                                                                                                                                                                                                            | Read alert rules, alert instances, silences, contact points, and notification policies using the provisioning API and use export with decrypted secrets.                                                                                                                        |
| Access to alert rules provisioning API: `fixed:alerting.provisioning:writer`             | `alert.provisioning:read` and `alert.provisioning:write`                                                                                                                                                                                                                                                                   | Manage all alert rules, notification policies, contact points, templates, in the organization using the provisioning API.                                                                                                                                                       |
| Set provisioning status: `fixed:alerting.provisioning.provenance:writer`                 | `alert.provisioning.provenance:write`                                                                                                                                                                                                                                                                                      | Set provisioning rules for Alerting resources. Should be used together with other regular roles (Notifications Writer and/or Rules Writer.)                                                                                                                                     |
| Contact Point Reader: `fixed:alerting.receivers:reader`                                  | `alert.notifications.receivers:read` for scope `receivers:*`                                                                                                                                                                                                                                                               | Read all contact points.                                                                                                                                                                                                                                                        |
| Contact Point Creator: `fixed:alerting.receivers:creator`                                | `alert.notifications.receivers:create` <br> `alert.notifications.receivers.test:create` for scope `receivers:uid:-`                                                                                                                                                                                                        | Create a new contact point. The user is automatically granted full access to the created contact point.                                                                                                                                                                         |
| Contact Point Writer: `fixed:alerting.receivers:writer`                                  | All permissions from `fixed:alerting.receivers:creator` <br> `alert.notifications.receivers:read`, `alert.notifications.receivers:write`, `alert.notifications.receivers:delete` and `alert.notifications.receivers.test:create` for scope `receivers:*`                                                                   | Create a new contact point and manage all existing contact points.                                                                                                                                                                                                              |
| Templates Reader: `fixed:alerting.templates:reader`                                      | `alert.notifications.templates:read`                                                                                                                                                                                                                                                                                       | Read all notification templates.                                                                                                                                                                                                                                                |
| Templates Writer: `fixed:alerting.templates:writer`                                      | `alert.notifications.templates:read`, `alert.notifications.templates:write`, `alert.notifications.templates:delete`, `alert.notifications.templates.test:write`                                                                                                                                                            | Create new and manage existing notification templates. Test templates with custom payloads.                                                                                                                                                                                     |
| Time Intervals Reader: `fixed:alerting.time-intervals:reader`                            | `alert.notifications.time-intervals:read`                                                                                                                                                                                                                                                                                  | Read all time intervals.                                                                                                                                                                                                                                                        |
| Time Intervals Writer: `fixed:alerting.time-intervals:writer`                            | `alert.notifications.time-intervals:read`, `alert.notifications.time-intervals:write`, `alert.notifications.time-intervals:delete`                                                                                                                                                                                         | Create new and manage existing time intervals.                                                                                                                                                                                                                                  |
| Notification Policies Reader: `fixed:alerting.routes:reader`                             | `notifications.alerting.grafana.app/routingtrees:get` for scope `notifications.alerting.grafana.app/routingtrees:*`                                                                                                                                                                                                        | Read all notification policies.                                                                                                                                                                                                                                                 |
| Notification Policies Writer: `fixed:alerting.routes:writer`                             | All permissions from `fixed:alerting.routes:reader` and<br>`notifications.alerting.grafana.app/routingtrees:create`, and<br>`notifications.alerting.grafana.app/routingtrees:update`, `notifications.alerting.grafana.app/routingtrees:delete` for scope `notifications.alerting.grafana.app/routingtrees:*`               | Create new and manage existing notification policies.                                                                                                                                                                                                                           |
| Inhibition Rules Reader: `fixed:alerting.inhibition-rules:reader`                        | `alert.notifications.inhibition-rules:read` for scope `inhibition-rules:*`                                                                                                                                                                                                                                                 | Read all inhibition rules.                                                                                                                                                                                                                                                      |
| Inhibition Rules Writer: `fixed:alerting.inhibition-rules:writer`                        | All permissions from `fixed:alerting.inhibition-rules:reader` and<br>`alert.notifications.inhibition-rules:write`, `alert.notifications.inhibition-rules:delete` for scope `inhibition-rules:*`                                                                                                                            | Create, update, and delete all inhibition rules.                                                                                                                                                                                                                                |
| Full admin access: `fixed:alerting:admin`                                                | All permissions from `fixed:alerting:writer` and<br>`alert.notifications.receivers.permissions:read`, `alert.notifications.receivers.permissions:write`, `alert.notifications.receivers:readSecrets`, `alert.notifications.receivers:updateProtected` for scope `receivers:*`                                              | Full write access in Grafana and all external providers, including their permissions, protected fields and secrets.                                                                                                                                                             |
| Enrichments Reader: `fixed:alerting.enrichments:reader`                                  | `alert.enrichments:read`                                                                                                                                                                                                                                                                                                   | Read per-rule [alert enrichment](https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/alert-enrichment/) configurations. Alert enrichment is a Grafana Cloud feature; global enrichment settings require the Admin basic role.              |
| Enrichments Writer: `fixed:alerting.enrichments:writer`                                  | `alert.enrichments:read`<br>`alert.enrichments:write`                                                                                                                                                                                                                                                                      | Create and manage per-rule [alert enrichment](https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/alert-enrichment/) configurations. Alert enrichment is a Grafana Cloud feature; global enrichment settings require the Admin basic role. |
| External Notifications Reader: `fixed:alerting.notifications.external:reader`            | `alert.notifications.external:read` for scope `datasources:*`                                                                                                                                                                                                                                                              | Read notification policies and contact points in external providers.                                                                                                                                                                                                            |
| External Notifications Writer: `fixed:alerting.notifications.external:writer`            | All permissions from `fixed:alerting.notifications.external:reader` and<br>`alert.notifications.external:write` for scope `datasources:*`                                                                                                                                                                                  | Add, update, and delete contact points and notification policies in external providers.                                                                                                                                                                                         |

## Create custom roles

Create custom roles of your own to manage permissions. Custom roles contain unique combinations of permissions, actions and scopes. Create a custom role when basic roles and fixed roles do not meet your permissions requirements.

For more information on creating custom roles, refer to [Create custom roles](https://grafana.com/docs/grafana/latest/administration/roles-and-permissions/access-control/manage-rbac-roles/#create-custom-roles).

{{< admonition type="note" >}}
It is not recommended to create custom roles that include `alerting.notifications.receiver` actions with a scope other than `receivers:*`. The UID used in the scope is not stable and changes whenever a contact point is renamed.
{{< /admonition >}}

### Examples

The following examples give you an idea of how you can combine permissions for Grafana Alerting.

A custom role for read access to alert rules in folder F:

<!-- prettier-ignore-start -->
```
PUT access-control/roles
{
	"name": "custom:alert_rules_reader",
	"displayName": "Alert rule reader in folder F",
	"description": "Read access to rules in folder F that use DS1 and DS2",
	"permissions": [
    	{
        	"action": "alert.rules:read",
        	"scope": "folders:uid:UID_F"
    	},
    	{
        	"action": "folders:read",
        	"scope": "folders:uid:UID_F"
    	}
	]
}
```
<!-- prettier-ignore-end -->

A custom role for write access to alert rules that uses simplified routing:

<!-- prettier-ignore-start -->
```
PUT access-control/roles
{
	"name": "custom:alert_rules_updater",
	"displayName": "Alert rules editor in folder F",
	"description": "Edit access to rules in folder F that use DS1 and DS2",
	"permissions": [
    	{
        	"action": "alert.rules:read",
        	"scope": "folders:uid:UID_F"
    	},
    	{
        	"action": "alert.rules:read",
        	"scope": "folders:uid:UID_F"
    	},
    	{
        	"action": "alert.rules:write",
        	"scope": "folders:uid:UID_F"
    	},
    	{
        	"action": "alert.rules:create",
        	"scope": "folders:uid:UID_F"
    	},
    	{
        	"action": "alert.notifications.receivers:list",
    	},
{
        	"action": "alert.notifications.time-intervals:read",
    	},
	]
}
```
<!-- prettier-ignore-end -->

{{< admonition type="note" >}}
Delete the last two permissions if you aren’t using simplified notification routing.
{{< /admonition >}}

## Assign roles

To assign roles, complete the following steps.

1. Navigate to Administration > Users and access > Users, Teams, or Service Accounts.
1. Search for the user, team or service account you want to add a role for.
1. Select the role you want to assign.

# Configure silence access

This article describes how Grafana Alerting controls access to silences and how to configure it for your teams. Use this article when you need to give some users full silence management while limiting others to silencing only their own rules.

Before you begin, ensure you have:

- Admin access to the Grafana organization
- Familiarity with [Grafana RBAC](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-rbac/) and [fixed roles](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-rbac/access-roles/)

## Silence types

Grafana Alerting has two types of silences:

- **General silences** — not attached to any specific alert rule. They can match alerts from any rule in the organization. Only users with org-scoped silence permissions can create them.
- **Rule-specific silences** — linked to a single alert rule via its UID. Access is controlled by the folder that contains the rule.

When you silence a firing alert directly from the alert list, Grafana creates a rule-specific silence automatically by adding the label matcher `__alert_rule_uid__=<rule UID>`. You can also create a rule-specific silence manually by including this matcher.

## Silence permissions

Grafana has two independent sets of permissions for silences. They serve different purposes and apply to different scopes.

### Org-scoped permissions

These permissions apply across the whole organization and cover both silence types.

| Action                   | Description                                    |
| ------------------------ | ---------------------------------------------- |
| `alert.instances:read`   | Read all silences — general and rule-specific  |
| `alert.instances:create` | Create any silence, including general silences |
| `alert.instances:write`  | Update or expire any silence                   |

The built-in `Editor` role includes all three via the `fixed:alerting.instances:writer` fixed role.

### Folder-scoped permissions

These permissions are scoped to a specific folder. Users with only these permissions can manage rule-specific silences for rules in the granted folder, but can't create general silences.

| Action                  | Description                                                                             |
| ----------------------- | --------------------------------------------------------------------------------------- |
| `alert.silences:read`   | Read rule-specific silences for rules in the folder, plus all general silences org-wide |
| `alert.silences:create` | Create rule-specific silences for rules in the folder                                   |
| `alert.silences:write`  | Update or expire rule-specific silences for rules in the folder                         |

{{< admonition type="note" >}}
`alert.silences:read` always includes all general silences, regardless of the folder scope. General silences can suppress notifications for any rule, so users managing rules in a folder need to see silences that may affect their alerts.
{{< /admonition >}}

## Fixed roles

The following fixed roles are relevant to silence management.

| Fixed role                                                        | Permissions                                                     | What it allows                                                |
| ----------------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------- |
| `fixed:alerting.instances:writer` (Silences Writer)               | `alert.instances:create`, `alert.instances:write` (org scope)   | Create and manage general and rule-specific silences org-wide |
| `fixed:alerting.instances:reader` (Instances and Silences Reader) | `alert.instances:read` (org scope)                              | Read all silences                                             |
| `fixed:alerting.rules:writer` (Rules Writer)                      | `alert.silences:create`, `alert.silences:write` for `folders:*` | Create and manage rule-specific silences                      |
| `fixed:alerting.rules:reader` (Rules Reader)                      | `alert.silences:read` for `folders:*`                           | Read rule-specific silences and all general silences          |

## Folder permission mapping in Grafana OSS

In Grafana OSS, custom RBAC isn't available. Instead, folder permissions automatically grant the corresponding silence access.

| Folder permission | Silence access                                                                     |
| ----------------- | ---------------------------------------------------------------------------------- |
| View              | Read all general silences; read rule-specific silences for rules in the folder     |
| Edit              | View access, plus create and update rule-specific silences for rules in the folder |
| Admin             | Same as Edit                                                                       |

In Grafana Enterprise and Grafana Cloud, folder permissions grant the same silence access as in OSS, but you can go further by assigning folder-scoped RBAC permissions independently of rule access — for example, granting silence write access without granting rule write access.

## Scenario: restrict users to rule-specific silences only

Use this setup when a team should be able to silence their own rules but must not create general silences that affect the whole organization.

Grant the following folder-scoped permissions for the team's folder:

| Permission              | Scope                  |
| ----------------------- | ---------------------- |
| `alert.silences:read`   | `folders:<FOLDER_UID>` |
| `alert.silences:create` | `folders:<FOLDER_UID>` |
| `alert.silences:write`  | `folders:<FOLDER_UID>` |

These permissions are included in `fixed:alerting.rules:writer` and are also granted automatically by the folder **Edit** permission.

**Do not** grant `alert.instances:create` or `alert.instances:write` at org scope — these allow creating general silences. They're bundled into `fixed:alerting.instances:writer` and the built-in `Editor` role, so avoid assigning either to users who should be folder-restricted.

With this configuration:

- Users can create silences only for rules in their folder.
- Users can read all general silences.
- Any attempt to create a general silence is rejected.

## Scenario: grant full silence access

Use this setup for on-call engineers or administrators who need full control over all silences.

Assign the following fixed roles:

| Fixed role                        | Grants                                                        |
| --------------------------------- | ------------------------------------------------------------- |
| `fixed:alerting.instances:writer` | Create and manage general and rule-specific silences org-wide |
| `fixed:alerting.rules:writer`     | Create and manage rule-specific silences across all folders   |

Alternatively, assign the built-in `Editor` role, which includes both.

With this configuration:

- Users can create general silences for org-wide maintenance windows.
- Users can create and expire rule-specific silences for any rule.
- Users see all silences across the organization.

## Security considerations

Keep the following in mind when configuring silence access.

- **Org-scoped write actions are powerful.** Granting `alert.instances:create` or `alert.instances:write` at org scope — even to a user who otherwise has only folder-level access — allows them to create or modify silences that affect the entire organization. Review all custom and inherited roles that include these actions.
- **General silences are always visible to folder users.** You can't hide general silences from users who have any folder-scoped `alert.silences:read` permission. If general silences carry sensitive context (such as the name of an active incident), use clear, consistent naming conventions.
- **Use rule-specific silences for team isolation.** In multi-team or multi-tenant deployments, rule-specific silences are the recommended pattern. They respect folder boundaries, which general silences don't.

## Next steps

- [Configure silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/) — create, edit, and expire silences
- [Manage access using roles](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-rbac/access-roles/) — full reference for fixed and custom roles
- [Manage access using folders](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-rbac/access-folders/) — folder-based access for alert rules and silences

# Configure roles and permissions

This guide explains how to configure roles and permissions for Grafana Alerting for Grafana OSS users. You'll learn how to manage access using roles, folder permissions, and contact point permissions.

A user is any individual who can log in to Grafana. Each user is associated with a role that includes permissions. Permissions determine the tasks a user can perform in the system. For example, the Admin role includes permissions for an administrator to create and delete users.

For more information, refer to [Organization roles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/#organization-roles).

## Manage access using roles

Grafana OSS has four basic roles: `Admin`, `Editor`, `Viewer`, and `None`.

The following table describes the access each role provides for Grafana Alerting.

| Role   | Access                                                                                                                                                                                             |
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Admin  | Write access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), and provisioning, as well as assign roles. |
| Editor | Write access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), and provisioning.                          |
| Viewer | Read access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences).                                             |
| None   | No access to alert rules or notification resources. Limit access to only the required resources by assigning fixed or custom roles, or granting folder or contact point permissions.               |

## Assign roles

To assign roles, an admin needs to complete the following steps.

1. Navigate to **Administration** > **Users and access** > **Users, Teams, or Service Accounts**.
1. Search for the user, team or service account you want to add a role for.
1. Add the role you want to assign.

## Manage access using folder permissions

You can extend the access provided by a role to alert rules and rule-specific silences by assigning permissions to individual folders.

This allows different users, teams, or service accounts to have customized access to modify or silence alert rules in specific folders.

Refer to the following table for details on the additional access provided by folder permissions:

| Folder permission | Additional Access                                                                                       |
| ----------------- | ------------------------------------------------------------------------------------------------------- |
| Admin             | Same additional access as Edit.                                                                         |
| Edit              | Write access to alert rules and their rule-specific silences _only_ in the given folder and subfolders. |
| View              | No additional access: all permissions already contained in Viewer role.                                 |

{{< admonition type="note" >}}
You can't use folders to customize access to notification resources.
{{< /admonition >}}

To manage folder permissions, complete the following steps:

1. In the left-side menu, click **Dashboards**.
1. Hover your mouse cursor over a folder and click **Go to folder**.
1. Click **Manage permissions** from the Folder actions menu.
1. Update or add permissions as required.

## Manage access to contact points

Extend or limit the access provided by a role to contact points by assigning permissions to individual contact points.

This allows different users, teams, or service accounts to have customized access to read or modify specific contact points.

Refer to the following table for details on the additional access provided by contact point permissions.

| Contact point permission | Additional Access                                                                                                                             |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| Admin                    | Same additional access as Edit and manage permissions for the contact point. User should have additional permissions to read users and teams. |
| Edit                     | Update or delete the contact point                                                                                                            |
| View                     | View and export contact point as well as select it on the Alert rule edit page                                                                |

### Assign contact point permissions

To manage contact point permissions, complete the following steps:

1. In the left-side menu, click **Notification configuration**, then select the **Contact points** tab.
1. Hover your mouse cursor over a contact point and click **More**.
1. Click **Manage permissions** from the actions menu.
1. Update or add permissions as required.

# Performance considerations and limitations

Grafana Alerting supports multi-dimensional alerting, where one alert rule can generate many alerts. For example, you can configure an alert rule to fire an alert every time the CPU of individual virtual machines max out. This topic discusses performance considerations resulting from multi-dimensional alerting.

Evaluating alerting rules consumes RAM and CPU to compute the output of an alerting query, and network resources to send alert notifications and write the results to the Grafana SQL database. The configuration of individual alert rules affects the resource consumption and, therefore, the maximum number of rules a given configuration can support.

The following section provides a list of alerting performance considerations.

- Frequency of rule evaluation consideration. The "Evaluate Every" property of an alert rule controls the frequency of rule evaluation. It is recommended to use the lowest acceptable evaluation frequency to support more concurrent rules.
- Cardinality of the rule's result set. For example, suppose you are monitoring API response errors for every API path, on every virtual machine in your fleet. This set has a cardinality of _n_ number of paths multiplied by _v_ number of VMs. You can reduce the cardinality of a result set - perhaps by monitoring errors-per-VM instead of for each path per VM.
- Complexity of the alerting query consideration. Queries that data sources can process and respond to quickly consume fewer resources. Although this consideration is less important than the other considerations listed above, if you have reduced those as much as possible, looking at individual query performance could make a difference.

Each evaluation of an alert rule generates a set of alert instances; one for each member of the result set. The state of all the instances is written to the `alert_instance` table in the Grafana SQL database. This number of write-heavy operations can cause issues when using SQLite.

Grafana Alerting exposes a metric, `grafana_alerting_rule_evaluations_total` that counts the number of alert rule evaluations. To get a feel for the influence of rule evaluations on your Grafana instance, you can observe the rate of evaluations and compare it with resource consumption. In a Prometheus-compatible database, you can use the query `rate(grafana_alerting_rule_evaluations_total[5m])` to compute the rate over 5 minute windows of time. It's important to remember that this isn't the full picture of rule evaluation. For example, the load is unevenly distributed if you have some rules that evaluate every 10 seconds, and others every 30 minutes.

These factors all affect the load on the Grafana instance, but you should also be aware of the performance impact that evaluating these rules has on your data sources. Alerting queries are often the vast majority of queries handled by monitoring databases, so the same load factors that affect the Grafana instance affect them as well.

## Limited rule sources support

Grafana Alerting can retrieve alerting and recording rules **stored** in most available Prometheus, Loki, Mimir, and Alertmanager compatible data sources.

It does not support reading or writing alerting rules from any other data sources but the ones previously mentioned at this time.

## Prometheus version support

The latest two minor versions of both Prometheus and Alertmanager are supported. We cannot guarantee that older versions work.

As an example, if the current Prometheus version is `2.31.1`, >= `2.29.0` is supported.

## The Grafana Alertmanager can only receive Grafana managed alerts

Grafana cannot be used to receive external alerts. You can only send alerts to the Grafana Alertmanager using Grafana managed alerts.

You have the option to send Grafana-managed alerts to an external Alertmanager, you can find this option in the Admin tab on the Alerting page.

For more information, refer to [this GitHub issue](https://github.com/grafana/grafana/issues/73447).

## High load on database caused by a high number of alert instances

If you have a high number of alert rules or alert instances, the load on the database can get very high.

Grafana performs one SQL update per alert rule after each evaluation. This update groups all alert instances belonging to the rule together and compresses them into a single protobuf-based row, which keeps database overhead low even for alert rules with many instances.

{{< admonition type="warning" >}}
Earlier versions of Grafana wrote alert instance state to the database uncompressed, one row per instance, and let you opt in to compressed storage with the `alertingSaveStateCompressed` feature toggle. Grafana 13.2 removes that feature toggle and the uncompressed storage path entirely.

If you're upgrading from a Grafana version where you had explicitly disabled `alertingSaveStateCompressed`, enable it and let Grafana run for at least one evaluation cycle _before_ you upgrade to Grafana 13.2 or later. Otherwise, any alert state still held in the old uncompressed format isn't read after the upgrade. This doesn't cause data loss beyond the alert state itself: Grafana re-evaluates every rule and rebuilds instance state from scratch, but you might see a brief, one-time gap in alert history and current state immediately after the upgrade.
{{< /admonition >}}

### Save state periodically

You can also reduce database load by writing states periodically instead of after every evaluation.

Enable the `alertingSaveStatePeriodic` feature toggle to save alert states at the interval specified by `state_periodic_save_interval` instead of after every rule evaluation. Grafana groups all alert instances by rule UID and compresses them together for efficient storage.

By default, Grafana saves the states every 5 minutes and on each shutdown. Note that `state_periodic_save_batch_size` and `state_periodic_save_jitter_enabled` don't apply here, since Grafana groups instances by rule UID rather than by batch.

```ini
[unified_alerting]
state_periodic_save_interval = 1m
```

If Grafana crashes or is force killed, then the database can be up to `state_periodic_save_interval` seconds out of date.
When Grafana restarts, the UI might show incorrect state for some alerts until the alerts are re-evaluated.
In some cases, alerts that were firing before the crash might fire again.
If this happens, Grafana might send duplicate notifications for firing alerts.

## Limit on the number of results per alert rule

Each alert rule evaluation generates one alert instance per series in the rule's query result set. Rules with high-cardinality result sets consume more CPU, memory, network, and database resources, and can produce a large number of alert instances (refer to [High load on database caused by a high number of alert instances](#high-load-on-database-caused-by-a-high-number-of-alert-instances)).

To prevent this, Grafana can limit the number of query evaluation results a single alert rule produces in one evaluation. When a rule's query returns more results than the limit, the evaluation fails with an error similar to the following:

```
query evaluation returned too many results: 12345 (limit: 10000)
```

The alert rule enters the [Error state](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/) and produces no alert instances for that evaluation until you reduce its result set below the limit.

In self-managed Grafana, set this limit using the [`alerting_rule_evaluation_results`](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana#alerting_rule_evaluation_results) option in the `[quota]` section. The default is `-1` (unlimited). In Grafana Cloud, Grafana Labs manages this limit.

To resolve the error, reduce the cardinality of the rule's result set:

- Aggregate the query so it returns fewer series. For example, sum or average by fewer labels.
- Add label filters so the query returns only the series you need to alert on.
- Split a single high-cardinality rule into several rules with narrower queries.

## Alert rule migrations for Grafana 11.6.0

When you upgrade to Grafana 11.6.0, a migration is performed on the `alert_rule_versions` table. If you experience a 11.6.0 upgrade that causes a migration failure, then your `alert_rule_versions` table has too many rows. To fix this, you need to truncated the `alert_rule_versions` table for the migration to complete.

# Export alerting resources

Export your alerting resources, such as alert rules, contact points, and notification policies for provisioning, automatically importing single folders and single groups.

There are distinct methods to export your alerting resources:

- [Grafana UI](#export-from-the-grafana-ui) exports in Terraform format and YAML or JSON formats for file provisioning.
- [HTTP Alerting API](#http-alerting-api) exports in JSON API format used by the HTTP Alerting API.
- [HTTP Alerting API - Export endpoints](#export-api-endpoints) exports in YAML or JSON formats for file provisioning.

{{< admonition type="note" >}}
Alerting resources imported through [file provisioning](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/file-provisioning) cannot be edited in the Grafana UI. This prevents changes made in the UI from being overridden by file provisioning during Grafana restarts.

If you need to modify provisioned alerting resources in Grafana, refer to [edit HTTP API alerting resources in the Grafana UI](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning#edit-resources-in-the-grafana-ui) or to [edit Terraform alerting resources in the Grafana UI](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/terraform-provisioning#enable-editing-resources-in-the-grafana-ui).
{{< /admonition >}}

## Export from the Grafana UI

The export options listed below enable you to download resources in YAML, JSON, or Terraform format, facilitating their provisioning through [configuration files](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/file-provisioning/) or [Terraform](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/terraform-provisioning/).

### Export alert rules

To export alert rules from the Grafana UI, complete the following steps.

1. Click **Alerts & IRM** -> **Alert rules**.
1. To export all Grafana-managed rules, click **Export rules**.
1. To export a folder, change the **View as** to **List**.
1. Select the folder you want to export and click the **Export rules folder** icon.
1. To export a group, change the **View as** to **Grouped**.
1. Find the group you want to export and click the **Export rule group** icon.
1. Choose the format to export in.

   The exported alert rule data appears in different formats - YAML, JSON, Terraform.

1. Click **Copy Code** or **Download**.

### Modify alert rule and export rule group without saving changes

{{< admonition type="note" >}} This feature is for Grafana-managed alert rules only. It is available to Admin, Viewer, and Editor roles. {{< /admonition >}}

Use the **Modify export** mode to edit and export an alert rule without updating it. The exported data includes all alert rules within the same alert group.

To export a modified alert rule without saving the modifications, complete the following steps from the Grafana UI.

1. Click **Alerts & IRM** -> **Alert rules**.
1. Locate the alert rule you want to edit and click **More** -> **Modify Export** to open the Alert Rule form.
1. From the Alert Rule form, edit the fields you want to change. Changes made are not applied to the alert rule.
1. Click **Export**.
1. Choose the format to export in.

   The exported alert rule group appears in different formats - YAML, JSON, Terraform.

1. Click **Copy Code** or **Download**.

### Export a new alert rule definition without saving changes

{{< admonition type="note" >}} You can only export in Terraform (HCL) format. {{< /admonition >}}

Add a new alert rule definition to an existing provisioned rule group rather than creating the code manually. You can then copy it to your Terraform pipeline, and quickly deploy and manage alert rules as part of your infrastructure as code.

An alert rule definition differs from an alert rule in the sense that you define your alert rule, but you do not save it. It only becomes a saved alert rule once you provision it using Terraform.

To export your alert rule definition into Terraform (HCL) format, complete the following steps.

1. Click **Alerts & IRM** -> **Alert rules**.
2. Click **Export rule definition**.
3. Fill out the alert rule details.
4. Choose a provisioned folder and group to add your alert rule definition to.
5. Click **Export**.
6. Copy and paste the code into your Terraform pipeline to create your new alert rule.

### Export contact points

To export contact points from the Grafana UI, complete the following steps.

1. Click **Alerts & IRM** -> **Contact points**.
1. Find the contact point you want to export and click **More** -> **Export**.
1. Choose the format to export in.

   The exported contact point appears in different formats - YAML, JSON, Terraform.

1. Click **Copy Code** or **Download**.

### Export notification template groups

Grafana currently doesn't offer an Export UI or [Export endpoint](#export-api-endpoints) for notification template groups, unlike other Alerting resources presented in this documentation.

However, you can export it by manually copying the content and name of the notification template group from the Grafana UI.

1. Click **Alerts & IRM** -> **Contact points** -> **Notification templates** tab.
1. Find the notification template group you want to export.
1. Copy the content and name.
1. Adjust it for the [file provisioning format](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/file-provisioning/#import-templates) or [Terraform resource](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/terraform-provisioning/#import-contact-points-and-templates).

### Export the notification policy tree

All notification policies are provisioned through a single resource: the root of the notification policy tree.

{{< docs/shared lookup="alerts/warning-provisioning-tree.md" source="grafana" version="<GRAFANA_VERSION>" >}}

To export the notification policy tree from the Grafana UI, complete the following steps.

1. Click **Alerts & IRM** -> **Notification policies**.
2. In the **Default notification policy** section, click **...** -> **Export**.
3. Choose the format to export in.

   The exported notification policy tree appears in different formats - YAML, JSON, Terraform.

4. Click **Copy Code** or **Download**.

You can also export individual [named routing trees](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy/#manage-multiple-notification-policy-trees).

1. Click **Alerts & IRM** -> **Notification policies**.
1. Find the routing tree you want to export in the list.
1. Click the **More actions** menu item and select **Export**.
1. Choose the export format and click **Copy Code** or **Download**.

### Export mute timings

To export mute timings from the Grafana UI, complete the following steps.

1. Click **Alerts & IRM** -> **Notification policies**, and then the **Mute timings** tab.
1. Find the mute timing you want to export and click **Export**.
1. Choose the format to export in.

   The exported contact point appears in different formats - YAML, JSON, Terraform.

1. Click **Copy Code** or **Download**.

## HTTP Alerting API

You can use the [Alerting HTTP API](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/) to return existing alerting resources in JSON and import them to another Grafana instance using the same endpoint.

| Resource                                                                                                                                                         | URI                                 |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| [Alert rules](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#alert-rules)                                   | /api/v1/provisioning/alert-rules    |
| [Contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#contact-points)                             | /api/v1/provisioning/contact-points |
| [Notification policy tree](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#notification-policies)            | /api/v1/provisioning/policies       |
| [Notification template groups](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#notification-template-groups) | /api/v1/provisioning/templates      |
| [Mute timings](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#mute-timings)                                 | /api/v1/provisioning/mute-timings   |

However, note the standard endpoints return a JSON format that is not compatible for provisioning through configuration files or Terraform, except the `/export` endpoints listed below.

### Export API endpoints

The **Alerting HTTP API** provides specific endpoints for exporting alerting resources in YAML or JSON, facilitating [provisioning via configuration files](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/file-provisioning/), or Terraform (HCL).

| Resource                 | Method / URI                                                         | Summary                                                                                                                                                                                                                                                                                                    |
| ------------------------ | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Alert rules              | GET /api/v1/provisioning/alert-rules/export                          | [Export all alert rules in provisioning file format](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#span-idroute-get-alert-rules-exportspan-export-all-alert-rules-in-provisioning-file-format-_routegetalertrulesexport_).                           |
| Alert rules              | GET /api/v1/provisioning/folder/:folderUid/rule-groups/:group/export | [Export an alert rule group in provisioning file format](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#span-idroute-get-alert-rule-group-exportspan-export-an-alert-rule-group-in-provisioning-file-format-_routegetalertrulegroupexport_).          |
| Alert rules              | GET /api/v1/provisioning/alert-rules/:uid/export                     | [Export an alert rule in provisioning file format](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#span-idroute-get-alert-rule-exportspan-export-an-alert-rule-in-provisioning-file-format-_routegetalertruleexport_).                                 |
| Contact points           | GET /api/v1/provisioning/contact-points/export                       | [Export all contact points in provisioning file format](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#span-idroute-get-contactpoints-exportspan-export-all-contact-points-in-provisioning-file-format-_routegetcontactpointsexport_).                |
| Notification policy tree | GET /api/v1/provisioning/policies/export                             | [Export the notification policy tree in provisioning file format](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#span-idroute-get-policy-tree-exportspan-export-the-notification-policy-tree-in-provisioning-file-format-_routegetpolicytreeexport_). |
| Mute timings             | GET /api/v1/provisioning/mute-timings/export                         | [Export all mute timings in provisioning file format](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#span-idroute-get-mute-timings-exportspan-export-all-mute-timings-in-provisioning-file-format-_routegetmutetimingsexport_).                       |
| Mute timings             | GET /api/v1/provisioning/mute-timings/:name/export                   | [Export a mute timing in provisioning file format](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/#span-idroute-get-mute-timing-exportspan-export-a-mute-timing-in-provisioning-file-format-_routegetmutetimingexport_).                               |

These endpoints accept a `download` parameter to download a file containing the exported resources.

<!-- prettier-ignore-start -->


<!-- prettier-ignore-end -->

# Use configuration files to provision alerting resources

Manage your alerting resources using configuration files that can be version controlled. When Grafana starts, it provisions the resources defined in your configuration files. [Provisioning](/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/) can create, update, or delete existing resources in your Grafana instance.

This guide outlines the steps and references to provision alerting resources using YAML files. For a practical demo, you can clone and try [this example using Grafana OSS and Docker Compose](https://github.com/grafana/provisioning-alerting-examples/tree/main/config-files).

{{< admonition type="note" >}}

- [Provisioning Grafana](/docs/grafana/<GRAFANA_VERSION>/administration/provisioning) with configuration files is not available in Grafana Cloud.

- You cannot edit provisioned resources from files in Grafana. You can only change the resource properties by changing the provisioning file and restarting Grafana or carrying out a hot reload. This prevents changes from being made to the resource that would be overwritten if a file is provisioned again or a hot reload is carried out.

- Provisioning using configuration files takes place during the initial set up of your Grafana system, but you can re-run it at any time using the [Grafana Admin API](/docs/grafana/<GRAFANA_VERSION>/developers/http_api/admin#reload-provisioning-configurations).

- Importing an existing alerting resource results in a conflict. First, when present, remove the resources you plan to import.
  {{< /admonition >}}

Details on how to set up the files and which fields are required for each object are listed below depending on which resource you are provisioning.

## Import alert rules

Create or delete alert rules using provisioning files in your Grafana instance(s).

1. Find the alert rule group in Grafana.
1. [Export](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources/#export-alert-rules) and download a provisioning file for your alert rules.
1. Copy the contents into a YAML or JSON configuration file and add it to the `provisioning/alerting` directory of the Grafana instance you want to import the alerting resources to.

   Example configuration files can be found below.

1. Restart your Grafana instance (or reload the provisioned files using the Admin API).

Here is an example of a configuration file for creating alert rules.

```yaml
# config file version
apiVersion: 1

# List of rule groups to import or update
groups:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the rule group
    name: my_rule_group
    # <string, required> name of the folder the rule group will be stored in
    folder: my_first_folder
    # <duration, required> interval that the rule group should be evaluated at
    interval: 60s
    # <list, required> list of rules that are part of the rule group
    rules:
      # <string, required> unique identifier for the rule. Should not exceed 40 symbols. Only letters, numbers, - (hyphen), and _ (underscore) allowed.
      - uid: my_id_1
        # <string, required> title of the rule that will be displayed in the UI
        title: my_first_rule
        # <string, required> which query should be used for the condition
        condition: A
        # <list, required> list of query objects that should be executed on each
        #                  evaluation - should be obtained through the API
        data:
          - refId: A
            datasourceUid: '__expr__'
            model:
              conditions:
                - evaluator:
                    params:
                      - 3
                    type: gt
                  operator:
                    type: and
                  query:
                    params:
                      - A
                  reducer:
                    type: last
                  type: query
              datasource:
                type: __expr__
                uid: '__expr__'
              expression: 1==0
              intervalMs: 1000
              maxDataPoints: 43200
              refId: A
              type: math
        # <string> UID of a dashboard that the alert rule should be linked to
        dashboardUid: my_dashboard
        # <int> ID of the panel that the alert rule should be linked to
        panelId: 123
        # <string> the state the alert rule will have when no data is returned
        #          possible values: "NoData", "Alerting", "OK", default = NoData
        noDataState: Alerting
        # <string> the state the alert rule will have when the query execution
        #          failed - possible values: "Error", "Alerting", "OK"
        #          default = Alerting
        execErrState: Alerting
        # <duration, required> for how long should the alert fire before alerting
        for: 60s
        # <map<string, string>> a map of strings to pass around any data
        annotations:
          some_key: some_value
        # <map<string, string> a map of strings that can be used to filter and
        #                      route alerts
        labels:
          team: sre_team_1
```

Here is an example of a configuration file for deleting alert rules.

```yaml
# config file version
apiVersion: 1

# List of alert rule UIDs that should be deleted
deleteRules:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> unique identifier for the rule
    uid: my_id_1
```

## Import contact points

Create or delete contact points using provisioning files in your Grafana instance(s).

1. Find the contact point in Grafana.
1. [Export](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources/#export-contact-points) and download a provisioning file for your contact point.
1. Copy the contents into a YAML or JSON configuration file and add it to the `provisioning/alerting` directory of the Grafana instance you want to import the alerting resources to.

   Example configuration files can be found below.

1. Restart your Grafana instance (or reload the provisioned files using the Admin API).

Here is an example of a configuration file for creating contact points.

```yaml
# config file version
apiVersion: 1

# List of contact points to import or update
contactPoints:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the contact point
    name: cp_1
    receivers:
      # <string, required> unique identifier for the receiver. Should not exceed 40 symbols. Only letters, numbers, - (hyphen), and _ (underscore) allowed.
      - uid: first_uid
        # <string, required> type of the receiver
        type: prometheus-alertmanager
        # <bool, optional> Disable the additional [Incident Resolved] follow-up alert, default = false
        disableResolveMessage: false
        # <object, required> settings for the specific receiver type
        settings:
          url: http://test:9000
```

Here is an example of a configuration file for deleting contact points.

```yaml
# config file version
apiVersion: 1

# List of receivers that should be deleted
deleteContactPoints:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> unique identifier for the receiver
    uid: first_uid
```

### Settings

Here are some examples of settings you can use for the different
contact point integrations.

{{< collapse title="Alertmanager" >}}

#### Alertmanager

```yaml
type: prometheus-alertmanager
settings:
  # <string, required>
  url: http://localhost:9093
  # <string>
  basicAuthUser: abc
  # <string>
  basicAuthPassword: abc123
```

{{< /collapse >}}

{{< collapse title="DingDing" >}}

#### DingDing

```yaml
type: dingding
settings:
  # <string, required>
  url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx
  # <string> options: link, actionCard
  msgType: link
  # <string>
  message: |
    {{ template "default.message" . }}
```

{{< /collapse >}}

{{< collapse title="Discord" >}}

#### Discord

```yaml
type: discord
settings:
  # <string, required>
  url: https://discord/webhook
  # <string>
  avatar_url: https://my_avatar
  # <bool>
  use_discord_username: false
  # <string>
  message: |
    {{ template "default.message" . }}
```

{{< /collapse >}}

{{< collapse title="E-Mail" >}}

#### E-Mail

```yaml
type: email
settings:
  # <string, required>
  addresses: me@example.com;you@example.com
  # <bool>
  singleEmail: false
  # <string>
  message: my optional message to include
  # <string>
  subject: |
    {{ template "default.title" . }}
```

{{< /collapse >}}

{{< collapse title="Google Chat" >}}

#### Google Chat

```yaml
type: googlechat
settings:
  # <string, required>
  url: https://google/webhook
  # <string>
  message: |
    {{ template "default.message" . }}
```

{{< /collapse >}}

{{< collapse title="Kafka" >}}

#### Kafka

```yaml
type: kafka
settings:
  # <string, required>
  kafkaRestProxy: http://localhost:8082
  # <string, required>
  kafkaTopic: topic1
```

{{< /collapse >}}

{{< collapse title="LINE" >}}

#### LINE

```yaml
type: line
settings:
  # <string, required>
  token: xxx
```

{{< /collapse >}}

{{< collapse title="MQTT" >}}

#### MQTT

```yaml
type: mqtt
settings:
  # <string, required>
  brokerUrl: tcp://127.0.0.1:1883
  # <string>
  clientId: grafana
  # <string, required>
  topic: grafana/alerts
  # <string>
  messageFormat: json
  # <string>
  username: grafana
  # <string>
  password: password1
  # <string>
  qos: 0
  # <bool>
  retain: false
  # <map>
  tlsConfig:
    # <bool>
    insecureSkipVerify: false
    # <string>
    clientCertificate: certificate in PEM format
    # <string>
    clientKey: key in PEM format
    # <string>
    caCertificate: CA certificate in PEM format
```

{{< /collapse >}}

{{< collapse title="Microsoft Teams" >}}

#### Microsoft Teams

```yaml
type: teams
settings:
  # <string, required>
  url: https://ms_teams_url
  # <string>
  title: |
    {{ template "default.title" . }}
  # <string>
  sectiontitle: ''
  # <string>
  message: |
    {{ template "default.message" . }}
```

{{< /collapse >}}

{{< collapse title="OpsGenie" >}}

#### OpsGenie

```yaml
type: opsgenie
settings:
  # <string, required>
  apiKey: xxx
  # <string, required>
  apiUrl: https://api.opsgenie.com/v2/alerts
  # <string>
  message: |
    {{ template "default.title" . }}
  # <string>
  description: some descriptive description
  # <bool>
  autoClose: false
  # <bool>
  overridePriority: false
  # <string> options: tags, details, both
  sendTagsAs: both
```

{{< /collapse >}}

{{< collapse title="PagerDuty" >}}

#### PagerDuty

```yaml
type: pagerduty
settings:
  # <string, required> the 32-character Events API key https://support.pagerduty.com/docs/api-access-keys#events-api-keys
  integrationKey: XXX
  # <string> options: critical, error, warning, info
  severity: critical
  # <string>
  class: ping failure
  # <string>
  component: Grafana
  # <string>
  group: app-stack
  # <string>
  summary: |
    {{ template "default.message" . }}
```

{{< /collapse >}}

{{< collapse title="Pushover" >}}

#### Pushover

```yaml
type: pushover
settings:
  # <string, required>
  apiToken: XXX
  # <string, required>
  userKey: user1,user2
  # <string>
  device: device1,device2
  # <string> options (high to low): 2,1,0,-1,-2
  priority: '2'
  # <string>
  retry: '30'
  # <string>
  expire: '120'
  # <string> the number of seconds before a message expires and is deleted automatically. Examples: 10s, 5m30s, 8h.
  ttl:
  # <string>
  sound: siren
  # <string>
  okSound: magic
  # <string>
  message: |
    {{ template "default.message" . }}
```

{{< /collapse >}}

{{< collapse title="Slack" >}}

#### Slack

```yaml
type: slack
settings:
  # <string, required>
  recipient: alerting-dev
  # <string, required>
  token: xxx
  # <string>
  username: grafana_bot
  # <string>
  icon_emoji: heart
  # <string>
  icon_url: https://icon_url
  # <string>
  mentionUsers: user_1,user_2
  # <string>
  mentionGroups: group_1,group_2
  # <string> options: here, channel
  mentionChannel: here
  # <string> Optionally provide a Slack incoming webhook URL for sending messages, in this case the token isn't necessary
  url: https://some_webhook_url
  # <string>
  endpointUrl: https://custom_url/api/chat.postMessage
  # <string>
  color: {{ if eq .Status "firing" }}#D63232{{ else }}#36a64f{{ end }}
  # <string>
  title: |
    {{ template "slack.default.title" . }}
  text: |
    {{ template "slack.default.text" . }}
```

{{< /collapse >}}

{{< collapse title="Sensu Go" >}}

#### Sensu Go

```yaml
type: sensugo
settings:
  # <string, required>
  url: http://sensu-api.local:8080
  # <string, required>
  apikey: xxx
  # <string>
  entity: default
  # <string>
  check: default
  # <string>
  handler: some_handler
  # <string>
  namespace: default
  # <string>
  message: |
    {{ template "default.message" . }}
```

{{< /collapse >}}

{{< collapse title="Telegram" >}}

#### Telegram

```yaml
type: telegram
settings:
  # <string, required>
  bottoken: xxx
  # <string, required>
  chatid: some_chat_id
  # <string>
  message: |
    {{ template "default.message" . }}
```

{{< /collapse >}}

{{< collapse title="Threema Gateway" >}}

#### Threema Gateway

```yaml
type: threema
settings:
  # <string, required>
  api_secret: xxx
  # <string, required>
  gateway_id: A5K94S9
  # <string, required>
  recipient_id: A9R4KL4S
```

{{< /collapse >}}

{{< collapse title="VictorOps" >}}

#### VictorOps

```yaml
type: victorops
settings:
  # <string, required>
  url: XXX
  # <string> options: CRITICAL, WARNING
  messageType: CRITICAL
```

{{< /collapse >}}

{{< collapse title="Webhook" >}}

#### Webhook

```yaml
type: webhook
settings:
  # <string, required>
  url: https://endpoint_url
  # <string> options: POST, PUT
  httpMethod: POST
  # <string>
  username: abc
  # <string>
  password: abc123
  # <string>
  authorization_scheme: Bearer
  # <string>
  authorization_credentials: abc123
  # <string>
  maxAlerts: '10'
  # <map>
  tlsConfig:
    # <bool>
    insecureSkipVerify: false
    # <string>
    clientCertificate: certificate in PEM format
    # <string>
    clientKey: key in PEM format
    # <string>
    caCertificate: CA certificate in PEM format
  hmacConfig:
    #<string>
    secret: secret-key
    #<string>
    header: X-Grafana-Alerting-Signature
    #<string>
    timestampHeader: X-Grafana-Alerting-Signature-Timestamp
```

{{< /collapse >}}

{{< collapse title="WeCom" >}}

#### WeCom

```yaml
type: wecom
settings:
  # <string, required>
  url: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxxx
  # <string>
  message: |
    {{ template "default.message" . }}
  # <string>
  title: |
    {{ template "default.title" . }}
```

{{< /collapse >}}

## Import notification template groups

Create or delete notification template groups using provisioning files in your Grafana instance(s).

1. Find the notification template group in Grafana.
1. [Export](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources/#export-notification-template-groups) a template group by copying the template content and name.
1. Copy the contents into a YAML or JSON configuration file and add it to the `provisioning/alerting` directory of the Grafana instance you want to import the alerting resources to.

   Example configuration files can be found below.

1. Restart your Grafana instance (or reload the provisioned files using the Admin API).

Here is an example of a configuration file for creating notification template groups.

```yaml
# config file version
apiVersion: 1

# List of templates to import or update
templates:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the template group, must be unique
    name: my_first_template
    # <string, required> content of the template group
    template: |
      {{ define "my_first_template" }}
        Custom notification message
      {{ end }}
```

Here is an example of a configuration file for deleting notification template groups.

```yaml
# config file version
apiVersion: 1

# List of alert rule UIDs that should be deleted
deleteTemplates:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the template group, must be unique
    name: my_first_template
```

## Import notification policies

Create or reset the notification policy tree using provisioning files in your Grafana instance(s).

In Grafana, the entire notification policy tree is considered a single, large resource. Add new specific policies as sub-policies under the root policy. Since specific policies may depend on each other, you cannot provision subsets of the policy tree; the entire tree must be defined in a single place.

{{< docs/shared lookup="alerts/warning-provisioning-tree.md" source="grafana" version="<GRAFANA_VERSION>" >}}

1. Find the notification policy tree in Grafana.
1. [Export](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources/#export-the-notification-policy-tree) and download a provisioning file for your notification policy tree.
1. Copy the contents into a YAML or JSON configuration file and add it to the `provisioning/alerting` directory of the Grafana instance you want to import the alerting resources to.

   Example configuration files can be found below.

1. Restart your Grafana instance (or reload the provisioned files using the Admin API).

Here is an example of a configuration file for creating notification policies.

```yaml
# config file version
apiVersion: 1

# List of notification policies
policies:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string> name of the contact point that should be used for this route
    receiver: grafana-default
    # <list> The labels by which incoming alerts are grouped together. For example,
    #        multiple alerts coming in for cluster=A and alertname=LatencyHigh would
    #        be batched into a single group.
    #
    #        To aggregate by all possible labels use the special value '...' as
    #        the sole label name, for example:
    #        group_by: ['...']
    #        This effectively disables aggregation entirely, passing through all
    #        alerts as-is. This is unlikely to be what you want, unless you have
    #        a very low alert volume or your upstream notification system performs
    #        its own grouping.
    group_by: ['...']
    # <list> a list of prometheus-like matchers that an alert rule has to fulfill to match the node (allowed chars
    #        [a-zA-Z_:])
    matchers:
      - alertname = Watchdog
      - service_id_X = serviceX
      - severity =~ "warning|critical"
    # <list> a list of grafana-like matchers that an alert rule has to fulfill to match the node
    object_matchers:
      - ['alertname', '=', 'CPUUsage']
      - ['service_id-X', '=', 'serviceX']
      - ['severity', '=~', 'warning|critical']
    # <list> Times when the route should be muted. These must match the name of a
    #        mute time interval.
    #        Additionally, the root node cannot have any mute times.
    #        When a route is muted it will not send any notifications, but
    #        otherwise acts normally (including ending the route-matching process
    #        if the `continue` option is not set)
    mute_time_intervals:
      - abc
    # <duration> How long to initially wait to send a notification for a group
    #            of alerts. Allows to collect more initial alerts for the same group.
    #            (Usually ~0s to few minutes), default = 30s
    group_wait: 30s
    # <duration> How long to wait before sending a notification about new alerts that
    #            are added to a group of alerts for which an initial notification has
    #            already been sent. (Usually ~5m or more), default = 5m
    group_interval: 5m
    # <duration>  How long to wait before sending a notification again if it has already
    #             been sent successfully for an alert. (Usually ~3h or more), default = 4h
    repeat_interval: 4h
    # <list> Zero or more child policies. The schema is the same as the root policy.
    # routes:
    #   # Another recursively nested policy...
    #   - receiver: another-receiver
    #     matchers:
    #       - ...
    #     ...
```

Here is an example of a configuration file for resetting the policy tree back to its default value:

```yaml
# config file version
apiVersion: 1

# List of orgIds that should be reset to the default policy
resetPolicies:
  - 1
```

## Import mute timings

Create or delete mute timings via provisioning files using provisioning files in your Grafana instance(s).

1. Find the mute timing in Grafana.
1. [Export](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources/#export-mute-timings) and download a provisioning file for your mute timing.
1. Copy the contents into a YAML or JSON configuration file and add it to the `provisioning/alerting` directory of the Grafana instance you want to import the alerting resources to.

   Example configuration files can be found below.

1. Restart your Grafana instance (or reload the provisioned files using the Admin API).

Here is an example of a configuration file for creating mute timings.

```yaml
# config file version
apiVersion: 1

# List of mute time intervals to import or update
muteTimes:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the mute time interval, must be unique
    name: mti_1
    # <list> time intervals that should trigger the muting
    #        refer to https://prometheus.io/docs/alerting/latest/configuration/#time_interval-0
    time_intervals:
      - times:
          - start_time: '06:00'
            end_time: '23:59'
        location: 'UTC'
        weekdays: ['monday:wednesday', 'saturday', 'sunday']
        months: ['1:3', 'may:august', 'december']
        years: ['2020:2022', '2030']
        days_of_month: ['1:5', '-3:-1']
```

Here is an example of a configuration file for deleting mute timings.

```yaml
# config file version
apiVersion: 1

# List of mute time intervals that should be deleted
deleteMuteTimes:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the mute time interval, must be unique
    name: mti_1
```

## Template variable interpolation

Provisioning interpolates environment variables using the `$variable` syntax.

```yaml
contactPoints:
  - orgId: 1
    name: My Contact Email Point
    receivers:
      - uid: 1
        type: email
        settings:
          addresses: $EMAIL
```

In this example, provisioning replaces `$EMAIL` with the value of the `EMAIL` environment variable or an empty string if it is not present. For more information, refer to [Using environment variables in the Provision documentation](/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#using-environment-variables).

In alerting resources, most properties support template variable interpolation, with a few exceptions:

- Alert rule annotations: `groups[].rules[].annotations`
- Alert rule time range: `groups[].rules[].relativeTimeRange`
- Alert rule query model: `groups[].rules[].data.model`
- Mute timings name: `muteTimes[].name`
- Mute timings time intervals: `muteTimes[].time_intervals[]`
- Notification template group name: `templates[].name`
- Notification template group content: `templates[].template`

Note for properties that support interpolation, you may unexpectedly substitute template variables when not intended. To avoid this, you can escape the `$variable` with `$$variable`.

For example, when provisioning a `subject` property in a `contactPoints.receivers.settings` object that is meant to use the `$labels` variable.

1. `subject: '{{ $labels }}'` will interpolate, incorrectly defining the subject as `subject: '{{ }}'`.
1. `subject: '{{ $$labels }}'` will not interpolate, correctly defining the subject as `subject: '{{ $labels }}'`.

## More examples

For more examples on the concept of this guide:

- Try provisioning alerting resources in Grafana OSS with YAML files through a demo project using [Docker Compose](https://github.com/grafana/provisioning-alerting-examples/tree/main/config-files) or [Kubernetes deployments](https://github.com/grafana/provisioning-alerting-examples/tree/main/kubernetes).
- Review the distinct options about how Grafana provisions resources in the [Provision Grafana documentation](/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/).
- For Helm support, review the examples provisioning alerting resources in the [Grafana Helm Chart documentation](https://github.com/grafana/helm-charts/blob/main/charts/grafana/README.md).

# Use Terraform to provision alerting resources

Use Terraform’s Grafana Provider to manage your alerting resources and provision them into your Grafana system. Terraform provider support for Grafana Alerting makes it easy to create, manage, and maintain your entire Grafana Alerting stack as code.

This guide outlines the steps and references to provision alerting resources with Terraform. For a practical demo, you can clone and try this [example using Grafana OSS and Docker Compose](https://github.com/grafana/provisioning-alerting-examples/tree/main/terraform).

To create and manage your alerting resources using Terraform, you have to complete the following tasks.

1. Create an API key to configure the Terraform provider.
1. Create your alerting resources in Terraform format by
   - [exporting configured alerting resources](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources/)
   - or writing the [Terraform Alerting schemas](https://registry.terraform.io/providers/grafana/grafana/latest/docs).
     > By default, you cannot edit provisioned resources. Enable [`disable_provenance` in the Terraform resource](#enable-editing-resources-in-the-grafana-ui) to allow changes in the Grafana UI.
1. Run `terraform apply` to provision your alerting resources.

Before you begin, you should have available a Grafana instance and [Terraform installed](https://www.terraform.io/downloads) on your machine.

## Create an API key and configure the Terraform provider

You can create a [service account token](/docs/grafana/<GRAFANA_VERSION>/administration/service-accounts/) to authenticate Terraform with Grafana. To create an API key for provisioning alerting resources, complete the following steps.

1. Create a new service account.
1. Assign the role or permission to access the [Alerting provisioning API](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/http-api-provisioning/).
1. Create a new service account token.
1. Name and save the token for use in Terraform.

You can now move to the working directory for your Terraform configurations, and create a file named `main.tf` like:

```main.tf
terraform {
    required_providers {
        grafana = {
            source = "grafana/grafana"
            version = ">= 4.47.0"
        }
    }
}

provider "grafana" {
    url = <grafana-url>
    auth = <api-key>
}
```

Replace the following values:

- `<grafana-url>` with the URL of the Grafana instance.
- `<api-key>` with the API token previously created.

This Terraform configuration installs the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) and authenticates against your Grafana instance using an API token. For other authentication alternatives including basic authentication, refer to the [`auth` option documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs#authentication).

For Grafana Cloud, refer to the [instructions to manage a Grafana Cloud stack with Terraform](/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/terraform-cloud-stack/). For role-based access control, refer to [Provisioning RBAC with Terraform](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/rbac-terraform-provisioning/) and the [alerting provisioning roles (`fixed:alerting.provisioning.*`)](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/rbac-fixed-basic-role-definitions/).

## Create Terraform configurations for alerting resources

[Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) enables you to manage the following alerting resources.

| Alerting resource                                                                                                        | Terraform resource                                                                                                               |
| ------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- |
| [Alert rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/)                                                  | [grafana_rule_group](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/rule_group)                   |
| [Contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/)                | [grafana_contact_point](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/contact_point)             |
| [Notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/)       | [grafana_message_template](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/message_template)       |
| [Notification policy tree](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy/) | [grafana_notification_policy](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/notification_policy) |
| [Mute timings](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/)                           | [grafana_mute_timing](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/mute_timing)                 |

In this section, we'll create Terraform configurations for each alerting resource and demonstrate how to link them together.

### Add alert rules

[Alert rules](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/) enable you to receive alerts by querying any backend Grafana data sources.

1. First, create a data source to query and a folder to store your rules in.

   In this example, the [TestData](/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/) data source is used.

   ```terraform
   resource "grafana_data_source" "<terraform_data_source_name>" {
       name = "TestData"
       type = "testdata"
   }

   resource "grafana_folder" "<terraform_folder_name>" {
       title = "My Rule Folder"
   }
   ```

   Replace the following field values:
   - `<terraform_data_source_name>` with the terraform name of the data source.
   - `<terraform_folder_name>` with the terraform name of the folder.

1. Create or find an alert rule you want to import in Grafana.

1. [Export](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources/) the alert rule group in Terraform format. This exports the alert rule group as [`grafana_rule_group` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/rule_group).

   You can edit the exported resource, or alternatively, consider creating the resource from scratch.

   ```terraform
   resource "grafana_rule_group" "<terraform_rule_group_name>" {
       name = "My Alert Rules"
       folder_uid = grafana_folder.<terraform_folder_name>.uid
       interval_seconds = 60
       org_id = 1

       rule {
           name = "My Random Walk Alert"
           condition = "C"
           for = "0s"

           // Query the datasource.
           data {
               ref_id = "A"
               relative_time_range {
                   from = 600
                   to = 0
               }
               datasource_uid = grafana_data_source.<terraform_data_source_name>.uid
               // `model` is a JSON blob that sends datasource-specific data.
               // It's different for every datasource. The alert's query is defined here.
               model = jsonencode({
                   intervalMs = 1000
                   maxDataPoints = 43200
                   refId = "A"
               })
           }

           // The query was configured to obtain data from the last 60 seconds. Let's alert on the average value of that series using a Reduce stage.
           data {
               datasource_uid = "__expr__"
               // You can also create a rule in the UI, then GET that rule to obtain the JSON.
               // This can be helpful when using more complex reduce expressions.
               model = <<EOT
   {"conditions":[{"evaluator":{"params":[0,0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["A"]},"reducer":{"params":[],"type":"last"},"type":"avg"}],"datasource":{"name":"Expression","type":"__expr__","uid":"__expr__"},"expression":"A","hide":false,"intervalMs":1000,"maxDataPoints":43200,"reducer":"last","refId":"B","type":"reduce"}
   EOT
               ref_id = "B"
               relative_time_range {
                   from = 0
                   to = 0
               }
           }

           // Now, let's use a math expression as our threshold.
           // We want to alert when the value of stage "B" above exceeds 70.
           data {
               datasource_uid = "__expr__"
               ref_id = "C"
               relative_time_range {
                   from = 0
                   to = 0
               }
               model = jsonencode({
                   expression = "$B > 70"
                   type = "math"
                   refId = "C"
               })
           }
       }
   }
   ```

   Replace the following field values:
   - `<terraform_rule_group_name>` with the name of the alert rule group.

   Note that the distinct Grafana resources are connected through `uid` values in their Terraform configurations. The `uid` value will be randomly generated when provisioning.

   To link the alert rule group with its respective data source and folder in this example, replace the following field values:
   - `<terraform_data_source_name>` with the terraform name of the previously defined data source.
   - `<terraform_folder_name>` with the terraform name of the previously defined folder.

1. Continue to add more Grafana resources or [use the Terraform CLI for provisioning](#provision-grafana-resources-with-terraform).

### Add contact points

[Contact points](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/) are the receivers of alert notifications.

1. Create or find the contact points you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.

1. [Export](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources/) the contact point in Terraform format. This exports the contact point as [`grafana_contact_point` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/contact_point)—edit it if necessary.

1. In this example, notifications are muted on weekends.

   ```terraform
    resource "grafana_contact_point" "<terraform_contact_point_name>" {
        name = "My contact point email"

        email {
            addresses               = ["<email_address>"]
        }
    }
   ```

   Replace the following field values:
   - `<terraform_contact_point_name>` with the terraform name of the contact point. It will be used to reference the contact point in other Terraform resources.
   - `<email_address>` with the email to receive alert notifications.

1. Continue to add more Grafana resources or [use the Terraform CLI for provisioning](#provision-grafana-resources-with-terraform).

### Add and enable notification templates

[Notification templates](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/) allow customization of alert notifications across multiple contact points.

1. Create or find the notification template group you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.

1. [Export](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources/) the notification template group as [`grafana_message_template` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/message_template).

   This example creates a notification template group named `custom_emails` that defines a `custom_email.message` template.

   ```terraform
    resource "grafana_message_template" "<terraform_message_template_name>" {
        name = "custom_emails"

        template = <<EOT
    {{ define "custom_email.message" }}
    Lorem ipsum - Custom alert!
    {{ end }}
    EOT
    }
   ```

   This enables contact points to use the notification templates (`{{ define "<NAME>"}}`) within the notification template group.

1. In the previous contact point, enable the template by setting the `email.message` property as follows.

   ```terraform
    resource "grafana_contact_point" "<terraform_contact_point_name>" {
        name = "My contact point email"

        email {
            addresses               = ["<email_address>"]
            message                 = "{{ template \"custom_email.message\" .}}"
        }
    }
   ```

1. Continue to add more Grafana resources or [use the Terraform CLI for provisioning](#provision-grafana-resources-with-terraform).

### Add mute timings

[Mute timings](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/) pause alert notifications during predetermined intervals.

1. Create or find the mute timings you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.

1. [Export](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources/) the mute timing in Terraform format. This exports the mute timing as [`grafana_mute_timing` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/mute_timing)—edit it if necessary.

1. This example turns off notifications on weekends.

   ```terraform
    resource "grafana_mute_timing" "<terraform_mute_timing_name>" {
        name = "No weekends"

        intervals {
            weekdays = ["saturday", "sunday"]
        }
    }
   ```

   Replace the following field values:
   - `<terraform_mute_timing_name>` with the name of the Terraform resource. It will be used to reference the mute timing in the Terraform notification policy tree.

1. Continue to add more Grafana resources or [use the Terraform CLI for provisioning](#provision-grafana-resources-with-terraform).

### Add the notification policy tree

[Notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy/) defines how to route alert instances to your contact points.

{{< docs/shared lookup="alerts/warning-provisioning-tree.md" source="grafana" version="<GRAFANA_VERSION>" >}}

1. Find the default notification policy tree. Alternatively, consider writing the resource in code as demonstrated in the example below.

1. [Export](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources/) the notification policy tree in Terraform format. This exports it as [`grafana_notification_policy` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/notification_policy)—edit it if necessary.

   ```terraform
   resource "grafana_notification_policy" "my_policy_tree" {
   contact_point = grafana_contact_point.<terraform_contact_point_name>.name
   ...

   policy {
       contact_point = grafana_contact_point.<terraform_contact_point_name>.name

       matcher {...}

       mute_timings = [grafana_mute_timing.<terraform_mute_timing_name>.name]
   }
   }
   ```

   To configure the mute timing and contact point previously created in the notification policy tree, replace the following field values:
   - `<terraform_data_source_name>` with the terraform name of the previously defined contact point.
   - `<terraform_folder_name>` with the terraform name of the previously defined mute timing.

1. Continue to add more Grafana resources or [use the Terraform CLI for provisioning](#provision-grafana-resources-with-terraform).

### Enable multiple notification policy trees

By default, Grafana routes all alerts to the default notification policy tree using the `grafana_notification_policy` resource as shown above.

[Multiple notification policy trees](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy/#manage-multiple-notification-policy-trees) let you split routing logic into separate routing trees, for example, one per team. Each routing tree is provisioned using its own [`grafana_apps_notifications_routingtree_v1beta1` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_notifications_routingtree_v1beta1).

1. Create a routing tree for each scope.

   ```terraform
   resource "grafana_apps_notifications_routingtree_v1beta1" "team_platform" {
       metadata {
           uid = "platform-routing-tree"
       }
       spec {
            # ...
       }
   }
   ```

   - `metadata.uid` sets the routing tree's unique identifier. Alert rules reference this value to route into this tree instead of the default policy tree.
   - For routing tree settings, refer to the [`grafana_apps_notifications_routingtree_v1beta1` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_notifications_routingtree_v1beta1).

1. To route notifications from an alert rule to a specific tree, set the `notification_settings.policy` option in the alert rule to the tree's `metadata.uid`.

   ```terraform
   resource "grafana_rule_group" "<terraform_rule_group_name>" {
        # ...
        rule {
            name = "My Random Walk Alert"
            # ...
            notification_settings {
                policy = grafana_apps_notifications_routingtree_v1beta1.team_platform.metadata.uid
            }
        }
        # ...
   }
   ```

   Alert rules that don't set `notification_settings.policy` continue to route through the default notification policy tree.

1. Continue to add more Grafana resources or [use the Terraform CLI for provisioning](#provision-grafana-resources-with-terraform).

### Enable editing resources in the Grafana UI

By default, you cannot edit resources provisioned via Terraform in Grafana. This ensures that your alerting stack always stays in sync with your Terraform code.

To make provisioned resources editable in the Grafana UI, enable the `disable_provenance` attribute on alerting resources.

```terraform
resource "grafana_contact_point" "my_contact_point" {
  name = "My Contact Point"

  disable_provenance = true
}

resource "grafana_message_template" "custom_notification_template_group" {
  name     = "custom_notification_template_group"
  template = "{{define \"template1\" }}Say{{ end }}{{define \"template2\" }}Hi!{{ end }}"

  disable_provenance = true
}
...
```

## Provision Grafana resources with Terraform

To create the previous alerting resources in Grafana with the Terraform CLI, complete the following steps.

1. Initialize the working directory containing the Terraform configuration files.

   ```shell
   terraform init
   ```

   This command initializes the Terraform directory, installing the Grafana Terraform provider configured in the `main.tf` file.

1. Apply the Terraform configuration files to provision the resources.

   ```shell
   terraform apply
   ```

   Before applying any changes to Grafana, Terraform displays the execution plan and requests your approval.

   ```shell
    Plan: 4 to add, 0 to change, 0 to destroy.

    Do you want to perform these actions?
    Terraform will perform the actions described above.
    Only 'yes' will be accepted to approve.

    Enter a value:
   ```

   Once you have confirmed to proceed with the changes, Terraform will create the provisioned resources in Grafana!

   ```shell
   Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
   ```

You can now access Grafana to verify the creation of the distinct resources.

## More examples

For more examples on the concept of this guide:

- Try the demo [provisioning alerting resources in Grafana OSS using Terraform and Docker Compose](https://github.com/grafana/provisioning-alerting-examples/tree/main/terraform).
- Review all the available options and examples of the Terraform Alerting schemas in the [Grafana Terraform Provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs).
- Review the [tutorial to manage a Grafana Cloud stack using Terraform](/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/terraform-cloud-stack/).

# As Code

# Create and manage your Grafana Cloud stack using Ansible

This guide shows you how to create a Grafana Cloud stack and add a data source, dashboard, and folder using the Ansible Collection for Grafana. You'll manage your Grafana infrastructure through Ansible playbooks.

## Before you begin

Before you begin, make sure you have the following available:

- A Grafana Cloud account
- [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/index.html) installed on your machine

## Install the Grafana Ansible collection

Install the Grafana Ansible collection:

```sh
ansible-galaxy collection install grafana.grafana
```

This collection provides all the modules needed to manage Grafana Cloud stacks and resources.

## Create a Cloud stack

First, create a Grafana Cloud Access Policy and get a token. You'll need this for the Ansible playbook to be able to create a Grafana Cloud stack. Refer to [Create a Grafana Cloud Access Policy](/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-policies/create-access-policies/).

Next, create an Ansible playbook file. This Ansible playbook creates a Grafana Cloud stack using the [Cloud stack module](https://docs.ansible.com/ansible/latest/collections/grafana/grafana/cloud_stack_module.html#ansible-collections-grafana-grafana-cloud-stack-module).

To do so, create a file named `cloud-stack.yml` and add the following:

```yaml
- name: Create Grafana Cloud stack
  connection: local
  hosts: localhost

  vars:
    grafana_cloud_api_key: '<CLOUD_ACCESS_POLICY_TOKEN>'
    stack_name: '<STACK_NAME>'
    org_name: '<ORG_NAME>'

  tasks:
    - name: Create a Grafana Cloud stack
      grafana.grafana.cloud_stack:
        name: '{{ stack_name }}'
        stack_slug: '{{ stack_name }}'
        cloud_api_key: '{{ grafana_cloud_api_key }}'
        org_slug: '{{ org_name }}'
        delete_protection: true
        state: present
      register: stack_result

    - name: Display stack URL
      debug:
        msg: 'Stack created at: {{ stack_result.url }}'
```

Replace the placeholders with your values:

- _`<CLOUD_ACCESS_POLICY_TOKEN>`_: Token from the Cloud Access Policy you created in the Grafana Cloud portal
- _`<STACK_NAME>`_: Name of your stack
- _`<ORG_NAME>`_: Name of the organization in Grafana Cloud

The playbook registers the stack creation result and displays the stack URL, which you'll need for subsequent resource management.

## Create an API key in your Grafana stack

Create an API key in the Grafana stack. You'll need this key to configure Ansible to create data sources, folders, and dashboards.

1. Log into your Grafana Cloud instance.
2. Click **Administration** and select **API keys**.
3. Click **Add API key**.
4. In **Key name**, enter a name for your API key.
5. In **Role**, select **Admin** or **Editor** to associate the role with this API key.
6. Click **Copy** to save it for later use.

## Add resources using playbooks

### Add a data source

The following steps use the InfluxDB data source. The required arguments vary depending on the type of data source you select.

Create a file named `data-source.yml`:

```yaml
- name: Add/Update data source
  connection: local
  hosts: localhost

  vars:
    grafana_url: 'https://<STACK_NAME>.grafana.net'
    grafana_api_key: '<GRAFANA_API_KEY>'
    data_source_config:
      name: '<DATA_SOURCE_NAME>'
      type: 'influxdb'
      url: '<DATA_SOURCE_URL>'
      user: '<USERNAME>'
      secureJsonData:
        password: '<PASSWORD>'
      database: '<DATABASE_NAME>'
      uid: '<UID>'
      access: 'proxy'

  tasks:
    - name: Create/Update Data source
      grafana.grafana.datasource:
        dataSource: '{{ data_source_config }}'
        grafana_url: '{{ grafana_url }}'
        grafana_api_key: '{{ grafana_api_key }}'
        state: present
```

Replace the placeholders with your values:

- _`<DATA_SOURCE_NAME>`_: Name of the data source to be added in Grafana
- _`<DATA_SOURCE_URL>`_: URL of your data source
- _`<USERNAME>`_: Username for authenticating with your data source
- _`<PASSWORD>`_: Password for authenticating with your data source
- _`<DATABASE_NAME>`_: Name of your database
- _`<UID>`_: UID for your data source in Grafana
- _`<STACK_NAME>`_: Name of your stack
- _`<GRAFANA_API_KEY>`_: API key created in the Grafana instance

### Add a folder

This playbook creates a folder in your Grafana instance using the [Folder module](https://docs.ansible.com/ansible/latest/collections/grafana/grafana/folder_module.html#ansible-collections-grafana-grafana-folder-module).

Create a file named `folder.yml`:

```yaml
- name: Add/Update Folders
  connection: local
  hosts: localhost

vars:
    grafana_url: 'https://<STACK_NAME>.grafana.net'
    grafana_api_key: '<GRAFANA_API_KEY>'
    folders:
      - title: '<FOLDER_NAME>'
        uid: '<UID>'

  tasks:
    - name: Create/Update a Folder in Grafana
      grafana.grafana.folder:
        title: '{{ item.title }}'
        uid: '{{ item.uid }}'
        grafana_url: '{{ grafana_url }}'
        grafana_api_key: '{{ grafana_api_key }}'
        state: present
      loop: '{{ folders }}'
```

Replace the placeholders with your values:

- _`<FOLDER_NAME>`_: Name of the folder to be added in Grafana
- _`<UID>`_: UID for your folder in Grafana
- _`<STACK_NAME>`_: Name of your stack
- _`<GRAFANA_API_KEY>`_: API key created in the Grafana instance

### Add a dashboard to the folder

This playbook iterates through the dashboard JSON source code files in the folder referenced in `dashboards_path` and adds them to the Grafana instance using the [Dashboard module](https://docs.ansible.com/ansible/latest/collections/grafana/grafana/dashboard_module.html#ansible-collections-grafana-grafana-dashboard-module).

Create a file named `dashboard.yml`:

```yaml
- name: Add/Update Dashboards
  connection: local
  hosts: localhost

  vars:
    grafana_url: 'https://<STACK_NAME>.grafana.net'
    grafana_api_key: '<GRAFANA_API_KEY>'
    dashboards_path: '<PATH_TO_DASHBOARD_FILES>' # Example "./dashboards"

  tasks:
    - name: Find dashboard files
      find:
        paths: '{{ dashboards_path }}'
        file_type: file
        recurse: true
        patterns: '*.json'
      register: files_matched
      no_log: true

    - name: Create list of dashboard file names
      set_fact:
        dashboard_file_names: '{{ dashboard_file_names | default([]) + [item.path] }}'
      loop: '{{ files_matched.files }}'
      no_log: true

    - name: Create/Update a dashboard
      grafana.grafana.dashboard:
        dashboard: "{{ lookup('ansible.builtin.file', item) }}"
        grafana_url: '{{ grafana_url }}'
        grafana_api_key: '{{ grafana_api_key }}'
        state: present
      loop: '{{ dashboard_file_names }}'
```

Replace the placeholders with your values:

- _`<PATH_TO_DASHBOARD_FILES>`_: Path to the folder containing dashboard JSON source code files
- _`<STACK_NAME>`_: Name of your stack
- _`<GRAFANA_API_KEY>`_: API key created in the Grafana instance

## Run the Ansible playbooks

In a terminal, run the following commands from the directory where all of the Ansible playbooks are located.

Create the Grafana Cloud stack:

```sh
ansible-playbook cloud-stack.yml
```

Add a data source to the Grafana stack:

```sh
ansible-playbook data-source.yml
```

Add a folder to the Grafana stack:

```sh
ansible-playbook folder.yml
```

Add a dashboard to the folder in your Grafana stack:

```sh
ansible-playbook dashboard.yml
```

## Validate your configuration

After you've run the Ansible playbooks, you can verify the following:

- The new Grafana Cloud stack is created and visible in the Cloud Portal.

  ![Cloud Portal](/static/img/docs/grafana-cloud/terraform/cloud_portal_tf.png)

- A new data source (InfluxDB in this example) is visible in the Grafana stack.

  ![InfluxDB datasource](/media/docs/grafana-cloud/screenshot-influxdb_datasource_tf.png)

- A new folder is available in your Grafana stack. In the following image, a folder named `Demos` was added.

  ![Folder](/media/docs/grafana-cloud/screenshot-folder_tf.png)

- A new dashboard is visible in the Grafana stack. In the following image, a dashboard named `InfluxDB Cloud Demos` was created inside the "Demos" folder.

  ![InfluxDB dashboard](/static/img/docs/grafana-cloud/terraform/influxdb_dashboard_tf.png)

## Next steps

You've successfully created a Grafana Cloud stack along with a data source, a folder, and a dashboard using Ansible. Your Grafana infrastructure is now managed through code.

To learn more about managing Grafana with Infrastructure as code:

- [Grafana Ansible collection documentation](https://docs.ansible.com/ansible/latest/collections/grafana/grafana/)
- [Ansible playbook best practices](https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html)
- [Grafana API documentation](/docs/grafana/latest/developer-resources/api-reference/http-api/)
- [Grafana Cloud API documentation](https://grafana.com/docs/grafana-cloud/developer-resources/api-reference/)
- [Infrastructure as Code with Terraform](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/terraform/)

# Automate dashboard provisioning with CI/CD

Managing Grafana dashboards manually can be inefficient and error-prone. With the Grafana Foundation SDK you can define dashboards using strongly typed code, commit them to version control systems, and automatically deploy them using GitHub Actions.

## Why automate?

Automating Grafana dashboard deployment eliminates the need for manual dashboard creation and updates, ensuring that dashboards remain consistent across environments.

By defending dashboards as code and managing them through CI/CD such as GitHub Actions, you will gain full version control, making it easy to track changes over time and roll back if needed. This also prevents duplication, as the workflow intelligently checks whether a dashboard exists before deciding to create or update it.

With this fully automated CI/CD pipeline, you can focus on improving your dashboards rather than manually uploading JSON files to Grafana.

{{< youtube id="cFnO8kVOaAI" >}}

You can find the full example source code in the [Introduction to the Foundation SDK](https://github.com/grafana/intro-to-foundation-sdk/tree/main/github-actions-example) GitHub repository.

## Overview

This guide shows you how to:

1. Generate a Grafana dashboard as code and format it for Kubernetes-style deployment
2. Use GitHub Actions to deploy, verify, and update the dashboard

By the end, you'll be able to provision your dashboard as code, and every change to your dashboard will be automatically created or updated in your Grafana instance without manual intervention.

## 1. Generate the dashboard JSON

Before deploying a dashboard, [define it as code using the Grafana Foundation SDK](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/foundation-sdk/#create-a-dashboard/).

Since Grafana exposes a Kubernetes resource compatible API, you need to make some changes to the code to output the dashboard JSON in the appropriate format.

This script:

- Generates a Grafana dashboard JSON file
- Wraps it in a Kubernetes-style API format (`apiVersion`, `kind`, `metadata`, `spec`)
- Saves it as `dashboard.json` for deployment

{{< code >}}

```go
package main

import (
	"encoding/json"
	"log"
	"os"

	"github.com/grafana/grafana-foundation-sdk/go/cog"
	"github.com/grafana/grafana-foundation-sdk/go/common"
	"github.com/grafana/grafana-foundation-sdk/go/dashboard"
)

type DashboardWrapper struct {
	APIVersion string              `json:"apiVersion"`
	Kind       string              `json:"kind"`
	Metadata   Metadata            `json:"metadata"`
	Spec       dashboard.Dashboard `json:"spec"`
}

type Metadata struct {
	Name string `json:"name"`
}

func main() {
	builder := dashboard.NewDashboardBuilder("My Dashboard").
		Uid("my-dashboard").
		Tags([]string{"generated", "foundation-sdk", "go"}).
		Refresh("5m").
		Time("now-1h", "now").
		Timezone(common.TimeZoneBrowser).
		WithRow(dashboard.NewRowBuilder("Overview"))

	dashboard, err := builder.Build()
	if err != nil {
		log.Fatalf("failed to build dashboard: %v", err)
	}

	dashboardWrapper := DashboardWrapper{
		APIVersion: "dashboard.grafana.app/v1",
		Kind:       "Dashboard",
		Metadata: Metadata{
			Name: *dashboard.Uid,
		},
		Spec: dashboard,
	}

	dashboardJson, err := json.MarshalIndent(dashboardWrapper, "", "  ")
	if err != nil {
		log.Fatalf("failed to marshal dashboard: %v", err)
	}

	err = os.WriteFile("dashboard.json", dashboardJson, 0644)
	if err != nil {
		log.Fatalf("failed to write dashboard to file: %v", err)
	}

	log.Printf("Dashboard JSON:\n%s", dashboardJson)
}
```

```typescript
import { DashboardBuilder, RowBuilder } from '@grafana/grafana-foundation-sdk/dashboard';
import * as fs from 'fs';

// Generate the dashboard JSON
const dashboard = new DashboardBuilder('My Dashboard')
  .uid('my-dashboard')
  .tags(['generated', 'foundation-sdk', 'typescript'])
  .refresh('5m')
  .time({ from: 'now-1h', to: 'now' })
  .timezone('browser')
  .withRow(new RowBuilder('Overview'))
  .build();

// Convert to Kubernetes-style format
const dashboardWrapper = {
  apiVersion: "dashboard.grafana.app/v1",
  kind: "Dashboard",
  metadata: {
    name: dashboard.uid!
  },
  spec: dashboard
};

// Save the formatted JSON to a file
const dashboardJSON = JSON.stringify(dashboardWrapper, null, 2);
fs.writeFileSync('dashboard.json', dashboardJSON, 'utf8');

console.log(`Dashboard JSON:\n${}`);
```

{{< /code >}}

## 2. Automate deployment with GitHub Actions

Next, set up GitHub Actions to automate the deployment of a Grafana dashboard using the Foundation SDK and the [`gcx` CLI tool](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/ai-tools/grafana-cli/) to:

- Extract the dashboard name from `dashboard.json`
- Check if the dashboard already exists within our Grafana instance
- Update it if it does, create it if it doesn’t

{{< admonition type="note" >}}
The following GitHub Action configuration assumes you are using a Go-based dashboard generator. If you're using one of the other languages that the Foundation SDK supports, modify the **Generate Dashboard JSON** step accordingly.
{{< /admonition >}}

The `.github/workflows/deploy-dashboard.yml` deploy workflow looks like:

```yaml
name: Deploy Grafana Dashboard

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: 1.24.6

      - name: Verify Go version
        run: go version

      - name: Download and Extract gcx
        run: |
          curl -L -o gcx-x86_64.tar.gz "https://github.com/grafana/gcx/releases/download/${{ vars.GCX_VERSION }}/gcx_Linux_x86_64.tar.gz"
          tar -xzf gcx-x86_64.tar.gz
          chmod +x gcx
          sudo mv gcx /usr/local/bin/gcx

      - name: Generate Dashboard JSON
        working-directory: ./github-actions-example
        run: go run main.go

      - name: Deploy Dashboard with gcx
        env:
          GRAFANA_SERVER: ${{ vars.GRAFANA_SERVER }}
          GRAFANA_STACK_ID: ${{ vars.GRAFANA_STACK_ID }}
          GRAFANA_TOKEN: ${{ secrets.GRAFANA_TOKEN }}
        run: |
          if [ -f dashboard.json ]; then
            echo "dashboard.json exists, deploying dashboard."
            gcx resources push dashboards --path ./dashboard.json
          else
            echo "dashboard.json does not exist."
            exit 1
          fi
        working-directory: ./github-actions-example
```

### 1. Checkout and set up Go

To set up Go:

- Check out the repository to access the project code.
- Install Go 1.24.6 using the `actions/setup-go` action.
- Verify Go is properly installed.

### 2. Download and install `gcx`

Next, download the `gcx` CLI from GitHub using the version defined in `vars.GCX_VERSION`. It unpacks the tarball, makes it executable, and moves it to a location in the system `PATH`.

### 3. Generate the dashboard JSON

Next, run the dashboard generator (`main.go`) from the `./github-actions-example` director to produce a `dashboard.json` file that contains the Grafana dashboard definition.

### 4. Deploy the dashboard with `gcx`

If `dashboard.json` already exists, it is deployed to your Grafana instance using:

```bash
gcx resources push dashboards --path ./dashboard.json
```

This command authenticates against Grafana using the following environment variables:

- `GRAFANA_SERVER`: Your Grafana instance URL
- `GRAFANA_STACK_ID`: Your Grafana stack ID
- `GRAFANA_TOKEN`: A Grafana service account token with sufficient permissions

### GitHub variables and secrets used

Verify these variables are configured in your repository under **Settings > Security > Secrets and variables > Actions**:

- `vars.GCX_VERSION`: Version of `gcx` to install
- `vars.GRAFANA_SERVER`: The URL of your Grafana instance
- `vars.GRAFANA_STACK_ID`: The stack ID in Grafana
- `secrets.GRAFANA_TOKEN`: Grafana API token

This action ensures that every push to `main` will regenerate and deploy your latest dashboard definition to Grafana.

# Foundation SDK key concepts

The Grafana Foundation SDK is built around a few core concepts that make your dashboards structured, reusable, and strongly typed.

## Builders

The SDK follows a builder pattern, which lets you compose dashboards step-by-step using chained method calls.
Almost every piece of the dashboard, including dashboards, panels, rows, queries, and variables, has its own `Builder` class.

Here are a few you've already seen:

- `DashboardBuilder` - Starts the dashboard definition and sets global configuration settings like title, UID, refresh interval, time range, etc.
- `PanelBuilder` - Creates individual visualizations like time series panels, stat panels, or log panels.
- `DataqueryBuilder` - Defines how a panel fetches data, for example, from Prometheus or the `testdata` plugin.

Builders can be chained, so you can fluently compose dashboards in a readable, structured way:

{{< code >}}

```go
stat.NewPanelBuilder().
  Title("Version").
  Datasource(testdataRef).
  ReduceOptions(common.NewReduceDataOptionsBuilder().
    Calcs([]string{"lastNotNull"}).
    Fields("/.*/")).
  WithTarget(
    testdata.NewDataqueryBuilder().
      ScenarioId("csv_content").
      CsvContent("version\nv1.2.3"),
  )
```

```typescript
new stat.PanelBuilder()
  .title('Version')
  .reduceOptions(new common.ReduceDataOptionsBuilder().calcs(['lastNotNull']).fields('/.*/'))
  .datasource(testdataRef)
  .withTarget(new testdata.DataqueryBuilder().scenarioId('csv_content').csvContent('version\nv1.2.3'));
```

{{< /code >}}

## Types

The Grafana Foundation SDK uses strong types under the hood to help catch mistakes before you deploy a broken dashboard.

For example:

- When setting a unit, you'll get autocomplete suggestions for valid Grafana units like `"percent"` or `"bps"`.
- When defining a time range, you'll be guided to provide the correct structure, like `from` and `to` values.
- When referencing data sources, you'll use a structured `DataSourceRef` object with defined `type` and `uid` fields.

This helps you:

- Avoid typos or unsupported configuration values
- Get full autocomplete and inline documentation in your IDE
- Write dashboards that are less error-prone and easier to maintain

Strong typing also makes it easier to build reusable patterns and components with confidence, especially in large codebases or teams.

## Options

Most builder methods accept simple values like strings or numbers, but others expect more structured option objects. These are used for things like:

- `ReduceDataOptions` - How to reduce time series data into single values (e.g. last, avg).
- `VizLegendOptions` - Configure how the legend of a panel is displayed.
- `CanvasElementOptions` - Define how the various components of a Canvas panel should be displayed.

Example using options:

{{< code >}}

```go
stat.NewPanelBuilder().
  ReduceOptions(common.NewReduceDataOptionsBuilder().
    Calcs([]string{"lastNotNull"}).
    Fields("/.*/"))
  )
```

```typescript
new stat.PanelBuilder().reduceOptions(new common.ReduceDataOptionsBuilder().calcs(['lastNotNull']).fields('/.*/'));
```

{{< /code >}}

By using option builders, you don't need to manually construct deeply nested configuration objects. Instead, the SDK gives you a typed and guided API that mirrors a dashboards internal structure, making it easier to configure complex options without guesswork or referring back to the JSON schema.

# Git Sync key concepts

{{< admonition type="note" >}}

Git Sync functionalities are constantly evolving. [Contact Grafana](https://grafana.com/help/) for support or to report any issues you encounter and help us improve this feature.

{{< /admonition >}}

## Key Git Sync components

Before you start using Git Sync, understand how the key Git Sync components relate:

- [Grafana instance](#grafana-instance)
- [Git repository](#git-repository)
- [Git Sync repository resource](#git-sync-repository-resource)
- [Git Sync connection resource](#git-sync-connection-resource)

### Grafana instance

A Grafana instance is a running Grafana server. Multiple instances can:

- Connect to the same Git repository using different Repository configurations.
- Sync from different branches of the same repository.
- Sync from different paths within the same repository.
- Sync from different repositories.

### Git repository

A Git repository is the external storage you want to sync your Grafana instance with. You can organize your Git repository in several ways:

- Single branch, multiple paths: Use different directories for different purposes. For example, `dev/`, `prod/`, or `team-a/`.
- Multiple branches: Use different branches for different environments or teams. For example, `main`, `develop`, or `team-a`.
- Multiple repositories: Use separate repositories for different teams or environments.

#### Flexible configuration for your Git Sync repositories

Git Sync repositories support different combinations of repository URL, branch, and path:

- Different Git repositories: Each environment or team can use its own repository.
  - Instance A: `repository: your-org/grafana-prod`.
  - Instance B: `repository: your-org/grafana-dev`.
- Different branches: Use separate branches within the same repository.
  - Instance A: `repository: your-org/grafana-manifests, branch: main`.
  - Instance B: `repository: your-org/grafana-manifests, branch: develop`.
- Different paths: Use different directory paths within the same repository.
  - Instance A: `repository: your-org/grafana-manifests, branch: main, path: production/`.
  - Instance B: `repository: your-org/grafana-manifests, branch: main, path: development/`.
- Any combination: Mix and match based on your workflow requirements.

### Git Sync repository resource

A repository resource is a Grafana configuration object that defines the connection between a group of repositories and your Grafana instance via Git Sync.

- Which Git repository to sync with your Grafana instance.
- Which branch to use.
- Which directory path to synchronize.
- Sync behavior and workflows.

Each repository resource creates bidirectional synchronization between a Grafana instance and a specific location in Git.

### Git Sync connection resource

A connection is the authentication setup between Grafana and an external Git provider, required when you're not using a Personal Access Token or a static token to authenticate. Use it to authorize access to your external repositories, and to generate or refresh the credentials for Git Sync. A single connection can be reused across multiple repositories.

For example, if you're using GitHub App to authenticate, the connection represents the app installation. Grafana uses this connection to authenticate with GitHub, create access tokens, and authorize repository access on your behalf.

The connection resource includes:

- The external provider configuration: The authentication mechanism Grafana uses to communicate with the Git provider (for example, a GitHub App).
- The provider authorization or installation: The authorized entity that allows Grafana to request tokens from the provider.
- Repository access scope: The set of repositories that the connection (and therefore, Grafana) is authorized to access in the Git provider.

## How does Git Sync behave?

Git Sync is bidirectional, and syncs a repository resource with your Grafana instance. You can modify provisioned resources both from the Grafana UI or from the synced GitHub repository, and changes will be reflected in both places:

1. Grafana monitors the specified Git location (repository, branch, and path).
2. Grafana creates a folder in Dashboards (typically named after the repository).
3. Grafana creates dashboards from dashboard JSON files in Git within this folder.
4. Grafana commits dashboard changes made in the UI back to Git.
5. Grafana pulls dashboard changes made in Git and updates dashboards in the UI.
6. Synchronization occurs at regular intervals (configurable), or instantly if you use webhooks.

You can find the provisioned dashboards organized in folders under **Dashboards**.

### Sync targets

With Git Sync you can place synced resources in Grafana in two ways:

- **Folder sync**: Grafana creates a folder named after the repository and places all synced resources inside it. Subdirectories in the repository become subfolders within that folder. **This is the default behavior**.
- **Folderless sync**: Grafana places synced resources at the top level, without creating a wrapper folder. Files at the repository path root become top-level resources, and subdirectories become top-level folders.

Use folder sync to keep each repository's resources grouped together under a dedicated folder. Use folderless sync when you want provisioned resources to appear at the top of your Dashboards view instead of nested inside a repository folder.

Both modes can coexist with each other and with resources that aren't managed by Git Sync.

The following examples use the same repository to show how the same files appear with each mode.

The repository `grafana-manifests` syncs from the path `grafana/`:

```
your-org/grafana-manifests/
└── grafana/
    ├── cpu-metrics.json
    └── team-platform/
        ├── _folder.json
        └── memory-usage.json
```

The `_folder.json` file stores the `team-platform` folder's stable UID and display name, so the folder keeps its identity and permissions if you move or rename it in the repository. Refer to [The Git Sync folder metadata file](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/use-git-sync/#the-git-sync-folder-metadata-file) for details.

The instance also has content that isn't managed by Git Sync: a manually created **Ops** folder and an **Ad-hoc dashboard**.

**With folder sync**, a repository folder wraps the synced resources, alongside the unprovisioned content:

```
Dashboards
├── 📁 grafana-manifests/      ← managed by Git Sync
│   ├── CPU Metrics Dashboard
│   └── 📁 team-platform/
│       └── Memory Usage Dashboard
├── 📁 Ops/                    ← not managed by Git Sync
│   └── Ops dashboard
└── Ad-hoc dashboard           ← not managed by Git Sync
```

**With folderless sync**, the same files map to the top level, next to the unprovisioned content:

```
Dashboards
├── CPU Metrics Dashboard      ← managed by Git Sync
├── 📁 team-platform/          ← managed by Git Sync
│   └── Memory Usage Dashboard
├── 📁 Ops/                    ← not managed by Git Sync
│   └── Ops dashboard
└── Ad-hoc dashboard           ← not managed by Git Sync
```

Folderless sync only manages the resources it provisions. The unprovisioned **Ops** folder and **Ad-hoc dashboard** are left untouched.

#### Multiple folderless repositories

Because folderless sync doesn't create a wrapper folder, several folderless repositories can sync to the top level at the same time. Each repository manages only the resources it provisions:

```
Dashboards
├── CPU Metrics Dashboard      ← managed by grafana-manifests
├── 📁 team-platform/          ← managed by grafana-manifests
│   └── Memory Usage Dashboard
├── Billing Overview           ← managed by finance-dashboards
├── 📁 invoices/               ← managed by finance-dashboards
│   └── Monthly Invoices
└── Ad-hoc dashboard           ← not managed by Git Sync
```

### Git Sync states

Your Grafana instance can be in one of the following Git Sync states:

- **Unprovisioned**: None of the instance's resources are being managed by Git Sync.
- **Partially provisioned**: Some of the resources are controlled by Git Sync.
- **Fully provisioned**: All supported resource types are managed by Git Sync. Unsupported resources are **not managed**.

## Example: Relationship between repository, branch, and path

Here's an example showing how the repository, branch, and path concepts work together:

**Configuration:**

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `main`
- **Path**: `team-platform/grafana/`

**In Git (on branch `main`):**

```
your-org/grafana-manifests/
├── .git/
├── README.md
├── team-platform/
│   └── grafana/
│       ├── cpu-metrics.json       ← Synced
│       ├── memory-usage.json      ← Synced
│       └── disk-io.json           ← Synced
├── team-data/
│   └── grafana/
│       └── pipeline-stats.json    ← Not synced (different path)
└── other-files.txt                ← Not synced (outside path)
```

**In the Grafana Dashboards view:**

```
Dashboards
└── 📁 grafana-manifests/
    ├── CPU Metrics Dashboard
    ├── Memory Usage Dashboard
    └── Disk I/O Dashboard
```

**Key takeaways:**

- Grafana only synchronizes files within the specified path (`team-platform/grafana/`).
- Grafana ignores files in other paths or at the repository root.
- The folder name in Grafana comes from the repository name.
- Dashboard titles come from the JSON file content, not the filename.

# Configure Git repository protection

For Git Sync to work correctly you need to configure permissions at two layers:

- At your Git provider level, to protect your repository, as described in this document.
- At the Grafana level for repository management and resource access. Refer to [Git Sync permissions and access control](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/permissions-grafana) for more information.

When you use Git Sync, your dashboard configurations are stored as code in a Git repository. You can configure Git repository protection to control who can access this source code and who can modify it. Repository protection works as an additional security layer after Grafana internal permissions.

## Required permissions at the Git provider level

Git Sync authentication credentials must have specific permissions at your Git provider:

**Required for all configurations**:

- Read access to repository contents
- Read access to branch information

**Required for writing changes**:

- Write access to create commits
- Permission to create pull requests (when branch protection is enabled)
- Permission to push to feature branches (for creating pull requests)

**Optional for instant synchronization**:

- Permission to create and manage webhooks

Refer to the [Git Sync setup documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup) for detailed instructions on configuring authentication for your Git provider.

## Control access to the dashboards source code

You can access dashboard source code through two paths, each with its own protection mechanism:

- **Grafana files endpoint**: Users can view and edit dashboard source code through the Grafana files API endpoint. Use Grafana folder and dashboard permissions to control this access. Refer to [Manage access to dashboard source code via the API](#manage-access-to-dashboard-source-code-via-the-api) for more information.

- **Git repository**: Users with repository access can view and modify dashboard files directly in Git. You can control repository permissions in your Git provider. Refer to [Control access to your Git repository ](#control-access-to-your-git-repository) for more information.

Protect both access points to secure your dashboard configurations.

## Manage access to dashboard source code via the API

### View dashboard source code

You can view dashboard source code through the files endpoint if you have **Viewer** permission or higher on the dashboard or its parent folder. This allows you to:

- List files in provisioned folders and branches
- Read dashboard JSON content
- View folder structure and organization

### Edit dashboard source code

You can modify dashboard source code through the files endpoint if you have **Editor** or **Admin** permission on the dashboard or its parent folder. This allows you to:

- Create new dashboard files
- Update existing dashboard content
- Delete dashboards
- Modify folder structure

When you save changes through this endpoint, Git Sync commits those changes to the Git repository (or creates pull requests if branch protection is enabled), subject to Git repository permissions.

For detailed information about configuring folder and dashboard permissions in Grafana, refer to [Git Sync permissions and access control](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/permissions-grafana).

## Control access to your Git repository

Access control at the Git provider level determines who can view and modify your dashboard source code. Configure repository access based on your security and compliance requirements.

### Read access (repository visibility)

Read access controls who can view the dashboard source code stored in your repository. This includes dashboard JSON files, folder structure, and any other files in the repository.

**Public repositories**: Anyone can view the repository contents, including dashboard configurations and any data or queries they contain. Only use public repositories if your dashboards contain no sensitive information.

**Private repositories**: Only authorized users can view repository contents. This protects dashboard configurations, queries, and any embedded credentials or sensitive data from public access.

For Git Sync to function, the authentication credentials configured in Grafana must have read access to pull dashboard changes from Git to Grafana.

### Write access (push permissions)

Write access controls who can push changes to your repository. This determines who can modify dashboard source code, either directly or through pull requests.

**Direct write access**: Users with write permission can push commits directly to branches. For Git Sync to push dashboard changes from Grafana to Git, the authentication credentials must have write access to the repository.

**Protected branches**: Branch protection rules can restrict direct writes and require changes to go through pull requests with review and approval, even for users with write access.

Refer to the [Git Sync setup documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup) for detailed instructions on configuring authentication for your Git provider.

## Control how changes are written in Git Sync

Git Sync supports different modes for writing dashboard changes to your repository, from allowing direct commits to requiring formal review processes.

### Read-only mode

Configure the repository as read-only in Grafana to prevent any writes to Git from the Grafana UI. Dashboards sync from Git to Grafana, but users cannot save changes back to Git.

**Use when**: Git is the single source of truth and all changes must be made through direct Git commits or CI/CD processes.

### Direct commit mode

When the repository allows writes and branch protection is not enabled, Git Sync commits dashboard changes directly to the configured branch without review.

**Use when**: Rapid iteration is needed and changes don't require formal review, such as in development environments.

### Pull request mode

When branch protection is enabled at your Git provider, Git Sync prompts you to go create pull requests in your Git repository instead of committing directly. Changes require review and approval before merging to the main branch.

**Use when**: Changes require review and approval, such as in production environments or when multiple teams collaborate on dashboards.

## Protect your branch

Branch protection rules at your Git provider enforce how changes are made to specific branches. When enabled on the branch that Git Sync targets, these rules require Git Sync to create pull requests instead of pushing commits directly.

**Common use cases for branch protection**:

- Production environments requiring change approval
- Compliance requirements for audit trails and review
- Multi-team environments where changes need visibility

Branch protection can enforce various controls such as requiring pull requests before merging, setting reviewer approval requirements, running automated validation checks, preventing force pushes, and restricting who can push directly to protected branches.

Refer to your Git provider's documentation for specific instructions on configuring branch protection rules.

## Code review assignments

Many Git providers support `CODEOWNERS` files that automatically assign reviewers to pull requests based on which files are changed. When Git Sync creates a pull request, the Git provider can use the `CODEOWNERS` file to assign the appropriate team or users for review.

This ensures dashboard changes are reviewed by the teams responsible for those dashboards, based on folder path or file patterns.

Refer to your Git provider's documentation for instructions on configuring `CODEOWNERS` files.

## Troubleshooting

For permissions troubleshooting refer to [Troubleshoot permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/permissions-grafana#troubleshoot-permissions) in the Git Sync permissions documentation.

# Git Sync permissions and access control

For Git Sync to work correctly you need to configure permissions at two layers:

- At the Grafana level for repository management and resource access, as described in this document.
- At your Git provider level, to protect your repository. Refer to [Repository protection for Git Sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/permissions-git) for more information.

## Grafana role-based permissions

Git Sync integrates with the Grafana standard role-based permission model, which has three levels:

1. **Organization-level:** Default permissions of the `Admin`, `Editor`, or `Viewer` role. Refer to [Roles and permissions](ref:roles-and-permissions) for more details.
2. **Folder-level:** They also apply to all dashboards within it. Refer to [Folder permissions](ref:manage-folder-permissions) for more details.
3. **Dashboard-level:** Refer to [Dashboard permissions](ref:manage-dashboard-permissions) for more details.

{{< admonition type="note" >}}

Dashboard-level permissions override folder-level permissions, which override organization-level roles.

{{< /admonition >}}

The following applies for Git Sync:

- **Permissions don't sync**: Folder and dashboard permissions are managed in Grafana only and don't sync to Git. You must configure permissions separately in each Grafana instance that uses the repository
- **Dashboard changes**: When users with appropriate dashboard or folder permissions modify dashboard content, those changes automatically sync to Git, or create pull requests if branch protection is enabled
- **Folder structure**: Creating, renaming, or deleting folders syncs to Git

## Use org-level permissions in Git Sync

If you apply org-level permissions, users can do the following with Git Sync:

### Admin users

If Grafana OSS/Enterprise, you need to have the `Admin` role to set up and manage Git Sync repositories and connections. In Grafana Cloud, the equivalent role is **Grafana Cloud Admin** or **Admin** at the organization level.

As an Admin you can:

- Configure new Git Sync repositories and connections
- Update repository settings (URL, branch, path, sync interval, webhook configuration)
- Delete repository connections
- Manage authentication credentials and Git provider connections
- Manually trigger sync operations (pull from Git)
- View sync status, logs, and statistics
- Access the Provisioning admin UI at **Administration > General > Provisioning**

### Editor users

Users with the `Editor` role can work with provisioned dashboards and folders. Their specific capabilities depend on the folder-level and dashboard-level permissions assigned to them.

**Organization-level capabilities**:

- View dashboard preview links in pull requests
- Push their dashboard and folder changes to Git, including opening pull requests, via the jobs API

{{< admonition type="note" >}}
**Only Admins can trigger a manual sync (pull from Git) and orphan-resource clean-up**. Editors can push their own changes but can't pull from Git on demand. Refer to [Job actions and required permissions](#job-actions-and-required-permissions) for more details.
{{< /admonition >}}

**Resource access** depends on folder/dashboard permissions:

- **Folder Editor or Admin**: Create, edit, and delete dashboards within the folder; create subfolders; changes sync to Git
- **Folder Viewer**: View dashboards only within that folder
- **Dashboard Editor or Admin**: Edit specific dashboards; changes sync to Git (even without folder edit access)
- **Dashboard Viewer**: View specific dashboards only

Editors don't need access to the Provisioning admin UI or repository configuration. Refer to [Configure folder and dashboard permissions](#configure-folder-and-dashboard-permissions) and [Configure fine-grained access control (RBAC)](#configure-fine-grained-access-control-rbac) for details.

### Viewer users

Users with the `Viewer` role can view provisioned resources. Their access to specific dashboards and folders depends on the permissions assigned to them.

**Organization-level capabilities**:

- Read Git Sync settings, via the `provisioning.settings:read` permission
- View dashboard preview links in pull requests

**Resource access** depends on folder and dashboard permissions:

- **Folder Viewer**: View all dashboards and subfolders within that folder
- **Dashboard Viewer**: View specific dashboards (even if they don't have folder access)
- Cannot edit dashboards or manage Git Sync repositories

## Configure folder and dashboard permissions

Folder-level role permissions determine who can view, edit, or delete provisioned resources.

These roles grant Grafana permissions (`dashboards:read`, `dashboards:write`, `folders:create`...) that are checked when users interact with provisioned resources through the Git Sync files endpoint. Dashboards within a provisioned folder inherit the folder's permissions.

When Git Sync creates a provisioned folder, it assigns these default permissions:

| Grafana Role | Folder Permission |
| ------------ | ----------------- |
| Admin        | Admin             |
| Editor       | Editor            |
| Viewer       | Viewer            |

**Folder-level `Viewer` users**:

- View dashboards and folders
- Cannot create, edit, or delete resources

**Folder-level `Editor` users**:

- Have all `Viewer` permissions
- Create, edit and delete dashboards
- Create subfolders
- When an `Editor` saves dashboard changes, Git Sync automatically commits the changes to Git, or creates a pull request if branch protection is enabled

**Folder-level `Admin` users**:

- Have all `Editor` permissions
- Update folder settings, rename and delete folders
- Modify folder permissions
- Full control over the folder and all its contents

### Modify folder-level permissions

Folder permissions in Grafana are attached to a folder's **UID**, not to its path in the Git repository. Git Sync stores this UID in the folder's `_folder.json` metadata file (in the `metadata.name` field). For the metadata file schema and why it exists, refer to [The Git Sync folder metadata file](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/use-git-sync/#the-git-sync-folder-metadata-file).

Because permissions reference the folder UID, keep the following in mind:

- **Create the folders first.** A folder must exist in Grafana—and have a `_folder.json` file with a stable UID—before you can assign permissions to it. Let Git Sync create and sync the folders, then apply the permissions that reference the resulting UIDs. If you try to set a permission for a folder that hasn't been synced yet, there's no UID to attach it to.
- **Permissions don't sync to Git.** You set them per Grafana instance. Every instance that syncs the same repository shares the same folder UIDs (from `_folder.json`), so you can apply the same permission definitions to each instance.

{{< admonition type="note" >}}
To safely modify permissions, each provisioned folder must include a `_folder.json` metadata file with the folder's UID. Without this file, the folder's UID is derived from its repository path, so folder permissions can be lost if the folder is moved to a different path in the Git repository.

For folders created from the Grafana UI, the metadata file is added automatically. If your folder is missing the metadata file, the UI shows a warning with instructions on how to add it. Refer to [The Git Sync folder metadata file](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/use-git-sync/#the-git-sync-folder-metadata-file) for details about this file and why it exists.
{{< /admonition >}}

You can set folder permissions manually or as code. In both cases, apply them only after Git Sync has created the folders.

#### Set permissions manually

- **Grafana UI**: Navigate to the folder and select **Folder actions > Manage permissions**. Add or edit permissions for roles, teams, or users.
- **HTTP API**: Use the [Folder/Dashboard permissions API](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/dashboard_permissions/). Reference the folder by the UID stored in its `_folder.json` file.
- **RBAC (Enterprise/Cloud)**: Use [Role-Based Access Control](ref:rbac) for fine-grained permission management.

#### Set permissions with Terraform

You can manage folder permissions as code with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). Because Git Sync—not Terraform—creates the folders, reference each folder by the UID from its `_folder.json` file (the `metadata.name` value). Apply the permissions only after Git Sync has synced the folders, so the folder UIDs already exist.

Use [`grafana_folder_permission_item`](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/folder_permission_item) to manage a single permission entry. This is additive, so it only manages the entries you declare and leaves other permissions untouched:

```terraform
# The UID comes from the folder's _folder.json metadata file ("metadata.name").
locals {
  team_platform_folder_uid = "team-platform-abc123"
}

resource "grafana_folder_permission_item" "team_platform_editor" {
  folder_uid = local.team_platform_folder_uid
  team       = grafana_team.platform.id # an existing team
  permission = "Edit"
}

resource "grafana_folder_permission_item" "team_platform_viewer" {
  folder_uid = local.team_platform_folder_uid
  role       = "Viewer"
  permission = "View"
}
```

Alternatively, use [`grafana_folder_permission`](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/folder_permission) to declare the full set of permissions for a folder in a single resource. This resource is authoritative: any permission that isn't listed is removed when you apply the configuration.

{{< admonition type="note" >}}
Don't manage these folders with the `grafana_folder` resource—they're owned by Git Sync. To avoid hardcoding UIDs, read them from the `_folder.json` files in your repository, or look them up with a [`grafana_folder` data source](https://registry.terraform.io/providers/grafana/grafana/latest/docs/data-sources/folder).
{{< /admonition >}}

## Configure fine-grained access control (RBAC)

If you're a Grafana Enterprise or Grafana Cloud user with RBAC enabled, Git Sync provides fine-grained permissions that allow more granular control over Git Sync operations. You can create custom roles that combine Git Sync permissions in different ways. Refer to [Role-Based Access Control](ref:rbac) documentation for instructions on creating and managing custom roles.

### How basic roles map to RBAC permissions

Understanding which permissions each basic role receives helps you create custom roles or understand exactly what access users have. The tables below show which Git Sync permissions are granted to each role by default.

{{< admonition type="note" >}}
The `provisioning.settings:read` permission is granted to all roles (Viewer and above) and allows viewing Git Sync system settings, which is necessary for the UI to display properly. This does not grant access to modify settings or manage repositories.
{{< /admonition >}}

#### Admin role

Users with the `Admin` role receive full access to Git Sync infrastructure:

| Permission Category      | Specific Permissions                                                                                                                              | What This Allows                                                                                                                        |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **Repositories**         | `provisioning.repositories:create`<br>`provisioning.repositories:read`<br>`provisioning.repositories:write`<br>`provisioning.repositories:delete` | Create new repositories<br>View repository configurations<br>Update repository settings (branch, path, interval)<br>Delete repositories |
| **Connections**          | `provisioning.connections:create`<br>`provisioning.connections:read`<br>`provisioning.connections:write`<br>`provisioning.connections:delete`     | Create Git provider connections<br>View connection details<br>Update connection settings<br>Delete connections                          |
| **Jobs**                 | `provisioning.jobs:create`<br>`provisioning.jobs:read`<br>`provisioning.jobs:write`<br>`provisioning.jobs:delete`                                 | Trigger manual syncs<br>View sync jobs<br>Modify sync job settings<br>Cancel/delete sync jobs                                           |
| **History & Monitoring** | `provisioning.historicjobs:read`<br>`provisioning.stats:read`                                                                                     | View sync job history<br>View Git Sync statistics and metrics                                                                           |
| **Settings**             | `provisioning.settings:read`                                                                                                                      | View Git Sync system settings                                                                                                           |

#### Editor role

Users with the `Editor` role can manage sync operations but not infrastructure configuration:

| Permission Category  | Specific Permissions                                                                                              | What This Allows                                                                                             |
| -------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| **Jobs**             | `provisioning.jobs:create`<br>`provisioning.jobs:read`<br>`provisioning.jobs:write`<br>`provisioning.jobs:delete` | Push changes and open pull requests<br>View sync jobs<br>Modify sync job settings<br>Cancel/delete sync jobs |
| **Read-Only Access** | `provisioning.repositories:read`<br>`provisioning.settings:read`                                                  | View repository configurations<br>View Git Sync settings                                                     |

{{< admonition type="note" >}}
`provisioning.jobs:create` allows editors to create jobs to push their changes, subject to the relevant `dashboards:*` and `folders:*` permissions. However, **it doesn't allow admin-only job actions** such as a manual **pull** from Git or orphan-resource cleanup, which require the `provisioning.repositories:write` permission. Refer to [Job actions and required permissions](#job-actions-and-required-permissions) for more details.
{{< /admonition >}}

`Editors` can access resources based on the folder/dashboard assignments:

- `dashboards:create`, `dashboards:read`, `dashboards:write`, `dashboards:delete` - On folders/dashboards where assigned Folder Editor or Dashboard Editor
- `folders:create`, `folders:read`, `folders:write`, `folders:delete` - On folders where assigned Folder Editor

#### Viewer role

Users with Viewer role have read-only access to Git Sync:

| Permission Category  | Specific Permissions                                             | What This Allows                                                           |
| -------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------- |
| **Read-Only Access** | `provisioning.repositories:read`<br>`provisioning.settings:read` | View repository configurations<br>View Git Sync settings (required for UI) |

`Viewers` can access resources based on the folder/dashboard assignments:

- `dashboards:read` - On folders/dashboards where assigned Folder Viewer or Dashboard Viewer
- `folders:read` - On folders where assigned Folder Viewer

#### None role

The `None` basic role, also called **No basic role** in the UI, doesn't have any Git Sync permissions by default, as it doesn't inherit the default permissions of Viewer, Editor, or Admin. Without explicit grants, as a `None` user you can't read Git Sync settings or repository configurations, configure repositories, submit jobs, or read and modify provisioned dashboards and folders. In any case, you can use RBAC to assign specific permissions to your `None` users.

- **Resource access:** Refer to [RBAC for dashboards and folders](#rbac-for-dashboards-and-folders).
- **Job access:** Refer to [Job actions and required permissions](#job-actions-and-required-permissions).
- **Repository subresource access:** Refer to [Repository subresource access](#repository-subresource-access).

### RBAC for dashboards and folders

Provisioned dashboards and folders use the Grafana standard permission model. To modify provisioned resources, you will need permissions over your dashboard and folder.

The following applies for Git Sync:

- You need standard `dashboards:*` and `folders:*` permissions to work with provisioned resources
- `Editors` and `Viewers` need `provisioning.settings:read` and `provisioning.repositories:read` to view Git Sync configuration
- You do **not** need repository write/delete or connection permissions to edit dashboards
- Dashboard-level permissions override folder-level permissions
- Changes made by users with appropriate permissions automatically sync to Git

### Job actions and required permissions

Git Sync operations run as jobs. You need the `provisioning.jobs:create` permission to create any job. Moreover, **some job actions are restricted to administrators** and require the `provisioning.repositories:write` permission. This prevents editors from triggering repository-wide operations even though they hold `provisioning.jobs:create`.

| Job action                          | Required permission                                                                   | Who can run it                                            |
| ----------------------------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| Push changes / open pull request    | `provisioning.jobs:create` plus the relevant `dashboards:*` / `folders:*` permissions | Editors and Admins, on the resources they can modify      |
| Migrate resources                   | `provisioning.jobs:create` plus read/write on the affected resource types             | Editors and Admins with the required resource permissions |
| Manual sync (pull from Git)         | `provisioning.repositories:write`                                                     | Admins only                                               |
| Release / delete orphaned resources | `provisioning.repositories:write`                                                     | Admins only                                               |

### Repository subresource access

The repository API exposes several subresources. The following table shows the permission each one is gated on.

The `refs` subresource lists the repository's branches and commits, and two distinct flows legitimately need it:

- **Editors pushing changes**: When an editor saves changes or opens a pull request through the push-job flow, Grafana needs the branch list. This is authorized with `provisioning.jobs:create`.
- **Admins configuring a repository**: When an admin or repository owner sets up or edits a repository and picks a target branch, Grafana updates the `Repository` resource info. This is authorized with `provisioning.repositories:write`.

Because the `repositories` resource has no Editor tier (`repositories:read` is granted to Viewer and above, while `write`, `create`, and `delete` are admin-only), `refs` accepts either of these checks, and viewers satisfy neither.

| Subresource                      | Purpose                                    | Required permission                                                     | Who can access it       |
| -------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------- | ----------------------- |
| `files`                          | Read and write provisioned resource files  | Authenticated access, then standard `dashboards:*` / `folders:*` checks | All authenticated users |
| `refs`                           | List repository branches and commits       | `provisioning.repositories:write` **or** `provisioning.jobs:create`     | Admins and Editors      |
| `resources`, `history`, `status` | Repository management and inspection views | `provisioning.repositories:write`                                       | Admins only             |

{{< admonition type="note" >}}
The management and inspection views (`resources`, `history`, `status`) are gated on `provisioning.repositories:write` rather than `repositories:read`, because there's no admin-only read action on the `repositories` resource. As a result, only users who can manage a repository can inspect its management views; Viewers and Editors can't access these views.
{{< /admonition >}}

## Configure Git repository protection

After you've configured your Grafana permissions, set up the appropriate permissions at your Git provider to write changes. Repository protection settings control write access, branch protection rules, and code review requirements.

For detailed information about configuring repository write access and branch protection, refer to [Repository protection for Git Sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/permissions-git).

## Troubleshoot permissions

### "Permission denied" when saving a dashboard

**Cause**: User lacks **Editor** or **Admin** permission on the provisioned folder.

**Solution**:

1. Verify the user's folder-level permissions in Grafana
1. Navigate to **Folder settings > Permissions**
1. Grant the user or their team **Editor** or **Admin** role

### Git Sync fails with "403 Forbidden" or "Unauthorized"

**Cause**: The Git provider authentication credentials lack the required repository permissions.

**Solution**:

1. Verify the authentication credentials (GitHub App, Personal Access Token) have **read and write** permissions on the repository
1. If using a GitHub App or OAuth app, verify it is installed and authorized for the target repository
1. Check that the credentials have permission to create pull requests (if branch protection is enabled)
1. Verify authentication credentials haven't expired
1. For expired or revoked tokens, generate new credentials and update the Git Sync connection configuration

### Dashboard changes commit directly instead of creating pull requests

**Cause**: Branch protection is not configured at the Git provider.

**Solution**: Enable branch protection rules at your Git provider to enforce pull request workflows or in your Grafana repository settings. Refer to your Git provider's documentation for instructions on configuring branch protection.

### Dashboard changes commit directly without review

**Cause**: Branch protection is not configured at the Git provider.

**Solution**:

1. Enable branch protection on the target branch at your Git provider
1. Configure the branch to require pull requests before merging
1. Verify the branch name in protection rules matches the branch configured in Grafana

### Pull requests not created when expected

**Cause**: Branch protection is not enabled or the authentication credentials lack pull request creation permission.

**Solution**:

1. Verify branch protection is enabled on the correct branch
1. Check that the credentials have permission to create pull requests
1. Ensure the branch name in Git Sync settings matches the protected branch exactly

# Work with provisioned dashboards in Git Sync

{{< admonition type="note" >}}

Git Sync functionalities are constantly evolving. [Contact Grafana](https://grafana.com/help/) for support or to report any issues you encounter and help us improve this feature.

{{< /admonition >}}

Using provisioning, you can choose to store your dashboard JSON files in either Git repositories using Git Sync or a local path, and manage them through the Grafana interface. Dashboards and folders synchronized using Git Sync or a local path are referred to as _provisioned_ resources. **Git Sync is the recommended method for provisioning your dashboards**.

## Manage dashboards provisioned with Git Sync

Using Git Sync, you can manage your dashboards in the UI and synchronize them with your configured Git repository. If you push a change in the repository, those changes are mirrored in your Grafana instance.

- Dashboards saved in your repository or local folder configured with Git Sync appear in a provisioned folder in Grafana.
- Any dashboard folders saved with Git Sync have a **Provisioned** label in the UI.
- To save any changes to provisioned resources, open a pull request or commit directly to an existing branch, such as the `main` branch.
  - Use pull requests to review changes to dashboards.
  - Preview the changes before merging.

Refer to [Key concepts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/key-concepts) for more details.

## Create a new dashboard

{{< admonition type="note" >}}

It may take a few minutes for your changes to reflect on your screen. If they don't, refresh the UI manually.

{{< /admonition >}}

You have the following options to add new dashboards in your Git Sync connection:

- Create them directly within Git Sync-managed folders in the Grafana UI
- Add them by committing JSON files to your Git repository

If you want to add an existing dashboard to your provisioned resources, refer to [Add non-provisioned resources from Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/add-resources/).

### Add a new dashboard from the Grafana UI

To add a new dashboard from the Grafana UI:

1. On the **Dashboards** tab, click on **New** on the top right corner and chose one of these options:
   - **New dashboard**. When you create a new dashboard in a provisioned folder associated with a Git repository, you follow the same process you use for any new dashboard. Refer to [Create a dashboard](http://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/create-dashboard/) for more information.
   - **Import dashboard**. Refer to [Add a dashboard using Import dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/add-resources/dashboards-copy/#add-a-dashboard-using-import-dashboards) for more details.
   - **Use template**. Create a dashboard using a Grafana template.

After you create the dashboard, follow these steps:

1. Click **Save dashboard**.
1. On the **Provisioned dashboard** panel, choose the options you want to use:
   - **Title**: The title of the dashboard.
   - **Description**: Description of the dashboard.
   - **Target folder**: The folder where you want to store the new dashboard.
   - **Branch**: Specify the branch name in your Git provider (for example, main).
   - **Repository folder**: The folder inside the repository. Leave empty for the repository root.
   - **Filename**: The name of the dashboard file.
   - **Comment**: Add a comment describing your changes for you commit.
1. Optional: Select the **Changes** tab to view the differences between the updates you made and the original resource.
1. Select **Save**, and the updated dashboard will load.
1. On the prompt, select **Open a pull request in GitHub** to open a new PR to your repository.
1. Follow your usual Git workflow to save and merge the PR to your repository.

## Edit dashboards

When you edit a provisioned resource, you're prompted to save or discard those changes. Saving changes requires opening a pull request in your Git repository.

To save dashboard changes:

1. Select **Edit** to update a provisioned dashboard.
1. Make your desired changes, and click **Save dashboard**.
1. On the **Provisioned dashboard** panel, choose the options you want to use:
   - **Branch**: Specify the branch name in your Git provider (for example, main).
   - **Repository folder**: The folder inside the repository. Leave empty for the repository root.
   - **Filename**: The name of the dashboard file.
   - **Comment**: Add a comment describing your changes for your commit.
1. Optional: Select the **Changes** tab to view the differences between the updates you made and the original resource.
1. Select **Save**, and the updated dashboard will load.
1. On the prompt, select **Open a pull request in GitHub** to open a new PR to your repository.
1. Follow your usual Git workflow to save and merge the PR to your repository.

## Preview banner

This section applies to Git repository connections (GitHub, GitHub Enterprise, GitLab, Bitbucket, and plain Git). It doesn't apply to local file provisioning, which has no branches or pull requests.

Whenever you're viewing a provisioned dashboard loaded from a branch other than your repository's configured branch, Grafana shows a **preview banner** at the top of the page. You'll see it right after you create, edit, delete, or move a dashboard from the Grafana UI, as well as when you open a dashboard from a pull request notification.

The banner's title reflects what changed:

- **A new resource has been created in a branch in your Git provider.**
- **A resource has been updated in a branch in your Git provider.**
- **A resource has been deleted in a branch in your Git provider.**
- **A resource has been moved in a branch in your Git provider.**

In every case, the rest of your organization keeps seeing the version on the configured branch until the branch is merged — what you're looking at is a private preview of your own change. The banner also shows the branch your change was pushed to and the branch it targets.

From the banner you can:

- Select **Open a pull request** (or **View pull request**, if one already exists) to open the pull request for this change, in a new tab.
- Select **View saved version**, when editing a dashboard that already exists in Grafana, to jump back to the version currently saved on the configured branch, so you can compare it against your draft.

If the branch behind the preview no longer exists in the repository, for example, the pull request was already merged or closed and its source branch deleted, selecting **Open a pull request** can't take you to a live pull request. Grafana instead shows **This branch no longer exists** and offers two ways forward:

- **Save to a new branch**: Reopens the save form with your draft intact, so you can commit it to a new branch and open a new pull request.
- **View the current version**: Navigates to the dashboard as currently saved in Grafana, if it exists.

{{< admonition type="note" >}}

How much of the banner is interactive depends on your repository type:

- **GitHub, GitHub Enterprise, GitLab, and Bitbucket** (enhanced integrations): the source and target branch names link directly to that branch in your Git provider, and the pull request action opens (or creates) a real pull request.
- **Plain Git** connections: branch names are shown, but since plain Git has no pull-request concept or per-branch view, the pull request action opens the repository itself instead.

Provisioned folders show a related banner, **This resource is behind the branch**, when the folder itself is out of sync with its configured branch. Select **Open in repository** to view the latest changes.

{{< /admonition >}}

## Remove dashboards

You can remove a provisioned dashboard either from the Grafana UI or from the repository:

- **From the Grafana UI**: Open the dashboard, go to its settings, and select **Delete**. On the **Provisioned dashboard** panel, choose the branch, comment, and other options the same way you would when [editing a dashboard](#edit-dashboards), then select **Delete**. On the prompt, select **Open a pull request** and follow your usual Git workflow to review and merge the change.
- **From the repository**: Delete the dashboard's file directly in Git. The Grafana UI updates when the changes from the Git repository sync.

To restore a deleted dashboard, revert the deletion (or re-add the file) directly in your Git repository and raise a PR as usual. Restoring resources from the UI isn't possible at the moment.

## Document folders with a README

You can document the contents or any other relevant piece of information of your provisioned folder in a `README.md` file stored alongside its resources in the repository. Grafana renders the README inline on the folder page, below the list of dashboards, so your team can see what's in the folder, how it's organized, and where to find the right dashboard without leaving Grafana.

Common uses include:

- **Runbooks**: Document how to respond to an alert or investigate an incident, with links to the dashboards teammates need.
- **Architectural diagrams**: Show how the system a folder monitors fits together, using [Mermaid](#render-mermaid-diagrams) diagrams that render inline.
- **Onboarding guidelines**: Explain what each dashboard is for and how a new team member should navigate the folder.

- When the folder contains a `README.md` file, Grafana renders its Markdown content. Relative links and images in the README resolve against the host repository.
- If the folder has no `README.md`, or if the folder is empty, you're prompted to action with an **Add README** button.
- You can edit the README any time. Select the edit pencil in the README header to open the file in your Git provider's editor and commit changes through your usual workflow. When you have several documents, the edit pencil targets the document that's currently open.

{{< admonition type="note" >}}

It may take a few minutes for your changes to reflect on your screen. If they don't, refresh the UI manually.

{{< /admonition >}}

### Browse multiple documents with tabs

A folder can hold more than one Markdown document. When it does, Grafana shows a tab for each document above the rendered content, similar to how a Git provider presents documentation files.

- Grafana lists any `.md` and `.markdown` files stored alongside the folder's resources.
- The `README`, `CONTRIBUTING.md`, and `SECURITY.md` files appear first, in that order, when they're present. All other documents follow in alphabetical order.
- The README renders by default. Select a tab to load that document.
- When there are more tabs than fit the width of the page, the extra tabs collapse into a **More** menu.

Use this to keep related documentation next to the dashboards it describes, such as a runbook, an onboarding guide, or contribution and ownership details in separate documents.

### Navigate to resources from Markdown links

Markdown links are context-aware. A link that points to a provisioned resource file, such as a dashboard JSON file or a folder, opens the matching page inside Grafana when you select it. This lets you move between documentation and dashboards the same way you move between provisioned resources.

When a link doesn't match a provisioned resource, Grafana opens the file in the host repository, so the same README still works when viewed on GitHub or GitLab.

### Render Mermaid diagrams

Grafana renders [Mermaid](https://mermaid.js.org/) diagrams defined in your Markdown documents. Use them to embed architectural diagrams, incident runbook flows, or data pipeline overviews next to the dashboards that monitor them. Use a fenced code block with the `mermaid` language to include a diagram:

````markdown
```mermaid
flowchart LR
  A[Ingest] --> B[Store]
  B --> C[Query]
  C --> D[Dashboard]
```
````

Grafana replaces the code block with the rendered diagram and matches it to your light or dark theme.

## Best practices

Follow these recommendations when working with provisioned dashboards:

- **Use GitHub pull requests for changes**: Maintain review processes for dashboard modifications.
- **Provide clear commit messages**: Describe your changes to help with tracking and collaboration.
- **Regularly sync your repository**: Keep Grafana up to date with the latest changes.
- **Review the Events tab**: Monitor sync status to ensure changes are applied correctly.
- **Add a folder README**: Document each folder's contents with a `README.md` so your teammates can find the right dashboard quickly. Use additional documents for runbooks, architectural diagrams, or onboarding guidelines that belong next to the dashboards.

Refer to [Work with provisioned repositories](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/use-git-sync) for general guidance about using Git Sync.

# Usage and performance limitations

{{< admonition type="note" >}}

Git Sync functionalities are constantly evolving. [Contact Grafana](https://grafana.com/help/) for support or to report any issues you encounter and help us improve this feature.

{{< /admonition >}}

## Performance considerations

When Git Sync is enabled, the database load might increase, especially if your Grafana instance has many folders and nested folders. Evaluate the performance impact, if any, in a non-production environment.

## Usage tiers and limits

The following Git Sync per-tier limits apply:

| Tier                                      | **Cloud - Free** | **Cloud - Other** | **On-prem OSS** | **On-prem Enterprise** |
| ----------------------------------------- | ---------------- | ----------------- | --------------- | ---------------------- |
| Amount of repositories                    | 1                | 10                | 10 (default)    | 10 (default)           |
| Amount of synced resources per repository | 20               | 1,000             | No limit        | No limit               |

{{< admonition type="note" >}}

On Grafana Cloud, tier limits can't be changed from configuration. Contact Support if you need a higher connection limit, or [shard by capacity](#shard-by-capacity-not-by-team) instead of using a single connection per team or microservice.

On self-managed Grafana (OSS or Enterprise), the repository count of **10** in the table is the **default**, not a hard ceiling. Raise or remove it with `[provisioning] max_repositories` in the Grafana configuration file. On-prem resources per repository stay unlimited by default (`max_resources_per_repository = 0`), and you can set a cap if you want one.

Refer to [Modify your usage limits](#modify-your-usage-limits) for more details.

{{< /admonition >}}

### Recommended limits

On Grafana Cloud, the repository-connection limit is a per-stack limit. On self-managed Grafana, the effective limit is whatever you set for `max_repositories` (default `10`).

**Do not sync more than 1,000 resources per repository connection as of today.** This isn't an arbitrary cap: beyond roughly 1,000 resources per connection, the sync workflow puts noticeable load on Grafana itself, which may result in slower syncs and increased database load.

Both of these limits are early figures. As Git Sync matures, these limits will raise by orders of magnitude. The goal is for Git Sync to support around 100 repository connections per stack, and up to roughly 1,000 in the longer term, with similar increases to the number of resources you can sync per connection.

If these limits are affecting your use of Git Sync, [get in touch](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/contribute/#communicate-with-grafana) and explain your situation. You may also share any ideas or suggestions.

For details on usage and storage limits, refer to [Dashboard and folder limits](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/manage-invoices/understand-your-invoice/usage-limits/#other-usage-limits).

### Modify your usage limits

Before changing your usage limits, study your specific use case. Design the repository structure carefully, and determine how many repositories and how many resources you can support. For example, setting over 1,000 resources per repository may impact your system's performance.

In many cases, splitting a single repository across multiple connections, as described in [Scale beyond 1,000 resources per repository](#scale-beyond-1000-resources-per-repository), is a better option than raising the limits.

#### Grafana Cloud

On Grafana Cloud, limits are enforced per tier and can't be edited from configuration. The 10-connection limit can be increased slightly on request, but splitting a single repository across multiple connections is the recommended way to sync more resources without changing tier limits.

Limit increases aren't granted automatically. When you request one, Support will ask you to describe your use case to understand why the current limits aren't enough, and assess whether the increase is necessary and safe for your stack's performance.

#### Grafana OSS/Enterprise

In Grafana On-prem (OSS or Enterprise), the table value of **10** repositories is only the default, not a hard ceiling. Resources per repository remain unlimited by default. The 1,000-resources-per-connection figure is a performance recommendation, not an on-prem default cap.

If you're an on-prem user you can change the limits directly through configuration settings, as described below.

To customize the limits use one of these configuration settings:

- Use `max_repositories` to set how many repositories you can sync. Default is `10`. Set to `0` for unlimited repositories. Refer to [`max_repositories`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#max_repositories) in the Configure Grafana section to learn more.
- Use `max_resources_per_repository` to set the amount of resources per repository to sync. Default is `0` (unlimited). Refer to [`max_resources_per_repository`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#max_resources_per_repository) in the Configure Grafana section to learn more.

### Scale beyond 1,000 resources per repository

If a single repository holds more than 1,000 resources, you don't have to raise the per-repository limit. Instead, connect the same repository multiple times, with each connection pointing to a different folder (path) in the repository. Each connection syncs its own subset of resources and counts toward the 1,000-resource recommendation independently.

Because a stack allows up to 10 repository connections, this lets you sync up to roughly 10,000 resources from a single repository while keeping each connection within the recommended range.

When you split a single repository across several connections, use [folderless sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/key-concepts#sync-targets) for each connection. With folder sync (the default), every connection creates its own wrapper folder named after the repository, so splitting the repository would change your folder hierarchy and nest resources one level deeper. Folderless sync places each connection's resources at the top level, so the split has no effect on how your dashboards and folders are organized in Grafana.

If you need more than that, the 10-connection limit can be increased slightly for Cloud stacks on request. Contact Support to discuss your use case.

On Grafana Cloud, syncing this many resources may also exceed your stack's maximum number of dashboards. If it does, you'll need to increase that limit as well. Refer to [Dashboard and folder limits](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/manage-invoices/understand-your-invoice/usage-limits/#other-usage-limits) to review and adjust your dashboard limits.

#### Shard by capacity, not by team

When you have many teams or tenants, it's tempting to create one connection per team so each team maps to its own connection. Avoid this: it consumes connections quickly, doesn't scale as teams grow, and on Grafana Cloud a single stack can't be granted the hundreds of connections this would require.

Instead, shard by capacity. Create one repository and group teams into a small number of shard folders, each holding up to about 1,000 resources, and connect each shard folder separately. For example, a customer with 190 teams and 900 resources fits comfortably in a single shard today:

```
your-org/grafana-manifests/
├── shard-1/        ← ~900 resources today, connected now
│   ├── team-a/
│   ├── team-b/
│   └── ...
├── shard-2/        ← empty for now, add a connection when shard-1 approaches 1,000
├── shard-3/        ← add later as you keep growing
└── shard-4/        ← ...up to 10 shards / connections per stack
```

As the number of resources grows, add `shard-2`, `shard-3`, and later shards, and connect each one. You can move teams between shards at any time to balance the load, so you only pay for the connections you actually need and can grow up to the 10-connection limit without restructuring your repository.

If sharding isn't practical for your setup, try raising the per-connection resource limit modestly, for example from 1,000 to a limit in the 1,200 - 1,500 range. This is a small adjustment for a bit of extra headroom, not an order-of-magnitude increase. Still, do not go beyond 1,500 resources per connection because of the performance impact on Grafana. For substantially larger scales, sharding remains the recommended approach.

## Git Sync across multiple organizations

Git Sync works across multiple organizations for self-managed Grafana instances. You can set up Git Sync independently in each of your organizations, and what you sync in one organization does not affect another. Teams that share a single Grafana instance across separate organizations can each manage their own provisioning from Git.

This feature is available starting in Grafana 13.0.4, but not supported until Grafana 13.1.1.

## Compatible Git providers

Git Sync is available for any Git provider through a Pure Git repository type, and has specific enhanced integrations for GitHub, GitLab and Bitbucket.

| **Provider**                         | **Available in**       | **Authentication**                              |
| ------------------------------------ | ---------------------- | ----------------------------------------------- |
| Pure Git                             | Cloud, OSS, Enterprise | Personal Access Token or OAuth App              |
| GitHub                               | Cloud, OSS, Enterprise | Personal Access Token, GitHub App, or OAuth App |
| GitHub Enterprise (Server and Cloud) | Cloud, Enterprise      | Personal Access Token, GitHub App, or OAuth App |
| GitLab                               | Cloud, Enterprise      | Personal Access Token or OAuth App              |
| Bitbucket                            | Cloud, Enterprise      | API token with scopes or OAuth App              |

Note that Pure Git, GitLab and Bitbucket are supported in Grafana v12.4.x or later only. Refer to [Enable Git providers](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/set-up-before#enable-git-providers) to set them up. To make OAuth Apps available in self-managed Grafana, refer to [Enable OAuth connection types](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/set-up-before#enable-oauth-connection-types).

To learn more about Git, refer to [Getting Started - About Version Control](https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control) of the [Pro Git book](https://git-scm.com/book/en/v2) in the official Git documentation.

### The Pure Git repository type

The Pure Git repository type uses the [Smart HTTP protocol v2](https://git-scm.com/docs/protocol-v2) (Git over HTTPS), with no provider-specific logic. Pure Git delivers the core Git Sync workflow: your repository is the source of truth, you may edit dashboards in the UI, and Grafana stays in sync.

{{< admonition type="note" >}}

Pure Git only supports **Smart HTTP protocol v2**. Earlier protocol versions (v1, v0) and SSH transport are not supported.

Make sure your Git server supports protocol v2 over HTTPS. Some providers, like Azure DevOps, only use v1 and are therefore not compatible with Git Sync.

{{< /admonition >}}

However, Pure Git doesn't include any features that require provider APIs, such as webhook-driven instant sync, automated PR comments, or deep links to source files.

### Enhanced integrations: GitHub, GitHub Enterprise, GitLab, Bitbucket

If your Git provider is GitHub, GitHub Enterprise (Server and Cloud), GitLab, or Bitbucket, use the enhanced integration. Enhanced integrations understand the platform you're using, allowing workflows that feel native: automated pull request comments with dashboard previews, instant webhook-based sync, or direct navigation from Grafana to source files in your provider's UI.

The GitHub enhanced integration is the most feature-complete experience today. It's available in Grafana OSS, Enterprise, and Cloud, and enables richer pull request workflows, deeper linking between Grafana and GitHub, and tighter integration into review processes.

The GitLab and Bitbucket integrations are only available in Grafana Enterprise and Grafana Cloud. While they're less developed for the moment, expect continued improvements around pull request workflows, linking, and sync behavior.

## Resource support and compatibility

**Git Sync only supports dashboards and folders**. Alerts, data sources, panels and other resources are not supported yet.

If you're a Grafana Cloud user, you can check the [Grafana roadmap portal](https://grafana.ideas.aha.io/ideas) to learn about future improvements.

### Resource compatibility

If you're using Git Sync in Grafana OSS or Grafana Enterprise, some supported resources might be in an incompatible data format. If this happens, syncing will be blocked. Compatibility issues will be fixed with an upcoming migration tool.

A resource can be:

| Is the resource? | **Compatible**                                                             | **Incompatible**                                                                                |
| ---------------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| **Supported**    | The resource can be managed with Git Sync.                                 | The resource is supported but has compatibility issues. It **cannot** be managed with Git Sync. |
| **Unsupported**  | The resource is **not** supported and **cannot** be managed with Git Sync. | Not applicable.                                                                                 |

### Nested folders

Git Sync supports up to four nested folders within a repository.

## Known limitations

### Migration to Git Sync

**Full-instance sync is experimental.**

When migrating to resources to Git Sync, you can still create, edit or delete resources, but changes may not be exported. The duration of this process depends on the number of resources involved.

### Use existing resources

To add existing resources as provisioned Git Sync resources, you can use the Import dashboards tool, save them from the UI, save them as JSON files and commit them to the synced repository, or use `gcx`. Refer to [Export non-provisioned resources from Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/add-resources) for more details. When migrating existing dashboards, the folder structure will be replicated in the provisioned repository.

{{< admonition type="caution" >}}

Git Sync cannot take ownership of a resource while an unmanaged resource with the same UID exists in Grafana, so you may need to manually remove the original resources to ensure the transition to provisioned resources. Since deleting resources implies certain operational caveats, refer to [How to delete existing resources in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/add-resources/dashboards-migrate#step-2-delete-the-original-dashboards) for more information.

{{< /admonition >}}

### Restore resources

Restoring resources from the UI is currently not possible. As an alternative, you can restore dashboards directly in your GitHub repository by raising a PR, and they will be updated in Grafana.

# Work with provisioned repositories in Git Sync

{{< admonition type="note" >}}

Git Sync functionalities are constantly evolving. [Contact Grafana](https://grafana.com/help/) for support or to report any issues you encounter and help us improve this feature.

{{< /admonition >}}

After you sync your resources, Git Sync creates a dashboard that provides a summary of resources, health, pull status, webhook, sync jobs, resources, and files. To access it, follow these steps:

1. Log in to your Grafana server with an account that has the Grafana Admin flag set.
1. Select **Administration > General > Provisioning** in the left-side menu to access the Git Sync configuration screen.
1. Go to the **Repositories** tab, and locate the repository you want to work with. You can view the current status of the sync, run pulls, or update your settings.

Refer to [Work with provisioned dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/provisioned-dashboards) for more information about working with provisioned files.

## View the current status of your synchronized repository

Use the **View** section to see detailed information about the current status of your sync and [troubleshoot](#troubleshoot-synchronization) possible issues:

- The **Overview** tab contains information about the health of your repository's connection with Grafana, configuration options such as webhooks, or information on Git processes.

- The **Resources** tab lists the provisioned resources of the connection.

### Troubleshoot synchronization

{{< admonition type="note" >}}

Before you proceed to troubleshoot, understand the [Usage and performance known limitations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/usage-limits/).

{{< /admonition >}}

Monitor the **View** status page for synchronization issues and status updates. The status page displays the following events:

- **Sync started:** The synchronization process has begun.
- **Sync completed:** The synchronization process finished successfully.
- **Sync failed:** The synchronization process failed. Refer to the error details for troubleshooting.
- **Sync issues:** The synchronization process encountered issues.

**Dashboard sync errors**

- **Repository URL:** If dashboards don't sync, verify that the repository URL is correct and accessible from the Grafana instance.
- **Repository branch:** Verify that the configured repository branch exists and is correctly referenced.
- **Conflicts:** Check for conflicts in the repository that may prevent syncing.

**Dashboard import errors**

- Validate the JSON format of the dashboard files before importing.
- If the import fails, check Grafana logs for error messages and troubleshoot accordingly.

## Synchronize changes

To sync resources between the provisioned repositories and your Grafana instance, click **Pull** under the repository you want to sync. The synchronization process runs and completes.

Grafana overwrites existing dashboards with the same `uid`.

## Update or delete your settings

To update or delete your repository configuration after you complete setup:

1. Log in to your Grafana server with an account that has the Grafana Admin flag set.
1. Select **Administration > General > Provisioning**.
1. Go to the **Repositories** tab, and locate the repository you want to modify.
1. Select **Settings** to access the **Configure repository** screen:

- To modify your configuration, update any of the settings and select **Save**.
- To delete the repository, click **Delete**. You can either keep the synced resources or delete them.

{{< admonition type="note" >}}

It may take a few minutes for your changes to reflect on your screen. If they don't, refresh the UI manually.

{{< /admonition >}}

## Manage folder permissions

By default, users keep their roles in folders provisioned with Git Sync.

| Grafana Role | Folder Permission |
| ------------ | ----------------- |
| Admin        | Admin             |
| Editor       | Editor            |
| Viewer       | Viewer            |

Refer to [Git Sync permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/permissions-grafana) to understand and set up permissions in Git Sync.

### Modify folder permissions

{{< admonition type="note" >}}
To modify permissions, each provisioned folder must include the `_folder.json` metadata file, which gives the folder a stable UID. Without it, the folder's permissions are lost if you move or rename that folder in the Git repository. Refer to [The Git Sync folder metadata file](#the-git-sync-folder-metadata-file) for details about this file and why it exists.
{{< /admonition >}}

Folder permissions attach to the folder's UID (stored in `_folder.json`), not to its repository path, so you can only set them after Git Sync has created the folder. To add or modify folder permissions:

- From the UI, select **Folder actions > Manage permissions** on the top right corner.
- Using the API, refer to [Dashboard Permissions API](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/dashboard_permissions/).
- As code with Terraform, using the folder UID from `_folder.json`. Refer to [Modify folder-level permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/permissions-grafana/#modify-folder-level-permissions) for manual and Terraform examples.

### The Git Sync folder metadata file

Each provisioned folder in a synced repository contains a `_folder.json` metadata file at its root. This file is the folder's manifest: it stores the folder's stable UID and its display name, so both survive changes to the repository layout.

The file exists because a directory path isn't a stable identity. Without `_folder.json`, Grafana derives the folder UID from a hash of the directory path in the repository. If you move or rename that directory in Git, the hash changes and Grafana treats it as a different folder: everything attached to the old UID, such as custom folder permissions, bookmarks, and API references, is lost. With `_folder.json`, the UID travels with the directory, so you can reorganize your repository without breaking permissions or references.

The file also separates the folder's display name from the directory name. The directory keeps a filesystem-friendly name, for example `team-platform`, while the `spec.title` field holds the name shown in the Grafana UI, for example `Team Platform`.

Grafana manages this file for you:

- When you create a provisioned folder from the Grafana UI, Git Sync commits the `_folder.json` file together with the folder.
- When you rename a folder in Grafana, Git Sync updates the `spec.title` field in the file.
- If a folder is missing the file, for example because you created the directory directly in Git, Grafana falls back to the hash-derived UID and shows a warning in the UI with instructions on how to add the missing metadata.

The file has the following format:

```json
{
  "apiVersion": "<API_VERSION>",
  "kind": "Folder",
  "metadata": {
    "name": "<FOLDER_UID>"
  },
  "spec": {
    "title": "<FOLDER_UI_NAME>"
  }
}
```

Where:

- `<API_VERSION>` is the version of the folders API. For example, `folder.grafana.app/v1`.
- `<FOLDER_UID>` is the stable folder UID that Grafana uses for permissions, bookmarks, and API references. Treat this value as immutable: Grafana rejects folder UID changes, and changing it directly in Git breaks the link between the directory and the existing Grafana folder, with the same consequences as having no metadata file.
- `<FOLDER_UI_NAME>` is the display name shown in the Grafana UI. This field is optional. If it's not set, Grafana uses the directory name instead.

# Cherry-pick and copy individual dashboards

{{< admonition type="note" >}}

At the moment Git Sync only manages dashboards and folders. Alerts, data sources, and library panels are **not** supported yet.

To migrate your existing dashboards, refer to [Migrate existing dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/add-resources/dashboards-migrate).

{{< /admonition >}}

Use these methods to add a copy of one or more dashboards to a folder provisioned with Git Sync. Each copy is created with a new UID, so the original dashboards stay exactly as they are and nothing needs to be deleted. Existing links continue to point to the original dashboards, not the copies.

- [Add a dashboard using Import dashboards](#add-a-dashboard-using-import-dashboards)
- [Copy an existing dashboard from the Grafana UI](#copy-an-existing-dashboard-from-the-grafana-ui)

## Add a dashboard using Import dashboards

You can import dashboards directly into your Git Sync provisioned folders using the Grafana UI or the HTTP API.

![Import dashboard](/static/img/docs/ascode/gitsync-dashboards-import.png)

To access the Import dashboard tool from the Git Sync UI:

1. Go to the **Dashboards tab** of you connection.
1. On the top right corner, click **New**.
1. Select **Import dashboard** and you'll be redirected to the wizard.
1. Upload or paste the dashboard JSON.
1. Fill in the relevant fields, including the branch and repository folder, and press **Import**.
1. Open the pull request, follow your regular workflow, and merge. Note that it could take a few minutes until the imported dashboard appears.

Keep in mind the following:

- Importing an ordinary dashboard JSON creates a new dashboard with a new UID. To preserve the original UID instead (for a migration), provide a UID in the wizard or import a resource file that already sets `metadata.name`. Refer to [Migrate existing dashboards](#migrate-existing-dashboards-to-git-sync).
- UIDs are globally unique per org. Two repositories with dashboards sharing a UID will conflict.
- Two dashboards can share a title as long as they live at different paths in the repository. If a file with the same name already exists at the target path, the import is stopped before it overwrites anything.

For more information refer to [Import dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/import-dashboards/) in the Data Visualization documentation.

{{< admonition type="note" >}}

It may take a few minutes for your changes to reflect on your screen. If they don't, refresh the UI manually.

{{< /admonition >}}

## Copy an existing dashboard from the Grafana UI

You can also save a copy of dashboard directly from the Grafana UI to your provisioned folder. This creates a new dashboard with a new UID and leaves the original in place.

To do so, follow these steps:

1. Make sure the dashboard is in **Editable** mode.
1. Select **Save** or **Save as** from the top-right corner.
1. In the menu:
   - **Target folder**: Select the provisioned folder from your Grafana UI where you want to save the dashboard in.
   - **Branch**: Type in the name of the branch of the provisioned repository you want to work in, or create a new branch. Committing directly to `main` is not supported.
   - **Folder**: Type in the folder in your sync repository, if any.
   - Fill in the rest of the fields accordingly.
1. Click **Save**.
1. In your synced GitHub repository, merge the branch with the dashboard you want to sync.

# Migrate existing dashboards to Git Sync

{{< admonition type="note" >}}

At the moment Git Sync only manages dashboards and folders. Alerts, data sources, and library panels are **not** supported yet. Refer to [Before you begin](#before-you-begin) for details.

To copy a specific dashboard, refer to [Cherry-pick and copy individual dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/add-resources/dashboards-copy).

{{< /admonition >}}

If you migrate your existing non-provisioned dashboards under Git Sync, dashboards keep their UIDs, so existing links, references, and bookmarks keep working. Because the UID is preserved, Git Sync adopts the resource in place, and you must delete the original resource so Git Sync can take over its UID.

The migration follows these steps:

1. [Before you begin](#before-you-begin): Back up your instance and understand what Git Sync manages.
1. [Step 1: Export the resources to your repository](#step-1-export-the-resources-to-your-repository): Export with UIDs preserved and commit.
1. [Step 2: Delete the original dashboards](#step-2-delete-the-original-dashboards): Required so Git Sync can adopt each dashboard by UID.
1. [Step 3: Validate the migration](#step-3-validate-the-migration): Confirm the resources are synced before moving on.

## Before you begin

Because migrating involves deleting resources, read this section and plan carefully before you start.

To migrate safely, keep in mind the following:

- **Back up your instance first.** Export or snapshot your dashboards, folders, alert rules, and library panels before you delete anything. Deleted resources can't be restored from the Grafana UI.
- **Migrate folder by folder.** Start with a single folder, complete the full migration for it, and validate the result before moving to the next one. This limits the impact if something goes wrong and lets you get comfortable with the process.
- **Do not delete a folder that contains alerts or library panels**. Git Sync only manages dashboards and folders. Alerts, data sources, library panels, and other resources are not supported yet, and Git Sync will not recreate them. Deleting the folder deletes them permanently.

### Manage and delete migrated resources

With a folder or folderless sync, Git Sync recreates dashboards and folders:

- You'll have to delete each original dashboard you're migrating so Git Sync can take over its UID.
- You don't need to delete your original folders to migrate the dashboards inside them. Git Sync creates its own new folders when it syncs with your existing repository, deriving each folder's UID from the folder's **path in the repository**. These folders are independent from your existing ones, even if they share the same name.
- However, Git Sync doesn't recreate alerts or library panels. Deleting or recreating a folder to match your repository structure permanently deletes any alert rules and library panels it holds, and they are not restored.

Refer to [Step 2: Delete the original dashboards](#step-2-delete-the-original-dashboards) for more details.

{{< admonition type="note" >}}

Full-instance migrations have different cleanup behavior and can delete unmanaged folders, so follow the full-instance migration guidance instead.

{{< /admonition >}}

### Keep your original resources and set them apart

After a migration you'll have your original folder (holding any alerts and library panels) alongside the new Git Sync folder of the same name. To avoid confusion, rename your original folders or move them under a single top-level **Alerts & Library Panels** folder. This keeps the unsupported resources intact and clearly separated from the provisioned dashboards.

If you need links to your original folders to keep working, refer to [Preserve links to the original folders](#preserve-links-to-the-original-folders).

## Step 1: Export the resources to your repository

Export the dashboards you want to migrate so that each file keeps the dashboard's original UID (`metadata.name`), then commit the files to your Git repository. You can use either of the following:

- [Export with the Grafana CLI](#export-with-the-grafana-cli)
- [Export as a JSON resource file](#export-as-a-json-resource-file)

### Export with the Grafana CLI

You can export existing dashboards from the terminal or from agentic coding tools using the `gcx` CLI. With `gcx` you can download the resources you want to sync from Grafana, and then commit and push those files to your provisioned Git repository. Git Sync will then detect the commit, and synchronize with Grafana.

{{< admonition type="note" >}}

For more information refer to the [`gcx` documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/ai-tools/grafana-cli/gcx/).

{{< /admonition >}}

To export dashboards with `gcx`, follow these steps:

1. Set up the `gcx` context to point to your instance as documented in [Defining contexts](https://github.com/grafana/gcx/#1-authenticate).
1. Pull the resources you want to sync from the instance to your local repository:

   ```
   gcx resources pull dashboards --path <REPO_PATH>
   ```

1. Commit and push the resources to your Git repository:

   ```
   git add <DASHBOARDS_PATH>
   git commit -m "Add dashboards from Grafana"
   git push
   ```

   Where:
   - _<GIT_REPO>_: The path to the repository synced with Git Sync
   - _<DASHBOARDS_PATH>_: The path where the dashboards you want to export are located. The dashboards path must be under the repository

After you commit the resources, continue to [Step 2](#step-2-delete-the-original-dashboards).

### Export as a JSON resource file

To export a dashboard as a JSON resource file, you need to:

1. Export the dashboard as JSON.
1. Convert it to the Custom Resource Definition (CRD) format required by the Grafana App Platform.
1. Commit the converted file to your Git repository.

After you commit the file, continue to [Step 2](#step-2-delete-the-original-dashboards).

To export a dashboard as a JSON file it must follow this CRD structure:

```yaml
{
  'apiVersion': 'dashboard.grafana.app/v1',
  'kind': 'Dashboard',
  'metadata': { 'name': 'dcf2lve9akj8xsd' },
  'spec': { /* Original dashboard JSON goes here */ },
}
```

The structure includes:

- `apiVersion`: Specifies the API version. Both classic and `v2` JSON models are supported. For more information, refer to [Dashboard JSON model](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/view-dashboard-json-model/).
- `kind`: Identifies the resource type. For example, dashboard.
- `metadata`: Contains the dashboard identifier `name`, which must match the original dashboard UID so Git Sync can adopt it. You can find the identifier in the dashboard's URL or in the exported JSON.
- `spec`: Wraps your original dashboard JSON.

## Step 2: Delete the original dashboards

Delete each original dashboard you're migrating so Git Sync can take over its UID. Since the exported files keep the original UID, Git Sync can't provision any dashboard if an unmanaged dashboard with the same UID (`metadata.name`) still exists in Grafana.

When you delete a dashboard, keep in mind the following:

- You cannot restore deleted resources from the UI.
- Dashboard version history does not carry over.
- You need to reapply custom folder permissions. Refer to [Git Sync permissions and access control](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/permissions-grafana) for more details.

### Manage migrated folders

You don't need to delete your migrated folders, as Git Sync creates its own folders with new, path-derived UIDs. **Don't delete or recreate folders that contain alert rules or library panels**, since those resources are deleted permanently and Git Sync does not recreate them. See [Before you begin](#before-you-begin) for how to keep and set apart your original folders.

## Step 3: Validate the migration

1. Trigger a new pull to complete the sync. The dashboards are recreated as provisioned, with their original UIDs, so existing links keep working.
1. Confirm each dashboard appears in the provisioned folder and opens correctly. It may take a few minutes for changes to appear; if they don't, refresh the UI manually.
1. Confirm that the alert rules and library panels in your original folders are still present and working.

After you've validated one folder, repeat the process for the next one until the migration is complete.

## Preserve links to the original folders

By default, Git Sync gives each synced folder a new UID derived from its path in the repository. This means links and URLs that point to your original folders keep pointing to the original folders, not the new provisioned ones.

If you need existing folder links and URLs to resolve to the provisioned folders instead, you can pin a folder's UID with a folder metadata file so Git Sync reuses the original folder's UID.

### Save unsupported resources

Because this option reuses the original folder's UID, the synced folder collides with your existing unmanaged folder, and Git Sync can't take over a UID that still belongs to an unmanaged folder, with the sync failing with a conflict.

To avoid losing unsupported resources, complete these steps for each folder **before** you sync:

1. Create a new folder and move all alert rules, library panels, and other unsupported resources out of the original folder into it.

- Move alert rules and library panels out of the folder **before** you delete the folder, since Git Sync does not recreate alerts or library panels.

1. Delete the original folder. Its dashboards should already be exported to the repository from [Step 1](#step-1-export-the-resources-to-your-repository).
1. Reapply any custom permissions on the new provisioned folder, as folder permissions don't carry over. Refer to [Git Sync permissions and access control](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/permissions-grafana).

### Reuse the original folder's UID

{{< admonition type="note" >}}

Folder metadata requires the `provisioningFolderMetadata` feature, which is enabled by default. If your administrator has disabled it, `metadata.name` is ignored and folders always get a path-derived UID.

{{< /admonition >}}

To reuse an original folder's UID, add a `_folder.json` file to that folder's directory in the repository:

```json
{
  "apiVersion": "folder.grafana.app/v1beta1",
  "kind": "Folder",
  "metadata": { "name": "<ORIGINAL_FOLDER_UID>" },
  "spec": { "title": "<FOLDER_TITLE>" }
}
```

Where `<ORIGINAL_FOLDER_UID>` is the UID of your existing folder. You can find it in the folder's URL.

# Before you begin

Before you begin to set up Git Sync, ensure you have the following:

- A Grafana instance (Cloud, OSS, or Enterprise)
- Administration rights in your Grafana organization
- A [Git provider](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/usage-limits#compatible-providers)
- If you're [using webhooks or image rendering](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/set-up-extend), a public instance with external access
- Optional: The [Image Renderer service](https://github.com/grafana/grafana-image-renderer) to save image previews with your PRs

## Enable required feature toggles

The `provisioning` feature toggle is enabled by default in Grafana Cloud and, starting in Grafana v13, for OSS and Enterprise as well. No manual configuration is required.

For more information about feature toggles, refer to [Configure feature toggles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/).

## Enable Git providers

If you're using Grafana Enterprise v12.4.0 and want to set up Git Sync with pure Git, GitLab or Bitbucket, or if you're using Grafana OSS v12.4.0 and want to set up Git Sync with pure Git, add them to your configuration file:

1. Open your Grafana configuration file, either `grafana.ini` or `custom.ini`.
1. Add the available providers:

   ```ini
   [provisioning]
   repository_types = "git|github|bitbucket|gitlab|local"
   ```

1. Save the changes to the file and restart Grafana.

## Enable OAuth connection types

If you're using self-managed Grafana and the OAuth App connection type for your provider isn't available, add it to `connection_types` in your configuration file:

1. Open your Grafana configuration file, either `grafana.ini` or `custom.ini`.
1. Add the connection types:

   ```ini
   [provisioning]
   connection_types = "github|githubOAuth|gitOAuth"
   ```

1. Save the changes to the file and restart Grafana.

The available OAuth connection types are:

| **Connection type**     | **Provider**      | **Available in** |
| ----------------------- | ----------------- | ---------------- |
| `githubOAuth`           | GitHub            | OSS, Enterprise  |
| `gitOAuth`              | Pure Git          | OSS, Enterprise  |
| `githubEnterpriseOAuth` | GitHub Enterprise | Enterprise       |
| `gitlabOAuth`           | GitLab            | Enterprise       |
| `bitbucketOAuth`        | Bitbucket         | Enterprise       |

## Network connectivity and IP allowlisting

Git Sync requires network connectivity between your Grafana instance and Git server. Understanding the traffic patterns helps you configure firewall rules and allowlists correctly.

### Traffic types

Git Sync uses two types of network traffic:

- **Sync operations (pull and push)**: From Grafana to the Git Server
  - Egress traffic from Hosted Grafana IP addresses
  - Your Git servers must allow inbound traffic from these IP addresses
  - For a list of IP addresses to add to your Git server's allowlist, refer to [Hosted Grafana source IPs](https://grafana.com/docs/grafana-cloud/security-and-account-management/allow-list/#hosted-grafana)
- **Webhooks (instantaneous sync)**: From the Git Server to Grafana stack
  - Inbound traffic to the stack's public endpoint
  - Your Git server must be able to reach `*.grafana.net`
  - Required only if you're [using webhooks](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/set-up-extend)

### AWS PrivateLink and Private Data Source Connect

Git Sync doesn't route over AWS PrivateLink or Private Data Source Connect (PDC). Instead, Git Sync uses the normal public path from the Hosted Grafana IPs, which is independent from PrivateLink and PDC. AWS PrivateLink and PDC provide a separate tunnel for data source query traffic, from Grafana to your private databases or data sources.

**If you use AWS PrivateLink or PDC for data sources, you can still use Git Sync**. The two features neither interfere with nor depend on each other.

## Allow internal or private Git servers

{{< admonition type="note" >}}
This setting is available for Grafana v13.0.4 and Grafana v13.1.1 and later. It only applies to self-managed Grafana (OSS and Enterprise), but it's not configurable in Grafana Cloud.
{{< /admonition >}}

While public Git servers such as `github.com`, `gitlab.com`, and `bitbucket.org` resolve to public addresses and are always allowed, by default Git Sync rejects repository URLs with a host that resolves to a loopback, a private (RFC 1918), link-local, or an unspecified address. This protects your Grafana instance against server-side request forgery (SSRF).

If you connect Git Sync to a Git server on a private network such as a self-hosted GitHub Enterprise, GitLab, or Bitbucket instance reachable only through an internal address, add its host to the `allowed_git_urls` allowlist:

1. Open your Grafana configuration file, either `grafana.ini` or `custom.ini`.
1. Add each internal Git host to `allowed_git_urls` as a comma-separated list:

   ```ini
   [provisioning]
   allowed_git_urls = git.internal.example.com, ghe.example.com:8443
   ```

1. Save the changes to the file and restart Grafana.

Each entry can be a hostname, `host:port`, a full URL (only the host is used), a literal IP address, or a CIDR range. If possible, use specific hosts or narrow ranges, since a broad CIDR such as `10.0.0.0/8` re-exposes the entire private range that SSRF protection blocks.

**Only add hosts you trust** because an allowlisted host receives the configured Git token on every sync, fetch, and push.

## Resource and role permissions

By default, folders provisioned with Git Sync have these roles:

- Admin = Admin
- Editor = Editor
- Viewer = Viewer.

Refer to [Git Sync permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/permissions-grafana) for details on how to set up permissions in Git Sync. To modify them, refer to [Manage folder permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/use-git-sync#manage-folder-permissions).

Refer to [Roles and permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions) for more information about Grafana roles.

## Create a GitHub App

GitHub Apps are tools that extend GitHub functionality. They use fine-grained permissions and short-lived tokens, giving you more control over which repositories are being accessed. Find out more in the [GitHub Apps official documentation](https://docs.github.com/en/apps/overview).

If you chose to authenticate with a newly created GitHub App, you'll need the following parameters:

- GitHub App ID
- GitHub App Private Key
- GitHub App Installation ID

There are many ways to create a GitHub App. The following instructions are informative only, always refer to official GitHub documentation for more details.

To create the GitHub App, follow these steps:

1. Go to https://github.com/settings/apps and click on **New Github App**, or navigate directly to https://github.com/settings/apps/new
1. Fill in the following fields:
   - Name: Must be unique
   - Homepage URL: For example, your Grafana Cloud instance URL
1. Scroll down to the **Webhook** section and uncheck the **Active** box
1. In the **Permissions** section, go to **Repository permissions** and set these parameters:
   - **Administration**: Read-only permission (enables validation of branch protection rules against the configured branch when users can push directly to it; may be used in the future to check other repository settings and make the setup process smoother)
   - **Contents**: Read and write permission
   - **Metadata**: Read-only permission
   - **Pull requests**: Read and write permission
   - **Webhooks**: Read and write permission
1. Finally, under **Where can this GitHub App be installed?**, select **Only on this account**
1. Click on **Create Github App** to complete the process.

On the app page:

1. Copy the **AppID** from the **About** section
1. Select the **Generate private key** from the banner or scroll down to to the **Private Keys** section to generate a key
1. A PEM file containing your private key will be downloaded to your computer

Finally, install the app:

1. At the top left of the App page, click on **Install App**
1. Choose for which user you need to install it, you’ll be redirected to the repository selection screen
1. Choose for which repositories you want to install the app
1. Click **Install**.
1. On the installation page, copy **`installationID`** from the page URL https://github.com/settings/installations/installationID

You can now proceed to [Set up Git Sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/)!

## Create an OAuth App

An OAuth App lets Grafana act on your behalf in your Git provider. You create the app in your provider, then authorize it once from Grafana. Grafana stores the access token and refreshes it when the provider supports refresh tokens.

{{< admonition type="caution" >}}
An OAuth App connection acts as the user who authorized it. Every repository that uses the connection and every user who can run its sync jobs will use the access token of the user that authorized the connection. The token can reach every repository that user can access, not only those connected to Grafana. Removing the user from Grafana doesn't revoke the token. Revoke it in your Git provider.

To limit access, authorize the app with a dedicated account that can only access the repositories you sync. For GitHub, consider a GitHub App, which is scoped to the repositories where it's installed.
{{< /admonition >}}

If you chose to authenticate with an OAuth App, you need the following parameters:

- The client ID of the app.
- The client secret of the app.

Every OAuth App needs the Grafana callback URL. Grafana shows it in the setup wizard, and it has this format:

```text
<GRAFANA_URL>/admin/provisioning/connections/oauth-callback
```

Replace _`<GRAFANA_URL>`_ with the root URL of your Grafana instance, including any sub-path.

The following instructions are informative only. Always refer to the official documentation of your Git provider for more details.

### GitHub and GitHub Enterprise

To create a GitHub OAuth App, follow these steps:

1. In GitHub, go to **Settings > Developer settings > OAuth apps** and click **New OAuth App**. For GitHub Enterprise, use the same menu on your enterprise instance.
1. Enter an **Application name** and a **Homepage URL**, for example your Grafana instance URL.
1. Paste the Grafana callback URL in **Authorization callback URL**.
1. Click **Register application**.
1. Copy the **Client ID**, then click **Generate a new client secret** and copy the secret.

Grafana requests the `repo` scope when you authorize the app. For more details, refer to [Creating an OAuth app](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app).

### GitLab

To create a GitLab OAuth application, follow these steps:

1. In GitLab, create the application for your user or for a group:
   - For your user, select your avatar, then **Edit profile > Access > Applications > Add new application**.
   - For a group, go to the group and select **Settings > Applications**.
1. Enter a **Name** and paste the Grafana callback URL in **Redirect URI**.
1. Keep **Confidential** selected, and select the `api` scope.
1. Click **Save application**.
1. Copy the **Application ID** and the **Secret**.

For more details, refer to [Configure GitLab as an OAuth 2.0 authentication identity provider](https://docs.gitlab.com/integration/oauth_provider/).

### Bitbucket

To create a Bitbucket OAuth consumer, follow these steps:

1. In Bitbucket, go to **Workspace settings > Apps and features > OAuth consumers** and click **Add consumer**.
1. Enter a **Name** and paste the Grafana callback URL in **Callback URL**.
1. Set these permissions:
   - **Repositories**: Read and write permission
   - **Pull requests**: Read and write permission
   - **Webhooks**: Read and write permission
1. Click **Save**.
1. Select the consumer name to show the **Key** and the **Secret**, copy them, and note the name of the workspace.

For more details, refer to [Use OAuth on Bitbucket Cloud](https://support.atlassian.com/bitbucket-cloud/docs/use-oauth-on-bitbucket-cloud/).

### Other Git providers

For any other Git provider that supports the OAuth 2.0 authorization code flow, create an OAuth application with the Grafana callback URL as the redirect URI. In addition to the client ID and secret, you need:

- The **Authorization URL** of the provider, for example `https://git.example.com/oauth/authorize`.
- The **Token URL** of the provider, for example `https://git.example.com/oauth/token`.
- The **Scopes** that grant read and write access to your repositories. Refer to your provider's documentation.

Both URLs must use `https://`, unless you set `allow_insecure` in the `[provisioning]` section of your configuration file.

## Next steps

For further details on how Git Sync operates, refer to:

- [Git Sync key concepts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/key-concepts)
- [Git Sync supported resources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/#supported-resources)
- [Git Sync usage and performance limitations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/usage-limits)

# Set up Git Sync as code

You can also configure Git Sync using `gcx`, the Grafana CLI. Since Git Sync configuration is managed as code using Custom Resource Definitions (CRDs), you can create your required resources in YAML files and push them to Grafana using `gcx`. This approach enables automated, GitOps-style workflows for managing Git Sync configuration instead of using the Grafana UI.

For more information, refer to the following documents:

- [Repository resource](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/key-concepts#git-sync-repository-resource) and [Connection resource](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/key-concepts#git-sync-repository-resource) overview
- [Dashboard CRD Format](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/add-resources/dashboards-migrate)
- [Grafana CLI documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/ai-tools/grafana-cli/)

To provision resources from a local file system as code instead of a Git repository, refer to [Set up file provisioning as code](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/provision-resources/file-path-setup#set-up-file-provisioning-as-code).

## Set up Git Sync as code with the Grafana CLI

To set up Git Sync as code with `gcx`, follow these steps:

1. Understand [Usage and performance limitations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/usage-limits)
1. [Create the connection and repository CRDs](#create-the-resources-crds)
1. [Push the CRDs to Grafana](#push-the-resources-to-grafana)
1. [Manage repository resources](#manage-repository-resources)
1. [Verify setup](#verify-setup)

## Create the resources CRDs

If you're connecting with any of the [supported Git providers](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/usage-limits#compatible-git-providers) using a Personal Access Token, you need to create a repository resource to define the connection between your repositories and your Grafana instance.

If you're connecting to Git Sync with GitHub App or an OAuth App, in addition to the repository resource you need to create a connection resource as well.

### Create the connection resource

If you're connecting Git Sync with GitHub App, create a `connection.yaml` file defining your Git Sync connection configuration:

```yaml
apiVersion: provisioning.grafana.app/v0alpha1
kind: Connection
metadata:
  name: '<GITHUB_CONNECTION_NAME>'
  namespace: default
spec:
  title: '<REPOSITORY_TITLE>'
  type: github
  url: https://github.com
  github:
    appID: '<GITHUB_APP_ID>'
    installationID: '<GITHUB_INSTALL_ID>'
    serverUrl: '<GITHUB_ENTERPRISE_SERVER_URL>' # Only required for GitHub Enterprise
secure:
  privateKey:
    create: '<GITHUB_PRIVATE_KEY>'
```

Replace the placeholders with your values:

- _`<GITHUB_CONNECTION_NAME>`_: The name of your GitHub connection
- _`<REPOSITORY_TITLE>`_: Human-readable name displayed in Grafana UI
- _`<GITHUB_APP_ID>`_: GitHub App unique identifier
- _`<GITHUB_INSTALL_ID>`_: GitHub App installation id
- _`<GITHUB_PRIVATE_KEY>`_: GitHub Private Key

### Create an OAuth App connection resource

If you're connecting Git Sync with an [OAuth App](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/set-up-before/#create-an-oauth-app), create a `connection.yaml` file with the client ID and secret of the app, and the fields for your provider:

```yaml
apiVersion: provisioning.grafana.app/v0alpha1
kind: Connection
metadata:
  name: '<OAUTH_CONNECTION_NAME>'
  namespace: default
spec:
  title: '<CONNECTION_TITLE>'
  type: '<CONNECTION_TYPE>' # githubOAuth, githubEnterpriseOAuth, gitlabOAuth, bitbucketOAuth, or gitOAuth
  oauth:
    clientID: '<CLIENT_ID>'
  # GitHub Enterprise only:
  githubEnterpriseOAuth:
    serverUrl: '<GITHUB_ENTERPRISE_SERVER_URL>'
  # Bitbucket only:
  bitbucket:
    workspace: '<BITBUCKET_WORKSPACE>'
  # Pure Git only:
  gitOAuth:
    authURL: '<AUTHORIZATION_URL>'
    tokenURL: '<TOKEN_URL>'
    scopes:
      - '<SCOPE>'
secure:
  clientSecret:
    create: '<CLIENT_SECRET>'
```

Replace the placeholders with your values:

- _`<OAUTH_CONNECTION_NAME>`_: The name of your OAuth App connection
- _`<CONNECTION_TITLE>`_: Human-readable name displayed in Grafana UI
- _`<CONNECTION_TYPE>`_: The OAuth connection type for your provider
- _`<CLIENT_ID>`_: The client ID of the OAuth App
- _`<CLIENT_SECRET>`_: The client secret of the OAuth App
- _`<GITHUB_ENTERPRISE_SERVER_URL>`_: The URL of your GitHub Enterprise instance
- _`<BITBUCKET_WORKSPACE>`_: The workspace that the OAuth consumer belongs to
- _`<AUTHORIZATION_URL>`_, _`<TOKEN_URL>`_, _`<SCOPE>`_: The OAuth endpoints of your Git provider, and the scopes that grant read and write access to your repositories

An OAuth App connection needs a user to approve access in the browser. After you push the connection, it shows as **Disconnected** until you authorize it:

1. Select **Administration > General > Provisioning** and go to the **Connections** tab.
1. Select the connection and click **Reauthorize**.
1. Approve the access in the provider tab that opens.

### Create the repository resource

Next, create a `repository.yaml` file defining your Git Sync configuration. Depending on your Git provider and authentication method, add your Personal Access Token information or the connection name.

```yaml
apiVersion: provisioning.grafana.app/v0alpha1
kind: Repository
metadata:
  name: '<REPOSITORY_NAME>'
spec:
  sync:
    enabled: true
    intervalSeconds: 60
    target: folder
  workflows:
    - write
    - branch
  title: '<REPOSITORY_TITLE>'

# Git Sync for GitHub:
spec:
  type: github
  github:
    url: '<GIT_REPO_URL>'
    branch: '<BRANCH>'
    path: grafana/
  # GitHub App or OAuth App connection only:
  connection:
    name: '<GITHUB_CONNECTION_NAME>'
# GitHub Personal Access Token only:
secure:
  token: { create: 'GIT_PAT' }

# Git Sync for GitHub Enterprise:
spec:
  type: githubEnterprise
  githubEnterprise:
    url: '<GIT_REPO_URL>'
    branch: '<BRANCH>'
    path: grafana/
  # GitHub Enterprise App connection only:
  connection:
    name: '<GITHUB_ENTERPRISE_CONNECTION_NAME>'
# GitHub Personal Access Token only:
secure:
  token: { create: 'GIT_PAT' }

# GitLab Personal Access Token only:
spec:
  type: gitlab
  gitlab:
    url: '<GIT_REPO_URL>'
    branch: '<BRANCH>'
secure:
  token: { create: 'GIT_PAT' }

# Bitbucket Personal Access Token only:
spec:
  type: bitbucket
  bitbucket:
    url: '<GIT_REPO_URL>'
    branch: '<BRANCH>'
    tokenUser: tokenuser
secure:
  token: { create: 'GIT_PAT' }

# Pure Git only:
spec:
  type: git
  git:
    url: '<GIT_REPO_URL>'
    branch: '<BRANCH>'
    path: 'grafana/'
    tokenUser: tokenuser
secure:
  token: { create: 'GIT_PAT' }
```

To use an OAuth App connection, set `spec.connection.name` instead of `secure.token`. The repository type must match the connection type: `githubOAuth` for `github`, `githubEnterpriseOAuth` for `githubEnterprise`, `gitlabOAuth` for `gitlab`, `bitbucketOAuth` for `bitbucket`, and `gitOAuth` for `git`.

Replace the placeholders with your values:

- _`<REPOSITORY_NAME>`_: Unique identifier for this repository resource
- _`<REPOSITORY_TITLE>`_: Human-readable name displayed in Grafana UI
- _`<GIT_REPO_URL>`_: GitHub repository URL
- _`<BRANCH>`_: Branch to sync
- _`<GITHUB_CONNECTION_NAME>`_: The name of your GitHub connection
- _`<GITHUB_ENTERPRISE_CONNECTION_NAME>`_: The name of your GitHub Enterprise connection
- _`<GIT_PAT>`_: Git provider Personal Access Token

{{< admonition type="note" >}}

Git Sync supports two sync targets: `target: folder` (the default) creates a folder named after the repository and places synced resources inside it, while `target: folderless` places synced resources at the top level without creating a wrapper folder. Refer to [Sync targets](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/key-concepts/#sync-targets) for details.

{{< /admonition >}}

### Configuration parameters

The following configuration parameters are available:

| Field                                   | Description                                                     |
| --------------------------------------- | --------------------------------------------------------------- |
| `metadata.name`                         | Unique identifier for this repository resource                  |
| `spec.title`                            | Human-readable name displayed in Grafana UI                     |
| `spec.type`                             | Repository type (`github`, `githubEnterprise`)                  |
| `spec.github.url`                       | GitHub repository URL                                           |
| `spec.github.branch`                    | Branch to sync                                                  |
| `spec.github.path`                      | Directory path containing dashboards                            |
| `spec.github.generateDashboardPreviews` | Generate preview images (true/false) (Only available in GitHub) |
| `spec.sync.enabled`                     | Enable synchronization (true/false)                             |
| `spec.sync.intervalSeconds`             | Sync interval in seconds                                        |
| `spec.sync.target`                      | Where to place synced dashboards (`folder` or `folderless`)     |
| `spec.workflows`                        | Enabled workflows: `write` (direct commits), `branch` (PRs)     |
| `secure.token.create`                   | GitHub Personal Access Token                                    |

## Push the resources to Grafana

Before pushing any resources, configure `gcx` with your Grafana instance details. Refer to the [Grafana CLI documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/ai-tools/grafana-cli/) for setup instructions.

Push the repository configuration. If you're using GitHub App to connect Git Sync, push the connection resource configuration file as well.

```sh
gcx resources push --path <DIRECTORY>
```

The `--path` parameter has to point to the directory containing your `repository.yaml` and `connection.yaml` files.

After pushing, Grafana will:

1. Create the required resources (repository and, for GitHub App, connection)
1. Connect to your GitHub repository
1. Pull dashboards from the specified path
1. Begin syncing at the configured interval

## Manage repository resources

### List repositories

To list all repositories:

```sh
gcx resources get repositories
```

### Get repository details

To get details for a specific repository:

```sh
gcx resources get repository/<REPOSITORY_NAME>
gcx resources get repository/<REPOSITORY_NAME> -o json
gcx resources get repository/<REPOSITORY_NAME> -o yaml
```

### Update the repository

To update a repository:

```sh
gcx resources edit repository/<REPOSITORY_NAME>
```

### Delete the repository

To delete a repository:

```sh
gcx resources delete repository/<REPOSITORY_NAME>
```

## Verify setup

Check that Git Sync is working:

```sh
# List repositories
gcx resources get repositories

# Check Grafana UI
# Navigate to: Administration → Provisioning → Git Sync
```

# Set up instantaneous pulling and dashboard previews in Pull Requests

After [setup](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/), you can optionally extend Git Sync by enabling pull request notifications and image previews of dashboard changes.

| Capability                                       | Benefit                                                           | Requires                               |
| ------------------------------------------------ | ----------------------------------------------------------------- | -------------------------------------- |
| A table summarizing changes to your pull request | A convenient way to save changes back to GitHub                   | Webhooks configured                    |
| A dashboard preview image to a PR                | A snapshot of dashboard changes to a pull request outside Grafana | Image renderer and webhooks configured |

## Set up webhooks for real-time notification and pull request integration

Real-time notifications (or automatic pulling) is enabled and configured by default in Grafana Cloud.

In Grafana OSS/Enterprise, Git Sync uses webhooks to enable real-time updates from GitHub public repositories, or to enable pull request integrations. Without webhooks the polling interval is set during configuration, and is 60 seconds by default. You can set up webhooks with whichever service or tooling you prefer: Cloudflare Tunnels with a Cloudflare-managed domain, port-forwarding and DNS options, or a tool such as `ngrok`.

To set up webhooks:

1. Expose your Grafana instance to the public Internet.
   - Use port forwarding and DNS, `ngrok`, or any other method you prefer.
   - The permissions set in your GitHub access token provide the authorization for this communication.
1. After you have the public URL, add it to your Grafana configuration file:
   ```ini
   [server]
   root_url = https://<PUBLIC_DOMAIN>
   ```
1. Replace _`<PUBLIC_DOMAIN>`_ with your public domain.

To check the configured webhooks, go to **Administration > General > Provisioning** and click the **View** link for your GitHub repository.

### Webhook limits

Git Sync registers its own webhooks for each repository connection, so when several Grafana instances sync the same repository, each connection adds to this total. Since [Git providers limit the amount of webhooks](#webhook-limitations-per-provider) you can set for each repository, when the limit is exceeded, your Git provider rejects new webhooks.

For example, in GitHub you can get an error such as:

```
GitHub API error (HTTP 422: Validation Failed: The "pull_request" event cannot have more than 20 hooks; The "push" event cannot have more than 20 hooks)
```

If you hit this limit, you can either:

- Remove unused or duplicate webhooks from your repository's **Settings > Webhooks** page.
- Disable webhook integration for connections that don't need real-time sync so those instances poll on an interval instead. Refer to [Webhook options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/#webhook-options) to disable webhook integration.

#### Webhook limitations per provider

The following limits apply:

- GitHub limits each repository to 20 webhooks per event type, such `push` and `pull_request`. For more information refer to the official [GitHub webhooks documentation](https://docs.github.com/en/webhooks/types-of-webhooks).
- GitLab defaults to 100 webhooks for each project, 50 for each group, although subgroup webhooks are not counted towards parent group limits. Refer to [GitLab settings](https://docs.gitlab.com/user/gitlab_com/#other-limits) for more information.
- BitBucket limits to 50 webhooks per repository. Refer to [BitBucket webhook documentation](https://support.atlassian.com/bitbucket-cloud/docs/manage-webhooks/#Create-webhooks) for more details.

### Set a publicly-reachable URL

If you need to point `[server] root_url` to an internal address (for example, when Grafana runs behind a private ingress in a Kubernetes cluster), set the publicly-reachable URL with `[provisioning] public_root_url` instead. This URL is used both to register webhook callbacks with the Git provider and as the base for screenshot images embedded in pull-request comments, which the Git provider's servers fetch from the public internet.

```ini
[server]
root_url = http://internal.cluster.local

[provisioning]
public_root_url = https://<PUBLIC_DOMAIN>
```

The per-repository `spec.webhook.baseUrl` field still overrides `public_root_url` for webhook registration, and screenshot URLs always use `public_root_url` (or `root_url` when unset).

### Expose necessary paths only

If your security setup doesn't permit publicly exposing the Grafana instance, you can either choose to allowlist the Git provider's IP addresses, or expose only the necessary paths.

For information about the traffic between Grafana and your Git server, refer to [Network connectivity and IP allowlisting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/set-up-before/#network-connectivity-and-ip-allowlisting).

In RegExp, the necessary paths required to be exposed are `/apis/provisioning\.grafana\.app/v0(alpha1)?/namespaces/[^/]+/repositories/[^/]+/(webhook|render/.*)$`.

## Set up image rendering for dashboard previews

{{< admonition type="caution" >}}

Available in Grafana OSS and Grafana Enterprise only.

{{< /admonition >}}

To set up image rendering to add visual previews of dashboard updates directly in pull requests, **install the Grafana Image Renderer** in your Grafana instance. For more information and installation instructions, refer to the [Image Renderer service](https://github.com/grafana/grafana-image-renderer).

Image rendering requires webhooks.

## Next steps

To learn more about using Git Sync refer to the following documents:

- [Set up Git Sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/)
- [Export resources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/add-resources/)
- [Work with provisioned repositories](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/use-git-sync/)
- [Work with provisioned dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/provisioned-dashboards/)
- [Git Sync deployment scenarios](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/scenarios/)

# Set up Git Sync with Terraform

You can also configure Git Sync via the Grafana provisioning app platform using Terraform.

## Before you begin

Before you begin, make sure to have the following:

- A Grafana Cloud account or an on-prem Grafana instance.
- Administrator permissions in your Grafana stack/instance.
- Terraform 1.11 or later installed on your machine. Refer to the [Terraform install documentation](https://developer.hashicorp.com/terraform/install) to learn more.

{{< admonition type="note" >}}
Save all of the following Terraform configuration files in the same directory.
{{< /admonition >}}

## Configure the Grafana provider

Use this Terraform configuration to configure the [Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) with the authentication required to set up Git Sync.

1. Create a service account and token in Grafana. For more information refer to [Service account tokens](https://grafana.com/docs/grafana/latest/administration/service-accounts/#service-account-tokens) or [Create and manage a Grafana Cloud stack using Terraform](https://grafana.com/docs/grafana-cloud/as-code/terraform/terraform-cloud-stack/).

1. Make sure that the token has the Admin or the `Provisioning:Repositories` writer permission.

1. Create a file named `main.tf` and add the following:

   ```terraform
      terraform {
        required_providers {
          grafana = {
            source  = "grafana/grafana"
            version = ">= 4.28.1"
          }
        }
      }

      provider "grafana" {
        url = "https://<your-stack>.grafana.net/"
        auth = var.grafana_service_account_token
        stack_id = var.grafana_stack_id
      }
   ```

Replace the following field values:

- `<Stack-URL>` with the URL of your Grafana stack (for example, `https://my-stack.grafana.net/`)
- `<Service-account-token>` with the service account token that you created (or `<USERNAME>:<PASSWORD>` for basic auth)
- `<Stack-ID>` with the Grafana stack ID, if you are using a Grafana Cloud stack

{{< admonition type="caution" >}}

The Git Sync Terraform resources are App Platform resources and talk directly to the Grafana stack's API.

**Do not use Grafana Cloud provider settings in this configuration**. Configure the provider with `url` and `auth` (not `cloud_api_url` and `cloud_access_policy_token`). If you're using Grafana Cloud, set `stack_id` to your stack ID.

{{< /admonition >}}

## Create the resources to use Git Sync

You need two resources for configure and manage Git Sync:

- The repository resource configures the Git repository to sync Grafana resources with. For examples, refer to [Repository resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_provisioning_repository_v0alpha1) in the Terraform registry.
- The connection resource configures your Git provider credentials. For examples, refer to [Connection resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_provisioning_connection_v0alpha1) in the Terraform registry.

For better understanding of the required resources, refer to [Git Sync key concepts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/key-concepts).

# Git Sync for development and production environments

Use separate Grafana instances for development and production. Each syncs with different Git locations to test dashboards before production.

## Use it for

- **Staged deployments**: You need to test dashboard changes before production deployment.
- **Change control**: You require approvals before dashboards reach production.
- **Quality assurance**: You verify dashboard functionality in a non-production environment.
- **Risk mitigation**: You minimize the risk of breaking production dashboards.

## Architecture

```
┌────────────────────────────────────────────────────────────┐
│              GitHub Repository                             │
│   Repository: your-org/grafana-manifests                 │
│   Branch: main                                             │
│                                                            │
│   grafana-manifests/                                     │
│   ├── dev/                                                │
│   │   ├── dashboard-new.json    ← Development dashboards │
│   │   └── dashboard-test.json                            │
│   │                                                       │
│   └── prod/                                               │
│       ├── dashboard-stable.json  ← Production dashboards │
│       └── dashboard-approved.json                        │
└────────────────────────────────────────────────────────────┘
           ↕                                 ↕
    Git Sync (dev/)              Git Sync (prod/)
           ↕                                 ↕
┌─────────────────────┐          ┌─────────────────────┐
│  Dev Grafana        │          │  Prod Grafana       │
│                     │          │                     │
│  Repository:        │          │  Repository:        │
│  - path: dev/       │          │  - path: prod/      │
│                     │          │                     │
│  Creates folder:    │          │  Creates folder:    │
│  "grafana-manifests"│         │  "grafana-manifests"│
└─────────────────────┘          └─────────────────────┘
```

## Repository structure

**In Git:**

```
your-org/grafana-manifests
├── dev/
│   ├── dashboard-new.json
│   └── dashboard-test.json
└── prod/
    ├── dashboard-stable.json
    └── dashboard-approved.json
```

**In Grafana Dashboards view:**

**Dev instance:**

```
Dashboards
└── 📁 grafana-manifests/
    ├── New Dashboard
    └── Test Dashboard
```

**Prod instance:**

```
Dashboards
└── 📁 grafana-manifests/
    ├── Stable Dashboard
    └── Approved Dashboard
```

- Both instances create a folder named "grafana-manifests" (from repository name)
- Each instance only shows dashboards from its configured path (`dev/` or `prod/`)
- Dashboards appear with their titles from the JSON files

## Configuration parameters

Development:

- Repository: `your-org/grafana-manifests`
- Branch: `main`
- Path: `dev/`

Production:

- Repository: `your-org/grafana-manifests`
- Branch: `main`
- Path: `prod/`

## How it works

1. Developers create and modify dashboards in development.
2. Git Sync commits changes to `dev/`.
3. You review changes in Git.
4. You promote approved dashboards from `dev/` to `prod/`.
5. Production syncs from `prod/`.
6. Production dashboards update.

## Alternative: Use branches

Instead of using different paths, you can configure instances to use different branches:

**Development instance:**

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `develop`
- **Path**: `grafana/`

**Production instance:**

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `main`
- **Path**: `grafana/`

With this approach:

- Development changes go to the `develop` branch
- Use Git merge or pull request workflows to promote changes from `develop` to `main`
- Production automatically syncs from the `main` branch

## Alternative: Use separate repositories for stricter isolation

For stricter isolation, use completely separate repositories:

**Development instance:**

- **Repository**: `your-org/grafana-manifests-dev`
- **Branch**: `main`
- **Path**: `grafana/`

**Production instance:**

- **Repository**: `your-org/grafana-manifests-prod`
- **Branch**: `main`
- **Path**: `grafana/`

# Git Sync for high availability environments

## Primary–replica scenario

Use a primary Grafana instance and one or more replicas synchronized with the same Git location to enable failover.

### Use it for

- **Automatic failover**: You need service continuity when the primary instance fails.
- **High availability**: Your organization requires guaranteed dashboard availability.
- **Simple HA setup**: You want high availability without the complexity of active–active.
- **Maintenance windows**: You perform updates while another instance serves traffic.
- **Business continuity**: Dashboard access can't tolerate downtime.

### Architecture

```
┌─────────────────────────────────────────────────────┐
│           GitHub Repository                         │
│   Repository: your-org/grafana-manifests          │
│   Branch: main                                      │
│                                                     │
│   grafana-manifests/                              │
│   └── shared/                                      │
│       ├── dashboard-metrics.json                   │
│       ├── dashboard-alerts.json                    │
│       └── dashboard-logs.json                      │
└─────────────────────────────────────────────────────┘
              ↕                           ↕
        Git Sync (shared/)        Git Sync (shared/)
              ↕                           ↕
┌────────────────────┐          ┌────────────────────┐
│  Master Grafana    │          │  Replica Grafana   │
│    (Active)        │          │    (Standby)       │
│                    │          │                    │
│  Repository:       │          │  Repository:       │
│  - path: shared/   │          │  - path: shared/   │
└────────────────────┘          └────────────────────┘
         │                               │
         └───────────┬───────────────────┘
                     ↓
          ┌──────────────────────┐
          │  Reverse Proxy       │
          │  (Failover)          │
          └──────────────────────┘
```

### Repository structure

**In Git:**

```
your-org/grafana-manifests
└── shared/
    ├── dashboard-metrics.json
    ├── dashboard-alerts.json
    └── dashboard-logs.json
```

**In Grafana Dashboards view (both instances):**

```
Dashboards
└── 📁 grafana-manifests/
    ├── Metrics Dashboard
    ├── Alerts Dashboard
    └── Logs Dashboard
```

- Master and replica instances show identical folder structure.
- Both sync from the same `shared/` path.
- Reverse proxy routes traffic to master (active) instance.
- If master fails, proxy automatically fails over to replica (standby).
- Users see the same dashboards regardless of which instance is serving traffic.

### Configuration parameters

Both master and replica instances use identical parameters:

**Master instance:**

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `main`
- **Path**: `shared/`

**Replica instance:**

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `main`
- **Path**: `shared/`

### How it works

1. Both instances stay synchronized through Git.
2. Reverse proxy routes traffic to primary.
3. Users edit on primary. Git Sync commits changes.
4. Both instances pull latest changes to keep replica in sync.
5. On primary failure, proxy fails over to replica.

### Failover considerations

- Health checks and monitoring.
- Continuous syncing to minimize data loss.
- Plan failback (automatic or manual).

## Load balancer scenario

Run multiple active Grafana instances behind a load balancer. All instances sync from the same Git location.

### Use it for

- **High traffic**: Your deployment needs to handle significant user load.
- **Load distribution**: You want to distribute user requests across instances.
- **Maximum availability**: You need service continuity during maintenance or failures.
- **Scalability**: You want to add instances as load increases.
- **Performance**: Users need fast response times under heavy load.

### Architecture

```
┌─────────────────────────────────────────────────────┐
│           GitHub Repository                         │
│   Repository: your-org/grafana-manifests          │
│   Branch: main                                      │
│                                                     │
│   grafana-manifests/                              │
│   └── shared/                                      │
│       ├── dashboard-metrics.json                   │
│       ├── dashboard-alerts.json                    │
│       └── dashboard-logs.json                      │
└─────────────────────────────────────────────────────┘
              ↕                           ↕
        Git Sync (shared/)        Git Sync (shared/)
              ↕                           ↕
┌────────────────────┐          ┌────────────────────┐
│  Grafana Instance 1│          │  Grafana Instance 2│
│    (Active)        │          │    (Active)        │
│                    │          │                    │
│  Repository:       │          │  Repository:       │
│  - path: shared/   │          │  - path: shared/   │
└────────────────────┘          └────────────────────┘
         │                               │
         └───────────┬───────────────────┘
                     ↓
          ┌──────────────────────┐
          │  Load Balancer       │
          │  (Round Robin)       │
          └──────────────────────┘
```

### Repository structure

**In Git:**

```
your-org/grafana-manifests
└── shared/
    ├── dashboard-metrics.json
    ├── dashboard-alerts.json
    └── dashboard-logs.json
```

**In Grafana Dashboards view (all instances):**

```
Dashboards
└── 📁 grafana-manifests/
    ├── Metrics Dashboard
    ├── Alerts Dashboard
    └── Logs Dashboard
```

- All instances show identical folder structure.
- All instances sync from the same `shared/` path.
- Load balancer distributes requests across all active instances.
- Any instance can serve read requests.
- Any instance can accept dashboard modifications.
- Changes propagate to all instances through Git.

### Configuration parameters

All instances use identical parameters:

**Instance 1:**

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `main`
- **Path**: `shared/`

**Instance 2:**

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `main`
- **Path**: `shared/`

### How it works

1. All instances stay synchronized through Git.
2. Load balancer distributes incoming traffic across all active instances.
3. Users can view dashboards from any instance.
4. When a user modifies a dashboard on any instance, Git Sync commits the change.
5. All other instances pull the updated dashboard during their next sync cycle, or instantly if webhooks are configured.
6. If one instance fails, load balancer stops routing traffic to it and remaining instances continue serving.

### Important considerations

- **Eventually consistent**: Due to sync intervals, instances may briefly have different dashboard versions.
- **Concurrent edits**: Multiple users editing the same dashboard on different instances can cause conflicts.
- **Database sharing**: Instances should share the same backend database for user sessions, preferences, and annotations.
- **Stateless design**: Design for stateless operation where possible to maximize load balancing effectiveness.

# Git Sync with regional replication

Deploy multiple Grafana instances across regions. Synchronize them with the same Git location to ensure consistent dashboards everywhere.

## Use it for

- **Geographic distribution**: You deploy Grafana close to users in different regions.
- **Latency reduction**: Users need fast dashboard access from their location.
- **Data sovereignty**: You keep dashboard data in specific regions.
- **High availability**: You need dashboard availability across regions.
- **Consistent experience**: All users see the same dashboards regardless of region.

## Architecture

```
┌─────────────────────────────────────────────────────┐
│           GitHub Repository                         │
│   Repository: your-org/grafana-manifests          │
│   Branch: main                                      │
│                                                     │
│   grafana-manifests/                              │
│   └── shared/                                      │
│       ├── dashboard-global.json                    │
│       ├── dashboard-metrics.json                   │
│       └── dashboard-logs.json                      │
└─────────────────────────────────────────────────────┘
              ↕                           ↕
       Git Sync (shared/)         Git Sync (shared/)
              ↕                           ↕
┌────────────────────┐          ┌────────────────────┐
│  US Region         │          │  EU Region         │
│  Grafana           │          │  Grafana           │
│                    │          │                    │
│  Repository:       │          │  Repository:       │
│  - path: shared/   │          │  - path: shared/   │
└────────────────────┘          └────────────────────┘
```

## Repository structure

**In Git:**

```
your-org/grafana-manifests
└── shared/
    ├── dashboard-global.json
    ├── dashboard-metrics.json
    └── dashboard-logs.json
```

**In Grafana Dashboards view (all regions):**

```
Dashboards
└── 📁 grafana-manifests/
    ├── Global Dashboard
    ├── Metrics Dashboard
    └── Logs Dashboard
```

- All regional instances (US, EU, etc.) show identical folder structure
- Same folder name "grafana-manifests" in every region
- Same dashboards synced from the `shared/` path appear everywhere
- Users in any region see the exact same dashboards with the same titles

## Configuration parameters

All regions:

- Repository: `your-org/grafana-manifests`
- Branch: `main`
- Path: `shared/`

## How it works

1. All regional instances pull dashboards from `shared/`.
2. Any region’s change commits to Git.
3. Other regions pull updates during the next sync (or via webhooks).
4. Changes propagate across regions per sync interval.

## Considerations

- **Write conflicts**: If users in different regions modify the same dashboard simultaneously, Git uses last-write-wins.
- **Primary region**: Consider designating one region as the primary location for making dashboard changes.
- **Propagation time**: Changes propagate to all regions within the configured sync interval, or instantly if webhooks are configured.
- **Network reliability**: Ensure all regions have reliable connectivity to the Git repository.

# Git Sync in a Grafana instance shared by multiple teams

Use a single Grafana instance with multiple Repository resources, one per team. Each team manages its own dashboards while sharing Grafana.

## Use it for

- **Team autonomy**: Different teams manage their own dashboards independently.
- **Organizational structure**: Dashboard organization aligns with team structure.
- **Resource efficiency**: Multiple teams share Grafana infrastructure.
- **Cost optimization**: You reduce infrastructure costs while maintaining team separation.
- **Collaboration**: Teams can view each other’s dashboards while managing their own.

## Architecture

```
┌─────────────────────────┐  ┌─────────────────────────┐
│  Platform Team Repo     │  │  Data Team Repo         │
│  platform-dashboards    │  │  data-dashboards        │
│                         │  │                         │
│  platform-dashboards/   │  │  data-dashboards/       │
│  └── grafana/           │  │  └── grafana/           │
│      ├── k8s.json       │  │      ├── pipeline.json  │
│      └── infra.json     │  │      └── analytics.json │
└─────────────────────────┘  └─────────────────────────┘
           ↕                            ↕
    Git Sync (grafana/)          Git Sync (grafana/)
           ↕                            ↕
        ┌──────────────────────────────────────┐
        │       Grafana Instance               │
        │                                      │
        │  Repository 1:                       │
        │  - repo: platform-dashboards         │
        │  → Creates "platform-dashboards"     │
        │                                      │
        │  Repository 2:                       │
        │  - repo: data-dashboards             │
        │  → Creates "data-dashboards"         │
        └──────────────────────────────────────┘
```

## Repository structure

**In Git (separate repositories):**

**Platform team repository:**

```
your-org/platform-dashboards
└── grafana/
    ├── dashboard-k8s.json
    └── dashboard-infra.json
```

**Data team repository:**

```
your-org/data-dashboards
└── grafana/
    ├── dashboard-pipeline.json
    └── dashboard-analytics.json
```

**In Grafana Dashboards view:**

```
Dashboards
├── 📁 platform-dashboards/
│   ├── Kubernetes Dashboard
│   └── Infrastructure Dashboard
└── 📁 data-dashboards/
    ├── Pipeline Dashboard
    └── Analytics Dashboard
```

- Two separate folders created (one per Repository resource).
- Folder names derived from repository names.
- Each team has complete control over their own repository.
- Teams can independently manage permissions, branches, and workflows in their repos.
- All teams can view each other's dashboards in Grafana but manage only their own.

## Configuration parameters

**Platform team repository:**

- **Repository**: `your-org/platform-dashboards`
- **Branch**: `main`
- **Path**: `grafana/`

**Data team repository:**

- **Repository**: `your-org/data-dashboards`
- **Branch**: `main`
- **Path**: `grafana/`

## How it works

1. Each team has their own Git repository for complete autonomy.
2. Each repository resource in Grafana creates a separate folder.
3. Platform team dashboards sync from `your-org/platform-dashboards` repository.
4. Data team dashboards sync from `your-org/data-dashboards` repository.
5. Teams can independently manage their repository settings, access controls, and workflows.
6. All teams can view each other's dashboards in Grafana but edit only their own.

## Scale to more teams

Adding additional teams is straightforward. For a third team, create a new repository and configure:

- **Repository**: `your-org/security-dashboards`
- **Branch**: `main`
- **Path**: `grafana/`

This creates a new "security-dashboards" folder in the same Grafana instance.

## Alternative: Shared repository with different paths

For teams that prefer sharing a single repository, use different paths to separate team dashboards:

**In Git:**

```
your-org/grafana-manifests
├── team-platform/
│   ├── dashboard-k8s.json
│   └── dashboard-infra.json
└── team-data/
    ├── dashboard-pipeline.json
    └── dashboard-analytics.json
```

**Configuration:**

**Platform team:**

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `main`
- **Path**: `team-platform/`

**Data team:**

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `main`
- **Path**: `team-data/`

This approach provides simpler repository management but less isolation between teams.

## Alternative: Different branches per team

For teams wanting their own branch in a shared repository:

**Platform team:**

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `team-platform`
- **Path**: `grafana/`

**Data team:**

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `team-data`
- **Path**: `grafana/`

This allows teams to use Git branch workflows for collaboration while sharing the same repository.

# Single instance Git Sync

Use a single Grafana instance synchronized with a Git repository. This is the foundation for Git Sync and helps you understand bidirectional synchronization.

## Use it for

- **Getting started**: You want to learn how Git Sync works before implementing complex scenarios.
- **Personal projects**: Individual developers manage their own dashboards.
- **Small teams**: You have a simple setup without multiple environments or complex workflows.
- **Development environments**: You need quick prototyping and testing.

## Architecture

```
┌─────────────────────────────────────────────────────┐
│           GitHub Repository                         │
│   Repository: your-org/grafana-manifests          │
│   Branch: main                                      │
│                                                     │
│   grafana-manifests/                              │
│   └── grafana/                                     │
│       ├── dashboard-1.json                         │
│       ├── dashboard-2.json                         │
│       └── dashboard-3.json                         │
└─────────────────────────────────────────────────────┘
                        ↕
              Git Sync (bidirectional)
                        ↕
        ┌─────────────────────────────┐
        │    Grafana Instance         │
        │                             │
        │  Repository Resource:       │
        │  - url: grafana-manifests  │
        │  - branch: main             │
        │  - path: grafana/           │
        │                             │
        │  Creates folder:            │
        │  "grafana-manifests"       │
        └─────────────────────────────┘
```

## Repository structure

**In Git:**

```
your-org/grafana-manifests
└── grafana/
    ├── dashboard-1.json
    ├── dashboard-2.json
    └── dashboard-3.json
```

**In Grafana Dashboards view:**

```
Dashboards
└── 📁 grafana-manifests/
    ├── Dashboard 1
    ├── Dashboard 2
    └── Dashboard 3
```

- A folder named "grafana-manifests" (from repository name) contains all synced dashboards.
- Each JSON file becomes a dashboard with its title displayed in the folder.
- Users browse dashboards organized under this folder structure.

## Configuration parameters

Configure your Grafana instance to synchronize with:

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `main`
- **Path**: `grafana/`

## How it works

1. **From Grafana to Git**: When users create or modify dashboards in Grafana, Git Sync commits changes to the `grafana/` directory on the `main` branch.
2. **From Git to Grafana**: When dashboard JSON files are added or modified in the `grafana/` directory, Git Sync pulls these changes into Grafana.

# Testing sandboxes with Git Sync

Use a sandbox branch that users can push to directly, without opening a pull request for every change. A sandbox branch is a low-friction space for short-lived projects and experimentation, while keeping every dashboard version-controlled in Git. When work is ready, you promote it to the `main` branch.

A sandbox is a branch in the same repository, synced to Grafana with the [`write` workflow](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/set-up-code/#configuration-parameters) enabled. With the `write` workflow, saving a dashboard commits directly to the sandbox branch, so the editing experience feels close to using Grafana without Git Sync. The difference is that every change is still committed to Git, so nothing is lost and history is preserved.

This scenario uses a single **shared** sandbox branch that everyone works in. A shared sandbox uses only one Repository connection regardless of how many users work in it, which keeps you well within the per-stack limit of 10 repository connections. If you need stricter isolation, you can instead give each user their own sandbox branch. Refer to [Individual or shared sandboxes](#individual-or-shared-sandboxes).

## Use it for

- **Short-lived projects**: You need a temporary space for work that may or may not reach production.
- **Shared experimentation**: A team experiments freely in one space without affecting shared production dashboards.
- **Fast iteration**: You want the low-friction editing of a non-provisioned instance, but with the safety of Git history.
- **Staged promotion**: You develop in a sandbox, then promote finished work to the `main` branch.

## Architecture

```
┌────────────────────────────────────────────────────────────┐
│              GitHub Repository                             │
│   Repository: your-org/grafana-manifests                 │
│                                                            │
│   Branch: main       ← Promoted, reviewed work            │
│   └── grafana/                                             │
│       ├── dashboard-stable.json                            │
│       └── dashboard-approved.json                          │
│                                                            │
│   Branch: sandbox    ← Direct pushes, shared experiments  │
│   └── grafana/                                             │
│       ├── alice/dashboard-wip.json                         │
│       └── bob/dashboard-test.json                          │
└────────────────────────────────────────────────────────────┘
                        ↕
              Git Sync (write workflow)
                        ↕
              ┌─────────────────────────┐
              │    Grafana Instance     │
              │                         │
              │  Repository Resource:   │
              │  - branch: sandbox      │
              │  - path: grafana/       │
              │  - workflows: [write]   │
              │                         │
              │  Save = direct commit   │
              │  to sandbox             │
              └─────────────────────────┘
```

## Repository structure

**In Git:**

```
your-org/grafana-manifests
├── (branch: main)
│   └── grafana/
│       ├── dashboard-stable.json
│       └── dashboard-approved.json
└── (branch: sandbox)
    └── grafana/
        ├── alice/
        │   └── dashboard-wip.json
        └── bob/
            └── dashboard-test.json
```

**In Grafana Dashboards view (synced to `sandbox`):**

```
Dashboards
└── 📁 grafana-manifests/
    ├── 📁 alice/
    │   └── Work in Progress Dashboard
    └── 📁 bob/
        └── Test Dashboard
```

- A folder named "grafana-manifests" (from the repository name) contains the dashboards synced from the sandbox branch.
- Users can create subfolders inside it, such as `alice/` and `bob/`, to keep their work separate within the shared branch. Subfolders map to subdirectories under the synced path, so no extra Repository connection is needed.
- Only dashboards on the configured branch and path appear in the instance.
- Saving a dashboard commits directly to the sandbox branch.

## Configuration parameters

Configure the Repository resource to sync with the sandbox branch and enable the `write` workflow:

- **Repository**: `your-org/grafana-manifests`
- **Branch**: `sandbox`
- **Path**: `grafana/`
- **Workflows**: `write`

In the Repository resource, this looks like:

```yaml
spec:
  github:
    url: 'https://github.com/your-org/grafana-manifests'
    branch: 'sandbox'
    path: grafana/
  workflows:
    - write
```

{{< admonition type="note" >}}

If you enable only the `write` workflow, saving a dashboard always commits directly to the sandbox branch. To also allow users to open pull requests from Grafana, add the `branch` workflow: `workflows: [write, branch]`. Refer to [Configuration parameters](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/set-up-code/#configuration-parameters) for details.

{{< /admonition >}}

## How it works

1. Create a sandbox branch in your repository, for example `sandbox`.
2. Configure a Repository resource that syncs that branch with the `write` workflow enabled.
3. Users create and edit dashboards in Grafana, optionally in their own subfolder. Each save commits directly to the sandbox branch.
4. The experience feels similar to using Grafana without Git Sync, but every change is committed to Git.
5. When work is ready, you promote it to the `main` branch. Refer to [Promote a sandbox to main](#promote-a-sandbox-to-main).

## Individual or shared sandboxes

A sandbox can be shared by everyone or scoped to a single user. Each sandbox is a branch synced through its own Repository connection, so the choice is shaped by the per-stack limit of 10 repository connections. Refer to [Usage and performance limitations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/usage-limits/) for details.

- **Shared sandbox (recommended)**: Use a single branch, such as `sandbox`, synced through one Repository connection that all users share. This uses only one connection regardless of how many users work in it, so it scales to any number of users. Separate users' work with subfolders (for example, `alice/`, `bob/`) if you want some organization within the shared branch. The trade-off is less isolation, since everyone commits to the same branch. This is the setup described above.
- **Individual sandboxes**: Give each user their own branch, such as `sandbox/alice` and `sandbox/bob`, each synced through a separate Repository connection. This provides the most isolation, but every sandbox consumes one of your 10 connections, so per-user sandboxes don't scale to many users. Reserve them for a small number of active users or short-lived projects, and delete the branch and its connection when the work is done to free the connection.

Both approaches promote to `main` the same way. Refer to [Promote a sandbox to main](#promote-a-sandbox-to-main).

## Protect sandbox branches

Even though sandbox branches allow direct pushes, protect them against destructive operations. We recommend blocking force pushes on sandbox branches in your Git provider.

Force pushes rewrite branch history and can silently discard commits that Git Sync relies on. Blocking them keeps history append-only, so Grafana and Git stay consistent and you can always trace how a dashboard reached its current state.

- **GitHub**: Add a branch protection rule (or ruleset) for the sandbox branch (for example, `sandbox` or the pattern `sandbox/*` if you use individual sandboxes) and disable **Allow force pushes**.
- **GitLab**: Set the sandbox branches as protected and disallow force push.
- **Bitbucket**: Add a branch permission for the sandbox branch that prevents rewriting history.

You typically keep pull requests optional on sandbox branches so users can push directly, while still blocking force pushes.

## Promote a sandbox to main

When sandbox work is ready, promote it to the `main` branch. Because everything already lives in Git, promotion is a standard Git operation. Choose the option that fits your review requirements:

### Option 1: Open a pull request from the sandbox branch (recommended)

Use your Git provider to open a pull request from the sandbox branch into `main`.

1. In your Git provider, open a pull request from `sandbox` into `main`.
2. Review the changes as a normal Git diff and request approvals as needed.
3. Merge the pull request. The instance syncing `main` picks up the promoted dashboards on its next sync.

This keeps a full review and audit trail for everything that reaches `main`, and is the recommended path when `main` feeds production.

If you also enable the `branch` workflow on the sandbox Repository resource, users can open these pull requests directly from Grafana when they save, instead of switching to the Git provider.

### Option 2: Cherry-pick or merge selected changes

When only some dashboards are ready, promote them selectively instead of merging the whole branch.

1. Cherry-pick the relevant commits from the sandbox branch onto a branch based on `main`, or copy the specific dashboard files.
2. Open a pull request into `main` and review it.
3. Merge to complete the promotion.

Use this when a shared sandbox contains a mix of finished and experimental work.

### Option 3: Promote through a separate instance or path

When `main` is synced to a production instance, treat the sandbox as the development stage of a promotion flow.

1. Develop in the sandbox instance (synced to the sandbox branch with the `write` workflow).
2. Promote approved dashboards to `main` using a pull request (Option 1) or by copying files into the `main` path.
3. The production instance syncing `main` updates automatically.

This mirrors the [development and production environments](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/scenarios/dev-prod/) scenario, with the sandbox branch acting as the development stage.

### Option 4: Promote from the Grafana UI

If the `main` branch is synced to a folder in the same instance (through a second Repository connection) or to a separate instance, users can move a dashboard into it directly from Grafana, without touching Git:

- **Save a copy**: Open the sandbox dashboard, select **Save as** (save a copy), and choose the folder backed by the `main` branch as the destination. Git Sync commits the copy to `main`.
- **Import dashboard**: Export the dashboard from the sandbox (**Export** as JSON), then use **Dashboards > New > Import** and save it into the folder backed by the `main` branch. Git Sync commits it to `main`.

This suits users who prefer working entirely in the UI. Note that these paths commit to `main` without a pull request, so use them only when `main` allows direct writes; when `main` requires review, prefer Option 1.

After promotion, you can delete short-lived sandbox branches, or keep a long-lived shared sandbox and continue iterating.

## Learn more

Refer to the following documents to learn more:

- [Git Sync for development and production environments](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/scenarios/dev-prod/)
- [Git Sync key concepts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/key-concepts/)
- [Configure Git Sync as code](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/set-up-code/)
- [Manage provisioned dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/provisioned-dashboards/)
- [Usage and performance limitations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/usage-limits/)

# Manage Grafana dashboards with GitOps using ArgoCD

This guide shows you how to set up a continuous deployment pipeline using ArgoCD to synchronize your Grafana dashboards with a Git repository. You'll use the Grafana Dashboard Custom Resource provided by the Grafana Operator to manage dashboard configurations declaratively.

## Prerequisites

Before you begin, make sure you have the following:

- An existing Grafana Cloud stack
- A Kubernetes cluster with Grafana Operator installed, as shown in [Grafana Operator Installation](/docs/grafana-cloud/as-code/grafana-operator/#installing-the-grafana-operator)
- ArgoCD installed on your Kubernetes cluster. Refer to the [ArgoCD Installation Guide](https://argo-cd.readthedocs.io/en/stable/getting_started/)
- A Git repository to store your dashboard configurations

## Set up your Git repository

Create a directory structure in your repository to organize your Grafana and dashboard configurations. For this tutorial, create a folder named `grafana`.

## Set up the Grafana Operator

The Grafana Operator allows you to authenticate with the Grafana instance using the Grafana Custom Resource (CR).

### Create the Grafana API Token Secret

Store the Grafana API Token in a secret. Create a file named `grafana-token.yml` in the `grafana` folder in your Git repository:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: grafana-cloud-credentials
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
stringData:
  GRAFANA_CLOUD_INSTANCE_TOKEN: '<GRAFANA_API_KEY>'
type: Opaque
```

Replace the placeholders with your values:

- _`<GRAFANA_API_KEY>`_: API key from your Grafana instance. To create an API key, refer to [Grafana API Key Documentation](/docs/grafana/latest/administration/service-accounts/migrate-api-keys/)
- _`<GRAFANA_OPERATOR_NAMESPACE>`_: Namespace where the `grafana-operator` is deployed in your Kubernetes cluster

### Configure the Grafana Custom Resource

Set up the connection to your Grafana Cloud instance. Create a file named `grafana-cloud.yml` in the `grafana` folder in your Git repository:

```yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: '<GRAFANA_CLOUD_STACK_NAME>'
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
  labels:
    dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
spec:
  external:
    url: https://<GRAFANA_CLOUD_STACK_NAME>.grafana.net/
    apiKey:
      name: grafana-cloud-credentials
      key: GRAFANA_CLOUD_INSTANCE_TOKEN
```

Replace the placeholders with your values:

- _`<GRAFANA_CLOUD_STACK_NAME>`_: Name of your Grafana Cloud Stack
- _`<GRAFANA_OPERATOR_NAMESPACE>`_: Namespace where the `grafana-operator` is deployed in your Kubernetes cluster

## Add dashboards to your Git repository

In your `grafana` directory, create a sub-folder called `dashboards`.

This guide shows you how to create three separate dashboards. For all dashboard configurations, replace the placeholders with your values:

- _`<GRAFANA_CLOUD_STACK_NAME>`_: Name of your Grafana Cloud Stack
- _`<GRAFANA_OPERATOR_NAMESPACE>`_: Namespace where the `grafana-operator` is deployed in your Kubernetes cluster

### Create a simple dashboard

Under the `dashboards` folder, create a file named `simple-dashboard.yaml`:

```yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: grafanadashboard-sample
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
spec:
  resyncPeriod: 30s
  instanceSelector:
    matchLabels:
      dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
  json: >
    {
    "id": null,
    "title": "Simple Dashboard",
    "tags": [],
    "style": "dark",
    "timezone": "browser",
    "editable": true,
    "hideControls": false,
    "graphTooltip": 1,
    "panels": [],
    "time": {
        "from": "now-6h",
        "to": "now"
    },
    "timepicker": {
        "time_options": [],
        "refresh_intervals": []
    },
    "templating": {
        "list": []
    },
    "annotations": {
        "list": []
    },
    "refresh": "5s",
    "schemaVersion": 17,
    "version": 0,
    "links": []
    }
```

### Create a dashboard from ConfigMap

Under the `dashboards` folder, create a file named `dashboard-from-cm.yaml`:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: dashboard-definition
  namespace: <GRAFANA_OPERATOR_NAMESPACE>
data:
  json: >
    {
    "id": null,
    "title": "Simple Dashboard from ConfigMap",
    "tags": [],
    "style": "dark",
    "timezone": "browser",
    "editable": true,
    "hideControls": false,
    "graphTooltip": 1,
    "panels": [],
    "time": {
        "from": "now-6h",
        "to": "now"
    },
    "timepicker": {
        "time_options": [],
        "refresh_intervals": []
    },
    "templating": {
        "list": []
    },
    "annotations": {
        "list": []
    },
    "refresh": "5s",
    "schemaVersion": 17,
    "version": 0,
    "links": []
    }
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: grafanadashboard-from-configmap
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
spec:
  instanceSelector:
    matchLabels:
      dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
  configMapRef:
    name: dashboard-definition
    key: json
```

### Create a dashboard from Grafana.com

Under the `dashboards` folder, create a file named `dashboard-from-id.yaml`:

```yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: node-exporter-latest
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
spec:
  instanceSelector:
    matchLabels:
      dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
  grafanaCom:
    id: 1860
```

## Configure ArgoCD to sync the Git repository

After you commit all changes to Git, log in to the ArgoCD user interface or use the CLI.

### Create an ArgoCD application

**Using the UI:**

1. Navigate to **New App** and complete the form with your Git repository details and the path to your `grafana` folder
2. Enable **Directory Recurse**
3. Set the sync policy to **Automatic**

**Using the CLI:**

Prepare an application manifest named `argo-application.yaml`:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: Grafana
  namespace: '<ARGOCD_NAMESPACE>'
spec:
  destination:
    name: ''
    namespace: ''
    server: 'https://kubernetes.default.svc'
  source:
    path: '<PATH_TO_GRAFANA_FOLDER>'
    repoURL: '<GIT_REPO_URL>'
    targetRevision: HEAD
    directory:
      recurse: true
  sources: []
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 2
    backoff:
      duration: 5s
      maxDuration: 3m0s
      factor: 2
```

Replace the placeholders with your values:

- _`<GIT_REPO_URL>`_: URL of your Git repository
- _`<PATH_TO_GRAFANA_FOLDER>`_: Path to the `grafana` folder in your repository
- _`<ARGOCD_NAMESPACE>`_: Namespace where ArgoCD is deployed in your Kubernetes cluster

Create the application in ArgoCD:

```sh
kubectl apply -f argo-application.yaml
```

## Verify sync status in ArgoCD

1. Monitor the newly created ArgoCD application to ensure it successfully syncs your dashboard configuration
2. Visit the ArgoCD dashboard and check the sync status. If it's successful, your Grafana dashboards should be up to date with the configuration from your Git repository

## Update your dashboards

To update an existing dashboard:

1. Make changes to the dashboard JSON configuration in your Git repository
2. Commit and push the changes
3. ArgoCD detects the update and synchronizes the changes to your Custom Resource
4. Grafana Operator then syncs changes to the Grafana instance

### Validate your dashboard updates

Log in to your Grafana dashboard and confirm that the changes are applied. You should see the dashboard updates reflected in the Grafana UI.

## Next steps

You've successfully set up a GitOps workflow to manage Grafana dashboards using ArgoCD and the Grafana Operator. Your dashboards are now version-controlled and can be consistently deployed across environments. This approach provides a reliable and auditable way to manage observability dashboards and scale your operations.

To learn more about managing Grafana using Grafana Operator:

- [Grafana Operator documentation](https://grafana.github.io/grafana-operator/docs/)
- [Grafana dashboard provisioning](/docs/grafana/latest/administration/provisioning/#dashboards)
- [ArgoCD best practices](https://argo-cd.readthedocs.io/en/stable/user-guide/best_practices/)

### Additional considerations

- You can install the Grafana Operator's Helm Chart using ArgoCD to manage your setup with GitOps
- You can follow a similar setup for Grafana Folders and other resources

# Manage folders, data sources, and dashboards using the Grafana Operator

This guide shows you how to manage data sources, folders, and dashboards using the Grafana Operator. You'll create these resources declaratively using Kubernetes custom resources.

## Prerequisites

Before you begin, make sure you have the following:

- An existing Grafana Cloud stack
- Grafana Operator installed in your cluster, as shown in [Grafana Operator Installation](/docs/grafana-cloud/as-code/grafana-operator/#installing-the-grafana-operator)

## Set up the Grafana Operator

The Grafana Operator allows you to authenticate with your Grafana instance using the Grafana Custom Resource (CR).

### Create the Grafana API Token Secret

Store the Grafana API Token in a secret with the following content in a file named `grafana-token.yml`:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: grafana-cloud-credentials
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
stringData:
  GRAFANA_CLOUD_INSTANCE_TOKEN: '<GRAFANA_API_KEY>'
type: Opaque
```

Replace the placeholders with your values:

- _`<GRAFANA_API_KEY>`_: API key from your Grafana instance. To create an API key, refer to [Grafana API Key Documentation](/docs/grafana/latest/administration/service-accounts/migrate-api-keys/)
- _`<GRAFANA_OPERATOR_NAMESPACE>`_: Namespace where the `grafana-operator` is deployed in your Kubernetes cluster

### Configure the Grafana Custom Resource

Set up connection to your Grafana Cloud instance. Create a file named `grafana-cloud.yml`:

```yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: '<GRAFANA_CLOUD_STACK_NAME>'
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
  labels:
    dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
spec:
  external:
    url: https://<GRAFANA_CLOUD_STACK_NAME>.grafana.net/
    apiKey:
      name: grafana-cloud-credentials
      key: GRAFANA_CLOUD_INSTANCE_TOKEN
```

Replace the placeholders with your values:

- _`<GRAFANA_CLOUD_STACK_NAME>`_: Name of your Grafana Cloud stack
- _`<GRAFANA_OPERATOR_NAMESPACE>`_: Namespace where the `grafana-operator` is deployed in your Kubernetes cluster

## Add a data source

{{< admonition type="note" >}}

This example uses the Prometheus data source. Note that the required arguments vary depending on the data source you select.

{{< /admonition >}}

### Create a data source configuration

Create and save a new YAML file `datasource.yml` with your data source's configuration:

```yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: '<DATA_SOURCE_NAME>'
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
spec:
  instanceSelector:
    matchLabels:
      dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
  allowCrossNamespaceImport: true
  datasource:
    access: proxy
    database: prometheus
    jsonData:
      timeInterval: 5s
      tlsSkipVerify: true
    name: '<DATA_SOURCE_NAME>'
    type: prometheus
    url: '<DATA_SOURCE_URL>'
```

Replace the placeholders with your values:

- _`<DATA_SOURCE_NAME>`_: Name of the data source to be added in Grafana
- _`<DATA_SOURCE_URL>`_: URL of your data source
- _`<GRAFANA_CLOUD_STACK_NAME>`_: Name of your Grafana Cloud stack
- _`<GRAFANA_OPERATOR_NAMESPACE>`_: Namespace where the `grafana-operator` is deployed in your Kubernetes cluster

### Add a dashboard to a folder

Use the following YAML definition to create a simple dashboard in the Grafana instance under a custom folder. If the folder defined under the `spec.folder` field doesn't exist, the operator creates it before placing the dashboard inside the folder.

Prepare the dashboard configuration. In `dashboard.yml`, define the dashboard and assign it to a folder:

```yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: '<FOLDER_NAME>'
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
spec:
  instanceSelector:
    matchLabels:
      dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
  folder: '<FOLDER_NAME>'
  json: >
    {
      "title": "as-code dashboard",
      "uid" : "ascode"
    }
```

Replace the placeholders with your values:

- _`<FOLDER_NAME>`_: Name of the folder in which you want the dashboard to be created
- _`<GRAFANA_CLOUD_STACK_NAME>`_: Name of your Grafana Cloud stack
- _`<GRAFANA_OPERATOR_NAMESPACE>`_: Namespace where the `grafana-operator` is deployed in your Kubernetes cluster

## Apply the Kubernetes manifests

In a terminal, run the following commands from the directory where all of the above Kubernetes YAML definitions are located.

Create Kubernetes Custom resources for all of the configurations:

```sh
kubectl apply -f grafana-token.yml grafana-cloud.yml datasource.yml dashboard.yml
```

## Validate your configuration

After you apply the configurations, verify that:

- A new data source is visible in Grafana. In the following image, a data source named `InfluxDB` was created.

  ![InfluxDB datasource](/static/img/docs/grafana-cloud/terraform/influxdb_datasource_tf.png)

- A new dashboard and folder have been created in Grafana. In the following image, a dashboard named `InfluxDB Cloud Demos` was created inside the `Demos` folder.

  ![InfluxDB dashboard](/static/img/docs/grafana-cloud/grizzly/grizzly-folder-dashboard-datasource.png)

## Next steps

You've successfully created a data source, folder, and dashboard using the Grafana Operator. Your Grafana resources are now managed declaratively through Kubernetes custom resources.

To learn more about managing Grafana:

- [Grafana Operator documentation](https://grafana.github.io/grafana-operator/docs/)
- [Grafana dashboard provisioning](/docs/grafana/latest/administration/provisioning/#dashboards)
- [Grafana data source provisioning](/docs/grafana/latest/administration/provisioning/#data-sources)

# Set up file provisioning

{{< admonition type="note" >}}

On-prem file provisioning is available in Grafana v12 and later for open source and Enterprise editions. It's **not available in Grafana Cloud**. This new file provisioning feature is only available for dashboards and doesn't replace classic provisioning for the time being.

For classic provisioning using configuration files refer to [Provision Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/).

{{< /admonition >}}

Use local file provisioning to include in your Grafana instance resources that you store in your local file system.

The local path mount is referred to as a `repository`. File provisioning uses the same `Repository` resource as [Git Sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/), with the repository `type` set to `local` and a `path` that points to a directory on the Grafana server's file system.

Using the local path lets you also use it with a tool like `fuse`, allowing you to mount S3 buckets as local paths. You can also use tools like `restic` to automatically back up your dashboards to your preferred backup storage solution.

You can set up file provisioning in two ways:

- [Using the UI](#set-up-file-provisioning-using-the-ui) through **Administration** > **Provisioning**.
- [As code](#set-up-file-provisioning-as-code) with `gcx`, the Grafana CLI, so you can manage the configuration in a GitOps-style workflow.

Both approaches require you to first [configure the permitted paths](#configure-permitted-paths) in the Grafana configuration file.

{{< admonition type="note" >}}

The UI refers to this feature as _file provisioning_ and to the storage location as _local storage_. In the `Repository` resource, the corresponding repository `type` value is `local` and the path is set in `spec.local.path`.

{{< /admonition >}}

## New file provisioning capabilities

Local file provisioning using **Administration** > **Provisioning** will eventually replace the traditional methods Grafana has used for referencing local file systems for dashboard files.

{{< admonition type="note" >}}
For production systems, use the `folderFromFilesStructure` capability instead of **Administration** > **Provisioning** to include dashboards from a local file system in your Grafana instance. Refer to [Provision Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#provision-folders-structure-from-filesystem-to-grafana) for more information.
{{< /admonition >}}

## Before you begin

To set up file provisioning, you need:

- Administration rights in your Grafana organization.
- A local directory to store your dashboards. If you want to use a GitHub repository, refer to [Set up Git Sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/git-sync-setup/).
- To update the `permitted_provisioning_paths` section of `custom.ini`.
- To understand the limitations of local file provisioning. Refer to [How it works](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/provision-resources#how-it-works) for more details.
- To set up file provisioning as code, you also need `gcx` configured with your Grafana instance details. Refer to the [Grafana CLI documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/ai-tools/grafana-cli/) for setup instructions.

## Configure permitted paths

The `provisioning` feature toggle is enabled by default in Grafana Cloud and, starting in Grafana v13, for OSS and Enterprise as well. No manual configuration of the feature toggle is required.

The local setting must be a relative path and its relative path must be configured in the `permitted_provisioned_paths` configuration option.
The configuration option is relative to your working directory, i.e. where you are running Grafana from; this is usually `/usr/share/grafana` or similar.

Local file paths can point to any directory that is permitted by the configuration.
The default paths is `devenv/dev-dashboards` and `conf/provisioning` in your `grafana` installation directory.

The path must behave as a standard file directory on the system of choice.
Any subdirectories are automatically included.

The values that you enter for the `permitted_provisioning_paths` become the base paths for those entered when you enter a local path in the **Connect to local storage** wizard or in the `spec.local.path` field of a `Repository` resource.

1. Open your Grafana configuration file, either `grafana.ini` or `custom.ini`. For file location based on operating system, refer to [Configuration file location](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/).
1. Locate or add a `[paths]` section. To add more than one location, use the pipe character (`|`) to separate the paths. The list should not include empty paths or trailing pipes. Add these values:

   ```ini
   [paths]
   ; This is devenv/dev-dashboards and conf/provisioning by default.
   permitted_provisioning_paths = grafana/ | /etc/grafana/provisioning/
   ```

1. Save the changes to the file and start Grafana.

## Set up file provisioning using the UI

To use file-based provisioning, you need the file path to the `grafana` folder where your dashboards are stored in the repository.

To start setting up file-based provisioning:

1. Log in to your Grafana server with an account that has the Grafana Admin flag set.
1. Select **Administration** in the left-side menu and then **Provisioning**.
1. Select [Configure file provisioning](#set-up-file-provisioning-using-the-ui).

### Connect to local storage

The local path can point to any directory that is permitted by the configuration.
Refer to [Configure permitted paths](#configure-permitted-paths) for information.

The starting path is always your working `grafana` directory.
The prefix that must be entered is determined by the locations configured in `permitted_provisioning_paths`.
The default paths are `devenv/dev-dashboards` and `conf/provisioning` in your `grafana` installation directory.
The value you enter in the Grafana UI must _begin_ with any of the configured values. For example, `conf/provisioning/test` is valid, but `conf/test` is not.

1. Enter the **Local path**, for example `grafana/`. This must begin with any of the configured `permitted_provisioned_paths`.
1. Select **Choose what to synchronize**.

The set up process verifies the path and provides an error message if a problem occurs.

### Choose what to synchronize

#### Synchronization limitations

To have access to full instance sync you must explicitly enable the option.

The following applies:

- You won't be able to create new alerts or library panels after setup is completed.
- If you opted for full instance sync and want to use alerts and library panels, you'll have to delete the provisioned repository and connect again with folder sync.

#### Set up synchronization

You can sync external resources into a new folder without affecting the rest of your instance.

To set up synchronization:

1. Select which resources you want to sync.

1. Enter a **Display name** for the repository connection. Resources stored in this connection appear under the chosen display name in the Grafana UI.

1. Click **Synchronize** to continue.

1. You can repeat this process for up to 10 connections.

{{< admonition type="note" >}}

Optionally, you can export any unmanaged resources into the provisioned folder. See how in [Synchronize with external storage](#synchronize-with-external-storage).

{{< /admonition >}}

### Synchronize with external storage

In this step you proceed to synchronize the resources selected in the previous step. Optionally, you can check the **Migrate existing resources** box to migrate your unmanaged dashboards to the provisioned folder.

Select **Begin synchronization** to start the process. After this one time step, all future updates are automatically saved to the local file path and provisioned back to the instance.

Note that during the initial synchronization, your dashboards will be temporarily unavailable. No data or configurations will be lost.
How long the process takes depends upon the number of resources involved.

### Choose additional settings

If you wish, you can make any files synchronized as as **Read only** so no changes can be made to the resources through Grafana.
Any resources made outside of Grafana and saved to the local repository will be reflected in Grafana.

Select **Finish**.

## Set up file provisioning as code

Because file provisioning is managed as code using the same `Repository` Custom Resource Definition (CRD) as Git Sync, you can define it in a YAML file and push it to Grafana with `gcx`. This approach enables automated, GitOps-style workflows for managing file provisioning instead of using the Grafana UI.

To set up file provisioning as code with `gcx`, follow these steps:

1. [Create the repository resource](#create-the-repository-resource).
1. [Push the resource to Grafana](#push-the-resource-to-grafana).
1. [Manage the repository resource](#manage-the-repository-resource).
1. [Verify your dashboards in Grafana](#verify-your-dashboards-in-grafana).

### Create the repository resource

Create a `repository.yaml` file defining your file provisioning configuration. Set the repository `type` to `local` and use `spec.local.path` for the directory that holds your dashboards.

```yaml
apiVersion: provisioning.grafana.app/v0alpha1
kind: Repository
metadata:
  name: '<REPOSITORY_NAME>'
spec:
  title: '<REPOSITORY_TITLE>'
  type: local
  local:
    path: '<LOCAL_PATH>'
  sync:
    enabled: true
    intervalSeconds: 60
    target: folder
  workflows:
    - write
```

Replace the placeholders with your values:

- _`<REPOSITORY_NAME>`_: Unique identifier for this repository resource
- _`<REPOSITORY_TITLE>`_: Human-readable name displayed in the Grafana UI
- _`<LOCAL_PATH>`_: Path to the directory that holds your dashboards. It must begin with one of the configured `permitted_provisioning_paths`, for example `grafana/`.

{{< admonition type="note" >}}

File provisioning supports two sync targets: `target: folder` (the default) creates a folder named after the repository and places synced resources inside it, while `target: folderless` places synced resources at the top level without creating a wrapper folder. Refer to [Sync targets](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/git-sync/key-concepts/#sync-targets) for details.

{{< /admonition >}}

### Configuration parameters

The following configuration parameters are available:

| Field                       | Description                                                                    |
| --------------------------- | ------------------------------------------------------------------------------ |
| `metadata.name`             | Unique identifier for this repository resource                                 |
| `spec.title`                | Human-readable name displayed in the Grafana UI                                |
| `spec.type`                 | Repository type. Set to `local` for file provisioning                          |
| `spec.local.path`           | Path to the directory that holds your dashboards, within a permitted path      |
| `spec.sync.enabled`         | Enable synchronization (true/false)                                            |
| `spec.sync.intervalSeconds` | Sync interval in seconds                                                       |
| `spec.sync.target`          | Where to place synced dashboards (`folder` or `folderless`)                    |
| `spec.workflows`            | Enabled workflows: `write` (direct changes). Leave empty for a read-only mount |

### Push the resource to Grafana

Before pushing any resources, configure `gcx` with your Grafana instance details. Refer to the [Grafana CLI documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/ai-tools/grafana-cli/) for setup instructions.

Push the repository configuration:

```sh
gcx resources push --path <DIRECTORY>
```

The `--path` parameter has to point to the directory containing your `repository.yaml` file.

After pushing, Grafana will:

1. Create the repository resource
1. Read the dashboards from the configured local path
1. Begin syncing at the configured interval

### Manage the repository resource

Use the following commands to manage file provisioning repositories.

To list all repositories:

```sh
gcx resources get repositories
```

To get details for a specific repository:

```sh
gcx resources get repository/<REPOSITORY_NAME>
gcx resources get repository/<REPOSITORY_NAME> -o json
gcx resources get repository/<REPOSITORY_NAME> -o yaml
```

To update a repository:

```sh
gcx resources edit repository/<REPOSITORY_NAME>
```

To delete a repository:

```sh
gcx resources delete repository/<REPOSITORY_NAME>
```

## Verify your dashboards in Grafana

To verify that your dashboards are available at the location that you specified, click **Dashboards**. The name of the dashboard is listed in the **Name** column.

# Create and manage dashboards using Terraform and GitHub Actions

Learn how to create and manage multiple dashboards represented as JSON source code for Grafana using Terraform and GitHub Actions.

## Prerequisites

Before you begin, ensure you have the following:

- A Grafana Cloud account, as shown in [Get started](https://grafana.com/docs/grafana-cloud/get-started/)
- A [GitHub](https://github.com/) repository

## Add dashboards to a GitHub repository

This guide shows you how to add dashboards for ElasticSearch, InfluxDB, and AWS EC2. You can use different dashboards according to your configured data sources.

1. In your GitHub repository, create a folder named `dashboards` in the root directory.

1. In the `dashboards` folder create three sub-folders. For this guide, we will create three sub-folders named `elasticsearch`, `influxdb`, and `aws`.

1. Add dashboard JSON source code to each of the three sub-folders.

## Terraform configuration for Grafana provider

This Terraform configuration configures the [Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) to provide necessary authentication when creating folders and dashboards in the Grafana instance.

1. Create a service account and token in the Grafana instance by following these steps:
   1. [Create a service account in Grafana](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/service-accounts/#create-a-service-account-in-grafana)
   1. [Add a token to a service account](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/service-accounts/#add-a-token-to-a-service-account-in-grafana)

1. Create a file named `main.tf` in the Git root directory and add the following code block:

   ```terraform
   terraform {
      required_providers {
         grafana = {
            source  = "grafana/grafana"
            version = ">= 2.9.0"
         }
      }
   }

   provider "grafana" {
      alias = "cloud"

      url   = "<Grafana-instance-url>"
      auth  = "<Grafana-Service-Account-token>"
   }
   ```

1. Replace the following field values:
   - `<Grafana-instance-url>` with the URL of your Grafana instance, for example `"https://my-stack.grafana.net/"`.
   - `<Grafana-Service-Account-token>` with a Service Account token from the Grafana instance.

## Terraform configuration for folders

This Terraform configuration creates three folders named `ElasticSearch`, `InfluxDB` and `AWS` in the Grafana instance using [grafana_folder (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/folder).

Create a file named `folders.tf` in the Git root directory and add the following code block:

```terraform
resource "grafana_folder" "ElasticSearch" {
  provider = grafana.cloud

  title = "ElasticSearch"
}

resource "grafana_folder" "InfluxDB" {
  provider = grafana.cloud

  title = "InfluxDB"
}

resource "grafana_folder" "AWS" {
  provider = grafana.cloud

  title = "AWS"
}
```

## Terraform configuration for dashboards

This Terraform configuration iterates through the JSON files in the three folders (`elasticsearch`, `influxdb` and `aws`) you created in the GitHub repository and adds them to the respective folders in the Grafana instance using a Grafana dashboard resource. Available resources include:

- [`grafana_dashboard` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/dashboard).
- For Kubernetes-style dashboards in Grafana v13 or later, use the appropriate resource depending on your dashboard's Kubernetes version. Refer to [Grafana resources](https://github.com/grafana/terraform-provider-grafana/blob/main/docs/resources) for details on the available resources, such as `grafana_apps_dashboard_dashboard_v1beta1`, `grafana_apps_dashboard_dashboard_v1`, or `grafana_apps_dashboard_dashboard_v2`.

For example, the dashboard represented as JSON source code in the `elasticsearch` folder in the GitHub repository will be created in the `ElasticSearch` folder in the Grafana instance.

Create a file named `dashboards.tf` in the Git root directory and add the following code block:

```terraform
resource "grafana_dashboard" "elasticsearch" {
  provider = grafana.cloud

  for_each    = fileset("${path.module}/dashboards/elasticsearch", "*.json")
  config_json = file("${path.module}/dashboards/elasticsearch/${each.key}")
  folder      = grafana_folder.ElasticSearch.id
}

resource "grafana_dashboard" "influxdb" {
  provider = grafana.cloud

  for_each    = fileset("${path.module}/dashboards/influxdb", "*.json")
  config_json = file("${path.module}/dashboards/influxdb/${each.key}")
  folder      = grafana_folder.InfluxDB.id
}

resource "grafana_dashboard" "aws" {
  provider = grafana.cloud

  for_each    = fileset("${path.module}/dashboards/aws", "*.json")
  config_json = file("${path.module}/dashboards/aws/${each.key}")
  folder      = grafana_folder.AWS.id
}
```

## GitHub workflow for managing dashboards using Terraform

This GitHub workflow consists of the following steps:

- Using the [actions/checkout@v3](https://github.com/actions/checkout) action, The GitHub repository is checked out so that the GitHub workflow can access it.
- The Terraform CLI is installed on the GitHub runner using the [hashicorp/setup-terraform@v1](https://github.com/hashicorp/setup-terraform) action.
- `terraform init` is run as a bash command in the GitHub runner to initialize a working directory containing Terraform configuration files.
- `terraform fmt -check` is run as a bash command in the GitHub runner to check if the Terraform configuration files are properly formatted. If the Terraform configuration files are not properly formatted, the workflow will fail at this step.
- `terraform plan` is run as a bash command in the GitHub runner to preview the changes that Terraform will make.
- Using [mshick/add-pr-comment@v1](https://github.com/mshick/add-pr-comment) action, the preview from Terraform plan is posted as a comment on the pull request. This helps in reviewing the changes that Terraform will make before the pull request is merged.
- `terraform apply -auto-approve` is run as a bash command in the GitHub runner to apply the Terraform configuration files. `-auto-approve` flag is added to the command to skip interactive approval of plan before applying and make the workflow automated.
  This step is run only when changes are committed to `main` branch. When a pull request is merged, the merge action creates a commit to the `main` branch which triggers the `terraform apply -auto-approve` step to execute.

1. In your GitHub repository, create a folder named `.github` in the root directory .

1. In the `.github` folder create a sub-folder named `workflows`.

1. To add the GitHub workflow to your GitHub repository, create a file named `terraform.yml` in the `workflows` directory and add the following code block:

   ````yaml
   name: Terraform

   on:
     push:
       branches:
         - 'main'
     pull_request:

   jobs:
     terraform:
       runs-on: ubuntu-latest

       steps:
         # Checkout the repository to the GitHub Actions runner
         - name: Checkout
           uses: actions/checkout@v3

         # Install the latest version of Terraform CLI
         - name: Setup Terraform
           uses: hashicorp/setup-terraform@v1

         # Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
         - name: Terraform Init
           run: terraform init

         # Checks that all Terraform configuration files adhere to a canonical format
         - name: Terraform Format
           run: terraform fmt -check

         # Previews the changes that Terraform will make
         - name: Plan Terraform
           id: plan
           continue-on-error: true
           run: terraform plan -input=false -no-color

         # Post the preview (terraform plan) from the previous step as a GitHub pull request comment
         - name: Post Plan to GitHub PR
           if: github.ref != 'refs/heads/main'
           uses: mshick/add-pr-comment@v1
           with:
             repo-token: ${{ secrets.GITHUB_TOKEN }}
             repo-token-user-login: 'github-actions[bot]'
             message: |
               Applying:

               ```
               ${{ steps.plan.outputs.stdout }}
               ```

         # Applies the terraform configuration files when the branch is `main`
         - name: Apply Terraform
           if: github.ref == 'refs/heads/main'
           id: apply
           continue-on-error: true
           run: |
             terraform apply -auto-approve
   ````

1. Commit the changes made to the `terraform.yml` in the previous step to the `main` branch in your GitHub repository. Once the changes are committed, The GitHub workflow you created should start to run automatically as the workflow we defined in the previous step runs when a pull request is created or when changes are committed to `main` branch.

## Managing the Terraform state

If you are not using a [Terraform backend](https://www.terraform.io/language/settings/backends/configuration) to store the `.tfstate` file, add the following code block to the end of the GitHub workflow file to make sure the Terraform state file is stored in Git.

```yaml
- name: commit the terraform state
  if: github.ref == 'refs/heads/main'
  uses: stefanzweifel/git-auto-commit-action@v4
  with:
    commit_message: Updating Terraform state
    file_pattern: terraform.tfstate
```

When you run `terraform apply`, Terraform automatically manages and updates the `terraform.tfstate` file to store state about your infrastructure and configuration.
This step uses the [stefanzweifel/git-auto-commit-action@v4](https://github.com/stefanzweifel/git-auto-commit-action) action to auto-commit the `terraform.tfstate` file for changes made by running the `terraform apply` step.

{{< admonition type="note" >}}
The Terraform state file (terraform.tfstate) shouldn't be stored in Git to avoid leakage of sensitive data. Instead, store Terraform state files using a remote backend like AWS S3 with proper RBAC. For more information, refer to [Terraform state](https://www.terraform.io/language/state).
{{< /admonition >}}

## Validation

Once the GitHub workflow run is successful, you should be able to verify the following:

- `ElasticSearch`, `InfluxDB` and `AWS` folders are created in the Grafana instance.

  ![Folders in Dashboards](/media/docs/grafana-cloud/screenshot-folders-github-action-tf.png)

- Dashboard represented as JSON source code from `elasticsearch` folder in GitHub are added under the `ElasticSearch` folder in the Grafana instance.

  ![ElasticSearch Folder](/media/docs/grafana-cloud/screenshot-elastic-folder-github-action-tf.png)

- Dashboard source code from the `influxdb` folder in GitHub is added under the `InfluxDB` folder in the Grafana instance.

  ![InfluxDB Folder](/media/docs/grafana-cloud/screenshot-influxdb-folder-github-action-tf.png)

- Dashboards from `aws` folder in GitHub are added under the `AWS` folder in the Grafana instance.

  ![AWS EC2 Folder](/media/docs/grafana-cloud/screenshots-aws-folder-github-action-tf.png)

## Summary

In this guide, you created a GitHub workflow using Terraform to manage dashboard source code. Using this workflow, the dashboards in the Grafana instance are synchronized with the JSON source code files for dashboards in GitHub.

To learn more about managing Grafana Cloud using Terraform, refer to the [Grafana provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

# Manage saved queries using Terraform

This guide shows you how to manage [saved queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/#saved-queries), also known as the query library, using the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). Managing saved queries as code lets you version-control your query library and keep it consistent across Grafana instances.

{{< admonition type="note" >}}
Saved queries are only available on Grafana Enterprise and Grafana Cloud.
{{< /admonition >}}

## Before you begin

Before you begin, ensure you have the following:

- A Grafana Enterprise or Grafana Cloud instance. For more information on setting up a Grafana Cloud account, refer to [Get started](https://grafana.com/docs/grafana-cloud/get-started/).
- Terraform installed on your machine. For more information on how to install Terraform, refer to the [Terraform install documentation](https://developer.hashicorp.com/terraform/install).
- A service account token with the **Writer** role for saved queries. For more information, refer to [Roles, permissions, and RBAC](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/#roles-permissions-and-rbac) and [Service account tokens](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/service-accounts/#service-account-tokens).

## Configure the Grafana provider

Create a file named `main.tf` and add the following to set up the [Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) with the authentication required to manage saved queries:

```terraform
terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 4.44.0"
    }
  }
}

provider "grafana" {
  url  = "<Grafana-URL>"
  auth = "<Service-account-token>"
}
```

Replace the following field values:

- `Grafana-URL` with the URL of your Grafana instance, for example `https://my-stack.grafana.net/`
- `Service-account-token` with the service account token that you created

## Create a saved query resource

Create a file named `saved-queries.tf` and add the following:

```terraform
resource "grafana_apps_queries_query_v1" "example" {
  metadata {
    uid = "example-saved-query"
  }

  spec {
    title       = "Requests per second"
    description = "Prometheus rate of HTTP requests"
    is_visible  = true
    tags        = ["http", "prometheus"]

    targets {
      properties_json = jsonencode({
        refId = "A"
        expr  = "rate(http_requests_total[$__rate_interval])"
        datasource = {
          type = "prometheus"
          uid  = "my-prometheus-uid"
        }
      })
    }
  }
}
```

Because the shape of a data source query depends on the data source, the query stored in each target (`properties_json`), a target's variable replacements (`variables_json`), and a variable's value list (`value_list_definition_json`) are passed as raw JSON strings. Use `jsonencode()` to construct them.

The `spec` block supports the following fields:

| Field         | Description                                                                                                     |
| ------------- | --------------------------------------------------------------------------------------------------------------- |
| `title`       | The display name of the saved query. Required.                                                                  |
| `description` | A longer description of the saved query.                                                                        |
| `is_visible`  | Whether the saved query is visible in the query library.                                                        |
| `is_locked`   | Whether the saved query is locked and can't be edited in the UI. This is for UI display only, not for security. |
| `tags`        | The tags used to filter the saved query.                                                                        |
| `targets`     | The query targets that make up the saved query. At least one target is required.                                |
| `vars`        | The template variables that can be interpolated into the query targets.                                         |

For the complete schema, refer to the [`grafana_apps_queries_query_v1` resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_queries_query_v1).

## Import an existing saved query

To bring an existing saved query under Terraform management, import it using its UID:

```shell
terraform import grafana_apps_queries_query_v1.example example-saved-query
```

## Summary

In this guide, you learned how to create and import saved queries using Terraform.

To learn more about saved queries, refer to [Saved queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/#saved-queries).

# Manage Cloud Provider Observability in Grafana Cloud using Terraform

Manage Cloud Provider Observability, including Amazon CloudWatch and Microsoft Azure resources, in Grafana Cloud using Terraform.
For more information on Cloud Provider Observability, refer to the [Cloud Provider Observability](/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/) documentation.

## Before you begin

Before you begin, ensure you have the following:

- A Grafana Cloud account
  - For more information on setting up a Grafana Cloud account, refer to [Get started](/docs/grafana-cloud/get-started/).
- Terraform installed on your machine
  - For more information on how to install Terraform, refer to the [Terraform install documentation](https://developer.hashicorp.com/terraform/install).
- Administrator permissions in your Grafana instance
  - For more information on assigning Grafana RBAC roles, refer to [Assign RBAC roles](/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-control/assign-rbac-roles/).

{{< admonition type="note" >}}
Save all of the following Terraform configuration files in the same directory.
{{< /admonition >}}

## Configure authentication for the Grafana Provider

The Grafana Provider is a logical abstraction of an upstream API that you can use to interact with Grafana Cloud resources.
You must configure it with the following information:

- A Grafana Cloud access policy token that includes the permissions the provider needs to access the Grafana Cloud Provider API.
- A regional Cloud Provider API endpoint to establish which Grafana Cloud stack you are accessing.

To configure authentication for the Grafana Provider:

1. Create a Grafana Cloud access policy and token.
   - To create an access policy for your organization, refer to the [Create an access policy for a stack steps](/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-policies/create-access-policies/#create-an-access-policy-for-a-stack) and use the following scopes listed for the supported Amazon CloudWatch or Microsoft Azure resources:
     - Amazon CloudWatch
       - Metrics scrape or resource metadata scrape
         - `integration-management:read`
         - `integration-management:write`
         - `stacks:read`
       - Metric streams
         - `metrics:write`
       - ALB access logs, logs with Lambda, or logs with Amazon Data Firehose
         - `logs:write`
   - Microsoft Azure
     - Serverless metrics
       - `integration-management:read`
       - `integration-management:write`
       - `stacks:read`
     - Logs with Azure functions
       - `logs:write`

1. Obtain the regional Cloud Provider API endpoint.
   - To obtain the regional Cloud Provider API endpoint, use your access policy token and the following command to return a list of all of the Grafana stacks you own, along with their respective Cloud Provider API hostnames:

   ```bash
   curl -sH "Authorization: Bearer @@@GRAFANA_CLOUD_ACCESS_POLICY_TOKEN@@@" "https://grafana.com/api/instances" | \
   jq '[.items[]|{stackName: .slug, clusterName:.clusterSlug, cloudProviderAPIURL: "https://cloud-provider-api-\(.clusterSlug).grafana.net"}]'
   ```

   {{< admonition type="note" >}}
   If this endpoint does not work, your Grafana Cloud Provider API endpoint may be using the new URL format.
   For more information on Grafana Cloud URLs, refer to [Determine Grafana Cloud URLs based on region](/docs/grafana-cloud/security-and-account-management/region-url-formats/).
   {{< /admonition >}}

1. Create a file named `cloud-provider.tf` and add the following code block:

   ```tf
   terraform {
       required_providers {
       grafana = {
           source  = "grafana/grafana"
       }
     }
   }

   provider "grafana" {
       cloud_provider_url      = "<@@@CLOUD_PROVIDER_URL@@@"
       cloud_access_policy_token     = "@@@CLOUD_ACCESS_POLICY_TOKEN@@@>"
   }
   ```

1. Create a `variables.tf` file and paste the `CLOUD_ACCESS_POLICY_TOKEN` and `CLOUD_PROVIDER_URL` variables with your values.
1. Run the following Terraform command:
   ```tf
   terraform apply -var-file="variables.tf"
   ```

## Configure your resources

To find instructions for configuring specific Amazon CloudWatch and Microsoft Azure resources in Cloud Provider Observability using Terraform, refer to the following documents:

- Amazon CloudWatch
  - [Metrics scrape](/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/aws/cloudwatch-metrics/config-cw-metric-scrape/): Pull CloudWatch metrics from multiple regions for your AWS account, without needing to install Grafana Alloy.
  - [Metric streams](/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/aws/cloudwatch-metrics/config-cw-metric-streams/#configure-metric-streams-with-terraform): Push metrics with CloudWatch metric streams using Amazon Data Firehose, providing real-time insights and scalability while simplifying configuration and reducing cost and manual effort.
  - [ALB access logs](/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/aws/logs/cloudwatch-logs/config-alb-access-logs-lambda/#configure-with-terraform): Send application load balancer access logs from AWS to Grafana Cloud using a Lambda function.
  - [Logs with Lambda](/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/aws/logs/cloudwatch-logs/config-cw-logs-lambda/#configure-with-terraform): Send logs to Grafana Cloud from multiple AWS services using a lambda-promtail function.
  - [Logs with Amazon Data Firehose](/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/aws/logs/firehose-logs/config-firehose-logs/#configure-with-terraform): Send logs from AWS to Grafana Cloud with Amazon Data Firehose and minimal infrastructure.
- Microsoft Azure
  - [Serverless metrics](/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/azure/collect-azure-serverless/config-azure-metrics-serverless/): Monitor your Azure resources without the need to configure or deploy a collector by using Cloud Provider Observability.
  - [Logs with Azure functions](/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/azure/config-azure-logs-azure-function/): Send Azure event logs to a Loki endpoint using an Azure function that subscribes to an Azure event hub.

## Grafana cloud provider resources

You can define the following Cloud Provider Observability resources and data sources using Terraform:

| Resource name                                             | Description                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `grafana_cloud_provider_aws_account`                      | Represents an AWS IAM role that authorizes Grafana Cloud to pull Amazon CloudWatch metrics for a set of regions. Usually, there's one of these resources per configured AWS account. For a full reference of this resource, refer to the [Terraform Grafana Provider reference documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_provider_aws_account).                                        |
| `grafana_cloud_provider_aws_cloudwatch_scrape_job`        | Represents a Grafana AWS scrape job. This configures Grafana to fetch a list of metrics and statistics for one or many AWS services, and for a given `grafana_cloud_provider_aws_account`. For a full reference of this resource, refer to the [Terraform Grafana Provider reference documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_provider_aws_cloudwatch_scrape_job).                    |
| `grafana_cloud_provider_aws_resource_metadata_scrape_job` | Represents a Grafana AWS Resource Metadata scrape job. This resource configures Grafana to fetch resource metadata for one or multiple AWS services, for a given `grafana_cloud_provider_aws_account`. For a full reference of this resource, refer to the [Terraform Grafana Provider reference documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_provider_aws_resource_metadata_scrape_job). |
| `grafana_cloud_provider_azure_credential`                 | A resource representing an Azure Service Principal credential used by Grafana Cloud to pull Azure Monitor metrics from one or more subscriptions. For a full reference of this resource, refer to the [Terraform Grafana Provider resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_provider_azure_credential).                                                                       |

# Create and manage a Grafana Cloud stack using Terraform

Learn how to add a data source, a dashboard, and a folder to a Grafana Cloud stack using Terraform.

## Prerequisites

Before you begin, ensure you have the following:

- A Grafana Cloud account, as shown in [Get started](/docs/grafana-cloud/get-started/)
- [Terraform](https://www.terraform.io/downloads) installed on your machine

{{< admonition type="note" >}}
All of the following Terraform configuration files should be saved in the same directory.
{{< /admonition >}}

## Create a Cloud stack

1. Create a Terraform configuration file.

   This Terraform configuration will create a Grafana Cloud stack and a second token needed for your Grafana instance.

   Create a file named `cloud-stack.tf` and add the following:

   ```terraform
   terraform {
     required_providers {
       grafana = {
         source  = "grafana/grafana"
         version = ">= 2.9.0"
       }
     }
   }
   // Step 1: Create a stack
   provider "grafana" {
     alias = "cloud"
     cloud_access_policy_token = "<cloud-access-token>"
   }


   resource "grafana_cloud_stack" "my_stack" {
     provider = grafana.cloud

     name               = "<stack-name>"
     slug               = "<stack-name>"
     region_slug        = "<region>" # Example "us","eu" etc
     delete_protection  = true
   }

   // Step 2: Create a service account and key for the stack
   resource "grafana_cloud_stack_service_account" "cloud_sa" {
     provider   = grafana.cloud
     stack_slug = grafana_cloud_stack.my_stack.slug

     name        = "<service-account-name>"
     role        = "Admin"
     is_disabled = false
   }

   resource "grafana_cloud_stack_service_account_token" "cloud_sa" {
     provider   = grafana.cloud
     stack_slug = grafana_cloud_stack.my_stack.slug

     name               = "terraform serviceaccount key"
     service_account_id = grafana_cloud_stack_service_account.cloud_sa.id
   }

   // Step 3: Create resources within the stack
   provider "grafana" {
     alias = "my_stack"

     url  = grafana_cloud_stack.my_stack.url
     auth = grafana_cloud_stack_service_account_token.cloud_sa.key
   }
   resource "grafana_folder" "my_folder" {
     provider = grafana.my_stack

     title = "Test Folder"
   }
   ```

1. Replace the following field values:
   - `<cloud-access-token>` with your Grafana Cloud Access Policy Token.
     To create a new one, refer to [Grafana Cloud Access Policies](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/).
     Add all stacks to the realms list.
     The scopes needed for the example are:
     - dashboards:read
     - orgs:read
     - stack-dashboards:read
     - stacks:read
     - dashboards:write
     - orgs:write
     - stack-dashboards:write
     - stacks:write
     - stack-service-accounts:write
     - dashboards:delete
     - stack-dashboards:delete
     - stacks:delete
   - `<stack-name>` with the name of your stack.
   - `<region>` with the region in which you want to create the stack. For example `us`, `eu`.
   - `<service-account-name>` with a name for the serviceaccount that will be created to use for operations within the stack/instance.

The first provider block, `grafana.cloud`, uses the Cloud Access Policy Token from the Cloud Portal and is referenced as a parameter when creating the Cloud stack and the token in the Grafana instance to provide the necessary authentication.

The second provider block, `grafana.my_stack`, is referenced as a parameter when creating resources inside the Grafana instance.

## Add a data source

This guide uses the InfluxDB data source. The required arguments for [grafana_data_source (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source) vary depending on the type of data source you select.

1. Create a file named `datasource.tf` and add the following:

   ```terraform
   resource "grafana_data_source" "<data-source-name>" {
     provider = grafana.my_stack

     type          = "influxdb"
     name          = "<data-source-name>"
     url           = "<data-source-url>"
     username      = "<username>"
     password      = "<password>"
     database_name = "<db-name>"
   }
   ```

1. Replace the following field values:
   - `<data-source-name>` with the name of the data source to be added in Grafana.
   - `<data-source-url>` with URL of your data source.
   - `<username>` with the username for authenticating with your data source.
   - `<password>` with password for authenticating with your data source.
   - `<db-name>` with name of your database.

## Add a folder

This Terraform configuration creates a folder in your Grafana instance using [grafana_folder (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/folder).

1. Create a file named `folder.tf` and add the following:

   ```terraform
   resource "grafana_folder" "<folder-name>" {
     provider = grafana.my_stack

     title = "<folder-name>"
   }
   ```

1. Replace the following field value:
   - `<folder-name>` with a name for the folder.

## Add a dashboard to the folder

This Terraform configuration creates a dashboard inside the folder created above in your Grafana instance using [grafana_dashboard (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/dashboard).

1. Create a file named `dashboard.tf` and add the following:

   ```terraform
   # Using a JSON file
   resource "grafana_dashboard" "dashboard" {
     provider = grafana.my_stack

     config_json = file("<file-name>.json")
     folder = grafana_folder.<folder-name>.id
   }
   ```

1. Replace the following field value:
   - `<file-name>` with the name of the JSON file that has the source code for the dashboard.

   The dashboard is represented by its JSON source code and referenced in the `config_json` parameter.

## Apply the Terraform configuration

In a terminal, run the following commands from the directory where all of the configuration files are located.

1. Initialize a working directory containing Terraform configuration files.

   ```shell
   terraform init
   ```

1. Preview the changes that Terraform will make.

   ```shell
   terraform plan
   ```

1. Apply the configuration files.

   ```shell
   terraform apply
   ```

## Validation

Once you apply the changes in the Terraform configurations, you should be able to verify the following:

- The new Grafana stack is created and visible in the Cloud Portal

  ![Cloud Portal](/static/img/docs/grafana-cloud/terraform/cloud_portal_tf.png)

- A service account key token is added in your Grafana instance. In the following image, the service account key token named "terraform serviceaccount key" was added by the [grafana_cloud_stack_service_account_token (Resource)](#create-a-cloud-stack).

  ![API Key](/media/docs/grafana-cloud/screenshot-api_key_tf.png)

- A new data source (InfluxDB in this example) is visible in the grafana instance.

  ![InfluxDB data source](/media/docs/grafana-cloud/screenshot-influxdb_datasource_tf.png)

- A new folder in Grafana. In the following image, a folder named "Demos" was added by the [grafana_folder (Resource)](./#add-a-folder).

  ![Folder](/media/docs/grafana-cloud/screenshot-folder_tf.png)

- A new dashboard in the Grafana instance. In the following image a dashboard named "InfluxDB Cloud Demos" was created inside the "Demos" folder.

  ![InfluxDB dashboard](/static/img/docs/grafana-cloud/terraform/influxdb_dashboard_tf.png)

## Summary

In this guide, you created a Grafana Cloud stack along with a data source, folder, and dashboard imported from a JSON file using Terraform.

To learn more about managing Grafana Cloud using Terraform, refer to the [Grafana provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

# Manage Fleet Management in Grafana Cloud using Terraform

Learn how to create [Grafana Fleet Management](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/) collectors and pipelines in Grafana Cloud using Terraform.
This guide shows you how to create an access policy and a token for Fleet Management, preregister a collector with remote attributes, and create a configuration pipeline.

Fleet Management supports [Grafana Alloy](https://grafana.com/docs/alloy/latest/) and the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/).
The two collector types connect to Fleet Management differently and use different configuration syntax, so this guide shows both paths side by side.
Follow the Alloy steps, the OpenTelemetry Collector steps, or both if you're managing a mixed fleet.

## Before you begin

Before you begin, ensure you have the following:

- A Grafana Cloud account, as shown in [Get started](https://grafana.com/docs/grafana-cloud/get-started/)
- Administrator permissions in your Grafana instance
- [Terraform](https://www.terraform.io/downloads) installed on your machine
- One of the following collectors installed on your machine:
  - [Alloy](https://grafana.com/docs/alloy/latest/set-up/install/)
  - A supported OpenTelemetry Collector distribution that includes the [OpAMP extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/opampextension) (for example, `otelcol-contrib`)
    If using the OpenTelemetry Collector, you must also install the [OpAMP Supervisor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/opampsupervisor).

{{< admonition type="note" >}}
All of the following Terraform configuration files should be saved in the same directory.
{{< /admonition >}}

## Configure a provider for Grafana Cloud

This Terraform configuration configures the [Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) to provide necessary authentication when interacting with the Cloud API.
The [`grafana_cloud_stack` (Data Source)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/data-sources/cloud_stack) is used to retrieve the user ID and URL details of your instance.

1. Create a Grafana Cloud access policy and token.
   To create a new one, refer to [Grafana Cloud Access Policies](https://grafana.com/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-policies/).
   Add your stack to the realms list.
   The scopes needed for the examples in this guide are:
   - `accesspolicies:read`
   - `accesspolicies:write`
   - `accesspolicies:delete`
   - `stacks:read`

1. Create a file named `cloud-provider.tf` and add the following code block:

   ```terraform
   terraform {
     required_providers {
       grafana = {
         source  = "grafana/grafana"
         version = ">= 3.19.0"
       }
     }
   }

   provider "grafana" {
     alias = "cloud"

     cloud_access_policy_token = "<CLOUD_ACCESS_POLICY_TOKEN>"
   }

   data "grafana_cloud_stack" "stack" {
     provider = grafana.cloud

     slug = "<STACK_SLUG>"
   }
   ```

1. Replace the following field values:
   - `<CLOUD_ACCESS_POLICY_TOKEN>` with the access policy token you created in the first step
   - `<STACK_SLUG>` with your stack slug, which is the subdomain where your Grafana Cloud instance is available: `https://<STACK_SLUG>.grafana.net`

## Create an access policy and token for Fleet Management

This Terraform configuration creates the following:

- An access policy named `fleet-management-policy` with `fleet-management:read` and `fleet-management:write` scopes, using [`grafana_cloud_access_policy` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_access_policy)
- A token named `fleet-management-token`, using [`grafana_cloud_access_policy_token` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_access_policy_token)

1. Create a file named `fm-access-policy.tf` and add the following code block:

   ```terraform
   resource "grafana_cloud_access_policy" "fm_policy" {
     provider = grafana.cloud

     name   = "fleet-management-policy"
     region = data.grafana_cloud_stack.stack.region_slug

     scopes = [
       "fleet-management:read",
       "fleet-management:write"
     ]

     realm {
       type       = "stack"
       identifier = data.grafana_cloud_stack.stack.id
     }
   }

   resource "grafana_cloud_access_policy_token" "fm_token" {
     provider = grafana.cloud

     name             = "fleet-management-token"
     region           = grafana_cloud_access_policy.fm_policy.region
     access_policy_id = grafana_cloud_access_policy.fm_policy.policy_id
   }
   ```

## Configure a provider for Fleet Management

This Terraform configuration configures the [Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) to provide necessary authentication when interacting with the Fleet Management API.

1. Create a file named `fm-provider.tf` and add the following code block:

   ```terraform
   locals {
     fm_id    = data.grafana_cloud_stack.stack.fleet_management_user_id
     fm_token = grafana_cloud_access_policy_token.fm_token.token
     fm_url   = data.grafana_cloud_stack.stack.fleet_management_url
   }

   provider "grafana" {
     alias = "fm"

     fleet_management_auth = "${local.fm_id}:${local.fm_token}"
     fleet_management_url  = local.fm_url
   }
   ```

## Create an access policy and token for your collector

Alloy and the OpenTelemetry Collector need properly scoped access to read remote configuration from Fleet Management and write telemetry to Grafana Cloud.

The Terraform provider creates the `alloy_token` for you when you run `terraform apply` later in this guide.

OpenTelemetry Collectors require an access token _before_ you can successfully apply the Terraform configurations.
You create the `otel_token` manually in this step, **but you won't need it until later in the guide**.

### Alloy

This Terraform configuration creates the following:

- An access policy named `alloy-policy` with `set:alloy-data-write` scope, using [`grafana_cloud_access_policy` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_access_policy)
- A token named `alloy-token`, using [`grafana_cloud_access_policy_token` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_access_policy_token)

1. Create a file named `alloy-access-policy.tf` and add the following code block:

   ```terraform
   resource "grafana_cloud_access_policy" "alloy_policy" {
     provider = grafana.cloud

     name   = "alloy-policy"
     region = data.grafana_cloud_stack.stack.region_slug

     scopes = [
       "set:alloy-data-write"
     ]

     realm {
       type       = "stack"
       identifier = data.grafana_cloud_stack.stack.id
     }
   }

   resource "grafana_cloud_access_policy_token" "alloy_token" {
     provider = grafana.cloud

     name             = "alloy-token"
     region           = grafana_cloud_access_policy.alloy_policy.region
     access_policy_id = grafana_cloud_access_policy.alloy_policy.policy_id
   }

   output "alloy_token" {
     value     = grafana_cloud_access_policy_token.alloy_token.token
     sensitive = true
   }
   ```

### OpenTelemetry Collector

This Terraform configuration creates the following:

- An access policy named `otel-policy` with `set:otel-data-write` scope, using [`grafana_cloud_access_policy` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_access_policy)
- A token named `otel-token`, using [`grafana_cloud_access_policy_token` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_access_policy_token)

1. Create a file named `otel-access-policy.tf` and add the following code block:

   ```terraform
   resource "grafana_cloud_access_policy" "otel_policy" {
     provider = grafana.cloud

     name   = "otel-policy"
     region = data.grafana_cloud_stack.stack.region_slug

     scopes = [
       "set:otel-data-write"
     ]

     realm {
       type       = "stack"
       identifier = data.grafana_cloud_stack.stack.id
     }
   }

   resource "grafana_cloud_access_policy_token" "otel_token" {
     provider = grafana.cloud

     name             = "otel-token"
     region           = grafana_cloud_access_policy.otel_policy.region
     access_policy_id = grafana_cloud_access_policy.otel_policy.policy_id
   }
   ```

## Create a configuration file for your collector

In this step, you create a local configuration file for Alloy or the OpAMP Supervisor.

### Alloy

This Terraform configuration creates an Alloy configuration file with the [`remotecfg` block](https://grafana.com/docs/grafana-cloud/send-data/alloy/reference/config-blocks/remotecfg/) for Fleet Management, using [`local_file` (Resource)](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file).

1. Create a file named `config.alloy.tftpl` and add the following content:

   ```alloy
   remotecfg {
       id             = "${collector_id}"
       url            = "${fm_url}"
       poll_frequency = "60s"

       basic_auth {
           username = "${fm_id}"
           password = sys.env("GCLOUD_RW_API_KEY")
       }
   }
   ```

1. Create a file named `alloy-config.tf` and add the following code block:

   ```terraform
   resource "local_file" "alloy_config" {
     filename = "<ALLOY_CONFIG_PATH>"
     content = templatefile(
       "config.alloy.tftpl",
       {
         collector_id = "prod_alloy_collector",
         fm_id        = local.fm_id,
         fm_url       = local.fm_url,
       },
     )
     directory_permission = "0644"
     file_permission      = "0644"
   }
   ```

1. Replace the following field values:
   - `<ALLOY_CONFIG_PATH>` with the path the Alloy configuration file should be written to, for example `config.alloy`

### OpenTelemetry Collector

This Terraform configuration creates a Supervisor configuration file that connects the OpenTelemetry Collector to Fleet Management over [OpAMP](https://github.com/open-telemetry/opamp-spec), using [`local_file` (Resource)](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file).

1. Create a file named `supervisor.yaml.tftpl` and add the following content:

   ```yaml
   server:
     endpoint: '${fm_url}/v1/opamp'
     headers:
       Authorization: 'Basic ${fm_auth_base64}'

   capabilities:
     reports_effective_config: true
     accepts_remote_config: true

   agent:
     executable: '<OTELCOL_EXECUTABLE_PATH>'
     instance_id: '${collector_id}'

   storage:
     directory: '<STORAGE_DIRECTORY>'
   ```

1. Create a file named `otel-supervisor-config.tf` and add the following code block:

   ```terraform
   resource "local_file" "otel_supervisor_config" {
     filename = "${path.module}/supervisor.yaml"
     content = templatefile(
       "supervisor.yaml.tftpl",
       {
         collector_id   = "<COLLECTOR_ID>"
         fm_auth_base64 = base64encode("${local.fm_id}:${grafana_cloud_access_policy_token.otel_token.token}"),
         fm_url         = local.fm_url,
       },
     )
     directory_permission = "0644"
     file_permission      = "0644"
   }
   ```

1. Replace the following field values:
   - `<OTELCOL_EXECUTABLE_PATH>` with the path to your OpenTelemetry Collector executable
   - `<STORAGE_DIRECTORY>` with the path to a writable directory to use for [persistent data storage](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/opampsupervisor/README.md#persistent-data-storage)
   - `<COLLECTOR_ID>` with the universally unique identifier (UUID) of the Collector.

## Create a Fleet Management collector

This step registers a collector with a remote attribute, using [`grafana_fleet_management_collector` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/fleet_management_collector).
The `collector_type` argument tells Fleet Management which configuration syntax and validation to expect for the collector: `ALLOY` (the default) or `OTEL`.

The following configurations only preregister the collector.
You must complete the [Run the collector](#run-the-collector) step for the collector to connect to Fleet Management and be assigned remote attributes.

### Alloy

1. Create a file named `fm-collector-alloy.tf` and add the following code block:

   ```terraform
   resource "grafana_fleet_management_collector" "fm_collector_alloy" {
     provider = grafana.fm

     id = "prod_alloy_collector"
     collector_type = "ALLOY"
     remote_attributes = {
       "env" = "PROD"
     }
     enabled = true
   }
   ```

### OpenTelemetry Collector

1. Create a file named `fm-collector-otel.tf` and add the following code block:

   ```terraform
   resource "grafana_fleet_management_collector" "fm_collector_otel" {
     provider = grafana.fm

     id = "<COLLECTOR_ID>"
     collector_type = "OTEL"
     remote_attributes = {
       "env" = "PROD"
     }
     enabled = true
   }
   ```

1. Replace `<COLLECTOR_ID>` with the same unique identifier used in the Supervisor configuration.

## Create a Fleet Management pipeline

The following Terraform configurations create pipelines with [matchers](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/introduction/glossary/#attribute-matching) for the collectors declared in the previous step, using [`grafana_fleet_management_pipeline` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/fleet_management_pipeline).
The `config_type` argument tells Fleet Management how to parse and validate the `contents` argument: `ALLOY` (the default) expects Alloy configuration syntax, and `OTEL` expects OpenTelemetry Collector YAML.

Fleet Management delivers a pipeline only to collectors of the matching `collector_type`, so an `ALLOY` pipeline and an `OTEL` pipeline can safely share the same matchers.

### Alloy profiling pipeline

This pipeline profiles Alloy itself and writes the profiles to [Grafana Cloud Profiles](https://grafana.com/docs/grafana-cloud/monitor-applications/profiles/).

1. Create a file named `profiling.alloy.tftpl` and add the following content:

   ```alloy
   // This pipeline scrapes pprof Go profiles from Alloy and sends them to Pyroscope.
   //
   // It requires the following environment variables to be set where Alloy is running:
   //   Required:
   //     * GCLOUD_RW_API_KEY: The Grafana Cloud API key with write access to Pyroscope.
   //   Optional:
   //     * ALLOY_ADDRESS: The address Alloy listens on. Defaults to 127.0.0.1:12345.
   pyroscope.scrape "alloy" {
       targets = [
           {
               "__address__" = coalesce(
                   sys.env("ALLOY_ADDRESS"),
                   "127.0.0.1:12345",
               ),
               "service_name" = "alloy",
           },
       ]
       forward_to = [pyroscope.write.grafana_cloud.receiver]

       profiling_config {
           profile.process_cpu {
               enabled = true
           }

           profile.memory {
               enabled = true
           }

           profile.mutex {
               enabled = true
           }

           profile.block {
               enabled = true
           }

           profile.goroutine {
               enabled = true
           }
       }
   }

   pyroscope.write "grafana_cloud" {
       endpoint {
           url = "${profiles_url}"

           basic_auth {
               username = "${profiles_id}"
               password = sys.env("GCLOUD_RW_API_KEY")
           }
       }
   }
   ```

1. Create a file named `fm-pipeline-alloy-profiles.tf` and add the following code block:

   ```terraform
   locals {
     profiles_id  = data.grafana_cloud_stack.stack.profiles_user_id
     profiles_url = data.grafana_cloud_stack.stack.profiles_url
   }

   resource "grafana_fleet_management_pipeline" "profiles" {
     provider = grafana.fm

     name        = "alloy_profiling"
     config_type = "ALLOY"
     contents = templatefile(
       "profiling.alloy.tftpl",
       {
         profiles_id  = local.profiles_id,
         profiles_url = local.profiles_url,
       },
     )
     matchers = [
       "env=\"PROD\""
     ]
     enabled = true
   }
   ```

### OpenTelemetry Collector host metrics pipeline

This pipeline collects host metrics with the [`hostmetrics` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) and writes them to Grafana Cloud over [OTLP](https://grafana.com/docs/grafana-cloud/send-data/otlp/send-data-otlp/).

1. Create a file named `hostmetrics.otel.yaml.tftpl` and add the following content:

   ```yaml
   # This pipeline collects host metrics and sends them to Grafana Cloud over OTLP.
   receivers:
     hostmetrics:
       scrapers:
         load:
         memory:

   processors:
     resourcedetection:
       detectors: ['env', 'system']
       override: false
     transform/add_host_metric_attributes:
       error_mode: ignore
       metric_statements:
         - context: datapoint
           statements:
             - set(attributes["host.name"], resource.attributes["host.name"])
             - set(attributes["os.type"], resource.attributes["os.type"])

   exporters:
     otlp_http/grafana_cloud:
       endpoint: '<OTLP_ENDPOINT>'
       headers:
         Authorization: 'Basic $${env:GCLOUD_BASIC_AUTH_BASE64}'
       sending_queue:
         enabled: true
         batch: {}

   service:
     pipelines:
       metrics/hostmetrics:
         receivers: [hostmetrics]
         processors: [resourcedetection, transform/add_host_metric_attributes]
         exporters: [otlp_http/grafana_cloud]
   ```

1. Replace `<OTLP_ENDPOINT>` with the OTLP endpoint for your stack, which you can find in your Grafana Cloud portal.

1. Create a file named `fm-pipeline-otel-hostmetrics.tf` and add the following code block:

   ```terraform
   resource "grafana_fleet_management_pipeline" "host_metrics" {
     provider = grafana.fm

     name        = "otel_host_metrics"
     config_type = "OTEL"
     contents = templatefile("hostmetrics.otel.yaml.tftpl", {})
     matchers = [
       "env=\"PROD\""
     ]
     enabled = true
   }
   ```

## Apply the Terraform configuration

In a terminal, run the following commands from the directory where all of the configuration files are located.

1. Initialize a working directory containing Terraform configuration files:

   ```shell
   terraform init
   ```

1. Preview the Terraform changes:

   ```shell
   terraform plan
   ```

1. Apply the configuration:

   ```shell
   terraform apply
   ```

## Run the collector

The final step is to start your collector.
Make sure to set an environment variable containing the access token.
The pipelines used in this guide won't work without it.

### Alloy

1. Run the following command to view the Alloy token:

   ```shell
   terraform output -raw alloy_token
   ```

1. Set the environment variable `GCLOUD_RW_API_KEY` to the value from the first step.
1. Run Alloy.
   To learn how to start or restart Alloy, refer to [Run Grafana Alloy](https://grafana.com/docs/alloy/latest/set-up/run/).

### OpenTelemetry Collector

1. Run the following command to export the Base64-encoded credentials the Collector uses to send telemetry to Grafana Cloud.
   Replace `<INSTANCE_ID>` with your actual instance ID, which you can find on the **API** tab in the Fleet Management application.

   ```shell
   export GCLOUD_BASIC_AUTH_BASE64=$(printf '%s' "<INSTANCE_ID>:$(terraform output -raw otel_token)" | base64 | tr -d '\n')
   ```

1. Start the Supervisor:

   ```shell
   ./opampsupervisor --config=supervisor.yaml
   ```

## Validation

After you apply the changes in the Terraform configurations and run your collector, you should be able to verify the following:

- A collector is added to the Fleet Management **Inventory tab**, showing the `collector_type` you configured (`ALLOY` or `OTEL`):

  {{< figure alt="The Inventory screen in the Fleet Management interface in Grafana Cloud which shows that a new OpenTelemetry Collector called `otelcol-contrib` is registered with attribute `env=PROD`, has a healthy status, and was last modified a few minutes ago." src="/media/docs/fleet-management/screenshot-fleet-management-terraform-validate-collector-otel.png" >}}

- A pipeline is added to the Fleet Management **Remote configuration tab**:

  {{< figure alt="The Remote configuration screen in the Fleet Management interface in Grafana Cloud which shows that a new OpenTelemetry configuration pipeline called `otel_host_metrics` is active." src="/media/docs/fleet-management/screenshot-fleet-management-terraform-validate-pipeline-otel.png" >}}

- Telemetry is being exported to Grafana Cloud. For example, host metrics are visible in Grafana Cloud Metrics:

  {{< figure alt="A dashboard in Grafana Cloud which shows OTLP host metrics data, including a graph for memory usage." src="/media/docs/fleet-management/screenshot-fleet-management-terraform-validate-host-metrics.png" >}}

## Summary

In this guide, you created an access policy and a token for Fleet Management, registered an Alloy instance or OpenTelemetry Collector with remote attributes, and created a configuration pipeline based on collector type, all using Terraform.

To learn more about managing Grafana Cloud using Terraform, refer to the [Grafana provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

# Manage Frontend Observability in Grafana Cloud with Terraform

Learn how to use Terraform to manage [Grafana Frontend Observability](https://grafana.com/docs/grafana-cloud/frontend-observability/) resources, such as your applications.
This guide shows you how to create an access policy and a token for Frontend Observability so that you can connect to the Frontend Observability API.

## Before you begin

Before you begin, ensure you have the following:

- A Grafana Cloud account, as shown in [Get started](https://grafana.com/docs/grafana-cloud/get-started/)
- [Terraform](https://www.terraform.io/downloads) installed on your machine
- Administrator permissions in your Grafana instance

{{< admonition type="note" >}}
All of the following Terraform configuration files should be saved in the same directory.
{{< /admonition >}}

## Configure a provider for Grafana Cloud

This Terraform configuration configures the [Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) to provide necessary authentication when interacting with the Cloud API.
The [`grafana_cloud_stack` (Data Source)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/data-sources/cloud_stack) is used to retrieve the details of your instance.

1. Create a Grafana Cloud access policy and token.
   To create a new one, refer to [Grafana Cloud Access Policies](https://grafana.com/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-policies/).
   Add your stack to the realms list.
   The scopes needed for the examples in this guide are:
   - `accesspolicies:read`
   - `accesspolicies:write`
   - `accesspolicies:delete`
   - `dashboards:read`
   - `dashboards:write`
   - `dashboards:delete`
   - `orgs:read`
   - `orgs:write`
   - `stacks:read`
   - `stacks:write`
   - `stacks:delete`
   - `stack-dashboards:read`
   - `stack-dashboards:write`
   - `stack-dashboards:delete`
   - `stack-service-accounts:write`

1. Create a file named `cloud-provider.tf` and add the following code block:

   ```terraform
   terraform {
     required_providers {
       grafana = {
         source  = "grafana/grafana"
       }
     }
   }

   provider "grafana" {
     alias = "cloud"

     cloud_access_policy_token = "<CLOUD_ACCESS_POLICY_TOKEN>"
   }

   data "grafana_cloud_stack" "stack" {
     provider = grafana.cloud

     slug = "<STACK_SLUG>"
   }
   ```

1. Replace the following field values:
   - `<CLOUD_ACCESS_POLICY_TOKEN>` with the access policy token you created in the first step
   - `<STACK_SLUG>` with your stack slug, which is the subdomain where your Grafana Cloud instance is available: `https://<STACK_SLUG>.grafana.net`

## Create an access policy and token for Frontend Observability

You must create a Terraform configuration with the following:

- An access policy with `frontend-observability:read`, `frontend-observability:write`, and `frontend-observability:delete` scopes, using [`grafana_cloud_access_policy` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_access_policy)
- A token named `frontend_o11y_api_access_token`, using [`grafana_cloud_access_policy_token` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_access_policy_token)

## Configure the provider to use the Frontend Observability API

After you have created the token, you can configure the provider as follows:

```terraform
provider "grafana" {
  frontend_o11y_api_access_token = "<access token from previous step>"
}
```

## Summary

In this guide, you created an access policy and a token for Frontend Observability using Terraform.

To learn more about managing Grafana Cloud using Terraform, refer to the [Grafana provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

# Manage Grafana IRM in Grafana Cloud using Terraform

Learn how to use Terraform to manage [Grafana IRM](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/) resources.
This guide shows you how to connect an integration, configure escalation policies, and add on-call schedules using Terraform.

To illustrate the use of IRM across multiple teams, this guide features examples with two teams: `Devs` and `SREs`.
Additionally, it includes the necessary steps to configure Slack for IRM.

{{< admonition type="note" >}}
Grafana IRM supports Terraform-based configuration for a limited set of resources, primarily those related to OnCall functionality.
These resources use the `grafana_oncall_` naming convention in Terraform. Additional IRM components are not yet configurable via Terraform.
{{< /admonition >}}

## Before you begin

Before you begin, ensure you have the following:

- A Grafana Cloud account, as shown in [Get started](https://grafana.com/docs/grafana-cloud/get-started/)
- [Terraform](https://www.terraform.io/downloads) installed on your machine
- Administrator permissions in your Grafana instance
- (Optional) Administrator permissions in your Slack workspace, if you plan to integrate Slack with Grafana IRM

{{< admonition type="note" >}}
All of the following Terraform configuration files should be saved in the same directory.
{{< /admonition >}}

## Connect Slack to Grafana IRM

Before including Slack settings in your Terraform setup, you must first configure the Slack integration with Grafana IRM.

To connect your Slack workspace to Grafana IRM, refer to the [Slack integration for Grafana IRM](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/configure/integrations/irm-slack/) documentation.

## Configure the Grafana provider

This Terraform configuration sets up the [Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) to provide necessary authentication when managing resources for Grafana IRM.

You can reuse a similar setup to the one described in [Creating and managing a Grafana Cloud stack using Terraform](../terraform-cloud-stack/) to set up a service account and a token.

1. Create a Service account and token in Grafana. To create a new one, refer to [Service account tokens](https://grafana.com/docs/grafana/latest/administration/service-accounts/#service-account-tokens).

1. Create a file named `main.tf` and add the following:

   ```terraform
   terraform {
     required_providers {
       grafana = {
         source  = "grafana/grafana"
         version = ">= 3.15.3"
       }
     }
   }

   provider "grafana" {
     alias = "oncall"

     url  = "<Stack-URL>"
     auth = "<Service-account-token>"
     oncall_url = "<OnCall-URL>"
   }
   ```

1. Replace the following field values:
   - `<Stack-URL>` with the URL of your Grafana stack
   - `<Service-account-token>` with the service account token that you created
   - `<OnCall-URL>` with the API URL found on the **Admin & API** tab of the IRM **Settings** page

{{< admonition type="note" >}}
If the service account has the right permissions, this provider setup also allows you to manage other Grafana resources.
{{< /admonition >}}

### Authentication via OnCall API tokens (deprecated)

OnCall API tokens are being deprecated.
While existing tokens will continue to work, we recommend using
[Grafana Cloud service account tokens](https://grafana.com/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/service-accounts/) for all new API authentication.

{{< collapse title="Authentication via OnCall API tokens" >}}
To use an existing OnCall API token:

1. Log into your Grafana Cloud instance
1. Select **Alerts & IRM** > **IRM**
1. Click **Settings**, and then select **Admin & API**
1. Locate the **Grafana IRM API** section
1. View, copy or revoke existing **OnCall API tokens**

1. Create a file named `main.tf` and add the following:

   ```terraform
   terraform {
     required_providers {
       grafana = {
         source  = "grafana/grafana"
         version = ">= 2.9.0"
       }
     }
   }

   provider "grafana" {
     alias = "oncall"

     oncall_access_token = "<OnCall-API-Token>"
     oncall_url = "<OnCall-URL>"
   }
   ```

1. Replace the following field values:
   - `<OnCall-API-Token>` with your existing OnCall API Token
   - `<OnCall-URL>` with the API URL found on the **Admin & API** tab of the IRM **Settings** page
     {{< /collapse >}}

## Add on-call schedules

This Terraform configuration sets up two on-call schedules, `SREs` and `Devs`, using the [`grafana_oncall_schedule` resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/oncall_schedule) to define the schedules within Grafana IRM.
Additionally, this configuration includes Slack channels to receive notifications for the on-call schedules of each team.

To learn more about managing on-call schedules, refer to the [On-call schedules documentation](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/manage/on-call-schedules/).

1. Create two new calendars in your calendar service, one for `Devs` and one for `SREs`

1. Locate and save the secret iCal URLs.
   For example, in a Google calendar, these URLs can be found in **Settings > Settings for my calendars > Integrate calendar**

1. Create a file named `schedule.tf` and add the following:

   ```terraform
   # Name of the Slack channel to notify about on-call schedules for Devs
   data "grafana_oncall_slack_channel" "Devs" {
     provider = grafana.oncall

     name = "<Devs-channel-name>"
   }

   # Name of the Slack channel to notify about on-call schedules for SREs
   data "grafana_oncall_slack_channel" "SREs" {
     provider = grafana.oncall

     name = "<SREs-channel-name>"
   }

   resource "grafana_oncall_schedule" "schedule_Devs" {
     provider = grafana.oncall

     name             = "Devs"
     type             = "ical"
     ical_url_primary = "<secret-iCal-URL-for-devs-calendar>"
     slack {
       channel_id = data.grafana_oncall_slack_channel.Devs.slack_id
     }
   }

   resource "grafana_oncall_schedule" "schedule_SREs" {
     provider = grafana.oncall

     name             = "SREs"
     type             = "ical"
     ical_url_primary = "<secret-iCal-URL-for-SREs-calendar>"
     slack {
       channel_id = data.grafana_oncall_slack_channel.SREs.slack_id
     }
   }
   ```

1. Replace the following field values:
   - `<Devs-channel-name>` with name of the Slack channel to notify about on-call schedules for `Devs`
   - `<SREs-channel-name>` with name of the Slack channel to notify about on-call schedules for `SREs`
   - `<secret-iCal-URL-for-devs-calendar>` with the secret iCal URL created in the first step for `Devs` Calendar
   - `<secret-iCal-URL-for-SREs-calendar>` with the secret iCal URL created in the first step for `SREs` Calendar

## Add escalation chains

This Terraform configuration creates two escalation chains named `SREs` and `Devs` in Grafana IRM using the [`grafana_oncall_escalation_chain` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/oncall_escalation_chain).
The configuration also adds the following three steps to each escalation chain using the [`grafana_oncall_escalation` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/oncall_escalation):

- Notify users from on-call schedule
- Wait for 5 minutes
- Notify default Slack channel

1. Create a file named `escalation-devs.tf` and add the following:

   ```terraform
   resource "grafana_oncall_escalation_chain" "Devs" {
     provider = grafana.oncall

     name = "Devs"
   }

   // Notify users from on-call schedule
   resource "grafana_oncall_escalation" "notify_schedule_step_Devs" {
     provider = grafana.oncall

     escalation_chain_id          = grafana_oncall_escalation_chain.Devs.id
     type                         = "notify_on_call_from_schedule"
     notify_on_call_from_schedule = grafana_oncall_schedule.schedule_Devs.id
     position                     = 0
   }

   // Wait step for 5 Minutes
   resource "grafana_oncall_escalation" "wait_step_Devs" {
     provider = grafana.oncall

     escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
     type                = "wait"
     duration            = 300
     position            = 1
   }

   // Notify default Slack channel step
   resource "grafana_oncall_escalation" "notify_step_Devs" {
     provider = grafana.oncall

     escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
     type                = "notify_whole_channel"
     important           = true
     position            = 2
   }
   ```

2. Create a file named `escalation-sre.tf` and add the following:

   ```terraform
   resource "grafana_oncall_escalation_chain" "SREs" {
     provider = grafana.oncall

     name = "SREs"
   }

   // Notify users from on-call schedule
   resource "grafana_oncall_escalation" "notify_schedule_step_SREs" {
     provider = grafana.oncall

     escalation_chain_id          = grafana_oncall_escalation_chain.SREs.id
     type                         = "notify_on_call_from_schedule"
     notify_on_call_from_schedule = grafana_oncall_schedule.schedule_SREs.id
     position                     = 0
   }

   // Wait step for 5 Minutes
   resource "grafana_oncall_escalation" "wait_step_SREs" {
     provider = grafana.oncall

     escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
     type                = "wait"
     duration            = 300
     position            = 1
   }

   // Notify default Slack channel step
   resource "grafana_oncall_escalation" "notify_step_SREs" {
     provider = grafana.oncall

     escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
     type                = "notify_whole_channel"
     important           = true
     position            = 2
   }
   ```

## Connect an integration to Grafana IRM

This Terraform configuration connects Alertmanager to Grafana IRM using the [`grafana_oncall_integration` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/oncall_integration).
It also adds the `Devs` escalation chain as the default route for alerts.

1. Create a file named `integrations.tf` and add the following:

   ```terraform
   resource "grafana_oncall_integration" "AlertManager" {
     provider = grafana.oncall

     name = "AlertManager"
     type = "alertmanager"
     default_route {
       escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
     }
   }
   ```

1. To configure Alertmanager, refer to the [Alertmanager integration for Grafana OnCall](https://grafana.com/docs/grafana-cloud/alerting-and-irm/oncall/integrations/alertmanager/) documentation.

## Set up a route to configure escalation behavior for alert group notifications

This Terraform configuration sets up a route to the Alertmanager integration using the `grafana_oncall_route` (Resource).
This route ensures that notifications for alerts with `\"namespace\" *: *\"ops-.*\"` in the payload are escalated to the `SREs` escalation chain.

Create a file named `routes.tf` and add the following:

```terraform
resource "grafana_oncall_route" "route_SREs" {
  provider = grafana.oncall

  integration_id      = grafana_oncall_integration.AlertManager.id
  escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
  routing_regex       = "\"namespace\" *: *\"ops-.*\""
  position            = 0
}
```

## Apply the Terraform configuration

In a terminal, run the following commands from the directory where all of the configuration files are located.

1. Initialize a working directory containing Terraform configuration files.

   ```shell
   terraform init
   ```

1. Preview the changes that Terraform will make.

   ```shell
   terraform plan
   ```

1. Apply the configuration files.

   ```shell
   terraform apply
   ```

## Validation

After you apply the changes in the Terraform configurations, you can verify the following:

- Two new Schedules named `Devs` and `SREs` are created in Grafana IRM:

  ![`Devs` and `SREs` OnCall schedules](/media/docs/grafana-cloud/alerting-and-irm/screenshot-oncall-schedules-tf.png)

- New Escalation chain named `SREs` is created in Grafana IRM:

  ![`SREs` escalation chain](/media/docs/grafana-cloud/alerting-and-irm/screenshot-oncall-escalation-sre-tf.png)

- New Escalation chain named `Devs` is created in Grafana IRM:

  ![`Devs` escalation chain](/media/docs/grafana-cloud/alerting-and-irm/screenshot-oncall-escalation-devs-tf.png)

- The Alertmanager integration is added and configured with escalation policies:

  ![Alertmanager integration for SREs escalation](/media/docs/grafana-cloud/alerting-and-irm/screenshot-oncall-alertmanager-tf.png)

## Summary

In this guide, you learned how to use Terraform to manage Grafana IRM by connecting an integration, configuring escalation policies, and setting up on-call schedules.

To learn more about managing Grafana Cloud using Terraform, refer to the [Grafana provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

# Install plugins in Grafana Cloud using Terraform

This guide shows you how to install plugins in Grafana Cloud using Terraform. For more information about Grafana plugins, refer to the [Find and use Grafana plugins](/docs/grafana-cloud/introduction/find-and-use-plugins/) documentation.

## Before you begin

Before you begin, ensure you have the following:

- A Grafana Cloud account; for more information on setting up a Grafana Cloud account, refer to [Get started](https://grafana.com/docs/grafana-cloud/get-started/).
- Terraform installed on your machine; for more information on how to install Terraform, refer to the [Terraform install documentation](https://developer.hashicorp.com/terraform/install).
- Administrator permissions in your Grafana instance; for more information on assigning Grafana RBAC roles, refer to [Assign RBAC roles](/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-control/assign-rbac-roles/).

{{< admonition type="note" >}}
Save all of the following Terraform configuration files in the same directory.
{{< /admonition >}}

## Configure the Grafana provider

Use this Terraform configuration to set up the [Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) to provide the authentication required to manage plugin resources.

1. Create a service account and token in Grafana. For more information on creating a service account and token, refer to [Service account tokens](https://grafana.com/docs/grafana/latest/administration/service-accounts/#service-account-tokens). You can also refer to [Creating and managing a Grafana Cloud stack using Terraform](../terraform-cloud-stack/) to set up a service account and a token.

1. Make sure that the token has the following permissions:

- `stack-plugins:read`
- `stack-plugins:write`
- `stack-plugins:delete`

Next, create a file named `main.tf` and add the following:

```terraform
   terraform {
     required_providers {
       grafana = {
         source  = "grafana/grafana"
         version = ">= 4.5.3"
       }
     }
   }

   provider "grafana" {
     cloud_api_url      = "<Stack-URL>"
     cloud_access_policy_token     = "<Service-account-token>"
   }
```

Replace the following field values:

- `Stack-URL` with the URL of your Grafana stack, for example `https://my-stack.grafana.net/`
- `Service-account-token` with the service account token that you created

## Create new plugin resource

Create a file named `plugins.tf` and add the following:

```terraform
resource "grafana_cloud_plugin_installation" "grafana-clock-panel" {
  stack_slug   = "<Your-Stack-Slug>"
  slug         = "grafana-clock-panel"
  version      = "latest"
}
```

## Summary

In this guide, you learned how to install a plugin in Grafana Cloud using Terraform.

To learn more about plugin installation, refer to the [Grafana provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/cloud_plugin_installation).

# Manage secrets in Grafana Cloud using Terraform

Use the Terraform Grafana provider to create and manage secure values in [Grafana Secrets Management](https://grafana.com/docs/grafana-cloud/platform/security-and-account-management/security-and-access/manage-secrets/) as code.
With Terraform, you keep secrets such as API keys, tokens, passwords, and certificates in version-controlled, declarative infrastructure alongside the rest of your Grafana stack.

This guide covers _secure values_, which you manage with the [`grafana_apps_secret_securevalue_v1beta1`](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_secret_securevalue_v1beta1) resource.
By default, Grafana stores secure values in the built-in _system keeper_, which requires no setup.
If you configure an additional keeper and set it as the active keeper for your namespace, Grafana stores new secure values there automatically.
For more information, refer to [Activate and use the keeper](#activate-and-use-the-keeper).
To store secure values in an external secret manager instead, [provision a keeper](#provision-a-keeper).

To create and manage secure values with Terraform, you complete the following tasks:

1. Create a service account token to authenticate the Terraform provider.
1. Configure the Grafana provider.
1. Define one or more secure values.
1. Run `terraform apply` to provision your secure values.

## Before you begin

To provision secure values with Terraform, you need the following:

- A Grafana Cloud instance that you can reach from the machine that runs Terraform.
- Permission to create service account tokens.

You also need the following tools at the minimum supported versions:

| Tool                                 | Minimum version                                                                                                               |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
| Terraform CLI                        | 1.11, for [write-only arguments](https://developer.hashicorp.com/terraform/language/resources/ephemeral#write-only-arguments) |
| `grafana/grafana` Terraform provider | 4.26.0                                                                                                                        |

If you don't have the Terraform CLI, refer to the [Terraform install documentation](https://developer.hashicorp.com/terraform/install).
The `grafana/grafana` provider downloads automatically when you run `terraform init` in [Provision the secure value with Terraform](#provision-the-secure-value-with-terraform).

{{< admonition type="note" >}}
All of the following Terraform configuration files should be saved in the same directory.
{{< /admonition >}}

## Create a service account token

Terraform authenticates against Grafana with a service account token.

To create a service account and token, follow these steps:

1. Create a service account and token in Grafana.
   To create them, refer to [Service account tokens](https://grafana.com/docs/grafana/latest/administration/service-accounts/#service-account-tokens).
   You can also refer to [Create and manage a Grafana Cloud stack using Terraform](../terraform-cloud-stack/) to set up a service account and token.

1. Make sure the service account has the role-based access control (RBAC) actions needed to manage secure values:
   - `secret.securevalues:create`
   - `secret.securevalues:read`
   - `secret.securevalues:write`
   - `secret.securevalues:delete`

   If you also provision a keeper (refer to [Provision a keeper](#provision-a-keeper)), the service account also needs the keeper actions, which cover activating a keeper:
   - `secret.keepers:create`
   - `secret.keepers:read`
   - `secret.keepers:write`
   - `secret.keepers:delete`

1. Copy the token and store it securely, because you can't view it again after you leave the page.

## Configure the Grafana provider

In your Terraform working directory, create a `main.tf` file:

```terraform
terraform {
  required_version = ">= 1.11.0"

  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 4.26.0"
    }
  }
}

variable "grafana_auth" {
  type      = string
  sensitive = true
}

provider "grafana" {
  url  = "<GRAFANA_URL>"
  auth = var.grafana_auth

  # Required for app-platform resources, including secure values.
  stack_id = <STACK_ID>
}
```

Replace the placeholders as follows:

- _`<GRAFANA_URL>`_ is the URL of your Grafana instance, for example `https://my-org.grafana.net`.
- _`<STACK_ID>`_ is your numeric Grafana Cloud stack ID, for example `123456`.

  To find your stack ID, open the [Grafana Cloud Portal](https://grafana.com/profile/org), select your stack, and read the numeric **Instance ID**. That value is your `stack_id`.

  You can also retrieve it with a `GET` request to `https://grafana.com/api/orgs/<ORG_SLUG>/instances`, where _`<ORG_SLUG>`_ is your Grafana Cloud organization name.
  The numeric `id` field in the response is the stack ID.
  This Cloud API endpoint authenticates with a [Cloud Access Policy token](https://grafana.com/docs/grafana-cloud/developer-resources/api-reference/cloud-api/), which is separate from the service account token that Terraform uses.

Provide each input variable through a matching environment variable that follows the Terraform `TF_VAR_<variable_name>` naming convention.
The part after `TF_VAR_` is the name of the variable in your configuration: Terraform maps `TF_VAR_grafana_auth` to the `grafana_auth` variable, `TF_VAR_external_api_key` to `external_api_key`, and so on.
You can also use a secrets manager or a `terraform.tfvars` file.
Don't set variables in a file that you commit.

For other authentication options, refer to the [Grafana provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs#authentication).
For end-to-end stack management, refer to [Create and manage a Grafana Cloud stack using Terraform](../terraform-cloud-stack/).

## Define a secure value

The following example stores an API key as a secure value and authorizes Synthetic Monitoring to decrypt it.
Add it to `main.tf`, or to any other `.tf` file in the same working directory.
Terraform loads all `.tf` files in the directory together.

```terraform
variable "external_api_key" {
  type      = string
  sensitive = true
  ephemeral = true # Keeps the value out of state and plan files.
}

resource "grafana_apps_secret_securevalue_v1beta1" "external_api_key" {
  metadata {
    uid = "external-api-key" # The secure value's name.
  }

  spec {
    description = "External API key"
    value       = var.external_api_key
    decrypters  = ["synthetic-monitoring"]
  }
}
```

Keep the following points in mind for this example:

- `metadata.uid` is the secure value's _name_ in Grafana.
  The resource also exposes a separate, read-only `metadata.uuid`, which is the Grafana-generated identifier rather than the name.
- `spec.description` is required and limited to 25 characters.
  Longer values fail at plan with a Terraform length validation error.
- `spec.value` is a [write-only argument](https://developer.hashicorp.com/terraform/language/resources/ephemeral#write-only-arguments) that requires Terraform 1.11 or later.
  Terraform sends the plaintext to Grafana on apply, but never writes it to state, plan output, or any other on-disk artifact.
  Combine it with an `ephemeral`, sensitive variable, as in the preceding example, so the value also stays out of `terraform plan` output.
- You don't pass a `namespace`. The provider derives it from `stack_id`.
- You don't pass a keeper name. Grafana stores the value in whichever keeper is the active keeper for your namespace, which is the system keeper by default.

### Schema reference

The full schema is on the [Terraform Registry](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_secret_securevalue_v1beta1).
The following tables describe the fields you use for secure values.

The `metadata` block contains the following fields:

| Field                                   | Required | Description                                                                   |
| --------------------------------------- | -------- | ----------------------------------------------------------------------------- |
| `uid`                                   | Yes      | The secure value's name. Up to 253 characters, DNS-compatible, and immutable. |
| `folder_uid`                            | No       | Not used for secure values. Leave it unset.                                   |
| `uuid`, `version`, `url`, `annotations` | No       | Read-only. Grafana populates these fields.                                    |

The `spec` block contains the following fields:

| Field         | Required                  | Description                                                                                                                                                                                                                                                                                            |
| ------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `description` | Yes, enforced server-side | 1 to 25 characters.                                                                                                                                                                                                                                                                                    |
| `value`       | Conditional               | Plaintext secret value, 1 to 24,576 bytes. Sensitive and [write-only](https://developer.hashicorp.com/terraform/language/resources/ephemeral#write-only-arguments) in Terraform 1.11 or later. Sent to Grafana on apply but never written to state or plan files. Set exactly one of `value` or `ref`. |
| `ref`         | Conditional               | Reference to a secret already stored in a third-party keeper, 1 to 1,024 characters. Valid only when the active keeper for the namespace isn't the system keeper. Set exactly one of `value` or `ref`.                                                                                                 |
| `decrypters`  | No                        | List of services permitted to decrypt this secure value. Up to 64 unique entries. Each entry must be a valid Kubernetes label value: letters, numbers, and the characters `.`, `-`, and `_`, with no slashes or spaces. For more information, refer to [Decrypters](#decrypters).                      |
| `value_hash`  | Computed                  | Read-only SHA-256 hash of the stored value. The provider uses it to detect rotation. Don't set it yourself.                                                                                                                                                                                            |

The `options` block is optional and contains the following fields:

| Field              | Required | Description                                                                                                                                                                                    |
| ------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `manager_identity` | No       | Overrides the manager identity stamped on this resource. The default is `grafana-terraform-provider`. Useful when multiple Terraform workspaces manage resources in the same Grafana instance. |
| `overwrite`        | No       | Set to `true` to overwrite an existing resource with the same UID when applicable.                                                                                                             |

At the top level, the resource exposes a read-only `id` field, which Grafana derives from the secure value's UUID.

### Decrypters

The `decrypters` list controls which Grafana services can decrypt a secret. If a service isn't in the list, it can't read the plaintext value, even if it has access to the secure value's metadata. An empty or unset list means no service can decrypt the value.

Grafana supports the following decrypters:

| Decrypter              | Consumer                    |
| ---------------------- | --------------------------- |
| `k6-cloud`             | Grafana Cloud k6            |
| `synthetic-monitoring` | Synthetic Monitoring checks |

Grant the minimum set of decrypters required for your use case.

## Provision the secure value with Terraform

To apply the configuration, follow these steps:

1. Initialize the working directory.
   This downloads the Grafana provider.

   ```shell
   terraform init
   ```

1. Set the environment variables, then preview and apply the changes:

   ```shell
   export TF_VAR_grafana_auth='<TOKEN>'
   export TF_VAR_external_api_key='<SECRET>'
   terraform plan
   terraform apply
   ```

   Replace the placeholders as follows:
   - _`<TOKEN>`_ is the service account token you created in [Create a service account token](#create-a-service-account-token).
   - _`<SECRET>`_ is the plaintext secret value that Terraform stores as the secure value.

   This keeps secrets out of Terraform files.
   If your shell writes history, prefer a secure secret-injection flow, such as CI/CD secrets, a secret manager, or a nested shell with history disabled.

   Terraform shows the execution plan and asks for confirmation:

   ```console
   Plan: 1 to add, 0 to change, 0 to destroy.

   Do you want to perform these actions?
     Terraform will perform the actions described above.
     Only 'yes' will be accepted to approve.

     Enter a value:
   ```

   After you confirm, Terraform creates the secure value:

   ```console
   Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
   ```

If a secure value with the same `metadata.uid` already exists in the namespace, for example from a previous apply or one created outside Terraform, the apply fails with `HTTP 409 - AlreadyExists`.
To bring the existing secure value under Terraform management instead of recreating it, [import it](#import-an-existing-secure-value).
To replace it on apply, set `overwrite = true` in the resource's `options` block.

### Verify that the plaintext stays out of state

Because `spec.value` is write-only, Terraform never writes the plaintext to state.
It writes only the `value_hash`.
To confirm this, search your local state file for the plaintext:

```shell
grep -c '<SECRET_VALUE>' terraform.tfstate
```

Replace _`<SECRET_VALUE>`_ with the plaintext secret you provisioned.
The search returns no matches:

```console
0
```

The stored `spec` shows `value` as `null` and retains only the hash:

```json
"spec": {
  "decrypters": ["synthetic-monitoring"],
  "description": "External API key",
  "ref": null,
  "value": null,
  "value_hash": "65f087ba121c13dec3fa0cb7697d4309ed5e624ba528a758bb9071bee220825a"
}
```

To verify the result, open **Administration** > **Secrets Management** in Grafana, or call the [Secrets Management HTTP API](https://grafana.com/docs/grafana-cloud/developer-resources/api-reference/http-api/secrets_management/) directly.
The namespace is `stacks-<STACK_ID>`:

```shell
curl -H "Authorization: Bearer <TOKEN>" \
  "<GRAFANA_URL>/apis/secret.grafana.app/v1beta1/namespaces/stacks-<STACK_ID>/securevalues/external-api-key"
```

The response includes the secure value's metadata, its `spec` with `description` and `decrypters`, and the keeper that stores it.
It never includes the plaintext `value`:

```json
{
  "kind": "SecureValue",
  "apiVersion": "secret.grafana.app/v1beta1",
  "metadata": {
    "name": "external-api-key",
    "namespace": "stacks-<STACK_ID>",
    "annotations": {
      "grafana.app/managedBy": "terraform",
      "grafana.app/managerId": "grafana-terraform-provider"
    }
  },
  "spec": {
    "description": "External API key",
    "decrypters": ["synthetic-monitoring"]
  },
  "status": {
    "version": 1,
    "keeper": "system"
  }
}
```

There is no `value` field in the response. The API doesn't return decrypted secret contents.

## Rotate a secure value

To rotate a secret, change the value, or the variable that feeds it, and run `terraform apply` again.

The provider computes a SHA-256 hash of your configured `spec.value` at plan time and stores it in `value_hash`. When the hash changes, Terraform plans an update. When the hash doesn't change, Terraform plans no change. This lets you rotate secrets without storing plaintext in state.

```terraform
resource "grafana_apps_secret_securevalue_v1beta1" "external_api_key" {
  metadata {
    uid = "external-api-key"
  }

  spec {
    description = "External API key"
    value       = var.external_api_key # Supply the new value here.
    decrypters  = ["synthetic-monitoring"]
  }
}
```

When the value changes, the recomputed `value_hash` drives an in-place update:

```console
  ~ spec {
      ~ value_hash = (sensitive value)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
```

If you apply again without changing the value, the hash matches and Terraform plans no change:

```console
No changes. Your infrastructure matches the configuration.
```

### Rotation considerations

Keep the following points in mind when you rotate secure values:

- **`value` is write-only.**
  The Grafana API never returns the plaintext, and the provider never reads it back.
  Only `value_hash` is stored in state.
- **Out-of-band value changes aren't directly detectable.**
  Because the plaintext is write-only, Terraform can't read the current secret contents from Grafana.
  Keep Terraform as your source of truth for rotations.
- **You can't switch a secure value between `value` and `ref` after creation.**
  The API rejects updates that try to change mode.
  To change mode, destroy and recreate the resource.
- **`metadata.uid` is immutable.**
  Changing it forces Terraform to destroy and recreate the secure value.
- **`decrypters` ordering is significant in Terraform.**
  Keep a stable order in your configuration to avoid noisy plans.
- **An update is a full replace with `PUT`, not a patch.**
  Always provide `description` and `decrypters` together with `value` or `ref` on update.
  If you omit them, the new spec replaces the old one in full.

### Handle the secret value safely

{{< admonition type="warning" >}}
Never check plaintext secret values into Terraform configuration.
{{< /admonition >}}

Instead, use one of the following approaches:

- A `sensitive` or `ephemeral` Terraform variable populated from a `TF_VAR_*` environment variable.
- A CI/CD secret injected at apply time.
- A dedicated secret store, such as HashiCorp Vault or AWS Secrets Manager, referenced from Terraform with the appropriate provider data source.

## Import an existing secure value

The `grafana_apps_secret_securevalue_v1beta1` resource supports import.
The import ID is the secure value name, which is the value of `metadata.uid`:

```shell
terraform import grafana_apps_secret_securevalue_v1beta1.external_api_key external-api-key
```

After you import a secure value, keep the following points in mind:

- `spec.value` is still write-only, and Terraform never imports it into state.
- For value-backed secure values, set `spec.value` in configuration before you apply.
  The first managed apply sets or rotates the value to whatever you configure.
- For reference-backed secure values, set `spec.ref` in configuration to match the existing reference.
- Because you must set exactly one of `value` or `ref`, your configuration must include one of them after import.

## Provision a keeper

A _keeper_ defines where Grafana stores the encrypted material for your secure values. By default, Grafana uses the built-in system keeper, which encrypts values at rest in the Grafana database and requires no setup.

To store secrets in an external secret manager instead, provision a keeper with the [`grafana_apps_secret_keeper_v1beta1`](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_secret_keeper_v1beta1) resource.
Grafana currently supports AWS Secrets Manager as a keeper.

Grafana connects to AWS Secrets Manager through cross-account role assumption with AWS Security Token Service, and never stores your AWS credentials.
You create an AWS Identity and Access Management (IAM) role in your AWS account that Grafana can assume, and the keeper references that role.

### Keeper requirements

In addition to the [requirements for secure values](#before-you-begin), you need the following:

- An AWS account with AWS Secrets Manager enabled in your target region.
- An IAM role named exactly `grafana-secrets-manager` that Grafana can assume, along with its role ARN and external ID.
  Grafana can assume only a role with that name.
  For the IAM role and trust policy that the role requires, refer to [Configure an AWS Secrets Manager keeper](https://grafana.com/docs/grafana-cloud/platform/security-and-account-management/security-and-access/configure-aws-secrets-manager-keeper/).

### Define a keeper

The following example provisions an AWS Secrets Manager keeper. Add it to `main.tf`, or to any other `.tf` file in the same working directory.

```terraform
resource "grafana_apps_secret_keeper_v1beta1" "aws_secrets_manager" {
  metadata {
    uid = "aws-secrets-manager" # The keeper's name.
  }

  spec {
    description = "AWS Secrets Manager keeper"

    aws {
      region = "<REGION>"

      assume_role {
        assume_role_arn = "arn:aws:iam::<ACCOUNT_ID>:role/grafana-secrets-manager"
        external_id     = "<EXTERNAL_ID>"
      }
    }
  }
}
```

Replace the placeholders as follows:

- _`<REGION>`_ is the AWS region that hosts your secrets, for example `us-east-1`.
- _`<ACCOUNT_ID>`_ is the ID of the AWS account that hosts your secrets.
- _`<EXTERNAL_ID>`_ is the external ID that your IAM role's trust policy requires.

The role name in `assume_role_arn` must be `grafana-secrets-manager`. Grafana can't assume a role with any other name.

Apply the keeper with the same `terraform apply` workflow you use for a secure value, described in [Provision the secure value with Terraform](#provision-the-secure-value-with-terraform).
Unlike a secure value, a keeper has no secret value, so it needs no `ephemeral` variable of its own: `TF_VAR_grafana_auth` for the provider is the only variable you export.
Creating a keeper doesn't make it active, so don't create secure values in the same apply until you add the activation resource in [Activate and use the keeper](#activate-and-use-the-keeper) and make new secrets depend on it.

### Keeper schema reference

The full schema is on the [Terraform Registry](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_secret_keeper_v1beta1). The following tables describe the fields you use for a keeper.

The `metadata` block contains the following fields:

| Field | Required | Description                                                             |
| ----- | -------- | ----------------------------------------------------------------------- |
| `uid` | Yes      | The keeper's name. Up to 253 characters, DNS-compatible, and immutable. |

The `spec` block contains the following fields:

| Field         | Required | Description                                                                                                                                            |
| ------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `description` | Yes      | Short description for the keeper, 1 to 253 characters.                                                                                                 |
| `aws`         | Yes      | AWS Secrets Manager configuration. Grafana currently supports only AWS, so this block is required. For the fields, refer to the following `aws` table. |

The `aws` block contains the following fields:

| Field         | Required | Description                                                                                                                       |
| ------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `region`      | Yes      | AWS region that hosts your secrets, for example `us-east-1`.                                                                      |
| `assume_role` | Yes      | Role that Grafana assumes to access AWS Secrets Manager. Required for AWS. `assume_role_arn` and `external_id` are both required. |

The `assume_role` block contains the following fields:

| Field             | Required | Description                                            |
| ----------------- | -------- | ------------------------------------------------------ |
| `assume_role_arn` | Yes      | ARN of the IAM role that Grafana assumes.              |
| `external_id`     | Yes      | External ID that the IAM role's trust policy requires. |

At the top level, the resource exposes a read-only `active` status that's `true` when the keeper is the active keeper for the namespace.

### Activate and use the keeper

Creating a keeper with Terraform doesn't make it active. Each namespace has at most one active keeper, which determines where Grafana stores new secure values.

To set a keeper as active, use the [`grafana_apps_secret_keeper_activation_v1beta1`](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_secret_keeper_activation_v1beta1) resource and reference the keeper's `uid`:

```terraform
resource "grafana_apps_secret_keeper_activation_v1beta1" "aws_secrets_manager" {
  metadata {
    uid = grafana_apps_secret_keeper_v1beta1.aws_secrets_manager.metadata.uid
  }
}
```

The activation resource already depends on the keeper because it references the keeper's `uid`.
Any secure value that should use this keeper must depend on the activation, otherwise Terraform can create the secret while the system keeper is still active:

```terraform
resource "grafana_apps_secret_securevalue_v1beta1" "external_api_key" {
  metadata {
    uid = "external-api-key"
  }

  spec {
    description = "External API key"
    value       = var.external_api_key
    decrypters  = ["synthetic-monitoring"]
  }

  depends_on = [grafana_apps_secret_keeper_activation_v1beta1.aws_secrets_manager]
}
```

Keep the following points in mind for this resource:

- Because a namespace has only one active keeper, applying an activation for a different keeper switches which keeper is active.
- Running `terraform destroy` on the activation resource reverts the namespace to the built-in system keeper.
  Destroying the keeper resource doesn't deactivate it first; destroy the activation resource as well.

You can also set the active keeper outside Terraform, in the Grafana UI under **Administration** > **Secrets Management**.

After the keeper is active, Grafana routes new secure values to it automatically.
A `value`-backed secure value like the one in [Define a secure value](#define-a-secure-value) needs no field changes to use the active keeper, but it still needs `depends_on` if you create it in the same configuration as the activation.
Destroying a `value`-backed secure value that lives in AWS Secrets Manager deletes the AWS secret immediately, with no recovery window.

To reference a secret that already exists in AWS Secrets Manager instead of sending a new value, use `ref` in place of `value`.
Set `ref` to the AWS secret name or the full ARN, not a path inside Grafana:

```terraform
resource "grafana_apps_secret_securevalue_v1beta1" "db_password" {
  metadata {
    uid = "db-password"
  }

  spec {
    description = "Production DB password"
    ref         = "grafana-secrets-manager/stacks-<STACK_ID>/db-password/1"
    decrypters  = ["synthetic-monitoring"]
  }

  depends_on = [grafana_apps_secret_keeper_activation_v1beta1.aws_secrets_manager]
}
```

Replace _`<STACK_ID>`_ with the same numeric stack ID you set on the provider.
You can also set `ref` to the full secret ARN, for example `arn:aws:secretsmanager:<REGION>:<ACCOUNT_ID>:secret:grafana-secrets-manager/stacks-<STACK_ID>/db-password/1-AbCdEf`.

If you use `ref` while the system keeper is active, the API returns the following error:

```console
tried to create secure value using reference with system keeper, references can only be used with 3rd party keepers
```

## Summary

In this guide, you created a service account token, configured the Grafana provider, and provisioned a secure value with Terraform.
You also learned how to rotate and import secure values, and how to provision and activate a keeper to store secrets in an external secret manager.

To learn more about managing Grafana Cloud using Terraform, refer to the [Grafana provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

## Next steps

- For the full secure value schema and more examples, refer to [`grafana_apps_secret_securevalue_v1beta1` on the Terraform Registry](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_secret_securevalue_v1beta1).
- For the full keeper schema, refer to [`grafana_apps_secret_keeper_v1beta1` on the Terraform Registry](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_secret_keeper_v1beta1).
- For the keeper activation resource, refer to [`grafana_apps_secret_keeper_activation_v1beta1` on the Terraform Registry](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/apps_secret_keeper_activation_v1beta1).
- For concepts, permissions, and limits, refer to [Manage secrets](https://grafana.com/docs/grafana-cloud/platform/security-and-account-management/security-and-access/manage-secrets/).
- For the underlying HTTP API, refer to the [Secrets Management API](https://grafana.com/docs/grafana-cloud/developer-resources/api-reference/http-api/secrets_management/).
- For more about write-only arguments, refer to [Ephemeral values](https://developer.hashicorp.com/terraform/language/resources/ephemeral#write-only-arguments) in the Terraform documentation.

# Create custom model rules using Terraform

Custom model rules in [Knowledge Graph](/docs/grafana-cloud/knowledge-graph/) allow you to define how entities are discovered and modeled based on Prometheus queries. These rules enable you to create custom entity types, define their relationships, and specify how they should be enriched with additional data.

For information about managing entities and relations in the Knowledge Graph UI, refer to [Manage entities and relations](/docs/grafana-cloud/knowledge-graph/configure/manage-entities-relations/).

## Basic custom model rules

Create a file named `custom-model-rules.tf` and add the following:

```terraform
# Basic custom model rule for services
resource "grafana_asserts_custom_model_rules" "basic_service" {
  provider = grafana.asserts

  name = "basic-service-model"

  rules {
    entity {
      type = "Service"
      name = "service"

      defined_by {
        query = "up{job!=''}"
        label_values = {
          service = "job"
        }
        literals = {
          _source = "up_query"
        }
      }
    }
  }
}
```

## Advanced service model with scope and lookup

Define service entities with environment scoping and relationship mappings:

```terraform
# Advanced service model with environment scoping
resource "grafana_asserts_custom_model_rules" "advanced_service" {
  provider = grafana.asserts

  name = "advanced-service-model"

  rules {
    entity {
      type = "Service"
      name = "workload | service | job"

      scope = {
        namespace = "namespace"
        env       = "asserts_env"
        site      = "asserts_site"
      }

      lookup = {
        workload  = "workload | deployment | statefulset | daemonset | replicaset"
        service   = "service"
        job       = "job"
        proxy_job = "job"
      }

      defined_by {
        query = "up{job!='', asserts_env!=''}"
        label_values = {
          service     = "service"
          job         = "job"
          workload    = "workload"
          namespace   = "namespace"
        }
        literals = {
          _source = "up_with_workload"
        }
      }

      defined_by {
        query    = "up{job='maintenance'}"
        disabled = true
      }
    }
  }
}
```

## Multi-entity model configuration

Define multiple entity types in a single configuration:

```terraform
# Multiple entity types in a single model
resource "grafana_asserts_custom_model_rules" "multi_entity" {
  provider = grafana.asserts

  name = "kubernetes-entities"

  rules {
    # Service entity
    entity {
      type = "Service"
      name = "service"

      scope = {
        namespace = "namespace"
        cluster   = "cluster"
      }

      defined_by {
        query = "up{service!=''}"
        label_values = {
          service   = "service"
          namespace = "namespace"
          cluster   = "cluster"
        }
      }
    }

    # Pod entity
    entity {
      type = "Pod"
      name = "Pod"

      scope = {
        namespace = "namespace"
        cluster   = "cluster"
      }

      lookup = {
        service   = "service"
        workload  = "workload"
      }

      defined_by {
        query = "kube_pod_info{pod!=''}"
        label_values = {
          Pod       = "pod"
          namespace = "namespace"
          cluster   = "cluster"
          service   = "service"
        }
        literals = {
          _entity_type = "Pod"
        }
      }
    }

    # Namespace entity
    entity {
      type = "Namespace"
      name = "namespace"

      scope = {
        cluster = "cluster"
      }

      defined_by {
        query = "kube_namespace_status_phase{namespace!=''}"
        label_values = {
          namespace = "namespace"
          cluster   = "cluster"
        }
      }
    }
  }
}
```

## Complex entity with enrichment

Create service entities with multiple data sources and enrichment:

```terraform
# Service entity with enrichment from multiple sources
resource "grafana_asserts_custom_model_rules" "enriched_service" {
  provider = grafana.asserts

  name = "enriched-service-model"

  rules {
    entity {
      type = "Service"
      name = "service"

      enriched_by = [
        "prometheus_metrics",
        "kubernetes_metadata",
        "application_logs"
      ]

      scope = {
        environment = "asserts_env"
        region      = "asserts_site"
        team        = "team"
      }

      lookup = {
        deployment = "workload"
        Pod        = "pod"
        container  = "container"
      }

      # Primary definition from service up metrics
      defined_by {
        query = "up{service!='', asserts_env!=''}"
        label_values = {
          service     = "service"
          environment = "asserts_env"
          region      = "asserts_site"
          team        = "team"
        }
        literals = {
          _primary_source = "service_up"
        }
      }

      # Secondary definition from application metrics
      defined_by {
        query = "http_requests_total{service!=''}"
        label_values = {
          service     = "service"
          environment = "environment"
          version     = "version"
        }
        literals = {
          _secondary_source = "http_metrics"
        }
      }

      # Disabled definition for testing
      defined_by {
        query    = "test_metric{service!=''}"
        disabled = true
      }
    }
  }
}
```

## Database and infrastructure entities

Define database and infrastructure entity models:

```terraform
# Database and infrastructure entity models
resource "grafana_asserts_custom_model_rules" "infrastructure" {
  provider = grafana.asserts

  name = "infrastructure-entities"

  rules {
    # Database entity
    entity {
      type = "Database"
      name = "database_instance"

      scope = {
        environment = "env"
        region      = "region"
      }

      lookup = {
        host     = "instance"
        port     = "port"
        db_name  = "database"
      }

      defined_by {
        query = "mysql_up{instance!=''}"
        label_values = {
          database_instance = "instance"
          database         = "database"
          env             = "environment"
          region          = "region"
        }
        literals = {
          _db_type = "mysql"
        }
        metric_value = "1"
      }

      defined_by {
        query = "postgres_up{instance!=''}"
        label_values = {
          database_instance = "instance"
          database         = "datname"
          env             = "environment"
        }
        literals = {
          _db_type = "postgresql"
        }
      }
    }

    # Load balancer entity
    entity {
      type = "LoadBalancer"
      name = "lb_instance"

      scope = {
        environment = "env"
      }

      defined_by {
        query = "haproxy_up{proxy!=''}"
        label_values = {
          lb_instance = "instance"
          proxy      = "proxy"
          env        = "environment"
        }
        literals = {
          _lb_type = "haproxy"
        }
      }
    }
  }
}
```

## Resource reference

### `grafana_asserts_custom_model_rules`

Manage Knowledge Graph custom model rules through the Grafana API. This resource allows you to define custom entity models based on Prometheus queries with advanced mapping and enrichment capabilities.

#### Arguments

| Name    | Type           | Required | Description                                                                                              |
| ------- | -------------- | -------- | -------------------------------------------------------------------------------------------------------- |
| `name`  | `string`       | Yes      | The name of the custom model rules. This field is immutable and forces recreation if changed.            |
| `rules` | `list(object)` | Yes      | The rules configuration containing entity definitions. Refer to [rules block](#rules-block) for details. |

#### Rules block

Each `rules` block supports the following:

| Name     | Type           | Required | Description                                                                     |
| -------- | -------------- | -------- | ------------------------------------------------------------------------------- |
| `entity` | `list(object)` | Yes      | List of entity definitions. Refer to [entity block](#entity-block) for details. |

#### Entity block

Each `entity` block supports the following:

| Name          | Type           | Required | Description                                                                                            |
| ------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------ |
| `type`        | `string`       | Yes      | The type of the entity (for example, Service, Pod, Namespace).                                         |
| `name`        | `string`       | Yes      | The name pattern for the entity. Can include pipe-separated alternatives.                              |
| `defined_by`  | `list(object)` | Yes      | List of queries that define this entity. Refer to [`defined_by` block](#defined_by-block) for details. |
| `disabled`    | `bool`         | No       | Whether this entity is disabled. Defaults to `false`.                                                  |
| `enriched_by` | `list(string)` | No       | List of enrichment sources for the entity.                                                             |
| `lookup`      | `map(string)`  | No       | Lookup mappings for the entity to relate different label names.                                        |
| `scope`       | `map(string)`  | No       | Scope labels that define the boundaries of this entity type.                                           |

#### `defined_by` block

Each `defined_by` block supports the following:

| Name           | Type          | Required | Description                                                               |
| -------------- | ------------- | -------- | ------------------------------------------------------------------------- |
| `query`        | `string`      | Yes      | The Prometheus query that defines this entity.                            |
| `disabled`     | `bool`        | No       | Whether this query is disabled. Defaults to `false`.                      |
| `label_values` | `map(string)` | No       | Label value mappings for extracting entity attributes from query results. |
| `literals`     | `map(string)` | No       | Literal value mappings for adding static attributes to entities.          |
| `metric_value` | `string`      | No       | Metric value to use from the query result.                                |

{{< admonition type="note" >}}
When `disabled = true` is set for a `defined_by` query, only the `query` field is used for matching. All other fields in the block are ignored.
{{< /admonition >}}

## Best practices

Consider the following best practices when creating custom model rules with Terraform.

### Entity models

- Design your entity models to reflect your actual infrastructure and application architecture
- Use descriptive names for custom model rules that indicate their purpose and scope
- Start with basic entity definitions and gradually add complexity as needed
- Define clear entity scopes using the `scope` parameter to organize entities by environment, region, or team

### Query design and performance

- Write efficient Prometheus queries that don't overload your monitoring system
- Test your Prometheus queries independently before using them in model rules
- Use specific label filters to reduce the scope of your queries where possible
- Consider the cardinality implications of your entity definitions
- Use the `disabled` flag to temporarily disable problematic queries during debugging

### Relationships and enrichment

- Use `lookup` mappings to establish relationships between different entity types
- Leverage `enriched_by` to specify additional data sources for entity enrichment
- Map Prometheus labels to entity attributes using clear and descriptive names
- Use meaningful `literals` to add static metadata that helps with entity identification

### Label and attribute management

- Establish consistent labeling conventions across your infrastructure
- Use `label_values` to extract dynamic attributes from your metrics
- Document the meaning and expected values of custom literals
- Ensure label names match across different entity definitions for proper relationship discovery

## Validation

After applying the Terraform configuration, verify that:

- Custom model rules are applied in your Knowledge Graph instance
- Entities are being discovered according to your defined queries
- Entity relationships and enrichment are working as expected
- Entity graphs display the correct entity types and connections
- Queries perform well without causing excessive load

## Related documentation

- [Manage entities and relations in Knowledge Graph](/docs/grafana-cloud/knowledge-graph/configure/manage-entities-relations/)
- [Get started with Terraform for Knowledge Graph](../getting-started/)
- [Knowledge graph basics](/docs/grafana-cloud/knowledge-graph/knowledge-graph-basics/)

# Get started with Terraform for Knowledge Graph

Learn how to configure Terraform to manage Grafana Cloud Knowledge Graph resources. This guide walks you through setting up the Grafana Terraform provider and preparing your environment.

## Before you begin

Before you begin, ensure you have the following:

- A Grafana Cloud account, as shown in [Get started](/docs/grafana-cloud/get-started/)
- [Terraform](https://www.terraform.io/downloads) installed on your machine
- Administrator permissions in your Grafana instance

{{< admonition type="note" >}}
You don't need to enable the Knowledge Graph before using Terraform. The [`grafana_asserts_stack` resource](../knowledge-graph-stack/) handles the full onboarding flow, including enabling the Knowledge Graph on your stack.
{{< /admonition >}}

{{< admonition type="note" >}}
All Terraform configuration files should be saved in the same directory.
{{< /admonition >}}

## Configure the Grafana provider

This Terraform configuration sets up the [Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) to provide necessary authentication when managing knowledge graph resources.

You can reuse a similar setup to the one described in [Creating and managing a Grafana Cloud stack using Terraform](/docs/grafana-cloud/as-code/terraform/terraform-cloud-stack/) to set up a service account and a token.

### Steps

1. Create a Service account and token in Grafana.

   To create a new one, refer to [Service account tokens](/docs/grafana/latest/administration/service-accounts/#service-account-tokens).

1. Create a file named `main.tf` and add the following:

   ```terraform
   terraform {
     required_providers {
       grafana = {
         source  = "grafana/grafana"
         version = ">= 2.9.0"
       }
     }
   }

   provider "grafana" {
     alias = "asserts"

     url      = "<Stack-URL>"
     auth     = "<Service-account-token>"
     stack_id = "<Stack-ID>"
   }
   ```

1. Replace the following field values:
   - `<Stack-URL>` with the URL of your Grafana stack (for example, `https://my-stack.grafana.net/`)
   - `<Service-account-token>` with the service account token that you created
   - `<Stack-ID>` with your Grafana Cloud stack ID

{{< admonition type="note" >}}
The `stack_id` parameter is required for Knowledge Graph resources to identify the stack where the resources belong.
{{< /admonition >}}

## Apply Terraform configurations

After creating your Terraform configuration files, apply them using the following commands:

1. Initialize a working directory containing Terraform configuration files:

   ```shell
   terraform init
   ```

1. Preview the changes that Terraform makes:

   ```shell
   terraform plan
   ```

1. Apply the configuration files:

   ```shell
   terraform apply
   ```

## Verify your setup

After applying the configuration, verify your setup by checking that:

- Terraform can authenticate with your Grafana Cloud stack
- The provider is properly configured with the correct stack ID
- No errors appear in the Terraform output

## Best practices

When managing Knowledge Graph resources with Terraform, consider the following best practices.

### Name conventions

- Use descriptive names that clearly indicate the purpose of each resource
- Follow a consistent naming pattern across your organization
- Include environment or team identifiers in names when appropriate

### Version control

- Store your Terraform configurations in version control (Git)
- Use separate directories or workspaces for different environments
- Document changes in commit messages

### State management

- Use remote state backends for team collaboration
- Enable state locking to prevent concurrent modifications
- Regularly back up your Terraform state files

### Security

- Never commit service account tokens or sensitive data to version control
- Use environment variables or secret management tools for credentials
- Rotate service account tokens regularly

## Next steps

Now that you have configured the Terraform provider, you can start managing knowledge graph resources:

- [Configure notification alerts](../notification-alerts/)
- [Define suppressed assertions](../suppressed-assertions/)
- [Create custom model rules](../custom-model-rules/)
- [Set up log configurations](../log-configurations/)
- [Configure thresholds](../thresholds/)
- [Configure the Knowledge Graph stack](../knowledge-graph-stack/)
- [Define Prometheus rules](../prometheus-rules/)

# Configure the Knowledge Graph stack using Terraform

The Knowledge Graph stack resource manages the full onboarding flow for your Grafana Cloud stack. It provisions API tokens, configures datasets based on available metrics, and enables the stack.

By default, datasets are auto-configured based on detected metrics. To manually configure datasets, for example when your metrics use non-standard label names, use the `dataset` block.

## Before you begin

Before you begin, ensure you have the following:

- A Grafana Cloud account, as shown in [Get started](/docs/grafana-cloud/get-started/)
- [Terraform](https://www.terraform.io/downloads) installed on your machine
- The Grafana Terraform provider configured, as shown in [Get started with Terraform for Knowledge Graph](../getting-started/)
- A Cloud Access Policy with the following scopes: `stacks:read`, `metrics:read`, `metrics:write`

## Dataset types

The following dataset types are available:

- **`kubernetes`**: Kubernetes metrics. Requires [Kubernetes Monitoring](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/) to be enabled on your stack.
- **`otel`**: Application Observability metrics. Requires [Application Observability](/docs/grafana-cloud/monitor-applications/application-observability/) to be enabled on your stack.
- **`prometheus`**: Standard Prometheus metrics.
- **`aws`**: Amazon Web Services metrics.

## Create the required tokens

Before you configure the Knowledge Graph stack, create the Cloud Access Policy and Grafana Service Account tokens that the resource requires.

Create a file named `knowledge-graph-stack.tf` and add the following:

```terraform
# Create a Cloud Access Policy with required scopes
resource "grafana_cloud_access_policy" "knowledge_graph" {
  name         = "knowledge-graph-stack-policy"
  display_name = "Knowledge Graph Stack Policy"

  scopes = [
    "stacks:read",
    "metrics:read",
    "metrics:write",
  ]

  realm {
    type       = "stack"
    identifier = var.stack_id
  }
}

# Create a token from the Cloud Access Policy
resource "grafana_cloud_access_policy_token" "knowledge_graph" {
  name             = "knowledge-graph-stack-token"
  access_policy_id = grafana_cloud_access_policy.knowledge_graph.policy_id
}

# Create a Grafana Service Account for dashboards and Grafana Managed Alerts
resource "grafana_cloud_stack_service_account" "knowledge_graph" {
  stack_slug  = var.stack_slug
  name        = "knowledge-graph-managed-alerts-sa"
  role        = "Admin"
  is_disabled = false
}

resource "grafana_cloud_stack_service_account_token" "knowledge_graph" {
  stack_slug         = var.stack_slug
  service_account_id = grafana_cloud_stack_service_account.knowledge_graph.id
  name               = "knowledge-graph-managed-alerts-token"
}

variable "stack_id" {
  description = "The Grafana Cloud stack ID"
  type        = string
}

variable "stack_slug" {
  description = "The Grafana Cloud stack slug"
  type        = string
}
```

Replace the following values:

- _`<STACK_ID>`_ with your Grafana Cloud stack ID
- _`<STACK_SLUG>`_ with your Grafana Cloud stack slug

## Basic stack configuration with auto-detected datasets

The simplest configuration provisions tokens and auto-detects datasets based on available metrics. This is the recommended approach when your metrics use standard label names.

Add the following to `knowledge-graph-stack.tf`:

```terraform
resource "grafana_asserts_stack" "main" {
  cloud_access_policy_token = grafana_cloud_access_policy_token.knowledge_graph.token
  grafana_token             = grafana_cloud_stack_service_account_token.knowledge_graph.key
}
```

When you apply this configuration, the resource:

1. Provisions API tokens for Grafana Cloud, Mimir, and the assertion detector
1. Auto-detects available datasets based on your stack's metrics
1. Enables the stack with the detected datasets

## Configure the dataset manually

Use manual configuration when your metrics use non-standard label names, for example a custom environment label. Each `dataset` block configures one dataset type.

```terraform
resource "grafana_asserts_stack" "manual" {
  cloud_access_policy_token = grafana_cloud_access_policy_token.knowledge_graph.token
  grafana_token             = grafana_cloud_stack_service_account_token.knowledge_graph.key

  dataset {
    type = "kubernetes"

    filter_group {
      env_label  = "deployment_environment"
      site_label = "cluster"

      env_label_values  = ["production", "staging"]
      site_label_values = ["us-east-1", "eu-west-1"]
    }
  }
}
```

{{< admonition type="note" >}}
When you specify one or more `dataset` blocks, auto-detection is skipped. You must define all datasets you want to configure.
{{< /admonition >}}

## Configure multiple datasets with filters

Configure multiple dataset types with custom label mappings and metric filters:

```terraform
resource "grafana_asserts_stack" "multi_dataset" {
  cloud_access_policy_token = grafana_cloud_access_policy_token.knowledge_graph.token
  grafana_token             = grafana_cloud_stack_service_account_token.knowledge_graph.key

  dataset {
    type = "kubernetes"

    filter_group {
      env_label  = "deployment_environment"
      site_label = "cluster"

      env_label_values  = ["production", "staging"]
      site_label_values = ["us-east-1", "eu-west-1"]
    }
  }

  dataset {
    type = "prometheus"

    filter_group {
      env_label = "environment"
      env_name  = "prod"

      filter {
        name     = "region"
        operator = "=~"
        values   = ["us-.*", "eu-.*"]
      }
    }
  }
}
```

## Resource reference

### `grafana_asserts_stack`

Manage the Knowledge Graph stack configuration through the Grafana API. This resource handles the full onboarding flow including token provisioning, dataset configuration, and stack enablement.

#### Arguments

| Name                        | Type           | Required | Description                                                                                                                                                                                                                       |
| --------------------------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cloud_access_policy_token` | `string`       | Yes      | A Grafana Cloud Access Policy token with the following scopes: `stacks:read`, `metrics:read`, `metrics:write`. Used for Grafana Cloud API access, Mimir authentication, and assertion detector webhook authentication. Sensitive. |
| `grafana_token`             | `string`       | No       | A Grafana Service Account token for installing dashboards and Grafana Managed Alerts. Create using `grafana_cloud_stack_service_account_token`. Sensitive.                                                                        |
| `dataset`                   | `list(object)` | No       | Manual dataset configuration. When specified, auto-detection is skipped. Refer to [dataset block](#dataset-block) for details.                                                                                                    |

#### Attributes

| Name      | Type     | Description                             |
| --------- | -------- | --------------------------------------- |
| `enabled` | `bool`   | Whether the stack is currently enabled. |
| `status`  | `string` | Current onboarding status of the stack. |
| `version` | `int`    | Configuration version number.           |

#### Dataset block

Each `dataset` block supports the following:

| Name               | Type           | Required | Description                                                                                              |
| ------------------ | -------------- | -------- | -------------------------------------------------------------------------------------------------------- |
| `type`             | `string`       | Yes      | The dataset type: `kubernetes`, `otel`, `prometheus`, or `aws`.                                          |
| `disabled_vendors` | `list(string)` | No       | List of vendors to disable for this dataset.                                                             |
| `filter_group`     | `list(object)` | No       | Filter groups for custom label mappings. Refer to [filter group block](#filter-group-block) for details. |

#### Filter group block

Each `filter_group` block supports the following:

| Name                | Type           | Required | Description                                                                                               |
| ------------------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------- |
| `env_label`         | `string`       | No       | The metric label name used for environment (for example, `env`, `environment`, `deployment_environment`). |
| `env_name`          | `string`       | No       | A friendly name for the environment.                                                                      |
| `site_label`        | `string`       | No       | The metric label name used for site or cluster.                                                           |
| `env_label_values`  | `list(string)` | No       | Specific values of the environment label to match.                                                        |
| `site_label_values` | `list(string)` | No       | Specific values of the site label to match.                                                               |
| `filter`            | `list(object)` | No       | Additional metric filters. Refer to [filter block](#filter-block) for details.                            |

#### Filter block

Each `filter` block supports the following:

| Name       | Type           | Required | Description                                    |
| ---------- | -------------- | -------- | ---------------------------------------------- |
| `name`     | `string`       | Yes      | The label name to filter on.                   |
| `operator` | `string`       | Yes      | The filter operator: `=`, `!=`, `=~`, or `!~`. |
| `values`   | `list(string)` | Yes      | The values to match against.                   |

#### Example

```terraform
resource "grafana_asserts_stack" "example" {
  cloud_access_policy_token = grafana_cloud_access_policy_token.knowledge_graph.token
  grafana_token             = grafana_cloud_stack_service_account_token.knowledge_graph.key

  dataset {
    type = "kubernetes"

    filter_group {
      env_label         = "deployment_environment"
      site_label        = "cluster"
      env_label_values  = ["production"]
      site_label_values = ["us-east-1"]
    }
  }
}
```

## Best practices

Consider the following best practices when configuring the Knowledge Graph stack with Terraform.

### Token management

- Create a dedicated Cloud Access Policy for the Knowledge Graph stack with only the required scopes
- Use separate Service Accounts for different purposes rather than sharing a single admin token
- Rotate tokens regularly and use sensitive variable handling in Terraform to avoid exposing tokens in logs
- Store tokens in a secret management system and reference them through Terraform variables

### Dataset configuration

- Start with auto-detected datasets and switch to manual configuration only when needed
- When using manual configuration, define all datasets you want to monitor
- Use specific `env_label_values` and `site_label_values` to limit the scope of monitoring
- Test dataset configurations in a non-production stack first

### Filter groups

- Use filter groups when your metrics use non-standard label names
- Match your `env_label` and `site_label` values to the actual label names in your Prometheus metrics
- Use regular expression operators (`=~`, `!~`) in filters for flexible value matching across regions or environments
- Keep filter configurations consistent across datasets for predictable behavior

## Validation

After applying the Terraform configuration, verify that:

- The stack status shows as enabled by checking the `enabled` output attribute
- The `status` attribute indicates successful onboarding
- Datasets are configured correctly in the Knowledge Graph UI
- Dashboards and alerting rules are installed when `grafana_token` is provided

You can inspect the stack status using Terraform outputs:

```terraform
output "stack_enabled" {
  value       = grafana_asserts_stack.main.enabled
  description = "Whether the Knowledge Graph stack is enabled"
}

output "stack_status" {
  value       = grafana_asserts_stack.main.status
  description = "Current onboarding status of the stack"
}

output "stack_version" {
  value       = grafana_asserts_stack.main.version
  description = "Configuration version number"
}
```

## Related documentation

- [Knowledge Graph documentation](/docs/grafana-cloud/knowledge-graph/)
- [Get started with Terraform for Knowledge Graph](../getting-started/)
- [Grafana Terraform Provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs)
- [Cloud Access Policies](/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/)
- [Service accounts](/docs/grafana/latest/administration/service-accounts/)

# Configure log correlation using Terraform

Log configurations in [Knowledge Graph](/docs/grafana-cloud/knowledge-graph/) allow you to define how log data is queried and correlated with entities. You can specify data sources, entity matching rules, label mappings, and filtering options for spans and traces.

For information about configuring log correlation in the Knowledge Graph UI, refer to [Configure logs correlation](/docs/grafana-cloud/knowledge-graph/configure/logs-correlation/).

## Basic log configuration

Create a file named `log-configs.tf` and add the following:

```terraform
# Basic log configuration for services
resource "grafana_asserts_log_config" "production" {
  provider = grafana.asserts

  name            = "production"
  priority        = 1000
  default_config  = false
  data_source_uid = "grafanacloud-logs"
  error_label     = "error"

  match {
    property = "asserts_entity_type"
    op       = "EQUALS"
    values   = ["Service"]
  }

  match {
    property = "environment"
    op       = "EQUALS"
    values   = ["production", "staging"]
  }

  entity_property_to_log_label_mapping = {
    "otel_namespace" = "service_namespace"
    "otel_service"   = "service_name"
    "environment"    = "env"
    "site"           = "region"
  }

  filter_by_span_id  = true
  filter_by_trace_id = true
}
```

## Log configuration with multiple match rules

Configure log correlation with multiple entity matching criteria:

```terraform
# Development environment log configuration
resource "grafana_asserts_log_config" "development" {
  provider = grafana.asserts

  name            = "development"
  priority        = 2000
  default_config  = true
  data_source_uid = "elasticsearch-dev"
  error_label     = "error"

  match {
    property = "asserts_entity_type"
    op       = "EQUALS"
    values   = ["Service"]
  }

  match {
    property = "environment"
    op       = "EQUALS"
    values   = ["development", "testing"]
  }

  match {
    property = "site"
    op       = "EQUALS"
    values   = ["us-east-1"]
  }

  match {
    property = "service"
    op       = "EQUALS"
    values   = ["api"]
  }

  entity_property_to_log_label_mapping = {
    "otel_namespace" = "service_namespace"
    "otel_service"   = "service_name"
    "environment"    = "env"
    "site"           = "region"
    "service"        = "app"
  }

  filter_by_span_id  = true
  filter_by_trace_id = true
}
```

## Minimal log configuration

Create a minimal configuration for all entities:

```terraform
# Minimal configuration for all entities
resource "grafana_asserts_log_config" "minimal" {
  provider = grafana.asserts

  name            = "minimal"
  priority        = 3000
  default_config  = false
  data_source_uid = "loki-minimal"

  match {
    property = "asserts_entity_type"
    op       = "IS_NOT_NULL"
    values   = []
  }
}
```

## Advanced log configuration with complex match rules

Configure logs with multiple operations and advanced match rules:

```terraform
# Advanced configuration with multiple operations
resource "grafana_asserts_log_config" "advanced" {
  provider = grafana.asserts

  name            = "advanced"
  priority        = 1500
  default_config  = false
  data_source_uid = "loki-advanced"
  error_label     = "level"

  match {
    property = "service_type"
    op       = "CONTAINS"
    values   = ["web", "api"]
  }

  match {
    property = "environment"
    op       = "NOT_EQUALS"
    values   = ["test"]
  }

  match {
    property = "team"
    op       = "IS_NOT_NULL"
    values   = []
  }

  entity_property_to_log_label_mapping = {
    "service_type"   = "type"
    "team"           = "owner"
    "environment"    = "env"
    "version"        = "app_version"
  }

  filter_by_span_id  = true
  filter_by_trace_id = false
}
```

## Resource reference

### `grafana_asserts_log_config`

Manage Knowledge Graph log configurations through the Grafana API.

#### Arguments

| Name                                   | Type           | Required | Description                                                                                  |
| -------------------------------------- | -------------- | -------- | -------------------------------------------------------------------------------------------- |
| `name`                                 | `string`       | Yes      | The name of the log configuration. This field is immutable and forces recreation if changed. |
| `priority`                             | `number`       | Yes      | Priority of the log configuration. Higher priority configurations are evaluated first.       |
| `default_config`                       | `bool`         | Yes      | Whether this is the default configuration. Default configurations cannot be deleted.         |
| `data_source_uid`                      | `string`       | Yes      | DataSource UID to be queried (for example, a Loki instance).                                 |
| `match`                                | `list(object)` | No       | List of match rules for entity properties. Refer to [match block](#match-block) for details. |
| `error_label`                          | `string`       | No       | Label name used to identify error logs.                                                      |
| `entity_property_to_log_label_mapping` | `map(string)`  | No       | Mapping of entity properties to log labels for correlation.                                  |
| `filter_by_span_id`                    | `bool`         | No       | Whether to filter logs by span ID for distributed tracing correlation.                       |
| `filter_by_trace_id`                   | `bool`         | No       | Whether to filter logs by trace ID for distributed tracing correlation.                      |

#### Match block

Each `match` block supports the following:

| Name       | Type           | Required | Description                                                                                                              |
| ---------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------------ |
| `property` | `string`       | Yes      | Entity property to match against.                                                                                        |
| `op`       | `string`       | Yes      | Operation to use for matching. One of: `EQUALS`, `NOT_EQUALS`, `CONTAINS`, `DOES_NOT_CONTAIN`, `IS_NULL`, `IS_NOT_NULL`. |
| `values`   | `list(string)` | Yes      | Values to match against. Can be empty for `IS_NULL` and `IS_NOT_NULL` operations.                                        |

#### Example

```terraform
resource "grafana_asserts_log_config" "example" {
  provider = grafana.asserts

  name            = "example-logs"
  priority        = 1000
  default_config  = false
  data_source_uid = "loki-prod"
  error_label     = "level"

  match {
    property = "asserts_entity_type"
    op       = "EQUALS"
    values   = ["Service", "Pod"]
  }

  entity_property_to_log_label_mapping = {
    "service"     = "app"
    "namespace"   = "k8s_namespace"
    "environment" = "env"
  }

  filter_by_span_id  = true
  filter_by_trace_id = true
}
```

## Best practices

Consider the following best practices when configuring log correlation with Terraform.

### Priority management

- Assign lower priority numbers to more specific configurations
- Higher priority configurations are evaluated first
- Use consistent priority ranges for different configuration types
- Document the reasoning behind priority assignments

### Data source configuration

- Ensure the data source UID matches your actual Loki or log aggregation system
- Test data source connectivity before applying configurations
- Use descriptive names for log configurations to indicate their purpose
- Consider using separate data sources for different environments

### Label map strategy

- Map entity properties consistently across all log configurations
- Use meaningful log label names that match your logging standards
- Document the mapping relationships in configuration comments
- Verify that mapped labels exist in your log data

### Match rules design

- Start with broad match rules and refine based on needs
- Use specific property names that exist in your entity model
- Test match rules with sample data before deploying
- Combine multiple match rules for precise entity targeting

### Distributed trace integration

- Enable `filter_by_span_id` and `filter_by_trace_id` when using OpenTelemetry
- Ensure your logs contain the appropriate trace and span ID labels
- Use consistent label names for trace IDs across your logging infrastructure
- Test trace correlation to verify it works as expected

## Validation

After applying the Terraform configuration, verify that:

- Log configurations are created in your Knowledge Graph instance
- Configurations appear in the Knowledge Graph UI under **Observability > Configuration > Logs**
- Log correlation works when drilling down from entities
- Label mappings correctly translate entity properties to log labels
- Match rules properly filter entities
- Trace and span ID filtering works for distributed tracing

## Related documentation

- [Configure logs correlation in Knowledge Graph](/docs/grafana-cloud/knowledge-graph/configure/logs-correlation/)
- [Get started with Terraform for Knowledge Graph](../getting-started/)
- [Loki documentation](/docs/loki/latest/)

# Configure notification alerts using Terraform

Notification alerts configurations in [Knowledge Graph](/docs/grafana-cloud/knowledge-graph/) allow you to manage how alerts are processed and routed. You can specify match labels to filter alerts, add custom labels, set duration requirements, and control silencing.

For information about configuring notification alerts in the Knowledge Graph UI, refer to [Configure notifications](/docs/grafana-cloud/knowledge-graph/configure/notifications/).

## Basic notification alerts configuration

Create a file named `alert-configs.tf` and add the following:

```terraform
# Basic alert configuration with silencing
resource "grafana_asserts_notification_alerts_config" "prometheus_remote_storage_failures" {
  provider = grafana.asserts

  name = "PrometheusRemoteStorageFailures"

  match_labels = {
    alertname   = "PrometheusRemoteStorageFailures"
    alertgroup  = "prometheus.alerts"
    asserts_env = "prod"
  }

  silenced = true
}

# High severity alert with specific job and context matching
resource "grafana_asserts_notification_alerts_config" "error_buildup_notify" {
  provider = grafana.asserts

  name = "ErrorBuildupNotify"

  match_labels = {
    alertname               = "ErrorBuildup"
    job                     = "acai"
    asserts_request_type    = "inbound"
    asserts_request_context = "/auth"
  }

  silenced = false
}
```

## Notification alerts with additional labels and duration

Configure alerts with custom labels and timing requirements:

```terraform
# Alert with additional labels and custom duration
resource "grafana_asserts_notification_alerts_config" "payment_test_alert" {
  provider = grafana.asserts

  name = "PaymentTestAlert"

  match_labels = {
    alertname         = "PaymentTestAlert"
    additional_labels = "asserts_severity=~\"critical\""
    alertgroup        = "alex-k8s-integration-test.alerts"
  }

  alert_labels = {
    testing = "onetwothree"
  }

  duration = "5m"
  silenced = false
}
```

## Latency and performance notification alerts

Monitor and alert on latency and performance issues:

```terraform
# Latency alert for shipping service
resource "grafana_asserts_notification_alerts_config" "high_shipping_latency" {
  provider = grafana.asserts

  name = "high shipping latency"

  match_labels = {
    alertname            = "LatencyP99ErrorBuildup"
    job                  = "shipping"
    asserts_request_type = "inbound"
  }

  silenced = false
}

# CPU throttling alert with warning severity
resource "grafana_asserts_notification_alerts_config" "cpu_throttling_sustained" {
  provider = grafana.asserts

  name = "CPUThrottlingSustained"

  match_labels = {
    alertname         = "CPUThrottlingSustained"
    additional_labels = "asserts_severity=~\"warning\""
  }

  silenced = true
}
```

## Infrastructure and service notification alerts

Configure alerts for infrastructure components and services:

```terraform
# Ingress error rate alert
resource "grafana_asserts_notification_alerts_config" "ingress_error" {
  provider = grafana.asserts

  name = "ingress error"

  match_labels = {
    alertname            = "ErrorRatioBreach"
    job                  = "ingress-nginx-controller-metrics"
    asserts_request_type = "inbound"
  }

  silenced = false
}

# MySQL Galera cluster alert
resource "grafana_asserts_notification_alerts_config" "mysql_galera_not_ready" {
  provider = grafana.asserts

  name = "MySQLGaleraNotReady"

  match_labels = {
    alertname = "MySQLGaleraNotReady"
  }

  silenced = false
}
```

## Resource reference

### `grafana_asserts_notification_alerts_config`

Manage Knowledge Graph notification alerts configurations through the Grafana API.

#### Arguments

| Name           | Type          | Required | Description                                                                                                                   |
| -------------- | ------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `name`         | `string`      | Yes      | The name of the notification alerts configuration. This field is immutable and forces recreation if changed.                  |
| `match_labels` | `map(string)` | No       | Labels to match for this notification alerts configuration. Used to filter which alerts this configuration applies to.        |
| `alert_labels` | `map(string)` | No       | Labels to add to alerts generated by this notification alerts configuration.                                                  |
| `duration`     | `string`      | No       | Duration for which the condition must be true before firing (for example, '5m', '30s'). Maps to 'for' in Knowledge Graph API. |
| `silenced`     | `bool`        | No       | Whether this notification alerts configuration is silenced. Defaults to `false`.                                              |

#### Example

```terraform
resource "grafana_asserts_notification_alerts_config" "example" {
  provider = grafana.asserts

  name = "ExampleAlert"

  match_labels = {
    alertname = "HighCPUUsage"
    job       = "monitoring"
  }

  alert_labels = {
    severity = "warning"
    team     = "platform"
  }

  duration = "5m"
  silenced = false
}
```

## Best practices

Consider the following best practices when managing notification alerts with Terraform.

### Label management

- Use specific and meaningful labels in `match_labels` to ensure precise alert filtering
- Leverage existing label conventions from your monitoring setup
- Consider using `asserts_env` and `asserts_site` labels for multi-environment setups

### Silence strategy

- Use the `silenced` parameter for temporary suppression rather than deleting notification alerts configurations
- Document the reason for silencing in your Terraform configuration comments
- Regularly review silenced configurations to ensure they're still needed

### Duration configuration

- Set appropriate duration values based on your alerting requirements
- Consider the nature of the monitored condition when choosing duration
- Use consistent duration formats across similar alert types

## Validation

After applying the Terraform configuration, verify that:

- Notification alerts configurations are created in your Knowledge Graph instance
- Configurations appear in the Knowledge Graph UI under **Observability > Rules > Notify**
- Match labels correctly filter the intended alerts
- Custom labels are properly applied to generated alerts

## Related documentation

- [Configure notifications in Knowledge Graph](/docs/grafana-cloud/knowledge-graph/configure/notifications/)
- [Get started with Terraform for Knowledge Graph](../getting-started/)
- [Configure alerts in Knowledge Graph](/docs/grafana-cloud/knowledge-graph/configure/alerts/)

# Configure profile correlation using Terraform

Profile configurations in [Knowledge Graph](/docs/grafana-cloud/knowledge-graph/) allow you to define how continuous profiling data is queried and correlated with entities. You can specify data sources, entity matching rules, and label mappings for performance profiling.

For information about configuring profile correlation in the Knowledge Graph UI, refer to [Configure profiles correlation](/docs/grafana-cloud/knowledge-graph/configure/profiles-correlation/).

## Basic profile configuration

Create a file named `profile-configs.tf` and add the following:

```terraform
# Basic profile configuration for services
resource "grafana_asserts_profile_config" "production" {
  provider = grafana.asserts

  name            = "production"
  priority        = 1000
  default_config  = false
  data_source_uid = "grafanacloud-profiles"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service"]
  }

  match {
    property = "deployment_environment"
    op       = "="
    values   = ["production", "staging"]
  }

  entity_property_to_profile_label_mapping = {
    "cluster"        = "k8s_cluster_name"
    "namespace"      = "k8s_namespace_name"
    "container"      = "k8s_container_name"
    "otel_service"   = "service_name"
    "otel_namespace" = "service_namespace"
  }
}
```

## Profile configuration with multiple match rules

Configure profile correlation with multiple entity matching criteria:

```terraform
# Development environment profile configuration
resource "grafana_asserts_profile_config" "development" {
  provider = grafana.asserts

  name            = "development"
  priority        = 2000
  default_config  = false
  data_source_uid = "pyroscope-dev"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service"]
  }

  match {
    property = "deployment_environment"
    op       = "="
    values   = ["development", "testing"]
  }

  match {
    property = "asserts_site"
    op       = "="
    values   = ["us-east-1"]
  }

  match {
    property = "service"
    op       = "="
    values   = ["api"]
  }

  entity_property_to_profile_label_mapping = {
    "cluster"        = "k8s_cluster_name"
    "namespace"      = "k8s_namespace_name"
    "container"      = "k8s_container_name"
    "otel_service"   = "service_name"
    "otel_namespace" = "service_namespace"
    "pod"            = "k8s_pod_name"
  }
}
```

## Minimal profile configuration

Create a minimal configuration for all entities:

```terraform
# Minimal configuration for all entities
resource "grafana_asserts_profile_config" "minimal" {
  provider = grafana.asserts

  name            = "minimal"
  priority        = 3000
  default_config  = false
  data_source_uid = "pyroscope-minimal"

  match {
    property = "asserts_entity_type"
    op       = "IS NOT NULL"
    values   = []
  }

  entity_property_to_profile_label_mapping = {
    "cluster"        = "k8s_cluster_name"
    "otel_service"   = "service_name"
    "otel_namespace" = "service_namespace"
  }
}
```

## Advanced profile configuration with complex match rules

Configure profiles with multiple operations and advanced match rules:

```terraform
# Advanced configuration with multiple operations
resource "grafana_asserts_profile_config" "advanced" {
  provider = grafana.asserts

  name            = "advanced"
  priority        = 1500
  default_config  = false
  data_source_uid = "pyroscope-advanced"

  match {
    property = "service_type"
    op       = "CONTAINS"
    values   = ["web", "api"]
  }

  match {
    property = "deployment_environment"
    op       = "<>"
    values   = ["test"]
  }

  match {
    property = "team"
    op       = "IS NOT NULL"
    values   = []
  }

  match {
    property = "cpu_threshold"
    op       = ">="
    values   = ["80"]
  }

  entity_property_to_profile_label_mapping = {
    "service_type"   = "service_type"
    "team"           = "team_owner"
    "environment"    = "deployment_env"
    "version"        = "app_version"
    "region"         = "cloud_region"
    "node"           = "k8s_node_name"
  }
}
```

## Resource reference

### `grafana_asserts_profile_config`

Manage Knowledge Graph profile configurations through the Grafana API.

#### Arguments

| Name                                       | Type           | Required | Description                                                                                      |
| ------------------------------------------ | -------------- | -------- | ------------------------------------------------------------------------------------------------ |
| `name`                                     | `string`       | Yes      | The name of the profile configuration. This field is immutable and forces recreation if changed. |
| `priority`                                 | `number`       | Yes      | Priority of the profile configuration. A lower number means a higher priority.                   |
| `default_config`                           | `bool`         | Yes      | Whether this is the default configuration. Default configurations cannot be deleted.             |
| `data_source_uid`                          | `string`       | Yes      | DataSource UID to be queried (for example, a Pyroscope instance).                                |
| `match`                                    | `list(object)` | No       | List of match rules for entity properties. Refer to [match block](#match-block) for details.     |
| `entity_property_to_profile_label_mapping` | `map(string)`  | No       | Mapping of entity properties to profile labels for correlation.                                  |

#### Match block

Each `match` block supports the following:

| Name       | Type           | Required | Description                                                                                                                  |
| ---------- | -------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `property` | `string`       | Yes      | Entity property to match against.                                                                                            |
| `op`       | `string`       | Yes      | Operation to use for matching. One of: `=`, `<>`, `<`, `>`, `<=`, `>=`, `IS NULL`, `IS NOT NULL`, `STARTS WITH`, `CONTAINS`. |
| `values`   | `list(string)` | Yes      | Values to match against. Can be empty for `IS NULL` and `IS NOT NULL` operations.                                            |

#### Example

```terraform
resource "grafana_asserts_profile_config" "example" {
  provider = grafana.asserts

  name            = "example-profiles"
  priority        = 1000
  default_config  = false
  data_source_uid = "pyroscope-prod"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service", "Pod"]
  }

  match {
    property = "deployment_environment"
    op       = "STARTS WITH"
    values   = ["prod"]
  }

  entity_property_to_profile_label_mapping = {
    "service"     = "service_name"
    "namespace"   = "k8s_namespace_name"
    "environment" = "deployment_env"
    "cluster"     = "k8s_cluster_name"
    "container"   = "k8s_container_name"
  }
}
```

## Best practices

Consider the following best practices when configuring profile correlation with Terraform.

### Priority management

- Assign lower priority numbers to more specific configurations
- A lower priority number means higher priority (configurations are evaluated in ascending priority order)
- Use consistent priority ranges for different configuration types
- Document the reasoning behind priority assignments

### Data source configuration

- Ensure the data source UID matches your actual Pyroscope or profiling system
- Test data source connectivity before applying configurations
- Use descriptive names for profile configurations to indicate their purpose
- Consider using separate data sources for different environments

### Label map strategy

- Map entity properties consistently across all profile configurations
- Use meaningful profile label names that match your profiling standards
- Document the mapping relationships in configuration comments
- Verify that mapped labels exist in your profile data
- Align label names with your continuous profiling setup (Pyroscope, Grafana Agent, etc.)

### Match rules design

- Start with broad match rules and refine based on needs
- Use specific property names that exist in your entity model
- Test match rules with sample data before deploying
- Combine multiple match rules for precise entity targeting
- Leverage comparison operators (`<`, `>`, `<=`, `>=`) for numeric thresholds or performance-based filtering

### Continuous profiling integration

- Ensure your profiling agents are configured with appropriate labels
- Use consistent label names across your profiling infrastructure
- Map both service-level and infrastructure-level properties for comprehensive correlation
- Consider including pod, node, and container information for Kubernetes deployments
- Test profile correlation to verify it works as expected

## Validation

After applying the Terraform configuration, verify that:

- Profile configurations are created in your Knowledge Graph instance
- Configurations appear in the Knowledge Graph UI under **Observability > Configuration > Profiles**
- Profile correlation works when drilling down from entities
- Label mappings correctly translate entity properties to profile labels
- Match rules properly filter entities
- Profile queries return expected results in the Knowledge Graph UI
- Flame graphs and profiling data display correctly when accessed through Knowledge Graph

## Related documentation

- [Configure profiles correlation in Knowledge Graph](/docs/grafana-cloud/knowledge-graph/configure/profiles-correlation/)
- [Get started with Terraform for Knowledge Graph](../getting-started/)
- [Pyroscope documentation](/docs/pyroscope/latest/)
- [Grafana Agent profiling](/docs/agent/latest/flow/reference/components/pyroscope.scrape/)

# Manage Prometheus rules using Terraform

Prometheus rules in the [Knowledge Graph](/docs/grafana-cloud/knowledge-graph/) allow you to define custom recording and alerting rules that are evaluated against your metrics data. Recording rules pre-compute frequently used or computationally expensive expressions and save the results as new time series. Alerting rules define conditions that, when met, trigger alerts.

Using the `grafana_asserts_prom_rule_file` resource, you can manage these rules as code, enabling version control, review processes, and consistent deployments across environments.

## Before you begin

To manage Prometheus rules using Terraform, you need:

- A Grafana Cloud stack with Knowledge Graph enabled
- [Terraform configured for Knowledge Graph](../getting-started/)
- An understanding of PromQL and Prometheus rule concepts

## Create a basic recording rule

Recording rules allow you to pre-compute PromQL expressions and store the results as new metrics. This is useful for expensive queries that you run frequently.

Create a file named `prom-rules.tf` and add the following:

```terraform
# Basic recording rule for request rate
resource "grafana_asserts_prom_rule_file" "request_rates" {
  provider = grafana.asserts

  name   = "request-rates"
  active = true

  group {
    name     = "request_rate_rules"
    interval = "30s"

    rule {
      record = "job:http_requests_total:rate5m"
      expr   = "sum(rate(http_requests_total[5m])) by (job)"

      labels = {
        aggregation = "job"
        source      = "custom"
      }
    }
  }
}
```

## Create alerting rules

Alerting rules define conditions that trigger alerts when met. Use the `alert` field instead of `record` to define an alerting rule.

```terraform
# Alerting rules for service health
resource "grafana_asserts_prom_rule_file" "service_alerts" {
  provider = grafana.asserts

  name   = "service-health-alerts"
  active = true

  group {
    name     = "service_health"
    interval = "1m"

    rule {
      alert    = "HighErrorRate"
      expr     = "sum(rate(http_requests_total{code=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) > 0.05"
      duration = "5m"

      labels = {
        severity = "critical"
        team     = "platform"
      }

      annotations = {
        summary     = "High error rate detected"
        description = "Error rate is above 5% for the last 5 minutes"
        runbook_url = "https://docs.example.com/runbooks/high-error-rate"
      }
    }

    rule {
      alert    = "ServiceDown"
      expr     = "up == 0"
      duration = "2m"

      labels = {
        severity = "critical"
      }

      annotations = {
        summary     = "Service is down"
        description = "{{ $labels.job }} has been down for more than 2 minutes"
      }
    }
  }
}
```

## Create multiple rule groups

Organize related rules into groups with their own evaluation intervals:

```terraform
# Multiple rule groups for comprehensive monitoring
resource "grafana_asserts_prom_rule_file" "comprehensive_rules" {
  provider = grafana.asserts

  name   = "comprehensive-monitoring"
  active = true

  # Latency recording rules
  group {
    name     = "latency_recording"
    interval = "30s"

    rule {
      record = "job:http_request_duration_seconds:p99"
      expr   = "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))"

      labels = {
        quantile = "0.99"
      }
    }

    rule {
      record = "job:http_request_duration_seconds:p95"
      expr   = "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))"

      labels = {
        quantile = "0.95"
      }
    }

    rule {
      record = "job:http_request_duration_seconds:p50"
      expr   = "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))"

      labels = {
        quantile = "0.50"
      }
    }
  }

  # Latency alerting rules
  group {
    name     = "latency_alerts"
    interval = "1m"

    rule {
      alert    = "HighP99Latency"
      expr     = "job:http_request_duration_seconds:p99 > 1"
      duration = "5m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "High P99 latency detected"
        description = "P99 latency for {{ $labels.job }} is above 1 second"
      }
    }

    rule {
      alert    = "CriticalLatency"
      expr     = "job:http_request_duration_seconds:p99 > 5"
      duration = "2m"

      labels = {
        severity = "critical"
      }

      annotations = {
        summary     = "Critical latency detected"
        description = "P99 latency for {{ $labels.job }} is above 5 seconds"
      }
    }
  }

  # Throughput rules
  group {
    name     = "throughput_rules"
    interval = "1m"

    rule {
      record = "job:http_requests:rate1m"
      expr   = "sum(rate(http_requests_total[1m])) by (job)"
    }

    rule {
      alert    = "LowThroughput"
      expr     = "job:http_requests:rate1m < 10"
      duration = "10m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "Low throughput detected"
        description = "Request rate for {{ $labels.job }} is below 10 requests per second"
      }
    }
  }
}
```

## Create resource utilization rules

Define rules to monitor resource utilization across your infrastructure:

```terraform
# Resource utilization monitoring rules
resource "grafana_asserts_prom_rule_file" "resource_utilization" {
  provider = grafana.asserts

  name   = "resource-utilization"
  active = true

  group {
    name     = "cpu_rules"
    interval = "30s"

    rule {
      record = "instance:cpu_utilization:avg5m"
      expr   = "1 - avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) by (instance)"
    }

    rule {
      alert    = "HighCPUUtilization"
      expr     = "instance:cpu_utilization:avg5m > 0.85"
      duration = "10m"

      labels = {
        severity = "warning"
        resource = "cpu"
      }

      annotations = {
        summary     = "High CPU utilization"
        description = "CPU utilization on {{ $labels.instance }} is above 85%"
      }
    }

    rule {
      alert    = "CriticalCPUUtilization"
      expr     = "instance:cpu_utilization:avg5m > 0.95"
      duration = "5m"

      labels = {
        severity = "critical"
        resource = "cpu"
      }

      annotations = {
        summary     = "Critical CPU utilization"
        description = "CPU utilization on {{ $labels.instance }} is above 95%"
      }
    }
  }

  group {
    name     = "memory_rules"
    interval = "30s"

    rule {
      record = "instance:memory_utilization:ratio"
      expr   = "1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)"
    }

    rule {
      alert    = "HighMemoryUtilization"
      expr     = "instance:memory_utilization:ratio > 0.85"
      duration = "10m"

      labels = {
        severity = "warning"
        resource = "memory"
      }

      annotations = {
        summary     = "High memory utilization"
        description = "Memory utilization on {{ $labels.instance }} is above 85%"
      }
    }
  }

  group {
    name     = "disk_rules"
    interval = "1m"

    rule {
      record = "instance:disk_utilization:ratio"
      expr   = "1 - (node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"})"
    }

    rule {
      alert    = "DiskSpaceLow"
      expr     = "instance:disk_utilization:ratio > 0.80"
      duration = "15m"

      labels = {
        severity = "warning"
        resource = "disk"
      }

      annotations = {
        summary     = "Disk space running low"
        description = "Disk utilization on {{ $labels.instance }} is above 80%"
      }
    }
  }
}
```

## Create Kubernetes-specific rules

Define rules for monitoring Kubernetes workloads:

```terraform
# Kubernetes workload monitoring rules
resource "grafana_asserts_prom_rule_file" "kubernetes_rules" {
  provider = grafana.asserts

  name   = "kubernetes-workloads"
  active = true

  group {
    name     = "kubernetes_pod_rules"
    interval = "30s"

    rule {
      record = "namespace:pod_restarts:rate1h"
      expr   = "sum(increase(kube_pod_container_status_restarts_total[1h])) by (namespace)"
    }

    rule {
      alert    = "PodCrashLooping"
      expr     = "rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 3"
      duration = "5m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "Pod is crash looping"
        description = "Pod {{ $labels.namespace }}/{{ $labels.pod }} is restarting frequently"
      }
    }

    rule {
      alert    = "PodNotReady"
      expr     = "kube_pod_status_ready{condition=\"true\"} == 0"
      duration = "10m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "Pod not ready"
        description = "Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for more than 10 minutes"
      }
    }
  }

  group {
    name     = "kubernetes_deployment_rules"
    interval = "1m"

    rule {
      record = "deployment:replicas_unavailable:count"
      expr   = "kube_deployment_status_replicas_unavailable"
    }

    rule {
      alert    = "DeploymentReplicasMismatch"
      expr     = "kube_deployment_spec_replicas != kube_deployment_status_replicas_available"
      duration = "10m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "Deployment replicas mismatch"
        description = "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for more than 10 minutes"
      }
    }
  }

  group {
    name     = "kubernetes_node_rules"
    interval = "1m"

    rule {
      alert    = "NodeNotReady"
      expr     = "kube_node_status_condition{condition=\"Ready\",status=\"true\"} == 0"
      duration = "5m"

      labels = {
        severity = "critical"
      }

      annotations = {
        summary     = "Kubernetes node not ready"
        description = "Node {{ $labels.node }} has been unready for more than 5 minutes"
      }
    }

    rule {
      alert    = "NodeMemoryPressure"
      expr     = "kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"} == 1"
      duration = "5m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "Node under memory pressure"
        description = "Node {{ $labels.node }} is under memory pressure"
      }
    }
  }
}
```

## Use conditional rule disabling

Disable specific rules in certain groups using the `disable_in_groups` field:

```terraform
# Rules with conditional disabling
resource "grafana_asserts_prom_rule_file" "conditional_rules" {
  provider = grafana.asserts

  name   = "conditional-alerting"
  active = true

  group {
    name     = "production_alerts"
    interval = "1m"

    rule {
      alert    = "HighErrorRate"
      expr     = "sum(rate(http_requests_total{code=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) > 0.01"
      duration = "5m"

      labels = {
        severity    = "critical"
        environment = "production"
      }

      annotations = {
        summary     = "High error rate in production"
        description = "Error rate is above 1% for the last 5 minutes"
      }

      # Disable this rule in staging group
      disable_in_groups = ["staging_alerts"]
    }
  }

  group {
    name     = "staging_alerts"
    interval = "1m"

    rule {
      alert    = "HighErrorRate"
      expr     = "sum(rate(http_requests_total{code=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) > 0.10"
      duration = "10m"

      labels = {
        severity    = "warning"
        environment = "staging"
      }

      annotations = {
        summary     = "High error rate in staging"
        description = "Error rate is above 10% for the last 10 minutes"
      }
    }
  }
}
```

## Manage inactive rule files

Temporarily disable an entire rules file without deleting it:

```terraform
# Inactive rules file (not evaluated)
resource "grafana_asserts_prom_rule_file" "experimental_rules" {
  provider = grafana.asserts

  name   = "experimental-rules"
  active = false  # Rules are not evaluated

  group {
    name     = "experimental_alerts"
    interval = "1m"

    rule {
      alert    = "ExperimentalAlert"
      expr     = "some_experimental_metric > 100"
      duration = "5m"

      labels = {
        severity = "info"
      }

      annotations = {
        summary = "Experimental alert triggered"
      }
    }
  }
}
```

## Resource reference

### `grafana_asserts_prom_rule_file`

Manage Prometheus recording and alerting rules through the Knowledge Graph API. This resource allows you to create and manage custom Prometheus rules that are evaluated against your metrics data.

#### Arguments

| Name     | Type           | Required | Description                                                                                      |
| -------- | -------------- | -------- | ------------------------------------------------------------------------------------------------ |
| `name`   | `string`       | Yes      | The name of the Prometheus rules file. This field is immutable and forces recreation if changed. |
| `active` | `bool`         | No       | Whether the rules file is active. Inactive rules are not evaluated. Defaults to `true`.          |
| `group`  | `list(object)` | Yes      | List of Prometheus rule groups. Refer to [group block](#group-block) for details.                |

#### Group block

Each `group` block contains a set of related rules with a shared evaluation interval:

| Name       | Type           | Required | Description                                                                                                           |
| ---------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------- |
| `name`     | `string`       | Yes      | The name of the rule group (for example, `latency_monitoring`).                                                       |
| `interval` | `string`       | No       | Evaluation interval for this group (for example, `30s`, `1m`). If not specified, uses the global evaluation interval. |
| `rule`     | `list(object)` | Yes      | List of Prometheus rules in this group. Refer to [rule block](#rule-block) for details.                               |

#### Rule block

Each `rule` block defines a recording or alerting rule. Either `record` or `alert` must be specified, but not both:

| Name                | Type          | Required    | Description                                                                                                                            |
| ------------------- | ------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `record`            | `string`      | Conditional | The name of the time series to output for recording rules. Required if `alert` is not specified.                                       |
| `alert`             | `string`      | Conditional | The name of the alert for alerting rules. Required if `record` is not specified.                                                       |
| `expr`              | `string`      | Yes         | The PromQL expression to evaluate.                                                                                                     |
| `duration`          | `string`      | No          | How long the condition must be true before firing the alert (for example, `5m`). Only for alerting rules. Maps to `for` in Prometheus. |
| `labels`            | `map(string)` | No          | Labels to attach to the resulting time series or alert.                                                                                |
| `annotations`       | `map(string)` | No          | Annotations to add to alerts (for example, `summary`, `description`). Only applicable for alerting rules.                              |
| `disable_in_groups` | `set(string)` | No          | List of group names where this rule should be disabled. Useful for conditional rule enablement.                                        |

#### Example

```terraform
resource "grafana_asserts_prom_rule_file" "example" {
  provider = grafana.asserts

  name   = "example-rules"
  active = true

  group {
    name     = "example_group"
    interval = "1m"

    # Recording rule
    rule {
      record = "job:http_requests:rate5m"
      expr   = "sum(rate(http_requests_total[5m])) by (job)"
    }

    # Alerting rule
    rule {
      alert    = "HighErrorRate"
      expr     = "job:http_errors:rate5m > 0.05"
      duration = "5m"

      labels = {
        severity = "critical"
      }

      annotations = {
        summary     = "High error rate detected"
        description = "Error rate for {{ $labels.job }} is above 5%"
      }
    }
  }
}
```

## Best practices

Consider the following best practices when managing Prometheus rules with Terraform.

### Rule organization

- Group related rules together in the same rule group
- Use descriptive names for rules that indicate their purpose
- Separate recording rules from alerting rules into different groups when they have different evaluation needs
- Use consistent naming conventions for recorded metrics, such as `level:metric:operation`

### Recording rules

- Use recording rules for expensive queries that are run frequently
- Pre-compute aggregations that are used by multiple dashboards or alerts
- Choose appropriate evaluation intervals based on how frequently the data changes
- Follow Prometheus naming conventions, such as `job:metric:aggregation`

### Alerting rules

- Set appropriate `duration` values to avoid flapping alerts
- Include meaningful labels like `severity` and `team` for routing
- Provide comprehensive annotations including `summary`, `description`, and `runbook_url`
- Use template variables in annotations such as `{{ $labels.job }}` to provide context

### Performance considerations

- Avoid overly complex PromQL expressions that may timeout
- Use appropriate evaluation intervals; don't evaluate faster than needed
- Test your PromQL queries in Grafana before adding them to Terraform
- Consider the cardinality impact of recording rules that create new time series

### Testing and validation

- Test rules in a non-production environment before deploying to production
- Use the `active = false` flag to stage rules without evaluating them
- Monitor the performance impact of new rules after deployment
- Review and prune unused rules periodically

## Validation

After applying the Terraform configuration, verify that:

- Rule files are created and visible in the Knowledge Graph rules configuration
- Recording rules are generating the expected time series
- Alerting rules are evaluating correctly
- Alert thresholds trigger notifications as expected
- Rule evaluation times are within acceptable ranges
- No errors appear in the rule evaluation logs

## Troubleshooting

### Rules not being evaluated

If your rules are not being evaluated:

1. Verify that `active = true` is set on the rule file
2. Check that the rule filename follows naming validation rules (alphanumeric, hyphens, underscores)
3. Ensure the PromQL expression is valid by testing it in Grafana

### Recording rule not producing data

If a recording rule is not producing data:

1. Verify the source metrics exist and have data
2. Check the PromQL expression returns results
3. Ensure the evaluation interval is appropriate
4. Review the rule evaluation logs for errors

### Alerts not firing

If alerting rules are not firing:

1. Verify the condition is actually met by querying the expression manually
2. Check that the `duration` period has elapsed
3. Ensure the alerting system is properly configured to receive alerts
4. Review annotation templates for syntax errors

### Import error when recreating rules

If you receive errors when recreating rules:

- The `name` field is immutable and forces recreation if changed
- Ensure no conflicting rule files exist with the same name
- Use `terraform import` to import existing resources if needed

## Related documentation

- [Get started with Terraform for Knowledge Graph](../getting-started/)
- [Create custom model rules using Terraform](../custom-model-rules/)
- [Prometheus recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/)
- [Prometheus alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)

# Configure suppressed assertions using Terraform

Suppressed assertions configurations allow you to disable specific alerts or assertions based on label matching in [Knowledge Graph](/docs/grafana-cloud/knowledge-graph/). This is useful for maintenance windows, test environments, or when you want to temporarily suppress certain types of alerts.

For information about suppressing insights in the Knowledge Graph UI, refer to [Suppress insights](/docs/grafana-cloud/knowledge-graph/troubleshoot-infra-apps/suppress-insights/).

## Basic suppressed assertions configuration

Create a file named `suppressed-assertions.tf` and add the following:

```terraform
# Basic suppressed alert configuration for maintenance
resource "grafana_asserts_suppressed_assertions_config" "maintenance_window" {
  provider = grafana.asserts

  name = "MaintenanceWindow"

  match_labels = {
    service     = "api-service"
    maintenance = "true"
  }
}

# Suppress specific alertname during deployment
resource "grafana_asserts_suppressed_assertions_config" "deployment_suppression" {
  provider = grafana.asserts

  name = "DeploymentSuppression"

  match_labels = {
    alertname = "HighLatency"
    job       = "web-service"
    env       = "staging"
  }
}

# Suppress alerts for specific test environment
resource "grafana_asserts_suppressed_assertions_config" "test_environment_suppression" {
  provider = grafana.asserts

  name = "TestEnvironmentSuppression"

  match_labels = {
    alertgroup  = "test.alerts"
    environment = "test"
  }
}
```

## Service-specific suppression configurations

Suppress alerts for specific services during maintenance or operational activities:

```terraform
# Suppress alerts for specific services during maintenance
resource "grafana_asserts_suppressed_assertions_config" "api_service_maintenance" {
  provider = grafana.asserts

  name = "APIServiceMaintenance"

  match_labels = {
    service     = "api-gateway"
    job         = "api-gateway"
    maintenance = "scheduled"
  }
}

# Suppress database alerts during backup operations
resource "grafana_asserts_suppressed_assertions_config" "database_backup" {
  provider = grafana.asserts

  name = "DatabaseBackupSuppression"

  match_labels = {
    service     = "postgresql"
    job         = "postgres-exporter"
    backup_mode = "active"
  }
}

# Suppress monitoring system alerts during updates
resource "grafana_asserts_suppressed_assertions_config" "monitoring_update" {
  provider = grafana.asserts

  name = "MonitoringSystemUpdate"

  match_labels = {
    service = "prometheus"
    job     = "prometheus"
    update  = "in_progress"
  }
}
```

## Environment and team-based suppression

Create suppression rules based on environment or team:

```terraform
# Suppress all alerts for development environment
resource "grafana_asserts_suppressed_assertions_config" "dev_environment" {
  provider = grafana.asserts

  name = "DevelopmentEnvironmentSuppression"

  match_labels = {
    environment = "development"
    team        = "platform"
  }
}

# Suppress alerts for specific team during their maintenance window
resource "grafana_asserts_suppressed_assertions_config" "team_maintenance" {
  provider = grafana.asserts

  name = "TeamMaintenanceWindow"

  match_labels = {
    team        = "backend"
    maintenance = "team_scheduled"
    timezone    = "UTC"
  }
}

# Suppress alerts for staging environment during testing
resource "grafana_asserts_suppressed_assertions_config" "staging_testing" {
  provider = grafana.asserts

  name = "StagingTestingSuppression"

  match_labels = {
    environment = "staging"
    testing     = "automated"
    job         = "integration-tests"
  }
}
```

## Alert type and severity-based suppression

Suppress alerts based on their type or severity:

```terraform
# Suppress low severity alerts during business hours
resource "grafana_asserts_suppressed_assertions_config" "low_severity_business_hours" {
  provider = grafana.asserts

  name = "LowSeverityBusinessHours"

  match_labels = {
    severity = "warning"
    timezone = "business_hours"
  }
}

# Suppress specific alert types during known issues
resource "grafana_asserts_suppressed_assertions_config" "known_issue_suppression" {
  provider = grafana.asserts

  name = "KnownIssueSuppression"

  match_labels = {
    alertname = "HighMemoryUsage"
    service   = "legacy-service"
    issue_id  = "LEG-123"
  }
}

# Suppress infrastructure alerts during planned maintenance
resource "grafana_asserts_suppressed_assertions_config" "infrastructure_maintenance" {
  provider = grafana.asserts

  name = "InfrastructureMaintenance"

  match_labels = {
    alertgroup = "infrastructure.alerts"
    maintenance_type = "planned"
    affected_services = "all"
  }
}
```

## Complex multi-label suppression

Define complex suppression rules with multiple labels:

```terraform
# Complex suppression for multi-service deployments
resource "grafana_asserts_suppressed_assertions_config" "multi_service_deployment" {
  provider = grafana.asserts

  name = "MultiServiceDeploymentSuppression"

  match_labels = {
    deployment_id = "deploy-2024-01-15"
    services      = "api,worker,frontend"
    environment   = "production"
    deployment_type = "blue_green"
  }
}

# Suppress alerts for specific cluster during maintenance
resource "grafana_asserts_suppressed_assertions_config" "cluster_maintenance" {
  provider = grafana.asserts

  name = "ClusterMaintenanceSuppression"

  match_labels = {
    cluster     = "production-cluster-1"
    maintenance = "cluster_upgrade"
    affected_nodes = "all"
    estimated_duration = "2h"
  }
}

# Suppress alerts for specific region during network issues
resource "grafana_asserts_suppressed_assertions_config" "regional_network_issue" {
  provider = grafana.asserts

  name = "RegionalNetworkIssueSuppression"

  match_labels = {
    region      = "us-west-2"
    issue_type  = "network"
    affected_services = "external_dependencies"
    incident_id = "NET-456"
  }
}
```

## Resource reference

### `grafana_asserts_suppressed_assertions_config`

Manage Knowledge Graph suppressed assertions configurations through the Grafana API.

#### Arguments

| Name           | Type          | Required | Description                                                                                                        |
| -------------- | ------------- | -------- | ------------------------------------------------------------------------------------------------------------------ |
| `name`         | `string`      | Yes      | The name of the suppressed assertions configuration. This field is immutable and forces recreation if changed.     |
| `match_labels` | `map(string)` | No       | Labels to match for this suppressed assertions configuration. Used to determine which alerts should be suppressed. |

#### Example

```terraform
resource "grafana_asserts_suppressed_assertions_config" "example" {
  provider = grafana.asserts

  name = "ExampleSuppression"

  match_labels = {
    alertname = "TestAlert"
    env       = "development"
  }
}
```

## Best practices

Consider the following best practices when managing suppressed assertions with Terraform.

### Suppression strategy

- Use suppression rules for temporary situations rather than permanent solutions
- Document the reason for suppression in your Terraform configuration comments
- Set expiration dates or reminders to review suppression rules
- Prefer fixing alert thresholds over suppressing recurring false positives

### Label match rules

- Be specific with match labels to avoid suppressing unintended alerts
- Test suppression rules in non-production environments first
- Use descriptive names that indicate the purpose and scope of the suppression
- Include relevant context in labels (for example, incident IDs, maintenance windows)

### Lifecycle management

- Regularly review active suppression rules to ensure they're still needed
- Remove or update suppression rules after maintenance windows or deployments
- Use version control to track when suppression rules were added and why
- Consider using time-based automation to enable or disable suppression rules

## Validation

After applying the Terraform configuration, verify that:

- Suppressed assertions configurations are active in your Knowledge Graph instance
- Configurations appear in the Knowledge Graph UI under **Observability > Rules > Suppress**
- Matching alerts are properly suppressed
- Suppression rules don't affect unintended alerts

## Related documentation

- [Suppress insights in Knowledge Graph](/docs/grafana-cloud/knowledge-graph/troubleshoot-infra-apps/suppress-insights/)
- [Get started with Terraform for Knowledge Graph](../getting-started/)
- [Configure notifications](/docs/grafana-cloud/knowledge-graph/configure/notifications/)

# Configure thresholds using Terraform

Threshold configurations in [Knowledge Graph](/docs/grafana-cloud/knowledge-graph/) allow you to define custom thresholds for request, resource, and health assertions. These configurations help you set specific limits and conditions for monitoring your services and infrastructure.

For information about managing thresholds in the Knowledge Graph UI, refer to [Manage thresholds](/docs/grafana-cloud/knowledge-graph/configure/manage-thresholds/).

## Basic threshold configuration

Create a file named `thresholds.tf` and add the following:

```terraform
# Basic threshold configuration with all three types
resource "grafana_asserts_thresholds" "basic" {
  provider = grafana.asserts

  request_thresholds = [{
    entity_name     = "payment-service"
    assertion_name  = "ErrorRatioBreach"
    request_type    = "inbound"
    request_context = "/charge"
    value           = 0.01
  }]

  resource_thresholds = [{
    assertion_name = "Saturation"
    resource_type  = "container"
    container_name = "worker"
    source         = "metrics"
    severity       = "warning"
    value          = 75
  }]

  health_thresholds = [{
    assertion_name = "ServiceDown"
    expression     = "up < 1"
    entity_type    = "Service"
  }]
}
```

## Request threshold configurations

Configure thresholds for different service request types and contexts:

```terraform
# Multiple request thresholds for different services
resource "grafana_asserts_thresholds" "request_thresholds" {
  provider = grafana.asserts

  request_thresholds = [
    {
      entity_name     = "api-service"
      assertion_name  = "ErrorRatioBreach"
      request_type    = "inbound"
      request_context = "/api/v1/users"
      value           = 0.02
    },
    {
      entity_name     = "api-service"
      assertion_name  = "LatencyP99ErrorBuildup"
      request_type    = "inbound"
      request_context = "/api/v1/orders"
      value           = 500
    },
    {
      entity_name     = "payment-gateway"
      assertion_name  = "RequestRateAnomaly"
      request_type    = "outbound"
      request_context = "/payment/process"
      value           = 1000
    }
  ]
}
```

## Resource threshold configurations

Define resource thresholds for different severity levels:

```terraform
# Resource thresholds for different severity levels
resource "grafana_asserts_thresholds" "resource_thresholds" {
  provider = grafana.asserts

  resource_thresholds = [
    {
      assertion_name = "Saturation"
      resource_type  = "container"
      container_name = "web-server"
      source         = "metrics"
      severity       = "warning"
      value          = 75
    },
    {
      assertion_name = "Saturation"
      resource_type  = "container"
      container_name = "web-server"
      source         = "metrics"
      severity       = "critical"
      value          = 90
    },
    {
      assertion_name = "ResourceRateBreach"
      resource_type  = "Pod"
      container_name = "database"
      source         = "logs"
      severity       = "warning"
      value          = 80
    }
  ]
}
```

## Health threshold configurations

Configure health checks with Prometheus expressions:

```terraform
# Health thresholds with Prometheus expressions
resource "grafana_asserts_thresholds" "health_thresholds" {
  provider = grafana.asserts

  health_thresholds = [
    {
      assertion_name = "ServiceDown"
      expression     = "up{job=\"api-service\"} < 1"
      entity_type    = "Service"
    },
    {
      assertion_name = "HighMemoryUsage"
      expression     = "memory_usage_percent > 85"
      entity_type    = "Service"
    },
    {
      assertion_name = "DatabaseConnectivity"
      expression     = "db_connection_pool_active / db_connection_pool_max > 0.9"
      entity_type    = "Service"
    }
  ]
}
```

## Comprehensive threshold configuration

Define comprehensive thresholds for production environments:

```terraform
# Production environment with comprehensive thresholds
resource "grafana_asserts_thresholds" "production" {
  provider = grafana.asserts

  request_thresholds = [
    {
      entity_name     = "frontend"
      assertion_name  = "ErrorRatioBreach"
      request_type    = "inbound"
      request_context = "/"
      value           = 0.005
    },
    {
      entity_name     = "backend-api"
      assertion_name  = "LatencyP99ErrorBuildup"
      request_type    = "inbound"
      request_context = "/api"
      value           = 200
    }
  ]

  resource_thresholds = [
    {
      assertion_name = "Saturation"
      resource_type  = "container"
      container_name = "frontend"
      source         = "metrics"
      severity       = "warning"
      value          = 70
    },
    {
      assertion_name = "Saturation"
      resource_type  = "container"
      container_name = "backend-api"
      source         = "metrics"
      severity       = "critical"
      value          = 85
    }
  ]

  health_thresholds = [
    {
      assertion_name = "ServiceDown"
      expression     = "up < 1"
      entity_type    = "Service"
    },
    {
      assertion_name = "NodeDown"
      expression     = "up{job=\"node-exporter\"} < 1"
      entity_type    = "Service"
    }
  ]
}
```

## Resource reference

### `grafana_asserts_thresholds`

Manage Knowledge Graph threshold configurations through the Grafana API. This resource allows you to define custom thresholds for request, resource, and health assertions.

#### Arguments

| Name                  | Type           | Required | Description                                                                                                              |
| --------------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------------ |
| `request_thresholds`  | `list(object)` | No       | List of request threshold configurations. Refer to [request thresholds block](#request-thresholds-block) for details.    |
| `resource_thresholds` | `list(object)` | No       | List of resource threshold configurations. Refer to [resource thresholds block](#resource-thresholds-block) for details. |
| `health_thresholds`   | `list(object)` | No       | List of health threshold configurations. Refer to [health thresholds block](#health-thresholds-block) for details.       |

#### Request thresholds block

Each `request_thresholds` block supports the following:

| Name              | Type     | Required | Description                                            |
| ----------------- | -------- | -------- | ------------------------------------------------------ |
| `entity_name`     | `string` | Yes      | The name of the entity to apply the threshold to.      |
| `assertion_name`  | `string` | Yes      | The name of the assertion to configure.                |
| `request_type`    | `string` | Yes      | The type of request (inbound, outbound).               |
| `request_context` | `string` | Yes      | The request context or path to apply the threshold to. |
| `value`           | `number` | Yes      | The threshold value.                                   |

#### Resource thresholds block

Each `resource_thresholds` block supports the following:

| Name             | Type     | Required | Description                                          |
| ---------------- | -------- | -------- | ---------------------------------------------------- |
| `assertion_name` | `string` | Yes      | The name of the assertion to configure.              |
| `resource_type`  | `string` | Yes      | The type of resource (container, Pod, node).         |
| `container_name` | `string` | Yes      | The name of the container to apply the threshold to. |
| `source`         | `string` | Yes      | The source of the metrics (metrics, logs).           |
| `severity`       | `string` | Yes      | The severity level (warning, critical).              |
| `value`          | `number` | Yes      | The threshold value.                                 |

#### Health thresholds block

Each `health_thresholds` block supports the following:

| Name             | Type     | Required | Description                                                                          |
| ---------------- | -------- | -------- | ------------------------------------------------------------------------------------ |
| `assertion_name` | `string` | Yes      | The name of the assertion to configure.                                              |
| `expression`     | `string` | Yes      | The Prometheus expression for the health check.                                      |
| `entity_type`    | `string` | Yes      | Entity type for the health threshold (for example, Service, Pod, Namespace, Volume). |
| `alert_category` | `string` | No       | Optional alert category label for the health threshold.                              |

#### Example

```terraform
resource "grafana_asserts_thresholds" "example" {
  provider = grafana.asserts

  request_thresholds = [{
    entity_name     = "api-service"
    assertion_name  = "ErrorRatioBreach"
    request_type    = "inbound"
    request_context = "/api/v1/users"
    value           = 0.02
  }]

  resource_thresholds = [{
    assertion_name = "Saturation"
    resource_type  = "container"
    container_name = "web-server"
    source         = "metrics"
    severity       = "warning"
    value          = 75
  }]

  health_thresholds = [{
    assertion_name = "ServiceDown"
    expression     = "up{job=\"api-service\"} < 1"
    entity_type    = "Service"
  }]
}
```

## Best practices

Consider the following best practices when configuring thresholds with Terraform.

### Threshold configuration management

- Set appropriate threshold values based on your service level objectives (SLOs)
- Use different severity levels (warning, critical) to create escalation paths
- Test threshold configurations in non-production environments first
- Monitor threshold effectiveness and adjust values based on actual performance data

### Request threshold best practices

- Configure request thresholds for critical user-facing endpoints
- Set different thresholds for different request types (inbound vs outbound)
- Consider request context when setting thresholds for specific API paths
- Use error ratio thresholds to catch service degradation early
- Review historical performance data to set realistic threshold values

### Resource threshold best practices

- Set resource thresholds based on your infrastructure capacity
- Use container-specific thresholds for microservices architectures
- Configure both warning and critical thresholds for gradual escalation
- Monitor resource utilization patterns to set realistic threshold values
- Consider seasonal or periodic patterns in resource usage

### Health threshold best practices

- Use Prometheus expressions that accurately reflect service health
- Test health check expressions independently before applying them
- Set up health thresholds for critical dependencies and external services
- Use composite expressions for complex health checks
- Ensure expressions perform efficiently without causing excessive load

### Value selection guidelines

- Start conservative and adjust based on real-world performance
- Use percentages (0-1 range) for ratio-based metrics
- Use milliseconds for latency thresholds
- Document the reasoning behind specific threshold values
- Review and update thresholds regularly based on system evolution

## Validation

After applying the Terraform configuration, verify that:

- Threshold configurations are applied in your Knowledge Graph instance
- Configurations appear in the Knowledge Graph UI under **Observability > Rules > Threshold**
- Request thresholds correctly identify breaches for specified services
- Resource thresholds trigger at appropriate severity levels
- Health thresholds accurately reflect service status
- Threshold values align with your SLO commitments

## Related documentation

- [Manage thresholds in Knowledge Graph](/docs/grafana-cloud/knowledge-graph/configure/manage-thresholds/)
- [Get started with Terraform for Knowledge Graph](../getting-started/)
- [Configure alerts in Knowledge Graph](/docs/grafana-cloud/knowledge-graph/configure/alerts/)

# Configure trace correlation using Terraform

Trace configurations in [Knowledge Graph](/docs/grafana-cloud/knowledge-graph/) allow you to define how trace data is queried and correlated with entities. You can specify data sources, entity matching rules, and label mappings for distributed tracing.

For information about configuring trace correlation in the Knowledge Graph UI, refer to [Configure traces correlation](/docs/grafana-cloud/knowledge-graph/configure/traces-correlation/).

## Basic trace configuration

Create a file named `trace-configs.tf` and add the following:

```terraform
# Basic trace configuration for services
resource "grafana_asserts_trace_config" "production" {
  provider = grafana.asserts

  name            = "production"
  priority        = 1000
  default_config  = false
  data_source_uid = "grafanacloud-traces"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service"]
  }

  match {
    property = "deployment_environment"
    op       = "="
    values   = ["production", "staging"]
  }

  entity_property_to_trace_label_mapping = {
    "cluster"        = "resource.k8s.cluster.name"
    "namespace"      = "resource.k8s.namespace"
    "container"      = "resource.container.name"
    "otel_service"   = "resource.service.name"
    "otel_namespace" = "resource.service.namespace"
  }
}
```

## Trace configuration with multiple match rules

Configure trace correlation with multiple entity matching criteria:

```terraform
# Development environment trace configuration
resource "grafana_asserts_trace_config" "development" {
  provider = grafana.asserts

  name            = "development"
  priority        = 2000
  default_config  = false
  data_source_uid = "tempo-dev"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service"]
  }

  match {
    property = "deployment_environment"
    op       = "="
    values   = ["development", "testing"]
  }

  match {
    property = "asserts_site"
    op       = "="
    values   = ["us-east-1"]
  }

  match {
    property = "service"
    op       = "="
    values   = ["api"]
  }

  entity_property_to_trace_label_mapping = {
    "cluster"        = "resource.k8s.cluster.name"
    "namespace"      = "resource.k8s.namespace"
    "container"      = "resource.container.name"
    "otel_service"   = "resource.service.name"
    "otel_namespace" = "resource.service.namespace"
    "pod"            = "span.k8s.pod.name"
  }
}
```

## Minimal trace configuration

Create a minimal configuration for all entities:

```terraform
# Minimal configuration for all entities
resource "grafana_asserts_trace_config" "minimal" {
  provider = grafana.asserts

  name            = "minimal"
  priority        = 3000
  default_config  = false
  data_source_uid = "tempo-minimal"

  match {
    property = "asserts_entity_type"
    op       = "IS NOT NULL"
    values   = []
  }

  entity_property_to_trace_label_mapping = {
    "cluster"        = "resource.k8s.cluster.name"
    "otel_service"   = "resource.service.name"
    "otel_namespace" = "resource.service.namespace"
  }
}
```

## Advanced trace configuration with complex match rules

Configure traces with multiple operations and advanced match rules:

```terraform
# Advanced configuration with multiple operations
resource "grafana_asserts_trace_config" "advanced" {
  provider = grafana.asserts

  name            = "advanced"
  priority        = 1500
  default_config  = false
  data_source_uid = "tempo-advanced"

  match {
    property = "service_type"
    op       = "CONTAINS"
    values   = ["web", "api"]
  }

  match {
    property = "deployment_environment"
    op       = "<>"
    values   = ["test"]
  }

  match {
    property = "team"
    op       = "IS NOT NULL"
    values   = []
  }

  match {
    property = "priority_level"
    op       = ">="
    values   = ["5"]
  }

  entity_property_to_trace_label_mapping = {
    "service_type"   = "resource.service.type"
    "team"           = "resource.team.owner"
    "environment"    = "resource.deployment.environment"
    "version"        = "resource.service.version"
    "region"         = "resource.cloud.region"
  }
}
```

## Resource reference

### `grafana_asserts_trace_config`

Manage Knowledge Graph trace configurations through the Grafana API.

#### Arguments

| Name                                     | Type           | Required | Description                                                                                    |
| ---------------------------------------- | -------------- | -------- | ---------------------------------------------------------------------------------------------- |
| `name`                                   | `string`       | Yes      | The name of the trace configuration. This field is immutable and forces recreation if changed. |
| `priority`                               | `number`       | Yes      | Priority of the trace configuration. A lower number means a higher priority.                   |
| `default_config`                         | `bool`         | Yes      | Whether this is the default configuration. Default configurations cannot be deleted.           |
| `data_source_uid`                        | `string`       | Yes      | DataSource UID to be queried (for example, a Tempo instance).                                  |
| `match`                                  | `list(object)` | No       | List of match rules for entity properties. Refer to [match block](#match-block) for details.   |
| `entity_property_to_trace_label_mapping` | `map(string)`  | No       | Mapping of entity properties to trace labels for correlation.                                  |

#### Match block

Each `match` block supports the following:

| Name       | Type           | Required | Description                                                                                                                  |
| ---------- | -------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `property` | `string`       | Yes      | Entity property to match against.                                                                                            |
| `op`       | `string`       | Yes      | Operation to use for matching. One of: `=`, `<>`, `<`, `>`, `<=`, `>=`, `IS NULL`, `IS NOT NULL`, `STARTS WITH`, `CONTAINS`. |
| `values`   | `list(string)` | Yes      | Values to match against. Can be empty for `IS NULL` and `IS NOT NULL` operations.                                            |

#### Example

```terraform
resource "grafana_asserts_trace_config" "example" {
  provider = grafana.asserts

  name            = "example-traces"
  priority        = 1000
  default_config  = false
  data_source_uid = "tempo-prod"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service", "Pod"]
  }

  match {
    property = "deployment_environment"
    op       = "STARTS WITH"
    values   = ["prod"]
  }

  entity_property_to_trace_label_mapping = {
    "service"     = "resource.service.name"
    "namespace"   = "resource.k8s.namespace"
    "environment" = "resource.deployment.environment"
    "cluster"     = "resource.k8s.cluster.name"
  }
}
```

## Best practices

Consider the following best practices when configuring trace correlation with Terraform.

### Priority management

- Assign lower priority numbers to more specific configurations
- A lower priority number means higher priority (configurations are evaluated in ascending priority order)
- Use consistent priority ranges for different configuration types
- Document the reasoning behind priority assignments

### Data source configuration

- Ensure the data source UID matches your actual Tempo or trace aggregation system
- Test data source connectivity before applying configurations
- Use descriptive names for trace configurations to indicate their purpose
- Consider using separate data sources for different environments

### Label map strategy

- Map entity properties consistently across all trace configurations
- Use OpenTelemetry semantic conventions for trace label names (e.g., `resource.service.name`, `resource.k8s.namespace`)
- Document the mapping relationships in configuration comments
- Verify that mapped labels exist in your trace data

### Match rules design

- Start with broad match rules and refine based on needs
- Use specific property names that exist in your entity model
- Test match rules with sample data before deploying
- Combine multiple match rules for precise entity targeting
- Leverage comparison operators (`<`, `>`, `<=`, `>=`) for numeric or version-based filtering

### Distributed trace integration

- Ensure your traces follow OpenTelemetry semantic conventions
- Use consistent label names for resource attributes across your tracing infrastructure
- Map both service-level and infrastructure-level properties for comprehensive correlation
- Test trace correlation to verify it works as expected

## Validation

After applying the Terraform configuration, verify that:

- Trace configurations are created in your Knowledge Graph instance
- Configurations appear in the Knowledge Graph UI under **Observability > Configuration > Traces**
- Trace correlation works when drilling down from entities
- Label mappings correctly translate entity properties to trace resource attributes
- Match rules properly filter entities
- Trace queries return expected results in the Knowledge Graph UI

## Related documentation

- [Configure traces correlation in Knowledge Graph](/docs/grafana-cloud/knowledge-graph/configure/traces-correlation/)
- [Get started with Terraform for Knowledge Graph](../getting-started/)
- [Tempo documentation](/docs/tempo/latest/)
- [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/)

# Datasources

# Data sources, plugins, and integrations

When working with Grafana, you'll encounter three key concepts: data sources, plugins, and integrations. Each one is essential in building effective monitoring solutions, but they serve distinct purposes, and are often confused with one another. This document clarifies the meaning of each concept and what each one does, when to use it, and how they work together to create observability solutions in Grafana.

The most important thing to remember: every data source is powered by a plugin. You _install_ a plugin, then _configure_ a data source. Core plugins like Prometheus come bundled with Grafana.

## Data sources

A data source is a connection to a specific database, monitoring system, service, or other external location that stores metrics, logs, traces, or other data. Examples include Prometheus, InfluxDB, PostgreSQL, and Amazon CloudWatch.

When you configure a data source in Grafana, you provide connection details, credentials, and endpoints that tell Grafana where to fetch data from. Data sources are the foundation for working with Grafana, without them, Grafana has nothing to visualize.

Every data source is powered by a data source plugin. Grafana comes bundled with core data source plugins (like Prometheus, Loki, and MySQL), and hundreds more are available in the [plugin catalog](https://grafana.com/grafana/plugins/) for systems ranging from databases and cloud providers to tools like Jira, GitHub, and Datadog. After you configure a data source, you can query it to build dashboards. For example, you can query a Prometheus data source to display CPU metrics or query CloudWatch to visualize AWS infrastructure performance.

## Plugins

A plugin extends the core functionality of Grafana. Plugins can add new data source types, visualization panels, or full-featured applications that integrate with Grafana. They make Grafana modular and extensible.

Plugins come in three types:

- **Data source plugins** enable Grafana to connect to specific types of databases or services. Core data source plugins like Prometheus, Loki, and MySQL come bundled with Grafana. You can install additional data source plugins to connect to other systems. Examples include MSSQL, Databricks, and MongoDB.

- **Panel plugins** control how data appears in Grafana dashboards. While Grafana comes with several built-in panel types (like graphs, single stats, and tables), panel plugins extend this by providing specialized ways to display data. Examples include pie chart, candlestick, and traffic light. Note that in some cases, panels don't rely on a data source at all. The **Text** panel can render static or templated content without querying data. Panels can also support user-driven actions. For example, the **Button** panel can trigger workflows or external calls.

- **App plugins** package data sources, panels, and custom pages into a single install. When you add an app plugin, you get a purpose-built experience for a specific use case. For example, the Azure Cloud Native Monitoring app gives you pre-configured data sources, dashboards, and a dedicated management page for your Azure environment, all in one install. If you're a developer, you can also [build your own app plugin](https://grafana.com/developers/plugin-tools/how-to-guides/app-plugins/).

## Integrations

_Integrations are exclusive to Grafana Cloud._ Integrations are pre-packaged monitoring solutions that bundle data collection instructions, pre-built dashboards, and alert and recording rules. They support metrics, logs, and traces. Unlike standalone data sources, integrations provide tested configuration and ready-to-use dashboards and alerts. For example, a Linux integration provides instructions and configuration for collecting metrics and logs from your hosts, creates dashboards for monitoring, and sets up common alerts—all working together out of the box.

## When to use each

Use a data source when:

- You want to connect Grafana to a specific system (for example, Prometheus or MySQL).
- You're building custom dashboards with hand-picked metrics and visualizations.
- Your monitoring needs are unique or not covered by pre-packaged integrations.

Use a plugin when:

- You need to connect to a system Grafana doesn't support natively.
- You want to add new functionality (visualizations, workflows, or app-style extensions).
- You have specialized or industry-specific requirements (for example, IoT).

Use an integration when:

- You're using Grafana Cloud and want a quick, pre-built setup.
- You prefer minimal configuration with ready-to-use dashboards and alerts.
- You're new to observability and want to learn what good monitoring looks like.

## Relationships and interactions

Data sources, plugins, and integrations work together to build observability solutions. The following diagram shows two paths: the custom path where you install plugins and configure data sources yourself, and the integration path where Grafana Cloud provides a pre-packaged setup.

{{< figure src="/media/docs/datasources/concepts-diagram-1.png" alt="Diagram showing the custom path and integration path for working with data in Grafana" >}}

<!-- ```text
  CUSTOM PATH                      INTEGRATION PATH
  (Self-hosted & Cloud)            (Cloud only)

  ┌────────────────────┐           ┌────────────────────┐
  │  Install plugin    │           │  Add integration   │
  └─────────┬──────────┘           └─────────┬──────────┘
            │ enables                        │ bundles
            ▼                                ▼
  ┌────────────────────┐           ┌────────────────────┐
  │  Configure data    │           │  Collection config, │
  │  source (URL,      │           │  dashboards,       │
  │  auth, endpoint)   │           │  alerts, rules     │
  └─────────┬──────────┘           └─────────┬──────────┘
            │ connects to                    │ ingests into
            ▼                                ▼
  ┌────────────────────┐           ┌────────────────────┐
  │  External backend  │           │  Grafana Cloud     │
  │  (Prometheus,      │           │  managed backends  │
  │   MySQL, etc.)     │           │  (metrics, logs,   │
  │                    │           │   traces)          │
  └─────────┬──────────┘           └─────────┬──────────┘
            │ queries                        │ queries
            ▼                                ▼
  ┌────────────────────┐           ┌────────────────────┐
  │  Build your own    │           │  Ready-to-use      │
  │  panels and        │           │  dashboards and    │
  │  dashboards        │           │  alerts            │
  └────────────────────┘           └────────────────────┘
``` -->

App plugins can combine both paths by bundling data source plugins, panel plugins, and custom pages into a single package.

The following examples show how these concepts work together in practice:

- Grafana includes a built-in Prometheus data source plugin. Configure a Prometheus data source by providing your server URL, then build a dashboard with a time series panel to monitor CPU usage across your hosts.

- Install the Databricks data source plugin. Configure the Databricks data source and run SQL queries against your Databricks workspace. Use the `Histogram` panel to visualize distributions in your query results, such as latency buckets, job durations, or model output scores.

- Install the Redis Application app plugin. This app provides a unified experience for monitoring Redis by working with your existing Redis data source. It adds custom pages for configuration and exploration, along with prebuilt dashboards, commands, and visualizations that help you analyze performance, memory usage, and key activity.

- If you're using Grafana Cloud, add the ClickHouse integration. This integration provides pre-built dashboards and alerts to monitor ClickHouse cluster metrics and logs, so you can visualize and analyze ClickHouse performance and health.

## Frequently asked questions

**What's the difference between a data source and a data source plugin?**

A data source plugin is a **software component that enables Grafana to communicate** with specific types of databases or services, like Prometheus, MySQL, or InfluxDB. A data source is **an actual configured connection** to one of these databases, including the credentials, URL, and settings needed to retrieve data.

Think of it this way: You _install_ a plugin but _configure_ a data source.

**Do I need a plugin to use a data source?**

You must install the plugin before you configure or use the data source. Each data source plugin has its own versioning and lifecycle. Grafana includes built-in core data sources, which can be thought of as pre-installed plugins.

**Can I use integrations in self-managed Grafana?**

No, integrations are exclusive to Grafana Cloud. In self-managed Grafana, you can replicate similar setups manually using data sources and dashboards.

**Aren't integrations just pre-built dashboards?**

No, integrations are much more than just dashboards. While dashboards are part of an integration, they're only one piece. Integrations typically include:

- Data collection setup (for example, pre-configured agents or exporters).
- Predefined metrics and queries tailored to the technology.
- Alerting rules and notifications to help detect common issues.
- Dashboards to visualize and explore that data.

**How do data sources and integrations differ in how they handle data?**

Data sources query data where it already lives. They connect Grafana to an external system or database, such as Prometheus, MySQL, or Elasticsearch, and fetch data on demand. You keep full control over your own data stores, schemas, and retention policies.

In contrast, integrations focus on getting data into Grafana Cloud's managed backends for metrics, logs, and traces. They ingest telemetry using pre-configured agents and pipelines. Instead of querying an external database, Grafana queries its own managed storage where the integration has placed the data.

## Summary reference

The following table compares the three core concepts at a glance.

| Concept         | What it is                                                                                                                                                                | When to use it                                                   | Example                                                              |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------- |
| **Data source** | A configured connection to an external database or service. Available in self-managed and Cloud.                                                                          | You want to query a specific system and build custom dashboards. | Prometheus, CloudWatch, PostgreSQL                                   |
| **Plugin**      | An extension that adds new data source types, visualizations, or app experiences. Available in self-managed and Cloud. Comes in three types: data source, panel, and app. | You need functionality that isn't built into Grafana.            | MongoDB data source, Geomap panel, Azure Cloud Native Monitoring app |
| **Integration** | A pre-packaged monitoring solution with collection configuration, dashboards, and alerts. Grafana Cloud only.                                                             | You want a quick, ready-to-use setup with minimal configuration. | Linux, Redis, NGINX                                                  |

For detailed documentation and how-to guides related to data sources, plugins, and integrations, refer to the following references:

**Data sources**:

- [Manage data sources](ref:data-source-management)

**Plugins**:

- [Browse the Grafana plugin catalog](https://grafana.com/grafana/plugins/)
- [Manage plugins](ref:plugin-management)

**Integrations**:

- [Grafana integrations](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/)
- [Install and manage integrations](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/install-and-manage-integrations/)

**For plugin developers**:

- [Plugin types and usage](https://grafana.com/developers/plugin-tools/key-concepts/plugin-types-usage)
- [Build a data source plugin](https://grafana.com/developers/plugin-tools/how-to-guides/data-source-plugins/)
- [Build a panel plugin](https://grafana.com/developers/plugin-tools/how-to-guides/panel-plugins/)
- [Build an app plugin](https://grafana.com/developers/plugin-tools/how-to-guides/app-plugins/)

# Troubleshoot general data source issues

This page provides solutions to common issues that apply across data sources in Grafana. For troubleshooting specific to a data source, refer to the troubleshooting page within each data source's documentation.

## Connection errors

These errors occur when Grafana can't reach the data source backend.

### "Connection refused" or timeout errors

**Symptoms:**

- **Save & test** fails with a connection error or timeout
- Panels show "error" or fail to load data
- Intermittent connectivity issues

**Possible causes and solutions:**

| Cause                     | Solution                                                                                                                                             |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| Incorrect URL             | Verify the data source URL is correct, including the protocol (`http://` or `https://`) and port number.                                             |
| Network or firewall rules | Ensure the Grafana server can reach the data source endpoint. Check that firewall rules allow outbound traffic on the required port.                 |
| Data source is down       | Verify the data source service is running and accepting connections.                                                                                 |
| DNS resolution failure    | Confirm the hostname resolves correctly from the Grafana server.                                                                                     |
| Private network access    | If the data source is on a private network and you're using Grafana Cloud, configure [Private data source connect](ref:private-data-source-connect). |

### TLS/SSL errors

**Symptoms:**

- Errors mentioning "certificate," "TLS handshake," or "x509"
- **Save & test** fails with SSL-related messages

**Solutions:**

1. Verify the data source is using a valid TLS certificate.
1. If you're using a self-signed certificate, enable **Skip TLS Verify** in the data source configuration (not recommended for production) or add the CA certificate to the list of trusted certificates in Grafana.
1. Ensure the certificate hasn't expired.
1. Confirm the certificate's Common Name or Subject Alternative Name matches the hostname in the data source URL.

## Authentication errors

These errors occur when credentials are invalid, missing, or lack the required permissions.

### "Unauthorized" or "Access denied"

**Symptoms:**

- **Save & test** fails with `401 Unauthorized` or `403 Forbidden`
- Queries return access denied messages
- Drop-down menus don't populate

**Possible causes and solutions:**

| Cause                       | Solution                                                                                                                                                   |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Invalid credentials         | Double-check the username, password, API key, or token. Regenerate credentials if necessary.                                                               |
| Expired credentials         | Create new credentials and update the data source configuration.                                                                                           |
| Insufficient permissions    | Ensure the account or API key has the permissions required by the data source. Refer to the specific data source's documentation for required permissions. |
| Wrong authentication method | Verify you've selected the correct authentication type for your setup.                                                                                     |

## Query errors

These errors occur when executing queries against a properly connected data source.

### "No data" or empty results

**Symptoms:**

- Queries execute without error but return no data
- Panels show "No data"
- Graphs are empty

**Possible causes and solutions:**

| Cause                           | Solution                                                                                          |
| ------------------------------- | ------------------------------------------------------------------------------------------------- |
| Time range doesn't contain data | Expand the dashboard time range or verify data exists in the data source for the selected period. |
| Incorrect query                 | Review the query syntax. Use the data source's query editor to build or validate the query.       |
| Wrong data source selected      | Verify you've selected the correct data source in the panel or Explore.                           |
| Permissions issue               | Ensure the credentials have read access to the specific resource or index being queried.          |

### Query timeout

**Symptoms:**

- Queries run for a long time then fail
- Error messages mention timeout or query limits

**Solutions:**

1. Narrow the dashboard time range to reduce the volume of data.
1. Add filters to the query to reduce the result set.
1. Break complex queries into smaller parts.
1. Increase the data source timeout setting if your data source legitimately needs more time.

## Data source configuration errors

### "Save & test" fails after provisioning

**Symptoms:**

- Provisioned data sources fail the connection test
- Errors appear after deploying provisioning YAML files

**Solutions:**

1. Verify the provisioning YAML syntax is correct. Refer to [Provision data sources](ref:provisioning-data-sources) for the expected format.
1. Ensure `secureJsonData` values (such as passwords, API keys, and tokens) are set correctly. These values can't be read back after being saved.
1. Check that the provisioning file is in the correct directory and that Grafana has read access.
1. Restart Grafana after making changes to provisioning files.

### Data source disappears or resets

**Symptoms:**

- Data source changes revert after Grafana restarts
- Data source configuration can't be saved through the UI

**Solutions:**

1. Provisioned data sources can't be edited through the UI. Make changes in the provisioning YAML file instead.
1. Verify no other provisioning file is overwriting your data source configuration.

## Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Restart Grafana for the change to take effect.
1. Reproduce the issue and review logs at `/var/log/grafana/grafana.log` (or your configured log location).
1. Look for entries related to your data source that include request and response details.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If the solutions on this page don't resolve your issue:

1. Check the troubleshooting page for your specific data source.
1. Search the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs.
1. Contact [Grafana Support](https://grafana.com/support/) if you're a Grafana Enterprise, Cloud Pro, or Cloud Advanced user.

When reporting issues, include:

- Grafana version and data source plugin version
- Exact error messages (redact sensitive information)
- Steps to reproduce the issue
- Relevant configuration (redact credentials)

# Amazon CloudWatch alerting

The Amazon CloudWatch data source supports [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/). You can create alert rules on CloudWatch metrics and CloudWatch Logs, so Grafana notifies you when a metric crosses a threshold or when a log query returns a value that meets your alert condition.

This is Grafana Alerting evaluated by Grafana, not CloudWatch alarms. To visualize the history of existing CloudWatch alarms instead, refer to [Amazon CloudWatch annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/annotations/).

## Before you begin

Before you create alert rules with CloudWatch data, ensure you have:

- A [configured Amazon CloudWatch data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/).
- Permissions to create alert rules in Grafana.
- Familiarity with [Grafana Alerting concepts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/).

## Supported query types for alerting

Alert rules require queries that return numeric data that Grafana can evaluate against a threshold. CloudWatch supports alerting for the following query types:

| Query type | Use case                                           | Notes                                                          |
| ---------- | -------------------------------------------------- | -------------------------------------------------------------- |
| Metrics    | Threshold-based alerts on CloudWatch metrics       | Returns time series data; best suited for alerting.            |
| Logs       | Alert on log patterns, error counts, or thresholds | Use the `stats` command to aggregate logs into numeric values. |

{{< admonition type="note" >}}
You can't create an alert rule from a metric math expression that references another query, such as `queryA * 2`. Define the metric directly in the alert query instead.
{{< /admonition >}}

## Create an alert rule

To create an alert rule using CloudWatch data:

1. Go to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for your alert rule.
1. In the **Define query and alert condition** section:
   - Select your Amazon CloudWatch data source.
   - Configure your query, for example, a Metrics query for `CPUUtilization` or a Logs query that uses the `stats` command.
   - Add a **Reduce** expression if your query returns multiple series.
   - Add a **Threshold** expression to define the alert condition.
1. Configure the **Set evaluation behavior** section:
   - Select or create a folder and evaluation group.
   - Set the evaluation interval.
   - Set the pending period.
1. Add labels and annotations to provide context for notifications.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example: metric threshold alert

This example fires when average EC2 CPU utilization exceeds 80%:

1. Create a new alert rule.
1. Configure the query:
   - **Namespace**: `AWS/EC2`
   - **Metric name**: `CPUUtilization`
   - **Statistic**: `Average`
   - **Dimensions**: `InstanceId`
1. Add expressions:
   - **Reduce**: Last, to get the most recent data point.
   - **Threshold**: Is above 80.
1. Set evaluation to run every 1 minute with a 5-minute pending period.
1. Save the rule.

## Example: log error count alert

This example alerts when the number of log entries that contain `Exception` exceeds a threshold. CloudWatch Logs alerting requires a query that returns numeric data, which you can produce with the `stats` command:

1. Create a new alert rule.
1. Configure the query:
   - Select **CloudWatch Logs** as the query mode.
   - Select the region and log groups to query.
   - Enter the following query:

     ```
     filter @message like /Exception/
         | stats count(*) as exceptionCount by bin(5m)
     ```

1. Add expressions:
   - **Reduce**: Max, to get the highest count in the period.
   - **Threshold**: Is above 10.
1. Set evaluation to run every 5 minutes.
1. Save the rule.

{{< admonition type="note" >}}
If you receive an error such as `input data must be a wide series but got ...`, make sure your query returns numeric data that can render in a time series panel.
{{< /admonition >}}

## CloudWatch Logs timeouts during alert evaluation

For CloudWatch Logs queries, Grafana polls AWS until the query completes or a timeout is reached. During alert evaluation, the timeout defined in the [Grafana configuration file](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/) takes precedence over the **Query Result Timeout** setting on the data source. Set the configuration timeout high enough for your log queries to finish.

## Best practices

Follow these recommendations to create reliable alerts with CloudWatch data.

### Test queries before alerting

Verify your query returns numeric data before you create an alert:

1. Go to **Explore**.
1. Select your Amazon CloudWatch data source.
1. Run the query you plan to use for alerting.
1. Confirm the result is numeric and suitable for threshold evaluation.

### Reduce multiple series

When a query returns multiple time series, use the **Reduce** expression to aggregate them into a single value with **Last**, **Mean**, **Max**, **Min**, or **Sum**.

### Handle no data conditions

Configure how the rule behaves when no data is returned under **Configure no data and error handling**. Choose **No Data**, **Alerting**, or **OK** based on whether missing data should be treated as a problem.

## Troubleshoot alerting

If your CloudWatch alerts don't work as expected, use the following sections to diagnose common issues.

### Alerts don't fire

- Confirm the query returns numeric data in Explore.
- Ensure the evaluation interval allows enough time for data to be available.
- Review the alert rule's health and any error messages in the Alerting UI.

### Log query alerts time out

- Increase the Grafana configuration file timeout, which takes precedence during alert evaluation.
- Narrow the time range or add filters to reduce the volume of data the query scans.

For more help, refer to [Troubleshoot Amazon CloudWatch](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/troubleshooting/).

## Related resources

- [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/)
- [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/)
- [Amazon CloudWatch query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/query-editor/)

# Amazon CloudWatch annotations

[Annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/) overlay rich event information on top of graphs. With the Amazon CloudWatch data source, annotations are based on [CloudWatch alarm history](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html). Each alarm history item within the dashboard time range—such as a state change, configuration update, or action—becomes an annotation, so you can correlate alarm activity with the rest of your dashboard data.

## Before you begin

Before you create CloudWatch annotations, ensure you have:

- A [configured Amazon CloudWatch data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/).
- One or more CloudWatch alarms in the AWS account and region you want to query.
- IAM permissions to describe alarms and alarm history. Refer to [Required permissions](#required-permissions).

## How CloudWatch annotations work

Unlike data sources that build annotations from free-form queries, CloudWatch annotations query existing CloudWatch alarms and return their state-change history. Grafana finds the matching alarms using one of two methods, retrieves each alarm's history with `cloudwatch:DescribeAlarmHistory`, and renders every history item as an annotation:

- **Match alarms by metric:** Grafana calls `cloudwatch:DescribeAlarmsForMetric` to find alarms attached to a specific metric.
- **Match alarms by prefix:** Grafana calls `cloudwatch:DescribeAlarms` with your alarm name and action prefixes, so AWS returns only alarms whose names and actions start with those prefixes. Grafana then filters the returned alarms by the namespace, metric name, dimensions, statistic, and period you specify.

Each annotation includes the alarm name as its title, the alarm history item type as a tag (for example, `StateUpdate`, `ConfigurationUpdate`, or `Action`), and the history summary as its text.

## Create an annotation query

To add a CloudWatch annotation to a dashboard:

1. Open the dashboard where you want to add annotations.
1. Click **Edit**, then click **Settings** in the top navigation.
1. Select the **Annotations** tab.
1. Click **Add annotation query**.
1. Enter a **Name** for the annotation, for example, `CloudWatch alarms`.
1. Select your **Amazon CloudWatch** data source.
1. Configure the query fields described in the following sections.
1. Click **Save dashboard**.

The annotation query editor provides the following fields:

| Field                      | Description                                                                                                                           |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| **Region**                 | The AWS region to query for alarms.                                                                                                   |
| **Namespace**              | The metric namespace, for example, `AWS/EC2`.                                                                                         |
| **Metric name**            | The name of the metric the alarm watches, for example, `CPUUtilization`.                                                              |
| **Statistic**              | The statistic the alarm uses, for example, `Average`.                                                                                 |
| **Dimensions**             | The dimensions that identify the resource, for example, `InstanceId`.                                                                 |
| **Period**                 | _Optional._ The minimum interval between data points, in seconds. Defaults to `300` when prefix matching is disabled.                 |
| **Enable Prefix Matching** | _Optional._ Match alarms by name and action prefix instead of by metric.                                                              |
| **Action**                 | Match only alarms whose actions start with this prefix. Available when prefix matching is enabled, and required for the query to run. |
| **Alarm Name**             | Match only alarms whose names start with this prefix. Available when prefix matching is enabled, and required for the query to run.   |

### Match alarms by metric

This is the default behavior, used when **Enable Prefix Matching** is off. Grafana returns annotations for any alarm attached to the metric you specify.

When you match by metric, **Region**, **Namespace**, **Metric name**, and **Statistic** are all required. If any of them are missing, the query returns an `invalid annotations query` error. Dimensions are optional, but adding them narrows the results to alarms on a specific resource.

For example, to annotate the history of the CPU utilization alarm on a single EC2 instance, set:

- **Region:** `us-east-1`
- **Namespace:** `AWS/EC2`
- **Metric name:** `CPUUtilization`
- **Statistic:** `Average`
- **Dimensions:** `InstanceId = i-0123456789abcdef0`

### Match alarms by prefix

Enable **Enable Prefix Matching** to find alarms by name and action prefix rather than by a specific metric. Grafana requests up to 100 alarms from `cloudwatch:DescribeAlarms` using the **Alarm Name** and **Action** prefixes, then filters the returned alarms by the namespace, metric name, dimensions, statistic, and period you specify. You must provide both an **Alarm Name** prefix and an **Action** prefix for the query to run.

Use prefix matching when you want to surface a group of related alarms, for example, all production alarms that trigger a specific SNS action. To do this, set **Enable Prefix Matching** on, then set **Alarm Name** to `prod-` and **Action** to the ARN prefix of your SNS topic, such as `arn:aws:sns:us-east-1:123456789012:prod-`.

## Required permissions

Annotation queries require the following CloudWatch API actions in the IAM policy attached to the role or user running the data source:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingAlarmsFromCloudWatch",
      "Effect": "Allow",
      "Action": ["cloudwatch:DescribeAlarms", "cloudwatch:DescribeAlarmsForMetric", "cloudwatch:DescribeAlarmHistory"],
      "Resource": "*"
    }
  ]
}
```

The metrics IAM policy on the [configure page](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/#iam-policy-examples) already includes these actions.

## Troubleshoot annotations

If annotations don't appear as expected, try the following solutions.

### Annotations don't appear

- Verify the alarm has state changes within the dashboard time range. Annotations come from alarm history, so an alarm that never changed state produces no annotations.
- Confirm the **Region**, **Namespace**, **Metric name**, **Statistic**, and **Dimensions** exactly match an existing alarm.
- Check that the IAM role or user has the `cloudwatch:DescribeAlarms`, `cloudwatch:DescribeAlarmsForMetric`, and `cloudwatch:DescribeAlarmHistory` permissions.

### `invalid annotations query` error

- This error appears when prefix matching is disabled and the query reaches the backend with an empty **Region**, **Namespace**, **Metric name**, or **Statistic**. Provide all four fields, or enable **Enable Prefix Matching**.
- In the dashboard annotation editor, an incomplete metric query is often skipped silently and produces no annotations rather than returning this error. If you see no annotations, confirm that all required fields are set.

## Related resources

- [Annotate visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/)
- [Amazon CloudWatch query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/query-editor/)
- [Configure the Amazon CloudWatch data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/)

# Configure AWS authentication

Grafana data source plugins make requests to AWS on behalf of an AWS Identity and Access Management (IAM) [role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) or IAM [user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html).
The IAM user or IAM role must have the associated policies to perform certain API actions for querying data source data.
Since these policies are specific to each data source, refer to individual data source plugin documentation for details.

The Grafana backend performs all AWS API requests server-side using the official [AWS SDK](https://github.com/aws/aws-sdk-go).

This document explores the following topics:

- [Select an authentication method](#select-an-authentication-method)
- [Assume a role](#assume-a-role)
- [Use a custom endpoint](#use-a-custom-endpoint)
- [Use an AWS credentials file](#use-an-aws-credentials-file)
- [Use EKS IAM roles for service accounts](#use-eks-iam-roles-for-service-accounts)

## Select an authentication method

Available authentication methods depend on your configuration and the environment where Grafana runs.

Open source Grafana enables the `AWS SDK Default`, `Credentials file`, and `Access & secret key` methods by default. Cloud Grafana enables `Access & secret key` and `Grafana Assume Role` by default. Users with server configuration access can enable or disable specific auth providers as needed. For more information, refer to the [`allowed_auth_providers` documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#allowed_auth_providers).

- `AWS SDK Default` uses the [default provider](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html) from the [AWS SDK for Go](https://github.com/aws/aws-sdk-go) without custom configuration.
  This method requires configuring AWS credentials outside Grafana through [the CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html), or by [attaching credentials directly to an EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html), [in an ECS task](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html), or for a [Service Account in a Kubernetes cluster](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). You can attach permissions directly to the data source with AWS SDK Default or combine it with the optional [`Assume Role ARN`](#assume-a-role) field.
- `Credentials file` maps to the [SharedCredentialsProvider](https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/#SharedCredentialsProvider) provider in the [AWS SDK for Go](https://github.com/aws/aws-sdk-go).
  This method reads the AWS shared credentials file for a specified profile.
  Unlike `AWS SDK Default` which also reads the shared credentials file, this option lets you specify a profile directly without environment variables.
  This option provides no fallback to other credential providers and fails if the file credentials are invalid.
- `Access & secret key` corresponds to the [StaticProvider](https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/#StaticProvider) and authenticates using a specified access key ID and secret key pair.
  This method doesn't provide fallback authentication and fails if the key pair is invalid. It's available in Grafana Cloud, though `Grafana Assume Role` is recommended there because it avoids managing long-term access keys.
- `Grafana Assume Role` - With this authentication method, Grafana Cloud users create an AWS IAM role that has a trust relationship with the Grafana AWS account. Grafana uses [STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) to generate temporary credentials on its behalf. This method eliminates the need to generate secret and access keys for users. Refer to [Use Grafana Assume Role](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/#use-grafana-assume-role) for more information.
- `Workspace IAM Role` corresponds to the [EC2RoleProvider](https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/ec2rolecreds/#EC2RoleProvider).
  The EC2RoleProvider retrieves credentials from a role attached to the EC2 instance running Grafana.
  While AWS SDK Default can achieve similar results, this option provides no fallback authentication. Amazon Managed Grafana enables this option by default.

## Assume a role

Specify an IAM role to assume in the **Assume Role ARN** field.

When you configure **Assume Role ARN**, Grafana uses the provided credentials to perform an [sts:AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) call. The primary authentication method only needs permission to assume the role, while the assumed role requires CloudWatch access permissions.

For example, you might use one set of long-term credentials across all AWS data sources but want to limit each data source's AWS access (such as separating staging and production data access). You could create separate credentials for each data source with distinct permissions, but this approach requires managing and rotating multiple secret and access keys across many data source instances.

Instead, assume role functionality lets you use one set of AWS credentials across all AWS data sources with a single permission: the ability to assume roles through STS. You then create separate IAM roles for each data source that specify temporary permissions. Since IAM roles are not credentials, they require no rotation and simplify management.

If the **Assume Role ARN** field is left empty, Grafana uses the provided credentials from the selected authentication method directly, and permissions to AWS data must be attached directly to those credentials. The **Assume Role ARN** field is optional for all authentication methods except for Grafana Assume Role.

To disable this feature in open source Grafana or Grafana Enterprise, refer to [`assume_role_enabled`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#assume_role_enabled).

### Use an external ID

{{< admonition type="note" >}}
You cannot use an external ID for the Grafana Assume Role authentication provider.
{{< /admonition >}}

To assume a role in another account created with an external ID, specify the external ID in the **External ID** field.

For more information, refer to the [AWS documentation on external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html).

## Use a custom endpoint

Use the **Endpoint** field to specify a custom endpoint URL that overrides the default AWS service API endpoint. Leave this field blank to use the default generated endpoint.

For more information about using service endpoints, refer to the [AWS service endpoints documentation](https://docs.aws.amazon.com/general/latest/gr/rande.html).

## Use an AWS credentials file

Create a file at `~/.aws/credentials`, the `HOME` path for the user running the `grafana-server` service.

{{< admonition type="note" >}}
If the credentials file appears to be in the correct location but isn't working, move your `aws` file to `/usr/share/grafana/` and set the credentials file permissions to `0644`.
{{< /admonition >}}

**Credentials file example:**

```bash
[default]
aws_access_key_id = asdsadasdasdasd
aws_secret_access_key = dasdasdsadasdasdasdsa
region = us-west-2
```

## Use EKS IAM roles for service accounts

EKS IAM roles for service accounts (IRSA) are an AWS EKS feature that allows pods to assume IAM roles without storing long-term credentials. When you configure IRSA in your EKS cluster, AWS injects temporary credentials into your pod as projected volume mounts.

In Grafana containers, the process runs as user `472` (`grafana`).
By default, Kubernetes mounts the projected credentials with permissions for the root user only.

To grant user `472` permission to access the credentials, and prevent fallback to the IAM role attached to the EC2 instance, set a [security context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for your pod.

**Security context example:**

```yaml
securityContext:
  fsGroup: 472
  runAsUser: 472
  runAsGroup: 472
```

## Use Grafana Assume Role

{{< admonition type="note" >}}
Grafana Assume Role is available only in Grafana Cloud. It's currently supported only for the Amazon CloudWatch and Amazon Athena data sources.
{{< /admonition >}}

The Grafana Assume Role authentication provider lets you access AWS without creating or managing long-term AWS IAM users or rotating access keys. Instead, you can create an IAM role that has permissions to access CloudWatch and trusts a Grafana AWS account.

The Grafana AWS account then makes a Security Token Service (STS) request to generate temporary credentials for your AWS data. This request includes an `externalID` unique to your Grafana Cloud account, which ensures users can access only their own AWS resources.

{{< admonition type="note" >}}
The trust policy names the shared Grafana AWS account as the trusted principal, but this doesn't grant every Grafana customer access to your role. The `sts:ExternalId` condition restricts the trust to your unique external ID, so AWS only allows an assume-role request that presents that exact value. Grafana sends your external ID only for your own data source requests. The external ID isn't a secret, but combined with the account-level trust it ensures one customer can't assume another customer's role. Keep the `sts:ExternalId` condition in the trust policy — removing it would let any account that Grafana trusts assume your role.
{{< /admonition >}}

For more information, refer to the [AWS documentation on external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html).

To use the Grafana Assume Role:

1. Create a new CloudWatch data source (or update an existing one) and select **Grafana Assume Role** as an authentication provider.
2. In the AWS Console, create a new IAM role, and under **Trusted entity type**, select **Another AWS account** as the trusted Entity.
3. Enter the Grafana account id (displayed in the instructions box on the **Settings** tab of the CloudWatch data source configuration) and check the **Require external ID** box.
4. Enter the external ID specified in the instructions box on the **Settings** tab of the CloudWatch data source configuration in Grafana. This external ID will be unique to your Grafana instance.
5. Attach any required permissions you would like Grafana to be able to access on your behalf (for example, CloudWatch Logs and CloudWatch Metrics policies).
6. Give the role a name and description, and click **Create role**.
7. Copy the ARN of the role you just created and paste it into the **Assume Role ARN** field on the **Settings** tab of CloudWatch data source configuration in Grafana.

**Sample Trust Relationship for an IAM role:**

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": {Grafana's AWS Account}
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": {External ID unique to your account}
                }
            }
        }
    ]
}
```

# Configure the Amazon CloudWatch data source

This document provides instructions for configuring the Amazon CloudWatch data source and explains available configuration options. For general information on adding and managing data sources, refer to [Data source management](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/).

## Before you begin

- You must have the `Organization administrator` role to configure the CloudWatch data source. Organization administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system.

- Grafana comes with a built-in CloudWatch data source plugin, so you do not need to install a plugin.

- Familiarize yourself with your CloudWatch security configuration and gather any necessary security certificates, client certificates, and client keys.

## Key concepts

If you're new to AWS, these terms are used throughout the configuration:

| Term                                  | Description                                                                                                                                                |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **IAM policy**                        | A JSON document attached to an identity that grants permissions to AWS API actions.                                                                        |
| **IAM role**                          | An identity with permissions that trusted entities can assume temporarily, instead of using long-lived keys.                                               |
| **Assume role**                       | An AWS mechanism that lets one identity temporarily take on the permissions of an IAM role, often used for cross-account access.                           |
| **External ID**                       | A shared value in a role's trust policy that prevents another party from assuming the role on your behalf.                                                 |
| **STS (Security Token Service)**      | The AWS service that issues the short-lived temporary credentials used when assuming a role.                                                               |
| **Cross-account observability (OAM)** | An AWS feature, managed through Observability Access Manager, that links monitoring and source accounts so you can query metrics and logs across accounts. |

## Add the CloudWatch data source

Complete the following steps to set up a new CloudWatch data source:

1. Click **Connections** in the left-side menu.
1. Click **Add new connection**
1. Type `CloudWatch` in the search bar.
1. Select the **CloudWatch data source**.
1. Click **Add new data source** in the upper right.

Grafana takes you to the **Settings** tab, where you will set up your CloudWatch configuration.

## Configure the data source in the UI

The following are configuration options for the CloudWatch data source.

| **Setting** | **Description**                                                                                                                            |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Name**    | The data source name. Sets the name you use to refer to the data source in panels and queries.                                             |
| **Default** | Toggle to select as the default name in dashboard panels. When you go to a dashboard panel, this will be the default selected data source. |

Grafana plugin requests to AWS are made on behalf of an AWS Identity and Access Management (IAM) role or IAM user.
The IAM user or IAM role must have the associated policies to perform certain API actions.

For authentication options and configuration details, refer to [AWS authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/).

| Setting                     | Description                                                                                                                                                                                                                  |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Authentication Provider** | Specify which AWS credentials chain to use. A Grafana plugin's requests to AWS are made on behalf of an IAM role or IAM user. The IAM user or IAM role must have the necessary policies to perform the required API actions. |

**Access & secret key:**

You must use both an access key ID and a secret access key to authenticate.

| Setting               | Description                  |
| --------------------- | ---------------------------- |
| **Access Key ID**     | Enter your key ID.           |
| **Secret Access Key** | Enter the secret access key. |

**Credentials file:**

When you select the **Credentials file** authentication provider, you can specify which profile to read from the AWS shared credentials file.

| Setting                      | Description                                                                                                               |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| **Credentials Profile Name** | The profile name in `~/.aws/credentials`, as specified in the credentials file. Leave blank to use the `default` profile. |

**Assume Role**:

| Setting             | Description                                                                                                                                          |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Assume Role ARN** | _Optional._ Specify the ARN of an IAM role to assume. This ensures the selected authentication method is used to assume the role, not used directly. |
| **External ID**     | If you're assuming a role in another AWS account that requires an external ID, specify it here.                                                      |

**Additional Settings:**

| Setting                          | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Endpoint**                     | _Optional_. Specify a custom endpoint for the AWS service.                                                                                                                                                                                                                                                                                                                                                                                                     |
| **Default Region**               | Specify the AWS region. Example: If the region is US West (Oregon), use `us-west-2`.                                                                                                                                                                                                                                                                                                                                                                           |
| **Namespaces of Custom Metrics** | Add one or more custom metric namespaces, separated by commas (for example,`Namespace1,Namespace2`). Grafana can't automatically load custom namespaces using the [CloudWatch GetMetricData API](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html). To make custom metrics available in the query editor, manually specify the namespaces in the `Namespaces of Custom Metrics` field in the data source configuration. |

**Proxy configuration:**

The **Proxy Configuration** settings appear only when per-data source HTTP proxy support is enabled with the `per_datasource_http_proxy_enabled` option in the Grafana configuration file. They let each CloudWatch data source use its own outbound HTTP proxy for AWS requests.

| Setting            | Description                                                                                                                                                                                                                                      |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Proxy Type**     | Select how the proxy is configured: `Environment (default)` uses the `HTTP_PROXY` and `HTTPS_PROXY` environment variables, `None` disables the proxy, and `URL` lets you specify a proxy URL. Don't set this when Secure Socks Proxy is enabled. |
| **Proxy URL**      | _Only when Proxy Type is `URL`._ The proxy URL, for example `https://localhost:3004`. Don't include the username or password in the URL.                                                                                                         |
| **Proxy Username** | _Only when Proxy Type is `URL`._ _Optional._ The proxy username.                                                                                                                                                                                 |
| **Proxy Password** | _Only when Proxy Type is `URL`._ _Optional._ The proxy password.                                                                                                                                                                                 |

**CloudWatch Logs**:

| Setting                  | Description                                                                                                                                                                                                                                                                                                                          |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Query Result Timeout** | Grafana polls CloudWatch Logs every second until AWS returns a `Done` status or the timeout is reached. An error is returned if the timeout is exceeded. For alerting, the timeout defined in the Grafana configuration file takes precedence. Enter a valid duration string, such as `30m`, `30s` or `200ms`. The default is `30m`. |
| **Default Log Groups**   | _Optional_. Specify the default log groups for CloudWatch Logs queries.                                                                                                                                                                                                                                                              |

**Application Signals trace link:**

| Setting         | Description                                                         |
| --------------- | ------------------------------------------------------------------- |
| **Data source** | Select the Application Signals data source from the drop-down menu. |

Grafana automatically creates a link to a trace in Application Signals data source if logs contain the `@xrayTraceId` field. To use this feature, you must already have an Application Signals data source configured. For details, refer to the [Application Signals data source docs](https://grafana.com/docs/plugins/grafana-x-ray-datasource/latest/). To view the Application Signals link, select the log row in either the Explore view or dashboard [Logs panel](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/logs/) to view the log details section.

To log the `@xrayTraceId`, refer to the [AWS Application Signals documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html). To provide the field to Grafana, your log queries must also contain the `@xrayTraceId` field, for example by using the query `fields @message, @xrayTraceId`.

**Private data source connect** - _Only for Grafana Cloud users._

| Setting                         | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Private data source connect** | Establishes a private, secured connection between a Grafana Cloud stack and data sources within a private network. Use the drop-down to locate the PDC URL. For setup instructions, refer to [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) and [Configure PDC](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc). Click **Manage private data source connect** to open your PDC connection page and view your configuration details. |

After configuring your Amazon CloudWatch data source options, click **Save & test** at the bottom to test the connection. When the test succeeds, Grafana confirms that it reached both CloudWatch APIs:

```
1. Successfully queried the CloudWatch metrics API.
2. Successfully queried the CloudWatch logs API.
```

{{< figure src="/media/docs/CloudWatch/CloudWatch-config-success-message.png" >}}

{{< admonition type="note" >}}
To troubleshoot issues while setting up the CloudWatch data source, check the `/var/log/grafana/grafana.log` file. Common issues include invalid credentials, missing regions and metrics-only credentials.
{{< /admonition >}}

### IAM policy examples

To read CloudWatch metrics and EC2 tags, instances, regions, and alarms, you must grant Grafana permissions via IAM.
You can attach these permissions to the IAM role or IAM user you configured in [AWS authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/).

**Metrics-only permissions:**

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "CloudWatch:DescribeAlarmsForMetric",
        "CloudWatch:DescribeAlarmHistory",
        "CloudWatch:DescribeAlarms",
        "CloudWatch:ListMetrics",
        "CloudWatch:GetMetricData",
        "CloudWatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourceMetricsFromPerformanceInsights",
      "Effect": "Allow",
      "Action": "pi:GetResourceMetrics",
      "Resource": "*"
    }
  ]
}
```

**Logs-only permissions:**

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:ListAggregateLogGroupSummaries",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}
```

**Metrics and logs permissions:**

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "CloudWatch:DescribeAlarmsForMetric",
        "CloudWatch:DescribeAlarmHistory",
        "CloudWatch:DescribeAlarms",
        "CloudWatch:ListMetrics",
        "CloudWatch:GetMetricData",
        "CloudWatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourceMetricsFromPerformanceInsights",
      "Effect": "Allow",
      "Action": "pi:GetResourceMetrics",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:ListAggregateLogGroupSummaries",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}
```

#### Cross-account observability permissions

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["oam:ListSinks", "oam:ListAttachedLinks"],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
```

{{< admonition type="note" >}}
Cross-account observability lets you retrieve metrics and logs across different accounts in a single region, but you can't query EC2 Instance Attributes across accounts because those come from the EC2 API and not the CloudWatch API.
{{< /admonition >}}

For more information on configuring authentication, refer to [Configure AWS authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/).

### Permissions reference

The following table explains why each permission is needed and whether it's required, so you can scope a policy to only the features you use. Grant `cloudwatch:GetMetricData` only for metrics, the `logs:` actions only for logs, and the EC2 and tag actions only if you use the related template variables. Granting a permission that you don't use has no effect on functionality.

| Permission                                                                                           | Purpose                                                                                                | Required                                                              |
| ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- |
| `cloudwatch:ListMetrics`                                                                             | Lists available metrics and populates the namespace, metric, and dimension fields in the query editor. | Required for metrics queries and metrics template variables.          |
| `cloudwatch:GetMetricData`                                                                           | Retrieves metric data points for queries and metric math expressions.                                  | Required for metrics queries.                                         |
| `cloudwatch:GetInsightRuleReport`                                                                    | Retrieves data from CloudWatch Contributor Insights rules.                                             | Optional. Only needed if you query Contributor Insights rules.        |
| `cloudwatch:DescribeAlarms`, `cloudwatch:DescribeAlarmsForMetric`, `cloudwatch:DescribeAlarmHistory` | Reads CloudWatch alarm configuration and history.                                                      | Optional. Only needed if you display alarm data.                      |
| `pi:GetResourceMetrics`                                                                              | Retrieves Performance Insights metrics for supported resources, such as RDS databases.                 | Optional. Only needed if you query Performance Insights metrics.      |
| `logs:DescribeLogGroups`                                                                             | Lists log groups and populates the log group selector.                                                 | Required for logs queries and the Log Groups template variable.       |
| `logs:StartQuery`, `logs:StopQuery`, `logs:GetQueryResults`                                          | Runs CloudWatch Logs Insights queries and retrieves their results.                                     | Required for logs queries.                                            |
| `logs:GetLogGroupFields`                                                                             | Retrieves the fields available in a log group for autocomplete.                                        | Optional. Improves the logs query editor experience.                  |
| `logs:GetLogEvents`                                                                                  | Retrieves individual log events.                                                                       | Optional. Only needed for queries that read raw log events.           |
| `logs:Unmask`                                                                                        | Reveals data masked by a CloudWatch Logs data protection policy.                                       | Optional. Only needed to view masked sensitive data.                  |
| `ec2:DescribeRegions`                                                                                | Lists available AWS regions.                                                                           | Optional. Only needed if you rely on dynamic region discovery.        |
| `ec2:DescribeInstances`, `ec2:DescribeTags`                                                          | Resolves EC2 instance attributes and tags for dimension and EC2 template variables.                    | Optional. Only needed for EC2 Instance Attributes template variables. |
| `ec2:DescribeVolumes`                                                                                | Resolves EBS volume IDs for the EBS Volume IDs template variable.                                      | Optional. Only needed for EBS Volume IDs template variables.          |
| `tag:GetResources`                                                                                   | Resolves resource ARNs for the Resource ARNs template variable.                                        | Optional. Only needed for Resource ARNs template variables.           |
| `oam:ListSinks`, `oam:ListAttachedLinks`                                                             | Discovers linked accounts for CloudWatch cross-account observability.                                  | Optional. Only needed for cross-account observability.                |

When you use an **Assume Role ARN**, attach these query permissions to the assumed role. The primary credentials only need permission to perform `sts:AssumeRole`. For details, refer to [Assume a role](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/#assume-a-role).

### CloudWatch Logs data protection

CloudWatch Logs can protect data by applying log group data protection policies. When data protection is enabled for a log group, any sensitive data that matches the identifiers you select is automatically masked. To view masked data, your IAM role or user must have the `logs:Unmask` permission. For more details, refer to [the AWS guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html) on masking sensitive log data.

### Configure the data source with grafana.ini

The Grafana [configuration file](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#aws) includes an `AWS` section where you can configure data source options:

| Configuration option                | Description                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `allowed_auth_providers`            | Specifies which authentication providers are allowed for the CloudWatch data source as a comma-separated list. The default is `default,keys,credentials`: `default` (AWS SDK Default), `keys` (Access and secret key), and `credentials` (Credentials file). The `ec2_iam_role` (EC2 IAM role) provider is also available but isn't enabled by default.                                                                                         |
| `assume_role_enabled`               | Allows you to disable `assume role (ARN)` in the CloudWatch data source. The assume role (ARN) is enabled by default in open source Grafana.                                                                                                                                                                                                                                                                                                    |
| `per_datasource_http_proxy_enabled` | Allows each CloudWatch data source instance to use its own HTTP proxy configuration for requests to AWS, instead of a shared proxy. Disabled by default. Set to `true` to enable.                                                                                                                                                                                                                                                               |
| `list_metrics_page_limit`           | Sets the limit of List Metrics API pages. When a custom namespace is specified in the query editor, the [List Metrics API](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_ListMetrics.html) populates the _Metrics_ field and _Dimension_ fields. The API is paginated and returns up to 500 results per page, and the data source also limits the number of pages to 500 by default. This setting customizes that limit. |

### Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system.
For more information about provisioning and available configuration options, refer to [Provision Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

**Using AWS SDK (default)**:

```yaml
apiVersion: 1
datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: default
      defaultRegion: eu-west-2
```

**Using credentials' profile name (non-default)**:

```yaml
apiVersion: 1

datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: credentials
      defaultRegion: eu-west-2
      customMetricsNamespaces: 'CWAgent,CustomNameSpace'
      profile: secondary
```

**Using `accessKey` and `secretKey`**:

```yaml
apiVersion: 1

datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: keys
      defaultRegion: eu-west-2
    secureJsonData:
      accessKey: '<your access key>'
      secretKey: '<your secret key>'
```

**Using AWS SDK Default and ARN of IAM Role to Assume:**

```yaml
apiVersion: 1
datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: default
      assumeRoleArn: arn:aws:iam::123456789012:root
      defaultRegion: eu-west-2
```

### Configure with Terraform

You can configure the CloudWatch data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

For more information about provisioning resources with Terraform, refer to [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/).

The following example uses the **Access and secret key** authentication method:

```hcl
resource "grafana_data_source" "cloudwatch" {
  name = "CloudWatch"
  type = "cloudwatch"

  json_data_encoded = jsonencode({
    authType      = "keys"
    defaultRegion = "eu-west-2"
  })

  secure_json_data_encoded = jsonencode({
    accessKey = "<YOUR_ACCESS_KEY>"
    secretKey = "<YOUR_SECRET_KEY>"
  })
}
```

The following example uses the **AWS SDK Default** authentication method with an assumed role:

```hcl
resource "grafana_data_source" "cloudwatch" {
  name = "CloudWatch"
  type = "cloudwatch"

  json_data_encoded = jsonencode({
    authType      = "default"
    assumeRoleArn = "arn:aws:iam::123456789012:role/<ROLE_NAME>"
    defaultRegion = "eu-west-2"
  })
}
```

Replace the placeholders with your own values:

- `<YOUR_ACCESS_KEY>` and `<YOUR_SECRET_KEY>`: The AWS access key ID and secret access key for the **Access and secret key** method.
- `<ROLE_NAME>`: The name of the IAM role to assume.

For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source).

# Amazon CloudWatch query editor

Grafana provides a query editor for the CloudWatch data source, which allows you to query, visualize, and alert on logs and metrics stored in Amazon CloudWatch. It is located on the [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/) page. For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/).

## Key concepts

If you're new to CloudWatch, these terms are used throughout this documentation:

| Term                  | Description                                                                                                   |
| --------------------- | ------------------------------------------------------------------------------------------------------------- |
| **Namespace**         | A container for CloudWatch metrics, usually one per AWS service, such as `AWS/EC2` or `AWS/Lambda`.           |
| **Metric**            | A time-ordered set of data points published to CloudWatch, such as `CPUUtilization`.                          |
| **Dimension**         | A name-value pair that identifies a specific resource for a metric, such as `InstanceId=i-1234567890abcdef0`. |
| **Statistic**         | An aggregation applied to metric data over a period, such as `Average`, `Sum`, or `Maximum`.                  |
| **Period**            | The length of time, in seconds, used to aggregate metric data points.                                         |
| **Metric math**       | Expressions that combine or transform metrics, such as computing a rate or a sum across metrics.              |
| **Metrics Insights**  | A SQL-like query engine for filtering and aggregating metrics in near real time.                              |
| **Search expression** | A CloudWatch expression that matches multiple metrics at query time, often used with wildcards.               |
| **Logs Insights QL**  | The CloudWatch Logs query language used to search and aggregate log data.                                     |

## Choose a query editing mode

The CloudWatch data source can query data from both CloudWatch metrics and CloudWatch Logs APIs, each with its own specialized query editor.

- [CloudWatch Metrics](#query-cloudwatch-metrics)
- [CloudWatch Logs](#query-cloudwatch-logs)

Select the API to query using the drop-down to the right of the **Region** setting.

## CloudWatch Metrics query editor components

The following are the components of the CloudWatch query editor.

| **Setting**     | **Description**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Region**      | Select an AWS region if it differs from the default.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| **Namespace**   | The AWS service namespace. Examples: `AWS/EC2`, `AWS/Lambda`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| **Metric name** | The name of the metric you want to visualize. Example: `CPUUtilization`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| **Statistic**   | Choose how to aggregate your data. Options are `Average`, `Maximum`, `Minimum`, `Sum`, `SampleCount`, and `IQM`. You can also enter a custom value, such as a percentile like `p99`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| **Dimensions**  | Select dimensions from the drop-down. Examples: `InstanceId`, `FunctionName`, `latency`. You can add several dimensions to your query.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| **Match exact** | _Optional_. When enabled, this option restricts query results to metrics that precisely match the specified dimensions and their values. All dimensions of the queried metric must be explicitly defined in the query to ensure an exact schema match. If disabled, the query will also return metrics that match the defined schema but possess additional dimensions.                                                                                                                                                                                                                                                                                                                   |
| **ID**          | _Optional_. Unique identifier required by the GetMetricData API for referencing queries in math expressions. Must start with a lowercase letter and can include letters, numbers, and underscores. If not specified, Grafana generates an ID using the pattern `query[refId]` (for example, `queryA`for the first query row).                                                                                                                                                                                                                                                                                                                                                             |
| **Period**      | The minimum time interval, in seconds, between data points. The default is `auto`. CloudWatch accepts periods of 1, 5, 10, 30, or any multiple of 60 seconds. When set to `auto` or left blank, Grafana calculates the period by dividing the time range in seconds by 2000, then rounding up to the next value among 60, 300, 900, 3600, 21600, and 86400. The available periods depend on the age of the data: because CloudWatch stores older metrics at a lower resolution under its [retention policy](https://aws.amazon.com/about-aws/whats-new/2016/11/cloudwatch-extends-metrics-retention-and-new-user-interface/), the shorter periods aren't available for older time ranges. |
| **Label**       | _Optional_. Add a customized time series legend name. The label field overrides the default metric legend name using CloudWatch dynamic labels. Time-based dynamic labels like ${MIN_MAX_TIME_RANGE} derive legend values from the current timezone in the time range picker. For the full list of label patterns and limitations, refer to [CloudWatch dynamic labels](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/graph-dynamic-labels.html).                                                                                                                                                                                                                        |

## Use Builder and Code modes

Both the Metric Search and Metric Insights query types provide two editing modes. Use the **Builder** and **Code** toggle in the query editor to switch between them.

### Builder mode

Builder mode lets you construct a query from drop-downs and form fields without writing any syntax:

- For **Metric Search** queries, select the namespace, metric name, statistic, and dimensions.
- For **Metric Insights** queries, select the namespace, metric name, filter, group, and order options using the [Metrics Insights keywords](#use-metrics-insights-keywords). Grafana constructs a SQL query based on your selections.

### Code mode

Code mode lets you write your query directly in a code editor:

- For **Metric Search** queries, write a metric math or `SEARCH` expression.
- For **Metric Insights** queries, write a Metrics Insights SQL query.

The code editor includes a built-in autocomplete feature that suggests keywords, aggregations, namespaces, metrics, labels, and label values.
Suggestions appear after typing a space, comma, or dollar (`$`) character, or by pressing <key>CTRL</key>+<key>Space</key>.

{{< admonition type="note" >}}
Template variables in the code editor can interfere with autocompletion.
{{< /admonition >}}

To run the query, click **Run query** above the code editor.

## Query CloudWatch metrics

You can create two types of queries in the CloudWatch query editor:

- [Metric Search](#metric-search-queries), which helps you retrieve and filter available metrics.
- [Metric Insights](#use-metric-insights-syntax), which uses the Metrics Insights feature to fetch time series data.

The query type you use depends on how you want to interact with AWS metrics. Use the drop-down in the upper middle of the query editor to select which type you want to create.

### Metric Search queries

Metric search queries help you discover and filter available metrics. These queries use wildcards and filters to find metrics without needing to know exact metric names.

A valid metric query requires a specified namespace, metric name, and at least one statistic. Dimensions are optional, but if included, you must provide both a `key` and a `value`.

The `Match Exact` option controls how dimension filtering is applied to metric queries. When you enable `match exact`, the query returns only metrics whose dimensions precisely match the specified criteria.

This requires the following:

- All dimensions present on the target metric must be explicitly specified.
- Dimensions you don't want to filter by must use a wildcard (\*) filter.
- The metric schema must match exactly as defined in the [CloudWatch metric schema](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/search-expression-syntax.html) documentation.

When `Match Exact` is disabled, you can specify any subset of dimensions for filtering. The query returns metrics that:

- Match the specified namespace and metric name.
- Match all defined dimension filters.
- May contain additional dimensions beyond those specified.

This mode provides more flexible querying but may return metrics with unexpected additional dimensions.

Because the data source retrieves metrics through the CloudWatch `GetMetricData` API, a single query returns data for up to 100 metrics matching your filter criteria, which is the per-request limit that AWS enforces.

Enhance metric queries using template variables to create dynamic, reusable dashboards.

#### Create dynamic queries with dimension wildcards

Use the asterisk (`*`) wildcard for dimension values to create dynamic queries that automatically monitor changing sets of AWS resources.

{{< figure src="/static/img/docs/cloudwatch/cloudwatch-dimension-wildcard-8.3.0.png" max-width="500px" caption="CloudWatch dimension wildcard" >}}

The query returns the average CPU utilization for all EC2 instances in the default region. With `Match Exact` disabled and `InstanceId` using a wildcard, the query retrieves metrics for any EC2 instance regardless of additional dimensions.

Auto-scaling events add new instances to the graph without manual instance ID tracking. As with other metric search queries, AWS returns up to 100 metrics per `GetMetricData` request.

Click the [**Query inspector**](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/#navigate-the-query-editor) button and select **Meta Data** to see the search expression that's automatically built to support wildcards.

To learn more about search expressions, refer to the [CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/search-expression-syntax.html).
The search expression is defined by default in such a way that the queried metrics must match the defined dimension names exactly.
This means that in the example, the query returns only metrics with exactly one dimension containing the name `InstanceId`.

{{< figure src="/static/img/docs/cloudwatch/cloudwatch-meta-inspector-8.3.0.png" max-width="500px" class="docs-image--right" caption="CloudWatch Meta Inspector" >}}

Disabling `Match Exact` includes metrics with additional dimensions and creates a search expression even without wildcards. Grafana searches for any metric matching at least the namespace, metric name, and all defined dimensions.

#### Use multi-value template variables

When defining dimension values based on multi-valued template variables, the data source uses a search expression to query for the matching metrics.
This enables the use of multiple template variables in one query, and also lets you use template variables for queries that have the `Match Exact` option disabled.

Search expressions are limited to 1,024 characters, so your query might fail if you have a long list of values.
We recommend using the asterisk (`*`) wildcard instead of the `All` option to query all metrics that have any value for a certain dimension name.

Multi-valued template variables are supported only for dimension values.
Using multi-valued template variables for `Region`, `Namespace`, or `Metric Name` is not supported.

#### Use metric math expressions

Create new time series metrics using mathematical functions on CloudWatch metrics. This supports arithmetic operators, unary subtraction, and other functions. For available functions, refer to [AWS Metric Math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html).

To apply arithmetic operations, assign a unique string ID to the raw metric, then reference this ID in the `Expression` field of the new metric.

{{< admonition type="note" >}}
If you use the expression field to reference another query, such as `queryA * 2`, you can't create an alert rule based on that query.
{{< /admonition >}}

### Query metrics across AWS monitoring accounts

When you select `Builder` mode within the Metric search editor, a new Account field is displayed. Use the `Account` field to specify which of the linked monitoring accounts to target for the given query. By default, the `All` option is specified, which will target all linked accounts.

While in `Code` mode, you can specify any math expression. If the Monitoring account badge displays in the query editor header, all `SEARCH` expressions entered in this field will be cross-account by default and can query metrics from linked accounts. Note that while queries run cross-account, the autocomplete feature currently doesn't fetch cross-account resources, so you'll need to manually specify resource names when writing cross-account queries.
You can limit the search to one or a set of accounts, as documented in the [AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html).

### Period macro

If you use a CloudWatch [`SEARCH`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/search-expression-syntax.html) expression, consider using the `$__period_auto` macro rather than specifying a period explicitly. The `$__period_auto` macro will resolve to a [CloudWatch period](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html) that is suitable for the chosen time range.

### Deep-link Grafana panels to the CloudWatch console

Left-clicking a time series in the panel displays a context menu with a link to `View in CloudWatch console`.
Clicking the link opens a new tab that takes you to the CloudWatch console and displays all metrics for that query.
If you're not logged in to the CloudWatch console, the link forwards you to the login page.
The link provided is valid for any account but displays the expected metrics only if you're logged in to the account that corresponds to the selected data source in Grafana.

{{< figure src="/media/docs/cloudwatch/cloudwatch-deep-link-v12.1.png" caption="CloudWatch deep linking" >}}

This feature is not available for metrics based on [metric math expressions](#use-metric-math-expressions).

### Use Metric Insights syntax

Metric Insights uses a dialect of SQL and this query syntax:

```sql
SELECT FUNCTION(MetricName)
FROM Namespace | SCHEMA(...)
[ WHERE labelKey OPERATOR labelValue [AND|...]]
[ GROUP BY labelKey [, ...]]
[ ORDER BY FUNCTION() [DESC | ASC] ]
[ LIMIT number]
```

For details about the Metrics Insights syntax, refer to the [AWS reference documentation](https://docs.aws.amazon.com/console/cloudwatch/metricsinsights-syntax).

For information about Metrics Insights limits, refer to the [AWS feature documentation](https://docs.aws.amazon.com/console/cloudwatch/metricsinsights).

You can also augment queries by using [template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/).

### Use Metrics Insights keywords

This table summarizes common Metrics Insights query keywords:

| Keyword      | Description                                                                                                                                                                                                  |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `FUNCTION`   | Required. Specifies the aggregate function to use, and also specifies the name of the metric to be queried. Valid values are AVG, COUNT, MAX, MIN, and SUM.                                                  |
| `MetricName` | Required. For example, `CPUUtilization`.                                                                                                                                                                     |
| `FROM`       | Required. Specifies the metric's source. You can specify either the metric namespace that contains the metric to be queried, or a SCHEMA table function. Namespace examples include `AWS/EC2`, `AWS/Lambda`. |
| `SCHEMA`     | Optional. Narrows the query results to only the metrics that are an exact match, or to metrics that do not match.                                                                                            |
| `WHERE`      | Optional. Filters the query results to only the metrics that match your specified expression. For example, `WHERE InstanceType != 'c3.4xlarge'`.                                                             |
| `GROUP BY`   | Optional. Groups the query results into multiple time series. For example, `GROUP BY ServiceName`.                                                                                                           |
| `ORDER BY`   | Optional. Specifies the order in which time series are returned. Options are `ASC`, `DESC`.                                                                                                                  |
| `LIMIT`      | Optional. Limits the number of time series returned.                                                                                                                                                         |

## Query CloudWatch Logs

The logs query editor helps you write CloudWatch Logs queries across a selected region and a set of log groups or Amazon CloudWatch Logs data sources.

You can query CloudWatch Logs using three supported query language options:

- **Logs Insights QL** - The AWS native query language specifically designed for CloudWatch Logs. It uses a SQL-like syntax with commands like `fields`, `filter`, `stats`, and `sort`. It's optimized for the CloudWatch log structure and offers built-in functions for parsing timestamps, extracting fields from JSON logs, and performing aggregations.
- **OpenSearch SQL** - OpenSearch SQL is a query language that uses a SQL-like syntax for querying data in OpenSearch. It supports standard SQL queries and is designed for users familiar with SQL.
- **OpenSearch PPL** - The OpenSearch query language is based on Elasticsearch's query DSL (Domain Specific Language). It uses a pipe-based syntax similar to Unix command-line tools or the Splunk search language, and supports complex boolean logic, range queries, wildcard matching, and full-text search capabilities.

**Create a CloudWatch Logs query:**

1. Select a region.
1. Select **CloudWatch Logs** from the query type drop-down.
1. Use the Logs Mode selector to choose between Logs Insights and Log Anomalies queries.

**Log Anomalies**

[Anomaly detection](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/LogsAnomalyDetection.html) uses machine-learning and pattern recognition to establish baselines of typical log content.  
The Log Anomalies query editor fetches the list of anomalies detected in your CloudWatch service. In order to query log anomalies in the editor, a log anomaly detector must be created in the AWS CloudWatch console first.
The log trend cell shows the number of occurrences of the pattern over the selected query time range.
The table shows 50 log anomalies at a time. If you would like to narrow down the list, you can filter anomalies by their ARN and suppressed state.

In addition to this, you can use the Logs Insights QL editor and the `anomaly` command together with the `patterns` command to define and display log anomalies in real time. See the [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/LogsAnomalyDetection-Insights.html) documentation for more info.

**Logs Insights**

The **Query language** drop-down is available only when **Logs Mode** is set to Logs Insights.

1. Select the query language you want to use in the **Query language** drop-down.
1. Specify the log groups, CloudWatch Logs data sources, or both that you want to query.
1. Use the main input area to write your logs query. Amazon CloudWatch only supports a subset of OpenSearch SQL and PPL commands. To find out more about the syntax supported, consult [Amazon CloudWatch Logs documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_Languages.html)

### Specify log groups to query

To target log groups in your query, use one of the following methods:

- **Log group name**: Select specific log groups. This is the default behavior. You can select up to 50 log groups.
  Click **Select log groups** to choose the target log groups for your query.
  When the **Monitoring account** badge appears in the query editor header, you can search and select log groups across multiple accounts.
  Use the **Accounts** filter to narrow results by account.
  For large lists, use prefix search to narrow the selection. The selector loads log groups in pages, so you can scroll to load more results when many log groups are available.
- **Name prefix**: Select the **Name prefix** query scope and provide up to five log group name prefixes. The query will run against log groups that have names that start with the specified prefixes. Each prefix must be at least three characters and must not include `*`. This option is only available for **Logs Insights QL**.
- **All log groups**: Select the **All log groups** query scope. This queries all log groups in the selected region. This option is only available for **Logs Insights QL**.

You can also apply optional filters when specifying log groups with the **Name prefix** or **All log groups** query scopes:

- **Class**: Filter by [log group class](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch_Logs_Log_Classes.html). Choose **Standard** or **Infrequent Access**.
- **Accounts**: Filter by AWS account. This option is only available when the **Monitoring account** badge appears in the query editor header. You can select up to 20 accounts.

### Specify Amazon CloudWatch Logs data sources to query

Amazon CloudWatch Logs data sources categorize logs by the service or application that generates them and by the type of log. They complement log groups and let you query logs without knowing individual log group names. For details about how Amazon CloudWatch categorizes logs, refer to [Data source discovery and management](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/data-source-discovery-management.html) in the AWS documentation.

Before you use the data source selector, ensure that the IAM role or IAM user for the Grafana data source has the `logs:ListAggregateLogGroupSummaries` permission. For the complete IAM policy examples, refer to [IAM policy examples](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/#iam-policy-examples).

To query logs by CloudWatch Logs data source, follow these steps:

1. Click **Select data sources**.
1. Search for a data source name or type.
1. Select up to 10 data sources.
1. Click **Apply selection**.

You can combine CloudWatch Logs data source selections with specific log groups or with the **Name prefix** and **All log groups** query scopes. When the **Monitoring account** badge appears, the selector includes data sources from the monitoring account and its linked source accounts.

The CloudWatch data source automatically adds the selected CloudWatch Logs data sources to Logs Insights QL and OpenSearch PPL queries.

{{< admonition type="note" >}}
When you use **Logs Insights QL**, **OpenSearch PPL**, or **OpenSearch SQL**, you must specify a region and define the logs to query by selecting log groups, selecting CloudWatch Logs data sources, or choosing a supported log group query scope.

In **OpenSearch SQL**, you can specify log groups in multiple ways. For details, refer to the [Query log groups and Amazon CloudWatch Logs data sources with OpenSearch SQL](#query-log-groups-with-opensearch-sql) section.
{{< /admonition >}}

Click **View in CloudWatch console** to interactively view, search, and analyze your log data in the CloudWatch Logs Insights console. If you're not logged in to the CloudWatch console, the link forwards you to the login page.

### Query log groups and Amazon CloudWatch Logs data sources with OpenSearch SQL {#query-log-groups-with-opensearch-sql}

When querying log groups and Amazon CloudWatch Logs data sources with OpenSearch SQL, you can use the `$__source` macro to automatically reference the log groups and CloudWatch Logs data sources selected in the query editor. This is the recommended approach as it allows you to manage both types of selection through the UI.

```sql
SELECT window.start, COUNT(*) AS exceptionCount
FROM `$__source`
WHERE `@message` LIKE '%Exception%'
```

The `$__source` macro expands to the proper syntax for the log groups and CloudWatch Logs data sources you've selected in the UI.

Alternatively, you can manually specify a single log group directly in the `FROM` clause:

```sql
SELECT window.start, COUNT(*) AS exceptionCount
FROM `log_group`
WHERE `@message` LIKE '%Exception%'
```

When querying multiple log groups you **must** use the `logGroups(logGroupIdentifier: [...])` syntax:

```sql
SELECT window.start, COUNT(*) AS exceptionCount
FROM `logGroups(logGroupIdentifier: ['LogGroup1', 'LogGroup2'])`
WHERE `@message` LIKE '%Exception%'
```

To reference log groups in a monitoring account, use ARNs instead of LogGroup names.

You can also write queries returning time series data by using the [`stats` command](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_Insights-Visualizing-Log-Data.html).
When making `stats` queries in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/), ensure you are in Metrics Explore mode.

### Create queries for alerting

Alerting requires queries that return numeric data, which CloudWatch Logs supports.
For example, you can enable alerts through the use of the `stats` command.

The following is a valid query for alerting on messages that include the text "Exception":

```
filter @message like /Exception/
    | stats count(*) as exceptionCount by bin(1h)
    | sort exceptionCount desc
```

{{< admonition type="note" >}}
If you receive an error like `input data must be a wide series but got ...` when trying to alert on a query, make sure that your query returns valid numeric data that can be output to a Time series panel.
{{< /admonition >}}

For more information on Grafana alerts, refer to [Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/).

## Common use cases

The following scenarios show how to combine the query editor's features to solve real monitoring problems.

- **Monitor an EC2 fleet:** Use a Metric Search query for `AWS/EC2` `CPUUtilization` with the `InstanceId` dimension set to a wildcard (`*`) and `Match Exact` disabled. The graph automatically includes new instances as your fleet scales.
- **Track Lambda errors and invocations:** Query the `AWS/Lambda` namespace for the `Errors` and `Invocations` metrics, then add a metric math expression such as `errors / invocations * 100` to chart an error rate.
- **Rank the busiest resources:** Use a Metrics Insights query with `ORDER BY` and `LIMIT` to return the top N resources by a metric, for example, the 10 EC2 instances with the highest average CPU utilization.
- **Investigate application errors:** Use a CloudWatch Logs query with `filter` and `stats` to count errors over time, then correlate the spikes with metric panels on the same dashboard.
- **Alert on log error rates:** Build a Logs query that returns numeric data with the `stats` command, then create an alert rule on the result. Refer to [Create queries for alerting](#create-queries-for-alerting).

## Cross-account observability

The CloudWatch plugin monitors and troubleshoots applications that span multiple accounts within a region. Cross-account observability enables seamless searching, visualization, and analysis of metrics and logs across account boundaries.

To enable cross-account observability, complete the following steps:

1. Go to the [Amazon CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html) and follow the instructions for enabling cross-account observability.

1. Add [two API actions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/#cross-account-observability-permissions) to the IAM policy attached to the role/user running the plugin.

Cross-account querying is available in the plugin through the **Logs**, **Metric search**, and **Metric Insights** modes.
After you have configured it, you'll see a **Monitoring account** badge in the query editor header.

{{< figure src="/static/img/docs/cloudwatch/cloudwatch-monitoring-badge-9.3.0.png" max-width="1200px" caption="Monitoring account badge" >}}

## Query caching

When you enable [query and resource caching](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/#query-and-resource-caching), Grafana temporarily stores the results of data source queries and resource requests. Query caching is available in CloudWatch Metrics in Grafana Cloud and Grafana Enterprise. It is not available in CloudWatch Logs Insights due to how query results are polled from AWS.

# CloudWatch template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
Grafana lists these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables.

For an introduction to templating and template variables, refer to [Templating](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/).

## Use query variables

You can specify these CloudWatch data source queries in the Variable edit view's **Query type** field.
Use them to fill a variable's options list with values like `regions`, `namespaces`, `metric names`, and `dimension keys/values`.

| Name                        | List returned                                                                                                                                    |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Regions**                 | All AWS regions.                                                                                                                                 |
| **Namespaces**              | All namespaces CloudWatch supports.                                                                                                              |
| **Metrics**                 | Metrics in the namespace. (Specify region or use "default" for custom metrics.)                                                                  |
| **Dimension Keys**          | Dimension keys in the namespace.                                                                                                                 |
| **Dimension Values**        | Dimension values matching the specified `region`, `namespace`, `metric`, and `dimension key`. Use dimension `filters` for more specific results. |
| **EBS Volume IDs**          | Volume IDs matching the specified `region` and `instance ID`.                                                                                    |
| **EC2 Instance Attributes** | Attributes matching the specified `region`, `attribute name`, and `filters`.                                                                     |
| **Resource ARNs**           | ARNs matching the specified `region`, `resource type`, and `tags`.                                                                               |
| **Statistics**              | All standard statistics.                                                                                                                         |
| **Log Groups**              | Log groups matching the specified `region` and optional log group `prefix`. The variable value is the log group ARN.                             |
| **Accounts**                | The IDs of linked monitoring accounts for the specified `region`. Available only when cross-account observability is enabled.                    |

When cross-account observability is enabled, the **Metrics**, **Dimension Keys**, **Dimension Values**, and **Log Groups** query types also show an optional **Account** field. Use it to target a specific linked monitoring account, or leave it unset to query all linked accounts. For more information, refer to [Cross-account observability](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/query-editor/#cross-account-observability).

For details on the available dimensions, refer to the [CloudWatch Metrics and Dimensions Reference](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CW_Support_For_AWS.html).

For details about the metrics CloudWatch provides, refer to the [CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/CW_Support_For_AWS.html).

### Use variables in queries

Use the Grafana variable syntax to include variables in queries. A query variable dynamically retrieves values from your data source using a query.
For details, refer to the [variable syntax documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/).

CloudWatch metric queries support variables in the **Region**, **Namespace**, **Metric name**, dimension key, and dimension value fields.

For example, if you create a variable named `region` with the **Regions** query type and a variable named `instance` with the **EC2 Instance Attributes** query type, you can reference them in a metric query by setting the query editor **Region** to `$region` and a dimension value to `$instance`. The dashboard then updates as you change the selected region and instance.

## Query EC2 instance attributes

The **EC2 Instance Attributes** query type retrieves instance metadata using the AWS [`ec2:DescribeInstances`](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html) API. Configure it with the following fields in the variable editor:

- **Region:** The AWS region to query.
- **Attribute name:** The instance attribute to return for each matching instance.
- **Filters:** _Optional._ Filters that limit which instances are returned.

{{< admonition type="note" >}}
The earlier `ec2_instance_attribute(region, attributeName, filters)` query syntax is still supported. Grafana automatically migrates existing variables that use it to the EC2 Instance Attributes query type.
{{< /admonition >}}

### Filters

Each filter has a key and a comma-separated list of matching values. Use the [filters supported by `ec2:DescribeInstances`](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html), such as instance properties. To filter by tag, use a key in the form `tag:<name>`, for example `tag:Environment`.

### Select attributes

A query returns one attribute per instance. You can select any attribute that has a single value and isn't an object or array, also known as a flat attribute. These attributes mirror the fields of the AWS EC2 [`Instance`](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_Instance.html) object, where you can find a definition of each one. Because the available attributes track the AWS API, the following list might not be exhaustive:

- `AmiLaunchIndex`
- `Architecture`
- `ClientToken`
- `EbsOptimized`
- `EnaSupport`
- `Hypervisor`
- `ImageId`
- `InstanceId`
- `InstanceLifecycle`
- `InstanceType`
- `KernelId`
- `KeyName`
- `LaunchTime`
- `Platform`
- `PrivateDnsName`
- `PrivateIpAddress`
- `PublicDnsName`
- `PublicIpAddress`
- `RamdiskId`
- `RootDeviceName`
- `RootDeviceType`
- `SourceDestCheck`
- `SpotInstanceRequestId`
- `SriovNetSupport`
- `SubnetId`
- `VirtualizationType`
- `VpcId`

Some attributes are objects rather than flat values. To select a nested value, use dot notation, for example `IamInstanceProfile.Arn`.

To select a tag value, prepend the tag name with `Tags.`. For example, select the `Name` tag by using `Tags.Name`.

# Troubleshoot Amazon CloudWatch data source issues

This document provides solutions to common issues you may encounter when configuring or using the Amazon CloudWatch data source. For configuration instructions, refer to [Configure CloudWatch](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/).

{{< admonition type="note" >}}
The data source health check validates both metrics and logs permissions. If your IAM policy only grants access to one of these (for example, metrics-only or logs-only), the health check displays a red status. However, the service you have permissions for is still usable—you can query metrics or logs based on whichever permissions are configured.
{{< /admonition >}}

## Authentication errors

These errors occur when AWS credentials are invalid, missing, or don't have the required permissions. Use the following table to match the error message you see to its most likely cause and the section that resolves it.

| Error message                                                                                                  | Likely cause                                                                                            | Refer to                                                                                                                                |
| -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `AccessDenied: User is not authorized to perform: cloudwatch:...`                                              | The IAM policy is missing required CloudWatch permissions.                                              | [Access Denied or Not authorized to perform this operation](#access-denied-or-not-authorized-to-perform-this-operation)                 |
| `InvalidClientTokenId` or `UnrecognizedClientException: The security token included in the request is invalid` | Credentials are invalid, deactivated, or rotated, or a transient Grafana Cloud incident is in progress. | [InvalidClientTokenId or UnrecognizedClientException](#invalidclienttokenid-or-unrecognizedclientexception)                             |
| `AccessDenied: ... not authorized to perform: sts:AssumeRole`                                                  | The role trust relationship, role ARN, or external ID is incorrect.                                     | [Unable to assume role](#unable-to-assume-role)                                                                                         |
| `AccessDenied: ... no VPC endpoint policy allows the cloudwatch:ListMetrics action`                            | A VPC endpoint policy or service control policy (SCP) blocks the action.                                | [Access blocked by a VPC endpoint policy or service control policy](#access-blocked-by-a-vpc-endpoint-policy-or-service-control-policy) |

### "Access Denied" or "Not authorized to perform this operation"

**Symptoms:**

- Save & test fails with "Access Denied"
- Queries return authorization errors
- Namespaces, metrics, or dimensions don't load

**Possible causes and solutions:**

| Cause                                            | Solution                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| IAM policy missing required permissions          | Attach the appropriate IAM policy to your user or role. For metrics, you need `cloudwatch:ListMetrics`, `cloudwatch:GetMetricData`, and related permissions. For logs, you need `logs:DescribeLogGroups`, `logs:StartQuery`, `logs:GetQueryResults`, and related permissions. Refer to [Configure CloudWatch](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/) for complete policy examples. |
| Incorrect access key or secret key               | Verify the credentials in the AWS Console under **IAM** > **Users** > your user > **Security credentials**. Generate new credentials if necessary.                                                                                                                                                                                                                                                                                    |
| Credentials have expired                         | For temporary credentials, generate new ones. For access keys, verify they haven't been deactivated or deleted.                                                                                                                                                                                                                                                                                                                       |
| Wrong AWS region                                 | Verify the default region in the data source configuration matches where your resources are located.                                                                                                                                                                                                                                                                                                                                  |
| IAM role lacks permissions in the queried region | AWS evaluates permissions per region. If your role only has CloudWatch access in some regions, querying another region returns `AccessDenied`, and the error might not name the region. Set the data source **Default Region** to a region where your role has permissions, and only query regions your IAM policy allows.                                                                                                            |
| Assume Role ARN is incorrect                     | Verify the role ARN format: `arn:aws:iam::<account-id>:role/<role-name>`. Check that the role exists in the AWS Console.                                                                                                                                                                                                                                                                                                              |

<!-- vale Grafana.Headings = NO -->

### "InvalidClientTokenId" or "UnrecognizedClientException"

<!-- vale Grafana.Headings = YES -->

**Symptoms:**

- Save & test fails with `InvalidClientTokenId: The security token included in the request is invalid`
- Queries fail with `UnrecognizedClientException: The security token included in the request is invalid`
- The error appears suddenly without any configuration change

**Possible causes and solutions:**

| Cause                                             | Solution                                                                                                                                                                                                                                 |
| ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Access key or secret key is invalid or mistyped   | Verify the credentials in the AWS Console under **IAM** > **Users** > your user > **Security credentials**. Re-enter them in the data source configuration.                                                                              |
| Credentials were deactivated, deleted, or rotated | Generate new credentials in AWS and update the data source configuration.                                                                                                                                                                |
| Temporary credentials expired                     | Generate new temporary credentials, or switch to a long-lived authentication method such as Assume Role.                                                                                                                                 |
| Transient Grafana Cloud incident                  | If multiple data sources fail at the same time with `InvalidClientTokenId` and your credentials are unchanged, check the [Grafana Cloud status page](https://status.grafana.com/) for an ongoing incident before you rotate credentials. |

### "Unable to assume role"

**Symptoms:**

- Authentication fails when using Assume Role ARN
- Error references STS or AssumeRole, for example, `AccessDenied: User is not authorized to perform: sts:AssumeRole`

**Solutions:**

1. Verify the trust relationship on the IAM role allows the Grafana credentials to assume it.
1. Check the trust policy includes the correct principal (the user or role running Grafana).
1. If you use the **Grafana Assume Role** authentication method, confirm the trust policy uses the exact Grafana AWS account ID and external ID shown in the instructions box on the data source's **Settings** tab. Entering a different account ID is a common mistake that causes the assume-role request to fail.
1. If using an external ID, ensure it matches exactly in both the role's trust policy and the Grafana data source configuration.
1. Verify the base credentials have the `sts:AssumeRole` permission.
1. Check that the role ARN is correct and the role exists.

**Example trust policy:**

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<your-account-id>:user/<grafana-user>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<your-external-id>"
        }
      }
    }
  ]
}
```

### "Save & test" passes but queries return no data

**Symptoms:**

- **Save & test** succeeds, but panels and the query editor return no data
- Namespaces, metrics, or dimensions don't load even though the connection test passed
- The issue appears only when you use an **Assume Role ARN**

**Possible causes and solutions:**

| Cause                                                   | Solution                                                                                                                                                                                                                                                                                                                               |
| ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The connection test only validates the assume-role step | When you configure an **Assume Role ARN**, **Save & test** confirms that the primary credentials can perform `sts:AssumeRole`. It doesn't verify that the assumed role has CloudWatch or Logs permissions. Attach the required query permissions to the assumed role, not only to the primary credentials.                             |
| The assumed role is missing CloudWatch permissions      | Add the metrics and logs actions, such as `cloudwatch:ListMetrics`, `cloudwatch:GetMetricData`, `logs:DescribeLogGroups`, and `logs:StartQuery`, to the assumed role's policy. Refer to [Configure CloudWatch](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/) for complete policy examples. |
| Permissions are attached to the wrong identity          | Confirm that the policy is attached to the role named in **Assume Role ARN**, not to the IAM user or role that provides the primary credentials.                                                                                                                                                                                       |

### Data source stops working after about an hour

**Symptoms:**

- The data source works initially, then queries fail after roughly an hour
- Errors reference an expired or invalid security token
- Saving the data source again, or restarting Grafana, temporarily restores access until it fails again

**Possible causes and solutions:**

| Cause                                               | Solution                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Temporary credentials expired and weren't refreshed | Methods that use temporary credentials, such as Assume Role or STS, issue short-lived tokens. If the data source uses static temporary credentials, switch to an authentication method that refreshes automatically, such as Assume Role with long-lived base credentials or an EKS/EC2 instance role. Refer to [AWS authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/). |

### Access blocked by a VPC endpoint policy or service control policy

**Symptoms:**

- Credentials are valid and the IAM policy grants the action, but requests still fail with `AccessDenied`
- The error references a VPC endpoint policy, for example, `AccessDenied: ... no VPC endpoint policy allows the cloudwatch:ListMetrics action`
- Access fails only from within a VPC or in a restricted account, such as a FedRAMP environment

**Possible causes and solutions:**

| Cause                                                              | Solution                                                                                                                                                                                                |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A VPC endpoint policy restricts which actions are allowed          | Update the policy on the CloudWatch, Logs, EC2, or STS VPC endpoints to allow the required actions, such as `cloudwatch:ListMetrics` and `cloudwatch:GetMetricData`.                                    |
| A service control policy (SCP) explicitly denies a required action | Review the service control policies applied to the account or organizational unit. An explicit `Deny` in an SCP overrides any `Allow` in an IAM policy, so the action must be permitted at both levels. |
| Requests route through the wrong endpoint                          | Confirm that the data source region and any custom endpoint match the VPC endpoints that permit the required actions.                                                                                   |

### AWS SDK Default authentication not working

**Symptoms:**

- Data source test fails when using AWS SDK Default
- Works locally but fails in production

**Solutions:**

1. Verify AWS credentials are configured in the environment where Grafana runs.
1. Check for credentials in the default locations:
   - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
   - Shared credentials file (`~/.aws/credentials`)
   - EC2 instance metadata (if running on EC2)
   - ECS task role (if running in ECS)
   - EKS service account (if running in EKS)
1. Ensure the Grafana process has permission to read the credentials file.
1. For EKS with IRSA, set the pod's security context to allow user `472` (`grafana`) to access the projected token. Refer to [AWS authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/) for details.

### Credentials file not found

**Symptoms:**

- Error indicates credentials file cannot be read
- Authentication fails with "Credentials file" option

**Solutions:**

1. Create the credentials file at `~/.aws/credentials` for the user running the `grafana-server` service.
1. Verify the file has correct permissions (`0644`).
1. If the file exists but isn't working, move it to `/usr/share/grafana/` and set permissions to `0644`.
1. Ensure the profile name in the data source configuration matches a profile in the credentials file.

## Connection errors

These errors occur when Grafana cannot reach AWS CloudWatch endpoints.

### "Request timed out" or connection failures

**Symptoms:**

- Data source test times out
- Queries fail with timeout errors
- Intermittent connection issues

**Solutions:**

1. Verify network connectivity from the Grafana server to AWS endpoints.
1. Check firewall rules allow outbound HTTPS (port 443) to AWS services.
1. If using a VPC, ensure proper NAT gateway or VPC endpoint configuration.
1. For Grafana Cloud connecting to private resources, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/).
1. Check if the default region is correct—incorrect regions may cause longer timeouts.
1. Increase the timeout settings if queries involve large data volumes.

### Custom endpoint configuration issues

**Symptoms:**

- Connection fails when using a custom endpoint
- Endpoint URL rejected

**Solutions:**

1. Verify the endpoint URL format is correct.
1. Ensure the endpoint is accessible from the Grafana server.
1. Check that the endpoint supports the required AWS APIs.
1. For VPC endpoints, verify the endpoint policy allows the required actions.

## CloudWatch Metrics query errors

These errors occur when querying CloudWatch Metrics.

### "No data" or empty results

**Symptoms:**

- Query executes without error but returns no data
- Charts show "No data" message

**Possible causes and solutions:**

| Cause                           | Solution                                                                                                   |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Time range doesn't contain data | Expand the dashboard time range. CloudWatch metrics have different retention periods based on resolution.  |
| Wrong namespace or metric name  | Verify the namespace (for example, `AWS/EC2`) and metric name (for example, `CPUUtilization`) are correct. |
| Incorrect dimensions            | Ensure dimension names and values match your AWS resources exactly.                                        |
| Match Exact enabled incorrectly | When Match Exact is enabled, all dimensions must be specified. Try disabling it to see if metrics appear.  |
| Period too large                | Reduce the period setting or set it to "auto" to ensure data points are returned for your time range.      |
| Custom metrics not configured   | Add custom metric namespaces in the data source configuration under **Namespaces of Custom Metrics**.      |

### "Metric not found" or metrics don't appear in drop-down

**Symptoms:**

- Expected metrics don't appear in the query editor
- Metric drop-down is empty for a namespace
- A specific metric is missing from the drop-down even though it exists in the CloudWatch console

**Solutions:**

1. Verify the metric exists in the selected region.
1. For custom metrics, add the namespace to **Namespaces of Custom Metrics** in the data source configuration.
1. Check that the IAM policy includes `cloudwatch:ListMetrics` permission.
1. CloudWatch limits `ListMetrics` to 500 results per page. To retrieve more metrics, increase the `list_metrics_page_limit` setting in the [Grafana configuration file](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/#configure-the-data-source-with-grafanaini).
1. Type the metric name directly into the **Metric name** field. For some AWS namespaces, the data source uses a predefined metric list, so newer or less common metrics might not appear in the drop-down even though you can still query them. If the metric exists in your CloudWatch console, enter its exact name to query it.
1. Use the Query Inspector to verify the API request and response.

### Dimension values not loading

**Symptoms:**

- Dimension value drop-down doesn't populate
- Wildcard searches return no results

**Solutions:**

1. Verify the IAM policy includes `cloudwatch:ListMetrics` permission.
1. Check that the namespace and metric are selected before dimension values can load.
1. For EC2 dimensions, ensure `ec2:DescribeTags` and `ec2:DescribeInstances` permissions are granted.
1. Dimension values require existing metrics—if no metrics match, no values appear.

### "Too many data points" or API throttling

**Symptoms:**

- Queries fail with throttling errors
- Performance degrades with multiple panels

**Solutions:**

1. Increase the period setting to reduce the number of data points.
1. Reduce the time range of your queries.
1. Use fewer dimensions or wildcard queries per panel.
1. Request a quota increase for `GetMetricData` requests per second in the [AWS Service Quotas console](https://console.aws.amazon.com/servicequotas/).
1. Enable query caching in Grafana to reduce API calls.

This section covers throttling at the query level. For account-wide rate limits and quota increases, refer to [API throttling errors](#api-throttling-errors).

### Metric math expression errors

**Symptoms:**

- Expression returns errors
- Referenced metrics not found

**Solutions:**

1. Verify each referenced metric has a unique ID set.
1. Check that metric IDs start with a lowercase letter and contain only letters, numbers, and underscores.
1. Ensure all referenced metrics are in the same query.
1. Verify the expression syntax follows [AWS Metric Math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) documentation.
1. Metric math expressions can't be used with Grafana alerting if they reference other query rows.

## CloudWatch Logs query errors

These errors occur when querying CloudWatch Logs.

### "Query failed" or logs don't appear

**Symptoms:**

- Log queries return errors
- No log data is displayed

**Solutions:**

1. Verify log group names are correct and exist in the selected region.
1. Check the IAM policy includes `logs:StartQuery`, `logs:GetQueryResults`, and `logs:DescribeLogGroups` permissions.
1. Ensure the time range contains log data.
1. Verify the query syntax is valid. For CloudWatch Logs Insights QL, test the query in the AWS Console.
1. Select the correct query language (Logs Insights QL, OpenSearch PPL, or OpenSearch SQL) based on your query syntax.

### Log query timeout

**Symptoms:**

- Query runs for a long time then fails
- Error mentions timeout

**Solutions:**

1. Increase the **Query Result Timeout** setting in the data source configuration (default is 30 minutes).
1. Narrow the time range to reduce the amount of data scanned.
1. Add filters to your query to limit results.
1. Break complex queries into smaller, more focused queries.
1. For alerting, the timeout defined in the [Grafana configuration file](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#unified_alerting) takes precedence.

### Log groups not appearing in selector

**Symptoms:**

- Log group selector is empty
- Can't find expected log groups

**Solutions:**

1. Verify the IAM policy includes `logs:DescribeLogGroups` permission.
1. Check that log groups exist in the selected region.
1. For cross-account observability, ensure proper IAM permissions for `oam:ListSinks` and `oam:ListAttachedLinks`.
1. Use prefix search to filter log groups if you have many groups.
1. Verify the selected account (for cross-account) contains the expected log groups.

### OpenSearch SQL query errors

**Symptoms:**

- OpenSearch SQL queries fail
- Syntax errors with SQL queries

**Solutions:**

1. Specify the log group identifier or ARN in the `FROM` clause:

   ```sql
   SELECT * FROM `log_group_name` WHERE `@message` LIKE '%error%'
   ```

1. For multiple log groups, use the `logGroups` function:

   ```sql
   SELECT * FROM `logGroups(logGroupIdentifier: ['LogGroup1', 'LogGroup2'])`
   ```

1. Amazon CloudWatch supports only a subset of OpenSearch SQL commands. Refer to the [CloudWatch Logs documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_Languages.html) for supported syntax.

## Template variable errors

These errors occur when using template variables with the CloudWatch data source.

### Variables return no values

**Symptoms:**

- Variable drop-down is empty
- Dashboard fails to load with variable errors

**Solutions:**

1. Verify the data source connection is working.
1. Check that the IAM policy includes permissions for the variable query type:
   - **Regions:** No additional permissions needed.
   - **Namespaces:** No additional permissions needed.
   - **Metrics:** Requires `cloudwatch:ListMetrics`.
   - **Dimension Values:** Requires `cloudwatch:ListMetrics`.
   - **EC2 Instance Attributes:** Requires `ec2:DescribeInstances`.
   - **EBS Volume IDs:** Requires `ec2:DescribeVolumes`.
   - **Resource ARNs:** Requires `tag:GetResources`.
   - **Log Groups:** Requires `logs:DescribeLogGroups`.
1. For dependent variables, ensure parent variables have valid selections.
1. Verify the region is set correctly (use "default" for the data source's default region).

For more information on template variables, refer to [CloudWatch template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/template-variables/).

### Multi-value template variables cause query failures

**Symptoms:**

- Queries fail when selecting multiple dimension values
- Error about search expression limits

**Solutions:**

1. Search expressions are limited to 1,024 characters. Reduce the number of selected values.
1. Use the asterisk (`*`) wildcard instead of selecting "All" to query all metrics for a dimension.
1. Multi-valued template variables are only supported for dimension values—not for Region, Namespace, or Metric Name.

## Cross-account observability errors

These errors occur when using CloudWatch cross-account observability features.

### Cross-account queries fail

**Symptoms:**

- Can't query metrics or logs from linked accounts
- Monitoring account badge doesn't appear

**Solutions:**

1. Verify cross-account observability is configured in the AWS CloudWatch console.
1. Add the required IAM permissions:

   ```json
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Action": ["oam:ListSinks", "oam:ListAttachedLinks"],
         "Effect": "Allow",
         "Resource": "*"
       }
     ]
   }
   ```

1. Check that the monitoring account and source accounts are properly linked in AWS.
1. Cross-account observability works within a single region—verify all accounts are in the same region.
1. EC2 Instance Attributes can't be queried across accounts because they use the EC2 API, not the CloudWatch API.

## Quota and pricing issues

These issues relate to AWS service quotas and cost management.

### API throttling errors

**Symptoms:**

- "Rate exceeded" errors
- Dashboard panels intermittently fail to load

**Solutions:**

1. Reduce the frequency of dashboard refreshes.
1. Increase the period setting to reduce `GetMetricData` requests.
1. Enable query caching in Grafana (available in Grafana Enterprise and Grafana Cloud).
1. Request a quota increase in the [AWS Service Quotas console](https://console.aws.amazon.com/servicequotas/).
1. Consider consolidating similar queries using metric math.

If throttling happens because a single query returns too many data points, refer to ["Too many data points" or API throttling](#too-many-data-points-or-api-throttling).

### Unexpectedly high CloudWatch costs

**Symptoms:**

- AWS CloudWatch costs are higher than expected
- Frequent API calls from Grafana

**Solutions:**

1. The `GetMetricData` API doesn't qualify for the CloudWatch API free tier.
1. Reduce dashboard auto-refresh frequency.
1. Increase the period setting to reduce data points returned.
1. Use query caching to reduce repeated API calls.
1. Review variable query settings—set variable refresh to "On dashboard load" instead of "On time range change."
1. Avoid using wildcards in dimensions when possible, as they generate search expressions with multiple API calls.

## Other common issues

These issues don't produce specific error messages but are commonly encountered.

### Custom metrics don't appear

**Symptoms:**

- Custom metrics from applications or agents don't show in the namespace drop-down
- Only standard AWS namespaces are visible

**Solutions:**

1. Add your custom metric namespace to the **Namespaces of Custom Metrics** field in the data source configuration.
1. Separate multiple namespaces with commas (for example, `CWAgent,CustomNamespace`).
1. Verify custom metrics have been published to CloudWatch in the selected region.

### Pre-configured dashboards not working

**Symptoms:**

- Imported dashboards show no data
- Dashboard variables don't load

**Solutions:**

1. Verify the data source name in the dashboard matches your CloudWatch data source.
1. Check that the dashboard's AWS region setting matches where your resources are located.
1. Ensure the IAM policy grants access to the required services (EC2, Lambda, RDS, etc.).
1. Verify resources exist and are emitting metrics in the selected region.

### Application Signals trace links not appearing

**Symptoms:**

- Log entries don't show Application Signals trace links
- `@xrayTraceId` field not appearing

**Solutions:**

1. Verify an Application Signals data source is configured and linked in the CloudWatch data source settings.
1. Ensure your logs contain the `@xrayTraceId` field.
1. Update log queries to include `@xrayTraceId` in the fields, for example: `fields @message, @xrayTraceId`.
1. Configure your application to log Application Signals trace IDs. Refer to the [AWS Application Signals documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html).

## Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location).
1. Look for CloudWatch-specific entries that include request and response details.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you've tried the solutions above and still encounter issues:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review the [CloudWatch plugin GitHub issues](https://github.com/grafana/grafana/issues) for known bugs.
1. Consult the [AWS CloudWatch documentation](https://docs.aws.amazon.com/cloudwatch/) for service-specific guidance.
1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user.
1. When reporting issues, include:
   - Grafana version
   - AWS region
   - Error messages (redact sensitive information)
   - Steps to reproduce
   - Query configuration (redact credentials and account IDs)

# Azure Monitor alerting

The Azure Monitor data source supports [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/) and [Grafana-managed recording rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/), allowing you to create alert rules based on Azure metrics, logs, traces, and resource data. You can monitor your Azure environment and receive notifications when specific conditions are met.

## Before you begin

- Ensure you have the appropriate permissions to create alert rules in Grafana.
- Verify your Azure Monitor data source is configured and working correctly.
- Familiarize yourself with [Grafana Alerting concepts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/).
- **Important**: Verify your data source uses a supported authentication method. Refer to [Authentication requirements](#authentication-requirements).

## Supported query types for alerting

All Azure Monitor query types support alerting and recording rules:

| Query type           | Use case                                           | Notes                                                    |
| -------------------- | -------------------------------------------------- | -------------------------------------------------------- |
| Metrics              | Threshold-based alerts on Azure resource metrics   | Best suited for alerting; returns time-series data       |
| Logs                 | Alert on log patterns, error counts, or thresholds | Use KQL to aggregate data into numeric values            |
| Azure Resource Graph | Alert on resource state or configuration changes   | Use count aggregations to return numeric data            |
| Traces               | Alert on trace data and application performance    | Use aggregations to return numeric values for evaluation |

{{< admonition type="note" >}}
Alert queries must return numeric data that Grafana can evaluate against a threshold. Queries that return only text or non-numeric data cannot be used directly for alerting.
{{< /admonition >}}

## Authentication requirements

Alerting and recording rules run as background processes without a user context. This means they require service-level authentication and don't work with all authentication methods.

| Authentication method                 | Supported                             |
| ------------------------------------- | ------------------------------------- |
| App Registration (client secret)      | ✓                                     |
| App Registration (client certificate) | ✓                                     |
| Managed Identity                      | ✓                                     |
| Workload Identity                     | ✓                                     |
| Current User                          | ✓ (with fallback service credentials) |

{{< admonition type="note" >}}
If you use **Current User** authentication, you must configure **fallback service credentials** for alerting and recording rules to function. User credentials aren't available for background operations, so Grafana uses the fallback credentials instead. Refer to [configure the data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/configure/) for details on setting up fallback credentials.
{{< /admonition >}}

## Create an alert rule

To create an alert rule using Azure Monitor data:

1. Go to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for your alert rule.
1. In the **Define query and alert condition** section:
   - Select your Azure Monitor data source.
   - Configure your query (for example, a Metrics query for CPU usage or a Logs query using KQL).
   - Add a **Reduce** expression if your query returns multiple series.
   - Add a **Threshold** expression to define the alert condition.
1. Configure the **Set evaluation behavior**:
   - Select or create a folder and evaluation group.
   - Set the evaluation interval (how often the alert is checked).
   - Set the pending period (how long the condition must be true before firing).
1. Add labels and annotations to provide context for notifications.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example: VM CPU usage alert

This example creates an alert that fires when virtual machine CPU usage exceeds 80%:

1. Create a new alert rule.
1. Configure the query:
   - **Service**: Metrics
   - **Resource**: Select your virtual machine
   - **Metric namespace**: `Microsoft.Compute/virtualMachines`
   - **Metric**: `Percentage CPU`
   - **Aggregation**: `Average`
1. Add expressions:
   - **Reduce**: Last (to get the most recent data point)
   - **Threshold**: Is above 80
1. Set evaluation to run every 1 minute with a 5-minute pending period.
1. Save the rule.

## Example: Error log count alert

This example alerts when error logs exceed a threshold using a KQL query:

1. Create a new alert rule.
1. Configure the query:
   - **Service**: Logs
   - **Resource**: Select your Log Analytics workspace
   - **Query**:
     ```kusto
     AppExceptions
     | where TimeGenerated > ago(5m)
     | summarize ErrorCount = count() by bin(TimeGenerated, 1m)
     ```
1. Add expressions:
   - **Reduce**: Max (to get the highest count in the period)
   - **Threshold**: Is above 10
1. Set evaluation to run every 5 minutes.
1. Save the rule.

## Example: Resource count alert

This example alerts when the number of running virtual machines drops below a threshold using Azure Resource Graph:

1. Create a new alert rule.
1. Configure the query:
   - **Service**: Azure Resource Graph
   - **Subscriptions**: Select your subscriptions
   - **Query**:

     ```kusto
     resources
     | where type == "microsoft.compute/virtualmachines"
     | where properties.extended.instanceView.powerState.displayStatus == "VM running"
     | summarize RunningVMs = count()
     ```

1. Add expressions:
   - **Reduce**: Last
   - **Threshold**: Is below 3
1. Set evaluation to run every 5 minutes.
1. Save the rule.

## Best practices

Follow these recommendations to create reliable and efficient alerts with Azure Monitor data.

### Use appropriate query intervals

- Set the alert evaluation interval to be greater than or equal to the minimum data resolution from Azure Monitor.
- Azure Monitor Metrics typically have 1-minute granularity at minimum.
- Avoid very short intervals (less than 1 minute) as they may cause evaluation timeouts or miss data points.

### Reduce multiple series

When your Azure Monitor query returns multiple time series (for example, CPU usage across multiple VMs), use the **Reduce** expression to aggregate them:

- **Last**: Use the most recent value
- **Mean**: Average across all series
- **Max/Min**: Use the highest or lowest value
- **Sum**: Total across all series

### Optimize Log Analytics queries

For Logs queries used in alerting:

- Use `summarize` to aggregate data into numeric values.
- Include appropriate time filters using `ago()` or `TimeGenerated`.
- Avoid returning large result sets; aggregate data in the query.
- Test queries in Explore before using them in alert rules.

### Handle no data conditions

Configure what happens when no data is returned:

1. In the alert rule, find **Configure no data and error handling**.
1. Choose an appropriate action:
   - **No Data**: Keep the alert in its current state
   - **Alerting**: Treat no data as an alert condition
   - **OK**: Treat no data as a healthy state

### Test queries before alerting

Always verify your query returns expected data before creating an alert:

1. Go to **Explore**.
1. Select your Azure Monitor data source.
1. Run the query you plan to use for alerting.
1. Confirm the data format and values are correct.
1. Verify the query returns numeric data suitable for threshold evaluation.

## Troubleshooting

If your Azure Monitor alerts aren't working as expected, use the following sections to diagnose and resolve common issues.

### Alerts not firing

- Verify the data source uses a supported authentication method. If using Current User authentication, ensure fallback service credentials are configured.
- Check that the query returns numeric data in Explore.
- Ensure the evaluation interval allows enough time for data to be available.
- Review the alert rule's health and any error messages in the Alerting UI.

### Authentication errors in alert evaluation

If you see authentication errors when alerts evaluate:

- Confirm the data source is configured with App Registration, Managed Identity, Workload Identity, or Current User with fallback service credentials.
- If using App Registration, verify the client secret hasn't expired.
- If using Current User, verify that fallback service credentials are configured and valid.
- Check that the service principal has appropriate permissions on Azure resources.

### Query timeout errors

- **Increase the data source timeout.** The default timeout is 30 seconds, but Azure Log Analytics queries can take up to 3 minutes. Open the Azure Monitor data source settings, scroll to **Misc**, and set **Timeout** to `300`. Refer to [Query timeout](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/configure/#query-timeout).
- Simplify complex KQL queries.
- Reduce the time range in Log Analytics queries.
- Add more specific filters to narrow result sets.
- Ensure the alert evaluation interval is long enough to accommodate the query duration.

For additional troubleshooting help, refer to [Troubleshoot Azure Monitor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/troubleshooting/).

## Additional resources

- [Grafana Alerting documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/)
- [Create alert rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/)
- [Azure Monitor query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/query-editor/)
- [Grafana-managed recording rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/)

# Azure Monitor annotations

[Annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/) overlay rich event information on top of graphs. You can use Azure Monitor Log Analytics queries to create annotations that mark important events, deployments, alerts, or other significant occurrences on your dashboards.

## Before you begin

- Ensure you have configured the Azure Monitor data source.
- You need access to a Log Analytics workspace containing the data you want to use for annotations.
- Annotations use Log Analytics (KQL) queries only. Metrics, Traces, and Azure Resource Graph queries are not supported for annotations.

## Create an annotation query

To add an Azure Monitor annotation to a dashboard:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Annotation query**.
1. Enter a name for the annotation query.
1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox.
1. Select a color for the annotation event markers.
1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed.
1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed.
1. Click **Open query editor** to open the **Annotation Query** dialog box.
1. Select your **Azure Monitor** data source from the **Data source** drop-down list.
1. Choose the **Logs** service.
1. Select a **Resource** (Log Analytics workspace or Application Insights resource).
1. Write a KQL query that returns the annotation data.
1. Configure the annotation query and field mappings.
1. (Optional) Click **Test annotation query** to ensure that the query is working properly.
1. Click **Close** when you've completed the query setup.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## Query requirements

Your KQL query should return columns that Grafana can use to create annotations:

| Column             | Required    | Description                                                                                      |
| ------------------ | ----------- | ------------------------------------------------------------------------------------------------ |
| `TimeGenerated`    | Yes         | The timestamp for the annotation. Grafana uses this to position the annotation on the time axis. |
| `Text`             | Recommended | The annotation text displayed when you hover over or click the annotation.                       |
| Additional columns | Optional    | Any other columns returned become annotation tags.                                               |

{{< admonition type="note" >}}
Always include a time filter in your query to limit results to the dashboard's time range. Use the `$__timeFilter()` macro.
{{< /admonition >}}

## Annotation query examples

The following examples demonstrate common annotation use cases.

### Azure Activity Log events

Display Azure Activity Log events such as resource modifications, deployments, and administrative actions:

```kusto
AzureActivity
| where $__timeFilter(TimeGenerated)
| where Level == "Error" or Level == "Warning" or CategoryValue == "Administrative"
| project TimeGenerated, Text=OperationNameValue, Level, ResourceGroup, Caller
| order by TimeGenerated desc
| take 100
```

### Deployment events

Show deployment-related activity:

```kusto
AzureActivity
| where $__timeFilter(TimeGenerated)
| where OperationNameValue contains "deployments"
| project TimeGenerated, Text=strcat("Deployment: ", OperationNameValue), Status=ActivityStatusValue, ResourceGroup
| order by TimeGenerated desc
```

### Application Insights exceptions

Mark application exceptions as annotations:

```kusto
AppExceptions
| where $__timeFilter(TimeGenerated)
| project TimeGenerated, Text=strcat(ProblemId, ": ", OuterMessage), SeverityLevel, AppRoleName
| order by TimeGenerated desc
| take 50
```

### Custom events from Application Insights

Display custom events logged by your application:

```kusto
AppEvents
| where $__timeFilter(TimeGenerated)
| where Name == "DeploymentStarted" or Name == "DeploymentCompleted"
| project TimeGenerated, Text=Name, AppRoleName
| order by TimeGenerated desc
```

### Security alerts

Show security-related alerts:

```kusto
SecurityAlert
| where $__timeFilter(TimeGenerated)
| project TimeGenerated, Text=AlertName, Severity=AlertSeverity, Description
| order by TimeGenerated desc
| take 50
```

### Resource health events

Display resource health status changes:

```kusto
AzureActivity
| where $__timeFilter(TimeGenerated)
| where CategoryValue == "ResourceHealth"
| project TimeGenerated, Text=OperationNameValue, Status=ActivityStatusValue, ResourceId
| order by TimeGenerated desc
```

### VM start and stop events

Mark virtual machine state changes:

```kusto
AzureActivity
| where $__timeFilter(TimeGenerated)
| where OperationNameValue has_any ("start", "deallocate", "restart")
| where ResourceProviderValue == "MICROSOFT.COMPUTE"
| project TimeGenerated, Text=OperationNameValue, VM=Resource, Status=ActivityStatusValue
| order by TimeGenerated desc
```

### Autoscale events

Show autoscale operations:

```kusto
AzureActivity
| where $__timeFilter(TimeGenerated)
| where OperationNameValue contains "autoscale"
| project TimeGenerated, Text=strcat("Autoscale: ", OperationNameValue), Status=ActivityStatusValue, ResourceGroup
| order by TimeGenerated desc
```

## Customize annotation appearance

After creating an annotation query, you can customize its appearance:

| Setting       | Description                                                                                              |
| ------------- | -------------------------------------------------------------------------------------------------------- |
| **Color**     | Choose a color for the annotation markers. Use different colors to distinguish between annotation types. |
| **Show in**   | Select which panels display the annotations.                                                             |
| **Filter by** | Add filters to limit when annotations appear.                                                            |

## Best practices

Follow these recommendations when creating annotations:

1. **Limit results**: Always use `take` or `limit` to restrict the number of annotations. Too many annotations can clutter your dashboard and impact performance.

2. **Use time filters**: Include `$__timeFilter()` to ensure queries only return data within the dashboard's time range.

3. **Create meaningful text**: Use `strcat()` or `project` to create descriptive annotation text that provides context at a glance.

4. **Add relevant tags**: Include columns like `ResourceGroup`, `Severity`, or `Status` that become clickable tags for filtering.

5. **Use descriptive names**: Name your annotations clearly (for example, "Production Deployments", "Critical Alerts") so dashboard users understand what they represent.

## Troubleshoot annotations

If annotations aren't appearing as expected, try the following solutions.

### Annotations don't appear

- Verify the query returns data in the selected time range.
- Check that the query includes a `TimeGenerated` column.
- Test the query in the Azure Portal Log Analytics query editor.
- Ensure the annotation is enabled (toggle is on).

### Too many annotations

- Add more specific filters to your query.
- Use `take` to limit results.
- Narrow the time range.

### Annotations appear at wrong times

- Verify the `TimeGenerated` column contains the correct timestamp.
- Check your dashboard's timezone settings.

# Configure the Azure Monitor data source

This document explains how to configure the Azure Monitor data source and the available configuration options.
For general information about data sources, refer to [Grafana data sources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/) and [Data source management](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/).

## Before you begin

Before configuring the Azure Monitor data source, ensure you have the following:

- **Grafana permissions:** You must have the `Organization administrator` role to configure data sources.
  Organization administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system or [using Terraform](#configure-with-terraform).

- **Azure prerequisites:** Depending on your chosen authentication method, you may need:
  - A Microsoft Entra ID (formerly Azure AD) app registration with a service principal (for App Registration authentication)
  - A Managed Identity enabled on your Azure VM or App Service (for Managed Identity authentication)
  - Workload identity configured in your Kubernetes cluster (for Workload Identity authentication)
  - Microsoft Entra ID authentication configured for Grafana login (for Current User authentication)

{{< admonition type="note" >}}
**Grafana Cloud users:** Managed Identity and Workload Identity authentication methods are not available in Grafana Cloud because they require Grafana to run on your Azure infrastructure. Use **App Registration** authentication instead.
{{< /admonition >}}

- **Azure RBAC permissions:** The identity used to authenticate must have the `Reader` role on the Azure subscription containing the resources you want to monitor.
  For Log Analytics queries, the identity also needs appropriate permissions on the Log Analytics workspaces to be queried.
  Refer to the [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current).

{{< admonition type="note" >}}
The Azure Monitor data source plugin is built into Grafana. No additional installation is required.
{{< /admonition >}}

## Add the data source

To add the Azure Monitor data source:

1. Click **Connections** in the left-side menu.
1. Click **Add new connection**.
1. Type `Azure Monitor` in the search bar.
1. Select **Azure Monitor**.
1. Click **Add new data source** in the upper right.

You're taken to the **Settings** tab where you can configure the data source.

## Choose an authentication method

The Azure Monitor data source supports five authentication methods. Choose based on where Grafana is hosted, your security requirements, and whether you need alerting:

| Authentication method                     | Best for                                   | Grafana Cloud | Supports alerting | Server configuration required |
| ----------------------------------------- | ------------------------------------------ | ------------- | ----------------- | ----------------------------- |
| **App Registration (client secret)**      | Any Grafana deployment                     | ✓             | ✓                 | No                            |
| **App Registration (client certificate)** | Any Grafana deployment                     | ✓             | ✓                 | No                            |
| **Managed Identity**                      | Grafana hosted in Azure (VMs, App Service) | ✗             | ✓                 | Yes                           |
| **Workload Identity**                     | Grafana in Kubernetes (AKS)                | ✗             | ✓                 | Yes                           |
| **Current User**                          | User-level access control                  | ✓             | Partial           | Yes                           |

{{< admonition type="note" >}}
**Current User** authentication doesn't support background operations like alerting, reporting, and recording rules. To use alerting with Current User, you must configure **fallback service credentials**. Alerts then run under the fallback credential's permissions. Refer to [Limitations and fallback credentials](#limitations-and-fallback-credentials) for details.
{{< /admonition >}}

## Configure authentication

Select one of the following authentication methods and complete the configuration.

### App Registration (client secret)

Use a Microsoft Entra ID app registration (service principal) to authenticate. This method works with any Grafana deployment.

#### App Registration prerequisites

1. Create an app registration in Microsoft Entra ID.
   Refer to the [Azure documentation for creating a service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#get-tenant-and-app-id-values-for-signing-in).

1. Create a client secret for the app registration.
   Refer to the [Azure documentation for creating a client secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret).

1. Assign the `Reader` role to the app registration on the subscription or resources you want to monitor.
   Refer to the [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current).

#### App Registration UI configuration

| Setting                     | Description                                                                                                                                |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Authentication**          | Select **App Registration**.                                                                                                               |
| **Azure Cloud**             | The Azure environment to connect to. Select **Azure** for the public cloud, or choose Azure Government or Azure China for national clouds. |
| **Directory (tenant) ID**   | The GUID that identifies your Microsoft Entra ID tenant.                                                                                   |
| **Application (client) ID** | The GUID for the app registration you created.                                                                                             |
| **Client secret**           | The secret key for the app registration. Keep this secure and rotate periodically.                                                         |
| **Default Subscription**    | Click **Load Subscriptions** to populate available subscriptions, then select your default.                                                |

#### Provision App Registration with YAML

```yaml
apiVersion: 1

datasources:
  - name: Azure Monitor
    type: grafana-azure-monitor-datasource
    access: proxy
    jsonData:
      azureAuthType: clientsecret
      cloudName: azuremonitor # See supported cloud names below
      tenantId: <tenant-id>
      clientId: <client-id>
      subscriptionId: <subscription-id> # Optional, default subscription
    secureJsonData:
      clientSecret: <client-secret>
    version: 1
```

### App Registration (client certificate)

Use a Microsoft Entra ID app registration with a certificate.

#### App Registration (client certificate) prerequisites

1. Create an app registration in Microsoft Entra ID.
   Refer to the [Azure documentation for creating a service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#get-tenant-and-app-id-values-for-signing-in).
1. Create or retrieve an existing certificate (see examples below for creating self-signed PEM or PFX certificates).
1. Upload a certificate to the app registration in PEM or PFX format. Refer to the [Azure documentation here](https://learn.microsoft.com/en-us/entra/identity-platform/how-to-add-credentials?source=recommendations&tabs=certificate).
1. Assign the `Reader` role to the app registration on the subscription or resources you want to monitor.
   Refer to the [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current).

#### Add the certificate to the app registration

Add the public certificate to your app registration in the Azure portal:

1. Open the app registration in Azure.
1. Select **Certificates & secrets**.
1. Select **Certificates**.
1. Click **Upload certificate**.
1. Upload the certificate file that contains the public certificate (`.cer`, `.crt`, or `.pem`).
1. Save your changes.

The uploaded certificate is used by Microsoft Entra ID to validate the client assertion signature from Grafana.

#### Certificate formats

The Azure Monitor data source supports the following certificate formats:

- **PEM:** Provide both the certificate and private key in PEM text format.
- **PFX:** Provide a base64-encoded PFX payload in the **Client Certificate** field, and provide the certificate password.

#### Create self-signed certificates

The following examples are for test and lab environments. For production, use certificates from your internal or public certificate authority.

Create a self-signed PEM certificate and private key with OpenSSL:

```sh
openssl genrsa -out key.pem 2048
openssl req -new -sha256 -key key.pem -out csr.csr
openssl req -x509 -sha256 -days 365 -key key.pem -in csr.csr -out certificate.pem
```

In this example:

- `certificate.pem` is uploaded to **Certificates & secrets** in Azure.
- `certificate.pem` is the client certificate added to Grafana.
- `key.pem` is the private key added to Grafana.

Create an encrypted, base64 encoded PFX certificate:

```sh
openssl genrsa -out key.pem 2048
openssl req -new -sha256 -key key.pem -out csr.csr
openssl req -x509 -sha256 -days 365 -key key.pem -in csr.csr -out certificate.pem
openssl pkcs12 -export -inkey key.pem -in certificate.pem -out certificate.pfx
openssl base64 -in certificate.pfx -out certificate.b64
```

Ensure you make note of the password you use to encrypt the certificate.

In this example:

- `certificate.pem` is uploaded to **Certificates & secrets** in Azure.
- `certificate.b64` is the encrypted, encoded file added to Grafana (in the client certificate field).

#### App Registration (client certificate) UI configuration

| Setting                     | Description                                                                                                                                |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Authentication**          | Select **App Registration (Client Certificate)**.                                                                                          |
| **Azure Cloud**             | The Azure environment to connect to. Select **Azure** for the public cloud, or choose Azure Government or Azure China for national clouds. |
| **Directory (tenant) ID**   | The GUID that identifies your Microsoft Entra ID tenant.                                                                                   |
| **Application (client) ID** | The GUID for the app registration you created.                                                                                             |
| **Format**                  | Select **PEM** or **PFX**.                                                                                                                 |
| **Client Certificate**      | Paste the certificate content (PEM text or base64-encoded PFX payload).                                                                    |
| **Private Key**             | Required only for PEM. Paste the private key PEM.                                                                                          |
| **Certificate Password**    | Required for encrypted PFX files.                                                                                                          |
| **Default Subscription**    | Click **Load Subscriptions** to populate available subscriptions, then select your default.                                                |

#### Provision App Registration (client certificate) with YAML

To provision the Azure Monitor data source with client certificate authentication, use the following configuration:

```yaml
apiVersion: 1

datasources:
  - name: Azure Monitor
    type: grafana-azure-monitor-datasource
    access: proxy
    jsonData:
      azureAuthType: clientcertificate
      cloudName: azuremonitor # See supported cloud names below
      tenantId: <tenant-id>
      clientId: <client-id>
      certificateFormat: pem # Use pem or pfx
      subscriptionId: <subscription-id> # Optional, default subscription
    secureJsonData:
      clientCertificate: <certificate-content>
      privateKey: <private-key-content> # pem only
      certificatePassword: <certificate-password> # pfx only, if set
    version: 1
```

### Managed Identity

Use Azure Managed Identity for secure, credential-free authentication when Grafana is hosted in Azure.

{{< admonition type="note" >}}
Managed Identity is available in [Azure Managed Grafana](https://azure.microsoft.com/en-us/products/managed-grafana) or self-managed Grafana deployed in Azure. It is not available in Grafana Cloud.
{{< /admonition >}}

#### Managed Identity prerequisites

- Grafana must be hosted in Azure (App Service, Azure VMs, or Azure Managed Grafana).
- Managed identity must be enabled on the Azure resource hosting Grafana.
- The managed identity must have the `Reader` role on the subscription or resources you want to monitor.

For details on Azure managed identities, refer to the [Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview).

#### Managed Identity Grafana server configuration

Enable managed identity in the Grafana server configuration:

```ini
[azure]
managed_identity_enabled = true
```

To use a user-assigned managed identity instead of the system-assigned identity, also set:

```ini
[azure]
managed_identity_enabled = true
managed_identity_client_id = <USER_ASSIGNED_IDENTITY_CLIENT_ID>
```

{{< admonition type="caution" >}}
If you've customized the `forward_settings_to_plugins` setting under `[azure]`, verify that `grafana-azure-monitor-datasource` is included. This setting controls which plugins receive the Azure configuration from the Grafana server. By default, all Grafana Labs Azure plugins are included. Missing this setting causes `401 Unauthorized` errors even when Managed Identity is correctly configured on your Azure resource.
{{< /admonition >}}

Refer to [Grafana Azure configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#azure) for more details.

#### Managed Identity UI configuration

| Setting                  | Description                                                                                         |
| ------------------------ | --------------------------------------------------------------------------------------------------- |
| **Authentication**       | Select **Managed Identity**. The directory ID, application ID, and client secret fields are hidden. |
| **Default Subscription** | Click **Load Subscriptions** to populate available subscriptions, then select your default.         |

{{< figure src="/media/docs/grafana/data-sources/screenshot-managed-identity-2.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor data source configured with Managed Identity" >}}

#### Provision Managed Identity with YAML

```yaml
apiVersion: 1

datasources:
  - name: Azure Monitor
    type: grafana-azure-monitor-datasource
    access: proxy
    jsonData:
      azureAuthType: msi
      subscriptionId: <subscription-id> # Optional, default subscription
    version: 1
```

### Workload Identity

Use Azure Workload Identity for secure authentication in Kubernetes environments like AKS.

#### Workload Identity prerequisites

- Grafana must be running in a Kubernetes environment with workload identity federation configured.
- The workload identity must have the `Reader` role on the subscription or resources you want to monitor.

For details, refer to the [Azure workload identity documentation](https://azure.github.io/azure-workload-identity/docs/).

#### Workload Identity Grafana server configuration

Enable workload identity in the Grafana server configuration:

```ini
[azure]
workload_identity_enabled = true
```

Optional configuration variables:

```ini
[azure]
workload_identity_enabled = true
workload_identity_tenant_id = <IDENTITY_TENANT_ID>    # Microsoft Entra ID tenant containing the managed identity
workload_identity_client_id = <IDENTITY_CLIENT_ID>    # Client ID if different from default
workload_identity_token_file = <TOKEN_FILE_PATH>      # Path to the token file
```

{{< admonition type="caution" >}}
If you've customized the `forward_settings_to_plugins` setting under `[azure]`, verify that `grafana-azure-monitor-datasource` is included. Refer to [Managed Identity Grafana server configuration](#managed-identity-grafana-server-configuration) for details.
{{< /admonition >}}

Refer to [Grafana Azure configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#azure) and the [Azure workload identity documentation](https://azure.github.io/azure-workload-identity/docs/) for more details.

#### Workload Identity UI configuration

| Setting                  | Description                                                                                          |
| ------------------------ | ---------------------------------------------------------------------------------------------------- |
| **Authentication**       | Select **Workload Identity**. The directory ID, application ID, and client secret fields are hidden. |
| **Default Subscription** | Click **Load Subscriptions** to populate available subscriptions, then select your default.          |

{{< figure src="/media/docs/grafana/data-sources/screenshot-workload-identity.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor data source configured with Workload Identity" >}}

#### Provision Workload Identity with YAML

```yaml
apiVersion: 1

datasources:
  - name: Azure Monitor
    type: grafana-azure-monitor-datasource
    access: proxy
    jsonData:
      azureAuthType: workloadidentity
      subscriptionId: <subscription-id> # Optional, default subscription
    version: 1
```

### Current User

Forward the logged-in Grafana user's Azure credentials to the data source for user-level access control.

{{< admonition type="note" >}}
Current User authentication is a [generally available feature](/docs/release-life-cycle/). Contact Grafana Support to enable this feature in Grafana Cloud.
{{< /admonition >}}

#### Current User prerequisites

Your Grafana instance must be configured with Microsoft Entra ID authentication. Refer to the [Microsoft Entra ID authentication documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/configure-authentication/azuread/).

#### Configure your Azure App Registration

The App Registration used for Grafana login requires additional configuration:

**Enable token issuance:**

1. In the Azure Portal, open your App Registration.
1. Select **Authentication** in the side menu.
1. Under **Implicit grant and hybrid flows**, check both **Access tokens** and **ID tokens**.
1. Save your changes.

**Add API permissions:**

1. In the Azure Portal, open your App Registration.
1. Select **API Permissions** in the side menu.
1. Ensure these permissions are present under **Microsoft Graph**: `openid`, `profile`, `email`, and `offline_access`.
1. Add the following permissions:
   - **Azure Service Management** > **Delegated permissions** > `user_impersonation`
   - **APIs my organization uses** > Search for **Log Analytics API** > **Delegated permissions** > `Data.Read`

Refer to the [Azure documentation](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-configure-app-access-web-apis) for more information.

**Update Grafana scopes:**

Update the `scopes` section in your Grafana Azure authentication configuration to include the `.default` scope:

```
.default openid email profile
```

#### Current User Grafana server configuration

Enable current user authentication in the Grafana server configuration:

```ini
[azure]
user_identity_enabled = true
```

By default, this also enables fallback service credentials. To disable fallback credentials at the instance level:

```ini
[azure]
user_identity_enabled = true
user_identity_fallback_credentials_enabled = false
```

{{< admonition type="caution" >}}
If you've customized the `forward_settings_to_plugins` setting under `[azure]`, verify that `grafana-azure-monitor-datasource` is included. Refer to [Managed Identity Grafana server configuration](#managed-identity-grafana-server-configuration) for details.
{{< /admonition >}}

#### Limitations and fallback credentials

Current User authentication doesn't support backend functionality like alerting, reporting, and recorded queries because user credentials aren't available for background operations.

To support these features, you must configure **fallback service credentials**, typically an App Registration with client secret. When enabled, Grafana uses the fallback credentials for backend operations. Operations using fallback credentials are limited to the permissions of those credentials, not the user's permissions.

{{< admonition type="caution" >}}
If you use Current User authentication without configuring fallback service credentials, alerting rules, recording rules, and reports fail with authentication errors. Refer to the [alerting documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/alerting/) for more details.
{{< /admonition >}}

{{< admonition type="note" >}}
Query and resource caching is disabled by default for data sources using Current User authentication.
{{< /admonition >}}

#### Current User UI configuration

| Setting                          | Description                                                                                 |
| -------------------------------- | ------------------------------------------------------------------------------------------- |
| **Authentication**               | Select **Current User**.                                                                    |
| **Default Subscription**         | Click **Load Subscriptions** to populate available subscriptions, then select your default. |
| **Fallback Service Credentials** | Enable and configure credentials for backend features like alerting.                        |

{{< figure src="/media/docs/grafana/data-sources/screenshot-current-user.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor data source configured with Current User authentication" >}}

#### Provision Current User with YAML

{{< admonition type="note" >}}
The `oauthPassThru` property is required for Current User authentication. The `disableGrafanaCache` property prevents returning cached responses for resources users don't have access to.
{{< /admonition >}}

```yaml
apiVersion: 1

datasources:
  - name: Azure Monitor
    type: grafana-azure-monitor-datasource
    access: proxy
    jsonData:
      azureAuthType: currentuser
      oauthPassThru: true
      disableGrafanaCache: true
      subscriptionId: <subscription-id> # Optional, default subscription
    version: 1
```

## Additional configuration options

These settings apply to all authentication methods.

### General settings

| Setting     | Description                                                                     |
| ----------- | ------------------------------------------------------------------------------- |
| **Name**    | The data source name used in panels and queries. Example: `azure-monitor-prod`. |
| **Default** | Toggle to make this the default data source for new panels.                     |

### Query timeout

The default data source timeout is 30 seconds. Azure Log Analytics queries can take up to 3 minutes on the Azure side, especially for complex KQL queries, large time ranges, or queries across multiple workspaces. If queries fail with `DatasourceError`, `context deadline exceeded`, or timeout errors, increase the timeout.

To change the timeout:

1. Open your Azure Monitor data source settings.
1. Scroll to the **Misc** section at the bottom of the configuration page.
1. Set the **Timeout** value. A value of `300` (300 seconds / 5 minutes) accommodates the maximum Azure Log Analytics server-side query duration.
1. Click **Save & test**.

{{< admonition type="caution" >}}
This timeout also affects alerting evaluations. If your alert rules use Log Analytics queries that exceed 30 seconds, you must increase this setting or alert evaluations fail with timeout errors. For alerting, also ensure your alert evaluation interval is long enough to accommodate the query duration.
{{< /admonition >}}

To set the timeout in a provisioning file, add `timeout` to the `jsonData` section:

```yaml
datasources:
  - name: Azure Monitor
    type: grafana-azure-monitor-datasource
    access: proxy
    jsonData:
      timeout: 300
```

### Enable Basic Logs

Toggle **Enable Basic Logs** to allow queries against [Basic Logs tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1) in supported Log Analytics Workspaces.

{{< admonition type="note" >}}
Querying Basic Logs tables incurs additional costs on a per-query basis.
{{< /admonition >}}

### Enable Batch API

Toggle **Enable Batch API** to query metrics for multiple resources in a single request using the [Azure Monitor Metrics Batch API](https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/migrate-to-batch-api). Instead of sending one request per resource, Grafana groups eligible metric queries and sends them together. This reduces the number of calls to Azure, which improves query performance and lowers the chance of throttling on dashboards that query many resources.

{{< admonition type="note" >}}
This setting is only available when the `datasources.azureMonitorBatchAPI` feature toggle is enabled.
{{< /admonition >}}

Because requests to the Batch API are sent to the `metrics.monitor.azure.com` data plane endpoint, you need the [Monitoring Reader](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/rest-api-walkthrough#retrieve-metric-values-multi-dimensional-api) role at the subscription scope to use it.

{{< admonition type="note" >}}
The Metrics Batch API has a free monthly allowance, after which additional calls incur costs. Refer to [Azure Monitor pricing](https://azure.microsoft.com/en-us/pricing/details/monitor/) for current rates.
{{< /admonition >}}

### Private data source connect (Grafana Cloud only)

If you're using Grafana Cloud and need to connect to Azure resources in a private network, use Private Data Source Connect (PDC).

1. Click the **Private data source connect** dropdown to select your PDC configuration.
1. Click **Manage private data source connect** to view your PDC connection details.

For more information, refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) and [Configure PDC](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc).

### Configure exemplars

You can link Azure Monitor metrics to traces in a tracing data source by configuring exemplars. This lets you navigate from a metric data point directly to related traces for deeper investigation.

To configure exemplars:

1. Open your Azure Monitor data source settings.
1. Scroll to the **Exemplar** configuration section.
1. Click **Add**.
1. Select a **Data source** from the drop-down. The data source must be a tracing backend such as Tempo, Zipkin, Jaeger, or X-Ray.
1. (Optional) Configure a **URL** label to customize the link.
1. Click **Save & test**.

After configuration, exemplar links appear on supported Azure Monitor metrics visualizations, allowing you to jump to the corresponding trace in your tracing data source.

## Supported cloud names

When provisioning the data source, use the following `cloudName` values:

| Azure Cloud                      | `cloudName` value        |
| -------------------------------- | ------------------------ |
| Microsoft Azure public cloud     | `azuremonitor` (default) |
| Microsoft Chinese national cloud | `chinaazuremonitor`      |
| US Government cloud              | `govazuremonitor`        |

{{< admonition type="note" >}}
For Current User authentication, the cloud names differ: use `AzureCloud` for public cloud, `AzureChinaCloud` for the Chinese national cloud, and `AzureUSGovernment` for the US Government cloud.
{{< /admonition >}}

## Verify the connection

After configuring the data source, click **Save & test**. A successful connection displays a message confirming that the credentials are valid and have access to the configured default subscription.

If the test fails, verify:

- Your credentials are correct (tenant ID, client ID, and client secret or certificate fields)
- The identity has the required Azure RBAC permissions
- For Managed Identity or Workload Identity, that the Grafana server configuration is correct
- Network connectivity to Azure endpoints

## Provision the data source

You can define and configure the Azure Monitor data source in YAML files as part of the Grafana provisioning system.
For more information about provisioning, refer to [Provisioning Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

### Provision quick reference

| Authentication method            | `azureAuthType` value | Required fields                                                  |
| -------------------------------- | --------------------- | ---------------------------------------------------------------- |
| App Registration (client secret) | `clientsecret`        | `tenantId`, `clientId`, `clientSecret`                           |
| App Registration (certificate)   | `clientcertificate`   | `tenantId`, `clientId`, `certificateFormat`, `clientCertificate` |
| Managed Identity                 | `msi`                 | None (uses VM identity)                                          |
| Workload Identity                | `workloadidentity`    | None (uses Pod identity)                                         |
| Current User                     | `currentuser`         | `oauthPassThru: true`, `disableGrafanaCache: true`               |

All methods support the optional `subscriptionId` field to set a default subscription.

For complete YAML examples, see the [authentication method sections](#configure-authentication) above.

## Configure with Terraform

You can configure the Azure Monitor data source using the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). This approach enables infrastructure-as-code workflows and version control for your Grafana configuration.

### Terraform prerequisites

- [Terraform](https://www.terraform.io/downloads) installed
- Grafana Terraform provider configured with appropriate credentials
- For Grafana Cloud: A [Cloud Access Policy token](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) with data source permissions

### Provider configuration

Configure the Grafana provider to connect to your Grafana instance:

```hcl
terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 2.0.0"
    }
  }
}

# For Grafana Cloud
provider "grafana" {
  url  = "<YOUR_GRAFANA_CLOUD_STACK_URL>"
  auth = "<YOUR_SERVICE_ACCOUNT_TOKEN>"
}

# For self-managed Grafana
# provider "grafana" {
#   url  = "http://localhost:3000"
#   auth = "<API_KEY_OR_SERVICE_ACCOUNT_TOKEN>"
# }
```

### Terraform examples

The following examples show how to configure the Azure Monitor data source for each authentication method.

**App Registration (client secret):**

```hcl
resource "grafana_data_source" "azure_monitor" {
  type = "grafana-azure-monitor-datasource"
  name = "Azure Monitor"

  json_data_encoded = jsonencode({
    azureAuthType  = "clientsecret"
    cloudName      = "azuremonitor"
    tenantId       = "<TENANT_ID>"
    clientId       = "<CLIENT_ID>"
    subscriptionId = "<SUBSCRIPTION_ID>"
  })

  secure_json_data_encoded = jsonencode({
    clientSecret = "<CLIENT_SECRET>"
  })
}
```

**App Registration (client certificate with PEM):**

```hcl
resource "grafana_data_source" "azure_monitor" {
  type = "grafana-azure-monitor-datasource"
  name = "Azure Monitor"

  json_data_encoded = jsonencode({
    azureAuthType     = "clientcertificate"
    cloudName         = "azuremonitor"
    tenantId          = "<TENANT_ID>"
    clientId          = "<CLIENT_ID>"
    certificateFormat = "pem"
    subscriptionId    = "<SUBSCRIPTION_ID>"
  })

  secure_json_data_encoded = jsonencode({
    clientCertificate = "<CERTIFICATE_CONTENT>"
    privateKey        = "<PRIVATE_KEY_CONTENT>"
  })
}
```

**App Registration (client certificate with PFX):**

```hcl
resource "grafana_data_source" "azure_monitor" {
  type = "grafana-azure-monitor-datasource"
  name = "Azure Monitor"

  json_data_encoded = jsonencode({
    azureAuthType     = "clientcertificate"
    cloudName         = "azuremonitor"
    tenantId          = "<TENANT_ID>"
    clientId          = "<CLIENT_ID>"
    certificateFormat = "pfx"
    subscriptionId    = "<SUBSCRIPTION_ID>"
  })

  secure_json_data_encoded = jsonencode({
    clientCertificate = "<CERTIFICATE_CONTENT_BASE64_ENCODED>"
    certificatePassword = "<CERTIFICATE_PASSWORD>"
  })
}
```

**Managed Identity:**

```hcl
resource "grafana_data_source" "azure_monitor" {
  type = "grafana-azure-monitor-datasource"
  name = "Azure Monitor"

  json_data_encoded = jsonencode({
    azureAuthType  = "msi"
    subscriptionId = "<SUBSCRIPTION_ID>"
  })
}
```

**Workload Identity:**

```hcl
resource "grafana_data_source" "azure_monitor" {
  type = "grafana-azure-monitor-datasource"
  name = "Azure Monitor"

  json_data_encoded = jsonencode({
    azureAuthType  = "workloadidentity"
    subscriptionId = "<SUBSCRIPTION_ID>"
  })
}
```

**Current User:**

```hcl
resource "grafana_data_source" "azure_monitor" {
  type = "grafana-azure-monitor-datasource"
  name = "Azure Monitor"

  json_data_encoded = jsonencode({
    azureAuthType       = "currentuser"
    oauthPassThru       = true
    disableGrafanaCache = true
    subscriptionId      = "<SUBSCRIPTION_ID>"
  })
}
```

**With Basic Logs enabled:**

Add `basicLogsEnabled = true` to any of the above configurations:

```hcl
resource "grafana_data_source" "azure_monitor" {
  type = "grafana-azure-monitor-datasource"
  name = "Azure Monitor"

  json_data_encoded = jsonencode({
    azureAuthType   = "clientsecret"
    cloudName       = "azuremonitor"
    tenantId        = "<TENANT_ID>"
    clientId        = "<CLIENT_ID>"
    subscriptionId  = "<SUBSCRIPTION_ID>"
    basicLogsEnabled = true
  })

  secure_json_data_encoded = jsonencode({
    clientSecret = "<CLIENT_SECRET>"
  })
}
```

For more information about the Grafana Terraform provider, refer to the [provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs) and the [`grafana_data_source` resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source).

# Azure Monitor query editor

Grafana provides a query editor for the Azure Monitor data source, which is located on the [Explore page](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/). You can also access the Azure Monitor query editor from a dashboard panel. Click the menu in the upper right of the panel and select **Edit**.

This document explains querying specific to the Azure Monitor data source.
For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/).

The Azure Monitor data source can query data from Azure Monitor Metrics and Logs, the Azure Resource Graph, and Application Insights Traces. Each source has its own specialized query editor.

## Before you begin

- Ensure you have [configured the Azure Monitor data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/configure/).
- Verify your credentials have appropriate permissions for the resources you want to query.

## Key concepts

If you're new to Azure Monitor, here are some key terms used throughout this documentation:

| Term                           | Description                                                                                                                                                                                                                                                                                                                               |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **KQL (Kusto Query Language)** | The query language used for Azure Monitor Logs and Azure Resource Graph. KQL uses a pipe-based syntax similar to Unix commands and is optimized for read-only data exploration. If you know SQL, the [SQL to Kusto cheat sheet](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/sqlcheatsheet) can help you get started. |
| **Log Analytics workspace**    | An Azure resource that collects and stores log data from your Azure resources, applications, and services. You query this data using KQL.                                                                                                                                                                                                 |
| **Application Insights**       | Azure's application performance monitoring (APM) service. It collects telemetry data like requests, exceptions, and traces from your applications.                                                                                                                                                                                        |
| **Metrics vs. Logs**           | **Metrics** are lightweight numeric values collected at regular intervals (for example, CPU percentage). **Logs** are detailed records of events with varying schemas (for example, request logs, error messages). Metrics use a visual query builder; Logs require KQL.                                                                  |

## Choose a query editor mode

The Azure Monitor data source's query editor has four modes depending on which Azure service you want to query:

- **Metrics** for [Azure Monitor Metrics](#query-azure-monitor-metrics)
- **Logs** for [Azure Monitor Logs](#query-azure-monitor-logs)
- **Traces** for [Application Insights Traces](#query-application-insights-traces)
- **Azure Resource Graph** for [Azure Resource Graph](#query-azure-resource-graph)

## Query Azure Monitor Metrics

Azure Monitor Metrics collects numeric data from [supported resources](https://docs.microsoft.com/en-us/azure/azure-monitor/monitor-reference), and you can query them to investigate your resources' health and usage and maximize availability and performance.

Monitor Metrics use a lightweight format that stores only numeric data in a specific structure and supports near real-time scenarios, making it useful for fast detection of issues.
In contrast, Azure Monitor Logs can store a variety of data types, each with their own structure.

{{< figure src="/static/img/docs/azure-monitor/query-editor-metrics.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor Metrics sample query visualizing CPU percentage over time" >}}

### Create a Metrics query

**To create a Metrics query:**

1. In a Grafana panel, select the **Azure Monitor** data source.
1. Select the **Metrics** service.
1. Select a resource from which to query metrics by using the subscription, resource group, resource type, and resource fields. Multiple resources can also be selected as long as they belong to the same subscription, region and resource type. Note that only a limited amount of resource types support this feature.
1. To select a different namespace than the default—for instance, to select resources like storage accounts that are organized under multiple namespaces—use the **Namespace** option.

   {{< admonition type="note" >}}
   Not all metrics returned by the Azure Monitor Metrics API have values.
   {{< /admonition >}}

   > The data source retrieves lists of supported metrics for each subscription and ignores metrics that never have values.

1. Select a metric from the **Metric** field.

Optionally, you can apply further aggregations or filter by dimensions.

1. Change the aggregation from the default average to show minimum, maximum, or total values.
1. Specify a custom time grain. By default, Grafana selects a time grain interval for you based on your selected time range.
1. For metrics with multiple dimensions, you can split and filter the returned metrics.
   For example, the Application Insights dependency calls metric supports returning multiple time series for successful and unsuccessful calls.

{{< figure src="/static/img/docs/azure-monitor/query-editor-metrics-dimensions.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor Metrics screenshot showing Dimensions" >}}

The available options change depending on what is relevant to the selected metric.

You can also augment queries by using [template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/template-variables/).

### Format legend aliases

You can change the legend label for Metrics by using aliases.
In the Legend Format field, you can combine aliases defined below any way you want.

For example:

- `Blob Type: {{ blobtype }}` becomes `Blob Type: PageBlob`, `Blob Type: BlockBlob`
- `{{ resourcegroup }} - {{ resourcename }}` becomes `production - web_server`

| Alias pattern                  | Description                                                                                            |
| ------------------------------ | ------------------------------------------------------------------------------------------------------ |
| `{{ subscriptionid }}`         | Replaced with the subscription ID.                                                                     |
| `{{ subscription }}`           | Replaced with the subscription name.                                                                   |
| `{{ resourcegroup }}`          | Replaced with the resource group.                                                                      |
| `{{ namespace }}`              | Replaced with the resource type or namespace, such as `Microsoft.Compute/virtualMachines`.             |
| `{{ resourcename }}`           | Replaced with the resource name.                                                                       |
| `{{ metric }}`                 | Replaced with the metric name, such as "Percentage CPU".                                               |
| _`{{ arbitraryDimensionID }}`_ | Replaced with the value of the specified dimension. For example, `{{ blobtype }}` becomes `BlockBlob`. |
| `{{ dimensionname }}`          | _(Legacy for backward compatibility)_ Replaced with the name of the first dimension.                   |
| `{{ dimensionvalue }}`         | _(Legacy for backward compatibility)_ Replaced with the value of the first dimension.                  |

### Filter with dimensions

Some metrics also have dimensions, which associate additional metadata.
Dimensions are represented as key-value pairs assigned to each value of a metric.
Grafana can display and filter metrics based on dimension values.

The data source supports the `equals`, `not equals`, and `starts with` operators as detailed in the [Monitor Metrics API documentation](https://docs.microsoft.com/en-us/rest/api/monitor/metrics/list).

For more information on multi-dimensional metrics, refer to the [Azure Monitor data platform metrics documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/data-platform-metrics#multi-dimensional-metrics) and [Azure Monitor filtering documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-charts#filters).

## Query Azure Monitor Logs

Azure Monitor Logs collects and organizes log and performance data from [supported resources](https://docs.microsoft.com/en-us/azure/azure-monitor/monitor-reference), and makes many sources of data available to query together with the [Kusto Query Language (KQL)](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/).

While Azure Monitor Metrics stores only simplified numerical data, Logs can store different data types, each with their own structure.
You can also perform complex analysis of Logs data by using KQL.

The Azure Monitor data source also supports querying of [Basic Logs](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1) tables (if they exist in your Log Analytics workspace). This feature must be enabled in the data source configuration.

{{< figure src="/static/img/docs/azure-monitor/query-editor-logs.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor Logs sample query comparing successful requests to failed requests" >}}

### Logs query builder (public preview)

{{< admonition type="note" >}}
The Logs query builder is a [public preview feature](/docs/release-life-cycle/). It may not be enabled in all Grafana environments.
{{< /admonition >}}

The Logs query builder provides a visual interface for building Azure Monitor Logs queries without writing KQL. This is helpful if you're new to KQL or want to quickly build queries.

**To enable the Logs query builder:**

1. Enable the `azureMonitorLogsBuilderEditor` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/) in your Grafana configuration.
1. Restart Grafana for the change to take effect.

**To switch between Builder and Code modes:**

When the feature is enabled, a **Builder / Code** toggle appears in the Logs query editor:

- **Builder**: Use the visual interface to select tables, columns, filters, and aggregations. The builder generates the KQL query for you.
- **Code**: Write KQL queries directly. Use this mode for complex queries that require full KQL capabilities.

New queries default to Builder mode. Existing queries that were created with raw KQL remain in Code mode. When you switch to Builder mode, the time-range setting automatically changes to **Dashboard** because the builder manages time filtering for you.

{{< admonition type="note" >}}
You can switch from Builder to Code mode at any time to view or edit the generated KQL. However, switching from Code to Builder mode may not preserve complex queries that can't be represented in the builder interface.
{{< /admonition >}}

### Create a Logs query

**To create a Logs query:**

1. In a Grafana panel, select the **Azure Monitor** data source.
1. Select the **Logs** service.
1. Select a resource to query. Multiple resources can be selected as long as they are of the same type.

   Alternatively, you can dynamically query all resources under a single resource group or subscription.
   {{< admonition type="note" >}}
   If a time span is specified in the query, the overlap between the query time span and the dashboard time range is used. Refer to the [API documentation for
   details.](https://learn.microsoft.com/en-us/rest/api/loganalytics/dataaccess/query/get?tabs=HTTP#uri-parameters)
   {{< /admonition >}}

1. Enter your KQL query.

You can also augment queries by using [template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/template-variables/).

**To create a Basic Logs query:**

1. Ensure that the data source has the **Enable Basic Logs** toggle enabled in the [data source configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/configure/#enable-basic-logs).
1. In a Grafana panel, select the **Azure Monitor** data source.
1. Select the **Logs** service.
1. Select a single resource to query. Basic Logs queries only support a single workspace resource.
1. Switch the **Logs** toggle from **Analytics** to **Basic**. A confirmation dialog box displays to notify you of potential additional costs.
1. Enter your KQL query.

{{< admonition type="caution" >}}
Basic Logs queries are billed on a per-query basis by Azure. Review the [Azure pricing documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1) before using this feature.
{{< /admonition >}}

Basic Logs queries have the following restrictions:

- **Time range:** The dashboard time range is always used. You can't specify a time range within the query itself.
- **Single resource:** Only a single Log Analytics workspace can be queried. Multi-resource selection isn't supported.
- **Alerting:** Basic Logs queries aren't available when creating alert rules.
- **KQL limitations:** Some KQL operators aren't supported. Refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1#limitations) for a full list of limitations.

You can also augment queries by using [template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/template-variables/).

### Logs result row limit

When a Logs query uses the **Logs** result format, Grafana truncates the results to 30,000 rows to prevent the browser tab from crashing, because the logs visualization renders every row at once. When truncation happens, Grafana returns the first 30,000 rows along with a notice that the limit was reached.

To remove this limit, enable the `azureMonitorDisableLogLimit` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/) in your Grafana configuration.

{{< admonition type="caution" >}}
Disabling the row limit can degrade browser performance for very large result sets. Prefer aggregating with `summarize` or narrowing the time range to reduce the number of rows returned.
{{< /admonition >}}

### Logs query examples

Azure Monitor Logs queries are written using the Kusto Query Language (KQL), a rich language similar to SQL.

The Azure documentation includes resources to help you learn KQL:

- [Log queries in Azure Monitor](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/log-query-overview)
- [Getting started with Kusto](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/concepts/)
- [Tutorial: Use Kusto queries in Azure Monitor](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/tutorial?pivots=azuremonitor)
- [SQL to Kusto cheat sheet](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/sqlcheatsheet)

{{< admonition type="note" >}}
**Time-range:** The time-range used for the query can be modified via the time-range switch:

- Selecting **Query** uses only time-ranges specified within the query.
- Selecting **Dashboard** uses only the Grafana dashboard time-range.
- If no time-range is specified in the query, the default Log Analytics time-range applies.

For more details, refer to the [Azure Monitor Logs API documentation](https://learn.microsoft.com/en-us/rest/api/loganalytics/dataaccess/query/get?tabs=HTTP#uri-parameters). If you previously used the `Intersection` option, it has been migrated to `Dashboard`.
{{< /admonition >}}

This example query returns a virtual machine's CPU performance, averaged over 5-minute time grains:

```kusto
Perf
// $__timeFilter is a special Grafana macro that filters the results to the time span of the dashboard
| where $__timeFilter(TimeGenerated)
| where CounterName == "% Processor Time"
| summarize avg(CounterValue) by bin(TimeGenerated, 5m), Computer
| order by TimeGenerated asc
```

Use time series queries for values that change over time, usually for graph visualizations such as the Time series panel.
Each query should return at least a `datetime` column and numeric value column.
The result must also be sorted in ascending order by the `datetime` column.

You can also create a query with at least one non-numeric, non-`datetime` column.
Azure Monitor considers those columns to be dimensions, and they become labels in the response.

For example, this query returns the aggregated count grouped by hour, Computer, and the CounterName:

```kusto
Perf
| where $__timeFilter(TimeGenerated)
| summarize count() by bin(TimeGenerated, 1h), Computer, CounterName
| order by TimeGenerated asc
```

You can also select additional number value columns, with or without multiple dimensions.
For example, this query returns a count and average value by hour, Computer, CounterName, and InstanceName:

```kusto
Perf
| where $__timeFilter(TimeGenerated)
| summarize Samples=count(), ["Avg Value"]=avg(CounterValue)
    by bin(TimeGenerated, $__interval), Computer, CounterName, InstanceName
| order by TimeGenerated asc
```

Use table queries with the Table panel to produce a list of columns and rows.
This query returns rows with the six specified columns:

```kusto
AzureActivity
| where $__timeFilter()
| project TimeGenerated, ResourceGroup, Category, OperationName, ActivityStatus, Caller
| order by TimeGenerated desc
```

### Use macros in Logs queries

To help you write queries, you can use several Grafana macros in the `where` clause:

| Macro                           | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `$__timeFilter()`               | Filters the results to the time range of the dashboard. Example: `TimeGenerated >= datetime(2018-06-05T18:09:58.907Z) and TimeGenerated <= datetime(2018-06-05T20:09:58.907Z)`.                                                                                                                                                                                                                                                                                                |
| `$__timeFilter(datetimeColumn)` | Like `$__timeFilter()`, but specifies a custom field to filter on.                                                                                                                                                                                                                                                                                                                                                                                                             |
| `$__timeFrom()`                 | Returns the start of the dashboard time range. Example: `datetime(2018-06-05T18:09:58.907Z)`.                                                                                                                                                                                                                                                                                                                                                                                  |
| `$__timeTo()`                   | Returns the end of the dashboard time range. Example: `datetime(2018-06-05T20:09:58.907Z)`.                                                                                                                                                                                                                                                                                                                                                                                    |
| `$__escapeMulti($myVar)`        | Escapes illegal characters in multi-value template variables. If `$myVar` has the values `'\\grafana-vm\Network(eth0)\Total','\\hello!'` as a string, use this to expand it to `@'\\grafana-vm\Network(eth0)\Total', @'\\hello!'`. If using single-value variables, escape the variable inline instead: `@'\$myVar'`.                                                                                                                                                          |
| `$__contains(colName, $myVar)`  | Expands multi-value template variables. If `$myVar` has the value `'value1','value2'`, use this to expand it to `colName in ('value1','value2')`. If using the `All` option, check the `Include All Option` checkbox, and type the value `all` in the `Custom all value` field. If `$myVar` has the value `all`, the macro instead expands to `1 == 1`. For template variables with many options, this avoids building a large "where..in" clause, which improves performance. |

Additionally, Grafana has the built-in [`$__interval` macro](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/#query-options), which calculates an interval in seconds.

## Query Azure Resource Graph

Azure Resource Graph (ARG) is an Azure service designed to extend Azure Resource Management with efficient resource exploration and the ability to query at scale across a set of subscriptions, so that you can more effectively govern an environment.
By querying ARG, you can query resources with complex filtering, iteratively explore resources based on governance requirements, and assess the impact of applying policies in a vast cloud environment.

{{< figure src="/static/img/docs/azure-monitor/query-editor-arg.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Resource Graph sample query listing virtual machines on an account" >}}

### Create a Resource Graph query

ARG queries are written in a variant of the [Kusto Query Language (KQL)](https://docs.microsoft.com/en-us/azure/governance/resource-graph/concepts/query-language), but not all Kusto language features are available in ARG.

An Azure Resource Graph query is formatted as table data.

If your Azure credentials grant you access to multiple subscriptions, you can choose multiple subscriptions before entering queries. It is also possible to run queries against the directory by changing the scope of the query.

{{< admonition type="note" >}}
Some queries that function at a directory level may not work at a subscription level and vice-versa.
{{< /admonition >}}

### Resource Graph query examples

The Azure documentation also includes [sample queries](https://docs.microsoft.com/en-gb/azure/governance/resource-graph/samples/starter) to help you get started.

**Sort results by resource properties:**

This query returns all resources in the selected subscriptions, but only the name, type, and location properties:

```kusto
Resources
| project name, type, location
| order by name asc
```

This query uses `order by` to sort the properties by the `name` property in ascending (`asc`) order.
You can change which property to sort by and the order (`asc` or `desc`).
This query uses `project` to show only the listed properties in the results.
You can use this to add or remove properties in your queries.

**Query resources with complex filtering:**

You can filter for Azure resources with a tag name and value.

For example, this query returns a list of resources with an `environment` tag value of `Internal`:

```kusto
Resources
| where tags.environment=~'internal'
| project name
```

This query uses `=~` in the `type` match to make the query case-insensitive.
You can also use `project` with other properties, or add or remove more.

**Group and aggregate the values by property:**

You can use `summarize` and `count` to define how to group and aggregate values by property.

For example, this query returns counts of healthy, unhealthy, and not applicable resources per recommendation:

```kusto
securityresources
| where type == 'microsoft.security/assessments'
| extend resourceId=id,
    recommendationId=name,
    resourceType=type,
    recommendationName=properties.displayName,
    source=properties.resourceDetails.Source,
    recommendationState=properties.status.code,
    description=properties.metadata.description,
    assessmentType=properties.metadata.assessmentType,
    remediationDescription=properties.metadata.remediationDescription,
    policyDefinitionId=properties.metadata.policyDefinitionId,
    implementationEffort=properties.metadata.implementationEffort,
    recommendationSeverity=properties.metadata.severity,
    category=properties.metadata.categories,
    userImpact=properties.metadata.userImpact,
    threats=properties.metadata.threats,
    portalLink=properties.links.azurePortal
| summarize numberOfResources=count(resourceId) by tostring(recommendationName), tostring(recommendationState)
```

In ARG, many nested properties (`properties.displayName`) are of a `dynamic` type and should be cast to a string with `tostring()` to operate on them.

### Use macros in Resource Graph queries

To help you write queries, you can use several Grafana macros in the `where` clause:

| Macro                           | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `$__timeFilter()`               | Expands to `timestamp ≥ datetime(2018-06-05T18:09:58.907Z) and timestamp ≤ datetime(2018-06-05T20:09:58.907Z)`, where the from and to date-time values are from the Grafana time picker.                                                                                                                                                                                                                                                                                          |
| `$__timeFilter(datetimeColumn)` | Expands to `datetimeColumn ≥ datetime(2018-06-05T18:09:58.907Z) and datetimeColumn ≤ datetime(2018-06-05T20:09:58.907Z)`, where the from and to date-time values are from the Grafana time picker.                                                                                                                                                                                                                                                                                |
| `$__timeFrom()`                 | Returns the start of the dashboard time range from the Grafana picker. Example: `datetime(2018-06-05T18:09:58.907Z)`.                                                                                                                                                                                                                                                                                                                                                             |
| `$__timeTo()`                   | Returns the end of the dashboard time range from the Grafana picker. Example: `datetime(2018-06-05T20:09:58.907Z)`.                                                                                                                                                                                                                                                                                                                                                               |
| `$__escapeMulti($myVar)`        | Escapes illegal characters from multi-value template variables. If `$myVar` has the values `'\\grafana-vm\Network(eth0)\Total','\\hello!'` as a string, this expands it to `@'\\grafana-vm\Network(eth0)\Total', @'\\hello!'`. If you use single-value variables, escape the variable inline instead: `@'\$myVar'`.                                                                                                                                                               |
| `$__contains(colName, $myVar)`  | Expands multi-value template variables. If `$myVar` has the value `'value1','value2'`, this expands it to `colName in ('value1','value2')`. If using the `All` option, then check the `Include All Option` checkbox and in the `Custom all value` field type in the following value: `all`. If `$myVar` has value `all`, this instead expands to `1 == 1`. For template variables with many options, this avoids building a large "where..in" clause, which improves performance. |

## Query Application Insights Traces

[Azure Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview?tabs=net) is a service that provides application performance monitoring (APM) features. Application insights can be used to collect metrics, telemetry, and trace logging data.

Application Insights stores trace data in an underlying Log Analytics workspace and traces can be [extended](https://learn.microsoft.com/en-us/azure/azure-monitor/app/api-custom-events-metrics) to provide additional supporting information as required by the application developer.

### Create a Traces query

**To create a Traces query:**

1. In a Grafana panel, select the **Azure Monitor** data source.
1. Select the **Traces** service.
1. Select a resource to query. Multiple resources can be selected as long as they are of the same type.

   {{< admonition type="note" >}}
   This query type only supports Application Insights resources.
   {{< /admonition >}}

1. (Optional) Specify an **Operation ID** value to filter traces.
1. (Optional) Specify **event types** to filter by.
1. (Optional) Specify **event properties** to filter by.
1. (Optional) Change the **Result format** to switch between tabular format and trace format.

   {{< admonition type="note" >}}
   Selecting the trace format filters events to only the `trace` type. Use this format with the Trace visualization.
   {{< /admonition >}}

Running a query returns all trace data within the time span specified by the panel or dashboard time range.

You can also augment queries by using [template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/template-variables/).

## Use queries for alerting and recording rules

All Azure Monitor query types (Metrics, Logs, Azure Resource Graph, and Traces) can be used with Grafana Alerting and recording rules.

For detailed information about creating alert rules, supported query types, authentication requirements, and examples, refer to [Azure Monitor alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/alerting/).

## Work with large Azure resource datasets

If a request exceeds the [maximum allowed value of records](https://docs.microsoft.com/en-us/azure/governance/resource-graph/concepts/work-with-data#paging-results), the result is paginated and only the first page of results are returned.
You can use filters to reduce the amount of records returned under that value.

## Next steps

- [Use template variables](../template-variables/) to create dynamic, reusable dashboards
- [Add annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/annotations/) to overlay events on your graphs
- [Set up alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/alerting/) to create alert rules based on Azure Monitor data
- [Troubleshoot](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/troubleshooting/) common query and configuration issues

# Azure Monitor template variables

Instead of hard-coding details such as resource group or resource name values in metric queries, you can use variables.
This helps you create more interactive, dynamic, and reusable dashboards.
Grafana refers to such variables as template variables.

For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/).

## Before you begin

- Ensure you have [configured the Azure Monitor data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/configure/).
- If you want template variables to auto-populate subscriptions, set a **Default Subscription** in the data source configuration.

## Create a template variable

To create a template variable for Azure Monitor:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Variable**.
1. Select **Query** as the variable type.
1. Enter a **Name** for your variable (for example, `subscription`, `resourceGroup`, or `resource`).
1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed.
1. Click **Open variable editor** to open the **Query Variable** dialog box.
1. In the **Data source** drop-down list, select your Azure Monitor data source.
1. In the **Query Type** drop-down list, select the appropriate query type (refer to [Available query types](#available-query-types)).
1. Configure any additional fields required by the selected query type.
1. Click **Preview** to preview the variable values.
1. Click **Close** to close the query variable editor dialog box.
1. Configure display options such as **Multi-value**, **Include All value**, or **Allow custom values** as needed.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## Available query types

The Azure Monitor data source provides the following query types for template variables:

| Query type              | Description                                                                                                                            |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Subscriptions**       | Returns a list of Azure subscriptions accessible to the configured credentials.                                                        |
| **Resource Groups**     | Returns resource groups for a specified subscription. Supports multi-value selection.                                                  |
| **Namespaces**          | Returns metric namespaces for the specified subscription. If a resource group is specified, returns only namespaces within that group. |
| **Regions**             | Returns Azure regions available for the specified subscription.                                                                        |
| **Resource Names**      | Returns resource names for a specified subscription, resource group, and namespace. Supports multi-value selection.                    |
| **Metric Names**        | Returns available metric names for a specified resource.                                                                               |
| **Workspaces**          | Returns Log Analytics workspaces for the specified subscription.                                                                       |
| **Logs**                | Executes a KQL query and returns the results as variable values. Refer to [Create a Logs variable](#create-a-logs-variable).           |
| **Custom Namespaces**   | Returns custom metric namespaces for a specified resource.                                                                             |
| **Custom Metric Names** | Returns custom metric names for a specified resource.                                                                                  |

{{< admonition type="note" >}}
Custom metrics cannot be emitted against a subscription or resource group. Select specific resources when retrieving custom metric namespaces or custom metric names.
{{< /admonition >}}

## Create cascading variables

Cascading variables (also called dependent or chained variables) allow you to create dropdown menus that filter based on previous selections. This is useful for drilling down from subscription to resource group to specific resource.

### Example: Subscription → Resource Group → Resource Name

**Step 1: Create a Subscription variable**

1. Create a variable named `subscription`.
1. Set **Query Type** to **Subscriptions**.

**Step 2: Create a Resource Group variable**

1. Create a variable named `resourceGroup`.
1. Set **Query Type** to **Resource Groups**.
1. In the **Subscription** field, select `$subscription`.

**Step 3: Create a Resource Name variable**

1. Create a variable named `resource`.
1. Set **Query Type** to **Resource Names**.
1. In the **Subscription** field, select `$subscription`.
1. In the **Resource Group** field, select `$resourceGroup`.
1. Select the appropriate **Namespace** for your resources (for example, `Microsoft.Compute/virtualMachines`).

Now when you change the subscription, the resource group dropdown updates automatically, and when you change the resource group, the resource name dropdown updates.

## Create a Logs variable

The **Logs** query type lets you use a KQL query to populate variable values. The query must return a single column of values.

**To create a Logs variable:**

1. Create a new variable with **Query Type** set to **Logs**.
1. Select a **Resource** (Log Analytics workspace or Application Insights resource).
1. Enter a KQL query that returns a single column.

### Logs variable query examples

| Query                                     | Returns                               |
| ----------------------------------------- | ------------------------------------- |
| `Heartbeat \| distinct Computer`          | List of virtual machine names         |
| `Perf \| distinct ObjectName`             | List of performance object names      |
| `AzureActivity \| distinct ResourceGroup` | List of resource groups with activity |
| `AppRequests \| distinct Name`            | List of application request names     |

You can reference other variables in your Logs query:

```kusto
workspace("$workspace").Heartbeat | distinct Computer
```

```kusto
workspace("$workspace").Perf
| where ObjectName == "$object"
| distinct CounterName
```

## Variable refresh options

Control when your variables refresh by setting the **Refresh** option:

| Option                   | Behavior                                                                                  |
| ------------------------ | ----------------------------------------------------------------------------------------- |
| **On dashboard load**    | Variables refresh each time the dashboard loads. Best for data that changes infrequently. |
| **On time range change** | Variables refresh when the dashboard time range changes. Use for time-sensitive queries.  |

For dashboards with many variables or complex queries, use **On dashboard load** to improve performance.

## Use variables in queries

After you create template variables, you can use them in your Azure Monitor queries by referencing them with the `$` prefix.

### Metrics query example

In a Metrics query, select your variables in the resource picker fields:

- **Subscription**: `$subscription`
- **Resource Group**: `$resourceGroup`
- **Resource Name**: `$resource`

### Logs query example

Reference variables directly in your KQL queries:

```kusto
Perf
| where ObjectName == "$object" and CounterName == "$metric"
| where TimeGenerated >= $__timeFrom() and TimeGenerated <= $__timeTo()
| where $__contains(Computer, $computer)
| summarize avg(CounterValue) by bin(TimeGenerated, $__interval), Computer
| order by TimeGenerated asc
```

## Multi-value variables

You can enable **Multi-value** selection for **Resource Groups** and **Resource Names** variables. When using multi-value variables in a Metrics query, all selected resources must:

- Belong to the same subscription
- Be in the same Azure region
- Be of the same resource type (namespace)

{{< admonition type="note" >}}
When a multi-value variable is used as a parameter in another variable query (for example, to retrieve metric names), only the first selected value is used. Ensure the first resource group and resource name combination is valid.
{{< /admonition >}}

## Troubleshoot template variables

If you encounter issues with template variables, try the following solutions. For additional troubleshooting, refer to the [Azure Monitor troubleshooting guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/troubleshooting/#template-variable-errors).

### Variable returns no values

- Verify the Azure Monitor data source is configured correctly and can connect to Azure.
- Check that the credentials have appropriate permissions to list the requested resources.
- For cascading variables, ensure parent variables have valid selections.

### Variable values are outdated

- Check the **Refresh** setting and adjust if needed.
- Click the refresh icon next to the variable dropdown to manually refresh.

### Variables break after a Grafana upgrade

If template variables return errors such as `Properties found in series but missing valueProp and textProp` after upgrading Grafana, this typically indicates a version-specific bug. Update to the latest patch release for your Grafana version. For Grafana Cloud, verify you're on the **stable** release channel. Refer to [Template variable errors](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/troubleshooting/#template-variable-errors) for detailed solutions.

### Multi-value selection not working in queries

- Ensure the resources meet the requirements (same subscription, region, and type).
- For Logs queries, use the `$__contains()` macro to handle multi-value variables properly.

# Troubleshoot Azure Monitor data source issues

This document provides solutions to common issues you may encounter when configuring or using the Azure Monitor data source, organized by the stage where the issue occurs.

## Connection and network errors

These errors indicate Grafana can't reach Azure endpoints. Verify connectivity before investigating authentication or query issues.

### "Connection refused" or timeout errors

**Symptoms:**

- Data source test fails with network errors
- Queries timeout without returning results

**Solutions:**

1. Verify network connectivity from Grafana to Azure endpoints.
1. Check firewall rules allow outbound HTTPS (port 443) to Azure services.
1. For private networks, ensure Private Link or VPN is configured correctly.
1. For Grafana Cloud, configure [Private Data Source Connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if accessing private resources.

### SSL/TLS certificate errors

**Symptoms:**

- Certificate validation failures
- SSL handshake errors

**Solutions:**

1. Ensure the system time is correct (certificate validation fails with incorrect time).
1. Verify corporate proxy isn't intercepting HTTPS traffic.
1. Check that required CA certificates are installed on the Grafana server.

## Authentication errors

These errors occur when credentials configured in the data source or in Azure are invalid, expired, or missing permissions.

### "Authorization failed" or "Access denied"

**Symptoms:**

- Save & test fails with "Authorization failed"
- Queries return "Access denied" errors
- Subscriptions don't load when clicking **Load Subscriptions**

**Possible causes and solutions:**

| Cause                                              | Solution                                                                                                                                                                                                                                                                    |
| -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| App registration doesn't have required permissions | Assign the `Reader` role to the app registration on the subscription or resource group you want to monitor. Refer to the [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). |
| Incorrect tenant ID, client ID, or client secret   | Verify the credentials in the Azure Portal under **App registrations** > your app > **Overview** (for IDs) and **Certificates & secrets** (for secret).                                                                                                                     |
| Client secret has expired                          | Create a new client secret in Azure and update the data source configuration.                                                                                                                                                                                               |
| Managed Identity not enabled on the Azure resource | For VMs, enable managed identity in the Azure Portal under **Identity**. For App Service, enable it under **Identity** in the app settings.                                                                                                                                 |
| Managed Identity not assigned the Reader role      | Assign the `Reader` role to the managed identity on the target subscription or resources.                                                                                                                                                                                   |

### "Invalid client secret" or "Client secret not found"

**Symptoms:**

- Authentication fails immediately after configuration
- Error message references invalid credentials

**Solutions:**

1. Ensure you copied the client secret **value**, not the secret ID. In Azure Portal under **Certificates & secrets**, the secret value is only shown once when created. The secret ID is a different identifier and won't work for authentication.
1. Verify the client secret was copied correctly (no extra spaces or truncation).
1. Check if the secret has expired in Azure Portal under **App registrations** > your app > **Certificates & secrets**.
1. Create a new secret and update the data source configuration.

### "Tenant not found" or "Invalid tenant ID"

**Symptoms:**

- Data source test fails with tenant-related errors
- Unable to authenticate

**Solutions:**

1. Verify the Directory (tenant) ID in Azure Portal under **Microsoft Entra ID** > **Overview**.
1. Ensure you're using the correct Azure cloud setting (Azure, Azure Government, or Azure China).
1. Check that the tenant ID is written using a valid GUID format.

### Client certificate authentication not working

**Symptoms:**

- Authentication fails when using App Registration with client certificate
- Error references invalid certificate or signature validation failure

**Possible causes and solutions:**

| Cause                                     | Solution                                                                                                                                                        |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Certificate has expired                   | Check the certificate expiration date in Azure Portal under **App registrations** > your app > **Certificates & secrets**. Upload a new certificate if expired. |
| Wrong certificate format                  | Verify you're using the correct format (PEM or PFX) and that the content matches what you selected. PEM requires separate certificate and private key fields.   |
| PFX password is incorrect                 | For encrypted PFX certificates, verify the password is correct.                                                                                                 |
| Private key doesn't match the certificate | Ensure the private key corresponds to the certificate uploaded to Azure. Regenerate the certificate and key pair if needed.                                     |
| Certificate not uploaded to Azure         | Verify the public certificate is uploaded to the app registration under **Certificates & secrets** > **Certificates** in the Azure Portal.                      |

## Grafana server configuration errors

These errors occur when the Grafana server `.ini` configuration is missing settings required by Managed Identity, Workload Identity, or Current User authentication. If your Azure credentials are correct but authentication still fails, check this section.

### Managed Identity not working

**Symptoms:**

- Managed Identity option is available but authentication fails
- Error: "Managed identity authentication is not available"

**Solutions:**

1. Verify `managed_identity_enabled = true` is set in the Grafana server configuration under `[azure]`.
1. Confirm the Azure resource hosting Grafana has managed identity enabled.
1. For user-assigned managed identity, ensure `managed_identity_client_id` is set correctly.
1. Verify the managed identity has the `Reader` role on the target resources.
1. Restart Grafana after changing server configuration.

### Workload Identity not working

**Symptoms:**

- Workload Identity authentication fails in Kubernetes/AKS environment
- Token file errors

**Solutions:**

1. Verify `workload_identity_enabled = true` is set in the Grafana server configuration.
1. Check that the service account is correctly annotated for workload identity.
1. Verify the federated credential is configured in Azure.
1. Ensure the token path is accessible to the Grafana Pod.
1. Check the workload identity webhook is running in the cluster.

### "401 Unauthorized" despite correct Azure setup

**Symptoms:**

- Authentication fails with `401 Unauthorized` even though Managed Identity or Workload Identity is correctly configured in Azure
- The managed identity has the correct RBAC roles
- `managed_identity_enabled` or `workload_identity_enabled` is set to `true` in the Grafana `.ini` file

**Cause:** The `forward_settings_to_plugins` setting under `[azure]` doesn't include the Azure Monitor plugin, so the plugin doesn't receive the Azure authentication settings from the Grafana server.

**Solutions:**

1. Check if you've customized the `forward_settings_to_plugins` setting under `[azure]` in your Grafana `.ini` file.
1. If customized, ensure `grafana-azure-monitor-datasource` is listed. For example:
   ```ini
   [azure]
   managed_identity_enabled = true
   forward_settings_to_plugins = grafana-azure-monitor-datasource
   ```
1. By default, Grafana includes all Grafana Labs Azure plugins. If you haven't customized this setting, the default includes the Azure Monitor plugin.
1. Restart Grafana after making changes.

## Query errors

These errors occur when executing queries against Azure Monitor services.

### "No data" or empty results

**Symptoms:**

- Query executes without error but returns no data
- Charts show "No data" message

**Possible causes and solutions:**

| Cause                             | Solution                                                                                                                         |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Time range doesn't contain data   | Expand the dashboard time range or verify data exists in Azure Portal.                                                           |
| Wrong resource selected           | Verify you've selected the correct subscription, resource group, and resource.                                                   |
| Metric not available for resource | Not all metrics are available for all resources. Check available metrics in Azure Portal under the resource's **Metrics** blade. |
| Metric has no values              | Some metrics only populate under certain conditions (for example, error counts when errors occur).                               |
| Permissions issue                 | Verify the identity has read access to the specific resource.                                                                    |

### "Bad request" or "Invalid query"

**Symptoms:**

- Query fails with 400 error
- Error message indicates query syntax issues

**Solutions for Logs queries:**

1. Validate your KQL syntax in the Azure Portal Log Analytics query editor.
1. Check for typos in table names or column names.
1. Ensure referenced tables exist in the selected workspace.
1. Verify the time range is valid (not in the future, not too far in the past for data retention).

**Solutions for Metrics queries:**

1. Verify the metric name is valid for the selected resource type.
1. Check that dimension filters use valid dimension names and values.
1. Ensure the aggregation type is supported for the selected metric.

### "Resource not found"

**Symptoms:**

- Query fails with 404 error
- Resource picker shows resources that can't be queried

**Solutions:**

1. Verify the resource still exists in Azure (it may have been deleted or moved).
1. Check that the subscription is correct.
1. Refresh the resource picker by re-selecting the subscription.
1. Verify the identity has access to the resource's resource group.

### Logs query timeout

**Symptoms:**

- Query runs for a long time then fails with `DatasourceError` or `context deadline exceeded`
- Alert evaluations fail with timeout errors
- Error mentions timeout or query limits

**Cause:** The default data source timeout in Grafana is 30 seconds, but Azure Log Analytics queries can take up to 3 minutes on the Azure side, especially for complex KQL queries or large time ranges.

**Solutions:**

1. **Increase the data source timeout.** Open the Azure Monitor data source settings, scroll to the **Misc** section, and increase the **Timeout** to `300` (seconds). Refer to [Query timeout](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/configure/#query-timeout) for details.
1. Narrow the time range to reduce data volume.
1. Add filters to reduce the result set.
1. Use `summarize` to aggregate data instead of returning raw rows.
1. Consider using Basic Logs for large datasets (if enabled).
1. Break complex queries into smaller parts.
1. For alerting, ensure the alert evaluation interval is long enough to accommodate the query duration plus the configured timeout.

### Logs query results are truncated

**Symptoms:**

- A Logs query returns exactly 30,000 rows even though more data exists
- Grafana displays a notice that the row limit was reached

**Cause:** When a Logs query uses the **Logs** result format, Grafana truncates the results to 30,000 rows to prevent the browser tab from crashing, because the logs visualization renders every row at once.

**Solutions:**

1. Aggregate the data in your query with `summarize` instead of returning raw rows.
1. Narrow the dashboard time range to reduce the number of rows returned.
1. Add filters to your KQL query to return only the rows you need.
1. To remove the limit, enable the `azureMonitorDisableLogLimit` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/). Disabling the limit can degrade browser performance for very large result sets.

### "Metrics not available" for a resource

**Symptoms:**

- Resource appears in picker but no metrics are listed
- Metric dropdown is empty

**Solutions:**

1. Verify the resource type supports Azure Monitor metrics.
1. Check if the resource is in a region that supports metrics.
1. Some resources require diagnostic settings to emit metrics—configure these in Azure Portal.
1. Try selecting a different namespace for the resource.

## Azure Resource Graph errors

These errors are specific to Azure Resource Graph (ARG) queries.

### "Query execution failed"

**Symptoms:**

- ARG query fails with execution errors
- Results don't match expected resources

**Solutions:**

1. Validate query syntax in Azure Portal Resource Graph Explorer.
1. Check that you have access to the subscriptions being queried.
1. Verify table names are correct (for example, `Resources`, `ResourceContainers`).
1. Some ARG features require specific permissions, check [ARG documentation](https://docs.microsoft.com/en-us/azure/governance/resource-graph/).

### Query returns incomplete results

**Symptoms:**

- Not all expected resources appear in results
- Results seem truncated

**Solutions:**

1. ARG queries are paginated. The data source handles pagination automatically, but very large result sets may be limited.
1. Add filters to reduce result set size.
1. Verify you have access to all subscriptions containing the resources.

## Application Insights Traces errors

These errors are specific to the Traces query type.

### "No traces found"

**Symptoms:**

- Trace query returns empty results
- Operation ID search finds nothing

**Solutions:**

1. Verify the Application Insights resource is collecting trace data.
1. Check that the time range includes when the traces were generated.
1. Ensure the Operation ID is correct (copy directly from another trace or log).
1. Verify the identity has access to the Application Insights resource.

## Basic Logs errors

These errors are specific to Basic Logs queries.

### Basic Logs toggle is not available

**Symptoms:**

- The **Analytics / Basic** toggle doesn't appear in the Logs query editor
- Unable to switch to Basic Logs mode

**Solutions:**

1. Verify **Enable Basic Logs** is toggled on in the data source configuration. Refer to [Enable Basic Logs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/configure/#enable-basic-logs).
1. Basic Logs isn't available when creating alert rules. If you're in the alerting UI, use Analytics mode instead.

### Basic Logs query returns errors

**Symptoms:**

- Query fails with "BadArgumentError" or similar errors
- Query returns unexpected results

**Solutions:**

1. Ensure you've selected only a single workspace resource. Basic Logs doesn't support multi-resource queries.
1. Remove any time-range filters from the query itself. Basic Logs always uses the dashboard time range.
1. Verify the table you're querying is configured with the Basic or Auxiliary log plan in Azure. Not all tables support Basic Logs.
1. Check that your KQL query uses only [supported operators](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1#limitations).

### Unexpected costs from Basic Logs

**Solutions:**

1. Basic Logs queries are billed per query by Azure, separate from Grafana costs. Review the [Azure pricing documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1).
1. Check your query frequency. Dashboard auto-refresh and variable changes can trigger repeated queries.
1. Consider switching infrequently queried tables back to Analytics mode to avoid per-query charges.

## Alerting errors

These errors occur when using Azure Monitor data in Grafana alert rules or recording rules. For general alerting configuration, refer to the [Azure Monitor alerting documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/alerting/).

### Alert failure with Current User authentication

**Symptoms:**

- Alert rules and recording rules fail with authentication errors
- Dashboard queries work, but alerting evaluations do not
- Error messages reference missing or invalid credentials during background evaluation

**Cause:** Current User authentication relies on the logged-in user's credentials, which aren't available for background operations like alerting.

**Solutions:**

1. Configure **fallback service credentials** in the data source settings. Typically this means adding an App Registration (client secret) as the fallback.
1. Verify the fallback credentials have the required Azure RBAC permissions on the resources used in your alert rules.
1. Refer to [Limitations and fallback credentials](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/configure/#limitations-and-fallback-credentials) for configuration details.

### Alert evaluation timeout

**Symptoms:**

- Alert rules fail with `DatasourceError` or `context deadline exceeded`
- Alerts work intermittently, failing when queries take longer than usual

**Solutions:**

1. **Increase the data source timeout.** The default is 30 seconds, but Azure Log Analytics queries can take up to 3 minutes. Open the Azure Monitor data source settings, scroll to **Misc**, and set **Timeout** to `300`. Refer to [Query timeout](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/configure/#query-timeout).
1. Ensure the alert evaluation interval is long enough to accommodate the query duration.
1. Simplify complex KQL queries used in alert rules.
1. Reduce the time range in Log Analytics queries.

## Template variable errors

For general template variable configuration, refer to the [template variables documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/template-variables/).

### "Properties found in series but missing `valueProp` and `textProp`"

**Symptoms:**

- Template variables using query types such as **Resource Groups**, **Resource Names**, or **Subscriptions** fail with this error
- Variables that previously worked stop returning values after a Grafana upgrade
- Dashboards that rely on cascading variables break

**Cause:** This error occurs when the Azure Monitor plugin's variable query response format doesn't match what Grafana expects. This was a known bug that affected specific Grafana versions.

**Solutions:**

1. **Check your Grafana version.** This issue was a software bug fixed in later releases. Update to the latest Grafana patch release for your version.
1. **Grafana Cloud users:** If you're on a release channel that doesn't yet include the fix, switch to the **stable** channel in your Cloud stack settings (**Stack Management** > **General** > **Release channel**) to receive the latest stable patch.
1. **Self-hosted users:** Download the latest patch release from the [Grafana download page](https://grafana.com/grafana/download).
1. After upgrading, reload the dashboard and re-run the variable queries.

### Variables return no values

**Solutions:**

1. Verify the data source connection is working (test it in the data source settings).
1. Check that parent variables (for cascading variables) have valid selections.
1. Verify the identity has permissions to list the requested resources.
1. For Logs variables, ensure the KQL query returns a single column.

### Variables stop working after a Grafana upgrade

**Symptoms:**

- Template variables that worked before an upgrade return errors or empty results
- Dashboard variable dropdowns show no options or display error indicators

**Solutions:**

1. Check the [Grafana release notes](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/whatsnew/) for breaking changes related to template variables or the Azure Monitor data source.
1. Verify the Azure Monitor plugin version is compatible with your Grafana version. Go to **Administration** > **Plugins and data** > **Plugins**, search for "Azure Monitor", and check the version.
1. Clear the browser cache and reload the dashboard.
1. Re-open the variable configuration and click **Run query** to verify it returns expected results.
1. If the issue persists, check for known bugs in the [Grafana GitHub issues](https://github.com/grafana/grafana/issues?q=is%3Aissue+label%3A%22datasource%2Fazure-monitor%22+template+variable) and consider rolling back to the previous Grafana version.

### Variables are slow to load

**Solutions:**

1. Set variable refresh to **On dashboard load** instead of **On time range change**.
1. Reduce the scope of variable queries (for example, filter by resource group instead of entire subscription).
1. For Logs variables, optimize the KQL query to return results faster.

## Enable debug logging

To capture detailed error information for troubleshooting Azure Monitor issues:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Restart Grafana for the change to take effect.
1. Reproduce the issue and review the logs at `/var/log/grafana/grafana.log` (or your configured log location).
1. Look for entries containing `azuremonitor` or `azure` for Azure Monitor-specific request and response details.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you've tried the solutions in this document and still encounter issues:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review the [Azure Monitor data source GitHub issues](https://github.com/grafana/grafana/issues) for known bugs.
1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user.
1. When reporting issues, include:
   - Grafana version
   - Error messages (redact sensitive information)
   - Steps to reproduce
   - Relevant configuration (redact credentials)

# Elasticsearch alerting

You can use Grafana Alerting with Elasticsearch to create alerts based on your Elasticsearch data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met.

For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/).

## Before you begin

Before creating alerts with Elasticsearch, ensure you have:

- An Elasticsearch data source configured in Grafana
- Appropriate permissions to create alert rules
- Understanding of the metrics you want to monitor

## Supported query types

Elasticsearch alerting works best with **metrics queries** that return time series data. To create a valid alert query:

- Use a **Date histogram** as the last bucket aggregation (under **Group by**)
- Select appropriate metric aggregations (Count, Average, Sum, Min, Max, etc.)

Queries that return time series data allow Grafana to evaluate values over time and trigger alerts when thresholds are crossed.

### Query types and alerting compatibility

| Query type                     | Alerting support | Notes                                                       |
| ------------------------------ | ---------------- | ----------------------------------------------------------- |
| Metrics with Date histogram    | Supported        | Recommended for alerting                                    |
| Metrics without Date histogram | Limited          | May not evaluate correctly over time                        |
| Logs                           | Not supported    | Use metrics queries instead                                 |
| Raw data                       | Not supported    | Use metrics queries instead                                 |
| Raw document (deprecated)      | Not supported    | Deprecated since Grafana v10.1. Use metrics queries instead |

## Create an alert rule

To create an alert rule using Elasticsearch:

1. Navigate to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for the alert rule.
1. Select your **Elasticsearch** data source.
1. Build your query using the query editor:
   - Add metric aggregations (for example, Average, Count, Sum)
   - Add a Date histogram under **Group by**
   - Optionally add filters using Lucene query syntax
1. Configure the alert condition (for example, when the average is above a threshold).
1. Set the evaluation interval and pending period.
1. Configure notifications and labels.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example alert queries

The following examples show common alerting scenarios with Elasticsearch.

### Alert on high error count

Monitor the number of error-level log entries:

1. **Query:** `level:error`
1. **Metric:** Count
1. **Group by:** Date histogram (interval: 1m)
1. **Condition:** When count is above 100

### Alert on average response time

Monitor API response times:

1. **Query:** `type:api_request`
1. **Metric:** Average on field `response_time`
1. **Group by:** Date histogram (interval: 5m)
1. **Condition:** When average is above 500 (milliseconds)

### Alert on unique user count drop

Detect drops in active users:

1. **Query:** `*` (all documents)
1. **Metric:** Unique count on field `user_id`
1. **Group by:** Date histogram (interval: 1h)
1. **Condition:** When unique count is below 100

## Limitations

When using Elasticsearch with Grafana Alerting, be aware of the following limitations:

### Template variables not supported

Alert queries cannot contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$hostname` or `$environment` won't be resolved.

If your dashboard query uses template variables, create a separate query for alerting with hard coded values.

### Logs queries not supported

Queries using the **Logs** metric type cannot be used for alerting. Convert your query to use metric aggregations with a Date histogram instead.

### Query complexity

Complex queries with many nested aggregations may timeout or fail to evaluate. Simplify queries for alerting by:

- Reducing the number of bucket aggregations
- Using appropriate time intervals
- Adding filters to limit the data scanned

## Best practices

Follow these best practices when creating Elasticsearch alerts:

- **Use specific filters:** Add Lucene query filters to focus on relevant data and improve query performance.
- **Choose appropriate intervals:** Match the Date histogram interval to your evaluation frequency.
- **Test queries first:** Verify your query returns expected results in Explore before creating an alert.
- **Set realistic thresholds:** Base alert thresholds on historical data patterns.
- **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor.

# Elasticsearch annotations

Annotations overlay event data on your dashboard graphs, helping you correlate log events with metrics.
You can use Elasticsearch as a data source for annotations to display events such as deployments, alerts, or other significant occurrences on your visualizations.

For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/annotate-visualizations/).

## Before you begin

Before creating Elasticsearch annotations, ensure you have:

- An Elasticsearch data source configured in Grafana
- Documents in Elasticsearch containing event data with timestamp fields
- Read access to the Elasticsearch index containing your events

## Create an annotation query

To add an Elasticsearch annotation to your dashboard:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Annotation query**.
1. Enter a name for the annotation query.
1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox.
1. Select a color for the annotation event markers.
1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed.
1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed.
1. Click **Open query editor** to open the **Annotation Query** dialog box.
1. Select your **Elasticsearch** data source from the **Data source** drop-down list.
1. Configure the annotation query and field mappings.
1. (Optional) Click **Test annotation query** to ensure that the query is working properly.
1. Click **Close** when you've completed the query setup.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## Query

Use the query field to filter which Elasticsearch documents appear as annotations. The query uses [Lucene query syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax).

**Examples:**

| Query                                    | Description                                          |
| ---------------------------------------- | ---------------------------------------------------- |
| `*`                                      | Matches all documents.                               |
| `type:deployment`                        | Shows only deployment events.                        |
| `level:error OR level:critical`          | Shows error and critical events.                     |
| `service:api AND environment:production` | Shows events for a specific service and environment. |
| `tags:release`                           | Shows events tagged as releases.                     |

You can use template variables in your annotation queries. For example, `service:$service` filters annotations based on the selected service variable.

## Field mappings

Field mappings tell Grafana which Elasticsearch fields contain the annotation data.

### Time

The **Time** field specifies which field contains the annotation timestamp.

- **Default:** `@timestamp`
- **Format:** The field must contain a date value that Elasticsearch recognizes.

### Time End

The **Time End** field specifies a field containing the end time for range annotations. Range annotations display as a shaded region on the graph instead of a single vertical line.

- **Default:** Empty (single-point annotations)
- **Use case:** Display maintenance windows, incidents, or any event with a duration.

### Text

The **Text** field specifies which field contains the annotation description displayed when you hover over the annotation.

- **Default:** `tags`
- **Tip:** Use a descriptive field like `message`, `description`, or `summary`.

### Tags

The **Tags** field specifies which field contains tags for the annotation. Tags help categorize and filter annotations.

- **Default:** Empty
- **Format:** The field can contain either a comma-separated string or an array of strings.

## Example: Deployment annotations

To display deployment events as annotations:

1. Create an annotation query with the following settings:
   - **Query:** `type:deployment`
   - **Time:** `@timestamp`
   - **Text:** `message`
   - **Tags:** `environment`

This configuration displays deployment events with their messages as the annotation text and environments as tags.

## Example: Range annotations for incidents

To display incidents with duration:

1. Create an annotation query with the following settings:
   - **Query:** `type:incident`
   - **Time:** `start_time`
   - **Time End:** `end_time`
   - **Text:** `description`
   - **Tags:** `severity`

This configuration displays incidents as shaded regions from their start time to end time.

# Configure the Elasticsearch data source

Grafana ships with the Elasticsearch data source preinstalled. You can create a variety of queries to visualize logs or metrics stored in Elasticsearch, and annotate graphs with log events stored in Elasticsearch. For details about updating the standalone plugin, refer to [Plugin updates](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/elasticsearch/#plugin-updates).

For instructions on how to add a data source to Grafana, refer to the [administration documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/).
Administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system.

## Before you begin

To configure the Elasticsearch data source, you need:

- **Grafana administrator permissions:** Only users with the organization `administrator` role can add data sources.
- **A supported Elasticsearch version:** v7.17 or later, v8.x, v9.x or Elastic Cloud Serverless.
- **Elasticsearch server URL:** The HTTP or HTTPS endpoint for your Elasticsearch instance, including the port (default: `9200`).
- **Authentication credentials:** Depending on your Elasticsearch security configuration, you need one of the following:
  - Username and password for basic authentication
  - API key
  - No credentials (if Elasticsearch security is disabled)
- **Network access:** Grafana must be able to reach your Elasticsearch server. For Grafana Cloud, consider using [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your Elasticsearch instance is in a private network.

## Elasticsearch permissions

When Elasticsearch security features are enabled, you must configure the following cluster privileges for the user or API key that Grafana uses to connect:

- **monitor** - Required for the connection health check. Grants access to the `_cluster/health` endpoint and retrieves the version information of the connected Elasticsearch instance.
- **view_index_metadata** - Required for accessing mapping definitions of indices.
- **read** - Grants the ability to perform search and retrieval operations on indices. This is essential for querying and extracting data from the cluster.

## Add the data source

To add the Elasticsearch data source, complete the following steps:

1. Click **Connections** in the left-side menu.
1. Under **Connections**, click **Add new connection**.
1. Enter `Elasticsearch` in the search bar.
1. Click **Elasticsearch** under the **Data source** section.
1. Click **Add new data source** in the upper right.

You are taken to the **Settings** tab where you set up your Elasticsearch configuration.

## Configuration options

Configure the following basic settings for the Elasticsearch data source:

- **Name** - The data source name. This is how you refer to the data source in panels and queries. Examples: `elastic-1`, `elasticsearch_metrics`.

- **Default** - Toggle on to make this the default data source. New panels and Explore queries use the default data source.

## Connection

Set the HTTP endpoint Grafana uses to reach your Elasticsearch cluster.

- **URL** - The URL of your Elasticsearch server, including the port. Examples: `http://localhost:9200`, `http://elasticsearch.example.com:9200`.

## Authentication

Choose how Grafana authenticates to Elasticsearch. Select an authentication method from the drop-down menu:

- **Basic authentication** - Enter the username and password for your Elasticsearch user.

- **Forward OAuth identity** - Forward the OAuth access token (and the OIDC ID token if available) of the user querying the data source.

- **No authentication** - Connect without credentials. Only use this option if your Elasticsearch instance doesn't require authentication.

- **Serverless API Key** - Enter your API key to connect to an Elastic serverless instance. For instructions on locating or generating your API key, refer to [Elastic Cloud API keys](https://www.elastic.co/docs/deploy-manage/api-keys/elastic-cloud-api-keys).

### API key authentication

To authenticate using an Elasticsearch API key, select **No authentication** and configure the API key using HTTP headers:

1. In the **HTTP headers** section, click **+ Add header**.
1. Set **Header** to `Authorization`.
1. Set **Value** to `ApiKey <your-api-key>`, replacing `<your-api-key>` with your base64-encoded Elasticsearch API key.

For information about creating API keys, refer to the [Elasticsearch API keys documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).

### Amazon OpenSearch Service and SigV4

Amazon OpenSearch Service is the successor to Amazon Elasticsearch Service. For Amazon OpenSearch Service, use the [OpenSearch data source](https://grafana.com/docs/plugins/grafana-opensearch-datasource/latest/) instead of this Elasticsearch data source.

If you still connect the Grafana Elasticsearch data source to a domain that requires AWS Signature Version 4 (AWS SigV4), for example a legacy Amazon Elasticsearch Service domain or another SigV4-protected OpenSearch-compatible endpoint, you must sign all requests with SigV4.

For details on AWS SigV4, refer to the [AWS documentation](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html).

To sign requests, enable SigV4 in the Grafana [configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#sigv4_auth_enabled). After SigV4 is enabled, configure it on the Elasticsearch data source configuration page. For more information about AWS authentication options, refer to [AWS authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/).

{{< figure src="/static/img/docs/v73/elasticsearch-sigv4-config-editor.png" max-width="500px" class="docs-image--no-shadow" caption="SigV4 configuration for AWS-hosted Elasticsearch-compatible domains" >}}

### TLS settings

{{< admonition type="note" >}}
Use TLS (Transport Layer Security) for an additional layer of security when working with Elasticsearch. For information on setting up TLS encryption with Elasticsearch, refer to [Configure TLS](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/configuring-tls.html#configuring-tls). You must add TLS settings to your Elasticsearch configuration file **prior** to setting these options in Grafana.
{{< /admonition >}}

- **Add self-signed certificate** - Check the box to authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file. Required for verifying self-signed TLS certificates.

- **TLS client authentication** - Check the box to authenticate with the TLS client, where the server authenticates the client. Add the `Server name`, `Client certificate` and `Client key`. The **ServerName** is used to verify the hostname on the returned certificate. The **Client certificate** can be generated from a Certificate Authority (CA) or be self-signed. The **Client key** can also be generated from a Certificate Authority (CA) or be self-signed. The client key encrypts the data between client and server.

- **Skip TLS certificate validation** - Check the box to bypass TLS certificate validation. Skipping TLS certificate validation is not recommended unless absolutely necessary or for testing purposes.

### HTTP headers

Click **+ Add header** to add one or more HTTP headers. HTTP headers pass additional context and metadata about the request/response.

- **Header** - Add a custom header. This allows custom headers to be passed based on the needs of your Elasticsearch instance.

- **Value** - The value of the header.

## Additional settings

Additional settings are optional settings that can be configured for more control over your data source.

### Advanced HTTP settings

- **Allowed cookies** - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default.

- **Timeout** - The HTTP request timeout. This must be in seconds. There is no default, so this setting is up to you.

### Elasticsearch details

The following settings are specific to the Elasticsearch data source.

- **Index name** - The name of your Elasticsearch index. You can use the following formats:
  - **Wildcard patterns** - Use `*` to match multiple indices. Examples: `logs-*`, `metrics-*`, `filebeat-*`.
  - **Time patterns** - Use date placeholders for time-based indices. Wrap the fixed portion in square brackets. Examples: `[logstash-]YYYY.MM.DD`, `[metrics-]YYYY.MM`.
  - **Specific index** - Enter the exact index name. Example: `application-logs`.
  - **Cross-cluster search** - Use the `cluster:index` pattern to query remote clusters configured in Elasticsearch. Example: `logs-cluster:logs-*`. Cross-cluster search is always available. No feature toggle is required.

- **Pattern** - Select the matching pattern if you use a time pattern in your index name. Options include:
  - no pattern
  - hourly
  - daily
  - weekly
  - monthly
  - yearly

Only select a pattern option if you have specified a time pattern in the Index name field.

- **Time field name** - Name of the time field. The default value is `@timestamp`. You can enter a different name.

- **Max concurrent shard requests** - Sets the number of shards being queried at the same time. The default is `5`. For more information on shards, refer to the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/scalability.html#scalability).

- **Min time interval** - Defines a lower limit for the auto group-by time interval. This value **must** be formatted as a number followed by a valid time identifier:

  | Identifier | Description |
  | ---------- | ----------- |
  | `y`        | year        |
  | `M`        | month       |
  | `w`        | week        |
  | `d`        | day         |
  | `h`        | hour        |
  | `m`        | minute      |
  | `s`        | second      |
  | `ms`       | millisecond |

We recommend setting this value to match your Elasticsearch write frequency.
For example, set this to `1m` if Elasticsearch writes data every minute.

You can also override this setting in a dashboard panel under its data source options. The default is `10s`.

- **X-Pack enabled** - Toggle to enable `X-Pack`-specific features and options, which provide the [query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/elasticsearch/query-editor/) with additional aggregations, such as `Rate` and `Top Metrics`.

- **Include frozen indices** - Toggle on when the `X-Pack enabled` setting is active. Includes frozen indices in searches. You can configure Grafana to include [frozen indices](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/frozen-indices.html) when performing search requests.

{{< admonition type="note" >}}
Frozen indices are [deprecated in Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/frozen-indices.html) since v7.14.
{{< /admonition >}}

### Logs

Configure which fields the data source uses for log messages and log levels.

- **Message field name** - The field that contains the log message content.

- **Level field name** - The field that contains log level or severity information. When specified, Grafana uses this field to determine the log level and color-code each log line. If the log doesn't have a level field, Grafana tries to match the content against [supported expressions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/logs-integration/#log-level). If Grafana can't determine the log level, it displays as unknown.

### Data links

Data links create a link from a specified field that can be accessed in the Explore logs view. You can add multiple data links by clicking **+ Add**.

Each data link configuration consists of:

- **Field** - Sets the name of the field used by the data link.

- **URL/query** - Sets the full link URL if the link is external. If the link is internal, this input serves as a query for the target data source. In both cases, you can interpolate the value from the field with the `${__value.raw }` macro.

- **URL Label** (Optional) - Sets a custom display label for the link. The link label defaults to the full external URL or name of the linked internal data source and is overridden by this setting.

- **Internal link** - Toggle on to set an internal link. For an internal link, you can select the target data source with a data source selector. This supports only tracing data sources.

## Private data source connect (PDC) and Elasticsearch

Use private data source connect (PDC) to connect to and query data within a secure network without opening that network to inbound traffic from Grafana Cloud. Refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) for more information on how PDC works and [Configure Grafana private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc) for steps on setting up a PDC connection.

If you use PDC with SigV4 (AWS Signature Version 4 Authentication), the PDC agent must allow internet egress to `sts.<region>.amazonaws.com:443`.

- **Private data source connect** - Click in the box to set the default PDC connection from the drop-down menu or create a new connection.

After you have configured your Elasticsearch data source options, click **Save & test** to test the connection. A successful connection displays the following message:

`Elasticsearch data source is healthy.`

## Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system.
For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/).

{{< admonition type="note" >}}
The previously used `database` field has now been [deprecated](https://github.com/grafana/grafana/pull/58647).
Use the `index` field in `jsonData` to store the index name.
Refer to the examples below.
{{< /admonition >}}

### Basic provisioning

```yaml
apiVersion: 1

datasources:
  - name: Elastic
    type: elasticsearch
    access: proxy
    url: http://localhost:9200
    jsonData:
      index: '[metrics-]YYYY.MM.DD'
      interval: Daily
      timeField: '@timestamp'
```

### Provision for logs

```yaml
apiVersion: 1

datasources:
  - name: elasticsearch-v7-filebeat
    type: elasticsearch
    access: proxy
    url: http://localhost:9200
    jsonData:
      index: '[filebeat-]YYYY.MM.DD'
      interval: Daily
      timeField: '@timestamp'
      logMessageField: message
      logLevelField: fields.level
      dataLinks:
        - datasourceUid: my_jaeger_uid # Target UID needs to be known
          field: traceID
          url: '$${__value.raw}' # Careful about the double "$$" because of env var expansion
```

## Provision the data source using Terraform

You can provision the Elasticsearch data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation.

### Basic Terraform example

The following example creates a basic Elasticsearch data source for metrics:

```hcl
resource "grafana_data_source" "elasticsearch" {
  name = "Elasticsearch"
  type = "elasticsearch"
  url  = "http://localhost:9200"

  json_data_encoded = jsonencode({
    index     = "[metrics-]YYYY.MM.DD"
    interval  = "Daily"
    timeField = "@timestamp"
  })
}
```

### Terraform example for logs

The following example creates an Elasticsearch data source configured for logs with a data link to Jaeger:

```hcl
resource "grafana_data_source" "elasticsearch_logs" {
  name = "Elasticsearch Logs"
  type = "elasticsearch"
  url  = "http://localhost:9200"

  json_data_encoded = jsonencode({
    index           = "[filebeat-]YYYY.MM.DD"
    interval        = "Daily"
    timeField       = "@timestamp"
    logMessageField = "message"
    logLevelField   = "fields.level"
    dataLinks = [
      {
        datasourceUid = grafana_data_source.jaeger.uid
        field         = "traceID"
        url           = "$${__value.raw}"
      }
    ]
  })
}
```

### Terraform example with basic authentication

The following example includes basic authentication:

```hcl
resource "grafana_data_source" "elasticsearch_auth" {
  name = "Elasticsearch"
  type = "elasticsearch"
  url  = "http://localhost:9200"

  basic_auth_enabled  = true
  basic_auth_username = "elastic_user"

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = var.elasticsearch_password
  })

  json_data_encoded = jsonencode({
    index     = "[metrics-]YYYY.MM.DD"
    interval  = "Daily"
    timeField = "@timestamp"
  })
}
```

For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source).

# Elasticsearch query editor

Grafana provides a query editor for Elasticsearch. Elasticsearch queries are in Lucene format.
For more information about query syntax, refer to [Lucene query syntax](https://www.elastic.co/guide/en/kibana/current/lucene-query.html) and [Query string syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax).

{{< admonition type="note" >}}
When composing Lucene queries, ensure that you use uppercase boolean operators: `AND`, `OR`, and `NOT`. Lowercase versions of these operators are not supported by the Lucene query syntax.
{{< /admonition >}}

{{< figure src="/static/img/docs/elasticsearch/elastic-query-editor-10.1.png" max-width="800px" class="docs-image--no-shadow" caption="Elasticsearch query editor" >}}

For general documentation on querying data sources in Grafana, including options and functions common to all query editors, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/).

## Aggregation types

Elasticsearch groups aggregations into three categories:

- **Bucket** - Bucket aggregations don't calculate metrics, they create buckets of documents based on field values, ranges and a variety of other criteria. Refer to [Bucket aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket.html) for additional information. Use bucket aggregations under `Group by` when creating a metrics query in the query builder.

- **Metrics** - Metrics aggregations perform calculations such as sum, average, min, etc. They can be single-value or multi-value. Refer to [Metrics aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics.html) for additional information. Use metrics aggregations in the metrics query type in the query builder.

- **Pipeline** - Pipeline aggregations work on the output of other aggregations rather than on documents or fields. Refer to [Pipeline aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline.html) for additional information.

## Select a query type

There are two types of queries you can create with the Elasticsearch query builder. Each type is explained in detail below.

Use the **Preserve query** toggle to keep your Lucene query when you switch between **Metrics**, **Logs**, **Raw Data**, and **Raw Document**. When the toggle is off (the default for new queries), switching query types clears the Lucene query. Grafana remembers your last Preserve query choice in the browser for new queries.

### Metrics query type

Metrics queries aggregate data and produce calculations such as count, min, max, and more. Click the metric box to view options in the drop-down menu. The default is `count`.

- **Alias** - Aliasing only applies to **time series queries**, where the last group is `date histogram`. This is ignored for any other type of query.

- **Metric** - Metrics aggregations include:
  - count - refer to [Value count aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-valuecount-aggregation.html)
  - average - refer to [Avg aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-avg-aggregation.html)
  - sum - refer to [Sum aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-sum-aggregation.html)
  - max - refer to [Max aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-max-aggregation.html)
  - min - refer to [Min aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-min-aggregation.html)
  - extended stats - refer to [Extended stats aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-extendedstats-aggregation.html)
  - percentiles - refer to [Percentiles aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-percentile-aggregation.html)
  - unique count - refer to [Cardinality aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html)
  - top metrics - refer to [Top metrics aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-metrics.html)
  - rate - refer to [Rate aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-rate-aggregation.html)

- **Sibling bucket aggregations** - These metrics compute an outer statistic over a per-group inner statistic within each time bucket. For example, **Sum Bucket** with an inner **Max** produces a sum of maxima across hosts. The following options are available:
  - Sum Bucket - refer to [Sum bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-sum-bucket-aggregation.html)
  - Max Bucket - refer to [Max bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-max-bucket-aggregation.html)
  - Min Bucket - refer to [Min bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-min-bucket-aggregation.html)
  - Average Bucket - refer to [Avg bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-avg-bucket-aggregation.html)

- **Pipeline aggregations** - Pipeline aggregations work on the output of other aggregations rather than on documents. The following pipeline aggregations are available:
  - moving function - Calculates a value based on a sliding window of aggregated values. Refer to [Moving function aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-movfn-aggregation.html).
  - derivative - Calculates the derivative of a metric. Refer to [Derivative aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-derivative-aggregation.html).
  - cumulative sum - Calculates the cumulative sum of a metric. Refer to [Cumulative sum aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-cumulative-sum-aggregation.html).
  - serial difference - Calculates the difference between values in a time series. Refer to [Serial differencing aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-serialdiff-aggregation.html).
  - bucket script - Executes a script on metric values from other aggregations. Refer to [Bucket script aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-script-aggregation.html).

You can select multiple metrics and group by multiple terms or filters when using the Elasticsearch query editor.

Use the **+ sign** to the right to add multiple metrics to your query. Click on the **eye icon** next to **Metric** to hide metrics, and the **garbage can icon** to remove metrics.

- **Group by options** - Create multiple group by options when constructing your Elasticsearch query. Date histogram is the default option. The following options are available in the drop-down menu:
  - terms - refer to [Terms aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html).
  - filter - refer to [Filter aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html).
  - `geo hash grid` - refer to the [`geohash_grid` aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-geohashgrid-aggregation.html).
  - date histogram - for time series queries. Refer to [Date histogram aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html).
  - histogram - Depicts frequency distributions. Refer to [Histogram aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-histogram-aggregation.html).
  - nested (experimental) - Refer to [Nested aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-nested-aggregation.html).

Each group by option has a different subset of options to further narrow your query.

The following options are specific to the **date histogram** bucket aggregation option.

- **Time field** - The field used for time-based queries. The default can be set when configuring the data source in the **Time field name** setting under [Elasticsearch details](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/elasticsearch/configure/#elasticsearch-details). The default is `@timestamp`.
- **Interval** - The time interval for grouping data. Select from the drop-down menu or enter a custom interval such as `30d` (30 days). The default is `Auto`.
- **Min doc count** - The minimum number of documents required to include a bucket. The default is `0`.
- **Trim edges** - Removes partial buckets at the edges of the time range. The default is `0`.
- **Offset** - Shifts the start of each bucket by the specified duration. Use positive (`+`) or negative (`-`) values. Examples: `1h`, `5s`, `1d`.
- **Timezone** - The timezone for date calculations. The default is `Coordinated Universal Time`.

Configure the following options for the **terms** bucket aggregation option:

- **Order** - Sets the order of data. Options are `top` or `bottom.`
- **Size** - Limits the number of documents, or size of the dataset. You can set a custom number or `no limit`.
- **Min doc count** - The minimum amount of data to include in your query. The default is `0`.
- **Order by** - Order terms by `term value`, `doc count` or `count`.
- **Missing** - Defines how documents missing a value should be treated. Missing values are ignored by default, but they can be treated as if they had a value. Refer to [Missing value](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_missing_value_5) in the Elasticsearch documentation for more information.

Configure the following options for the **filters** bucket aggregation option:

- **Query** - Specify the query to create a bucket of documents (data). Examples are `hostname:"hostname1"`, `product:"widget5"`. Use the \* wildcard to match any number of characters.
- **Label** - Add a label or name to the bucket.

Configure the following options for the **`geo hash grid`** bucket aggregation option:

- **Precision** - Specifies the number of characters of the `geo hash`.

Configure the following options for the **histogram** bucket aggregation option:

- **Interval** - The numeric interval for grouping values into buckets.
- **Min doc count** - The minimum number of documents required to include a bucket. The default is `0`.

The **nested** group by option is currently experimental, you can select a field and then settings specific to that field.

Click the **+ sign** to add multiple group by options. The data is grouped in order (first by, then by).

{{< figure src="/static/img/docs/elasticsearch/group-by-then-by-10.2.png" max-width="850px" class="docs-image--no-shadow" caption="Group by options" >}}

### Logs query type

Logs queries analyze Elasticsearch log data. You can configure the following options:

- **Logs Options/Limit** - Limits the number of logs to analyze. The default is `500`.

## Query options

Open the **Options** section in the query editor to configure query-level settings:

- **Limit** or **Size** - For Logs and Raw Data queries, sets the maximum number of documents to return. The default is `500`.
- **Include runtime fields** - When enabled, runtime fields defined in the index mapping are included in the response. This option is available in Builder mode.

## Raw query editor

The raw query editor lets you write Elasticsearch queries using the native [Elasticsearch Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html).

### Switch between Builder and Code modes

To access the raw query editor, click the **Code** toggle in the top-right corner of the query editor. You can switch between **Builder** and **Code** modes:

- **Builder** - Visual query builder with dropdown menus and forms
- **Code** - JSON editor for writing raw Elasticsearch DSL queries

### Write raw DSL queries

When in Code mode, you can write complete Elasticsearch query DSL in JSON format. The editor provides:

- **Syntax highlighting** for JSON
- **Auto-formatting** - Click the **Format** button or press `Shift+Alt+F` to format your query
- **Keyboard shortcuts** - Press `Ctrl+Enter` (or `Cmd+Enter` on Mac) to run the query
- **Real-time validation** - Invalid JSON is highlighted with error messages

### Time range handling

If you want to filter by time range in a dashboard, you need to use the `$__from` and `$__to` macros in your raw DSL.

An example query applying dashboard time range using the `@timestamp` field:

```json
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "gte": "$__from",
              "lte": "$__to",
              "format": "epoch_millis"
            }
          }
        }
      ]
    }
  }
}
```

### Supported query types

The raw query editor supports all query types:

- **Metrics queries** are used to query time series data with aggregations. The query parser automatically extracts bucket and metric aggregations from your DSL and uses them for response processing.
- **Logs queries** are used to query log data.

## ES|QL query editor

The ES|QL query editor lets you query Elasticsearch using [ES|QL (Elasticsearch Query Language)](https://www.elastic.co/docs/reference/query-languages/esql), a pipe-based query language. Unlike Lucene queries that rely on aggregation configuration in the builder UI, ES|QL lets you express filtering, aggregation, and transformation in a single query string.

For an introduction to ES|QL syntax and concepts, refer to [Get started with ES|QL queries](https://www.elastic.co/docs/reference/query-languages/esql/esql-getting-started) in the Elasticsearch documentation.

When an ES|QL query does not include a time filter, Grafana automatically adds a time range filter based on the dashboard or Explore time picker.

### Index selection

How the editor handles index selection depends on your data source configuration:

- **No index name configured:** You specify which index to query using the `FROM` command directly in your ES|QL query. This lets you query any index without creating a separate data source for each one.
- **Index name configured:** The editor automatically inserts ` FROM $__index` when the ES|QL field receives focus. You can override this and query a different index if needed.

### Editor features

The ES|QL code editor provides:

- **Code suggestions** - Auto-complete for ES|QL commands and functions
- **Error highlighting** - Syntax errors are highlighted in the editor and error messages from Elasticsearch are displayed directly
- **Syntax highlighting** - ES|QL keywords, operators, and values are color-coded for readability

### Example queries

The following examples show common ES|QL query patterns.

#### Basic aggregation

Count documents grouped by a field:

```esql
FROM logs-*
| STATS count = COUNT(*) BY host.name
| SORT count DESC
| LIMIT 10
```

#### Filter and aggregate over time

Filter by a field value and compute an average over time intervals:

```esql
FROM metrics-*
| WHERE service.name == "api-gateway"
| STATS avg_duration = AVG(transaction.duration.us) BY @timestamp = BUCKET(@timestamp, 1 minute)
| SORT @timestamp
```

#### Search log messages

Search for specific patterns in log data:

```esql
FROM logs-*
| WHERE message LIKE "*error*" AND log.level == "ERROR"
| KEEP @timestamp, message, host.name, log.level
| SORT @timestamp DESC
| LIMIT 100
```

#### PromQL metrics queries

When you select the **Metrics** query type, ES|QL also accepts the `PROMQL` source command. Grafana processes the response as time series frames, the same way it does for ES|QL `STATS` queries. `PROMQL` must be the first command in the query (leading comments are allowed):

```esql
PROMQL index=metrics-* step=1m avg(metrics.system.cpu.logical.count)
```

### Learn more about ES|QL

For more information about ES|QL syntax, commands, and functions, refer to the following Elasticsearch documentation:

- [ES|QL reference](https://www.elastic.co/docs/reference/query-languages/esql) - Overview of the ES|QL query language
- [ES|QL commands](https://www.elastic.co/docs/reference/query-languages/esql/commands) - Source and processing commands (`FROM`, `WHERE`, `STATS`, `EVAL`, `KEEP`, `SORT`, `LIMIT`, and more)
- [ES|QL functions and operators](https://www.elastic.co/docs/reference/query-languages/esql/functions-operators) - Aggregation, math, string, date, and other functions
- [ES|QL syntax](https://www.elastic.co/docs/reference/query-languages/esql/esql-syntax) - Identifiers, literals, operators, and special characters

## Use template variables

You can also augment queries by using [template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/elasticsearch/template-variables/).

Queries of `terms` have a 500-result limit by default.
To set a custom limit, set the `size` property in your query.

# Elasticsearch template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
Grafana lists these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
Grafana refers to such variables as template variables.

For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/) documentation.

## Use filters

Elasticsearch supports the use of filters.
You can use them to specify any number of key/value filters, and Grafana applies them automatically to all of your Elasticsearch queries.

Filters support the following operators:

| Operator | Description                                                   |
| -------- | ------------------------------------------------------------- |
| `=`      | Equals. Adds `AND field:"value"` to the query.                |
| `!=`     | Not equals. Adds `AND -field:"value"` to the query.           |
| `=~`     | Matches regex. Adds `AND field:/value/` to the query.         |
| `!~`     | Does not match regex. Adds `AND -field:/value/` to the query. |
| `>`      | Greater than. Adds `AND field:>value` to the query.           |
| `<`      | Less than. Adds `AND field:<value` to the query.              |

For more information, refer to the [Filter and Group by documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/filter-group-by/).

## Choose a variable syntax

The Elasticsearch data source supports two variable syntaxes for use in the **Query** field:

- `$varname`, such as `hostname:$hostname`, which is easy to read and write but doesn't let you use a variable in the middle of a word.
- `[[varname]]`, such as `hostname:[[hostname]]`

When the _Multi-value_ or _Include all value_ options are enabled, Grafana converts the labels from plain text to a Lucene-compatible condition.
For details, refer to the [Multi-value variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#multi-value-variables) documentation.

## Use variables in queries

You can use variables in the Lucene query field, metric aggregation fields, bucket aggregation fields, and the alias field.

### Variables in Lucene queries

Use variables to filter your Elasticsearch queries dynamically:

```
hostname:$hostname AND level:$level
```

### Chain or nest variables

You can create nested variables, where one variable's values depend on another variable's selection.

This example defines a variable named `$host` that only shows hosts matching the selected `$environment`:

```json
{ "find": "terms", "field": "hostname", "query": "environment:$environment" }
```

Whenever you change the value of the `$environment` variable via the drop-down, Grafana triggers an update of the `$host` variable to contain only hostnames filtered by the selected environment.

### Variables in aggregations

You can use variables in bucket aggregation fields to dynamically change how data is grouped. For example, use a variable in the **Terms** group by field to let users switch between grouping by `hostname`, `service`, or `datacenter`.

## Template variable examples

{{< figure src="/static/img/docs/elasticsearch/elastic-templating-query-7-4.png" max-width="500px" class="docs-image--no-shadow" caption="Query with template variables" >}}

In the above example, a Lucene query filters documents based on the `hostname` property using a variable named `$hostname`.
The example also uses a variable in the _Terms_ group by field input box, which you can use to quickly change how data is grouped.

## Create a query

Write the query using a custom JSON string, with the field mapped as a [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#keyword) in the Elasticsearch index mapping.

If the query is [multi-field](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) with both a `text` and `keyword` type, use `"field":"fieldname.keyword"` (sometimes `fieldname.raw`) to specify the keyword field in your query.

| Query                                                                          | Description                                                                                            |
| ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| `{"find": "fields", "type": "keyword"}`                                        | Returns a list of field names with the index type `keyword`.                                           |
| `{"find": "fields", "type": "number"}`                                         | Returns a list of numeric field names (includes `float`, `double`, `integer`, `long`, `scaled_float`). |
| `{"find": "fields", "type": "date"}`                                           | Returns a list of date field names.                                                                    |
| `{"find": "terms", "field": "hostname.keyword", "size": 1000}`                 | Returns a list of values for a keyword field. Uses the current dashboard time range.                   |
| `{"find": "terms", "field": "hostname", "query": "<Lucene query>"}`            | Returns a list of values filtered by a Lucene query. Uses the current dashboard time range.            |
| `{"find": "terms", "field": "status", "orderBy": "doc_count"}`                 | Returns values sorted by document count (descending by default).                                       |
| `{"find": "terms", "field": "status", "orderBy": "doc_count", "order": "asc"}` | Returns values sorted by document count in ascending order.                                            |

Queries of `terms` have a 500-result limit by default. To set a custom limit, set the `size` property in your query.

### Sort query results

By default, queries return results in term order (which can then be sorted alphabetically or numerically using the variable's Sort setting).

To produce a list of terms sorted by document count (a top-N values list), add an `orderBy` property of `doc_count`. This automatically selects a descending sort:

```json
{ "find": "terms", "field": "status", "orderBy": "doc_count" }
```

You can also use the `order` property to explicitly set ascending or descending sort:

```json
{ "find": "terms", "field": "hostname", "orderBy": "doc_count", "order": "asc" }
```

{{< admonition type="note" >}}
Elasticsearch [discourages](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order) sorting by ascending doc count because it can return inaccurate results.
{{< /admonition >}}

To keep terms in the document count order, set the variable's Sort drop-down to **Disabled**. You can alternatively use other sorting criteria, such as **Alphabetical**, to re-sort them.

# Troubleshoot issues with the Elasticsearch data source

This document provides troubleshooting information for common errors you may encounter when using the Elasticsearch data source in Grafana.

## Connection errors

The following errors occur when Grafana cannot establish or maintain a connection to Elasticsearch.

### Failed to connect to Elasticsearch

**Error message:** "Health check failed: Failed to connect to Elasticsearch"

**Cause:** Grafana cannot establish a network connection to the Elasticsearch server.

**Solution:**

1. Verify that the Elasticsearch URL is correct in the data source configuration.
1. Check that Elasticsearch is running and accessible from the Grafana server.
1. Ensure there are no firewall rules blocking the connection.
1. If using a proxy, verify the proxy settings are correct.
1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your Elasticsearch instance is not publicly accessible.

### Request timed out

**Error message:** "Health check failed: Elasticsearch data source is not healthy. Request timed out"

**Cause:** The connection to Elasticsearch timed out before receiving a response.

**Solution:**

1. Check the network latency between Grafana and Elasticsearch.
1. Verify that Elasticsearch is not overloaded or experiencing performance issues.
1. Increase the timeout setting in the data source configuration if needed.
1. Check if any network devices (load balancers, proxies) are timing out the connection.

### Failed to parse data source URL

**Error message:** "Failed to parse data source URL"

**Cause:** The URL entered in the data source configuration is not valid.

**Solution:**

1. Verify the URL format is correct (for example, `http://localhost:9200` or `https://elasticsearch.example.com:9200`).
1. Ensure the URL includes the protocol (`http://` or `https://`).
1. Remove any trailing slashes or invalid characters from the URL.

## Authentication errors

The following errors occur when there are issues with authentication credentials or permissions.

### Unauthorized (401)

**Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 401 Unauthorized"

**Cause:** The authentication credentials are invalid or missing.

**Solution:**

1. Verify that the username and password are correct.
1. If using an API key, ensure the key is valid and has not expired.
1. Check that the authentication method selected matches your Elasticsearch configuration.
1. Verify the user has the required permissions to access the Elasticsearch cluster.

### Forbidden (403)

**Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 403 Forbidden"

**Cause:** The authenticated user does not have permission to access the requested resource.

**Solution:**

1. Verify the user has read access to the specified index.
1. Check Elasticsearch security settings and role mappings.
1. Ensure the user has permission to access the `_cluster/health` endpoint.
1. If using SigV4 authentication with an AWS-hosted Elasticsearch-compatible domain, verify the IAM policy grants the required permissions.

## Cluster health errors

The following errors occur when the Elasticsearch cluster is unhealthy or unavailable.

### Cluster status is red

**Error message:** "Health check failed: Elasticsearch data source is not healthy"

**Cause:** The Elasticsearch cluster health status is red, indicating one or more primary shards are not allocated.

**Solution:**

1. Check the Elasticsearch cluster health using `GET /_cluster/health`.
1. Review Elasticsearch logs for errors.
1. Verify all nodes in the cluster are running and connected.
1. Check for unassigned shards using `GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason`.
1. Consider increasing the cluster's resources or reducing the number of shards.

### Bad Gateway (502)

**Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 502 Bad Gateway"

**Cause:** A proxy or load balancer between Grafana and Elasticsearch returned an error.

**Solution:**

1. Check the health of any proxies or load balancers in the connection path.
1. Verify Elasticsearch is running and accepting connections.
1. Review proxy/load balancer logs for more details.
1. Ensure the proxy timeout is configured appropriately for Elasticsearch requests.

## Index errors

The following errors occur when there are issues with the configured index or index pattern.

### Index not found

**Error message:** "Error validating index: `index_not_found`"

**Cause:** The specified index or index pattern does not match any existing indices.

**Solution:**

1. Verify the index name or pattern in the data source configuration.
1. Check that the index exists using `GET /_cat/indices`.
1. If using a time-based index pattern (for example, `[logs-]YYYY.MM.DD`), ensure indices exist for the selected time range.
1. Verify the user has permission to access the index.

### Time field not found

**Error message:** "Could not find time field '@timestamp' with type date in index"

**Cause:** The specified time field does not exist in the index or is not of type `date`.

**Solution:**

1. Verify the time field name in the data source configuration matches the field in your index.
1. Check the field mapping using `GET /<index>/_mapping`.
1. Ensure the time field is mapped as a `date` type, not `text` or `keyword`.
1. If the field name is different (for example, `timestamp` instead of `@timestamp`), update the data source configuration.

## Query errors

The following errors occur when there are issues with query syntax or configuration.

### Too many buckets

**Error message:** "Trying to create too many buckets. Must be less than or equal to: [65536]."

**Cause:** The query is generating more aggregation buckets than Elasticsearch allows.

**Solution:**

1. Reduce the time range of your query.
1. Increase the date histogram interval (for example, change from `10s` to `1m`). For nested aggregations with **Interval** set to `Auto`, newer plugin versions widen the auto interval when the query would exceed `max_buckets`.
1. Add filters to reduce the number of documents being aggregated.
1. Increase the `search.max_buckets` setting in Elasticsearch (requires cluster administrator access).
1. Update the Elasticsearch plugin if you are on an older version. Refer to [Plugin updates](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/elasticsearch/#plugin-updates).

### Required field missing

**Error message:** "Required one of fields [field, script], but none were specified."

**Cause:** A metric aggregation (such as Average, Sum, or Min) was added without specifying a field.

**Solution:**

1. Select a field for the metric aggregation in the query editor.
1. Ensure the selected field exists in your index and contains numeric data.

### Unsupported interval

**Error message:** "unsupported interval '&lt;interval&gt;'"

**Cause:** The interval specified for the index pattern is not valid.

**Solution:**

1. Use a supported interval: `Hourly`, `Daily`, `Weekly`, `Monthly`, or `Yearly`.
1. If you don't need a time-based index pattern, use `No pattern` and specify the exact index name.

## Version errors

The following errors occur when there are Elasticsearch version compatibility issues.

### Unsupported Elasticsearch version

**Error message:** "Support for Elasticsearch versions after their end-of-life (currently versions &lt; 7.17) was removed. Using unsupported version of Elasticsearch may lead to unexpected and incorrect results."

**Cause:** The Elasticsearch version is no longer supported by the Grafana data source.

**Solution:**

1. Upgrade Elasticsearch to a supported version (7.17+, 8.x, or 9.x).
1. Refer to [Elastic Product End of Life Dates](https://www.elastic.co/support/eol) for version support information.
1. Note that queries may still work, but Grafana does not guarantee functionality for unsupported versions.

### Elastic Cloud Serverless health check returns 410 Gone

**Error message:** Health check fails with status `410 Gone` when connecting to Elastic Cloud Serverless.

**Cause:** An older version of the Elasticsearch plugin used a health-check request that Elastic Cloud Serverless no longer accepts.

**Solution:**

1. Update the Elasticsearch plugin to **12.8.0** or later. Refer to [Plugin updates](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/elasticsearch/#plugin-updates).
1. Confirm `preinstall_auto_update` is enabled so Grafana installs the latest plugin on restart, or update manually from **Plugins and data** > **Plugins**.
1. Click **Save & test** again after the plugin update.

## Other common issues

The following issues don't produce specific error messages but are commonly encountered.

### Empty query results

**Cause:** The query returns no data.

**Solution:**

1. Verify the time range includes data in your index.
1. Check the Lucene query syntax for errors.
1. Test the query directly in Elasticsearch using the `_search` API.
1. Ensure the index contains documents matching your query filters.

### Slow query performance

**Cause:** Queries take a long time to execute.

**Solution:**

1. Reduce the time range of your query.
1. Add more specific filters to limit the data scanned.
1. Increase the date histogram interval.
1. Check Elasticsearch cluster performance and resource utilization.
1. Consider using index aliases or data streams for better query routing.

### Browser access mode disabled

**Error message:** Direct browser access is no longer available, or the data source fails when configured for browser access.

**Cause:** Browser access mode was removed in Grafana 9.2.0. The Elasticsearch data source must use Server (proxy) access so the Grafana backend connects to Elasticsearch.

**Solution:**

1. Open the data source configuration in Grafana.
1. Ensure access mode is set to **Server** (proxy). Provisioned data sources should use `access: proxy`.
1. Click **Save & test** to verify the connection.
1. If you still see browser CORS errors, upgrade Grafana and the Elasticsearch plugin. Older configurations that attempted direct browser-to-Elasticsearch requests are no longer supported.

## Get additional help

If you continue to experience issues after following this troubleshooting guide:

1. Check the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) for API-specific guidance.
1. Review the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Contact Grafana Support if you have an Enterprise license.

# Troubleshoot issues with the Elasticsearch data source

This document provides troubleshooting information for common errors you may encounter when using the Elasticsearch data source in Grafana.

## Connection errors

The following errors occur when Grafana cannot establish or maintain a connection to Elasticsearch.

### Failed to connect to Elasticsearch

**Error message:** "Health check failed: Failed to connect to Elasticsearch"

**Cause:** Grafana cannot establish a network connection to the Elasticsearch server.

**Solution:**

1. Verify that the Elasticsearch URL is correct in the data source configuration.
1. Check that Elasticsearch is running and accessible from the Grafana server.
1. Ensure there are no firewall rules blocking the connection.
1. If using a proxy, verify the proxy settings are correct.
1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your Elasticsearch instance is not publicly accessible.

### Request timed out

**Error message:** "Health check failed: Elasticsearch data source is not healthy. Request timed out"

**Cause:** The connection to Elasticsearch timed out before receiving a response.

**Solution:**

1. Check the network latency between Grafana and Elasticsearch.
1. Verify that Elasticsearch is not overloaded or experiencing performance issues.
1. Increase the timeout setting in the data source configuration if needed.
1. Check if any network devices (load balancers, proxies) are timing out the connection.

### Failed to parse data source URL

**Error message:** "Failed to parse data source URL"

**Cause:** The URL entered in the data source configuration is not valid.

**Solution:**

1. Verify the URL format is correct (for example, `http://localhost:9200` or `https://elasticsearch.example.com:9200`).
1. Ensure the URL includes the protocol (`http://` or `https://`).
1. Remove any trailing slashes or invalid characters from the URL.

## Authentication errors

The following errors occur when there are issues with authentication credentials or permissions.

### Unauthorized (401)

**Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 401 Unauthorized"

**Cause:** The authentication credentials are invalid or missing.

**Solution:**

1. Verify that the username and password are correct.
1. If using an API key, ensure the key is valid and has not expired.
1. Check that the authentication method selected matches your Elasticsearch configuration.
1. Verify the user has the required permissions to access the Elasticsearch cluster.

### Forbidden (403)

**Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 403 Forbidden"

**Cause:** The authenticated user does not have permission to access the requested resource.

**Solution:**

1. Verify the user has read access to the specified index.
1. Check Elasticsearch security settings and role mappings.
1. Ensure the user has permission to access the `_cluster/health` endpoint.
1. If using SigV4 authentication with an AWS-hosted Elasticsearch-compatible domain, verify the IAM policy grants the required permissions.

## Cluster health errors

The following errors occur when the Elasticsearch cluster is unhealthy or unavailable.

### Cluster status is red

**Error message:** "Health check failed: Elasticsearch data source is not healthy"

**Cause:** The Elasticsearch cluster health status is red, indicating one or more primary shards are not allocated.

**Solution:**

1. Check the Elasticsearch cluster health using `GET /_cluster/health`.
1. Review Elasticsearch logs for errors.
1. Verify all nodes in the cluster are running and connected.
1. Check for unassigned shards using `GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason`.
1. Consider increasing the cluster's resources or reducing the number of shards.

### Bad Gateway (502)

**Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 502 Bad Gateway"

**Cause:** A proxy or load balancer between Grafana and Elasticsearch returned an error.

**Solution:**

1. Check the health of any proxies or load balancers in the connection path.
1. Verify Elasticsearch is running and accepting connections.
1. Review proxy/load balancer logs for more details.
1. Ensure the proxy timeout is configured appropriately for Elasticsearch requests.

## Index errors

The following errors occur when there are issues with the configured index or index pattern.

### Index not found

**Error message:** "Error validating index: `index_not_found`"

**Cause:** The specified index or index pattern does not match any existing indices.

**Solution:**

1. Verify the index name or pattern in the data source configuration.
1. Check that the index exists using `GET /_cat/indices`.
1. If using a time-based index pattern (for example, `[logs-]YYYY.MM.DD`), ensure indices exist for the selected time range.
1. Verify the user has permission to access the index.

### Time field not found

**Error message:** "Could not find time field '@timestamp' with type date in index"

**Cause:** The specified time field does not exist in the index or is not of type `date`.

**Solution:**

1. Verify the time field name in the data source configuration matches the field in your index.
1. Check the field mapping using `GET /<index>/_mapping`.
1. Ensure the time field is mapped as a `date` type, not `text` or `keyword`.
1. If the field name is different (for example, `timestamp` instead of `@timestamp`), update the data source configuration.

## Query errors

The following errors occur when there are issues with query syntax or configuration.

### Too many buckets

**Error message:** "Trying to create too many buckets. Must be less than or equal to: [65536]."

**Cause:** The query is generating more aggregation buckets than Elasticsearch allows.

**Solution:**

1. Reduce the time range of your query.
1. Increase the date histogram interval (for example, change from `10s` to `1m`). For nested aggregations with **Interval** set to `Auto`, newer plugin versions widen the auto interval when the query would exceed `max_buckets`.
1. Add filters to reduce the number of documents being aggregated.
1. Increase the `search.max_buckets` setting in Elasticsearch (requires cluster administrator access).
1. Update the Elasticsearch plugin if you are on an older version. Refer to [Plugin updates](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/elasticsearch/#plugin-updates).

### Required field missing

**Error message:** "Required one of fields [field, script], but none were specified."

**Cause:** A metric aggregation (such as Average, Sum, or Min) was added without specifying a field.

**Solution:**

1. Select a field for the metric aggregation in the query editor.
1. Ensure the selected field exists in your index and contains numeric data.

### Unsupported interval

**Error message:** "unsupported interval '&lt;interval&gt;'"

**Cause:** The interval specified for the index pattern is not valid.

**Solution:**

1. Use a supported interval: `Hourly`, `Daily`, `Weekly`, `Monthly`, or `Yearly`.
1. If you don't need a time-based index pattern, use `No pattern` and specify the exact index name.

## Version errors

The following errors occur when there are Elasticsearch version compatibility issues.

### Unsupported Elasticsearch version

**Error message:** "Support for Elasticsearch versions after their end-of-life (currently versions &lt; 7.17) was removed. Using unsupported version of Elasticsearch may lead to unexpected and incorrect results."

**Cause:** The Elasticsearch version is no longer supported by the Grafana data source.

**Solution:**

1. Upgrade Elasticsearch to a supported version (7.17+, 8.x, or 9.x).
1. Refer to [Elastic Product End of Life Dates](https://www.elastic.co/support/eol) for version support information.
1. Note that queries may still work, but Grafana does not guarantee functionality for unsupported versions.

### Elastic Cloud Serverless health check returns 410 Gone

**Error message:** Health check fails with status `410 Gone` when connecting to Elastic Cloud Serverless.

**Cause:** An older version of the Elasticsearch plugin used a health-check request that Elastic Cloud Serverless no longer accepts.

**Solution:**

1. Update the Elasticsearch plugin to **12.8.0** or later. Refer to [Plugin updates](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/elasticsearch/#plugin-updates).
1. Confirm `preinstall_auto_update` is enabled so Grafana installs the latest plugin on restart, or update manually from **Plugins and data** > **Plugins**.
1. Click **Save & test** again after the plugin update.

## Other common issues

The following issues don't produce specific error messages but are commonly encountered.

### Empty query results

**Cause:** The query returns no data.

**Solution:**

1. Verify the time range includes data in your index.
1. Check the Lucene query syntax for errors.
1. Test the query directly in Elasticsearch using the `_search` API.
1. Ensure the index contains documents matching your query filters.

### Slow query performance

**Cause:** Queries take a long time to execute.

**Solution:**

1. Reduce the time range of your query.
1. Add more specific filters to limit the data scanned.
1. Increase the date histogram interval.
1. Check Elasticsearch cluster performance and resource utilization.
1. Consider using index aliases or data streams for better query routing.

### Browser access mode disabled

**Error message:** Direct browser access is no longer available, or the data source fails when configured for browser access.

**Cause:** Browser access mode was removed in Grafana 9.2.0. The Elasticsearch data source must use Server (proxy) access so the Grafana backend connects to Elasticsearch.

**Solution:**

1. Open the data source configuration in Grafana.
1. Ensure access mode is set to **Server** (proxy). Provisioned data sources should use `access: proxy`.
1. Click **Save & test** to verify the connection.
1. If you still see browser CORS errors, upgrade Grafana and the Elasticsearch plugin. Older configurations that attempted direct browser-to-Elasticsearch requests are no longer supported.

## Get additional help

If you continue to experience issues after following this troubleshooting guide:

1. Check the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) for API-specific guidance.
1. Review the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Contact Grafana Support if you have an Enterprise license.

# Google Cloud Monitoring alerting

The Google Cloud Monitoring data source supports [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/), allowing you to create alert rules based on GCP metrics and Service Level Objectives (SLOs). You can monitor your Google Cloud environment and receive notifications when specific conditions are met.

## Before you begin

Before you create alert rules, ensure the following:

- You have appropriate permissions to create alert rules in Grafana.
- Your Google Cloud Monitoring data source is configured and working correctly. Refer to [Configure the data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/configure/).
- You're familiar with [Grafana Alerting concepts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/).

## Supported query types for alerting

The following query types support alerting:

| Query type                         | Use case                                               | Notes                                              |
| ---------------------------------- | ------------------------------------------------------ | -------------------------------------------------- |
| **Builder**                        | Threshold-based alerts on GCP resource metrics         | Best suited for alerting; returns time-series data |
| **MQL**                            | Complex metric queries using Monitoring Query Language | Use for advanced filtering and aggregations        |
| **Service Level Objectives (SLO)** | Alert on SLO compliance, error budgets, or burn rate   | Monitor service reliability                        |
| **PromQL**                         | Prometheus-style queries on GCP metrics                | Familiar syntax for Prometheus users               |

{{< admonition type="note" >}}
Alert queries must return numeric data that Grafana can evaluate against a threshold. Queries that return only text or non-numeric data can't be used directly for alerting.
{{< /admonition >}}

## Authentication requirements

Alerting rules run as background processes without a user context. Both supported authentication methods work with alerting:

| Authentication method       | Supported |
| --------------------------- | --------- |
| Google JWT File             | ✓         |
| GCE Default Service Account | ✓         |

## Create an alert rule

To create an alert rule using Google Cloud Monitoring data:

1. Go to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for your alert rule.
1. In the **Define query and alert condition** section:
   - Select your Google Cloud Monitoring data source.
   - Configure your query (for example, a Builder query for CPU usage or an SLO query for error budget).
   - Add a **Reduce** expression if your query returns multiple series.
   - Add a **Threshold** expression to define the alert condition.
1. Configure the **Set evaluation behavior**:
   - Select or create a folder and evaluation group.
   - Set the evaluation interval (how often the alert is checked).
   - Set the pending period (how long the condition must be true before firing).
1. Add labels and annotations to provide context for notifications.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example: VM CPU usage alert

This example creates an alert that fires when Compute Engine VM CPU utilization exceeds 80%:

1. Create a new alert rule.
1. Configure the query:
   - **Query type**: Builder
   - **Project**: Select your GCP project
   - **Service**: Compute Engine
   - **Metric**: `instance/cpu/utilization`
   - **Group by function**: mean
1. Add expressions:
   - **Reduce**: Last (to get the most recent data point)
   - **Threshold**: Is above 0.8 (CPU utilization is returned as a decimal)
1. Set evaluation to run every 1 minute with a 5-minute pending period.
1. Save the rule.

## Example: SLO error budget alert

This example alerts when an SLO's error budget remaining drops below 20%:

1. Create a new alert rule.
1. Configure the query:
   - **Query type**: Service Level Objectives (SLO)
   - **Project**: Select your GCP project
   - **Service**: Select your SLO service
   - **SLO**: Select your SLO
   - **Selector**: SLO Error Budget Remaining
1. Add expressions:
   - **Reduce**: Last
   - **Threshold**: Is below 0.2 (20% remaining)
1. Set evaluation to run every 5 minutes.
1. Save the rule.

## Example: Cloud SQL memory alert

This example alerts when Cloud SQL instance memory usage exceeds 90%:

1. Create a new alert rule.
1. Configure the query:
   - **Query type**: Builder
   - **Project**: Select your GCP project
   - **Service**: Cloud SQL
   - **Metric**: `database/memory/utilization`
   - **Filter**: Add a filter for specific database instances if needed
1. Add expressions:
   - **Reduce**: Last
   - **Threshold**: Is above 0.9
1. Set evaluation to run every 1 minute.
1. Save the rule.

## Best practices

Follow these recommendations to create reliable and efficient alerts with Google Cloud Monitoring data.

### Use appropriate query intervals

- Set the alert evaluation interval to be greater than or equal to the minimum data resolution from Google Cloud Monitoring.
- Most GCP metrics have 1-minute granularity at minimum.
- Avoid very short intervals (less than 1 minute) as they may cause evaluation timeouts or miss data points.

### Reduce multiple series

When your query returns multiple time series (for example, CPU usage across multiple VMs), use the **Reduce** expression to aggregate them:

- **Last**: Use the most recent value
- **Mean**: Average across all series
- **Max/Min**: Use the highest or lowest value
- **Sum**: Total across all series

### Use appropriate alignment periods

For alerting queries, ensure the alignment period provides enough data points:

- Use "cloud monitoring auto" or "grafana auto" for most cases.
- For more precise control, set a fixed alignment period that matches your evaluation interval.

### Handle no data conditions

Configure what happens when no data is returned:

1. In the alert rule, find **Configure no data and error handling**.
1. Choose an appropriate action:
   - **No Data**: Keep the alert in its current state
   - **Alerting**: Treat no data as an alert condition
   - **OK**: Treat no data as a healthy state

### Test queries before alerting

Always verify your query returns expected data before creating an alert:

1. Go to **Explore**.
1. Select your Google Cloud Monitoring data source.
1. Run the query you plan to use for alerting.
1. Confirm the data format and values are correct.
1. Verify the query returns numeric data suitable for threshold evaluation.

## Recording rules

The Google Cloud Monitoring data source supports [Grafana-managed recording rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/). Recording rules periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.

Use recording rules to:

- Reduce query load on Google Cloud Monitoring by pre-computing complex aggregations.
- Create derived metrics from GCP data for use in alerts and dashboards.
- Import Google Cloud Monitoring data into a Prometheus-compatible database.

{{< admonition type="note" >}}
Grafana-managed recording rules write results to a Prometheus-compatible database (such as Grafana Mimir or the Grafana Cloud managed Prometheus). You must configure a target data source for storing the recorded metrics.
{{< /admonition >}}

For instructions on creating recording rules, refer to [Create recording rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/).

## Troubleshooting

If your Google Cloud Monitoring alerts aren't working as expected, use the following sections to diagnose and resolve common issues.

### Alerts not firing

- Check that the query returns numeric data in Explore.
- Ensure the evaluation interval allows enough time for data to be available.
- Verify the threshold is set correctly (remember that many GCP metrics return decimals, not percentages).
- Review the alert rule's health and any error messages in the Alerting UI.

### Authentication errors in alert evaluation

If you see authentication errors when alerts evaluate:

- Verify the service account has the **Monitoring Viewer** role.
- If using a JWT key file, ensure it hasn't been deleted or revoked.
- Check that the required APIs (Monitoring API, Cloud Resource Manager API) are enabled.

### Query timeout errors

- Increase the alignment period to reduce the number of data points.
- Reduce the time range in the query.
- Simplify complex MQL queries.
- Add filters to narrow the result set.

For additional troubleshooting help, refer to [Troubleshoot Google Cloud Monitoring](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/troubleshooting/).

## Additional resources

- [Grafana Alerting documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/)
- [Create alert rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/)
- [Create recording rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/)
- [Google Cloud Monitoring query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/query-editor/)

# Google Cloud Monitoring annotations

[Annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/) overlay rich event information on top of graphs. You can use annotations to mark important events, deployments, or incidents on your dashboards.

## Before you begin

Before you configure annotations, ensure you have the following:

- A configured Google Cloud Monitoring data source.
- A dashboard where you want to add annotations.

## Annotation limitations

Keep the following limitations in mind when using annotations:

- **Performance:** Rendering annotations is expensive. Limit the number of rows returned to maintain dashboard performance.
- **Native events:** There's no support for displaying Google Cloud Monitoring's native annotations and events. However, annotations work well with [custom metrics](https://cloud.google.com/monitoring/custom-metrics/) in Google Cloud Monitoring.

## Add an annotation query

To add an annotation query to a dashboard:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Annotation query**.
1. Enter a name for the annotation query.
1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox.
1. Select a color for the annotation event markers.
1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed.
1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed.
1. Click **Open query editor** to open the **Annotation Query** dialog box.
1. Select your Google Cloud Monitoring data source from the **Data source** drop-down list.
1. Configure the annotation query and field mappings.
1. (Optional) Click **Test annotation query** to ensure that the query is working properly.
1. Click **Close** when you've completed the query setup.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## Configure the annotation query

With the query editor for annotations, you can select a metric and filters to define which data points create annotations.

The **Title** and **Text** fields support templating and can use data returned from the query.

For example, the Title field could have the following text:

`{{metric.type}} has value: {{metric.value}}`

Example result: `monitoring.googleapis.com/uptime_check/http_status has this value: 502`

## Annotation patterns

Use the following patterns in the **Title** and **Text** fields to display metric data in your annotations:

| Pattern format           | Description                       | Example                          | Result                                            |
| ------------------------ | --------------------------------- | -------------------------------- | ------------------------------------------------- |
| `{{metric.value}}`       | Value of the metric/point.        | `{{metric.value}}`               | `555`                                             |
| `{{metric.type}}`        | Returns the full Metric Type.     | `{{metric.type}}`                | `compute.googleapis.com/instance/cpu/utilization` |
| `{{metric.name}}`        | Returns the metric name part.     | `{{metric.name}}`                | `instance/cpu/utilization`                        |
| `{{metric.service}}`     | Returns the service part.         | `{{metric.service}}`             | `compute`                                         |
| `{{metric.label.xxx}}`   | Returns the metric label value.   | `{{metric.label.instance_name}}` | `grafana-1-prod`                                  |
| `{{resource.label.xxx}}` | Returns the resource label value. | `{{resource.label.zone}}`        | `us-east1-b`                                      |

## Example: Annotate uptime check failures

To create annotations for uptime check failures:

1. Add an annotation query using the Google Cloud Monitoring data source.
1. Select the `monitoring.googleapis.com/uptime_check/check_passed` metric.
1. Add a filter for `check_passed = false`.
1. Set the **Title** to: `Uptime check failed: {{metric.label.check_id}}`
1. Set the **Text** to: `Region: {{resource.label.zone}}`

This creates an annotation marker on your graph each time an uptime check fails.

# Configure the Google Cloud Monitoring data source

This document provides instructions for configuring the Google Cloud Monitoring data source in Grafana.

## Before you begin

Before you begin, ensure you have the following:

- **Grafana permissions:** You must have the `Organization administrator` role to configure data sources.
- **GCP project:** A Google Cloud Platform project.
- **GCP permissions:** Permissions to create a service account or configure GCE default service account settings in your GCP project.

Grafana ships with the Google Cloud Monitoring data source preinstalled in both Grafana OSS and Grafana Enterprise, so there's nothing for you to install. For details on how the standalone plugin updates, refer to [Plugin updates](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/#plugin-updates).

## Set up GCP authentication

Before you can request data from Google Cloud Monitoring, you must configure authentication.
All requests to Google APIs are performed on the server-side by the Grafana backend.

For authentication options and configuration details, refer to [Google authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/google-authentication/).

When you configure Google authentication, note the following requirements specific to Google Cloud Monitoring.

### Configure a GCP service account

When you [create a Google Cloud Platform (GCP) service account and key file](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/google-authentication/#create-a-gcp-service-account-and-key-file), the service account must have the **Monitoring Viewer** role (**Role > Select a role > Monitoring > Monitoring Viewer**):

{{< figure src="/static/img/docs/v71/cloudmonitoring_service_account_choose_role.png" max-width="600px" class="docs-image--no-shadow" caption="Choose role" >}}

### Grant the GCE Default Service Account scope

If Grafana is running on a Google Compute Engine (GCE) virtual machine, when you [configure a GCE Default Service Account](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/google-authentication/#use-gce-default-service-account), you must also grant that service account access to the "Cloud Monitoring API" scope.

## Enable Google Cloud Platform APIs

Before you can request data from Google Cloud Monitoring, you must enable the necessary APIs in your GCP project.

1. Open the Monitoring and Cloud Resource Manager API pages:
   - [Monitoring API](https://console.cloud.google.com/apis/library/monitoring.googleapis.com)
   - [Cloud Resource Manager API](https://console.cloud.google.com/apis/library/cloudresourcemanager.googleapis.com)

1. On each page, click **Enable**.

   {{< figure src="/static/img/docs/v71/cloudmonitoring_enable_api.png" max-width="450px" class="docs-image--no-shadow" caption="Enable GCP APIs" >}}

## Add the data source

To add the Google Cloud Monitoring data source:

1. Click **Connections** in the left-side menu.
1. Click **Add new connection**.
1. Enter `Google Cloud Monitoring` in the search bar.
1. Select **Google Cloud Monitoring**.
1. Click **Add new data source** in the upper right.

You're taken to the **Settings** tab where you configure the data source.

## Configure the data source in the UI

The following are configuration options for the Google Cloud Monitoring data source.

| Setting             | Description                                                                                                                                                                                                 |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Name**            | Sets the name you use to refer to the data source in panels and queries.                                                                                                                                    |
| **Default**         | Sets whether the data source is pre-selected for new panels.                                                                                                                                                |
| **Universe Domain** | The universe domain to connect to. For more information, refer to the [Google Cloud universe domains documentation](https://cloud.google.com/docs/overview#universe_domains). Defaults to `googleapis.com`. |

### Authentication

Configure how Grafana authenticates with Google Cloud.

| Setting                 | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Authentication type** | Select the authentication method: **Google JWT File** to use a service account key file, **GCE Default Service Account** if Grafana runs on a GCE virtual machine, **Forward OAuth Identity** to authenticate as the Google-signed-in Grafana user, or **Workload Identity Federation** (Grafana Cloud only) to authenticate through an external OIDC identity provider. For a comparison of the methods, refer to [Supported authentication methods](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/google-authentication/#supported-authentication-methods). |

### Configure Google JWT File authentication

These settings appear when you select **Google JWT File** as the authentication type.

| Setting       | Description                                                                                                                                                                               |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **JWT token** | Upload or paste your Google JWT token. You can drag and drop a `.json` key file, click **Click to browse files** to upload, or use **Paste JWT Token** or **Fill In JWT Token manually**. |

### Configure Forward OAuth Identity

These settings appear when you select **Forward OAuth Identity** as the authentication type.

| Setting             | Description                                                                                                                                                                                            |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Default project** | Enter the GCP project ID to query. This field is required because the user's OAuth token doesn't carry a project context. The signed-in user must have the **Monitoring Viewer** role on this project. |

The Grafana Google authentication must request the following scopes so the forwarded token can read Cloud Monitoring data. Set the full list under `[auth.google]` in `grafana.ini` or `custom.ini`, or under **Scopes** in the SSO Settings UI:

```ini
[auth.google]
scopes = openid email profile https://www.googleapis.com/auth/monitoring.read
```

- `openid`, `email`, `profile`: the default Grafana sign-in scopes. Required for the user to log in.
- `https://www.googleapis.com/auth/monitoring.read`: required for the forwarded token to call the Cloud Monitoring API.

After you change the scopes, each user must sign out, revoke the existing grant at [https://myaccount.google.com/permissions](https://myaccount.google.com/permissions), and sign in again. Google reuses the previous consent until the grant is revoked, so existing sessions continue to hold tokens issued under the old scope set.

<!-- vale Grafana.Headings = NO -->

### Configure Workload Identity Federation

<!-- vale Grafana.Headings = YES -->

_Only available for Grafana Cloud._

These settings appear when you select **Workload Identity Federation** as the authentication type. This method lets each query run as the signed-in user by authenticating through an external OIDC identity provider, such as Okta, instead of a service account key. Setting it up spans Google Cloud, your Grafana Cloud stack, and the data source. For the complete procedure, refer to [Use Workload Identity Federation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/google-authentication/#use-workload-identity-federation).

| Setting                             | Description                                                                                                                                                                                                  |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Workload Identity Pool Provider** | Enter the full resource path of your provider, in the form `projects/<project-number>/locations/global/workloadIdentityPools/<pool-id>/providers/<provider-id>`. Use the project number, not the project ID. |
| **Service account email**           | _(Optional)_ Enter the service account to impersonate. Leave this blank if you granted the **Monitoring Viewer** role directly to the Workload Identity pool principal.                                      |
| **Default project**                 | Enter the GCP project ID where your Cloud Monitoring queries run.                                                                                                                                            |

{{< admonition type="note" >}}
Credentials from Workload Identity Federation are tied to the signed-in user's session, so features that run without a user present don't work. This includes alerting, scheduled reports, and public dashboards. If you rely on these features, use a service account key (Google JWT File) instead.
{{< /admonition >}}

### Configure service account impersonation

Use service account impersonation to have Grafana authenticate as a different service account than the one provided in the JWT token.

| Setting                            | Description                                                                                         |
| ---------------------------------- | --------------------------------------------------------------------------------------------------- |
| **Enable**                         | Toggle to enable service account impersonation.                                                     |
| **Service account to impersonate** | Enter the email address of the service account to impersonate when making requests to Google Cloud. |

### Private data source connect

_Only available for Grafana Cloud._

Use private data source connect (PDC) to connect to and query data within a secure network without opening that network to inbound traffic from Grafana Cloud. For more information on how PDC works, refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). For steps on setting up a PDC connection, refer to [Configure Grafana private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc).

| Setting                         | Description                                                                 |
| ------------------------------- | --------------------------------------------------------------------------- |
| **Private data source connect** | Select a PDC connection from the drop-down menu or create a new connection. |

### Save and test

Click **Save & test** to test the connection. A successful connection displays the following message:

`Successfully queried the Google Cloud Monitoring API.`

## Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system.
For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

### Provisioning examples

**Using the JWT (service account key file) authentication type:**

```yaml
apiVersion: 1

datasources:
  - name: Google Cloud Monitoring
    type: stackdriver
    access: proxy
    jsonData:
      tokenUri: https://oauth2.googleapis.com/token
      clientEmail: stackdriver@myproject.iam.gserviceaccount.com
      authenticationType: jwt
      defaultProject: my-project-name
      universeDomain: googleapis.com
    secureJsonData:
      privateKey: |
        -----BEGIN PRIVATE KEY-----
        POSEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCb1u1Srw8ICYHS
        ...
        yA+23427282348234=
        -----END PRIVATE KEY-----
```

**Using the JWT (service account private key path) authentication type:**

```yaml
apiVersion: 1

datasources:
  - name: Google Cloud Monitoring
    type: stackdriver
    access: proxy
    jsonData:
      tokenUri: https://oauth2.googleapis.com/token
      clientEmail: stackdriver@myproject.iam.gserviceaccount.com
      authenticationType: jwt
      defaultProject: my-project-name
      universeDomain: googleapis.com
      privateKeyPath: /etc/secrets/gce.pem
```

**Using GCE Default Service Account authentication:**

```yaml
apiVersion: 1

datasources:
  - name: Google Cloud Monitoring
    type: stackdriver
    access: proxy
    jsonData:
      authenticationType: gce
      universeDomain: googleapis.com
```

## Provision the data source using Terraform

You can provision the Google Cloud Monitoring data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation.

### Terraform prerequisites

Before you begin, ensure you have the following:

- [Terraform](https://www.terraform.io/downloads) installed.
- Grafana Terraform provider configured with appropriate credentials.
- For Grafana Cloud: A [Cloud Access Policy token](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) with data source permissions.

### Provider configuration

Configure the Grafana provider to connect to your Grafana instance:

```hcl
terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 2.0.0"
    }
  }
}

# For Grafana Cloud
provider "grafana" {
  url  = "<YOUR_GRAFANA_CLOUD_STACK_URL>"
  auth = "<YOUR_SERVICE_ACCOUNT_TOKEN>"
}

# For self-managed Grafana
# provider "grafana" {
#   url  = "http://localhost:3000"
#   auth = "<API_KEY_OR_SERVICE_ACCOUNT_TOKEN>"
# }
```

### Terraform examples

The following examples show how to configure the Google Cloud Monitoring data source for each authentication method.

**Using the JWT (service account key file) authentication type:**

```hcl
resource "grafana_data_source" "google_cloud_monitoring" {
  type = "stackdriver"
  name = "Google Cloud Monitoring"

  json_data_encoded = jsonencode({
    tokenUri           = "https://oauth2.googleapis.com/token"
    clientEmail        = "<SERVICE_ACCOUNT_EMAIL>"
    authenticationType = "jwt"
    defaultProject     = "<GCP_PROJECT_ID>"
    universeDomain     = "googleapis.com"
  })

  secure_json_data_encoded = jsonencode({
    privateKey = "<PRIVATE_KEY_CONTENT>"
  })
}
```

**Using the JWT (service account private key path) authentication type:**

```hcl
resource "grafana_data_source" "google_cloud_monitoring" {
  type = "stackdriver"
  name = "Google Cloud Monitoring"

  json_data_encoded = jsonencode({
    tokenUri           = "https://oauth2.googleapis.com/token"
    clientEmail        = "<SERVICE_ACCOUNT_EMAIL>"
    authenticationType = "jwt"
    defaultProject     = "<GCP_PROJECT_ID>"
    universeDomain     = "googleapis.com"
    privateKeyPath     = "/etc/secrets/gce.pem"
  })
}
```

**Using GCE Default Service Account authentication:**

```hcl
resource "grafana_data_source" "google_cloud_monitoring" {
  type = "stackdriver"
  name = "Google Cloud Monitoring"

  json_data_encoded = jsonencode({
    authenticationType = "gce"
    universeDomain     = "googleapis.com"
  })
}
```

For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source).

## Next steps

After you configure the Google Cloud Monitoring data source, you can:

- [Query GCP metrics](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/query-editor/) using the visual Builder, MQL, SLO, or PromQL query types.
- [Create template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/template-variables/) for dynamic, reusable dashboards.
- [Add annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/annotations/) to overlay GCP events on your graphs.
- [Set up alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/alerting/) to receive notifications based on GCP metrics and SLOs.
- [Explore your data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/) to investigate metrics without building a dashboard.
- [Build dashboards for GCP services](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/#import-dashboards-for-gcp-services) using community and Grafana-authored dashboards.

# Google authentication

This document explains how to configure authentication between Grafana and Google Cloud Platform (GCP). You must configure authentication before you can use the Google Cloud Monitoring data source to query metrics and SLOs.

All requests to Google APIs are performed on the server-side by the Grafana backend.

## Before you begin

Before you configure authentication, ensure you have the following:

- A Google Cloud Platform project with the Monitoring API enabled.
- Permissions to create service accounts or configure GCE instance settings in your GCP project.
- Access to the Grafana data source configuration page.

## Supported authentication methods

The Google Cloud Monitoring data source supports the following authentication methods:

| Method                            | Use case                                                                                                                 |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| **Google JWT File**               | Use when Grafana runs outside of GCP, or when you need explicit control over credentials.                                |
| **GCE Default Service Account**   | Use when Grafana runs on a Google Compute Engine VM with a configured service account.                                   |
| **Forward OAuth Identity**        | Use when you sign in to Grafana with Google and want each query to run as the signed-in user.                            |
| **Workload Identity Federation**  | Use on Grafana Cloud to let users authenticate with an external OIDC identity provider instead of a service account key. |
| **Service Account Impersonation** | Use when you need Grafana to act as a different service account than the one it authenticates with.                      |

## Use a Google service account key file

Use this method when Grafana runs outside of Google Cloud Platform, or when you need explicit control over which credentials are used.

Each Grafana data source connects to one GCP project by default. To visualize data from multiple GCP projects, create one data source per project, or use service account impersonation.

### Create a GCP service account and key file

To create a service account and download its key file:

1. Navigate to the [APIs and Services Credentials page](https://console.cloud.google.com/apis/credentials) in the GCP Console.
1. Click the **Create credentials** dropdown and select **Service account**.
1. In **Service account name**, enter a name for the account.
1. Click **Create and continue**.
1. In the **Grant this service account access to project** section, select the **Monitoring Viewer** role from the **Role** dropdown.
1. Click **Continue**, then click **Done**.
1. In the service accounts list, click the service account you created.
1. Go to the **Keys** tab and click **Add key** > **Create new key**.
1. Select **JSON** and click **Create**.

   A JSON key file downloads to your computer.

1. Store the key file securely. It grants access to your Google Cloud data.

### Upload the key file to Grafana

1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page.
1. Under **Authentication type**, select **Google JWT File**.
1. Upload the JSON key file using one of the available methods (drag and drop, browse, or paste).
1. Click **Save & test** to verify the connection.

### Grant access to multiple projects

You can configure a single service account to access multiple GCP projects:

1. Create a service account and key file following the steps above.
1. Note the service account email address (for example, `grafana-monitoring@my-project.iam.gserviceaccount.com`).
1. In each additional project you want to access:
   1. Navigate to **IAM & Admin** > **IAM**.
   1. Click **Grant access**.
   1. Enter the service account email address.
   1. Assign the **Monitoring Viewer** role.
   1. Click **Save**.
1. Use the original key file in Grafana. The service account can now access all configured projects.

## Use GCE Default Service Account

When Grafana runs on a Google Compute Engine (GCE) virtual machine, it can automatically retrieve credentials from the GCE metadata server. This method doesn't require you to create or manage key files.

### Prerequisites for GCE authentication

Before using this method, ensure the following:

- Grafana is running on a GCE virtual machine.
- The VM has a service account assigned with the **Monitoring Viewer** role.
- The VM has the **Cloud Monitoring API** scope enabled.

### Configure the GCE instance

1. In the GCP Console, navigate to **Compute Engine** > **VM instances**.
1. Stop the VM if it's running (you can't change scopes on a running VM).
1. Click the VM name, then click **Edit**.
1. Under **Service account**, select a service account with the **Monitoring Viewer** role.
1. Under **Access scopes**, select **Set access for each API** and enable the **Cloud Monitoring API** (read-only).
1. Click **Save** and restart the VM.

### Configure the data source

1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page.
1. Under **Authentication type**, select **GCE Default Service Account**.
1. Click **Save & test** to verify the connection.

For more information about GCE service accounts, refer to the [Google documentation on service accounts for instances](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances).

## Use Forward OAuth Identity

Use this method when your Grafana instance authenticates users with Google OAuth and you want each query to run as the signed-in user instead of as a shared service account. This enables per-user access control: a viewer who lacks Cloud Monitoring permissions on a project sees no data from that project, even when the dashboard's data source is shared.

Grafana forwards the user's existing Google OAuth access token as the bearer token on outgoing Cloud Monitoring API requests. No service account key is stored on the server.

### Prerequisites for Forward OAuth Identity

Before using this method, ensure the following:

- Your Grafana instance is configured to sign users in with [Google authentication](https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/google/).
- The Google OAuth client requests the `https://www.googleapis.com/auth/monitoring.read` scope on top of the default `openid email profile` scopes. Without this scope, the user's token is rejected by the Cloud Monitoring API with a `403` response.
- Each user who needs to query the data source has the **Monitoring Viewer** role (`roles/monitoring.viewer`) on the target GCP project.

### Configure the Google OAuth scope

Add the Cloud Monitoring read scope to your Google authentication configuration in `grafana.ini` or `custom.ini`:

```ini
[auth.google]
scopes = openid email profile https://www.googleapis.com/auth/monitoring.read
```

If you configure Google authentication through the Grafana SSO settings UI, add the same scope value to the **Scopes** field.

After you change the scopes, existing user sessions still hold tokens issued under the old scope set. Each affected user must sign out, revoke the existing grant at [https://myaccount.google.com/permissions](https://myaccount.google.com/permissions), and sign in again to consent to the new scope. Otherwise, queries continue to fail with `403`.

### Configure the data source

1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page.
1. Under **Authentication type**, select **Forward OAuth Identity**.
1. In **Default project**, enter the GCP project ID to query. This field is required because the user's OAuth token doesn't carry a project context.
1. Click **Save & test** to verify the connection.

{{< admonition type="note" >}}
Service account impersonation isn't compatible with Forward OAuth Identity. The data source authenticates as the signed-in user, so there's no service account to impersonate.
{{< /admonition >}}

## Use Workload Identity Federation

Use [Google Cloud Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) (WIF) to let Grafana users authenticate to Cloud Monitoring with an external identity provider (such as Okta or another OIDC provider) instead of a service account key.

{{< admonition type="note" >}}
This authentication method is available on **Grafana Cloud** only. Grafana Cloud exchanges the signed-in user's external OIDC token for a short-lived Google Cloud access token before the request reaches the plugin.
{{< /admonition >}}

Configuring Workload Identity Federation involves three systems: Google Cloud, your Grafana Cloud stack, and the data source itself.

### Configure Google Cloud

1. Create a [Workload Identity Pool and Provider](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-providers) that trusts your OIDC identity provider. When configuring the provider, set up attribute mappings so that `google.subject` maps to the relevant claim from your identity provider, such as `assertion.sub`. The exact mapping depends on your provider's claim format.
1. Grant the Cloud Monitoring permissions needed to run queries. How you grant them depends on whether you use service account impersonation:
   - **Without impersonation:** Grant the WIF pool principal the **Monitoring Viewer** role (`roles/monitoring.viewer`) directly.
   - **With impersonation:** Create a service account, grant it the **Monitoring Viewer** role, then grant the WIF pool principal the **Service Account Token Creator** role (`roles/iam.serviceAccountTokenCreator`) on that service account.

### Configure Grafana Cloud

1. Configure your Grafana Cloud stack's SSO integration against the same OIDC provider, so the signed-in user's identity is available for Grafana Cloud to exchange for a Google Cloud access token before the request reaches the plugin. Refer to [Configure OAuth2 authentication](https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/generic-oauth/) for setup details.

### Configure the data source

1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page.
1. Under **Authentication type**, select **Workload Identity Federation**.
1. In the **Workload Identity Pool Provider** field, enter the full resource path of your provider:

   `projects/<project-number>/locations/global/workloadIdentityPools/<pool-id>/providers/<provider-id>`

   {{< admonition type="note" >}}
   Use the **project number** (a numeric ID such as `123456789`), not the project ID (such as `my-project`). You can find the project number on the Google Cloud Console home page.
   {{< /admonition >}}

1. If you set up service account impersonation, enter the service account email in the **Service account email** field. If you granted permissions directly to the WIF pool, leave this blank.
1. In **Default project**, enter the GCP project ID where your Cloud Monitoring queries run.
1. Click **Save & test** to verify the connection.

{{< admonition type="note" >}}
Credentials from Workload Identity Federation are tied to the signed-in user's active session, so no long-lived credential is available to the Grafana backend. Any feature that runs without a user present won't work, including alerting, scheduled reports, and public dashboards. If you rely on these features, use a service account key (Google JWT File) instead.
{{< /admonition >}}

## Configure service account impersonation

Service account impersonation allows Grafana to authenticate as one service account but act as a different service account when making API requests. This is useful for:

- Accessing resources across multiple projects with a single configuration.
- Following the principle of least privilege by separating authentication from authorization.
- Auditing and tracking which service account accessed specific resources.

### Prerequisites for impersonation

The service account used by Grafana (the "caller") must have the following:

- The `iam.serviceAccounts.getAccessToken` permission on the target service account.
- This permission is included in the **Service Account Token Creator** role (`roles/iam.serviceAccountTokenCreator`).

The service account being impersonated (the "target") must have:

- The **Monitoring Viewer** role on the projects you want to access.

### Configure impersonation

1. In the GCP Console, grant the caller service account the **Service Account Token Creator** role on the target service account.
1. Grant the target service account the **Monitoring Viewer** role on the relevant projects.
1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page.
1. Configure authentication using either **Google JWT File** or **GCE Default Service Account**.
1. Enable **Service Account Impersonation**.
1. Enter the email address of the target service account.
1. Click **Save & test** to verify the connection.

For more information, refer to the [Google documentation on service account impersonation](https://cloud.google.com/iam/docs/service-account-impersonation).

# Google Cloud Monitoring query editor

This topic explains querying specific to the Google Cloud Monitoring data source.
For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/).

## Query types

The Google Cloud Monitoring query editor supports the following query types:

| Query type                                                            | Description                                                                                      |
| --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| [**Builder**](#query-metrics)                                         | Build metrics queries visually by selecting a service, metric, filters, and aggregation options. |
| [**MQL**](#use-the-monitoring-query-language)                         | Write queries using the Monitoring Query Language for advanced use cases.                        |
| [**Service Level Objectives (SLO)**](#query-service-level-objectives) | Query SLO data to track service reliability and error budgets.                                   |
| [**PromQL**](#query-with-promql)                                      | Write Prometheus-style queries against Google Cloud Monitoring metrics.                          |

## Query metrics

{{< figure src="/static/img/docs/google-cloud-monitoring/metric-query-builder-8-0.png" max-width="400px" class="docs-image--no-shadow" caption="Google Cloud Monitoring metrics query builder" >}}

The metrics query editor helps you select metrics, group and aggregate by labels and time, and use filters to specify which time series you want to query.

### Create a metrics query

1. Select **Builder** in the **Query type** drop-down.
1. Select a project from the **Project** drop-down.
1. Select a Google Cloud Platform service from the **Service** drop-down.
1. Select a metric from the **Metric** drop-down.
1. _(Optional)_ Use the plus and minus icons in the filter and group-by sections to add and remove filters or group-by clauses.

Google Cloud Monitoring supports several metrics types, such as `GAUGE`, `DELTA,` and `CUMULATIVE`.
Each supports different aggregation options, such as reducers and aligners. Additionally, metrics have specific value types that can be either scalar or a distribution.

The metrics query editor lists available aggregation methods for a selected metric, and sets a default aggregation, reducer and aligner when you select a metric.

In the case that the metric value type is a distribution, the aggregation will be set by default to the mean. For scalar value types, there is no aggregation by default.

For a full list of metrics, refer to the [Google Cloud metrics list](https://cloud.google.com/monitoring/api/metrics_gcp). For more details on metric kinds and value types, refer to [Metric kinds and value types](https://cloud.google.com/monitoring/api/v3/kinds-and-types).

{{< admonition type="note" >}}
Distribution metrics are typically best visualized as either a heatmap or histogram. When visualizing in this way, aggregation is not necessary. However, for other visualization types, performance degradation may be observed when attempting to query distribution metrics that are not aggregated due to the number of potential buckets that can be returned. For more information on how to visualize distribution metrics refer to [this page](https://cloud.google.com/monitoring/charts/charting-distribution-metrics).
{{< /admonition >}}

### Apply a filter

**To add and apply a filter:**

1. Click the plus icon in the filter section.
1. Select a field to filter by.
1. Enter a filter value, such as `instance_name = grafana-1`.

To remove the filter, click the trash icon.

#### Use simple wildcards

When you set the operator to `=` or `!=`, you can add wildcards to the filter value field.

For example, entering `us-*` captures all values that start with "us-", and entering `*central-a` captures all values that end with "central-a".
`*-central-*` captures all values with the substring of "-central-".

Simple wildcards perform better than regular expressions.

#### Use regular expressions

When you set the operator to `=~` or `!=~`, you can add regular expressions to the filter value field.

For example, entering `us-central[1-3]-[af]` matches all values that start with "us-central", then are followed by a number in the range of 1 to 3, a dash, and then either an "a" or an "f".

You don't need leading and trailing slashes when you create these regular expressions.

### Configure pre-processing options

The query editor displays pre-processing options when the selected metric has a metric type of `DELTA` or `CUMULATIVE`.

- The **Rate** option aligns and converts data points to a rate per time series.
- The **Delta** option aligns data points by their delta (difference) per time series.

### Group time series

To combine multiple time series and reduce the amount of data returned for a metric, specify a grouping and a function.

#### Group by

Use the **Group By** segment to group resource or metric labels, which reduces the number of time series and aggregates the results by group.

For example, if you group by `instance_name`, Grafana displays an aggregated metric for the specified Compute instance.

##### Metadata labels

Resource metadata labels contain information that can uniquely identify a resource in Google Cloud.
A time series response returns metadata labels only if they're part of the Group By segment in the time series request.

There's no API for retrieving metadata labels, so you can't populate the Group By list with the metadata labels that are available for the selected service and metric.
However, the **Group By** field list comes with a pre-defined set of common system labels.

You can't pre-define user labels, but you can enter them manually in the **Group By** field.
If you include a metadata label, user label, or system label in the **Group By** segment, you can create filters based on it and expand its value on the **Alias** field.

#### Group by function

To combine the time series in the group into a single time series, select a grouping function.

### Align data

You can align all data points received in a fixed length of time by applying an alignment function to combine those data points, and then assigning a timestamp to the result.

#### Select an alignment function

During alignment, all data points are received in a fixed interval.
Within each interval, as defined by the [alignment period](#define-the-alignment-period), and for each time series, the data is aggregated into a single point.
The value of that point is determined by the type of alignment function you use.

For more information on alignment functions, refer to [alignment metric selector](https://cloud.google.com/monitoring/charts/metrics-selector#alignment).

#### Define the alignment period

You can set the **Alignment Period** field to group a metric by time if you've chosen an aggregation.

The default, called "cloud monitoring auto", uses GCP Google Cloud Monitoring's default groupings.
These enable you to compare graphs in Grafana with graphs in the Google Cloud Monitoring UI.

The default values for "cloud monitoring auto" are:

- 1m for time ranges < 23 hours
- 5m for time ranges >= 23 hours and < 6 days
- 1h for time ranges >= 6 days

The other automatic option is "grafana auto", which automatically sets the Group By time depending on the time range chosen and width of the time series panel.

For more information about "grafana auto", refer to [Interval variable](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#add-an-interval-variable).

You can also choose fixed time intervals to group by, like `1h` or `1d`.

### Set alias patterns

The **Alias By** helps you control the format of legend keys.
By default, Grafana shows the metric name and labels, which can be long and difficult to read.

You can use patterns in the alias field to customize the legend key's format:

| Alias pattern                    | Description                                      | Alias pattern example             | Example result                                    |
| -------------------------------- | ------------------------------------------------ | --------------------------------- | ------------------------------------------------- |
| `{{metric.label.xxx}}`           | Returns the metric label value.                  | `{{metric.label.instance_name}}`  | `grafana-1-prod`                                  |
| `{{metric.type}}`                | Returns the full Metric Type.                    | `{{metric.type}}`                 | `compute.googleapis.com/instance/cpu/utilization` |
| `{{metric.name}}`                | Returns the metric name part.                    | `{{metric.name}}`                 | `instance/cpu/utilization`                        |
| `{{metric.service}}`             | Returns the service part.                        | `{{metric.service}}`              | `compute`                                         |
| `{{resource.label.xxx}}`         | Returns the resource label value.                | `{{resource.label.zone}}`         | `us-east1-b`                                      |
| `{{resource.type}}`              | Returns the name of the monitored resource type. | `{{resource.type}}`               | `gce_instance`                                    |
| `{{metadata.system_labels.xxx}}` | Returns the metadata system label value.         | `{{metadata.system_labels.name}}` | `grafana`                                         |
| `{{metadata.user_labels.xxx}}`   | Returns the metadata user label value.           | `{{metadata.user_labels.tag}}`    | `production`                                      |

#### Alias pattern examples

**Using metric labels:**

You can select from a list of metric and resource labels for a metric in the Group By drop-down.
You can then include them in the legend key using alias patterns.

For example, given this value of Alias By: `{{metric.type}} - {{metric.label.instance_name}}`

An expected result would look like: `compute.googleapis.com/instance/cpu/usage_time - server1-prod`

**Using the Monitored Resource Type:**

You can also resolve the name of the Monitored Resource Type with the `resource.type` alias.

For example, given this value of Alias By: `{{resource.type}} - {{metric.type}}`

An expected result would look like: `gce_instance - compute.googleapis.com/instance/cpu/usage_time`

### Deep-link from Grafana panels to the Google Cloud Console Metrics Explorer

{{< figure src="/static/img/docs/v71/cloudmonitoring_deep_linking.png" max-width="500px" class="docs-image--no-shadow" caption="Google Cloud Monitoring deep linking" >}}

You can click on a time series in the panel to access a context menu, which contains a link to **View in Metrics Explorer in Google Cloud Console**.
The link points to the Google Cloud Console's Metrics Explorer and runs the Grafana panel's query there.

If you select the link, you first select an account in the Google Account Chooser.
Google then redirects you to the Metrics Explorer.

The provided link is valid for any account, but displays the query only if your account has access to the query's specified GCP project.

### Understand automatic unit detection

Grafana issues one query to the Cloud Monitoring API per query editor row, and each API response includes a unit.
Grafana attempts to convert the returned unit into a unit compatible with its time series panel.

If successful, Grafana displays the unit on the panel's Y-axis.
If the query editor rows return different units, Grafana uses the unit from the last query editor row in the time series panel.

### Use the Monitoring Query Language

The Monitoring Query Language (MQL) query builder helps you query and display MQL results in time series format.
To understand basic MQL concepts, refer to [Introduction to Monitoring Query Language](https://cloud.google.com/monitoring/mql).

#### Create an MQL query

**To create an MQL query:**

1. Select **MQL** in the **Query type** drop-down.
1. Select a project from the **Project** drop-down.
1. Enter your MQL query in the text area.
1. _(Optional)_ Configure the **Graph period** setting.

Press `Shift+Enter` to run the query.

#### Configure MQL options

The following options are available for MQL queries:

| Setting          | Description                                                                                                                                            |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Alias by**     | Control the format of legend keys. Refer to [Set alias patterns for MQL queries](#set-alias-patterns-for-mql-queries) for available patterns.          |
| **Graph period** | Enable the toggle to override the default time period. Select a period from the drop-down to control the granularity of the returned time series data. |

### Set alias patterns for MQL queries

MQL queries use the same alias patterns as [metric queries](#set-alias-patterns).

However, `{{metric.service}}` is not supported, and `{{metric.type}}` and `{{metric.name}}` show the time series key in the response.

## Query Service Level Objectives

{{< figure src="/static/img/docs/google-cloud-monitoring/slo-query-builder-8-0.png" max-width="400px" class="docs-image--no-shadow" caption="Service Level Objectives (SLO) query editor" >}}

The SLO query builder helps you visualize SLO data in time series format.
To understand basic concepts in service monitoring, refer to the [Google Cloud Monitoring documentation](https://cloud.google.com/monitoring/service-monitoring).

### Create an SLO query

**To create an SLO query:**

1. Select **Service Level Objectives (SLO)** in the **Query type** drop-down.
1. Select a project from the **Project** drop-down.
1. Select an [SLO service](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/services) from the **Service** drop-down.
1. Select an [SLO](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/services.serviceLevelObjectives) from the **SLO** drop-down.
1. Select a [time series selector](https://cloud.google.com/stackdriver/docs/solutions/slo-monitoring/api/timeseries-selectors#ts-selector-list) from the **Selector** drop-down.

Time series selectors in Grafana use descriptive names that map to system names that Google uses in the Service Monitoring documentation:

| Selector drop-down value       | Corresponding time series selector |
| ------------------------------ | ---------------------------------- |
| **SLI Value**                  | `select_slo_health`                |
| **SLO Compliance**             | `select_slo_compliance`            |
| **SLO Error Budget Remaining** | `select_slo_budget_fraction`       |
| **SLO Burn Rate**              | `select_slo_burn_rate`             |

### Alias patterns for SLO queries

The **Alias By** field helps you control the format of legend keys for SLO queries.

| Alias pattern  | Description                   | Example result      |
| -------------- | ----------------------------- | ------------------- |
| `{{project}}`  | Returns the GCP project name. | `myProject`         |
| `{{service}}`  | Returns the service name.     | `myService`         |
| `{{slo}}`      | Returns the SLO.              | `latency-slo`       |
| `{{selector}}` | Returns the selector.         | `select_slo_health` |

### Alignment period and group-by time for SLO queries

SLO queries use the same alignment period functionality as [metric queries](#define-the-alignment-period).

## Query with PromQL

The PromQL query type allows you to query Google Cloud Monitoring metrics using Prometheus Query Language (PromQL) syntax. This is useful if you're familiar with PromQL from Prometheus or Grafana Mimir and want to use the same query syntax with Google Cloud Monitoring data.

For more information about PromQL support in Google Cloud Monitoring, refer to the [Google Cloud documentation on PromQL](https://cloud.google.com/monitoring/promql).

### Create a PromQL query

To create a PromQL query:

1. Select **PromQL** in the **Query type** drop-down.
1. Select a project from the **Project** drop-down.
1. Enter your PromQL query in the text area.

### Configure PromQL options

The following options are available for PromQL queries:

| Setting      | Description                                                                                                                                                                                       |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Min step** | Defines the lower bounds on the interval between data points. For example, set this to `1h` to hint that measurements are taken hourly. Supports the `$__interval` and `$__rate_interval` macros. |

### PromQL query examples

The following examples show common PromQL query patterns for Google Cloud Monitoring:

**Query CPU utilization for Compute Engine instances:**

```promql
compute_googleapis_com:instance_cpu_utilization
```

**Filter by label:**

```promql
compute_googleapis_com:instance_cpu_utilization{instance_name="my-instance"}
```

**Calculate the rate of a counter metric:**

```promql
rate(logging_googleapis_com:log_entry_count[5m])
```

# Google Cloud Monitoring template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
Grafana refers to such variables as template variables.

For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/) documentation.

## Use query variables

Variables of the type _Query_ help you query Google Cloud Monitoring for various types of data.
The Google Cloud Monitoring data source provides the following **Query Types**:

| Name                               | List returned                                                         |
| ---------------------------------- | --------------------------------------------------------------------- |
| **Metric Types**                   | Metric type names available for the specified service.                |
| **Labels Keys**                    | Keys for `metric label` and `resource label` in the specified metric. |
| **Labels Values**                  | Values for the label in the specified metric.                         |
| **Resource Types**                 | Resource types for the specified metric.                              |
| **Aggregations**                   | Aggregations (cross-series reducers) for the specified metric.        |
| **Aligners**                       | Aligners (per-series aligners) for the specified metric.              |
| **Alignment periods**              | All alignment periods available in the query editor in Grafana.       |
| **Selectors**                      | Selectors for SLO (Service Level Objectives) queries.                 |
| **SLO Services**                   | Service Monitoring services for SLO queries.                          |
| **Service Level Objectives (SLO)** | SLOs for the specified SLO service.                                   |

## Use variables in queries

Use Grafana's variable syntax to include variables in queries.
For details, refer to the [variable syntax documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/).

# Troubleshoot Google Cloud Monitoring data source issues

This document provides solutions to common issues you may encounter when configuring or using the Google Cloud Monitoring data source. For configuration instructions, refer to [Configure Google Cloud Monitoring](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/configure/).

## Authentication errors

These errors occur when GCP credentials are invalid, missing, or don't have the required permissions.

### "Permission denied" or "Access denied"

**Symptoms:**

- Save & test fails with permission errors
- Queries return authorization errors
- Projects, metrics, or labels don't load

**Possible causes and solutions:**

| Cause                                        | Solution                                                                                                                                                                                                                                                                      |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Service account missing required permissions | Assign the **Monitoring Viewer** role to the service account in the GCP Console under **IAM & Admin** > **IAM**. Refer to [Configure Google Cloud Monitoring](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/configure/) for details. |
| Incorrect service account key file           | Verify the JSON key file was downloaded correctly and contains valid credentials. Generate a new key if necessary.                                                                                                                                                            |
| Service account key has been deleted         | Check the service account in GCP Console under **IAM & Admin** > **Service Accounts**. If the key was deleted, create a new one.                                                                                                                                              |
| Wrong project selected                       | Verify the default project in the data source configuration matches a project the service account has access to.                                                                                                                                                              |
| APIs not enabled                             | Enable the Monitoring API and Cloud Resource Manager API in the GCP Console. Refer to [Configure Google Cloud Monitoring](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/configure/) for links.                                       |

### "Invalid JWT" or "JWT token error"

**Symptoms:**

- Authentication fails when using Google JWT File
- Error message references invalid or malformed JWT

**Solutions:**

1. Verify you uploaded the complete JSON key file, not just the private key portion.
1. Check that the JSON file is properly formatted and not corrupted.
1. Ensure the key file contains all required fields: `type`, `project_id`, `private_key_id`, `private_key`, `client_email`, `client_id`, `auth_uri`, `token_uri`.
1. Generate a new service account key and re-upload it.

### GCE Default Service Account not working

**Symptoms:**

- Data source test fails when using GCE Default Service Account
- Works with JWT but fails with GCE authentication

**Solutions:**

1. Verify Grafana is running on a Google Compute Engine (GCE) virtual machine.
1. Check that the GCE instance has the **Cloud Monitoring API** scope enabled.
1. Verify the GCE default service account has the **Monitoring Viewer** role.
1. If the VM was created without the required scope, you may need to stop the instance, edit it to add the scope, and restart.

### Service account impersonation errors

**Symptoms:**

- Authentication fails when service account impersonation is enabled
- Error: "Unable to impersonate service account"

**Solutions:**

1. Verify the primary service account has the `roles/iam.serviceAccountTokenCreator` role on the target service account.
1. Check that the target service account email is entered correctly.
1. Ensure the target service account has the **Monitoring Viewer** role.
1. Verify both service accounts are in projects that have the required APIs enabled.

### Forward OAuth Identity returns 403

**Symptoms:**

- Save & Test fails with `403 Forbidden` when using Forward OAuth Identity.
- Queries succeed for some users but not others.

**Solutions:**

1. Confirm you're signed in to Grafana with Google, not as the local `admin` user. The Forward OAuth Identity method only forwards a token if the current session has one.
1. Verify the Grafana Google authentication includes the `https://www.googleapis.com/auth/monitoring.read` scope. Refer to [Configure the Google OAuth scope](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/google-authentication/#configure-the-google-oauth-scope).
1. After you add the scope, sign out of Grafana, revoke the existing grant at [https://myaccount.google.com/permissions](https://myaccount.google.com/permissions), and sign back in. Google reuses the previous consent and doesn't reissue tokens with new scopes until the grant is revoked.
1. Verify the signed-in user has the **Monitoring Viewer** role (`roles/monitoring.viewer`) on the project set in **Default project**.
1. Check the **Default project** field is populated. The user's OAuth token doesn't carry a project context, so the data source can't fall back to a project from the credentials.

## Connection errors

These errors occur when Grafana cannot reach Google Cloud Monitoring endpoints.

### "Request timed out" or connection failures

**Symptoms:**

- Data source test times out
- Queries fail with timeout errors
- Intermittent connection issues

**Solutions:**

1. Verify network connectivity from the Grafana server to Google Cloud endpoints (`monitoring.googleapis.com`).
1. Check firewall rules allow outbound HTTPS (port 443) to Google Cloud services.
1. For Grafana Cloud connecting to private resources, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/).
1. Check if a corporate proxy is blocking connections to Google Cloud.

### "SSL certificate problem"

**Symptoms:**

- SSL/TLS handshake errors
- Certificate verification failures

**Solutions:**

1. Ensure the system time is correct on the Grafana server.
1. Verify the Grafana server has up-to-date CA certificates installed.
1. Check if a corporate proxy is intercepting HTTPS traffic.

## Metrics query errors

These errors occur when querying Google Cloud Monitoring metrics.

### "No data" or empty results

**Symptoms:**

- Query executes without error but returns no data
- Charts show "No data" message

**Possible causes and solutions:**

| Cause                           | Solution                                                                                                              |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Time range doesn't contain data | Expand the dashboard time range. GCP metrics have different retention periods based on the metric type.               |
| Wrong project selected          | Verify you've selected the correct project in the query editor.                                                       |
| Incorrect metric type           | Verify the service, metric type, and metric are correct. Check available metrics in the GCP Console Metrics Explorer. |
| Missing labels/filters          | Some metrics require specific label filters to return data. Try removing filters to see if data appears.              |
| Resource not emitting metrics   | Verify the resource exists and is actively emitting metrics. Some metrics only populate under certain conditions.     |

### Metrics don't appear in drop-down

**Symptoms:**

- Expected metrics don't appear in the query editor
- Metric drop-down is empty for a service

**Solutions:**

1. Verify the metric exists in the selected project and region.
1. Check that the service account has the **Monitoring Viewer** role.
1. Some metrics are only available for specific resource types. Check the [Google Cloud metrics list](https://cloud.google.com/monitoring/api/metrics_gcp).
1. Use the Query Inspector to verify the API request and response.

### Label values not loading

**Symptoms:**

- Label value drop-down doesn't populate
- Filters can't be applied

**Solutions:**

1. Verify the service account has the **Monitoring Viewer** role.
1. Ensure a project, service, and metric are selected before label values can load.
1. Label values are populated from existing metric data. If no metrics match the current selection, no values appear.

### MQL query errors

**Symptoms:**

- Monitoring Query Language (MQL) queries fail with syntax errors
- MQL query returns unexpected results

**Solutions:**

1. Validate your MQL syntax using the [MQL reference documentation](https://cloud.google.com/monitoring/mql/reference).
1. Check for typos in metric types, label names, or function names.
1. Ensure time range syntax is valid in your query.
1. Test the query in the GCP Console Metrics Explorer before using it in Grafana.

### "Too many data points" or API throttling

**Symptoms:**

- Queries fail with quota errors
- Performance degrades with multiple panels

**Solutions:**

1. Increase the alignment period to reduce the number of data points.
1. Reduce the time range of your queries.
1. Use fewer metric queries per panel.
1. Request a quota increase in the GCP Console under **APIs & Services** > **Quotas**.
1. Enable query caching in Grafana to reduce API calls.

## SLO query errors

These errors are specific to Service Level Objective (SLO) queries.

### SLO services don't appear

**Symptoms:**

- SLO service selector is empty
- Can't find expected SLO services

**Solutions:**

1. Verify SLOs are defined in Google Cloud Monitoring for the selected project.
1. Check that the service account has access to view SLOs.
1. Ensure the project has services configured in the Service Monitoring section of the GCP Console.

### SLO query returns no data

**Symptoms:**

- SLO query executes but returns no data
- SLO values show as empty

**Solutions:**

1. Verify the SLO exists and is active in the GCP Console.
1. Check that the time range includes periods when the SLO had data.
1. Ensure the selected SLO selector (SLI value, compliance, error budget, etc.) is appropriate for the SLO type.
1. Some SLOs may not have data if the underlying service hasn't received traffic.

## Template variable errors

These errors occur when using template variables with the Google Cloud Monitoring data source.

### Variables return no values

**Symptoms:**

- Variable drop-down is empty
- Dashboard fails to load with variable errors

**Solutions:**

1. Verify the data source connection is working.
1. Check that the service account has permissions to list the requested resources.
1. For dependent variables, ensure parent variables have valid selections.
1. Verify the project is selected correctly in the variable query.

### Variables are slow to load

**Symptoms:**

- Dashboard takes a long time to load
- Variable selectors are slow to populate

**Solutions:**

1. Set variable refresh to **On dashboard load** instead of **On time range change**.
1. Reduce the scope of variable queries (filter by specific project or service).
1. Limit the number of dependent variables in a chain.

For more information on template variables, refer to the [template variables documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/template-variables/).

## Dashboard issues

These issues occur with dashboards that use the Google Cloud Monitoring data source. As of plugin version 12.6.1, the plugin no longer bundles curated dashboards, so the data source **Dashboards** tab doesn't list them. To build dashboards, use community and Grafana-authored dashboards from the [Grafana dashboards catalog](https://grafana.com/grafana/dashboards/?dataSource=stackdriver) or recreate panels with the [query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/google-cloud-monitoring/query-editor/).

### Imported dashboards show no data

**Symptoms:**

- Imported dashboards show empty panels
- Template variables don't load

**Solutions:**

1. Verify the data source name in the dashboard matches your Google Cloud Monitoring data source.
1. Check that the service account has access to the projects shown in the project variable.
1. Ensure the resources (Compute Engine instances, Cloud SQL, and similar) exist and are emitting metrics.
1. Verify the required GCP services are enabled in your project.

## Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location).
1. Look for Google Cloud Monitoring-specific entries that include request and response details.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you've tried the solutions above and still encounter issues:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review [Google Cloud Monitoring issues on GitHub](https://github.com/grafana/grafana/issues) for known bugs.
1. Consult the [Google Cloud Monitoring documentation](https://cloud.google.com/monitoring/docs) for service-specific guidance.
1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user.
1. When reporting issues, include:
   - Grafana version
   - GCP project (redact if sensitive)
   - Error messages (redact sensitive information)
   - Steps to reproduce
   - Query configuration (redact credentials)

# Graphite alerting

You can use Grafana Alerting with Graphite to create alerts based on your time series data. This lets you monitor metrics, detect anomalies, and receive notifications when specific conditions are met.

For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/).

## Before you begin

Before creating alerts with Graphite, ensure you have:

- A Graphite data source configured in Grafana. Refer to [Configure the Graphite data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/graphite/configure/).
- Permission to create alert rules.
- An understanding of the metrics you want to monitor.

## Supported queries

Grafana Alerting evaluates any Graphite query that returns time series data. Grafana runs the query on the backend, reduces each series to a single value, and compares that value against the condition you define.

| Query pattern                  | Alerting support | Notes                                                                        |
| ------------------------------ | ---------------- | ---------------------------------------------------------------------------- |
| Single metric path             | Yes              | The most predictable option for alerting.                                    |
| Aggregated series              | Yes              | Use `sumSeries()`, `averageSeries()`, or similar to combine matching series. |
| Wildcard metric path           | Yes              | Returns one series per match. Reduce or aggregate before the condition.      |
| Tag query with `seriesByTag()` | Yes              | Requires Graphite 1.1 or later.                                              |

## Create an alert rule

To create an alert rule using Graphite:

1. Navigate to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for the alert rule.
1. In the query section, select your **Graphite** data source.
1. Build your query:
   - Click **Select metric** to choose the metric path, or click the pencil icon to write the query in code.
   - Add Graphite functions to aggregate or shape the series, for example `sumSeries()` or `summarize()`.
1. Add a **Reduce** expression to convert the series to a single value, for example `Last` or `Mean`.
1. Add a **Threshold** expression that defines when the alert fires, for example when the value is above a limit.
1. Set the evaluation interval and pending period.
1. Configure notifications and labels.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example alert queries

The following examples show common alerting scenarios with Graphite. Each query returns a single series that the alert condition reduces to one value.

### Alert on high CPU usage

Average CPU usage across every core on a host and alert when it exceeds 90%:

```text
averageSeries(servers.web01.cpu.*.percent.user)
```

**Condition:** Reduce with `Mean`, then alert when the value is above `90`.

### Alert on low free disk space

Monitor available disk space and alert when it drops below 10 GB:

```text
servers.web01.disk.root.free_bytes
```

**Condition:** Reduce with `Last`, then alert when the value is below `10737418240`.

### Alert on high error rate

Combine error counts across a service and alert on spikes:

```text
summarize(sumSeries(stats.myservice.errors.count), '5min', 'sum', false)
```

**Condition:** Reduce with `Last`, then alert when the value is above `100`.

### Alert on request latency

Monitor 95th percentile latency and alert when it exceeds 500 ms:

```text
stats.timers.myservice.request.latency.p95
```

**Condition:** Reduce with `Mean`, then alert when the value is above `500`.

### Alert across many hosts

Use a wildcard to evaluate every matching host and alert when any host crosses the threshold:

```text
aliasByNode(servers.*.cpu.percent.user, 1)
```

Because the wildcard returns one series per host, configure the alert rule to evaluate each series separately so you get a distinct alert instance for each host that breaches the threshold.

## Limitations

When using Graphite with Grafana Alerting, be aware of the following limitations.

### Template variables not supported

Alert queries can't contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$server` or `$environment` aren't resolved.

If your dashboard query uses template variables, create a separate query for alerting with hard-coded values.

### Nested queries not supported

Alert queries can't reference another query by letter, such as `#A`. Build a self-contained query for each alert rule instead of chaining query references.

### Consolidation affects results

Grafana consolidates data points so that a query doesn't return more points than needed. Consolidation can change the value the alert evaluates. Use `consolidateBy()` or `summarize()` to control how Graphite aggregates points, and match the interval to your evaluation frequency.

### Metrics that stop reporting

When a metric stops reporting, the query returns no data rather than a zero value, so a threshold alone can't detect the gap. To alert when a service stops sending metrics, set the alert rule's **No Data** state to **Alerting**. Configure this behavior in the alert rule under the evaluation options. For more information, refer to [Configure no data and error handling](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/#configure-no-data-and-error-handling).

## Best practices

Follow these best practices when creating Graphite alerts:

- **Return a single series:** Aggregate wildcard queries with `sumSeries()` or `averageSeries()`, or evaluate each series separately when you want per-target alerts.
- **Control consolidation:** Use `consolidateBy()` or `summarize()` so the evaluated value matches what you expect.
- **Test queries first:** Verify your query returns the expected values in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/) before creating an alert.
- **Set realistic thresholds:** Base thresholds on historical data patterns to avoid false positives.
- **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor.

## Next steps

- [Build queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/graphite/query-editor/) to explore your metrics before creating alerts.
- [Troubleshoot Graphite data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/graphite/troubleshooting/) if alerts aren't firing as expected.

# Graphite annotations

Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics. You can use Graphite as a data source for annotations to display events such as deployments, incidents, or maintenance windows on your visualizations.

For general information about annotations in Grafana, refer to [Annotate visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/).

## Common use cases

Graphite annotations help you explain changes in your metrics by marking when related events happened. Common use cases include:

- **Deployments:** Mark each release so you can see whether a deployment changed latency, error rate, or throughput.
- **Incidents and outages:** Mark when incidents start and resolve to correlate them with metric anomalies.
- **Maintenance windows:** Mark planned maintenance so expected dips aren't mistaken for problems.
- **Configuration changes:** Mark when you change a feature flag or setting to track its effect over time.
- **Scaling events:** Mark when you add or remove capacity to explain shifts in load or performance.

Each use case maps to one of the two query modes described in this page: a metric query when an existing metric already marks the event, or an events query when you record discrete events in Graphite.

## Before you begin

Before creating Graphite annotations, ensure you have:

- A configured Graphite data source in Grafana.
- Metrics or events in Graphite that represent the events you want to annotate.
- Read access to the Graphite instance.

## Annotation query modes

Graphite supports two ways to query annotations. You configure both in the annotation query editor:

| Mode             | Field                    | Description                                                                                                                                         |
| ---------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Metric query** | **Graphite Query**       | Runs a standard Graphite metric query. Each returned data point with a non-zero value becomes an annotation, and the series name becomes the title. |
| **Events query** | **Graphite events tags** | Queries the Graphite events API and filters by one or more event tags. Each matching event becomes an annotation.                                   |

Use the metric query mode when a metric is non-zero only at the moments you want to mark. A counter that stays above zero after the first event marks every interval, because Grafana creates an annotation for every non-zero point. Use the events query mode when you record discrete events in Graphite and tag them.

## Create an annotation query

To add a Graphite annotation to your dashboard:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Annotation query**.
1. Enter a name for the annotation query.
1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox.
1. Select a color for the annotation event markers.
1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed.
1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed.
1. Click **Open query editor** to open the **Annotation Query** dialog box.
1. Select the **Graphite** data source from the **Data source** drop-down list.
1. Configure one of the annotation query modes:
   - In the **Graphite Query** field, enter a metric query, for example `statsd.application.counters.*.count`.
   - In the **Graphite events tags** field, enter one or more event tags, for example `deploy`.
1. (Optional) Click **Test annotation query** to ensure that the query is working properly.
1. Click **Close** when you've completed the query setup.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

{{< admonition type="note" >}}
Enter a value in either the **Graphite Query** field or the **Graphite events tags** field, not both. If you provide a metric query, Grafana uses the metric query mode and ignores the tags.
{{< /admonition >}}

## Metric query annotations

In metric query mode, Grafana runs the value in the **Graphite Query** field as a standard Graphite query over the dashboard time range. Grafana converts the results into annotations using these rules:

- Every data point with a non-zero value creates an annotation. Data points with a value of zero or `null` are skipped.
- The annotation timestamp is the timestamp of the data point.
- The annotation title is the series name returned by Graphite.

Because every non-zero data point creates an annotation, use a metric that only produces values at the moments you want annotated. Metrics that report continuous values produce an annotation at every step interval and flood your dashboard.

### Example metric queries

The following examples show common ways to turn a metric into annotations.

#### Deployment markers

Record deployments as Graphite events and query them by tag. Refer to [Events query annotations](#events-query-annotations).

If you only have a counter, query the increase between points. `nonNegativeDerivative()` is zero while the counter is flat and non-zero only when it goes up, so annotations appear at each deploy:

```text
nonNegativeDerivative(deploys.myservice.count)
```

#### Service restarts

Use the same pattern for restarts. Query a metric that is non-zero only when a restart happens, or record restarts as events:

```text
nonNegativeDerivative(stats.myservice.restart.count)
```

#### Error spikes above a threshold

Because Grafana skips zero and `null` data points, use `removeBelowValue()` to annotate only the moments a metric crosses a threshold. The following query annotates every interval where the error count is 100 or greater:

```text
removeBelowValue(stats.myservice.errors.count, 100)
```

Values below 100 become `null` and are skipped, so annotations appear only during error spikes.

#### Host down

Annotate while a host is down by inverting an up metric so that a down state produces a non-zero value:

```text
offset(scale(hosts.web01.up, -1), 1)
```

This query returns `1` when `up` is `0`. Grafana skips the zero points from when the host is up, and creates an annotation for each interval the host stays down.

## Events query annotations

In events query mode, Grafana queries the Graphite events API and filters events by the tags you provide in the **Graphite events tags** field. Grafana converts each matching event into an annotation using these rules:

- The annotation timestamp is the event's `when` value.
- The annotation title is the event's `what` value.
- The tags you query with are attached to the annotation.

Enter multiple tags separated by a space. Grafana sends those tags to the Graphite events API without `set=union`, so an event must include every tag.

You can filter events in the following ways:

| Value         | Result                                                                                     |
| ------------- | ------------------------------------------------------------------------------------------ |
| Empty         | Returns all events in the dashboard time range.                                            |
| A single tag  | Returns events that include that tag, for example `deploy`.                                |
| Multiple tags | Returns events that include every tag. `deploy myservice` matches events tagged with both. |
| A wildcard    | Returns events whose tags match the pattern, for example `web*`.                           |

### Create events in Graphite

Before you can query events, record them in Graphite. Graphite exposes an HTTP endpoint for creating events:

```sh
curl -X POST http://<GRAPHITE_HOST>/events/ \
  -H "Content-Type: application/json" \
  -d '{
    "what": "Deployed myservice v2.5.0",
    "tags": ["deploy", "myservice"],
    "data": "Deployed new feature flags and performance improvements"
  }'
```

Replace _`<GRAPHITE_HOST>`_ with your Graphite server address. The `what` field becomes the annotation title, and the `tags` field is what you filter on in the **Graphite events tags** field.

### Example events queries

The following examples show what to enter in the **Graphite events tags** field for common scenarios.

| Goal                        | Value              | Result                                                     |
| --------------------------- | ------------------ | ---------------------------------------------------------- |
| Show all events             | Empty              | Displays every event in the dashboard time range.          |
| Show all deployments        | `deploy`           | Displays events tagged `deploy`.                           |
| Deployments for one service | `deploy myservice` | Displays events tagged with both `deploy` and `myservice`. |
| All production events       | `prod*`            | Displays events whose tags match the `prod*` pattern.      |
| Incidents                   | `incident`         | Displays events tagged `incident`.                         |

### Correlate multiple event types

To compare different kinds of events on the same dashboard, add more than one annotation query and give each a distinct color. For example:

1. Add a **Deployments** annotation query with the tag `deploy`.
1. Add an **Incidents** annotation query with the tag `incident`.
1. Add a **Maintenance** annotation query with the tag `maintenance`.

Each query renders in its own color, so you can see how deployments, incidents, and maintenance windows line up with your metrics.

## Use template variables in annotations

You can use template variables in both annotation query modes to filter annotations based on dashboard variable selections. Grafana resolves the variables at query time using the current dashboard variable values.

Metric query with a variable:

```text
nonNegativeDerivative(deploys.$service.count)
```

Events query with a variable:

```text
deploy $environment
```

For more information about template variables, refer to [Graphite template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/graphite/template-variables/).

## End-to-end example: correlate deployments with latency

This walkthrough shows how to combine a Graphite events annotation query with a metric panel so you can see whether deployments affect request latency. The result is a time series panel that plots latency with a vertical marker at each deployment.

### Before you start

Ensure you have the following:

- A Graphite data source configured in Grafana.
- A latency metric in Graphite, for example `stats.timers.myservice.request.latency.p95`.
- Permission to create events in Graphite.

### Step 1: Record a deployment event

When your pipeline deploys the service, post an event to Graphite tagged `deploy` and `myservice`:

```sh
curl -X POST http://<GRAPHITE_HOST>/events/ \
  -H "Content-Type: application/json" \
  -d '{
    "what": "Deployed myservice v2.5.0",
    "tags": ["deploy", "myservice"],
    "data": "Rolled out connection pool changes"
  }'
```

Replace _`<GRAPHITE_HOST>`_ with your Graphite server address.

### Step 2: Build the latency panel

1. Create a dashboard and add a **Time series** panel.
1. Select the **Graphite** data source.
1. In the query editor, build the latency query, for example `stats.timers.myservice.request.latency.p95`.
1. Confirm the panel plots your latency data over the selected time range.

### Step 3: Add the deployment annotation query

1. In the dashboard toolbar, click the **Add new element** icon (blue plus sign), then click **Annotation query**.
1. Enter `Deployments` as the name and select a color that stands out against the latency line.
1. Click **Open query editor** and select the **Graphite** data source.
1. In the **Graphite events tags** field, enter `deploy myservice`.
1. (Optional) Click **Test annotation query** to confirm the query returns events.
1. Click **Close**, then **Save**.

### Step 4: Interpret the result

The panel now shows a vertical marker at each deployment. Hover over a marker to see the deployment title from the event's `what` value. If latency rises immediately after a marker, the deployment is a likely cause. If latency is steady across markers, deployments aren't degrading performance.

To extend the dashboard, add a second annotation query tagged `incident` in a different color, so you can tell whether latency changes follow deployments or unrelated incidents.

## Troubleshoot annotation issues

The following section addresses common issues you might encounter when using Graphite annotations.

### Annotations don't appear

**Possible causes and solutions:**

| Cause                                 | Solution                                                                                                          |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Time range doesn't include the events | Expand the dashboard time range to include the event timestamps.                                                  |
| Metric query returns only zero values | Annotations are created only for non-zero data points. Verify the metric produces non-zero values at event times. |
| No matching events                    | Confirm events exist in Graphite for the supplied tags and time range.                                            |
| Both fields are populated             | Clear one of the fields. Grafana uses the metric query mode whenever the **Graphite Query** field has a value.    |

### Annotation title is empty

In events query mode, Grafana uses the event's `what` value as the title. Ensure the `what` field is populated when you create events in Graphite.

For more troubleshooting guidance, refer to [Troubleshoot Graphite data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/graphite/troubleshooting/).

# Configure the Graphite data source

This document provides instructions for configuring the Graphite data source and explains available configuration options. For general information on managing data sources, refer to [Data source management](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/).

## Before you begin

- You must have the `Organization administrator` role to configure the Graphite data source.
  Organization administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system.

- Grafana comes with a built-in Graphite data source plugin, eliminating the need to install a plugin.

- Familiarize yourself with your Graphite security configuration and gather any necessary security certificates and client keys.

## Add the Graphite data source

To configure basic settings for the data source, complete the following steps:

1. Click **Connections** in the left-side menu.
1. Click **Add new connection**
1. Type `Graphite` in the search bar.
1. Select the **Graphite data source**.
1. Click **Add new data source** in the upper right.

Grafana takes you to the **Settings** tab, where you set up your Graphite configuration.

## Configuration options in the UI

Following is a list of configuration options for Graphite.

| Setting     | Description                                                                                                                                |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Name**    | The display name for the data source. This is how you reference it in panels and queries. For example, `graphite-1` or `graphite-metrics`. |
| **Default** | When enabled, sets this data source as the default for dashboard panels. Grafana automatically selects it when you create new panels.      |

**HTTP:**

| Setting             | Description                                                                                                                                                                                    |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **URL**             | Sets the HTTP protocol, IP, and port of your `graphite-web` or `graphite-api` installation. Because the access method is set to _Server_, the URL must be accessible from the Grafana backend. |
| **Allowed cookies** | By default, Grafana removes forwarded cookies. Specify cookie names here to allow them to be forwarded to the data source.                                                                     |
| **Timeout**         | Sets the HTTP request timeout in seconds.                                                                                                                                                      |

**Auth:**

| **Setting**                 | **Description**                                                                                                               |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| **Basic Auth**              | Toggle on to enable basic authentication to the data source.                                                                  |
| &nbsp;&nbsp;**User**        | Sets the username used for basic authentication.                                                                              |
| &nbsp;&nbsp;**Password**    | Enter the password used for basic authentication.                                                                             |
| **With Credentials**        | Toggle on to include cookies and authentication headers in cross-origin requests.                                             |
| **TLS Client Auth**         | Toggle on to enable TLS client authentication (both server and client are verified).                                          |
| &nbsp;&nbsp;**ServerName**  | The server name used to verify the hostname on the certificate returned by the server.                                        |
| &nbsp;&nbsp;**Client Cert** | Client certificate generated by a Certificate Authority (CA) or self-signed.                                                  |
| &nbsp;&nbsp;**Client Key**  | Private key used to encrypt communication between the client and server. Also generated by a CA or self-signed.               |
| **With CA Cert**            | Toggle on to authenticate with a CA certificate.                                                                              |
| &nbsp;&nbsp;**CA Cert**     | CA certificate used to validate the server certificate.                                                                       |
| **Skip TLS Verify**         | Toggle on to bypass TLS certificate validation. Not recommended unless necessary or for testing purposes.                     |
| **Forward OAuth Identity**  | Toggle on to forward the user's upstream OAuth identity to the data source. Grafana includes the access token in the request. |

**Custom HTTP Headers:**

Pass along additional information and metadata about the request or response.

| **Setting** | **Description**                                                                                            |
| ----------- | ---------------------------------------------------------------------------------------------------------- |
| **Header**  | Add a custom header. This allows custom headers to be passed based on the needs of your Graphite instance. |
| **Value**   | The value of the header.                                                                                   |

**Graphite details:**

| **Setting**               | **Description**                                                                                                                                                                                                                             |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Version**               | Select your Graphite version from the drop-down. This controls which functions are available in the Graphite query editor. Use `1.1.x` for Grafana Cloud Graphite.                                                                          |
| **Graphite backend type** | Select the Graphite backend type. Choosing `Metrictank` enables additional features like query processing metadata. (`Metrictank` is a multi-tenant time series engine compatible with Graphite.) Use `Default` for Grafana Cloud Graphite. |
| **Rollup indicator**      | Toggle on to display an info icon in panel headers when data aggregation occurs. Only available when `Metrictank` is selected.                                                                                                              |

**Label mappings:**

Label mappings are the rules you define to tell Grafana how to pull pieces of the Graphite metric path into Loki labels when switching data sources. They are currently only supported between Graphite and Loki queries.

When you change your data source from Graphite to Loki, your queries are automatically mapped based on the rules you define. To create a mapping, specify the full path of the metric and replace the nodes you want to map with label names, using parentheses. The corresponding label values are extracted from your Graphite query during the data source switch.

Grafana automatically maps all Graphite tags to labels, even if you haven't defined explicit mappings. When using matching patterns with `{}` (for example, `metric.{a,b}.value`), Grafana converts them to the Loki regular expression matching syntax. If your queries include functions, Graphite extracts the relevant metrics and tags, then matches them against your mappings.

For example, given the mapping `servers.(cluster).(server).*`:

| **Graphite Query**                                       | **Mapped to Loki Query**         |
| -------------------------------------------------------- | -------------------------------- |
| `alias(servers.west.001.cpu,1,2)`                        | `{cluster="west", server="001"}` |
| `alias(servers.*.{001,002}.*,1,2)`                       | `{server=~"(001\|002)"}`         |
| `interpolate(seriesByTag('foo=bar', 'server=002'), inf)` | `{foo="bar", server="002"}`      |

| **Setting**                     | **Description**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Private data source connect** | _Only for Grafana Cloud users._ Establishes a private, secured connection between a Grafana Cloud stack and data sources within a private network. Use the drop-down to locate the PDC URL. For setup instructions, refer to [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) and [Configure PDC](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc). Click **Manage private data source connect** to open your PDC connection page and view your configuration details. |

After configuring your Graphite data source options, click **Save & test** at the bottom to test the connection. You should see a confirmation dialog box that says:

**Data source is working**

## Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system.
For more information about provisioning, and for lists of common configuration options and JSON data options, refer to [Provisioning data sources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

### YAML example

The following example provisions a Graphite data source:

```yaml
apiVersion: 1

datasources:
  - name: Graphite
    type: graphite
    access: proxy
    url: http://localhost:8080
    jsonData:
      graphiteVersion: '1.1'
      graphiteType: default
```

The following `jsonData` options are supported:

| Option                   | Type    | Description                                                                                   |
| ------------------------ | ------- | --------------------------------------------------------------------------------------------- |
| `graphiteVersion`        | string  | The Graphite version. Supported values are `0.9`, `1.0`, and `1.1`. Defaults to `1.1`.        |
| `graphiteType`           | string  | The Graphite backend type. Supported values are `default` and `metrictank`.                   |
| `rollupIndicatorEnabled` | boolean | Displays an info icon in panel headers when data is aggregated. Applies only to `metrictank`. |

## Provision with Terraform

You can provision the Graphite data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation.

### Terraform example

The following example provisions a Graphite data source:

```hcl
terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 2.0.0"
    }
  }
}

provider "grafana" {
  url  = "<YOUR_GRAFANA_URL>"
  auth = "<YOUR_SERVICE_ACCOUNT_TOKEN>"
}

resource "grafana_data_source" "graphite" {
  type = "graphite"
  name = "Graphite"
  url  = "http://localhost:8080"

  json_data_encoded = jsonencode({
    # Graphite version: 0.9, 1.0, or 1.1
    graphiteVersion = "1.1"
    # Backend type: default or metrictank
    graphiteType = "default"
  })
}
```

### Terraform example with basic authentication

The following example provisions a Graphite data source with basic authentication:

```hcl
resource "grafana_data_source" "graphite_auth" {
  type                = "graphite"
  name                = "Graphite"
  url                 = "http://localhost:8080"
  basic_auth_enabled  = true
  basic_auth_username = "<USERNAME>"

  json_data_encoded = jsonencode({
    graphiteVersion = "1.1"
    graphiteType    = "default"
  })

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = "<PASSWORD>"
  })
}
```

Replace the following placeholders:

- _`<YOUR_GRAFANA_URL>`_: Your Grafana instance URL (for example, `https://your-org.grafana.net` for Grafana Cloud)
- _`<YOUR_SERVICE_ACCOUNT_TOKEN>`_: A service account token with data source permissions
- _`<USERNAME>`_: The username for basic authentication
- _`<PASSWORD>`_: The password for basic authentication

# Graphite query editor

Grafana includes a Graphite-specific query editor to help you build queries.
The query editor helps you quickly navigate the metric space, add functions, and change function parameters.
It supports a variety of Graphite queries, including complex nested queries, through the use of query references.

For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/).

## Query editor elements

The query editor consists of the following elements:

- **Series**: A series in Graphite is a unique time-series dataset, represented by a specific metric name and timestamped values. Click **select metric** to choose a metric from the drop-down.

- **Functions**: Graphite uses functions to manipulate data. Click the **+ sign** to view a list of functions in the drop-down. You can add multiple functions to a query.

To view the raw query, click the **Pencil icon** in the upper right. Click the **Pencil icon** again to continue adding series and functions.

## Choose metrics to query

Click **select metric** to browse the available metrics. You can navigate using your mouse or arrow keys. You can also select a wildcard.

{{< figure src="/static/img/docs/graphite/graphite-query-editor-still.png" animated-gif="/static/img/docs/graphite/graphite-query-editor.gif" >}}

## Functions

Click the **+ sign** next to **Function** to add a function from the drop-down. You can also search by typing the first few letters of the function name.

After selecting a function, Grafana adds it to your query and automatically places your cursor in the first parameter field.

To edit a parameter, click it to open an editable text box.

To remove a function, click it, then click the **X icon** that appears above it.

{{< figure src="/static/img/docs/graphite/graphite-functions-still.png" animated-gif="/static/img/docs/graphite/graphite-functions-demo.gif" >}}

Some functions like `aliasByNode` support an optional second argument. To add this argument, hover your mouse over the argument and a dialog box opens. To remove the second optional parameter, click it to delete it.

Refer to [Functions](https://graphite.readthedocs.io/en/latest/functions.html) in the Graphite documentation for more information.

{{< admonition type="warning" >}}
Some functions accept a second argument, which can itself be another function that returns a series. If you need to add a second argument that is a function, Grafana recommends using a series reference from a second query instead of embedding the function directly.

Currently, the query editor does not support parsing a second function argument when switching between the query builder and the code editor.
{{< /admonition >}}

### Sort labels

If the same labels appear on multiple graphs, they may be sorted differently and assigned different colors.

To ensure consistent sorting and coloring, use the `sortByName()` function to order labels alphabetically.

### Modify metric names in tables or charts

Use `alias` functions, such as `aliasByNode()` or `aliasSub()`, to change metric names on Grafana tables or graphs.

### Consolidate data points

Grafana consolidates all Graphite metrics so that Graphite doesn't return more data points than there are pixels in the graph.
By default, Grafana consolidates data points using the `avg` function.
To control how Graphite consolidates metrics, use the Graphite `consolidateBy()` function.

{{< admonition type="note" >}}
Grafana calculates legend summary values like `max`, `min`, and `total` on the client side, after data has been calculated.
Depending on the consolidation function used, only one or two of these values may be accurate at the same time.
{{< /admonition >}}

### Combine time series

To combine time series, click **Combine** in the **Functions** list.

### Select and explore data with tags

Graphite supports tagged series, so you can filter data by tag key-value pairs instead of only by metric path.

When exploring data, previously selected tags filter the remaining result set.
To select data, use the `seriesByTag` function, which takes tag expressions (`=`, `!=`, `=~`, `!=~`) to filter the time series.

The Grafana query builder does this for you automatically when you select a tag.

{{< admonition type="note" >}}
Regular expression searches can be slow on high-cardinality tags, so try to use other tags to reduce the scope first. To help reduce the results, start by filtering on a particular name or namespace.
{{< /admonition >}}

## Nested queries

Grafana lets you reference one query from another using its query letter, similar to how cell references work in a spreadsheet.

For example, if you add a second query and want to build on the results of query A, you can reference it using `#A`.

This approach lets you build compound or nested queries so your panels are more flexible and easier to manage.

## Use wildcards to make fewer queries

To display multiple time series on the same graph, use wildcards in your query to return all matching series at once.

For example, to monitor CPU utilization across a variety of metrics, you can use a single query like `cpu.percent.*.g` to retrieve all matching time series.
This approach is more efficient than writing separate queries for each series, such as `cpu.percent.user.g`, `cpu.percent.system.g`, and others, which would result in multiple queries to the data source.

## Example queries

The following examples show common query patterns. Each one combines a metric path with one or more Graphite functions.

### Rename series for readable legends

Wildcard queries return series whose names are full metric paths. Use `aliasByNode()` to display a specific path segment instead. The following query labels each series with the second path node, for example the host name:

```text
aliasByNode(servers.*.cpu.percent.user, 1)
```

### Aggregate across many series

Combine matching series into a single line with an aggregation function. The following query sums request counts across all hosts:

```text
sumSeries(servers.*.requests.count)
```

Use `averageSeries()`, `maxSeries()`, or `minSeries()` when you want the average, maximum, or minimum instead of the sum.

### Convert a counter to a per-second rate

For ever-increasing counters, use `perSecond()` to plot the rate of change:

```text
perSecond(servers.web01.nginx.requests.count)
```

### Summarize into time buckets

Use `summarize()` to roll data into fixed intervals, which is useful for bar charts and reducing noise. The following query totals errors in five-minute buckets:

```text
summarize(sumSeries(servers.*.errors.count), '5min', 'sum', false)
```

### Control consolidation

When a panel shows a long time range, Grafana consolidates points. Use `consolidateBy()` to choose how points combine so peaks aren't averaged away:

```text
consolidateBy(servers.web01.response.max, 'max')
```

### Show the top series

Use a filter function to reduce a large wildcard result to the most significant series. The following query keeps the five hosts with the highest peak CPU:

```text
highestMax(servers.*.cpu.percent.user, 5)
```

### Select series by tag

Use `seriesByTag()` to select data by tag expressions instead of a metric path:

```text
seriesByTag('name=requests.count', 'env=production')
```

### Build on another query

Reference a query by its letter to reuse its result. If query A returns per-host request counts, query B can sum them:

```text
sumSeries(#A)
```

## Apply annotations

[Annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/) overlay rich event information on top of graphs. Graphite supports two ways to query annotations: a regular metric query and a Graphite events query.

For details on both modes, example queries, and how to create events in Graphite, refer to [Graphite annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/graphite/annotations/).

## Integration with Loki

When you change the data source to Loki in Explore, your Graphite queries are automatically converted to Loki queries. Loki label names and values are extracted based on the mapping information defined in your Graphite data source configuration. Grafana automatically transforms queries that use tags with `seriesByTag()` without requiring additional setup.

# Graphite template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
Grafana lists these variables in drop-down selection boxes at the top of the dashboard to help you change the data displayed in your dashboard.
Grafana refers to such variables as template variables.

For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/) documentation.

## Use query variables

Graphite populates **Query** variables. You can still use other variable types, such as **Custom**, **Interval**, and **Text box**, on a dashboard that queries Graphite. Those types don't send a query to Graphite.

To create a Graphite query variable, choose the query type from the **Select query type** drop-down and enter your query in the **Query** field. Grafana supports three query types for Graphite variables:

| Query type            | Description                                                                                                        | Example                                        |
| --------------------- | ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------- |
| **Default Query**     | Runs a metric-find query and returns metric nodes, or tag keys and values when you use `tags()` or `tag_values()`. | `tag_values(apps.*.requests.count, app)`       |
| **Value Query**       | Runs the query through the render endpoint and returns the numeric values of the resulting series.                 | `groupByNode(apps.*.requests.count, 1, 'sum')` |
| **Metric Name Query** | Runs the query through the render endpoint and returns the series names of the result.                             | `highestAverage(apps.*.requests.count, 5)`     |

### Choose a variable syntax

The Graphite data source supports two variable syntaxes for use in the **Query** field.

![Variable syntax example](/static/img/docs/v2/templated_variable_parameter.png)

Grafana allows two ways to reference variables in a query:

| **Syntax**   | **Example**                              |
| ------------ | ---------------------------------------- |
| `$varname`   | `apps.frontend.$server.requests.count`   |
| `${varname}` | `apps.frontend.${server}.requests.count` |

- **Shorthand syntax (`$varname`)** is convenient for standalone path segments but doesn't work when the variable is adjacent to other characters (for example, `cpu$coreLoad`).
- **Full syntax (`${varname}`)** is more flexible and works in any part of the string, including embedded within words.

Choose the format that best fits the structure of your Graphite metric path.

## Use tag variables

Grafana supports tag-based variables for Graphite, allowing you to dynamically populate drop-downs based on tag keys and values in your metric series. To do this, use the Graphite functions `tags()` and `tag_values()` in your variable queries.

| Query                                  | Description                                                                                |
| -------------------------------------- | ------------------------------------------------------------------------------------------ |
| `tags()`                               | Returns a list of all tag keys in the Graphite database.                                   |
| `tags(server=~backend*)`               | Returns tag keys only from series that match the provided filter expression.               |
| `tag_values(server)`                   | Returns all values for the specified tag key.                                              |
| `tag_values(server, server=~backend*)` | Returns tag values for a given key, filtered to only those that appear in matching series. |

You can use multiple filter expressions, and those expressions can include other Grafana variables. For example:

```text
tag_values(server, server=~backend*, app=~${apps:regex})
```

This query returns all server tag values from series where the `server` tag matches `backend*` and the `app` tag matches the regular expression-filtered values from another variable, `${apps}`.

For details, refer to the [Graphite docs on the autocomplete API for tags](http://graphite.readthedocs.io/en/latest/tags.html#auto-complete-support).

**Using regular expression formatting and the equal tilde operator `=~`:**

```text
server=~${servers:regex}
```

This query tells Grafana to format the selected values in the `servers` variable as a regular expression (for example, `(server1|server2)` if two servers are selected).

For more information, refer to [Advanced variable format options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/#advanced-variable-format-options).

### Filter with multiple expressions

When using multi-value variables in tag queries, append `${var:regex}` to the variable name to apply regular expression formatting.

```text
tag_values(server, app=~${apps:regex})
```

This query returns only series where the app tag matches the selected values in `${apps}`, formatted as a regular expression. `=~` is the regular expression operator.

Non-tag queries use the default `glob` formatting for multi-value variables.

## Use other query variables

When writing queries, use the **Default Query** type to retrieve dynamic metric values.

For example, the query `prod.servers.*` populates the variable with all values that exist at the wildcard position (`*`).

Note that the results include only the values found at the last level of the query path.

To return full metric paths that match your query, use the `expand()` function:

```text
expand(*.servers.*)
```

### Compare expanded and non-expanded metric search results

When querying Graphite metrics in Grafana, you can choose between using an **expanded** or **non-expanded** query:

- **Expanded queries** (using the `expand()` function) return the **full metric paths** that match your query.
- **Non-expanded queries** return only the **last segment** of each matching metric path, which limits your ability to extract or filter based on deeper parts of the metric name.

Expanded queries are especially useful when working with regular expressions to match or extract specific parts of the metric path.

Suppose your Graphite database contains the following metrics:

- `prod.servers.001.cpu`
- `prod.servers.002.cpu`
- `test.servers.001.cpu`

The following table illustrates the difference between expanded and non-expanded queries:

| **Non-expanded query** | **Results**    | **Expanded query**        | **Expanded results**                                                   |
| ---------------------- | -------------- | ------------------------- | ---------------------------------------------------------------------- |
| `*`                    | `prod`, `test` | `expand(*)`               | `prod`, `test`                                                         |
| `*.servers`            | `servers`      | `expand(*.servers)`       | `prod.servers`, `test.servers`                                         |
| `test.servers`         | `servers`      | `expand(test.servers)`    | `test.servers`                                                         |
| `*.servers.*`          | `001`, `002`   | `expand(*.servers.*)`     | `prod.servers.001`, `prod.servers.002`, `test.servers.001`             |
| `test.servers.*`       | `001`          | `expand(test.servers.*)`  | `test.servers.001`                                                     |
| `*.servers.*.cpu`      | `cpu`          | `expand(*.servers.*.cpu)` | `prod.servers.001.cpu`, `prod.servers.002.cpu`, `test.servers.001.cpu` |

{{< admonition type="note" >}}
A non-expanded query works like an expanded query but returns only the final segment of each matched metric.
{{< /admonition >}}

Grafana also supports **nested variables**, which allow you to reference other variables in a query.

For example:

```text
apps.$app.servers.*
```

This query uses the selected value of the `$app` variable to dynamically filter the metric path. The variable `$app` contains one or more application names and `servers.*` matches all servers for the given application.

### Filter query variable results with `__searchFilter`

Grafana provides the variable `__searchFilter`, which you can use to dynamically filter query results based on what the user types into the variable drop-down.
When the drop-down is empty or blank, `__searchFilter` defaults to `*`, which means it returns all possible values. If you type a string, Grafana replaces `__searchFilter` with that input.

To use `__searchFilter` as part of the query field to enable searching for `server` while the user types in the drop-down select box:

Query:

```text
apps.$app.servers.$__searchFilter
```

TagValues:

```text
tag_values(server, server=~${__searchFilter:regex})
```

# Troubleshoot Graphite data source issues

This document provides solutions for common issues you might encounter when using the Graphite data source.

## Connection issues

Use the following troubleshooting steps to resolve connection problems between Grafana and your Graphite server.

**Data source test fails with "Unable to connect":**

If the data source test fails, verify the following:

- The URL in your data source configuration is correct and accessible from the Grafana server.
- The Graphite server is running and accepting connections.
- Any firewall rules or network policies allow traffic between Grafana and the Graphite server.
- If using TLS, ensure your certificates are valid and properly configured.

To test connectivity, run the following command from the Grafana server:

```sh
curl -v <GRAPHITE_URL>/render
```

Replace _`<GRAPHITE_URL>`_ with your Graphite server URL. A successful connection returns a response from the Graphite server.

**Authentication errors:**

If you receive 401 or 403 errors:

- Verify your Basic Auth username and password are correct.
- Ensure the **With Credentials** toggle is enabled if your Graphite server requires cookies for authentication.
- Check that your TLS client certificates are valid and match what the server expects.

For detailed authentication configuration, refer to [Configure the Graphite data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/graphite/configure/).

**Deprecation notice about browser access mode:**

If the data source configuration page shows a deprecation notice about browser access mode:

- The data source is configured to use browser (direct) access, which is deprecated and slated for removal.
- Edit the data source so it uses _Server_ (proxy) access instead. Server access routes requests through the Grafana backend, which avoids browser cross-origin restrictions.
- Click **Save & test** to re-test the connection after switching.

## Query issues

Use the following troubleshooting steps to resolve problems with Graphite queries.

**No data returned:**

If your query returns no data:

- Verify the metric path exists in your Graphite server by testing directly in the Graphite web interface.
- Check that the time range in Grafana matches when data was collected.
- Ensure wildcards in your query match existing metrics.
- Confirm your query syntax is correct for your Graphite version.

**HTTP 500 errors with HTML content:**

Graphite-web versions before 1.6 return HTTP 500 errors with full HTML stack traces when a query fails. If you see error messages containing HTML tags:

- Check the Graphite server logs for the full error details.
- Verify your query syntax is valid.
- Ensure the requested time range doesn't exceed your Graphite server's capabilities.
- Check that all functions used in your query are supported by your Graphite version.

**Parser errors in the query editor:**

If the query editor displays parser errors:

- Check for unbalanced parentheses in function calls.
- Verify that function arguments are in the correct format.
- Ensure metric paths don't contain unsupported characters.

**Queries don't map correctly when switching from Graphite to Loki:**

Label mappings apply only when you switch between Graphite and Loki queries. If mapped queries produce unexpected labels or no results:

- Verify your label mappings are defined in the Graphite data source configuration.
- Remember that Grafana automatically maps all Graphite tags to labels, so you only need explicit mappings for non-tag metric nodes.
- Check that multi-value patterns such as `{001,002}` map to the expected regular expression, for example `{server=~"(001|002)"}`.

For details, refer to the label mappings section in [Configure the Graphite data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/graphite/configure/).

For query syntax help, refer to [Graphite query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/graphite/query-editor/).

## Version and feature issues

Use the following troubleshooting steps to resolve problems related to Graphite versions and features.

**Functions missing from the query editor:**

If expected functions don't appear in the query editor:

- Verify the correct Graphite version is selected in the data source configuration.
- The available functions depend on the configured version. For example, tag-based functions require Graphite 1.1 or later.
- If using a custom Graphite installation with additional functions, ensure the version setting matches your server.

**Tag-based queries not working:**

If `seriesByTag()` or other tag functions fail:

- Confirm your Graphite server is version 1.1 or later.
- Verify the Graphite version setting in your data source configuration matches your actual server version.
- Check that tags are properly configured in your Graphite server.

## Template variable issues

Use the following troubleshooting steps when dashboard variables don't behave as expected.

**Variable not interpolated in a query:**

- When a variable is adjacent to other characters, use the full syntax `${varname}` instead of `$varname`. For example, use `cpu.${core}.load` rather than `cpu.$coreLoad`.
- Confirm the variable name matches exactly, including case.

**Multi-value variable returns no data in tag queries:**

- Tag queries expect regular expression formatting for multi-value variables. Append `:regex` to the variable, for example `tag_values(server, app=~${apps:regex})`.
- Non-tag queries use the default glob formatting and don't require `:regex`.

**Dependent variable is empty:**

- Define the parent variable above the child variable in the dashboard variable list so it resolves first.
- Verify the parent variable has a selected value. An empty parent produces an empty child query.

For details, refer to [Graphite template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/graphite/template-variables/).

## Performance issues

Use the following troubleshooting steps to address slow queries or timeouts.

**Queries timing out:**

If queries consistently time out:

- Increase the **Timeout** setting in the data source configuration.
- Reduce the time range of your query.
- Use more specific metric paths instead of broad wildcards.
- Consider using `summarize()` or `consolidateBy()` functions to reduce the amount of data returned.
- Check your Graphite server's performance and resource utilization.

**Slow autocomplete in the query editor:**

If metric path autocomplete is slow:

- This often indicates a large number of metrics in your Graphite server.
- Use more specific path prefixes to narrow the search scope.
- Check your Graphite server's index performance.

## MetricTank backend issues

If you're using MetricTank as your Graphite backend, use the following troubleshooting steps.

**Rollup indicator not appearing:**

If the **Rollup indicator** doesn't display when expected:

- Verify **Metrictank** is selected as the Graphite backend type in the data source configuration.
- Ensure the **Rollup indicator** toggle is enabled.
- The indicator only appears when data aggregation actually occurs.

**Unexpected data aggregation:**

If you see unexpected aggregation in your data:

- Check the data aggregation (`rollup`) configuration in your MetricTank instance.
- Adjust the time range or use `consolidateBy()` to control aggregation behavior.
- Review the query processing metadata in the panel inspector for details on how data was processed.

## Enable debug logging

To capture detailed request and response information when troubleshooting Graphite issues:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Restart Grafana to apply the change.
1. Reproduce the issue, then review the logs in `/var/log/grafana/grafana.log` (or your configured log location) for Graphite-related entries that include the request URL and response details.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you continue to experience issues:

- Check the [Grafana community forums](https://community.grafana.com/) for similar issues and solutions.
- Review the [Graphite documentation](https://graphite.readthedocs.io/) for additional configuration options.
- Contact [Grafana Support](https://grafana.com/support/) if you're an Enterprise, Cloud Pro, or Cloud Advanced customer.

When reporting issues, include the following information:

- Grafana version
- Graphite version (for example, 1.1.x) and backend type (Default or MetricTank)
- Authentication method (Basic Auth, TLS, or none)
- Error messages (redact sensitive information)
- Steps to reproduce the issue
- Relevant configuration such as data source settings, timeout values, and Graphite version setting (redact passwords and other credentials)
- Sample query (if applicable, with sensitive data redacted)

# InfluxDB alerting

You can use Grafana Alerting with InfluxDB to create alerts based on your time-series data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met.

For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/).

## Before you begin

Before creating alerts with InfluxDB, ensure you have:

- An InfluxDB data source configured in Grafana
- Appropriate permissions to create alert rules
- Understanding of the metrics you want to monitor

## Supported query types

InfluxDB alerting supports all three query languages: InfluxQL, SQL, and Flux. Queries must return time-series data for Grafana to evaluate values over time and trigger alerts when thresholds are crossed.

### Query language compatibility

| Query language | Alerting support | Notes                                                           |
| -------------- | ---------------- | --------------------------------------------------------------- |
| InfluxQL       | Yes              | Use aggregation functions with GROUP BY time.                   |
| Flux           | Yes              | Use `aggregateWindow()` for time-based aggregation.             |
| SQL            | Yes              | Use `$__timeFilter` and `$__dateBin` macros for time filtering. |

## Create an alert rule

To create an alert rule using InfluxDB:

1. Navigate to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for the alert rule.
1. Select your **InfluxDB** data source.
1. Build your query using the query editor.
1. Configure the alert condition (for example, when the average is above a threshold).
1. Set the evaluation interval and pending period.
1. Configure notifications and labels.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example alert queries

The following examples show common alerting scenarios with InfluxDB.

### InfluxQL: Alert on high CPU usage

Monitor CPU usage and alert when it exceeds a threshold:

1. **Query:** Select the `cpu` measurement with `mean("usage_system")` as the aggregation.
1. **GROUP BY:** `time($__interval)`
1. **Condition:** When mean is above 90.

In raw query mode:

```sql
SELECT mean("usage_system") FROM "cpu" WHERE $timeFilter GROUP BY time($__interval) fill(null)
```

### Flux: Alert on memory pressure

Monitor memory usage with a Flux query:

```flux
from(bucket: "telegraf")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "mem")
  |> filter(fn: (r) => r["_field"] == "used_percent")
  |> aggregateWindow(every: v.windowPeriod, fn: mean)
  |> yield(name: "mean")
```

Set the condition to alert when the mean value exceeds 85.

### SQL: Alert on disk usage

Monitor disk usage with an SQL query:

```sql
SELECT $__dateBin(time), mean(used_percent)
FROM disk
WHERE $__timeFilter(time)
GROUP BY $__dateBin(time)
```

Set the condition to alert when the mean value exceeds 80.

## Limitations

When using InfluxDB with Grafana Alerting, be aware of the following limitations:

### Template variables not supported

Alert queries can't contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$hostname` or `$region` aren't resolved.

If your dashboard query uses template variables, create a separate query for alerting with hard-coded values.

### Query complexity

Complex queries with many nested functions or large result sets may timeout or fail to evaluate. Simplify queries for alerting by:

- Reducing the time range
- Using appropriate aggregation intervals
- Adding filters to limit the data scanned

## Handle transient errors and timeouts

Brief InfluxDB latency spikes or network interruptions can cause alert rule evaluations to fail. By default, a failed evaluation moves the rule into an error state, which can trigger a large number of spurious notifications at once even though your data is healthy.

To make alert rules resilient to transient errors:

1. In the alert rule settings, expand **Configure no data and error handling** and set **Alert state if execution error or timeout** to **Keep Last State**. The rule keeps its previous state through brief evaluation failures instead of firing an error alert.
1. Set a **Pending period** so the alert condition must be met for a sustained period across multiple evaluations before the alert fires.
1. If evaluations fail because queries run out of time, increase the data source timeout. Refer to [Advanced HTTP settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/configure/#advanced-http-settings) for details.
1. Simplify or narrow alert queries so they complete well within the timeout. Refer to [Query complexity](#query-complexity) for guidance.

For more information about error state handling, refer to [No data and error handling](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/#configure-no-data-and-error-handling).

## Best practices

Follow these best practices when creating InfluxDB alerts:

- **Use specific filters:** Add WHERE clauses or Flux filters to focus on relevant data and improve query performance.
- **Choose appropriate intervals:** Match the GROUP BY time interval to your evaluation frequency.
- **Test queries first:** Verify your query returns expected results in Explore before creating an alert.
- **Set realistic thresholds:** Base alert thresholds on historical data patterns.
- **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor.

If you encounter errors when creating or evaluating alert rules, refer to [Troubleshoot InfluxDB data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/troubleshooting/#alerting-errors).

# InfluxDB annotations

Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics. You can use InfluxDB as a data source for annotations to display events such as deployments, alerts, or other significant occurrences on your visualizations.

For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/).

## Before you begin

Before creating InfluxDB annotations, ensure you have:

- An InfluxDB data source configured in Grafana
- Data in InfluxDB containing event information with timestamp fields
- Read access to the InfluxDB database or bucket containing your events

## Create an annotation query

To add an InfluxDB annotation to your dashboard:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Annotation query**.
1. Enter a name for the annotation query.
1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox.
1. Select a color for the annotation event markers.
1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed.
1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed.
1. Click **Open query editor** to open the **Annotation Query** dialog box.
1. Select your **InfluxDB** data source from the **Data source** drop-down list.
1. Configure the annotation query and field mappings.
1. (Optional) Click **Test annotation query** to ensure that the query is working properly.
1. Click **Close** when you've completed the query setup.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## InfluxQL annotations

For InfluxQL-configured data sources, write an InfluxQL query in the **InfluxQL Query** field. Your query **must** include `WHERE $timeFilter` to scope the results to the dashboard's time range.

### Field mappings

Field mappings tell Grafana which InfluxDB columns contain the annotation data. If your query returns only one column, you don't need to enter anything in the field mapping fields.

| Field       | Description                                                                                                                                           |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Text**    | The column containing the annotation description displayed when you hover over the annotation.                                                        |
| **Tags**    | The column containing tags for the annotation. The value can be a comma-separated string. Tags help categorize and filter annotations.                |
| **TimeEnd** | The column containing an end time for range annotations. Range annotations display as a shaded region on the graph instead of a single vertical line. |

### InfluxQL annotation query example

The following query retrieves deployment events and displays them as annotations:

```sql
SELECT title, description
FROM events
WHERE $timeFilter
ORDER BY time ASC
```

### Range annotation example

To display events with duration as shaded regions:

```sql
SELECT description, tags, end_time
FROM maintenance_windows
WHERE $timeFilter
ORDER BY time ASC
```

Configure the field mappings as follows:

- **Text:** `description`
- **Tags:** `tags`
- **TimeEnd:** `end_time`

## Flux annotations

For Flux-configured data sources, annotations use the standard Flux query editor. Write a Flux query that returns data frames with time, text, and optional tag fields.

### Flux annotation query example

```flux
from(bucket: "events")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "deployments")
  |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
```

Grafana processes the resulting data frames as annotation events. Ensure the query returns a `_time` column and at least one text column.

## SQL annotations

SQL-configured data sources (InfluxDB 3.x) use the standard Grafana annotation query editor. Write an SQL query that returns a time column and at least one text column. Use the `$__timeFilter(time)` macro to scope results to the dashboard time range.

### SQL annotation query example

```sql
SELECT time, title, description
FROM events
WHERE $__timeFilter(time)
ORDER BY time ASC
```

### SQL range annotation example

To display events with duration as shaded regions, return both a start time and an end time:

```sql
SELECT time, end_time, description, tags
FROM maintenance_windows
WHERE $__timeFilter(time)
ORDER BY time ASC
```

Configure the field mappings:

- **Text:** `description`
- **Tags:** `tags`
- **TimeEnd:** `end_time`

## Use template variables in annotations

You can use template variables in your annotation queries to filter annotations based on dashboard variable selections. For example:

```sql
SELECT title, description
FROM events
WHERE $timeFilter AND environment = '$environment'
ORDER BY time ASC
```

If your annotations aren't appearing or you encounter errors, refer to [Troubleshoot InfluxDB data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/troubleshooting/#annotation-errors).

# InfluxDB query editor

The InfluxDB query editor helps you build and run queries against your InfluxDB data source. You can access it from the [Explore page](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/) or from any dashboard panel by clicking the ellipsis in the upper right of the panel and selecting **Edit**.

The editor supports three query languages: SQL, InfluxQL, and Flux. Each has its own editing experience and macro system. SQL and InfluxQL provide both a visual builder mode and a code editing mode, while Flux provides a code editor only. You can also use the query editor to retrieve [log data](#query-logs).

For general information about Grafana query editors, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/).

To compare InfluxQL and SQL queries for the same data, explore this demo dashboard on Grafana Play:

{{< docs/play title="Influx: Bar Charts and Time Series using Mixed queries" url="https://play.grafana.org/d/dea43745-7fac-4bf4-85d0-d06b168de92b/" >}}

## Before you begin

- Ensure you have [configured the InfluxDB data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/configure/).
- Verify your credentials have appropriate permissions to query the target database or bucket.

## Key concepts

If you're new to InfluxDB, these terms are used throughout the query editor:

| Term                 | Description                                                                                      |
| -------------------- | ------------------------------------------------------------------------------------------------ |
| **Measurement**      | A logical grouping of fields, tags, and timestamps, similar to a table in a relational database. |
| **Field**            | A column in a measurement that stores the actual data values (numbers, strings, Boolean values). |
| **Tag**              | A column used for metadata and indexing. Tags are indexed and optimized for filtering.           |
| **Retention policy** | An InfluxDB 1.x setting that controls how long data is stored before automatic deletion.         |
| **Bucket**           | An InfluxDB 2.x and 3.x storage location that combines a database and retention policy.          |

## Choose a query editor

The InfluxDB data source has three query editors, each corresponding to the query language selected in the [data source configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/configure/):

- [InfluxQL](#influxql-query-editor)
- [SQL](#sql-query-editor)
- [Flux](#flux-query-editor)

Editor options vary based on query language.

## InfluxQL query editor

The InfluxQL query editor helps you select metrics and tags to create InfluxQL queries. There are two modes: `visual editor mode` and `raw query mode`. To switch between the two modes click the **pencil icon** in the upper right.

Visual query editor mode contains the following components:

- **FROM** - Select a measurement to query.
- **WHERE** - Select filters by clicking the **+ sign**. You can enter regular expressions or use the text input for regular expression tag values.
- **SELECT** - Select fields and functions from the drop-down. You can add multiple fields and functions by clicking the **+ sign**.
- **GROUP BY** - Select a tag from the drop-down menu.
- **TIMEZONE** - _Optional_ Group data by a specific timezone.
- **ORDER BY TIME** - Sort data by time in either ascending or descending order.
- **LIMIT** - _Optional_ Limits the number of rows returned by the query.
- **SLIMIT** - _Optional_ Limits the number of series returned by the query. Refer to [SLIMIT clause](https://docs.influxdata.com/influxdb/cloud/query-data/influxql/explore-data/limit-and-slimit/#slimit-clause) for more information.
- **FORMAT AS** - Select a format option from the drop-down menu: time series, table, or logs.
- **ALIAS** - Add an alias. Refer to [Alias patterns](#alias-patterns) for more information.

### Raw query editor mode

You can write raw InfluxQL queries by switching to raw query mode. Click the pencil in the upper right of the query editor to switch modes. When you switch to visual editor mode, any changes made in raw query mode are lost.

If you use raw query mode, your query must include `WHERE $timeFilter`. You should also provide a group by time and an aggregation function. Otherwise, InfluxDB may return hundreds of thousands of data points, potentially causing your browser to hang.

### Match by regular expressions

You can enter regular expressions for metric names or tag filter values.
Wrap the regular expression pattern in forward slashes (`/`), as shown in this example: `/measurement/`.

Grafana automatically adjusts the filter tag condition to use the InfluxDB regular expression match condition operator (`=~`).

### Field and aggregation functions

In the `SELECT` row, you can specify which fields and functions to use.

If you **group by time** you must use an aggregation function. Certain functions such as `derivative` also require an aggregation function.

The query editor input generates an InfluxDB `SELECT` clause. For example:

```sql
SELECT derivative(mean("value"), 10s) / 10 AS "REQ/s"
FROM....
```

You can also use a \* in a SELECT statement to select all fields.

```sql
SELECT * FROM <measurement_name>
```

### GROUP BY results

To group results by a tag, specify the tag in the **GROUP BY** row:

1. Click the **+ sign** in the GROUP BY row.
1. Select a tag from the drop-down.

You can GROUP BY multiple options.

To remove a GROUP BY option click the **X icon** next to the option.

### Alias patterns

| Alias pattern     | Replaced with                                                                                                                                                                                      |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `$m`              | Measurement name.                                                                                                                                                                                  |
| `$measurement`    | Measurement name.                                                                                                                                                                                  |
| `$1` - `$9`       | Part of measurement name (if you separate your measurement name with dots).                                                                                                                        |
| `$col`            | Column name.                                                                                                                                                                                       |
| `$tag_exampletag` | The value of the `exampletag` tag. The syntax is `$tag_yourTagName` and must start with `$tag_`. To use your tag as an alias in the ALIAS BY field, you must use the tag to group by in the query. |

You can also use the `[[tag_hostname]]` pattern replacement syntax.

For example, entering the value `Host: [[tag_hostname]]` in the ALIAS BY field replaces it with the `hostname` tag value for each legend value.

An example legend value is `Host: server1`.

## SQL query editor

Grafana supports the [SQL query language](https://docs.influxdata.com/influxdb/cloud-serverless/query-data/sql/) for InfluxDB 3.x and newer cloud products (Cloud Serverless, Cloud Dedicated, Clustered). SQL is InfluxData's recommended query language for new deployments.

The SQL query editor provides two modes:

- **Builder mode** - A visual query builder that helps you construct SQL queries by selecting tables, columns, filters, and aggregations from drop-down menus. This mode is useful for building queries without writing SQL directly.
- **Code mode** - A text editor for writing raw SQL queries with autocomplete and syntax highlighting. Switch between modes using the toggle at the top of the query editor.

SQL queries connect to InfluxDB using the FlightSQL (gRPC) protocol, which provides high-performance data transfer for large result sets.

### Format output

The SQL query editor supports two output formats:

- **Time series** (default) - Returns data as time-series frames. The query must include a timestamp column.
- **Table** - Returns data as a table frame, useful for non-time-series queries or when you want to display raw results.

Select the format from the **Format** drop-down in the query editor.

### Macros

You can use macros in your query to automatically substitute them with values from the Grafana context.

| Macro example                 | Replaced with                                                                                                                                                                                       |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `$__timeFrom`                 | The start of the currently active time selection, such as `cast('2020-06-11T13:31:00Z' as timestamp)`.                                                                                              |
| `$__timeTo`                   | The end of the currently active time selection, such as `cast('2020-06-11T14:31:00Z' as timestamp)`.                                                                                                |
| `$__timeFilter(<column>)`     | A time range filter applied to the specified column. Expands to `<column> >= $__timeFrom AND <column> <= $__timeTo`.                                                                                |
| `$__interval`                 | An interval string that corresponds to the Grafana calculated interval based on the time range of the active time selection, such as `interval '5 second'`.                                         |
| `$__dateBin(<column>)`        | Applies the [`date_bin`](https://docs.influxdata.com/influxdb/cloud-serverless/reference/sql/functions/time-and-date/#date_bin) function using `$__interval`. Column must be a timestamp.           |
| `$__dateBinAlias(<column>)`   | Applies the [`date_bin`](https://docs.influxdata.com/influxdb/cloud-serverless/reference/sql/functions/time-and-date/#date_bin) function with a `_binned` suffix alias. Column must be a timestamp. |
| `$__timeGroup(<column>)`      | Groups results by a time interval using `date_bin()` with `$__interval`. Column must be a timestamp.                                                                                                |
| `$__timeGroupAlias(<column>)` | Groups results by a time interval using `date_bin()` with `$__interval` and adds a `_binned` suffix alias. Column must be a timestamp.                                                              |

### SQL query examples

**Basic time-series query:**

```sql
SELECT time, usage_system, usage_user
FROM cpu
WHERE $__timeFilter(time)
ORDER BY time
```

**Aggregated time-series query with date_bin:**

```sql
SELECT
  $__dateBin(time) AS time,
  mean(usage_system) AS avg_system,
  mean(usage_user) AS avg_user
FROM cpu
WHERE $__timeFilter(time)
GROUP BY $__dateBin(time)
ORDER BY time
```

**Using JOINs across tables:**

SQL supports JOINs, which aren't available in InfluxQL or Flux:

```sql
SELECT
  c.time,
  c.usage_system,
  m.used_percent AS mem_used
FROM cpu c
INNER JOIN mem m ON c.time = m.time AND c.host = m.host
WHERE $__timeFilter(c.time)
ORDER BY c.time
```

**Filtering with WHERE clauses:**

```sql
SELECT $__dateBin(time) AS time, mean(usage_system) AS avg_system
FROM cpu
WHERE $__timeFilter(time)
  AND host = 'server01'
  AND cpu = 'cpu-total'
GROUP BY $__dateBin(time)
ORDER BY time
```

For the full list of supported SQL statements, operators, and functions, refer to InfluxData's [SQL reference documentation](https://docs.influxdata.com/influxdb/cloud-serverless/reference/sql/).

## Flux query editor

Grafana supports Flux when running InfluxDB v1.8 and higher.
If your data source is [configured for Flux](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/configure/), you can use
the [Flux](https://docs.influxdata.com/flux/v0/) in the query editor, which serves as
a text editor for raw Flux queries with macro support.

For more information and connection details, refer
to [InfluxDB 1.8 API compatibility](https://github.com/influxdata/influxdb-client-go/#influxdb-18-api-compatibility).

### Use macros

You can enter macros in the query to replace them with values from the Grafana context.

<!-- vale Grafana.Spelling = NO -->

Macros support copying and pasting from [InfluxData Chronograf](https://www.influxdata.com/time-series-platform/chronograf/).

<!-- vale Grafana.Spelling = YES -->

| Macro example      | Replaced with                                                                                                                                                   |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `v.timeRangeStart` | The start of the currently active time selection, such as `2020-06-11T13:31:00Z`.                                                                               |
| `v.timeRangeStop`  | The end of the currently active time selection, such as `2020-06-11T14:31:00Z`.                                                                                 |
| `v.windowPeriod`   | An interval string compatible with Flux that corresponds to the Grafana calculated interval based on the time range of the active time selection, such as `5s`. |
| `v.defaultBucket`  | The **Default Bucket** value from the data source configuration.                                                                                                |
| `v.organization`   | The data source configuration's **Organization** setting.                                                                                                       |

For example, consider the following Flux query:

```flux
from(bucket: v.defaultBucket)
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "cpu" or r["_measurement"] == "swap")
  |> filter(fn: (r) => r["_field"] == "usage_system" or r["_field"] == "free")
  |> aggregateWindow(every: v.windowPeriod, fn: mean)
  |> yield(name: "mean")
```

Grafana interpolates this Flux query into the following and sends it to InfluxDB, with the interval and time period values changing according to the active time selection:

```flux
from(bucket: "grafana")
  |> range(start: 2020-06-11T13:59:07Z, stop: 2020-06-11T14:59:07Z)
  |> filter(fn: (r) => r["_measurement"] == "cpu" or r["_measurement"] == "swap")
  |> filter(fn: (r) => r["_field"] == "usage_system" or r["_field"] == "free")
  |> aggregateWindow(every: 2s, fn: mean)
  |> yield(name: "mean")
```

To view the interpolated version of a query with the Query inspector, refer to [Panel Inspector](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/panel-inspector/).

## Query logs

You can query and display log data from InfluxDB in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/) and in the dashboard [Logs panel](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/logs/).

**InfluxQL:** Select an InfluxDB data source in the query editor. Under the **Select measurement field** next to the **FROM** section, choose a measurement containing your log data, then choose the appropriate fields that display the log message. Add any additional filters by clicking the **+ sign** next to the **WHERE** field. Set **FORMAT AS** to **Logs**.

**SQL:** Write a SQL query that returns a timestamp column and a text column containing the log message. Set the **Format** to **Table**. For example:

```sql
SELECT time, message, level, host
FROM syslog
WHERE $__timeFilter(time)
ORDER BY time DESC
```

**Flux:** Use a Flux query with the `_value` field containing the log message.

After InfluxDB returns the results, the log panel displays log rows along with a bar chart. The x-axis represents time, while the y-axis shows the frequency or count.

## Next steps

- [Use template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/template-variables/) for dynamic dashboards.
- [Set up annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/annotations/) to overlay events on your graphs.
- [Create alert rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/alerting/) based on your InfluxDB queries.

# InfluxDB template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana displays these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables.

For general information about variables, refer to [Variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/).

## Before you begin

- [Configure the InfluxDB data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/configure/).
- Understand [Grafana template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/).

## Supported variable types

| Variable type | Supported           |
| ------------- | ------------------- |
| Query         | Yes                 |
| Custom        | Yes                 |
| Data source   | Yes                 |
| Ad-hoc filter | Yes (InfluxQL only) |

To switch a dashboard between InfluxDB instances, use a data source variable. Don't use a query variable that returns data source names as a workaround. Query-based workarounds can break after Grafana upgrades because they depend on internal identifiers. Refer to [Add a data source variable](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#add-a-data-source-variable) for details.

## Where you can use variables

Variable support differs by query editor and mode:

| Editor                 | Variable support                                                                                  |
| ---------------------- | ------------------------------------------------------------------------------------------------- |
| InfluxQL visual editor | Drop-downs include your template variables alongside values from your database.                   |
| InfluxQL raw mode      | Full support.                                                                                     |
| SQL builder mode       | Drop-downs list only tables and columns from your database. Switch to code mode to use variables. |
| SQL code mode          | Full support. Press Ctrl+Space to show variable suggestions.                                      |
| Flux code editor       | Full support.                                                                                     |

## Create a query variable

By adding a query template variable, you can write an InfluxDB metadata exploration query. These queries return results such as measurement names, key names, and key values.

For more information, refer to [Add a query variable](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#add-a-query-variable).

### InfluxQL query variable examples

To create a variable containing all values for the `hostname` tag, use the following query in the **Query** field:

```sql
SHOW TAG VALUES WITH KEY = "hostname"
```

You can fetch key names for a given measurement:

```sql
SHOW TAG KEYS [FROM <measurement_name>]
```

You can list available measurements:

```sql
SHOW MEASUREMENTS
```

### Flux query variable examples

For Flux-configured data sources, write a Flux query that returns a single column of values:

```flux
import "influxdata/influxdb/schema"
schema.tagValues(bucket: v.defaultBucket, tag: "hostname")
```

### SQL query variable examples

For SQL-configured data sources (InfluxDB 3.x), write an SQL query that returns a single column of values:

```sql
SELECT DISTINCT hostname FROM cpu
```

You can list available tables:

```sql
SHOW TABLES
```

You can list columns in a specific table:

```sql
SHOW COLUMNS FROM cpu
```

## Scope variables to the dashboard time range

By default, metadata queries such as `SHOW TAG VALUES` return values from the entire retention period, not just the dashboard time range. If a host or sensor stopped reporting long ago, its values still appear in the variable drop-down as long as they exist within retention.

To limit variable results to the dashboard time range:

1. Include a time condition in the variable query. Variable queries interpolate the same time macros as regular queries: `$timeFilter` for InfluxQL, `$__timeFilter(<column>)` for SQL, and `v.timeRangeStart` and `v.timeRangeStop` for Flux.
1. Set the variable's **Refresh** option to **On time range change** so the values update when the dashboard time range changes.

**InfluxQL example:**

```sql
SHOW TAG VALUES WITH KEY = "hostname" WHERE $timeFilter
```

**SQL example:**

```sql
SELECT DISTINCT hostname FROM cpu WHERE $__timeFilter(time)
```

## Chain or nest variables

You can create nested variables, sometimes called [chained variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#chained-variables).

For example, if you have a variable named `region`, you can configure the `hosts` variable to display only hosts from the selected region.

**InfluxQL:**

```sql
SHOW TAG VALUES WITH KEY = "hostname"  WHERE region = '$region'
```

**SQL:**

```sql
SELECT DISTINCT hostname FROM cpu WHERE region = '$region'
```

If you have a variable containing key names, you can use it in a **GROUP BY** clause. This allows you to adjust the grouping by selecting from the variable list at the top of the dashboard.

## Use filters

InfluxDB supports the use of filters for InfluxQL. They allow you to define multiple key/value filters, which Grafana automatically applies to all your InfluxDB queries. Filters also support expressions.

To add filters:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Filter and Group by**.
1. Enter a **Name** for the filter.
1. Select an option in the **Display** drop-down list to control where on the dashboard the filter is displayed.
1. Select your InfluxDB data source in the **Data source** drop-down list.

For more information, refer to the [Filter and Group by documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/filter-group-by/).

## Choose a variable syntax

The InfluxDB data source supports two variable syntaxes for use in the **Query** field:

- **`$<varname>`** - Use this syntax for standalone variable references. It doesn't allow you to use a variable in the middle of a word or expression.
- **`${varname}`** - Use this syntax when you want to interpolate a variable in the middle of an expression.

**InfluxQL examples:**

```sql
SELECT mean("value") FROM "logins" WHERE "hostname" =~ /^$host$/ AND $timeFilter GROUP BY time($__interval), "hostname"
```

```sql
SELECT mean("value") FROM "logins" WHERE "hostname" =~ /^${host}$/ AND $timeFilter GROUP BY time($__interval), "hostname"
```

When you enable the **Multi-value** or **Include all value** options with InfluxQL, Grafana converts the labels from plain text to a regular expression-compatible string, so you must use `=~` instead of `=` and wrap the variable in a regular expression such as `/^$host$/`.

For example, if `$host` is a multi-value variable and you select `server1` and `server2`, Grafana joins the values with `|`, wraps them in parentheses, and escapes any special characters. The interpolated query looks like this:

```sql
SELECT mean("value") FROM "logins" WHERE "hostname" =~ /^(server1|server2)$/ AND $timeFilter GROUP BY time($__interval), "hostname"
```

**SQL examples:**

```sql
SELECT $__dateBin(time), mean(usage_system) FROM cpu WHERE $__timeFilter(time) AND host = '$host' GROUP BY $__dateBin(time)
```

```sql
SELECT $__dateBin(time), mean(usage_system) FROM cpu WHERE $__timeFilter(time) AND host IN ($host) GROUP BY $__dateBin(time)
```

When you enable the **Multi-value** option with SQL, use the `IN` operator instead of `=` to match multiple values.

### Prevent unwanted value escaping

When you use a variable inside a regular expression, or when the variable is multi-value, Grafana escapes special characters in the values so the query remains valid. If you need the literal, unmodified value instead, use the `raw` format option:

```sql
SELECT mean("value") FROM "requests" WHERE "path" = '${path:raw}' AND $timeFilter GROUP BY time($__interval)
```

This is useful for custom variables whose values intentionally contain characters that Grafana would otherwise escape. For all available format options, refer to [Advanced variable format options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/#advanced-variable-format-options).

### Templated dashboard example

To view an example of a templated dashboard, refer to this [InfluxDB example dashboard](https://play.grafana.org/d/f62a0410-5abb-4dd8-9dfc-caddfc3e2ffd/eccb2445-b0a2-5e83-8e0f-6d5ea53ad575).

# Troubleshoot InfluxDB data source issues

This document provides solutions to common issues you may encounter when configuring or using the InfluxDB data source. Issues are organized to follow the typical setup and usage workflow. For configuration instructions, refer to [Configure the InfluxDB data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/configure/).

## Connection errors

The following errors occur when Grafana can't establish or maintain a connection to InfluxDB.

### "Plugin health check failed" or "An error occurred within the plugin"

**Symptoms:**

- All panels using InfluxDB return "An error occurred within the plugin"
- Adding a new InfluxDB data source fails with "Plugin health check failed"
- Connection settings appear blank in the UI

**Possible causes and solutions:**

| Cause                   | Solution                                                                                                                                                                                                                                                                     |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Platform outage         | If all InfluxDB panels fail at the same time, check the [Grafana Cloud status page](https://status.grafana.com/) for active incidents before investigating your own configuration. The issue may be a transient platform problem that resolves without customer-side action. |
| Authentication failure  | Check the Grafana server logs for auth-related errors. Verify your credentials haven't expired or been rotated.                                                                                                                                                              |
| Transient network issue | The error may self-resolve. Wait a few minutes and retry **Save & test**. If the issue persists, check network connectivity between Grafana and InfluxDB.                                                                                                                    |

### Failed to connect to InfluxDB

**Error message:** `error performing influxQL query` or `error performing flux query` or `error performing sql query`

**Cause:** Grafana can't establish a network connection to the InfluxDB server.

**Solution:**

1. Verify that the InfluxDB URL is correct in the data source configuration.
1. Check that InfluxDB is running and accessible from the Grafana server.
1. Ensure the URL includes the protocol (`http://` or `https://`).
1. Verify the port is correct (the InfluxDB default API port is `8086`).
1. Ensure there are no firewall rules blocking the connection.
1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your InfluxDB instance is not publicly accessible.

### PDC connection fails with "no such host"

**Error message:** `socks connect tcp ... -> influxdb.host:8086: dial tcp: lookup ... no such host`

**Cause:** When using [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/), the InfluxDB URL can't be resolved through the SOCKS proxy tunnel.

**Solution:**

1. Don't use `127.0.0.1` or `localhost` as the InfluxDB URL. PDC tunnels traffic over a SOCKS proxy, which can't resolve loopback addresses.
1. Use the machine's LAN IP address or a resolvable hostname instead.
1. Verify the hostname is resolvable from the network where the PDC agent is running.
1. If the error appeared suddenly without configuration changes, check the [Grafana Cloud status page](https://status.grafana.com/) for active incidents.

### PDC connection stops working after certificate renewal fails

**Symptoms:**

- A previously working PDC-connected data source stops working
- The PDC agent logs show errors renewing or signing the SSH certificate
- Queries fail with SOCKS proxy connection errors

**Cause:** The PDC agent uses a Grafana Cloud API token to periodically renew its SSH certificate. If the token is incorrect, has expired, or has been deleted, certificate renewal fails and the connection stops working when the current certificate expires.

**Solution:**

1. Check the PDC agent logs for authentication or certificate signing errors.
1. Verify the API token in the PDC agent configuration is valid. If the token has expired or been deleted, generate a new one from your PDC connection page in Grafana Cloud and update the agent configuration.
1. Restart the PDC agent after updating the token.
1. Refer to [Troubleshoot PDC issues](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/troubleshooting/) for agent error codes, log interpretation, and token management guidance.

### TLS certificate errors

**Error message:** `x509: certificate signed by unknown authority` or similar TLS verification errors

**Cause:** InfluxDB presents a TLS certificate that isn't signed by a certificate authority (CA) that the Grafana server trusts. This is common with self-signed certificates and internal corporate certificate authorities. TLS verification applies end to end, so this error also occurs when you connect through PDC.

**Solution:**

1. The InfluxDB data source supports TLS configuration for each data source instance. In the data source settings, expand **Auth and TLS/SSL Settings**, enable **CA cert**, and paste your CA certificate. Refer to [Auth and TLS/SSL settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/configure/#auth-and-tlsssl-settings) for details.
1. If your InfluxDB server requires client certificates, enable **TLS client auth** and provide the server name, client certificate, and client key.
1. As a last resort, you can enable **Skip TLS verify** to bypass certificate validation. Grafana doesn't recommend this for production use because it disables protection against man-in-the-middle attacks.

### Request timed out

**Error message:** `context deadline exceeded` or `request timeout` or `dial tcp <IP>:<port>: i/o timeout`

**Cause:** The connection to InfluxDB timed out before receiving a response. This is common after Grafana upgrades when infrastructure changes (such as database host migrations) happen at the same time.

**Solution:**

1. Verify network connectivity from the Grafana server to your InfluxDB endpoint. Check DNS resolution, firewall rules, and port access.
1. Confirm the InfluxDB host IP address or hostname hasn't changed. This is especially important after infrastructure migrations or Grafana upgrades.
1. Check the network latency between Grafana and InfluxDB.
1. Verify that InfluxDB is not overloaded or experiencing performance issues.
1. Increase the timeout setting in the data source configuration under **Advanced HTTP Settings**.
1. Reduce the time range or complexity of your query.

## Authentication errors

The following errors occur when there are issues with authentication credentials or permissions.

### Unauthorized (401)

**Error message:** "401 Unauthorized" or "authorization failed"

**Cause:** The authentication credentials are invalid or missing. Transient network issues can also surface as authentication errors even when your credentials are valid.

**Solution:**

1. If the error is intermittent or appeared without a configuration change, wait a few minutes and retry **Save & test** to rule out a transient network issue before rotating credentials.
1. Verify that the token or password is correct in the data source configuration.
1. For Flux and SQL, ensure the token has not expired.
1. For InfluxQL with InfluxDB 2.x, verify the token is set as an `Authorization` header with the value `Token <your-token>`.
1. For InfluxDB 1.x, verify the username and password are correct.
1. Check that the token has the required permissions to access the specified bucket or database.

### Forbidden (403)

**Error message:** "403 Forbidden" or "access denied"

**Cause:** The authenticated user or token doesn't have permission to access the requested resource.

**Solution:**

1. Verify the token has read access to the specified bucket or database.
1. Check the token's permissions in the InfluxDB UI under **API Tokens**.
1. Ensure the organization ID is correct for Flux queries.
1. For InfluxQL with InfluxDB 2.x, verify the DBRP mapping is configured correctly.

## Configuration errors

The following errors occur when the data source is not configured correctly.

### URL not configured

**Error message:** `missing URL from datasource configuration`

**Cause:** The data source URL field is empty.

**Solution:**

1. Open the data source configuration in Grafana.
1. Enter the full URL of your InfluxDB instance in the **URL** field, including the protocol and port (for example, `http://localhost:8086`).
1. Click **Save & test** to verify the connection.

### Unknown influx version

**Error message:** "unknown influx version"

**Cause:** The query language is not properly configured in the data source settings.

**Solution:**

1. Open the data source configuration in Grafana.
1. Verify that a valid query language is selected: **Flux**, **InfluxQL**, or **SQL**.
1. Match the query language to your InfluxDB version:

| InfluxDB version                         | Recommended query language | Notes                                                                                                                                                       |
| ---------------------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1.x                                      | InfluxQL                   | Flux is available from 1.8+ but InfluxQL is the primary language.                                                                                           |
| 2.x (OSS/Cloud)                          | Flux                       | InfluxQL is also available via the v1 compatibility API, but requires [DBRP mapping](https://docs.influxdata.com/influxdb/cloud/query-data/influxql/dbrp/). |
| 3.x / Cloud Dedicated / Cloud Serverless | SQL or InfluxQL            | Flux is not supported on InfluxDB 3.x.                                                                                                                      |

Each query language uses a different API endpoint. If you select the wrong language for your InfluxDB version, health checks and queries fail.

### Invalid data source info received

**Error message:** `invalid data source info received`

**Cause:** The data source configuration is incomplete or corrupted.

**Solution:**

1. Delete and recreate the data source.
1. Ensure all required fields are populated based on your query language:
   - **Flux:** URL, Organization, Token, Default Bucket
   - **InfluxQL:** URL, Database, User, Password
   - **SQL:** URL, Database, Token

### DBRP mapping required

**Error message:** "database not found" or queries return no data with InfluxQL on InfluxDB 2.x

**Cause:** InfluxQL queries on InfluxDB 2.x require a Database and Retention Policy (DBRP) mapping.

**Solution:**

1. Create a DBRP mapping in InfluxDB using the CLI or API.
1. Refer to [Manage DBRP Mappings](https://docs.influxdata.com/influxdb/cloud/query-data/influxql/dbrp/) for guidance.
1. Verify the database name in Grafana matches the DBRP mapping.

### Browser access mode disabled

**Error message:** `Direct browser access in the InfluxDB datasource is no longer available. Switch to server access mode.`

**Cause:** The data source is configured for direct browser access, which is no longer supported.

**Solution:**

1. Open the data source configuration in Grafana.
1. Change the access mode to **Server (default)**.
1. Click **Save & test** to verify the connection.

### Content Security Policy (CSP) violation

**Symptoms:**

- CSP violation errors in the browser console referencing the InfluxDB plugin
- `net::ERR_ABORTED` on proxy requests
- The InfluxDB plugin attempts direct browser-to-InfluxDB connections

**Cause:** You're running an outdated version of Grafana. Browser access mode was removed in Grafana 9.2.0, and older versions may attempt direct browser connections that violate CSP policies. Current versions of the data source support only server (proxy) access, where the Grafana server sends all queries to InfluxDB and the browser never connects to InfluxDB directly.

**Solution:**

1. Upgrade to the latest stable Grafana release. The InfluxDB data source requires Grafana 12.3.0 or later.
1. After upgrading, verify the data source access mode is set to **Server (default)**.

## Health check errors

The following errors occur when clicking **Save & test** to validate the data source connection. Each query language uses a different health check query.

### Flux health check errors

**"error performing flux query"**

**Cause:** The health check query `buckets()` failed to execute.

**Solution:**

1. Verify the InfluxDB URL is correct and reachable.
1. Check that the token is valid and has not expired.
1. Ensure the organization ID is correct.

**"error reading buckets"**

**Cause:** The `buckets()` query executed but returned an error.

**Solution:**

1. Verify the token has permission to list buckets.
1. Check that the organization ID matches the token's organization.

**"error getting flux query buckets"**

**Cause:** The `buckets()` query executed without error but returned no data.

**Solution:**

1. Verify the token has permission to list buckets.
1. Check that the organization ID is correct.
1. Ensure InfluxDB is running and accessible.

### InfluxQL health check errors

**"error performing influxQL query"**

**Cause:** The health check query `SHOW MEASUREMENTS` failed to execute.

**Solution:**

1. Verify the InfluxDB URL is correct and reachable.
1. Check the username and password (or token for InfluxDB 2.x).
1. Verify the database name exists.

**"error reading influxDB"**

**Cause:** The `SHOW MEASUREMENTS` query executed but returned an error.

**Solution:**

1. Verify the database name is correct.
1. Check that the user has permission to run `SHOW MEASUREMENTS`.
1. For InfluxDB 2.x, verify DBRP mapping is configured.

**"error connecting InfluxDB influxQL"**

**Cause:** The health check completed but the response couldn't be processed.

**Solution:**

1. Verify the database name is correct.
1. Check that the user has permission to run `SHOW MEASUREMENTS`.
1. Ensure the database exists and contains measurements.
1. For InfluxDB 2.x, verify DBRP mapping is configured.

### SQL health check errors

**"error performing sql query"**

**Cause:** The health check query `select 1` failed to execute against the FlightSQL endpoint.

**Solution:**

1. Verify the InfluxDB URL is correct. The SQL health check connects via gRPC (FlightSQL).
1. Check the token is valid and has the required permissions.
1. If using TLS, verify the certificate configuration. Toggle **Insecure Connection** if connecting without TLS.
1. Ensure the InfluxDB 3.x instance is running and the FlightSQL endpoint is accessible.

### 0 measurements found

**Error message:** `datasource is working. 0 measurements found`

**Cause:** The connection is successful, but the database contains no measurements.

**Solution:**

1. Verify you are connecting to the correct database.
1. Check that data has been written to the database.
1. If the database is new, add some test data to verify the connection.

## Query errors

The following errors occur when there are issues with query syntax or execution.

### Query syntax error

**Error message:** "error parsing query: found THING" or "failed to parse query: found WERE, expected ; at line 1, char 38"

**Cause:** The query contains invalid syntax.

**Solution:**

1. Check your query syntax for typos or invalid keywords.
1. For InfluxQL, verify the query follows the correct syntax:

   ```sql
   SELECT <field> FROM <measurement> WHERE <condition>
   ```

1. For SQL, verify your query uses standard SQL syntax supported by InfluxDB 3.x. Common issues include using InfluxQL-specific syntax (such as `GROUP BY time()`) in SQL mode. Refer to the [InfluxDB SQL reference](https://docs.influxdata.com/influxdb/cloud-serverless/reference/sql/) for supported functions.
1. For Flux, ensure proper pipe-forward syntax and function calls.
1. Use the InfluxDB UI or CLI to test your query directly.

### Query timeout limit exceeded

**Error message:** "query-timeout limit exceeded"

**Cause:** The query took longer than the configured timeout limit in InfluxDB.

**Solution:**

1. Reduce the time range of your query.
1. Add more specific filters to limit the data scanned.
1. Increase the query timeout setting in InfluxDB if you have administrator access.
1. Optimize your query to reduce complexity.

### Too many series or data points

**Error message:** "max-series-per-database limit exceeded" or "A query returned too many data points and the results have been truncated"

**Cause:** The query is returning more data than the configured limits allow.

**Solution:**

1. Reduce the time range of your query.
1. Add filters to limit the number of series returned.
1. Increase the **Max series** setting in the data source configuration under **Advanced Database Settings**.
1. Use aggregation functions to reduce the number of data points.
1. For SQL, use `$__dateBin(time)` to aggregate data into time buckets and reduce cardinality.
1. For Flux, use `aggregateWindow()` to downsample data.

### FlightSQL errors (SQL query language)

**Error message:** Messages prefixed with `"flightsql: "` followed by a gRPC error description.

**Cause:** The SQL (FlightSQL) backend encountered an error communicating with InfluxDB 3.x.

**Possible causes and solutions:**

| Error code         | Cause                                                    | Solution                                              |
| ------------------ | -------------------------------------------------------- | ----------------------------------------------------- |
| `InvalidArgument`  | The SQL query syntax is invalid.                         | Check your SQL query for syntax errors.               |
| `PermissionDenied` | The token doesn't have access to the requested resource. | Verify the token has read access to the database.     |
| `NotFound`         | The requested table or database doesn't exist.           | Check the database name and table name in your query. |
| `Unavailable`      | The InfluxDB server is unreachable.                      | Verify InfluxDB is running and the URL is correct.    |
| `Unauthenticated`  | The token is missing, invalid, or expired.               | Update the token in the data source configuration.    |

### No time column found

**Error message:** "no time column found"

**Cause:** The query result doesn't include a time column, which is required for time-series visualization.

**Solution:**

1. Ensure your query includes a time field.
1. For Flux, verify the query includes `_time` in the output.
1. For SQL, ensure the query returns a timestamp column.
1. Check that the time field is not being filtered out or excluded.

## Annotation errors

The following errors occur when using InfluxDB annotations on dashboards.

### "Query missing in annotation definition"

**Cause:** The annotation query field is empty.

**Solution:**

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Dashboard options** icon to open the sidebar.
1. Expand the **Annotations** section.
1. Select the InfluxDB annotation.
1. Click **Open query editor** to open the **Annotation Query** dialog box.
1. Enter a valid query in the **InfluxQL Query** field. The query must include `WHERE $timeFilter`. For example:

   ```sql
   SELECT title, description FROM events WHERE $timeFilter ORDER BY time ASC
   ```

### "Flux requires the standard annotation query"

**Cause:** A Flux data source is using the legacy InfluxQL annotation editor instead of the standard Flux query editor.

**Solution:**

1. Delete the existing annotation query.
1. Create a new annotation query and select your Flux-configured InfluxDB data source.
1. Write a Flux query that returns data frames with time and text fields. For example:

   ```flux
   from(bucket: "events")
     |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
     |> filter(fn: (r) => r["_measurement"] == "deployments")
   ```

### Annotations don't appear on the graph

**Cause:** Annotations are configured but aren't visible on the dashboard.

**Solution:**

1. Verify the annotation query returns data by testing it in Explore.
1. Check the dashboard time range covers the time period of your annotation events.
1. Ensure the annotation toggle is enabled in the dashboard (check the annotation icon in the top menu bar).
1. For InfluxQL, confirm the query includes `WHERE $timeFilter`.
1. If your query returns multiple columns, verify the field mappings (**Text**, **Tags**, **TimeEnd**) are set correctly.

## Alerting errors

The following errors occur when using InfluxDB queries with Grafana Alerting.

### Alert rule fails with template variable errors

**Cause:** The alert query contains template variables such as `$hostname` or `$region`.

**Solution:**

Alert queries can't use template variables because Grafana evaluates alert rules on the backend without dashboard context. Replace template variables with hard-coded values:

1. Open the alert rule.
1. Replace any `$variable` references with literal values.
1. Save the alert rule.

If you need the same query in both a dashboard panel and an alert rule, maintain two separate queries: one with variables for the dashboard and one with hard-coded values for alerting.

### Alert evaluation returns "no data"

**Cause:** The alert query doesn't return time-series data that Grafana can evaluate.

**Solution:**

1. Test the query in Explore first to verify it returns data.
1. For InfluxQL, ensure the query uses an aggregation function (such as `mean`, `sum`, `count`) with `GROUP BY time($__interval)`.
1. For Flux, use `aggregateWindow()` to produce time-bucketed results.
1. For SQL, use `$__dateBin(time)` or `$__timeGroup(time)` to aggregate by time.
1. Check that the alert evaluation time range contains data. Alerts use a fixed time range, not the dashboard's time picker.
1. Verify the data source connection is working by clicking **Save & test** in the data source settings.

## Template variable errors

The following issues occur when using template variables with InfluxDB queries.

### Variable drop-down shows stale or historical values

**Cause:** Metadata queries such as `SHOW TAG VALUES` return values from the entire retention period, not just the dashboard time range. Hosts or sensors that stopped reporting long ago still appear in the drop-down.

**Solution:**

1. Add a time condition to the variable query, such as `WHERE $timeFilter` for InfluxQL or `WHERE $__timeFilter(time)` for SQL.
1. Set the variable's **Refresh** option to **On time range change**.
1. Refer to [Scope variables to the dashboard time range](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/template-variables/#scope-variables-to-the-dashboard-time-range) for examples.

### Multi-select variable breaks the query

**Cause:** When a variable has the **Multi-value** or **Include all value** option enabled, Grafana interpolates the selected values as a regular expression group, such as `(server1|server2)`. Queries that compare with `=` or that don't wrap the variable in a regular expression fail or return no data.

**Solution:**

1. For InfluxQL, use the `=~` operator and wrap the variable in a regular expression: `"hostname" =~ /^$host$/`.
1. For SQL, use the `IN` operator: `host IN ($host)`.
1. Refer to [Choose a variable syntax](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/template-variables/#choose-a-variable-syntax) for working examples.

### Variable values are escaped unexpectedly

**Cause:** Grafana escapes special characters in variable values when the variable is multi-value or used inside a regular expression. Custom variable values that intentionally contain special characters, such as paths or expressions, arrive at InfluxDB modified.

**Solution:**

Use the `raw` format option, such as `${path:raw}`, to interpolate the literal value without escaping. Refer to [Prevent unwanted value escaping](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/template-variables/#prevent-unwanted-value-escaping) for details.

### Variable drop-down contains a blank or duplicate entry

**Cause:** The variable query returns duplicate values. Grafana deduplicates results, but depending on the plugin version, duplicates can surface as a blank entry in the drop-down.

**Solution:**

1. Update the variable query to return unique values. For SQL, use `SELECT DISTINCT`. For InfluxQL, `SHOW` metadata queries already return unique values, so check for duplicates across retention policies.
1. Alternatively, use the variable's **Regex** option to filter the results.

## Other common issues

The following issues don't produce specific error messages but are commonly encountered during day-to-day use.

### "Data source was not found"

**Symptoms:**

- Dashboard panels display "data source \<UID\> was not found"
- Manually re-running queries in the panel editor works after you select the data source again

**Cause:** Dashboard panels reference an old or deleted data source UID. This happens when a data source is deleted and recreated, since the new data source gets a different UID. Dashboard schema migrations can also leave panels with invalid data source references, so verify your panels after a dashboard is migrated to a new schema version.

**Solution:**

1. Find the current UID of your InfluxDB data source. Navigate to **Connections** > **Data sources**, open the data source, and copy the UID from the page URL.
1. To find every stale reference at once, open the dashboard's JSON model from the dashboard settings and search for the UID shown in the panel error message.
1. Fix the references by editing each affected panel and selecting the correct InfluxDB data source from the drop-down again, or by replacing the stale UID in the JSON model and saving the dashboard.
1. To avoid this issue, update existing data sources instead of deleting and recreating them.

### 404 Not Found when sending Telegraf metrics to Grafana Cloud

**Error message:** "404 Not Found" when Telegraf writes to the Grafana Cloud InfluxDB-compatible endpoint.

**Cause:** The Telegraf `influxdb_v2` output plugin isn't compatible with the Grafana Cloud metrics endpoint. This commonly occurs when using PrivateLink or the standard InfluxDB-compatible write endpoint.

**Solution:**

1. Switch the Telegraf output plugin from `influxdb_v2` to `influxdb` (v1) in your Telegraf configuration.
1. Ensure the endpoint URL and credentials match those shown in your Grafana Cloud InfluxDB configuration page.
1. Restart Telegraf after making the change.

### Empty query results

**Cause:** The query returns no data.

**Solution:**

1. Verify the time range includes data in your database.
1. Check that the measurement and field names are correct. For SQL, table names in InfluxDB 3.x are case-sensitive.
1. Test the query directly in the InfluxDB UI or CLI.
1. Ensure filters are not excluding all data.
1. For SQL, verify the `$__timeFilter(time)` macro is included so the query uses the dashboard time range.
1. For InfluxQL, verify the retention policy contains data for the selected time range.

### Numeric values stored as strings display a blank panel

**Symptoms:**

- Time series panels show no data even though the query returns results
- Values appear in the table view but can't be graphed

**Cause:** InfluxDB sets a field's data type when the field is first written. If numeric values were written as strings, for example quoted in line protocol, InfluxDB returns them as strings and Grafana can't graph them. This is a data issue in InfluxDB, not in Grafana.

**Solution:**

1. As a workaround, add the [Convert field type](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/#convert-field-type) transformation to the panel and convert the field to **Number**.
1. To fix the root cause, correct the data type at write time. InfluxDB can't change the type of an existing field, so you may need to write the data to a new field or measurement.
1. In older Grafana versions, the **Convert field type** transformation converted null values to zeros. Current versions preserve null values. If you see nulls displayed as zeros after conversion, upgrade Grafana.

### Panel values differ between view and edit mode

**Cause:** Grafana calculates the query interval (`$__interval`) from the time range and the maximum number of data points, which is based on the panel's width. A panel rendered in edit mode has a different width than in the dashboard view, so queries that group by `time($__interval)` can aggregate into different bucket sizes and display different values. Both results are correct. They're aggregated at different resolutions.

**Solution:**

1. To make values consistent, set a fixed **Min interval** or **Max data points** value in the panel's query options.
1. Alternatively, use an explicit interval in the query, such as `GROUP BY time(1m)` instead of `GROUP BY time($__interval)`.

### Legend or tooltip colors don't match the series

**Cause:** The query returns multiple series with the same name. This commonly happens when a query doesn't group by the tags that differentiate series, or when an alias hides the distinguishing information. Grafana assigns colors and tooltip values by series name, so duplicate names cause the legend and hover tooltip to display mismatched colors or values.

**Solution:**

1. Group by the tags that make each series unique, such as `GROUP BY "hostname"`.
1. Use alias patterns such as `$tag_hostname` in the **ALIAS** field so each series has a unique display name. Refer to [Alias patterns](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/query-editor/#alias-patterns) for details.
1. Use the panel inspector to confirm each returned series has a distinct name.

### Slow query performance

**Cause:** Queries take a long time to execute.

**Solution:**

1. Reduce the time range of your query.
1. Add more specific filters to limit the data scanned.
1. Increase the **Min time interval** setting to reduce the number of data points.
1. Check InfluxDB server performance and resource utilization.
1. For SQL, use `$__dateBin(time)` with aggregation functions to downsample data. Add `WHERE` clauses to narrow the query scope.
1. For Flux, use `aggregateWindow()` to downsample data before visualization.
1. Consider using continuous queries or tasks to pre-aggregate data.

### InfluxDB receives a high volume of queries from Grafana

**Symptoms:**

- Unexpected query load on your InfluxDB server
- You're unsure whether the traffic originates from Grafana

**Cause:** Every panel on a dashboard sends its queries to InfluxDB each time the dashboard refreshes. Query volume scales with the number of panels, the number of queries per panel, the refresh interval, and the number of people viewing the dashboard at the same time. For example, a dashboard with 20 panels refreshing every 10 seconds generates at least 120 queries per minute for each viewer.

**Solution:**

1. Increase the dashboard refresh interval or turn off auto-refresh where it isn't needed.
1. Reduce the number of panels and queries per dashboard.
1. Increase the **Min time interval** setting to reduce the resolution of each query.
1. In Grafana Enterprise and Grafana Cloud, enable [query caching](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/#query-and-resource-caching) so identical queries within the cache window are served from cache instead of hitting InfluxDB.
1. To confirm whether traffic originates from your Grafana Cloud instance, compare the source IP addresses against the [Grafana Cloud allowlist](https://grafana.com/docs/grafana-cloud/platform/security-and-account-management/security-and-access/allow-list/).

### Data appears delayed or missing recent points

**Cause:** The visualization doesn't show the most recent data.

**Solution:**

1. Check the dashboard time range and refresh settings.
1. Verify the **Min time interval** is not set too high.
1. Ensure InfluxDB has finished writing the data.
1. Check for clock synchronization issues between Grafana and InfluxDB.

## Enable debug logging

{{< admonition type="caution" >}}
InfluxDB plugin versions earlier than 13.1.0 could write API tokens to Grafana server logs in plain text at default log levels. Before troubleshooting with logs, upgrade to plugin version 13.1.0 or later. Treat log files as sensitive, and rotate any tokens that may have been exposed.
{{< /admonition >}}

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location).
1. Look for InfluxDB-specific entries that include request and response details.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you've tried the solutions in this guide and still encounter issues:

1. Check the [InfluxDB documentation](https://docs.influxdata.com/) for API-specific guidance.
1. Review the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review [InfluxDB data source issues on GitHub](https://github.com/grafana/grafana-influxdb-datasource/issues) for known bugs.
1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user.
1. When reporting issues, include:
   - Grafana version
   - InfluxDB data source plugin version, found on the **Administration > Plugins** page
   - InfluxDB version and product (OSS, Cloud, Enterprise)
   - Query language (Flux, InfluxQL, or SQL)
   - Error messages (redact sensitive information)
   - Steps to reproduce
   - Relevant configuration such as data source settings, HTTP method, and TLS settings (redact tokens, passwords, and other credentials)

# Configure the Jaeger data source

This document explains how to configure the Jaeger data source in Grafana, including connection settings, authentication, trace correlation, and provisioning.

## Before you begin

Before configuring the data source, ensure you have:

- **Grafana permissions:** Organization administrator role
- **Jaeger instance:** A running Jaeger instance with an accessible query endpoint (default: `http://localhost:16686`)

## Add the data source

To add the Jaeger data source:

1. Click **Connections** in the left-side menu.
1. Click **Add new connection**.
1. Type `Jaeger` in the search bar.
1. Select **Jaeger**.
1. Click **Add new data source**.

## Connection settings

Configure the basic connection options for the data source.

| Setting     | Description                                                        |
| ----------- | ------------------------------------------------------------------ |
| **Name**    | The name used to refer to the data source in panels and queries.   |
| **Default** | Toggle to make this the default data source for new panels.        |
| **URL**     | The URL of your Jaeger instance, such as `http://localhost:16686`. |

## Authentication

The Jaeger data source supports the following authentication methods.

### Basic authentication

To use basic authentication:

1. Select **Basic auth** from the drop-down.
1. Enter the **User** name.
1. Enter the **Password**.

### Forward OAuth identity

Select **Forward OAuth identity** from the drop-down to forward the user's upstream OAuth identity to the data source. This is useful when Jaeger is behind an authenticating proxy.

### Custom headers

You can add custom HTTP headers to requests sent to the Jaeger instance. Use this to pass authorization tokens or other required headers.

To add custom headers, expand the **Additional settings** section and configure them under **Custom HTTP headers**.

### TLS configuration

To configure TLS/mTLS for the connection, expand the **Additional settings** section and configure the following options:

| Setting             | Description                                                 |
| ------------------- | ----------------------------------------------------------- |
| **TLS client auth** | Toggle to enable client certificate authentication.         |
| **With CA cert**    | Toggle to provide a custom CA certificate.                  |
| **Skip TLS verify** | Toggle to skip TLS certificate verification (testing only). |
| **Server name**     | Server name for TLS certificate validation.                 |

### Private data source connect

**Private data source connect** - _Only for Grafana Cloud users._

{{< admonition type="note" >}}
This section is only visible when the Grafana instance has the secure SOCKS proxy feature enabled.
{{< /admonition >}}

Private data source connect, or PDC, allows you to establish a private, secured connection between a Grafana Cloud instance, or stack, and data sources secured within a private network. Click the drop-down to locate the URL for PDC. For more information regarding Grafana PDC refer to [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) and [Configure Grafana private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc) for instructions on setting up a PDC connection.

Click **Manage private data source connect** to open your PDC connection page and view your configuration details.

## Trace to logs

{{< figure src="/media/docs/tempo/tempo-trace-to-logs-9-4.png" max-width="800px" class="docs-image--no-shadow" caption="Trace to logs configuration settings" >}}

The **Trace to logs** setting configures the [trace to logs feature](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/trace-integration/) that links trace spans to related log entries.

{{< admonition type="note" >}}
If you use Grafana Cloud, open a [support ticket in the Cloud Portal](/profile/org#support) to access this feature.
{{< /admonition >}}

There are two ways to configure trace to logs:

- **Simple configuration:** Use a default query with tag-based filtering.
- **Custom query:** Write a query using [template variable syntax](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/) to interpolate values from the trace or span.

### Use a simple configuration

1. Select the target data source from the drop-down list.
1. Set **Span start time shift** and **Span end time shift** to widen or shift the time range if log timestamps don't exactly match span timestamps.
1. Configure **Tags** to use in the logs query. Tags must be present in span attributes or resources for the link to appear. You can remap tag names if the target data source doesn't allow dots in labels (for example, remap `http.status` to `http_status`).
1. Optionally, toggle on **Filter by trace ID** or **Filter by span ID** to further filter logs.

### Configure a custom query

1. Select the target data source from the drop-down list.
1. Set **Span start time shift** and **Span end time shift**.
1. Optionally, configure **Tags** to map. Use the `${__tags}` variable in your custom query to interpolate mapped tags. If you don't map tags, you can still reference any span tag directly, such as `method="${__span.tags.method}"`.
1. Toggle on **Use custom query**.
1. Write a custom query using the variables listed in this table. The link only appears when all variables resolve to non-empty values.

### Variables for custom queries

To use a variable, wrap it in `${}`. For example: `${__span.name}`.

| Variable               | Description                                                                                                                            |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| **\_\_tags**           | Interpolates mapped tags as a label matcher string in the target data source syntax. Only includes tags present in the span.           |
| **\_\_span.spanId**    | The ID of the span.                                                                                                                    |
| **\_\_span.traceId**   | The ID of the trace.                                                                                                                   |
| **\_\_span.duration**  | The duration of the span.                                                                                                              |
| **\_\_span.name**      | The name of the span.                                                                                                                  |
| **\_\_span.tags**      | Namespace for span tags. Access a specific tag with `${__span.tags.version}`. For tags with dots, use `${__span.tags["http.status"]}`. |
| **\_\_trace.traceId**  | The ID of the trace.                                                                                                                   |
| **\_\_trace.duration** | The duration of the trace.                                                                                                             |
| **\_\_trace.name**     | The name of the trace.                                                                                                                 |

### Trace to logs settings

| Setting                   | Description                                                                                                                                                                |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Data source**           | The target logs data source. You can select Loki or Splunk logs data sources.                                                                                              |
| **Span start time shift** | Shifts the start time for the logs query based on the span's start time. Use time units such as `5s`, `1m`, `3h`. Use negative values to extend to the past. Default: `0`. |
| **Span end time shift**   | Shifts the end time for the logs query based on the span's end time. Default: `0`.                                                                                         |
| **Tags**                  | Tags to use in the logs query. Default: `cluster`, `hostname`, `namespace`, `pod`, `service.name`, `service.namespace`.                                                    |
| **Filter by trace ID**    | Toggles whether to append the trace ID to the logs query.                                                                                                                  |
| **Filter by span ID**     | Toggles whether to append the span ID to the logs query.                                                                                                                   |
| **Use custom query**      | Toggles the use of a custom query with variable interpolation.                                                                                                             |
| **Query**                 | The custom query. Use variable interpolation to customize with values from the span.                                                                                       |

## Trace to metrics

The **Trace to metrics** setting lets you link trace spans to related metrics queries.

To configure trace to metrics:

1. Select the target metrics data source from the drop-down list.
1. Configure **Tags** to map span attributes to metric label names. For example, map `k8s.pod` to `pod`.
1. Create linked queries using the `$__tags` keyword to interpolate the mapped tags.

| Setting         | Description                                                                                                                                     |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| **Data source** | The target metrics data source.                                                                                                                 |
| **Tags**        | Maps span attribute names to metric label names. For example, map `k8s.pod` to `pod`. Use the `$__tags` keyword to interpolate tags in queries. |

Each linked query consists of:

- **Link Label:** _(Optional)_ A descriptive label for the linked query.
- **Query:** The query to run when navigating from a trace to the metrics data source. Use `$__tags` to interpolate tags. For example, the query `requests_total{$__tags}` with tags `k8s.pod=pod` and `cluster` produces `requests_total{pod="nginx-554b9", cluster="us-east-1"}`.

## Node graph

The **Node graph** setting is located under **Additional settings**, which is collapsed by default. Expand the **Additional settings** section on the data source configuration page to access it.

The **Node graph** setting enables the [Node Graph visualization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/node-graph/), which is disabled by default.

When enabled, Grafana displays the Node Graph above the trace view, showing the relationships between services in the trace.

## Span bar

The **Span bar** setting is located under **Additional settings**, which is collapsed by default. Expand the **Additional settings** section on the data source configuration page to access it.

The **Span bar** setting controls what additional information appears in the span bar row of the trace view.

| Option       | Description                                                                                     |
| ------------ | ----------------------------------------------------------------------------------------------- |
| **None**     | Adds nothing to the span bar row.                                                               |
| **Duration** | _(Default)_ Displays the span duration on the span bar row.                                     |
| **Tag**      | Displays a span tag value on the span bar row. Specify the tag key to use, such as `component`. |

## Trace ID time parameters

The **Query Trace by ID with Time Params** setting is located under **Additional settings**, which is collapsed by default. Expand the **Additional settings** section on the data source configuration page to access it.

The **Enable Time Parameters** toggle controls whether Grafana sends `start` and `end` time parameters when querying a trace by ID. Enable this if your Jaeger instance benefits from time-bounded trace lookups.

## Verify the connection

Click **Save & test** to verify the data source connection. A successful test displays the message **Data source is working**.

If the test fails, refer to [Troubleshoot Jaeger data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/jaeger/troubleshooting/) for help resolving common errors.

## Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, refer to [Provisioning Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

### Provisioning example

```yaml
apiVersion: 1

datasources:
  - name: Jaeger
    type: jaeger
    uid: jaeger-ds
    url: http://localhost:16686
    access: proxy
    basicAuth: true
    basicAuthUser: <USERNAME>
    isDefault: false
    jsonData:
      tracesToLogsV2:
        datasourceUid: 'loki'
        spanStartTimeShift: '1h'
        spanEndTimeShift: '-1h'
        tags:
          - key: 'job'
          - key: 'instance'
          - key: 'pod'
          - key: 'namespace'
        filterByTraceID: false
        filterBySpanID: false
        customQuery: true
        query: 'method="$${__span.tags.method}"'
      tracesToMetrics:
        datasourceUid: 'prom'
        spanStartTimeShift: '-2m'
        spanEndTimeShift: '2m'
        tags:
          - key: 'service.name'
            value: 'service'
          - key: 'job'
        queries:
          - name: 'Sample query'
            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))'
      nodeGraph:
        enabled: true
      traceIdTimeParams:
        enabled: true
      spanBar:
        type: 'Duration'
    secureJsonData:
      basicAuthPassword: <PASSWORD>
```

Replace `<USERNAME>` and `<PASSWORD>` with your Jaeger credentials.

## Provision with Terraform

You can provision the Jaeger data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation.

### Terraform example

The following example provisions a Jaeger data source:

```hcl
terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 2.0.0"
    }
  }
}

provider "grafana" {
  url  = "<YOUR_GRAFANA_URL>"
  auth = "<YOUR_SERVICE_ACCOUNT_TOKEN>"
}

resource "grafana_data_source" "jaeger" {
  type = "jaeger"
  name = "Jaeger"
  url  = "http://localhost:16686"

  json_data_encoded = jsonencode({
    nodeGraph = {
      enabled = true
    }
    traceIdTimeParams = {
      enabled = true
    }
    tracesToLogsV2 = {
      datasourceUid    = "loki"
      filterByTraceID  = true
      filterBySpanID   = false
      tags = [
        { key = "service.name", value = "service" },
        { key = "job" }
      ]
    }
  })
}
```

### Terraform example with basic authentication

The following example provisions a Jaeger data source with basic authentication:

```hcl
resource "grafana_data_source" "jaeger_auth" {
  type                = "jaeger"
  name                = "Jaeger"
  url                 = "http://localhost:16686"
  basic_auth_enabled  = true
  basic_auth_username = "<USERNAME>"

  json_data_encoded = jsonencode({
    nodeGraph = {
      enabled = true
    }
    traceIdTimeParams = {
      enabled = true
    }
  })

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = "<PASSWORD>"
  })
}
```

Replace the following placeholders:

- _`<YOUR_GRAFANA_URL>`_: Your Grafana instance URL (for example, `https://your-org.grafana.net` for Grafana Cloud)
- _`<YOUR_SERVICE_ACCOUNT_TOKEN>`_: A service account token with data source permissions
- _`<USERNAME>`_: The username for basic authentication
- _`<PASSWORD>`_: The password for basic authentication

For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source).

# Jaeger query editor

This document explains how to use the Jaeger query editor to search for traces, query by trace ID, visualize service dependencies, and import trace files.

For general information about querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/).

## Before you begin

- Ensure you have [configured the Jaeger data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/jaeger/configure/).
- Verify the connection is working by clicking **Save & test** in the data source settings.

## Query types

The Jaeger query editor supports the following query types:

- **Search:** Find traces by service, operation, tags, and duration.
- **TraceID:** Query a specific trace by its ID.
- **Dependency graph:** Visualize service dependencies within a time range.
- **Import trace:** Upload a JSON trace file for visualization.

## Search for traces

To search for traces:

1. Select **Search** from the **Query type** selector.
1. Fill out the search form:

| Field              | Description                                                                                                                            |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Service Name**   | Select a service from the drop-down list, or type to filter. Supports template variables.                                              |
| **Operation Name** | Select an operation for the chosen service. Select **All** to query all operations. This field is disabled until you select a service. |
| **Tags**           | Enter tags in [`logfmt`](https://brandur.org/logfmt) format, such as `error=true db.statement="select * from User"`.                   |
| **Min Duration**   | Filter traces with a duration greater than this value. Use formats like `1.2s`, `100ms`, or `500us`.                                   |
| **Max Duration**   | Filter traces with a duration less than this value. Use the same format as **Min Duration**.                                           |
| **Limit**          | Maximum number of traces to return.                                                                                                    |

{{< figure src="/static/img/docs/explore/jaeger-search-form.png" class="docs-image--no-shadow" caption="Jaeger query editor showing a search query" >}}

## Query by trace ID

To query a specific trace:

1. Select **TraceID** from the **Query type** selector.
1. Enter the trace ID into the **Trace ID** field.
1. Press **Shift+Enter** to run the query.

{{< figure src="/static/img/docs/explore/jaeger-trace-id.png" class="docs-image--no-shadow" caption="Jaeger query editor with TraceID selected" >}}

## Visualize the dependency graph

The dependency graph query type displays service dependencies as a [Node Graph](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/node-graph/). It shows how services communicate and the volume of calls between them.

To visualize the dependency graph:

1. Select **Dependency graph** from the **Query type** selector.
1. Set the dashboard time range to the period you want to analyze.
1. Run the query.

The dependency graph uses the dashboard time range to query the Jaeger `/api/dependencies` endpoint. Grafana displays the result as a Node Graph with:

- **Nodes:** Each node represents a service.
- **Edges:** Each edge represents calls between services, with the call count shown as the edge label.

{{< admonition type="note" >}}
Your Jaeger instance must have dependency data available. If the graph is empty, verify that Jaeger is collecting and processing dependency information for the selected time range.
{{< /admonition >}}

## Import a trace

You can upload a JSON file that contains a single trace and visualize it in Grafana. If the file contains multiple traces, Grafana visualizes the first trace.

To import a trace:

1. Click **Import trace** in the query editor.
1. Select a JSON file in the Jaeger trace format.

{{< figure src="/static/img/docs/explore/jaeger-upload-json.png" class="docs-image--no-shadow" caption="Jaeger data source with import trace selected" >}}

### Trace JSON example

The JSON file must follow the Jaeger trace format with a `data` array containing trace objects:

```json
{
  "data": [
    {
      "traceID": "2ee9739529395e31",
      "spans": [
        {
          "traceID": "2ee9739529395e31",
          "spanID": "2ee9739529395e31",
          "flags": 1,
          "operationName": "CAS",
          "references": [],
          "startTime": 1616095319593196,
          "duration": 1004,
          "tags": [
            {
              "key": "sampler.type",
              "type": "string",
              "value": "const"
            }
          ],
          "logs": [],
          "processID": "p1",
          "warnings": null
        }
      ],
      "processes": {
        "p1": {
          "serviceName": "loki-all",
          "tags": [
            {
              "key": "jaeger.version",
              "type": "string",
              "value": "Go-2.25.0"
            }
          ]
        }
      },
      "warnings": null
    }
  ],
  "total": 0,
  "limit": 0,
  "offset": 0,
  "errors": null
}
```

## Query data via gRPC endpoint (experimental)

Jaeger offers an alternative method for querying data that uses their gRPC service over HTTP. For detailed information about the API and setup requirements, refer to the [Jaeger API documentation](https://www.jaegertracing.io/docs/2.12/architecture/apis/#query-json-over-http).

The following queries are supported through the gRPC endpoint:

- Service search
- Operation search
- Trace ID search

To enable gRPC querying for Jaeger within Grafana, enable the `jaegerEnableGrpcEndpoint` feature flag. This feature is experimental. Grafana Cloud customers should contact support to request access and provide feedback on this feature.

## Use template variables

The Jaeger query editor supports [Grafana template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) for creating dynamic, reusable dashboards. You can use template variables in the following query fields:

- **Service Name**
- **Operation Name**
- **Trace ID**
- **Tags**
- **Min Duration**
- **Max Duration**

Use standard Grafana variable syntax such as `$variable` or `${variable}` in these fields. For example, set **Service Name** to `$service` to let dashboard users select the service from a drop-down list.

# Troubleshoot Jaeger data source issues

This document provides solutions to common issues you may encounter when configuring or using the Jaeger data source. For configuration instructions, refer to [Configure the Jaeger data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/jaeger/configure/).

## Connection errors

These errors occur when Grafana can't reach the Jaeger instance.

### "Connection refused" or timeout errors

**Symptoms:**

- **Save & test** fails with a connection error
- Queries fail with network errors
- Service and operation drop-downs don't load

**Possible causes and solutions:**

| Cause                    | Solution                                                                                                                                                                                      |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Jaeger isn't running     | Verify Jaeger is running and accessible at the configured URL.                                                                                                                                |
| Incorrect URL            | Check the **URL** setting in the data source configuration. The default Jaeger query endpoint is `http://localhost:16686`.                                                                    |
| Firewall blocking access | Ensure firewall rules allow traffic from the Grafana server to the Jaeger endpoint.                                                                                                           |
| Network segmentation     | For Grafana Cloud accessing private Jaeger instances, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). |

### "request failed: 404 Not Found"

**Symptoms:**

- **Save & test** returns a 404 error
- Queries fail with "request failed" messages

**Solutions:**

1. Verify the Jaeger URL doesn't include a trailing path such as `/api`. The correct URL is the base endpoint, for example `http://localhost:16686`.
1. Confirm the Jaeger query service is running and listening on the expected port.
1. If using a reverse proxy, verify the proxy forwards requests to the correct Jaeger endpoint.

### TLS errors

**Symptoms:**

- Certificate validation errors during **Save & test**
- Errors mentioning `x509` or certificate verification

**Solutions:**

1. Verify your TLS certificates are valid and not expired.
1. Ensure the CA certificate is configured in the data source **Additional settings** if using a private CA.
1. For testing, you can temporarily toggle on **Skip TLS verify** in the data source settings. Don't use this in production.
1. Verify the **Server name** setting matches the certificate's Common Name or Subject Alternative Name.

### Timeout errors

**Symptoms:**

- Queries take a long time then fail
- Errors mention timeout

**Solutions:**

1. Increase the timeout value in **Additional settings** > **Timeout**.
1. Verify network latency between Grafana and Jaeger is acceptable.
1. Check that the Jaeger query service isn't overloaded.

## Authentication errors

These errors occur when credentials are invalid or missing.

### "401 Unauthorized" or "403 Forbidden"

**Symptoms:**

- **Save & test** fails with authorization errors
- Queries return access denied messages
- Services and operations don't load in drop-downs

**Possible causes and solutions:**

| Cause                    | Solution                                                                                         |
| ------------------------ | ------------------------------------------------------------------------------------------------ |
| Invalid credentials      | Verify the **User** and **Password** values in the data source configuration.                    |
| Missing authentication   | If Jaeger requires authentication, toggle on the appropriate auth method.                        |
| Incorrect custom headers | Verify custom header names and values in **Additional settings**.                                |
| OAuth forwarding issues  | If using **Forward OAuth identity**, verify the upstream OAuth provider is configured correctly. |

## Query errors

These errors occur when executing queries against Jaeger.

### "traceID is empty"

**Symptoms:**

- The query fails with "traceID is empty" when using the TraceID query type

**Solutions:**

1. Enter a valid trace ID in the **Trace ID** field.
1. Verify the trace ID is a valid hexadecimal string.
1. If using a template variable, verify the variable resolves to a non-empty value.

### No data in search results

**Symptoms:**

- Search query returns no traces
- The trace list is empty

**Possible causes and solutions:**

| Cause                              | Solution                                                                                                       |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| Time range doesn't contain data    | Expand the dashboard or Explore time range. Verify traces exist in Jaeger for the selected period.             |
| Service or operation doesn't exist | Verify the selected service and operation names match what's in Jaeger.                                        |
| Tags syntax error                  | Ensure tags use [`logfmt`](https://brandur.org/logfmt) format: `error=true db.statement="select * from User"`. |
| Duration format error              | Use supported duration formats: `1.2s`, `100ms`, `500us`.                                                      |
| Limit set too low                  | Increase the **Limit** value or leave it empty for the default.                                                |

### "The JSON file uploaded is not in a valid Jaeger format"

**Symptoms:**

- Importing a trace file fails with a format error

**Solutions:**

1. Verify the JSON file follows the Jaeger trace format with a `data` array at the root level.
1. Ensure the file contains at least one trace object in the `data` array.
1. Validate the JSON syntax using a JSON linter.

Expected format:

```json
{
  "data": [
    {
      "traceID": "<TRACE_ID>",
      "spans": [...],
      "processes": {...}
    }
  ]
}
```

## Dependency graph errors

These errors relate to the dependency graph query type.

### Empty dependency graph

**Symptoms:**

- The dependency graph query returns no nodes or edges
- The Node Graph panel is empty

**Possible causes and solutions:**

| Cause                  | Solution                                                                                                                                                               |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No dependency data     | Verify Jaeger is collecting and processing service dependency information.                                                                                             |
| Time range too narrow  | Expand the dashboard time range. Dependencies are calculated over the selected period.                                                                                 |
| Jaeger storage backend | Some Jaeger storage backends may not support dependency queries. Refer to the [Jaeger documentation](https://www.jaegertracing.io/docs/) for storage-specific details. |

## gRPC endpoint errors

These errors occur when using the gRPC query endpoint.

### gRPC queries fail

**Symptoms:**

- Queries don't use the gRPC endpoint
- Search, service, and operation queries still use the REST API

**Solutions:**

1. Verify the `jaegerEnableGrpcEndpoint` feature flag is enabled in Grafana. This feature is experimental.
1. Grafana Cloud customers should contact support to request access.
1. Note that search and dependency graph queries currently use the REST endpoint even when the feature flag is enabled. Only service search, operation search, and trace ID queries use gRPC.

## Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Reproduce the issue.
1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location).
1. Look for Jaeger-specific entries that include request URLs, response status codes, and error details.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you've tried the solutions in this document and still encounter issues:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs related to the Jaeger data source.
1. Refer to the [Jaeger documentation](https://www.jaegertracing.io/docs/) for service-specific guidance.
1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Advanced user.
1. When reporting issues, include:
   - Grafana version
   - Jaeger version
   - Error messages (redact sensitive information)
   - Steps to reproduce
   - Relevant data source configuration (redact credentials)

# Loki alerting

The Loki data source supports [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/). You can alert on your logs so Grafana notifies you when a LogQL metric query crosses a threshold, for example when the rate of error logs spikes.

Loki works with two kinds of alert rules:

- **Grafana-managed alert rules:** Grafana evaluates the rule against a LogQL metric query. This is the most common approach and works with any Loki data source.
- **Data source-managed alert rules:** The Loki ruler stores and evaluates the rule. Use these when you want alert rules to live alongside your Loki deployment. Enable **Manage alert rules in Alerting UI** on the data source to create and edit them from Grafana.

## Before you begin

Before you create alert rules with Loki data, ensure you have:

- A [configured Loki data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/).
- Permissions to create alert rules in Grafana.
- Familiarity with [Grafana Alerting concepts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/).
- For data source-managed rules, a Loki deployment with the ruler enabled, and the **Manage alert rules in Alerting UI** setting turned on in the data source configuration. To route the resulting alerts, add a separate [Alertmanager data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/alertmanager/).

## Supported queries for alerting

Alert rules require queries that return numeric data that Grafana can evaluate against a threshold. Wrap a Loki log query in a LogQL metric function, such as `rate` or `count_over_time`, to produce numeric time series.

{{< admonition type="note" >}}
Use a metric query for alerting, not a plain log query. A log query returns log lines, which Grafana can't evaluate against a threshold.
{{< /admonition >}}

The following queries are common starting points for alerting. Replace the label matchers with values from your own logs.

Rate of error lines per second:

```logql
sum(rate({app="my-app"} |= `error` [5m]))
```

Error rate broken down by status code, to alert per endpoint or service:

```logql
sum by (status) (rate({app="nginx"} | logfmt | status=~`5..` [5m]))
```

Count of slow requests, for a latency alert:

```logql
sum(count_over_time({app="my-app"} | logfmt | duration > 1s [5m]))
```

Detect a service that stopped logging by alerting when the count drops to zero:

```logql
sum(count_over_time({app="my-app"} [10m]))
```

## Create a Grafana-managed alert rule

To create a Grafana-managed alert rule using Loki data:

1. Go to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for your alert rule.
1. In the **Define query and alert condition** section:
   - Select your Loki data source.
   - Enter a LogQL metric query.
   - Add a **Reduce** expression if your query returns multiple series.
   - Add a **Threshold** expression to define the alert condition.
1. Configure the **Set evaluation behavior** section:
   - Select or create a folder and evaluation group.
   - Set the evaluation interval and pending period.
1. Add labels and annotations to provide context for notifications.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example: alert on error log rate

This example fires when the rate of log lines that contain `error` exceeds a threshold:

1. Create a new alert rule.
1. Configure the query, replacing the label matcher with your own:

   ```logql
   sum(rate({app="my-app"} |= `error` [5m]))
   ```

1. Add expressions:
   - **Reduce**: Last, to get the most recent value.
   - **Threshold**: Is above `10`.
1. Set evaluation to run every 1 minute with a 5-minute pending period.
1. Save the rule.

## Manage data source-managed alert rules

When you turn on **Manage alert rules in Alerting UI** for the Loki data source, you can create and edit alert and recording rules that the Loki ruler stores and evaluates. These rules appear in the Grafana Alerting UI alongside your Grafana-managed rules, grouped under the Loki data source.

The Loki data source configuration doesn't include Alertmanager routing. To route the resulting alerts, add a separate [Alertmanager data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/alertmanager/). For more information, refer to [Configure the Loki data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/#alerting) and [Data source-managed alert rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-data-source-managed-rule/).

## Best practices

Follow these recommendations to create reliable alerts with Loki data.

### Test queries before alerting

Verify your query returns numeric data before you create an alert:

1. Go to **Explore**.
1. Select your Loki data source.
1. Run the LogQL metric query you plan to use for alerting.
1. Confirm the result is numeric and suitable for threshold evaluation.

### Reduce a time series to a single value

A Grafana-managed rule evaluates a single number against the threshold. A `range` query returns a time series with many points, which the rule can't evaluate directly. Either set the query **Type** to `instant`, or add a **Reduce** expression, such as **Last** or **Mean**, between the query and the **Threshold** expression. Without one of these, the rule can evaluate incorrectly.

### Set a pending period to avoid false positives

A transient condition, such as a brief network blip or a momentary spike in errors, can cross the threshold for a single evaluation. Set a pending period so the rule fires only after the condition holds for a sustained duration. A pending period of a few evaluation intervals reduces false positives from short-lived conditions.

### Handle no data versus zero results

A LogQL metric query returns _no series_ when no log lines match, rather than returning a value of `0`. Grafana evaluates a rule that receives no series as **No Data**, not **Normal**, which can cause an alert to enter the No Data state instead of resolving.

To avoid unexpected No Data states:

- Apply the threshold in the alert condition rather than writing a query so restrictive that it returns nothing. For example, query the error rate and compare it in a **Threshold** expression, instead of writing a query that only returns lines above the threshold.
- Set the No Data behavior under **Configure no data and error handling** to the state you want. The options are **No Data**, **Alerting**, **Normal**, and **Keep Last State**. Choose **Normal** if the absence of matching logs isn't a problem.

{{< admonition type="note" >}}
Editing and saving an alert rule resets its state to **Normal**. The rule re-enters **Pending** or **Alerting** on the next evaluation if the condition is still met. This is standard [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/) behavior and isn't specific to the Loki data source.
{{< /admonition >}}

## Troubleshoot alerting

If your Loki alerts don't work as expected, use the following sections to diagnose common issues.

### Alerts don't fire

- Confirm the query is a LogQL metric query that returns numeric data in Explore.
- Ensure the evaluation interval allows enough time for data to be available.
- Review the alert rule's health and any error messages in the Alerting UI.

### Alert enters No Data unexpectedly

- The query returned no series because no log lines matched. A LogQL metric query returns no series rather than `0` when nothing matches. Set the No Data behavior under **Configure no data and error handling**, or restructure the query so it returns a value that you compare with a threshold.
- Confirm logs exist for the query and time range in Explore.

### Alert fires on brief spikes

- Add or increase the pending period so the rule fires only when the condition holds over several evaluations.
- Widen the range in the metric query, for example from `[1m]` to `[5m]`, so short-lived spikes have less effect.

### Data source-managed rules don't appear

- Confirm **Manage alert rules in Alerting UI** is enabled in the Loki data source configuration.
- Verify the Loki ruler is enabled and reachable from Grafana.

For more help, refer to [Troubleshoot Loki issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/troubleshooting/).

## Related resources

- [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/)
- [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/)
- [Loki query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/query-editor/)

# Loki annotations

[Annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/) overlay rich event information on top of graphs. With the Loki data source, annotations are built from log queries: each log line that a LogQL query returns within the dashboard time range becomes an annotation, so you can correlate log events with the rest of your dashboard data.

## Before you begin

Before you create Loki annotations, ensure you have:

- A [configured Loki data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/).
- A [LogQL](https://grafana.com/docs/loki/latest/logql/) log query that returns the events you want to annotate.

## How Loki annotations work

Unlike data sources that build annotations from a separate query language, Loki annotations use the same LogQL log queries you use elsewhere. Grafana runs your log query for the dashboard time range and turns each returned log line into an annotation. By default, Grafana uses the log content as the annotation text and the log stream labels as tags, so you don't need to create any additional mapping.

You can only use log queries as a source for annotations. Metric queries aren't supported.

## Create an annotation query

To add a Loki annotation to a dashboard:

1. Open the dashboard where you want to add annotations.
1. Click **Edit**, then click **Settings** in the top navigation.
1. Select the **Annotations** tab.
1. Click **Add annotation query**.
1. Enter a **Name** for the annotation, for example, `Deploy events`.
1. Select your **Loki** data source.
1. Enter a LogQL log query and configure the optional fields described in the following table.
1. Click **Save dashboard**.

The annotation query editor provides the following fields:

| Field     | Description                                                                                                                                                       |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Query** | The LogQL log query that selects the log lines to annotate.                                                                                                       |
| **Title** | _Optional._ A literal name or a template for the annotation title. Reference labels with `{{label}}`, for example `{{level}}`.                                    |
| **Tags**  | _Optional._ A comma-separated list of label keys to use as annotation tags, for example `app,level`. When empty, Grafana uses the log stream labels as tags.      |
| **Text**  | _Optional._ A literal string or a template for the annotation text. Reference labels with `{{label}}`. When empty, Grafana uses the log line content as the text. |

Template fields resolve against the labels on each returned log line, including parsed fields. To reference a value that isn't an indexed label, add a parser such as `logfmt` or `json` to your query so the field is available.

### Example

To annotate deployment events, configure the annotation query with the following values:

| Field     | Value                               |
| --------- | ----------------------------------- |
| **Query** | `{app="deploy-bot"} \|= "deployed"` |
| **Title** | `{{app}} deployed`                  |
| **Tags**  | `app,env`                           |
| **Text**  | `{{message}}`                       |

Each matching log line becomes an annotation titled with the deploying app, tagged with the `app` and `env` labels, and described with the parsed `message` field. Because the **Text** template references a parsed field, the query needs a parser, for example:

```logql
{app="deploy-bot"} |= `deployed` | logfmt
```

## Troubleshoot annotations

If annotations don't appear as expected, try the following solutions.

### Annotations don't appear

- Verify the log query returns results in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/) for the same time range. Annotations come from returned log lines, so a query that returns no logs produces no annotations.
- Confirm the query is a log query, not a metric query.
- Widen the dashboard time range to include the log events you expect.

### Too many annotations appear

- Add label filters or a line filter to your LogQL query to narrow the returned log lines.
- Reduce the maximum number of lines the query returns.

## Related resources

- [Annotate visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/)
- [Loki query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/query-editor/)
- [Configure the Loki data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/)

# Configure the Loki data source

This document explains how to configure the Loki data source and describes the available configuration options. For general information on adding and managing data sources, refer to [Data source management](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/).

Grafana includes built-in support for Loki, so you don't need to install a plugin.

## Before you begin

Before you configure the data source, ensure you have:

- **Grafana permissions:** The `Organization administrator` role to add and configure data sources. Administrators can also configure the data source with [YAML provisioning](#provision-the-data-source).
- **Loki server URL:** The address of your Loki server, including the port. Loki listens on port `3100` by default.
- **Authentication details:** Any credentials or certificates your Loki server requires, such as a basic auth user and password or TLS certificates.

{{< admonition type="note" >}}
Use TLS (Transport Layer Security) for an additional layer of security when working with Loki. For information on setting up TLS encryption with Loki, refer to [Grafana Loki configuration parameters](https://grafana.com/docs/loki/latest/configure/).
{{< /admonition >}}

## Add the Loki data source

To add the Loki data source:

1. Click **Connections** in the left-side menu.
1. Click **Add new connection**.
1. Type `Loki` in the search bar.
1. Select the **Loki** data source.
1. Click **Add new data source** in the upper right.

Grafana takes you to the **Settings** tab, where you set up your Loki configuration.

## Configure the data source in the UI

The following sections describe the configuration options available for the Loki data source.

The first options set the name of your connection:

| Setting     | Description                                                                                                            |
| ----------- | ---------------------------------------------------------------------------------------------------------------------- |
| **Name**    | The data source name. This is how you refer to the data source in panels and queries. Examples: `loki-1`, `loki_logs`. |
| **Default** | Toggle to make this the default data source. Grafana selects it by default in new panels and queries.                  |

### Connection

Configure how Grafana connects to your Loki server.

| Setting | Description                                                                                                                                                                                |
| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **URL** | The base URL of your Loki server. If Loki runs locally, use `http://localhost:3100`. If Loki runs on a networked server, use its URL and port, for example `http://loki.example.com:3100`. |

{{< admonition type="note" >}}
Enter only the base URL. Don't append API paths such as `/loki/api/v1/push`. That endpoint sends logs to Loki with an agent like Grafana Alloy; it isn't used to query Loki from the data source.

If your Grafana instance runs on Grafana Cloud, a `localhost` or private network address refers to Grafana servers rather than your network, so it can't reach a self-hosted Loki. Use [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) to query Loki on a private network.
{{< /admonition >}}

### Authentication

Configure how Grafana authenticates with your Loki server. Select an authentication method and provide any required credentials.

| Setting                    | Description                                                                                                                                     |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| **Basic authentication**   | The most common authentication method. Enter the user name and password for your Loki server.                                                   |
| **Forward OAuth Identity** | Toggle on to forward the OAuth access token, and the OIDC ID token if available, of the user querying the data source.                          |
| **With credentials**       | Toggle on to send credentials such as cookies or authentication headers with cross-site requests.                                               |
| **TLS Client Auth**        | Toggle on to authenticate with a client certificate. Provide the **Server name**, **Client cert**, and **Client key**.                          |
| **With CA cert**           | Toggle on to verify a self-signed TLS certificate. Follow the instructions from your certificate authority (CA) to obtain the certificate file. |
| **Skip TLS verify**        | Toggle on to bypass TLS certificate verification. Skipping verification isn't recommended unless absolutely necessary or for testing.           |

{{< admonition type="note" >}}
For Grafana Cloud-hosted Loki, use **Basic authentication** with your Grafana Cloud user ID as the user name and a Cloud Access Policy token as the password. The token's [access policy](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) must include the `logs:read` scope. Create tokens in the Grafana Cloud Portal. A token's value is shown only once, so copy it when you create it.
{{< /admonition >}}

#### Custom HTTP headers

Add custom HTTP headers to pass values that your Loki instance requires.

| Setting    | Description                    |
| ---------- | ------------------------------ |
| **Header** | The name of the custom header. |
| **Value**  | The value of the header.       |

For a multi-tenant Loki, one configured with `auth_enabled: true`, add the `X-Scope-OrgID` header with your tenant ID so Loki knows which tenant to query. Without it, queries against a multi-tenant Loki fail with an authentication error or return no data.

### Additional settings

The **Additional settings** section is collapsible and contains optional settings that give you more control over the data source. It's open by default and includes advanced HTTP settings, alerting, query, and derived field options.

#### Advanced HTTP settings

Configure additional HTTP behavior for requests to Loki.

| Setting             | Description                                                                                                                   |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| **Allowed cookies** | Specify cookies by name that should be forwarded to the data source. By default, the Grafana proxy deletes forwarded cookies. |
| **Timeout**         | The HTTP request timeout, in seconds. There's no default, so set a value that suits your queries.                             |

#### Secure Socks Proxy

The **Secure Socks Proxy** settings appear only when the secure SOCKS data source proxy is enabled in the Grafana configuration file. When enabled, you can route data source requests through a secure SOCKS proxy. For more information, refer to [Configure a data source connection proxy](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/proxy/).

#### Alerting

Configure how the data source works with [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/).

| Setting                               | Description                                                                                                                                                                                                      |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Manage alert rules in Alerting UI** | Toggle on to manage data source-managed alert rules for the Loki data source in the Grafana Alerting UI. These rules are stored and evaluated by the Loki ruler. The default follows your Grafana configuration. |

To manage other alerting resources, such as the alerts these rules generate, add an [Alertmanager data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/alertmanager/). For more information on alerting with Loki, refer to [Loki alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/alerting/).

#### Queries

Configure query behavior for the data source.

<!-- vale Grafana.WordList = NO -->
<!-- vale Grafana.Spelling = NO -->

| Setting           | Description                                                                                                                                                                                                       |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Maximum lines** | Sets the maximum number of log lines Loki returns. Increase the limit for a larger result set for ad hoc analysis. Decrease the limit if your browser is slow when displaying log results. The default is `1000`. |

<!-- vale Grafana.Spelling = YES -->
<!-- vale Grafana.WordList = YES -->

#### Derived fields

Use derived fields to extract new fields from your logs and create a link from the value of the field.

For example, you can link to your tracing backend directly from your logs, or link to a user profile page when a log line contains a corresponding `userId`. These links appear in the [log details](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/logs-integration/#labels-and-detected-fields).

You can add multiple derived fields.

{{< admonition type="note" >}}
If you use Grafana Cloud, you can request modifications to this feature by opening a support ticket from the Grafana Cloud Portal.
{{< /admonition >}}

Each derived field has the following options:

| Setting                      | Description                                                                                                                                                                                                                                |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Name**                     | The field name. Displayed as a label in the log details.                                                                                                                                                                                   |
| **Type**                     | The type of the derived field. Choose **Regex in log line** to parse a value from the log content, or **Label** to match a label key.                                                                                                      |
| **Regex**                    | When the type is **Regex in log line**, a regular expression that parses part of the log message. Include a capture group; Grafana uses the first captured group as the field value.                                                       |
| **Label**                    | When the type is **Label**, the input matches as a regular expression against label keys, so a pattern like `trace[_]?id` matches variations such as `traceid` and `trace_id`. Matches any label: indexed, parsed, or structured metadata. |
| **URL/query**                | The full link URL for external links, or a query for the target data source for internal links. Interpolate the field value with the `${__value.raw}` macro.                                                                               |
| **URL Label**                | _Optional._ A custom display label for the link. Overrides the default label, which is the full external URL or the name of the linked internal data source.                                                                               |
| **Internal link**            | Toggle on to define an internal link to a tracing data source. Select the target data source from the selector.                                                                                                                            |
| **Open in new tab**          | Toggle on to open the link in a new tab or window.                                                                                                                                                                                         |
| **Show example log message** | Paste an example log line to test the regular expression of your derived fields.                                                                                                                                                           |

{{< admonition type="caution" >}}
Using complex regular expressions in either type can affect browser performance when processing large volumes of logs. Use simpler patterns when possible.
{{< /admonition >}}

{{< admonition type="note" >}}
A derived field produces a single value per log line, so you can't combine multiple labels or capture groups into one field or link.

For an internal link, the derived field only builds a link to the target data source using the extracted value. The trace must already be ingested into the target data source, such as Tempo, for the link to resolve. If a matching trace doesn't exist, the link opens the target data source but returns no trace.
{{< /admonition >}}

##### Troubleshoot interpolation

Use the debug section to see what your fields extract and how the URL interpolates. Click **Show example log message** to display a text area where you can enter a log message.

{{< figure src="/static/img/docs/v75/loki_derived_fields_settings.png" class="docs-image--no-shadow" max-width="800px" caption="Screenshot of the derived fields debug section" >}}

The new field with the link appears in the log details:

{{< figure src="/static/img/docs/explore/data-link-9-4.png" max-width="800px" caption="Data link in Explore" >}}

## Verify the connection

Click **Save & test** at the bottom of the settings. Grafana attempts to connect to your Loki server and query its labels. When the test succeeds, Grafana displays a success message confirming the data source is working. If the test fails, review the error message and refer to [Troubleshoot Loki issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/troubleshooting/).

{{< admonition type="note" >}}
To troubleshoot configuration and other issues, check the log file located at `/var/log/grafana/grafana.log` on Unix systems, or in `<grafana_install_dir>/data/log` on other platforms and manual installations.
{{< /admonition >}}

## Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

### Provisioning examples

The following example provisions a basic Loki data source:

```yaml
apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://localhost:3100
    jsonData:
      timeout: 60
      maxLines: 1000
```

**Using basic authorization and a derived field:**

You must escape the dollar (`$`) character in YAML values because it can be used to interpolate environment variables:

```yaml
apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://localhost:3100
    basicAuth: true
    basicAuthUser: my_user
    jsonData:
      maxLines: 1000
      derivedFields:
        # Field with internal link pointing to data source in Grafana.
        # datasourceUid value can be anything, but it should be unique across all defined data source uids.
        - datasourceUid: my_jaeger_uid
          matcherRegex: "traceID=(\\w+)"
          name: TraceID
          # url will be interpreted as query for the datasource
          url: '$${__value.raw}'
          # optional for URL Label to set a custom display label for the link.
          urlDisplayLabel: 'View Trace'

        # Field with external link.
        - matcherRegex: "traceID=(\\w+)"
          name: TraceID
          url: 'http://localhost:16686/trace/$${__value.raw}'
    secureJsonData:
      basicAuthPassword: test_password
```

**Using a Jaeger data source:**

In this example, the Jaeger data source's `uid` value should match the Loki data source's `datasourceUid` value.

```yaml
datasources:
  - name: Jaeger
    type: jaeger
    url: http://jaeger-tracing-query:16686/
    access: proxy
    # UID should match the datasourceUid in derivedFields.
    uid: my_jaeger_uid
```

## Configure with Terraform

You can configure the Loki data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

For more information about provisioning resources with Terraform, refer to [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/).

The following example provisions a Loki data source with a maximum line limit:

```hcl
resource "grafana_data_source" "loki" {
  name = "Loki"
  type = "loki"
  url  = "http://localhost:3100"

  json_data_encoded = jsonencode({
    maxLines = 1000
  })
}
```

The following example uses basic authentication and defines a derived field that links to a Jaeger data source:

```hcl
resource "grafana_data_source" "loki" {
  name           = "Loki"
  type           = "loki"
  url            = "http://localhost:3100"
  basic_auth_enabled  = true
  basic_auth_username = "my_user"

  json_data_encoded = jsonencode({
    maxLines = 1000
    derivedFields = [
      {
        datasourceUid   = "my_jaeger_uid"
        matcherRegex    = "traceID=(\\w+)"
        name            = "TraceID"
        url             = "$${__value.raw}"
        urlDisplayLabel = "View Trace"
      }
    ]
  })

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = "<LOKI_PASSWORD>"
  })
}
```

Replace the placeholders with your own values:

- `<LOKI_PASSWORD>`: The password for the basic authentication user.

For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source).

# Loki query editor

The Loki data source's query editor helps you create [log](#create-a-log-query) and [metric](#create-a-metric-query) queries that use the Loki query language, [LogQL](https://grafana.com/docs/loki/latest/logql/).

For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/).

## Choose a query editing mode

The Loki query editor has two modes:

- [Builder mode](#builder-mode), which provides a visual query designer.
- [Code mode](#code-mode), which provides a feature-rich editor for writing queries.

To switch between the editor modes, select the corresponding **Builder** and **Code** tabs.

To run a query, select **Run queries** located at the top of the editor.

{{< admonition type="note" >}}
To run Loki queries in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/), select **Run query**.
{{< /admonition >}}

Each mode is synchronized, so you can switch between them without losing your work, although there are some limitations. Builder mode doesn't support some complex queries.
When you switch from Code mode to Builder mode with such a query, the editor displays a warning message that explains how you might lose parts of the query if you continue.
You can then decide whether you still want to switch to Builder mode.

You can also augment queries by using [template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/template-variables/).

To see example LogQL log and metric queries, explore this demo dashboard on Grafana Play. Open any panel's editor to view its query:

{{< docs/play title="Loki NGINX Service Mesh - Pattern version" url="https://play.grafana.org/d/-lPcmrp7z/" >}}

## Toolbar elements

The query editor toolbar contains the following elements:

- **Kick start your query** - Click to see a list of sample queries that help you quickly start creating LogQL queries. These include log query starters and metric query starters. Click the arrow next to each to see the available query options, then continue to complete your query.

- **Label browser** - Use the Loki label browser to navigate through your labels and values, and build queries.

To navigate Loki and build a query:

1. Choose labels to locate.
1. Search for the values of your selected labels.

   The search field supports fuzzy search, and the label browser also supports faceting to list only possible label combinations.

1. Select the **Show logs** button to display log lines based on the selected labels, or select the **Show logs rate** button to show the rate based on metrics such as requests per second. Additionally, you can validate the selector by clicking the **Validate selector** button. Click **Clear** to start from the beginning.

{{< figure src="/static/img/docs/explore/Loki_label_browser.png" class="docs-image--no-shadow" max-width="800px" caption="The Loki label browser" >}}

- **Explain query** - Toggle to display a step-by-step explanation of all query components and operations.

{{< figure src="/static/img/docs/prometheus/explain-results.png" max-width="500px" class="docs-image--no-shadow" caption="Explain results" >}}

- **Builder/Code** - Click the corresponding **Builder** or **Code** tab on the toolbar to select an editor mode.

## Builder mode

Builder mode helps you build queries using a visual interface without needing to manually enter LogQL. This option is best for users who have limited or no previous experience working with Loki and LogQL.

### Label filters

Select labels and their values from the dropdown list.
When you select a label, Grafana retrieves available values from the server.

Use the `+` button to add a label and the `x` button to remove a label. You can add multiple labels.

Select comparison operators from the following options:

- `=` - equal to
- `!=` - is not equal
- `=~` - matches a regular expression
- `!~` - doesn't match a regular expression

Select values by using the dropdown, which displays all possible values based on the label selected.

### Operations

Select the `+ Operations` button to add operations to your query.
The query editor groups operations into related sections, and you can type while the operations dropdown is open to search and filter the list.

The query editor displays a query's operations as boxes in the operations section.
Each operation's header displays its name, and additional action buttons appear when you hover your cursor over the header:

| Button                                                                                                                  | Action                                                            |
| ----------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| {{< figure src="/static/img/docs/v95/loki_operation_replace.png" class="docs-image--no-shadow" max-width="30px" >}}     | Replaces the operation with different operation of the same type. |
| {{< figure src="/static/img/docs/v95/loki_operation_description.png" class="docs-image--no-shadow" max-width="30px" >}} | Opens the operation's description tooltip.                        |
| {{< figure src="/static/img/docs/v95/loki_operation_remove.png" class="docs-image--no-shadow" max-width="30px" >}}      | Removes the operation.                                            |

The query editor groups operations into the following sections:

- Aggregations - see [Built-in aggregation operators](https://grafana.com/docs/loki/latest/logql/metric_queries/#built-in-aggregation-operators)
- Range functions - see [Range Vector aggregation](https://grafana.com/docs/loki/latest/logql/metric_queries/#range-vector-aggregation)
- Formats - see [Log queries](https://grafana.com/docs/loki/latest/logql/log_queries/#log-queries)
- Binary operations - see [Binary operators](https://grafana.com/docs/loki/latest/logql/#binary-operators)
- Label filters - see [Label filter expression](https://grafana.com/docs/loki/latest/logql/log_queries/#label-filter-expression)
- Line filters - see [Line filter expression](https://grafana.com/docs/loki/latest/logql/log_queries/#label-filter-expression)

Some operations make sense only when used in a specific order. If adding an operation would result in a nonsensical query, the query editor adds the operation to the correct place.
To reorder operations manually, drag the operation box by its name and drop it into the desired place. For more information, refer to [Order of operations](https://grafana.com/docs/loki/latest/logql/#order-of-operations).

### Hints

In some cases, the query editor can detect which operations are most appropriate for a selected log stream. When it does, it shows a hint next to the **+ Operations** button. Click the hint to add the operations to your query.

### Builder mode example

The following steps build a metric query that counts the rate of error lines for the `payments` app, without writing any LogQL:

1. Under **Label filters**, select the `app` label and the value `payments`.
1. Select **+ Operations**, then from **Line filters** add a **Line contains** operation and enter `error`.
1. Select **+ Operations** again, then from **Range functions** add **Rate** with a range of `[5m]`.
1. From **Aggregations**, add **Sum** to combine the results into a single series.

The builder produces the equivalent LogQL query:

```logql
sum(rate({app="payments"} |= `error` [5m]))
```

Switch to [Code mode](#code-mode) at any time to view or refine the generated query.

## Code mode

In **Code mode**, you can write complex queries using a text editor with autocompletion, syntax highlighting, and query validation.
It also provides the [label browser](#toolbar-elements) to further help you write queries.

For more information about the Loki query language, refer to the [Loki documentation](https://grafana.com/docs/loki/latest/logql/).

### Use autocompletion

Code mode's autocompletion feature works automatically while typing.

The query editor can autocomplete static functions, aggregations, and keywords, and also dynamic items like labels.
The autocompletion dropdown includes documentation for the suggested items where available.

## Options

The following options are the same for both **Builder** and **Code** mode:

- **Legend** - Controls the time series name, using a name or pattern. For example, `{{hostname}}` is replaced with the label value for the label `hostname`.

- **Type** - Selects the query type to run. The `instant` type queries against a single point in time, using the **To** time from the time range. The `range` type queries over the selected range of time.

- **Line limit** - Defines the upper limit for the number of log lines returned by a query. The default is `1000`.

- **Direction** - Determines the search order. **Backward** is a backward search starting at the end of the time range. **Forward** is a forward search starting at the beginning of the time range. The default is **Backward**.

- **Step** - Sets the step parameter of Loki metric queries. The default value equals the value of the `$__auto` variable, which Grafana calculates using the time range and the width of the graph in pixels.

## Create a log query

Loki log queries return the contents of the log lines.
You can query and display log data from Loki via [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/), and with the [Logs panel](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/logs/) in dashboards.

To display the results of a log query, select the Loki data source, then enter a LogQL query.

For more information about log queries and LogQL, refer to the [Loki log queries documentation](https://grafana.com/docs/loki/latest/logql/log_queries/).

### Log query examples

A log query has two parts: a stream selector that chooses log streams by label, and an optional pipeline that filters and parses the matching lines. Replace the label matchers with values from your own logs.

Return every log line from the `payments` app:

```logql
{app="payments"}
```

Keep only lines that contain the word `error`, and drop health-check noise:

```logql
{app="payments"} |= `error` != `healthcheck`
```

Match lines with a regular expression, for example any 4xx or 5xx status:

```logql
{app="nginx"} |~ `status=(4|5)..`
```

Parse structured logs, then filter on an extracted field. This example parses `logfmt` logs and keeps requests slower than 500 ms:

```logql
{app="payments"} | logfmt | duration > 500ms
```

Parse JSON logs and keep only error-level entries:

```logql
{namespace="prod", app="checkout"} | json | level=`error`
```

**Use case: debug a failing service.** Combine a namespace and app selector with a parser and label filter to isolate the errors for one service during an incident:

```logql
{namespace="prod", app="checkout"} |= `error` | logfmt | status >= 500
```

### Show log context

In Explore, you can retrieve the context surrounding your log results by clicking the **Show context** button. You can then investigate the logs from the same log stream that came before and after the log message you're interested in.

The initial log context query is created from all labels defining the stream for the selected log line. You can use the log context query editor to widen the search by removing one or more of the label filters from the log stream. If you used a parser in your original query, you can refine your search by using extracted label filters.

You can also adjust the size of the time window that the log context loads, so you can expand or narrow the range of surrounding logs without leaving the context view.

To reduce the repetition of selecting and removing the same labels when examining multiple log context windows, Grafana stores your selected labels and applies them to each open context window. This lets you navigate through various log context windows without having to reapply your filters.

To reset filters and use the initial log context query, click the **Revert to initial query** button next to the query preview.

### Tail live logs

Loki supports live tailing of logs in real-time in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/).

Live tailing relies on two WebSocket connections: one between the browser and Grafana server, and another between the Grafana server and Loki server.

To start tailing logs click the **Live** button in the top right corner of the Explore view.
{{< figure src="/static/img/docs/v95/loki_tailing.png" class="docs-image--no-shadow" max-width="80px" >}}

#### Proxying examples

If you use reverse proxies, configure them accordingly to use live tailing:

**Using Apache2 for proxying between the browser and the Grafana server:**

```
ProxyPassMatch "^/(api/datasources/proxy/\d+/loki/api/v1/tail)" "ws://127.0.0.1:3000/$1"
```

**Using NGINX:**

This example provides a basic NGINX proxy configuration.
It assumes that the Grafana server is available at `http://localhost:3000/`, the Loki server is running locally without proxy, and your external site uses HTTPS.
If you also host Loki behind an NGINX proxy, repeat the following configuration for Loki.

In the `http` section of NGINX configuration, add the following map definition:

```
  map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
  }
```

In your `server` section, add the following configuration:

```
  location ~ /(api/datasources/proxy/\d+/loki/api/v1/tail) {
      proxy_pass          http://localhost:3000$request_uri;
      proxy_set_header    Host              $host;
      proxy_set_header    X-Real-IP         $remote_addr;
      proxy_set_header    X-Forwarded-for   $proxy_add_x_forwarded_for;
      proxy_set_header    X-Forwarded-Proto "https";
      proxy_set_header    Connection        $connection_upgrade;
      proxy_set_header    Upgrade           $http_upgrade;
  }

  location / {
      proxy_pass          http://localhost:3000/;
      proxy_set_header    Host              $host;
      proxy_set_header    X-Real-IP         $remote_addr;
      proxy_set_header    X-Forwarded-for   $proxy_add_x_forwarded_for;
      proxy_set_header    X-Forwarded-Proto "https";
  }
```

## Create a metric query

You can use LogQL to wrap a log query with functions that create metrics from your logs.

For more information about metric queries, refer to the [Loki metric queries documentation](https://grafana.com/docs/loki/latest/logql/metric_queries/).

### Metric query examples

Metric queries apply a range aggregation, such as `rate` or `count_over_time`, over a log query and an optional range like `[5m]`. You can then wrap the result in an aggregation operator, such as `sum` or `topk`, to group or rank the series.

Count the rate of error lines per second across the `payments` app:

```logql
sum(rate({app="payments"} |= `error` [5m]))
```

Break the request rate down by status code:

```logql
sum by (status) (rate({app="nginx"} | logfmt [5m]))
```

Count how many timeouts occurred in the last hour:

```logql
count_over_time({app="payments"} |= `timeout` [1h])
```

Calculate the 95th percentile request duration per route. This example parses `logfmt` logs and unwraps the numeric `duration` field:

```logql
quantile_over_time(0.95, {app="payments"} | logfmt | unwrap duration [5m]) by (route)
```

**Use case: find the noisiest endpoints.** Rank the top 10 request paths by log volume over the last hour:

```logql
topk(10, sum by (path) (count_over_time({app="nginx"} | logfmt [1h])))
```

## Query splitting

To keep large queries responsive, Grafana automatically splits Loki queries that cover a long time range into a series of smaller sub-queries. Grafana runs the sub-queries and merges the results, and the query progresses incrementally so you see partial results as they arrive. Query splitting applies to both log and metric queries and requires no configuration.

## Structured metadata

Loki stores three types of labels: indexed labels, parsed fields, and structured metadata. The query editor and log details treat all three types as labels you can filter on and use in [derived fields](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/#derived-fields). For more information, refer to the [Loki structured metadata documentation](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/).

## Apply annotations

You can use Loki log queries as a source for dashboard annotations. For details, refer to [Loki annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/annotations/).

# Loki template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
Grafana refers to such variables as template variables.

For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/) documentation.

## Use query variables

Variables of the type _Query_ help you query Loki for lists of labels or label values.
The Loki data source provides a form to select the type of values expected for a given variable.

The form has these options:

| Query type   | Example label | Example stream selector | List returned                                                    |
| ------------ | ------------- | ----------------------- | ---------------------------------------------------------------- |
| Label names  | Not required  | Not required            | Label names.                                                     |
| Label values | `label`       |                         | Label values for `label`.                                        |
| Label values | `label`       | `log stream selector`   | Label values for `label` in the specified `log stream selector`. |

### Query variable example

To create a variable that lists the values of the `app` label:

1. Add a new variable of type _Query_ and select the Loki data source.
1. Set **Query type** to **Label values** and **Label** to `app`.
1. Optionally, enter a **Stream selector** such as `{namespace="$namespace"}` to return only the values that appear within another variable's selection. This creates a chained variable that updates when the parent variable changes.

Reference the variable in a query with its name, prefixed by `$`. For a single-value variable, use an exact match:

```logql
{app="$app"}
```

When a variable has **Multi-value** or **Include All** enabled, Grafana joins the selected values into a regular expression, such as `payments|checkout`. Use the regular expression match operator `=~` so the query matches any of the selected values:

```logql
{app=~"$app"}
```

## Format variable values

By default, the Loki data source formats variable values based on the variable's settings:

- A single-value variable is inserted as-is, without escaping.
- A **Multi-value** or **Include All** variable has each value escaped for use in a regular expression and joined with `|`, for example `payments|checkout`.

This default suits the regular expression match operator `=~`, but it can produce unexpected results when a custom or static value contains characters that Grafana escapes, or when you insert a value into a context that isn't a regular expression. To control the formatting, use a [format option](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/#advanced-variable-format-options) in the variable syntax:

| Syntax         | Result                                                                                                     |
| -------------- | ---------------------------------------------------------------------------------------------------------- |
| `${app:raw}`   | Inserts the value with no escaping. Use this for custom or static values that must pass through unchanged. |
| `${app:pipe}`  | Joins multiple values with `\|` without regular expression escaping.                                       |
| `${app:regex}` | Escapes the values for a regular expression and joins them with `\|`.                                      |

For example, to use a raw custom value in an exact-match selector:

```logql
{app="${app:raw}"}
```

For the complete list of format options, refer to [Advanced variable format options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/#advanced-variable-format-options).

<!-- vale Grafana.WordList = NO -->
<!-- vale Grafana.Spelling = NO -->

## Use ad hoc filters

Loki supports ad hoc filters. Use them to specify any number of key/value filters that Grafana applies automatically to all of your Loki queries, without editing each query.

For example, if you add an ad hoc filter for `level = error`, Grafana appends the matcher to the stream selector of every Loki query on the dashboard, so a query like `{app="payments"}` runs as `{app="payments", level="error"}`.

For more information, refer to [Add ad hoc filters](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#add-ad-hoc-filters).

<!-- vale Grafana.Spelling = YES -->
<!-- vale Grafana.WordList = YES -->

## Use the `$__auto` variable for Loki metric queries

Consider using the `$__auto` variable in your Loki metric queries, which will automatically be substituted with the [step value](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/query-editor/#options) for range queries, and with the selected time range's value (computed from the starting and ending times) for instant queries.

Use it as the range in a range aggregation:

```logql
sum(rate({app="payments"} |= `error` [$__auto]))
```

{{< admonition type="note" >}}
Prefer `$__auto` over `$__range` for the range in a metric query. The `$__range` variable resolves to the entire selected time range, so using it as a range vector, such as `[$__range]`, makes each data point aggregate over the whole window. This can scan far more data and look back further than you intend.
{{< /admonition >}}

For more information about variables, refer to [Global built-in variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#global-variables).

## Known limitations

Keep the following limitations in mind when you use template variables with the Loki data source:

- **Only indexed labels are available for label variable queries.** A _Query_ variable of type **Label values** returns the values of indexed stream labels. Values that exist only as parsed fields or [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) aren't available, because the variable query doesn't run a log pipeline to extract them.
- **Long time ranges can return incomplete values.** Label and label-value variable queries run over the dashboard's selected time range. Over a long range with high log volume, the query can time out or hit the limits configured in Loki and return an incomplete list of values. Narrow the time range, or add a stream selector to the variable query, so it scans less data.

## Label extraction and indexing in Loki

Labels are key to how Loki aggregates and queries logs. Labels are `key-value` pairs that add contextual information to log entries, and Loki uses them to organize, filter, and search log data efficiently.

### Label extraction

When it ingests logs, Loki extracts labels from log lines using regular expressions, so you can define custom patterns that match your log formats. For example, given a log line like the following:

```text
2023-07-25 12:34:56 INFO: Request from IP A.B.C.D to endpoint /api/data
```

You can define a regular expression that extracts the log level (`INFO`), IP address (`A.B.C.D`), and endpoint (`/api/data`) as labels. You can then use those labels to filter and aggregate log entries.

### Index labels

Loki indexes the extracted labels. The index maps labels to their log entries, so Loki can retrieve logs by label without scanning the entire dataset. For example, if a `job` label represents the services in your application, Loki indexes each job's logs separately, so you can query a single job quickly.

Combine Loki indexed labels with Grafana template variables to build dynamic queries. Use template variables to select and filter logs by labels such as job names, instance IDs, or severity levels, so you can explore and visualize your log data interactively.

# Troubleshoot Loki data source issues

This document provides solutions to common issues you might encounter when configuring or using the Loki data source. For configuration instructions, refer to [Configure the Loki data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/).

{{< admonition type="note" >}}
This guide covers the Loki data source in Grafana, such as connecting to Loki, running queries, and configuring features. It doesn't cover troubleshooting the Loki service itself. For help with a Loki deployment, including ingestion and query performance, refer to [Manage and debug errors](https://grafana.com/docs/loki/latest/operations/troubleshooting/) in the Loki documentation.
{{< /admonition >}}

## Connection errors

These errors occur when Grafana can't reach your Loki server.

### Save & test fails

**Symptoms:**

- The data source test fails or times out.
- Queries fail with network errors.

**Possible causes and solutions:**

| Cause                          | Solution                                                                                                                                                                                                                              |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Incorrect URL or port          | Verify the **URL** setting. Loki listens on port `3100` by default, for example `http://localhost:3100`.                                                                                                                              |
| API path in the URL            | Enter only the base Loki URL. Don't append API paths such as `/loki/api/v1/push`, which sends logs to Loki rather than querying it.                                                                                                   |
| Localhost URL on Grafana Cloud | On Grafana Cloud, a `localhost` or private address points to Grafana servers, not your network. Use [PDC](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) to reach a self-hosted Loki. |
| Missing tenant header          | For a multi-tenant Loki, one with `auth_enabled: true`, add the `X-Scope-OrgID` custom HTTP header with your tenant ID.                                                                                                               |
| Network or firewall block      | Confirm the Grafana server can reach the Loki server, and that firewall rules allow the connection.                                                                                                                                   |
| Authentication failure         | Verify the credentials, tokens, or TLS certificates in the data source configuration. For Grafana Cloud Loki, confirm the access policy token includes the `logs:read` scope.                                                         |
| TLS verification failure       | If Loki uses a self-signed certificate, add the CA certificate with **With CA cert**, or enable **Skip TLS verify** for testing.                                                                                                      |

## Query errors

These errors occur when running queries against Loki.

### No data or empty results

**Symptoms:**

- A query runs without error but returns no data.
- Panels show a "No data" message.

**Possible causes and solutions:**

| Cause                             | Solution                                                                                  |
| --------------------------------- | ----------------------------------------------------------------------------------------- |
| Time range doesn't contain logs   | Expand the dashboard time range or confirm logs exist in Loki for that period.            |
| Label selector matches no streams | Verify your stream selector labels and values with the label browser in the query editor. |
| Logs exceed retention             | Confirm the logs you expect are within the retention period configured in Loki.           |

### Query timeout or "too many outstanding requests"

**Symptoms:**

- A query runs for a long time and then fails.
- Errors mention timeouts, query limits, or too many outstanding requests.

**Solutions:**

1. Narrow the time range to reduce the volume of data scanned.
1. Add label filters or a line filter to reduce the result set.
1. Lower the **Maximum lines** setting on the data source or the line limit on the query.
1. Review the query and rate limits configured on your Loki deployment.

### "maximum of series reached for a single query"

**Symptoms:**

- A metric query fails with an error about the maximum number of series.

**Solutions:**

1. Add label matchers to reduce the number of streams the query touches.
1. Aggregate with `sum by (...)` to reduce cardinality.
1. Avoid high-cardinality labels in aggregations.

## Performance issues

These issues relate to slow queries or rate limits.

### Queries are slow or get rate limited

**Solutions:**

1. Narrow the time range and add label filters so queries scan less data.
1. Enable query caching, available in Grafana Enterprise and Grafana Cloud.
1. Review the query, ingestion, and rate limits on your Loki deployment.

## Template variable errors

These errors occur when using template variables with the data source.

### Variables return no values

**Solutions:**

1. Confirm the data source connection works by running **Save & test**.
1. Verify the label or stream selector in the variable query returns values in the query editor.
1. For cascading variables, confirm parent variables have valid selections.

## Derived field and log-to-trace link issues

These issues affect [derived fields](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/#derived-fields), including the links from logs to traces.

### The derived field or trace link doesn't appear

- Confirm the regular expression matches your log line format. A pattern that doesn't match produces no value and no link. Use **Show example log message** in the data source configuration to test the pattern against a real log line.
- For a **Label** type derived field, confirm a label key matches the pattern. The match runs against label keys, not values.
- Check that the field value is captured. For a **Regex in log line** type, Grafana uses the first capture group, so the expression must include one.

### The trace link opens but shows no trace

- The derived field only builds a link to the target data source using the extracted value; it doesn't create the trace. Confirm the trace is ingested into the target data source, such as Tempo, and that the extracted ID matches the trace ID stored there.
- Verify the extracted value is the correct ID. An overly permissive regular expression can capture the wrong part of the log line. Anchor the pattern and make it specific, for example match `traceID=(\w+)` rather than a broad `(\w+)`.

### Can't edit derived fields on a provisioned data source

Data sources created through provisioning are read-only in the UI, so you can't change their derived fields from the data source configuration page.

- For self-managed Grafana, edit the derived field configuration in the provisioning YAML file and restart Grafana, or reload provisioning.
- For Grafana Cloud, open a support ticket from the Grafana Cloud Portal to request changes to a provisioned data source.

### A single link needs values from more than one field

A derived field produces a single value per log line, so you can't combine multiple labels or capture groups into one field or link. Create a separate derived field for each value you need to surface.

## Live tailing issues

These issues affect real-time log tailing in Explore.

### Live tailing doesn't stream logs

Live tailing relies on WebSocket connections between the browser and Grafana, and between Grafana and Loki. Reverse proxies often block these connections.

**Solutions:**

1. Confirm your reverse proxy forwards WebSocket connections for the tail endpoint. Refer to the proxy examples in the [Loki query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/query-editor/#tail-live-logs).
1. Verify that intermediate load balancers and firewall rules allow WebSocket upgrades.
1. Confirm Loki itself is reachable and that the tail endpoint isn't blocked.

## Label-based access control (LBAC) issues

Label-based access control (LBAC) filters the logs a team can query based on labels. It's available on Grafana Cloud and Grafana Enterprise, where it requires Grafana Enterprise Logs (GEL). If you can't enable LBAC, or teams see more logs than expected, check the following.

### Can't enable LBAC on a provisioned data source

{{< admonition type="note" >}}
You can't enable LBAC on a provisioned or auto-provisioned Loki data source. Provisioning of LBAC data sources isn't supported.
{{< /admonition >}}

To use LBAC, create a new Loki data source manually and configure it with basic authentication, using your Cloud Access Policy (CAP) token as the password. Point it at the same Loki endpoint as the provisioned data source, then define LBAC rules on the new data source's **Permissions** tab. For the full procedure, refer to [Configure LBAC for data sources for Loki](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/configure-teamlbac-for-loki/).

### Teams can still query all logs

- LBAC applies only when a team has at least one rule. If a team has no LBAC rules, its members can query all logs. Confirm each team has the rules you intend.
- Grafana permissions are additive. A user's basic role, such as Viewer, can grant access to the data source independently of team-scoped rules. Remove role and team permissions that aren't required so that only the intended teams have **Query** permission on the LBAC data source.
- Remove any label selectors from the Cloud Access Policy used by the data source. CAP label selectors override LBAC rules.

For details and limitations, refer to [LBAC for data sources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/teamlbac/).

## Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Review logs in `/var/log/grafana/grafana.log`, or in your configured log location.
1. Look for Loki-specific entries that include request and response details.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you've tried these solutions and still encounter issues:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review the [Loki data source GitHub issues](https://github.com/grafana/grafana-loki-datasource/issues) for known bugs.
1. Consult the [Loki documentation](https://grafana.com/docs/loki/latest/) for service-specific guidance.
1. When reporting issues, include:
   - Grafana and Loki versions
   - Error messages, with sensitive information redacted
   - Steps to reproduce
   - Relevant configuration, with credentials redacted

# Microsoft SQL Server alerting

You can use Grafana Alerting with Microsoft SQL Server to create alerts based on your SQL Server data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met.

For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/).

## Before you begin

Before creating alerts with Microsoft SQL Server, ensure you have:

- A [configured Microsoft SQL Server data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/configure/)
- Appropriate permissions to create alert rules
- Understanding of the metrics you want to monitor

## Supported query types

Microsoft SQL Server alerting works with **time series queries** that return numeric data over time. Table-formatted queries are not supported in alert rule conditions.

To create a valid alert query:

- Include a `time` column that returns an SQL `datetime`/`datetime2` or a Unix epoch timestamp
- Return numeric values for the metrics you want to alert on
- Sort results by the time column

For more information on writing time series queries, refer to [Microsoft SQL Server query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/query-editor/).

### Query format requirements

| Query format | Alerting support | Notes                                    |
| ------------ | ---------------- | ---------------------------------------- |
| Time series  | Yes              | Required for alerting                    |
| Table        | No               | Convert to time series format for alerts |

## Create an alert rule

To create an alert rule using Microsoft SQL Server:

1. Navigate to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for the alert rule.
1. Select your **Microsoft SQL Server** data source.
1. Build your query using the query editor:
   - Set the **Format** to **Time series**.
   - Include a time column using the `$__timeGroup()` or `$__timeGroupAlias()` macro.
   - Add numeric columns for the values to monitor.
   - Use `$__timeFilter()` to filter data by the evaluation time range.
1. Configure the alert condition (for example, when the average is above a threshold).
1. Set the evaluation interval and pending period.
1. Configure notifications and labels.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example alert queries

The following examples show common alerting scenarios with Microsoft SQL Server.

### Alert on high error count

Monitor the number of errors over time:

```sql
SELECT
  $__timeGroupAlias(created_at, '1m'),
  COUNT(*) AS error_count
FROM error_logs
WHERE $__timeFilter(created_at)
  AND level = 'error'
GROUP BY $__timeGroup(created_at, '1m')
ORDER BY 1
```

**Condition:** When `error_count` is above 100.

### Alert on average response time

Monitor API response times:

```sql
SELECT
  $__timeGroupAlias(request_time, '5m'),
  AVG(response_time_ms) AS avg_response_time
FROM api_requests
WHERE $__timeFilter(request_time)
GROUP BY $__timeGroup(request_time, '5m')
ORDER BY 1
```

**Condition:** When `avg_response_time` is above 500 (milliseconds).

### Alert on low order volume

Detect drops in order activity:

```sql
SELECT
  $__timeGroupAlias(order_date, '1h'),
  COUNT(*) AS order_count
FROM orders
WHERE $__timeFilter(order_date)
GROUP BY $__timeGroup(order_date, '1h')
ORDER BY 1
```

**Condition:** When `order_count` is below 10.

### Alert on high CPU utilization

Monitor server resource metrics:

```sql
SELECT
  $__timeGroupAlias(recorded_at, '5m'),
  AVG(cpu_percent) AS avg_cpu
FROM sys_metrics
WHERE $__timeFilter(recorded_at)
GROUP BY $__timeGroup(recorded_at, '5m')
ORDER BY 1
```

**Condition:** When `avg_cpu` is above 85.

## Limitations

When using Microsoft SQL Server with Grafana Alerting, be aware of the following limitations.

### Template variables not supported

Alert queries cannot contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$hostname` or `$environment` aren't resolved.

If your dashboard query uses template variables, create a separate query for alerting with hard-coded values.

### Table format not supported

Queries using the **Table** format cannot be used for alerting. Set the query format to **Time series** and ensure your query returns a time column.

### Query timeout

Complex queries with large datasets may time out during alert evaluation. Optimize queries for alerting by:

- Adding appropriate `WHERE` clauses to limit data
- Using indexes on time and filter columns
- Reducing the time range evaluated

### Current User authentication not supported

If your data source uses Azure Entra ID **Current User** authentication, alerting, reporting, and recorded queries are not supported. These features require backend-level credentials that don't rely on a specific user's session.

## Best practices

Follow these best practices when creating Microsoft SQL Server alerts:

- **Use time series format:** Always set the query format to time series for alert queries.
- **Include time filters:** Use the `$__timeFilter()` macro to limit data to the evaluation window.
- **Use MSSQL macros:** Prefer `$__timeGroupAlias()` and `$__timeGroup()` over manual time-bucketing expressions.
- **Optimize queries:** Add indexes on columns used in `WHERE` clauses and `GROUP BY`.
- **Test queries first:** Verify your query returns expected results in Explore before creating an alert.
- **Set realistic thresholds:** Base alert thresholds on historical data patterns.
- **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor.

# Microsoft SQL Server annotations

Annotations overlay event markers on your dashboard visualizations, helping you correlate metric behavior with events like deployments, incidents, or configuration changes. You can use the Microsoft SQL Server data source to create annotation queries that display SQL query results as annotation events.

For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/).

## Before you begin

Before you create MSSQL annotations, ensure you have:

- A [configured Microsoft SQL Server data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/configure/)
- A table containing event data with at least a time column and a description column
- Familiarity with [T-SQL](https://learn.microsoft.com/en-us/sql/t-sql/language-reference) query syntax

## Create an annotation query

To add a Microsoft SQL Server annotation to your dashboard:

1. Navigate to your dashboard and click **Dashboard settings** (gear icon).
1. Select **Annotations** in the left menu.
1. Click **Add annotation query**.
1. Enter a **Name** for the annotation.
1. Select your **Microsoft SQL Server** data source from the **Data source** drop-down.
1. Write an SQL query that returns the required columns.
1. Click **Save dashboard**.

## Annotation query columns

Your annotation query must return columns with specific names. Grafana uses these names to map query results to annotation fields.

| Column    | Required | Description                                                                                            |
| --------- | -------- | ------------------------------------------------------------------------------------------------------ |
| `time`    | Yes      | The date/time of the event. Can be a native SQL date/time type or a Unix epoch value in seconds.       |
| `timeend` | No       | The end time for region annotations. Same format as `time`. Creates a shaded region instead of a line. |
| `text`    | Yes      | The event description displayed in the annotation tooltip.                                             |
| `tags`    | No       | Comma-separated string used for event tags. Tags help categorize and filter annotations.               |

## Example: Annotation with epoch time

Given a table that stores events with Unix epoch timestamps:

```sql
CREATE TABLE [events] (
  time_sec bigint,
  description nvarchar(100),
  tags nvarchar(100),
)
```

Query to display events as annotations:

```sql
SELECT
  time_sec as time,
  description as [text],
  tags
FROM
  [events]
WHERE
  $__unixEpochFilter(time_sec)
ORDER BY 1
```

The `$__unixEpochFilter` macro automatically filters events to the dashboard's selected time range.

## Example: Region annotation with start and end times

To display annotations as shaded regions (spanning a duration), include both `time` and `timeend` columns:

```sql
SELECT
  time_sec as time,
  time_end_sec as timeend,
  description as [text],
  tags
FROM
  [events]
WHERE
  $__unixEpochFilter(time_sec)
ORDER BY 1
```

## Example: Annotation with native `datetime` column

If your table uses native SQL `datetime` or `datetime2` columns instead of epoch values:

```sql
SELECT
  time,
  measurement as text,
  convert(varchar, valueOne) + ',' + convert(varchar, valueTwo) as tags
FROM
  metric_values
WHERE
  $__timeFilter(time)
ORDER BY 1
```

The `$__timeFilter` macro works with native SQL date/time types and filters to the dashboard time range.

## Example: Deployment annotations

Display deployment events on your graphs:

```sql
SELECT
  deployed_at as time,
  'Deployed ' + version + ' to ' + environment as [text],
  environment as tags
FROM
  [deployments]
WHERE
  $__timeFilter(deployed_at)
ORDER BY 1
```

## Example: Maintenance window annotations

Display maintenance windows as shaded regions:

```sql
SELECT
  start_time as time,
  end_time as timeend,
  'Maintenance: ' + description as [text],
  'maintenance' as tags
FROM
  [maintenance_windows]
WHERE
  $__timeFilter(start_time)
ORDER BY 1
```

## Use template variables

You can use [template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/template-variables/) in your annotation queries to make them dynamic. For example, filter events by a selected server:

```sql
SELECT
  time_sec as time,
  description as [text],
  tags
FROM
  [events]
WHERE
  $__unixEpochFilter(time_sec)
  AND server IN ($server)
ORDER BY 1
```

## Macros

Use these macros in your annotation queries to filter by the dashboard time range:

| Macro                        | Description                                                             |
| ---------------------------- | ----------------------------------------------------------------------- |
| `$__timeFilter(column)`      | Filters by time range using a native SQL `datetime`/`datetime2` column. |
| `$__unixEpochFilter(column)` | Filters by time range using a column with Unix epoch timestamps.        |

For the full list of available macros, refer to the [Microsoft SQL Server query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/query-editor/#macros).

## Best practices

Follow these best practices when creating Microsoft SQL Server annotations:

- **Use time filters:** Always include `$__timeFilter()` or `$__unixEpochFilter()` to limit results to the dashboard time range.
- **Keep queries efficient:** Add indexes on time columns and filter columns to improve query performance.
- **Use meaningful text:** Include descriptive information in the `text` column to make annotations useful at a glance.
- **Organize with tags:** Use consistent tag values to categorize annotations and enable filtering.
- **Test queries first:** Verify your query returns expected results in Explore before adding it as an annotation.

## Get help

If your annotations aren't appearing or behaving as expected, refer to [Troubleshoot Microsoft SQL Server data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/troubleshooting/) for common solutions.

# Configure the Microsoft SQL Server data source

This document provides instructions for configuring the Microsoft SQL Server data source and explains available configuration options. For general information on adding and managing data sources, refer to [Grafana data sources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/) and [Data source management](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/).

{{< admonition type="note" >}}
This page documents the **Microsoft SQL Server data source**, which lets you query an existing SQL Server database and visualize the results in Grafana. This is different from the [Microsoft SQL Server integration](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/integration-reference/integration-mssql/) (available in Grafana Cloud), which uses Grafana Alloy to collect performance metrics _about_ your SQL Server instance (CPU, memory, connections, and similar). If you want to monitor the health of SQL Server itself, refer to the integration documentation instead.
{{< /admonition >}}

## Before you begin

Before configuring the Microsoft SQL Server data source, ensure you have the following:

- **Grafana permissions:** You must have the `Organization administrator` role to add and configure data sources. Users with the `Editor` or `Viewer` role cannot access the data source configuration form unless granted additional permissions through [RBAC](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/). Organization administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system.

- **A running SQL Server instance:** Microsoft SQL Server 2012 or newer, Azure SQL Database, or Azure SQL Managed Instance.

- **Network access:** Grafana must be able to reach your SQL Server. The default port is `1433`.

- **Authentication credentials:** Depending on your authentication method, you need one of:
  - SQL Server login credentials (username and password).
  - Windows/Kerberos credentials and configuration (not supported in Grafana Cloud).
  - Azure Entra ID app registration or managed identity.

- **Security certificates:** If using encrypted connections, gather any necessary TLS/SSL certificates.

{{< admonition type="note" >}}
The Microsoft SQL Server data source plugin is preinstalled in Grafana, so no additional installation is required. As of Grafana 13.2, it's packaged as a standalone plugin that updates independently of Grafana releases. Refer to [Plugin updates](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/#plugin-updates) for details.
{{< /admonition >}}

{{< admonition type="tip" >}}
**Grafana Cloud users:** If your SQL Server is in a private network, you can configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) to establish connectivity.
{{< /admonition >}}

## Add the MSSQL data source

To add the MSSQL data source, complete the following steps:

1. Click **Connections** in the left-side menu.
1. Click **Add new connection**
1. Type `Microsoft SQL Server` in the search bar.
1. Select **Microsoft SQL Server** under data source.
1. Click **Add new data source** in the upper right.

Grafana takes you to the **Settings** tab, where you set up your Microsoft SQL Server configuration.

## Configure the data source in the UI

Following are configuration options for the Microsoft SQL Server data source.

<!-- vale Grafana.Spelling = NO -->

{{< admonition type="warning" >}}
Kerberos is not supported in Grafana Cloud.
{{< /admonition >}}

<!-- vale Grafana.Spelling = YES -->

| **Setting** | **Description**                                                                                                                       |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| **Name**    | The data source name. Sets the name you use to refer to the data source in panels and queries. Examples: `MSSQL-1`, `MSSQL_Sales1`.   |
| **Default** | Toggle to select as the default name in dashboard panels. When you go to a dashboard panel, this is the default selected data source. |

**Connection:**

| Setting      | Description                                                                                                                                                                                                                                                         |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Host**     | Sets the IP address or hostname (and optional port) of your MSSQL instance. The default port is `0`, which uses the driver's default. You can include additional connection properties (for example, `ApplicationIntent`) by separating them with semicolons (`;`). |
| **Database** | Sets the name of the MSSQL database to connect to.                                                                                                                                                                                                                  |

{{< admonition type="caution" >}}
If you add `ApplicationIntent=ReadOnly` to the Host field to route queries to a read-only replica, be aware that read-only replicas may be significantly slower than the primary. This can cause dashboard load times of several minutes. Only use `ApplicationIntent=ReadOnly` if your replica is sized for the query workload and you've tested performance.
{{< /admonition >}}

**TLS/SSL encryption:**

The **Encrypt** setting determines whether and to what extent a secure TLS/SSL connection is negotiated with the server.

| Encrypt setting | Description                                                                                      | When to use                                                                                       |
| --------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------- |
| **Disable**     | Data sent between the client and server is **not encrypted**.                                    | Legacy environments where encryption isn't supported or required. Not recommended for production. |
| **False**       | The default setting. Only the login packet is encrypted; **all other data is sent unencrypted**. | Development environments or when SQL Server doesn't have a valid certificate.                     |
| **True**        | **All data** sent between the client and server is **encrypted**.                                | Production environments. Required for Azure SQL Database.                                         |

When **Encrypt** is set to **True**, the following additional TLS options are available:

| Setting                            | Description                                                                                                                                                                                                                                    | Default                   |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| **Skip TLS Verify**                | If enabled, the server's certificate chain and hostname aren't validated. Equivalent to `trustServerCertificate=true` in the connection string.                                                                                                | Disabled                  |
| **TLS/SSL Root Certificate**       | Path to a PEM-encoded CA certificate file used to verify the server's certificate. Required when the SQL Server uses a certificate signed by a private CA.                                                                                     | System CA bundle          |
| **Hostname in server certificate** | The expected hostname in the server's TLS certificate. Use this when the certificate's Common Name (CN) or Subject Alternative Name (SAN) doesn't match the configured hostname (for example, when connecting through a load balancer or PDC). | Value from **Host** field |

**Choosing the right encryption configuration:**

| Scenario                                              | Encrypt          | Skip TLS Verify | Root Certificate     | Notes                                                                                                                 |
| ----------------------------------------------------- | ---------------- | --------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Azure SQL Database                                    | True             | No              | Not needed           | Azure SQL always requires encryption and uses a publicly trusted certificate.                                         |
| SQL Server with public CA certificate                 | True             | No              | Not needed           | Certificate is already in the system trust store.                                                                     |
| SQL Server with private/self-signed certificate       | True             | No              | Path to CA cert      | Upload or reference the CA certificate that signed the SQL Server certificate.                                        |
| SQL Server with unverified certificate (testing only) | True             | Yes             | Not needed           | Skips certificate validation entirely. Don't use in production.                                                       |
| SQL Server 2008/2008R2 without valid certificate      | Disable or False | N/A             | N/A                  | Older versions may not support TLS negotiation.                                                                       |
| Grafana Cloud with PDC                                | True             | No              | Not needed (usually) | PDC tunnels are encrypted at the SSH layer; the TLS certificate must still be valid from the PDC agent's perspective. |

{{< admonition type="note" >}}
If you're using an older version of Microsoft SQL Server like 2008 or 2008R2, you may need to set Encrypt to **Disable** or **False** to connect.
{{< /admonition >}}

{{< admonition type="caution" >}}
Grafana builds using BoringCrypto (FIPS-compliant builds) enforce FIPS 140-3 cipher requirements. If your SQL Server only supports older cipher suites (for example, TLS 1.0 or RC4-based ciphers), connections from a FIPS-enabled Grafana instance fail with a TLS handshake error. Upgrade your SQL Server TLS configuration to support TLS 1.2 with FIPS-approved cipher suites.
{{< /admonition >}}

**Authentication:**

<!-- vale Grafana.Spelling = NO -->

| Authentication Type                                   | Description                                                                                                                     | Credentials / Fields                                                                                                                                                                                                      |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **SQL Server Authentication**                         | Default method to connect to MSSQL. Use a SQL Server or Windows login in `DOMAIN\User` format.                                  | **Username**: SQL Server username. **Password**: SQL Server password.                                                                                                                                                     |
| **Windows Authentication** (Integrated Security)      | Uses the logged-in Windows user's credentials via single sign-on. Available only when SQL Server allows Windows Authentication. | No input required; uses the logged-in Windows user's credentials.                                                                                                                                                         |
| **Windows AD** (Username/Password)                    | Authenticates a domain user with their Active Directory username and password.                                                  | **Username**: `user@example.com`. **Password**: Active Directory password.                                                                                                                                                |
| **Windows AD** (Keytab)                               | Authenticates a domain user using a keytab file.                                                                                | **Username**: `user@example.com`. **Keytab file path**: Path to your keytab file.                                                                                                                                         |
| **Windows AD** (Credential Cache)                     | Uses a Kerberos credential cache already loaded in memory (for example, from a prior `kinit` command). No file needed.          | **Credential cache path**: Path to in-memory credential (for example, `/tmp/krb5cc_1000`).                                                                                                                                |
| **Windows AD** (Credential Cache File)                | Authenticates a domain user using a credential cache file (`.ccache`).                                                          | **Username**: `user@example.com`. **Credential cache file path**: for example, `/home/grot/cache.json`.                                                                                                                   |
| **Azure Entra ID (formerly Azure AD) Authentication** | Authenticates the data source using Azure authentication methods. The UI label is **Azure AD Authentication**.                  | For supported methods and configuration steps, refer to [Azure Entra ID authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/configure/#azure-entra-id-formerly-azure-ad-authentication). |

<!-- vale Grafana.Spelling = YES -->

{{< admonition type="note" >}}
As of Grafana v13.0, passwords and usernames that contain semicolons (`;`) or closing braces (`}`) are handled correctly. If you previously encountered connection failures due to special characters in credentials, upgrade to Grafana v13.0 or later.
{{< /admonition >}}

**Additional settings:**

Additional settings are optional settings you configure for more control over your data source. This includes connection limits, connection timeout, group-by time interval, and Secure Socks Proxy.

**Connection limits:**

Grafana maintains a connection pool for each configured MSSQL data source. These settings control pool behavior and directly affect performance, especially under concurrent load from dashboards and alerting.

| Setting           | Description                                                                                                                                                                                  |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Max open**      | The maximum number of open connections to the database. If set to `0`, there is no limit. If `max open` is greater than `0` and less than `max idle`, `max idle` is adjusted to match.       |
| **Auto max idle** | When enabled, automatically sets `max idle` to match `max open`. If `max open` isn't set, it defaults to `100`.                                                                              |
| **Max idle**      | The maximum number of idle connections in the pool. If `max open` is set and is lower than `max idle`, then `max idle` is reduced to match. If set to `0`, no idle connections are retained. |
| **Max lifetime**  | The maximum time (in seconds) a connection can be reused before being closed and replaced. If set to `0`, connections are reused indefinitely.                                               |

{{< admonition type="caution" >}}
If **Max open** is set too low, concurrent dashboard panels and alert rule evaluations compete for connections. Alerting queries may fail with "failed to connect to server" while dashboards continue to work, because alert evaluations use a separate execution path that can't wait indefinitely for a free connection. Increase **Max open** if you see intermittent alert failures.
{{< /admonition >}}

**Connection details:**

| **Setting**            | **Description**                                                                                                                                                                                                                                                     |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Min time interval**  | Specifies the lower bound for the auto-generated `GROUP BY` time interval. Grafana recommends matching this value to the data write frequency. For example, use `1m` if data is written every minute. Refer to [Min time interval](#min-time-interval) for details. |
| **Connection timeout** | Specifies the maximum number of seconds to wait when attempting to connect to the database before timing out. A value of `0` (the default) disables the timeout.                                                                                                    |

**Windows AD Advanced settings:**

<!-- vale Grafana.Spelling = NO -->

| Setting                   | Description                                                                                                                                                                                                             | Default              |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
| **UDP Preference Limit**  | Defines the maximum packet size (in bytes) that Kerberos libraries attempt to send over UDP before retrying with TCP. A value of `1` forces all communication to use TCP.                                               | `1` (always use TCP) |
| **DNS Lookup KDC**        | Controls whether DNS `SRV` records are used to locate [Key Distribution Centers (KDCs)](https://web.mit.edu/kerberos/krb5-latest/doc/admin/realm_config.html#key-distribution-centers) and other servers for the realm. | `true`               |
| **krb5 config file path** | Specifies the path to the Kerberos configuration file used by the [MIT krb5 package](https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html).                                                       | `/etc/krb5.conf`     |

{{< admonition type="note" >}}
Windows AD (Kerberos) authentication is only available for self-managed Grafana installations. It is not supported in Grafana Cloud. For Grafana Cloud deployments connecting to SQL Server, use SQL Server Authentication or Azure Entra ID.
{{< /admonition >}}

**Kerberos SPN requirements:**

For Kerberos authentication to succeed, a Service Principal Name (SPN) must be registered for the SQL Server instance that matches the hostname Grafana uses to connect. This is especially important when connecting through Availability Group Listeners or DNS aliases.

Register SPNs using the `setspn` command on the SQL Server host:

```text
setspn -S MSSQLSvc/<SQL_SERVER_FQDN>:1433 <DOMAIN>\<SERVICE_ACCOUNT>
setspn -S MSSQLSvc/<SQL_SERVER_FQDN> <DOMAIN>\<SERVICE_ACCOUNT>
```

If Grafana connects through a Listener or alias, register an additional SPN for that name. Verify registered SPNs with `setspn -L <DOMAIN>\<SERVICE_ACCOUNT>`.

<!-- vale Grafana.Spelling = YES -->

### Connect to on-premises SQL Server from Grafana Cloud

If your SQL Server instance is in a private network (on-premises, VPN, or VPC), Grafana Cloud cannot reach it directly over the internet. You must use **Private data source connect (PDC)** to establish a secure tunnel between your Grafana Cloud stack and your private network.

{{< admonition type="note" >}}
PDC is required for any SQL Server that isn't publicly accessible from the internet. Firewall allowlisting alone is not sufficient because Grafana Cloud doesn't use static IP addresses for data source connections.
{{< /admonition >}}

**Network prerequisites:**

- **Port 22 (SSH):** The PDC agent on your network must be able to reach the Grafana Cloud PDC endpoint over port 22 for the SSH tunnel.
- **Port 1433 (SQL Server):** The PDC agent must be able to reach your SQL Server instance on port 1433 (or your custom SQL Server port).
- **DNS resolution:** The PDC agent must be able to resolve the SQL Server hostname from within your private network.

**Set up PDC for Microsoft SQL Server:**

1. In your Grafana Cloud stack, navigate to **Connections** > **Private data source connect**.
1. Create a new PDC connection or use an existing one.
1. Install and configure the PDC agent on a host within your private network that has access to the SQL Server instance. For detailed instructions, refer to [Configure Grafana private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc).
1. Return to the Microsoft SQL Server data source configuration.
1. Under **Private data source connect**, select your PDC connection from the drop-down.
1. Enter the SQL Server hostname as it's resolvable from within your private network (not a public DNS name).
1. Click **Save & test** to verify connectivity.

Click **Manage private data source connect** to open your PDC connection page and view your configuration details.

For more information about PDC, refer to [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/).

{{< admonition type="caution" >}}
If you previously connected using a self-managed Grafana instance and are migrating to Grafana Cloud, you must configure PDC before your private SQL Server is accessible. Direct network access methods that worked with self-managed Grafana (VPN peering, private endpoints) don't apply to Grafana Cloud.
{{< /admonition >}}

After configuring your MSSQL data source options, click **Save & test** at the bottom to test the connection. You should see a confirmation dialog box that says:

**Database Connection OK**

### Azure Entra ID (formerly Azure AD) Authentication

The following Azure authentication methods are supported:

- App Registration (service principal)
- Managed Identity
- Current User authentication
- Azure Entra Password

The Azure SQL Server that you are connecting to must support Azure Entra authentication. For configuration details, refer to the [Azure SQL documentation](https://learn.microsoft.com/en-us/azure/azure-sql/database/authentication-aad-configure?view=azuresql&tabs=azure-portal).

#### Required server configuration

Before you can use any Azure Entra ID authentication method, you must enable Azure authentication in the Grafana server configuration. Add the following to your `grafana.ini` or `custom.ini` file:

```ini
[azure]
azure_auth_enabled = true
```

Depending on the method you choose, additional flags are required (documented in each section below). After changing the `.ini` file, restart the Grafana server for the changes to take effect.

#### Current User authentication

This is the recommended authentication mechanism when working with SQL Server instances that are hosted in Azure. It allows users to be authenticated to and query the database using their own credentials rather than long-lived credentials.

This authentication method requires your Grafana instance to be configured with Azure Entra ID (formerly Active Directory) authentication for login. With Azure Entra ID login, this method can be used to forward the currently logged in user's credentials to the data source. The user's credentials are then used when requesting data from the data source. For details on how to configure your Grafana instance using Azure Entra, refer to the [documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/azuread/).

{{< admonition type="note" >}}
Additional configuration is required to ensure that the App Registration used to login a user via Azure provides an access token with the permissions required by the data source.

The App Registration must be configured to issue both **Access Tokens** and **ID Tokens**.

1. In the Azure Portal, open the App Registration that requires configuration.
2. Select **Authentication** in the side menu.
3. Under **Implicit grant and hybrid flows** check both the **Access tokens** and **ID tokens** boxes.
4. Save the changes to ensure the App Registration is updated.

The App Registration must also be configured with additional **API Permissions** to provide authenticated users with access to the APIs utilised by the data source.

1. In the Azure Portal, open the App Registration that requires configuration.
1. Select **API Permissions** in the side menu.
1. Ensure the `openid`, `profile`, `email`, and `offline_access` permissions are present under the **Microsoft Graph** section. If not, they must be added.
1. Select **Add a permission** and choose the following permissions. They must be added individually. Refer to the [Azure documentation](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-configure-app-access-web-apis) for more information.
   - Select **APIs my organization uses** > Search for **Azure SQL** and select it > **Delegated permissions** > `user_impersonation` > **Add permissions**

After all permissions have been added, the Azure authentication section in Grafana must be updated. The `scopes` section must be updated to include the `.default` scope to ensure that a token with access to all APIs declared on the App Registration is requested by Grafana. After updated the scopes value should equal: `.default openid email profile`.
{{< /admonition >}}

This method of authentication doesn't inherently support all backend functionality as a user's credentials won't be in scope. Affected functionality includes alerting, reporting, and recorded queries. Also, note that query and resource caching is disabled by default for data sources using current user authentication.

**To enable current user authentication for Grafana:**

1. Set the `user_identity_enabled` flag in the `[azure]` section of the [Grafana server configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#azure).

   ```ini
   [azure]
   user_identity_enabled = true
   ```

2. In the SQL Server data source configuration, set **Authentication** to **Azure AD Authentication** and the Azure Authentication type to **Current User**.

#### App Registration

Use an app registration (service principal) to authenticate the data source with a client ID and secret. This is the recommended method for automated, non-interactive access to Azure SQL.

**Prerequisites:**

- An app registration created in Azure Entra. For instructions, refer to the [Azure documentation for service principals](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#get-tenant-and-app-id-values-for-signing-in).
- The tenant ID, client (application) ID, and a client secret from the app registration.

**Step 1: Add the app registration as a database user**

1. Connect to your Azure SQL database as a user with administrative permissions (the user must have the ability to read your Azure Entra directory, for example, by possessing the `Directory Readers` role).
1. Run the following SQL to create a user for the app registration:

   ```sql
   CREATE USER [<APP_REGISTRATION_NAME>] FROM EXTERNAL PROVIDER;
   ```

1. Grant the created user appropriate permissions. Grafana recommends read-only access:

   ```sql
   ALTER ROLE db_datareader ADD MEMBER [<APP_REGISTRATION_NAME>];
   ```

**Step 2: Configure the Grafana server**

Ensure `azure_auth_enabled` is set in your `grafana.ini`:

```ini
[azure]
azure_auth_enabled = true
```

**Step 3: Configure the data source**

1. In the SQL Server data source configuration, set **Authentication** to **Azure AD Authentication** and the Azure Authentication type to **App Registration**.
1. Set **Azure Cloud** to **Azure** (for the Azure public cloud).
1. Enter the **Directory (tenant) ID**, **Application (client) ID**, and **Client Secret** from your app registration.

#### Managed Identity

{{< admonition type="note" >}}
Managed Identity is available only in [Azure Managed Grafana](https://azure.microsoft.com/en-us/products/managed-grafana) or self-managed Grafana deployed in Azure. It is not available in Grafana Cloud.
{{< /admonition >}}

You can use managed identity to configure SQL Server in Grafana if you host Grafana in Azure (such as an App Service or with Azure Virtual Machines) and have managed identity enabled on your VM.
This lets you securely authenticate data sources without manually configuring credentials.
For details on Azure managed identities, refer to the [Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview).

**To enable managed identity for Grafana:**

1. Set both `azure_auth_enabled` and `managed_identity_enabled` in the `[azure]` section of the [Grafana server configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#azure):

   ```ini
   [azure]
   azure_auth_enabled = true
   managed_identity_enabled = true
   ```

1. In the SQL Server data source configuration, set **Authentication** to **Azure AD Authentication** and the Azure Authentication type to **Managed Identity**.

   This hides the directory ID, application ID, and client secret fields, and the data source uses managed identity to authenticate to SQL Server.

1. Optionally, set `managed_identity_client_id` in the `[azure]` section to use a user-assigned managed identity instead of the default system-assigned identity.

Ensure that the managed identity is added to your Azure SQL instance as a user.

### Azure Entra Password

{{< admonition type="warning" >}}
Azure Entra Password is not a recommended authentication mechanism as it requires configuration using a single users password. Consider an alternative authentication method such as current user authentication or app registration.
{{< /admonition >}}

You can connect to an Azure SQL database using the username and password of a user that has permissions in the desired database. This also requires an app registration to be configured with access to the database.

**To enable Azure Entra password for Grafana:**

1. Set the `azure_entra_password_credentials_enabled` flag in the `[azure]` section of the [Grafana server configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#azure).

   ```ini
   [azure]
   azure_entra_password_credentials_enabled = true
   ```

2. In the SQL Server data source configuration, set **Authentication** to **Azure AD Authentication** and the Azure Authentication type to **Azure Entra Password**.
3. Set the **User ID** value to the username of the user in the Azure SQL database.
4. Set the **Application Client ID** to the client ID of the app registration that has been added to the Azure SQL database
5. Set the **Password** value to the password of the user in the Azure SQL database.

### Min time interval

The **Min time interval** setting defines a lower limit for the [`$__interval`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#__interval) and [`$__interval_ms`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#__interval_ms) variables.

This value _must_ be formatted as a number followed by a valid time identifier:

| Identifier | Description |
| ---------- | ----------- |
| `y`        | year        |
| `M`        | month       |
| `w`        | week        |
| `d`        | day         |
| `h`        | hour        |
| `m`        | minute      |
| `s`        | second      |
| `ms`       | millisecond |

Grafana recommends setting this value to match your Microsoft SQL Server write frequency.
For example, use `1m` if Microsoft SQL Server writes data every minute.

You can also override this setting in a dashboard panel under its data source options.

### Database user permissions

When adding a data source, ensure the database user you specify has only SELECT permissions on the relevant database and tables. Grafana does not validate the safety of queries, which means they can include potentially harmful SQL statements, such as `USE otherdb`; or `DROP TABLE user;`, which could get executed. To minimize this risk, Grafana strongly recommends creating a dedicated MSSQL user with restricted permissions.

```sql
CREATE USER grafanareader WITH PASSWORD 'password'
GRANT SELECT ON dbo.YourTable3 TO grafanareader
```

Also, ensure that the user doesn't have any unwanted privileges from the public role.

### Diagnose connection issues

If you use older versions of Microsoft SQL Server, such as 2008 and 2008R2, you might need to disable encryption before you can connect the data source.

Grafana recommends that you use the latest available service pack for optimal compatibility.

### Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to [Provision Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/).

#### Example

```yaml
apiVersion: 1

datasources:
  - name: MSSQL
    type: mssql
    url: localhost:1433
    user: grafana
    jsonData:
      database: grafana
      maxOpenConns: 100
      maxIdleConns: 100
      maxIdleConnsAuto: true
      connMaxLifetime: 14400
      connectionTimeout: 0
      encrypt: 'false'
    secureJsonData:
      password: 'Password!'
```

### Configure with Terraform

You can configure the Microsoft SQL Server data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation.

#### Terraform example

The following example creates a basic Microsoft SQL Server data source:

```hcl
resource "grafana_data_source" "mssql" {
  name = "MSSQL"
  type = "mssql"
  url  = "localhost:1433"
  user = "grafana"

  json_data_encoded = jsonencode({
    database           = "grafana"
    maxOpenConns       = 100
    maxIdleConns       = 100
    maxIdleConnsAuto   = true
    connMaxLifetime    = 14400
    connectionTimeout  = 0
    encrypt            = "false"
  })

  secure_json_data_encoded = jsonencode({
    password = "Password!"
  })
}
```

For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source).

## Next steps

After configuring your Microsoft SQL Server data source, you can:

- [Write queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/query-editor/) using the query editor to explore and visualize your data
- [Create template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/template-variables/) to build dynamic, reusable dashboards
- [Add annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/annotations/) to overlay SQL Server events on your graphs
- [Set up alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/alerting/) to create alert rules based on your SQL Server data
- [Troubleshoot issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/troubleshooting/) if you encounter problems with your data source

# Microsoft SQL Server query editor

The MSSQL query editor lets you write Transact-SQL queries directly in Grafana, with built-in macros and formatting options designed for time series and table visualizations. You can open it from [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/) or from any dashboard panel in edit mode.

This page covers MSSQL-specific query features. For general querying concepts shared across all data sources, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/). For Transact-SQL syntax, refer to [Write Transact-SQL statements](https://learn.microsoft.com/en-us/sql/t-sql/tutorial-writing-transact-sql-statements?view=sql-server-ver17) and [Transact-SQL reference](https://learn.microsoft.com/en-us/sql/t-sql/language-reference?view=sql-server-ver17) in the Microsoft documentation.

The Microsoft SQL Server query editor has two modes:

- [Builder mode](#builder-mode)
- [Code mode](#code-mode)

To switch between the editor modes, select the corresponding **Builder** and **Code** tabs in the upper right.

![MSSQL query builder](/media/mssql/mssql-query-editor-v12.png)

{{< admonition type="warning" >}}
When switching from **Code** mode to **Builder** mode, any changes made to your SQL query aren't saved and aren't shown in the builder interface. You can choose to copy your code to the clipboard or discard the changes.
{{< /admonition >}}

To run a query, select **Run query** in the upper right of the editor.

In addition to writing queries, the query editor also allows you to create and use:

- [Macros](#macros)
- [Stored procedures](#use-stored-procedures)

## Builder mode

**Builder mode** allows you to build queries using a visual interface. This mode is great for users who prefer a guided query experience or are just getting started with SQL.

{{< figure alt="MSSQL builder mode>"  src="/media/docs/mssql/mssql-builder-mode-v12.png" class="docs-image--no-shadow" >}}

The following components help you build a T-SQL query:

- **Format** - Select a format response from the drop-down for the MSSQL query. The default is **Table**. Refer to [Table queries](#table-queries) and [Time series queries](#time-series-queries) for more information and examples. If you select the **Time series** format option, you must include a `time` column.

- **Dataset** - Select a database to query from the drop-down. Grafana automatically populates the drop-down with all databases the user has access to. If a database is set in the **Database** field on the data source configuration page or via a provisioning file, users are limited to querying only that database.

  Note that `tempdb`, `model`, `msdb`, and `master` system databases are not included in the query editor drop-down.

- **Table** - Select a table from the drop-down. After selecting a database, the next drop-down displays all available tables in that database.

- **Data operations** - _Optional_. Select an aggregation or a macro from the drop-down. You can add multiple data operations by clicking the **+ sign**. Click the **garbage can icon** to remove data operations.
  - **Column** - Select a column on which to run the aggregation.
  - **Interval** - Select an interval from the drop-down. You'll see this option when you choose a `time group` macro from the drop-down.
  - **Fill** - _Optional_. Add a `FILL` method to populate missing time intervals with default values (such as NULL, 0, or a specified value) when no data exists for those intervals. This ensures continuity in the time series, avoiding gaps in visualizations.
  - **Alias** - _Optional_. Add an alias from the drop-down. You can also add your own alias by typing it in the box and clicking **Enter**. Remove an alias by clicking the **X**.

- **Filter** - Toggle to add filters.
  - **Filter by column value** - _Optional_. If you toggle **Filter** you can add a column to filter by from the drop-down. To filter by additional columns, click the **+ sign** to the right of the condition drop-down. You can choose a variety of operators from the drop-down next to the condition. When multiple filters are added, use the `AND` or `OR` operators to define how conditions are evaluated. `AND` requires all conditions to be true, while `OR` requires any condition to be true. Use the second drop-down to select the filter value. To remove a filter, click the **X icon** next to it. If you select a `date-type` column, you can use macros from the operator list and choose `timeFilter` to insert the `$\_\_timeFilter` macro into your query with the selected date column.

    After selecting a date type column, you can choose Macros from the operators list and select `timeFilter`, which adds the `$\_\_timeFilter` macro to the query with the selected date column. Refer to [Macros](#macros) for more information.

- **Group** - Toggle to add a `GROUP BY` column.
  - **Group by column** - Select a column to filter by from the drop-down. Click the **+sign** to filter by multiple columns. Click the **X** to remove a filter.
- **Order** - Toggle to add an `ORDER BY` statement.
  - **Order by** - Select a column to order by from the drop-down. Select ascending (`ASC`) or descending (`DESC`) order.
  - **Limit** - You can add an optional limit on the number of retrieved results. Default is 50.
- **Preview** - Toggle for a preview of the SQL query generated by the query builder. Preview is toggled on by default.

For additional detail about using formats, refer to [Table queries](#table-queries) and [Time series queries](#time-series-queries).

## Code mode

{{< figure src="/static/img/docs/v92/sql_code_editor.png" class="docs-image--no-shadow" >}}

**Code mode** lets you build complex queries using a text editor with helpful features like autocompletion and syntax highlighting.

This mode is ideal for advanced users who need full control over the SQL query or want to use features not available in visual query mode. It's especially useful for writing subqueries, using macros, or applying advanced filtering and formatting. You can switch back to visual mode, but note that some custom queries may not be fully compatible.

### Code mode toolbar features

Code mode has several features in a toolbar located in the editor's lower-right corner.

- To reformat the query, click the brackets button (`{}`).
- To expand the code editor, click the chevron button pointing downward.
- To run the query, click the **Run query** button or use the keyboard shortcut **<key>Ctrl</key>/<key>Cmd</key> + <key>Enter</key>/<key>Return</key>**.

### Use autocompletion

Code mode's autocompletion feature works automatically while typing.
To manually trigger autocompletion, use the keyboard shortcut <key>Ctrl</key>/<key>Cmd</key> + <key>Space</key>.

Code mode supports autocompletion of tables, columns, SQL keywords, standard SQL functions, Grafana template variables, and Grafana macros.

{{< admonition type="note" >}}
You can't autocomplete columns until you've specified a table.
{{< /admonition >}}

## Macros

To simplify syntax and to allow for dynamic components, such as date range filters, you can add macros to your query.

Use macros in the `SELECT` clause to simplify the creation of time series queries.
From the **Data operations** drop-down, choose a macro such as `$\_\_timeGroup` or `$\_\_timeGroupAlias`. Then, select a time column from the **Column** drop-down and a time interval from the **Interval** drop-down. This generates a time-series query based on your selected time grouping.

{{< admonition type="warning" >}}
Time macros (`$__time`, `$__timeFilter`, etc.) don't support time zone parameters in Microsoft SQL Server and always expand to UTC values. If your timestamps aren't stored in UTC (common with `datetime`/`datetime2` types), convert them to UTC in your SQL query using `AT TIME ZONE … AT TIME ZONE 'UTC'` rather than passing a time zone argument to a macro.
{{< /admonition >}}

| **Macro**                                              | **Description**                                                                                                                                                                                                                            |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `$__time(dateColumn)`                                  | Renames the specified column to `_time`. Example: `dateColumn AS time`                                                                                                                                                                     |
| `$__timeEpoch(dateColumn)`                             | Converts a `DATETIME` column to a Unix timestamp and renames it to `_time`. Example: `DATEDIFF(second, '1970-01-01', dateColumn) AS time`                                                                                                  |
| `$__timeFilter(dateColumn)`                            | Adds a time range filter for the specified column. Example: `dateColumn BETWEEN '2017-04-21T05:01:17Z' AND '2017-04-21T05:06:17Z'`                                                                                                         |
| `$__timeFrom()`                                        | Returns the start of the current time range. Example: `'2017-04-21T05:01:17Z'`                                                                                                                                                             |
| `$__timeTo()`                                          | Returns the end of the current time range. Example: `'2017-04-21T05:06:17Z'`                                                                                                                                                               |
| `$__timeGroup(dateColumn, '5m'[, fillValue])`          | Groups the specified time column into intervals (for example, 5 minutes). Optionally fills gaps with a value like `0`, `NULL`, or `previous`. Example: `CAST(ROUND(DATEDIFF(second, '1970-01-01', time_column)/300.0, 0) AS bigint) * 300` |
| `$__timeGroup(dateColumn, '5m', 0)`                    | Same as above, with `0` used to fill missing data points.                                                                                                                                                                                  |
| `$__timeGroup(dateColumn, '5m', NULL)`                 | Same as above, with `NULL` used for missing data points.                                                                                                                                                                                   |
| `$__timeGroup(dateColumn, '5m', previous)`             | Same as above, using the previous value to fill gaps. If no previous value exists, `NULL` is used.                                                                                                                                         |
| `$__timeGroupAlias(dateColumn, '5m')`                  | Same as `$__timeGroup`, but also adds an alias to the resulting column.                                                                                                                                                                    |
| `$__unixEpochFilter(dateColumn)`                       | Adds a time range filter using Unix timestamps. Example: `dateColumn > 1494410783 AND dateColumn < 1494497183`                                                                                                                             |
| `$__unixEpochFrom()`                                   | Returns the start of the current time range as a Unix timestamp. Example: `1494410783`                                                                                                                                                     |
| `$__unixEpochTo()`                                     | Returns the end of the current time range as a Unix timestamp. Example: `1494497183`                                                                                                                                                       |
| `$__unixEpochNanoFilter(dateColumn)`                   | Adds a time range filter using nanosecond-precision Unix timestamps. Example: `dateColumn > 1494410783152415214 AND dateColumn < 1494497183142514872`                                                                                      |
| `$__unixEpochNanoFrom()`                               | Returns the start of the current time range as a nanosecond Unix timestamp. Example: `1494410783152415214`                                                                                                                                 |
| `$__unixEpochNanoTo()`                                 | Returns the end of the current time range as a nanosecond Unix timestamp. Example: `1494497183142514872`                                                                                                                                   |
| `$__unixEpochGroup(dateColumn, '5m', [fillMode])`      | Same as `$__timeGroup`, but for Unix timestamps. Optional `fillMode` controls how to handle missing points.                                                                                                                                |
| `$__unixEpochGroupAlias(dateColumn, '5m', [fillMode])` | Same as above, but adds an alias to the grouped column.                                                                                                                                                                                    |

### View the interpolated query

The query editor includes a **Generated SQL** link that appears after you run a query while editing a panel. Click this link to view the raw interpolated SQL that Grafana executed, including any macros that were expanded during query processing.

## Table queries

To create a Table query, set the **Format** option in the query editor to [**Table**](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/table/). This allows you to write any valid SQL query, and the Table panel displays the results using the returned columns and rows.

**Example:**

```sql
CREATE TABLE [event] (
  time_sec bigint,
  description nvarchar(100),
  tags nvarchar(100),
)
```

```sql
CREATE TABLE [mssql_types] (
  c_bit bit, c_tinyint tinyint, c_smallint smallint, c_int int, c_bigint bigint, c_money money, c_smallmoney smallmoney, c_numeric numeric(10,5),
  c_real real, c_decimal decimal(10,2), c_float float,
  c_char char(10), c_varchar varchar(10), c_text text,
  c_nchar nchar(12), c_nvarchar nvarchar(12), c_ntext ntext,
  c_datetime datetime,  c_datetime2 datetime2, c_smalldatetime smalldatetime, c_date date, c_time time, c_datetimeoffset datetimeoffset
)

INSERT INTO [mssql_types]
SELECT
  1, 5, 20020, 980300, 1420070400, '$20000.15', '£2.15', 12345.12,
  1.11, 2.22, 3.33,
  'char10', 'varchar10', 'text',
  N'☺nchar12☺', N'☺nvarchar12☺', N'☺text☺',
  GETDATE(), CAST(GETDATE() AS DATETIME2), CAST(GETDATE() AS SMALLDATETIME), CAST(GETDATE() AS DATE), CAST(GETDATE() AS TIME), SWITCHOFFSET(CAST(GETDATE() AS DATETIMEOFFSET), '-07:00')
```

**Example query with output:**

```sql
SELECT * FROM [mssql_types]
```

{{< figure src="/static/img/docs/v51/mssql_table_query.png" max-width="500px" class="docs-image--no-shadow" >}}

Use the keyword `AS` to define an alias in your query to rename a column or table.

**Example query with output:**

```sql
SELECT
  c_bit AS [column1], c_tinyint AS [column2]
FROM
  [mssql_types]
```

{{< figure src="/static/img/docs/v51/mssql_table_result.png" max-width="1489px" class="docs-image--no-shadow" >}}

## Time series queries

{{< admonition type="note" >}}
Store timestamps in UTC to avoid issues with time shifts in Grafana when using non-UTC timezones.
{{< /admonition >}}

To create a time series query, set the **Format** option in the query editor to **Time series**. The query must include a column named `time`, which should contain either an SQL `datetime` value or a numeric value representing Unix epoch time in seconds. The result set must be sorted by the `time` column for panels to visualize the data correctly.

A time series query returns results in [wide data frame format](https://grafana.com/developers/plugin-tools/key-concepts/data-frames#wide-format).

- Any column except `time` or of the type `string` transforms into value fields in the data frame query result.
- Any string column transforms into field labels in the data frame query result.

You can enable macro support in the `SELECT` clause to simplify time series query creation. Use the **Data operations** drop-down to choose a macro such as `$\_\_timeGroup` or `$\_\_timeGroupAlias`, then select a time column from the Column drop-down and a time interval from the Interval drop-down. This generates a time-series query based on your selected time grouping.

{{< docs/shared source="grafana" lookup="datasources/sql-query-builder-macros.md" version="<GRAFANA_VERSION>" >}}

### Create a metric query

For backward compatibility, there's an exception to the above rule for queries that return three columns and include a string column named `metric`.
Instead of transforming the `metric` column into field labels, it becomes the field name, and then the series name is formatted as the value of the `metric` column.
Refer to the example with the `metric` column below.

To optionally customize the default series name formatting, refer to [Standard options definitions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-standard-options/#display-name).

**Example with `metric` column:**

```sql
SELECT
  $__timeGroupAlias(time_date_time, '5m'),
  min("value_double"),
  'min' as metric
FROM test_data
WHERE $__timeFilter(time_date_time)
GROUP BY time
ORDER BY 1
```

Data frame result:

```text
+---------------------+-----------------+
| Name: time          | Name: min       |
| Labels:             | Labels:         |
| Type: []time.Time   | Type: []float64 |
+---------------------+-----------------+
| 2020-01-02 03:05:00 | 3               |
| 2020-01-02 03:10:00 | 6               |
+---------------------+-----------------+
```

### Time series query examples

**Use the fill parameter in the $\_\_timeGroupAlias macro to convert null values to be zero instead:**

```sql
SELECT
  $__timeGroupAlias(createdAt, '5m', 0),
  sum(value) as value,
  hostname
FROM test_data
WHERE
  $__timeFilter(createdAt)
GROUP BY
  time,
  hostname
ORDER BY 1
```

Given the data frame result in the following example and using the graph panel, you get two series named _value 10.0.1.1_ and _value 10.0.1.2_. To render the series with a name of _10.0.1.1_ and _10.0.1.2_, use a [Standard options definitions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-standard-options/#display-name) display name value of `${__field.labels.hostname}`.

Data frame result:

```text
+---------------------+---------------------------+---------------------------+
| Name: time          | Name: value               | Name: value               |
| Labels:             | Labels: hostname=10.0.1.1 | Labels: hostname=10.0.1.2 |
| Type: []time.Time   | Type: []float64           | Type: []float64           |
+---------------------+---------------------------+---------------------------+
| 2020-01-02 03:05:00 | 3                         | 4                         |
| 2020-01-02 03:10:00 | 6                         | 7                         |
+---------------------+---------------------------+---------------------------+
```

**Use multiple columns:**

```sql
SELECT
  $__timeGroupAlias(time_date_time, '5m'),
  min(value_double) as min_value,
  max(value_double) as max_value
FROM test_data
WHERE $__timeFilter(time_date_time)
GROUP BY time
ORDER BY 1
```

Data frame result:

```text
+---------------------+-----------------+-----------------+
| Name: time          | Name: min_value | Name: max_value |
| Labels:             | Labels:         | Labels:         |
| Type: []time.Time   | Type: []float64 | Type: []float64 |
+---------------------+-----------------+-----------------+
| 2020-01-02 03:04:00 | 3               | 4               |
| 2020-01-02 03:05:00 | 6               | 7               |
+---------------------+-----------------+-----------------+
```

## Annotations

You can use Microsoft SQL Server queries as annotation sources to overlay events on your dashboard graphs. For detailed instructions, query examples, and best practices, refer to [Microsoft SQL Server annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/annotations/).

## Use stored procedures

Stored procedures have been verified to work with Grafana queries. However, note that there is no special handling or extended support for stored procedures, so some edge cases may not behave as expected.

Stored procedures can be used in table, time series, and annotation queries, provided that the returned data matches the expected column names and formats described in the relevant previous sections in this document.

{{< admonition type="note" >}}
Grafana macro functions do not work inside stored procedures.
{{< /admonition >}}

{{< figure src="/static/img/docs/v51/mssql_metrics_graph.png" class="docs-image--no-shadow docs-image--right" >}}

For the following examples, the database table is defined in [Time series queries](#time-series-queries). Let's say that we want to visualize four series in a graph panel, such as all combinations of columns `valueOne`, `valueTwo` and `measurement`. Graph panel to the right visualizes what we want to achieve. To solve this, you need to use two queries:

**First query:**

```sql
SELECT
  $__timeGroup(time, '5m') as time,
  measurement + ' - value one' as metric,
  avg(valueOne) as valueOne
FROM
  metric_values
WHERE
  $__timeFilter(time)
GROUP BY
  $__timeGroup(time, '5m'),
  measurement
ORDER BY 1
```

**Second query:**

```sql
SELECT
  $__timeGroup(time, '5m') as time,
  measurement + ' - value two' as metric,
  avg(valueTwo) as valueTwo
FROM
  metric_values
GROUP BY
  $__timeGroup(time, '5m'),
  measurement
ORDER BY 1
```

### Stored procedure with epoch time format

You can define a stored procedure to return all the data needed to render multiple series (for example, 4) in a graph panel.

In the following example, the stored procedure accepts two parameters, `@from` and `@to`, both of type `int`. These parameters represent a time range (from/to) in epoch time format and are used to filter the results returned by the procedure.

The query inside the procedure simulates the behavior of `$__timeGroup(time, '5m')` by grouping timestamps into 5-minute intervals. While the expressions for time grouping are somewhat verbose, they can be extracted into reusable SQL Server functions to simplify the procedure.

```sql
CREATE PROCEDURE sp_test_epoch(
  @from int,
  @to 	int
)	AS
BEGIN
  SELECT
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int) as time,
    measurement + ' - value one' as metric,
    avg(valueOne) as value
  FROM
    metric_values
  WHERE
    time >= DATEADD(s, @from, '1970-01-01') AND time <= DATEADD(s, @to, '1970-01-01')
  GROUP BY
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int),
    measurement
  UNION ALL
  SELECT
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int) as time,
    measurement + ' - value two' as metric,
    avg(valueTwo) as value
  FROM
    metric_values
  WHERE
    time >= DATEADD(s, @from, '1970-01-01') AND time <= DATEADD(s, @to, '1970-01-01')
  GROUP BY
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int),
    measurement
  ORDER BY 1
END
```

Then, in your graph panel, you can use the following query to call the stored procedure with the time range dynamically populated by Grafana:

```sql
DECLARE
  @from int = $__unixEpochFrom(),
  @to int = $__unixEpochTo()

EXEC dbo.sp_test_epoch @from, @to
```

This uses Grafana built-in macros to convert the selected time range into epoch time (`$__unixEpochFrom()` and `$__unixEpochTo()`), which are passed to the stored procedure as input parameters.

### Stored procedure with `datetime` format

You can define a stored procedure to return all the data needed to render four series in a graph panel.

In the following example, the stored procedure accepts two parameters, `@from` and `@to`, of the type `datetime`. These parameters represent the selected time range and are used to filter the returned data.

The query within the procedure mimics the behavior of `$__timeGroup(time, '5m')` by grouping data into 5-minute intervals. These expressions can be verbose, but you may extract them into reusable SQL Server functions for improved readability and maintainability.

```sql
CREATE PROCEDURE sp_test_datetime(
  @from datetime,
  @to 	datetime
)	AS
BEGIN
  SELECT
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int) as time,
    measurement + ' - value one' as metric,
    avg(valueOne) as value
  FROM
    metric_values
  WHERE
    time >= @from AND time <= @to
  GROUP BY
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int),
    measurement
  UNION ALL
  SELECT
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int) as time,
    measurement + ' - value two' as metric,
    avg(valueTwo) as value
  FROM
    metric_values
  WHERE
    time >= @from AND time <= @to
  GROUP BY
    cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int),
    measurement
  ORDER BY 1
END

```

To call this stored procedure from a graph panel, use the following query with Grafana built-in macros to populate the time range dynamically:

```sql
DECLARE
  @from datetime = $__timeFrom(),
  @to datetime = $__timeTo()

EXEC dbo.sp_test_datetime @from, @to
```

## Next steps

After building your queries, you can:

- [Use template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/template-variables/) to create dynamic, reusable dashboards
- [Add annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/annotations/) to overlay SQL Server events on your graphs
- [Set up alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/alerting/) to create alert rules based on your query results
- [Troubleshoot issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/troubleshooting/) if queries return unexpected results

# Microsoft SQL Server template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
Grafana displays these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
Grafana refers to such variables as **template variables**.

For general information on using variables in Grafana, refer to [Add variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/).

For an introduction to templating and template variables, refer to [Templating](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/).

## Query variable

A query variable in Grafana dynamically retrieves values from your data source using a query. With a query variable, you can write a SQL query that returns values such as measurement names, key names, or key values that are shown in a drop-down select box.

For example, the following query returns all values from the `hostname` column:

```sql
SELECT hostname FROM host
```

A query can return multiple columns, and Grafana automatically generates a list using the values from those columns. For example, the following query returns values from both the `hostname` and `hostname2` columns, which are included in the variable's drop-down list.

```sql
SELECT [host].[hostname], [other_host].[hostname2] FROM host JOIN other_host ON [host].[city] = [other_host].[city]
```

You can also create a key/value variable using a query that returns two columns named `__text` and `__value`.

- The `__text` column defines the label shown in the drop-down.

- The `__value` column defines the value passed to panel queries.

This is useful when you want to display a user-friendly label (like a hostname) but use a different underlying value (like an ID).

Note that the values in the `_text` column should be unique. If there are duplicates, Grafana uses only the first matching entry.

```sql
SELECT hostname __text, id __value FROM host
```

You can also create nested variables, where one variable depends on the value of another. For example, if you have a variable named `region`, you can configure a `hosts` variable to only show hosts from the selected region. If `region` is a multi-value variable, use the `IN` operator instead of `=` to match against multiple selected values.

```sql
SELECT hostname FROM host WHERE region IN ($region)
```

## Use variables in queries

Grafana automatically quotes template variable values only when the template variable is a `multi-value`.

When using a multi-value variable, use the `IN` comparison operator instead of `=` to match against multiple values.

Grafana supports two syntaxes for using variables in queries:

- **`$<varname>` syntax**

Example with a template variable named `hostname`:

```sql
SELECT
  atimestamp time,
  aint value
FROM table
WHERE $__timeFilter(atimestamp) and hostname in($hostname)
ORDER BY atimestamp
```

- **`[[varname]]` syntax**

Example with a template variable named `hostname`:

```sql
SELECT
  atimestamp as time,
  aint as value
FROM table
WHERE $__timeFilter(atimestamp) and hostname in([[hostname]])
ORDER BY atimestamp
```

## Format variables for SQL queries

When using template variables inside SQL queries, the formatting option you choose determines how Grafana renders the variable value. Choosing the correct format prevents SQL injection risks and avoids broken queries from incorrect quoting.

### Use `sqlstring` for safe single-value interpolation

The `${var:sqlstring}` format wraps the variable value in single quotes and escapes any internal single quotes. This is the recommended approach for single-value variables used in `WHERE` clauses:

```sql
SELECT * FROM events
WHERE hostname = ${hostname:sqlstring}
```

If `hostname` is set to `server01`, this expands to:

```sql
SELECT * FROM events
WHERE hostname = 'server01'
```

{{< admonition type="caution" >}}
Don't manually wrap variables in quotes (for example, `'${var}'`). Since Grafana v11.3, multi-value variables are automatically quoted when used with the `IN` operator. Manually adding quotes results in double-quoting (for example, `''value''`), which breaks queries.
{{< /admonition >}}

### Multi-value variable formatting

When a variable allows multiple selections, Grafana provides several formatting options:

| Format                | Syntax               | Output for selections `a`, `b` | Use case                       |
| --------------------- | -------------------- | ------------------------------ | ------------------------------ |
| Default (auto-quoted) | `$var` or `${var}`   | `'a','b'`                      | `IN` clauses (recommended)     |
| `sqlstring`           | `${var:sqlstring}`   | `'a'` (single value only)      | Equality comparisons           |
| `csv`                 | `${var:csv}`         | `a,b`                          | Unquoted lists                 |
| `singlequote`         | `${var:singlequote}` | `'a','b'`                      | Explicit single-quote wrapping |
| `doublequote`         | `${var:doublequote}` | `"a","b"`                      | MSSQL identifier quoting       |
| `pipe`                | `${var:pipe}`        | `a\|b`                         | Regex patterns                 |
| `raw`                 | `${var:raw}`         | `a,b`                          | No escaping (use with caution) |

For multi-value variables in SQL `WHERE` clauses, the default formatting with `IN` is typically correct:

```sql
SELECT * FROM events
WHERE hostname IN ($hostname)
```

If `hostname` has values `server01` and `server02` selected, this expands to:

```sql
SELECT * FROM events
WHERE hostname IN ('server01','server02')
```

### Disable quoting for multi-value variables

To output multi-value variables without automatic quoting, use the `csv` format:

```text
${servers:csv}
```

This outputs the values as an unquoted comma-separated list. Use this only when you handle quoting yourself or when the values are numeric.

Refer to [Advanced variable format options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/#advanced-variable-format-options) for the full list of formatting options.

## Handle the "All" option in SQL queries

When a variable includes the **All** option (enabled in variable settings), you need to handle the case where the user selects "All" differently from individual selections.

### Pattern: Use a custom "All" value with `LIKE`

Set the **Custom all value** field in the variable settings to `%`, then use `LIKE` in your query:

```sql
SELECT * FROM events
WHERE hostname LIKE ${hostname:sqlstring}
```

When the user selects "All", `hostname` resolves to `%`, and the `LIKE '%'` matches all rows. When a specific value is selected, `LIKE 'server01'` matches only that value.

### Pattern: Conditional `WHERE` clause with "All"

For `IN` clauses where `%` doesn't work, use a conditional approach. Set the **Custom all value** to `ALL`, then write:

```sql
SELECT * FROM events
WHERE (
  ${hostname:raw} = 'ALL'
  OR hostname IN ($hostname)
)
```

When the user selects "All", the first condition is true and all rows match. When specific values are selected, the `IN` clause filters normally.

### Pattern: Chained variables with "All"

When you have chained (dependent) variables and the parent uses "All", apply the same pattern. For example, if `region` feeds into `hostname`:

Variable query for `hostname`:

```sql
SELECT hostname FROM host
WHERE (${region:raw} = 'ALL' OR region IN ($region))
```

Panel query:

```sql
SELECT $__timeGroupAlias(time, '5m'), COUNT(*) as count
FROM events
WHERE $__timeFilter(time)
  AND (${region:raw} = 'ALL' OR region IN ($region))
  AND (${hostname:raw} = 'ALL' OR hostname IN ($hostname))
GROUP BY $__timeGroup(time, '5m')
ORDER BY 1
```

## Conditional SQL clauses

Grafana doesn't have built-in support for conditionally omitting entire `WHERE` clauses based on variable selection. Use one of these patterns to achieve conditional filtering.

### Pattern: Always-true condition

Use a condition that evaluates to true when the variable should be ignored:

```sql
SELECT * FROM events
WHERE $__timeFilter(time)
  AND (${env:raw} = 'ALL' OR environment IN ($env))
  AND (${sev:raw} = 'ALL' OR severity IN ($sev))
```

### Pattern: CASE expression for optional filters

For more complex scenarios, use a `CASE` expression:

```sql
SELECT * FROM events
WHERE $__timeFilter(time)
  AND 1 = CASE
    WHEN ${hostname:raw} = 'ALL' THEN 1
    WHEN hostname IN ($hostname) THEN 1
    ELSE 0
  END
```

## Known behaviors and limitations

Be aware of the following behaviors when using template variables with Microsoft SQL Server.

### Automatic quoting in multi-value variables

Since Grafana v11.3, multi-value variables used with the `IN` operator are automatically quoted. If you previously wrote `WHERE col IN ('${var}')` with manual quotes, remove the manual quotes and use `WHERE col IN ($var)` instead.

### Comments inside string literals

The Grafana SQL query parser strips SQL comments (`--` and `/* */`) before macro interpolation. In some cases, `--` inside a quoted string literal could be incorrectly interpreted as a comment. If your query contains strings with `--`, verify the query expands correctly by clicking **Generated SQL** after running the query.

### Variables in alerting queries

Template variables are not supported in alerting queries. Grafana evaluates alert rules on the backend without dashboard context. If your dashboard query uses variables, create a separate alerting query with hard-coded values. For more information, refer to [Microsoft SQL Server alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/alerting/#template-variables-not-supported).

# Troubleshoot Microsoft SQL Server data source issues

This document provides solutions to common issues you may encounter when configuring or using the Microsoft SQL Server (MSSQL) data source in Grafana.

## Configuration errors

These errors occur when setting up or accessing the data source configuration in Grafana.

### Configuration form not displaying

**Symptoms:** The data source settings page shows only **Delete** and **Back** buttons with no configuration fields.

**Cause:** The logged-in user has insufficient permissions to configure data sources. Only users with the `Organization administrator` role (or a custom RBAC role with `datasources:write` permissions) can access the data source configuration form.

**Solution:**

1. Ask an organization administrator to grant you the `Organization administrator` role, or assign you a custom RBAC role with data source write permissions.
1. Alternatively, ask an administrator to configure the data source on your behalf or use [provisioning](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/) to define the data source in YAML.

## Connection errors

These errors occur when Grafana cannot establish or maintain a connection to the Microsoft SQL Server.

### Unable to connect to the server

**Error message:** `Unable to open tcp connection` or `dial tcp: connection refused`

**Cause:** Grafana cannot establish a network connection to the SQL Server.

**Solution:**

1. Verify that the SQL Server is running and accessible.
1. Check that the host and port are correct in the data source configuration. The default SQL Server port is `1433`.
1. Ensure there are no firewall rules blocking the connection between Grafana and SQL Server.
1. Verify that SQL Server is configured to allow remote connections.
1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your SQL Server instance is not publicly accessible.

### Connection timeout

**Error message:** "Connection timed out" or "I/O timeout"

**Cause:** The connection to SQL Server timed out before receiving a response.

**Solution:**

1. Check the network latency between Grafana and SQL Server.
1. Verify that SQL Server is not overloaded or experiencing performance issues.
1. Increase the **Connection timeout** setting in the data source configuration under **Additional settings**.
1. Check if any network devices (load balancers, proxies) are timing out the connection.

### Encryption-related connection failures

**Error message:** "TLS handshake failed" or "certificate verify failed"

**Cause:** There is a mismatch between the encryption settings in Grafana and what the SQL Server supports or requires.

**Solution:**

1. For older versions of SQL Server (2008, 2008R2), set the **Encrypt** option to **Disable** or **False** in the data source configuration.
1. If Encrypt is **True**, verify that the SQL Server has a valid TLS certificate.
1. Check that the certificate is trusted by the Grafana server (either in the system CA bundle or specified in **TLS/SSL Root Certificate**).
1. If the certificate's hostname doesn't match the configured **Host** value (for example, when using a load balancer or PDC), set **Hostname in server certificate** to match the certificate's CN or SAN.
1. As a temporary workaround for testing, enable **Skip TLS Verify** (not recommended for production).
1. Ensure you're using the latest available service pack for your SQL Server version for optimal compatibility.

### TLS handshake failures with FIPS-enabled Grafana

**Error message:** "TLS handshake failed" or "no cipher suite supported by both client and server"

**Cause:** Grafana builds using BoringCrypto (FIPS-compliant builds) enforce FIPS 140-3 cipher requirements. SQL Server instances that only support older cipher suites (TLS 1.0, RC4, or 3DES) are incompatible with these builds.

**Solution:**

1. Upgrade the SQL Server instance to support TLS 1.2 with FIPS-approved cipher suites (AES-based).
1. On the SQL Server host, verify TLS 1.2 is enabled in the Windows registry or SQL Server Network Configuration.
1. If you cannot upgrade SQL Server, use a non-FIPS Grafana build.
1. Verify which TLS versions your SQL Server supports by checking the SQL Server error log for "TLS handshake" entries.

### TLS certificate errors through PDC

**Error message:** "certificate verify failed" or "x509: certificate signed by unknown authority" when connecting through Private data source connect

**Cause:** The SQL Server's TLS certificate must be verifiable from the PDC agent's perspective, not from Grafana Cloud. If the SQL Server uses a private CA certificate, the PDC agent host must trust that CA.

**Solution:**

1. Install the private CA certificate on the PDC agent host's system trust store.
1. Alternatively, specify the CA certificate path in the data source's **TLS/SSL Root Certificate** field (the path must be accessible from the PDC agent, not from Grafana Cloud).
1. If the certificate's hostname doesn't match what the PDC agent sees, set **Hostname in server certificate** to the correct value.
1. Verify the certificate is valid (not expired) from the PDC agent host: `openssl s_client -connect <SQL_SERVER_HOST>:1433 -starttls mssql`.

### Named instance connection issues

**Error message:** "Cannot connect to named instance" or connection fails when using instance name

**Cause:** Grafana cannot resolve the SQL Server named instance.

**Solution:**

1. Use the format `hostname\instancename` or `hostname\instancename,port` in the **Host** field.
1. Verify that the SQL Server Browser service is running on the SQL Server machine.
1. If the Browser service is unavailable, specify the port number directly: `hostname,port`.
1. Check that UDP port 1434 is open if using the SQL Server Browser service.

## Private data source connect (PDC) errors

These errors occur when using Grafana Cloud with Private data source connect to reach an on-premises or private-network SQL Server.

### "No such host" when connecting through PDC

**Error message:** `dial tcp: lookup sqlserver.internal.example.com: no such host`

**Cause:** The PDC agent cannot resolve the SQL Server hostname from its network. This commonly occurs when:

- The hostname in the data source configuration is a public DNS name that doesn't resolve inside the private network.
- The PDC agent host doesn't have access to the internal DNS server.
- A cluster or Availability Group Listener DNS entry is misconfigured.

**Solution:**

1. In the data source **Host** field, use the hostname as resolvable from the PDC agent's network (for example, the internal FQDN or IP address, not a public DNS name).
1. Verify the PDC agent host can resolve the hostname: run `nslookup <SQL_SERVER_HOST>` from the agent machine.
1. If using a cluster or Availability Group Listener, confirm the listener DNS record exists and resolves to the correct IP from the agent's network.
1. Check that the PDC agent host has the correct DNS servers configured.

### PDC connection breaks after maintenance or IP changes

**Cause:** The PDC SSH tunnel endpoint IP changed after infrastructure maintenance, breaking the established connection.

**Solution:**

1. Restart the PDC agent service to re-establish the SSH tunnel with the updated endpoint.
1. Verify the PDC agent status shows a healthy connection in **Connections** > **Private data source connect** in your Grafana Cloud stack.
1. If your network uses IP-based firewall rules for outbound traffic, update the allowlist. PDC endpoint IP addresses can change during maintenance windows.
1. Consider using DNS-based allowlisting instead of IP-based rules where possible to avoid future disruptions.

### Credentials not persisting with PDC enabled

**Cause:** A known issue can cause the data source username to not persist when PDC is enabled.

**Solution:**

1. After enabling PDC, save the data source configuration first, then re-enter the username and password and save again.
1. Click **Save & test** to verify the credentials are stored and the connection succeeds.
1. If the issue persists, try disabling PDC, saving credentials, then re-enabling PDC.

### Cannot connect to private SQL Server from Grafana Cloud

**Cause:** Attempting to connect to an on-premises or private-network SQL Server from Grafana Cloud without PDC configured.

**Solution:**

1. Grafana Cloud requires [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) for any SQL Server that isn't publicly accessible.
1. Firewall allowlisting alone doesn't work because Grafana Cloud data source connections don't use static IP addresses.
1. Install and configure a PDC agent on a host in your private network that can reach the SQL Server on port 1433.
1. Ensure the PDC agent host allows outbound connections on port 22 to the Grafana Cloud PDC endpoint.

For setup instructions, refer to [Connect to on-premises SQL Server from Grafana Cloud](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/configure/#connect-to-on-premises-sql-server-from-grafana-cloud).

## Authentication errors

These errors occur when there are issues with authentication credentials or permissions.

### Login failed for user

**Error message:** "Login failed for user 'username'" or "Authentication failed"

**Cause:** The authentication credentials are invalid or the user doesn't have permission to access the database.

**Solution:**

1. Verify that the username and password are correct.
1. Check that the user exists in SQL Server and is enabled.
1. Ensure the user has access to the specified database.
1. For Windows Authentication, verify that the credentials are in the correct format (`DOMAIN\User`).
1. Check that the SQL Server authentication mode allows the type of login you're using (SQL Server Authentication, Windows Authentication, or Mixed Mode).

### Login failed for user '' (empty username)

**Error message:** "Login failed for user ''". The saved username appears blank when Grafana connects to SQL Server.

**Cause:** A known issue on certain Grafana release channels caused the SQL Server Authentication username to appear saved in the UI but send an empty string to the server during connection. This was identified on the "fast" release channel.

**Solution:**

1. Switch to the "steady" release channel if you're on "fast" and encounter this issue.
1. After upgrading or switching channels, open the data source configuration, re-enter the username and password, and click **Save & test** to confirm the credentials are stored correctly.
1. If the issue persists after re-saving, clear your browser cache and try again.

### Connection fails when password contains special characters

**Error message:** "Login failed for user" or "Connection string parse error" when credentials contain semicolons (`;`) or closing braces (`}`)

**Cause:** In Grafana versions prior to v13.0, semicolons and closing braces in usernames or passwords were not properly escaped in the MSSQL connection string, causing authentication failures.

**Solution:**

1. Upgrade to Grafana v13.0 or later, which correctly handles these special characters.
1. If you cannot upgrade, change the SQL Server password to avoid `;` and `}` characters.

### Access denied to database

**Error message:** `Cannot open database 'dbname' requested by the login`

**Cause:** The authenticated user doesn't have permission to access the specified database.

**Solution:**

1. Verify that the database name is correct in the data source configuration.
1. Ensure the user is mapped to the database with appropriate permissions.
1. Grant at least `SELECT` permission on the required tables:

   ```sql
   USE [your_database]
   GRANT SELECT ON dbo.YourTable TO [your_user]
   ```

1. Check that the user doesn't have any conflicting permissions from the public role.

<!-- vale Grafana.Spelling = NO -->

### Windows Authentication (Kerberos) issues

**Error message:** "Kerberos authentication failed" or "Cannot initialize Kerberos"

**Cause:** Kerberos configuration is incorrect or incomplete.

**Solution:**

1. Verify that the Kerberos configuration file (`krb5.conf`) path is correct in the data source settings. The default path is `/etc/krb5.conf`.
1. For keytab authentication, ensure the keytab file exists and is readable by the Grafana service account.
1. Check that the realm and KDC settings are correct in the Kerberos configuration.
1. Verify DNS is correctly resolving the KDC servers.
1. Ensure the service principal name (SPN) is registered for the SQL Server instance.

{{< admonition type="note" >}}
Kerberos authentication is not supported in Grafana Cloud. Use SQL Server Authentication or Azure Entra ID instead.
{{< /admonition >}}

### `KDC_ERR_C_PRINCIPAL_UNKNOWN` with Availability Group listeners

**Error message:** "KDC_ERR_C_PRINCIPAL_UNKNOWN" when connecting through a SQL Server Availability Group Listener

**Cause:** The SPN registered for the SQL Server instance doesn't match the Availability Group Listener's DNS name. Kerberos authentication requires an SPN that matches the hostname the client connects to.

**Solution:**

1. Register an SPN for the Availability Group Listener DNS name:

   ```text
   setspn -S MSSQLSvc/<LISTENER_FQDN>:1433 <DOMAIN>\<SERVICE_ACCOUNT>
   setspn -S MSSQLSvc/<LISTENER_FQDN> <DOMAIN>\<SERVICE_ACCOUNT>
   ```

1. Verify the SPN is correctly registered: `setspn -L <DOMAIN>\<SERVICE_ACCOUNT>`.
1. Ensure `krb5.conf` includes the correct realm mapping for the listener's domain.
1. If using multiple realms, add cross-realm trust entries to `krb5.conf`.

### Untrusted domain errors with Windows Authentication

**Error message:** "The login is from an untrusted domain and cannot be used with Windows authentication"

**Cause:** The Grafana server's domain isn't trusted by the SQL Server's domain, or the Kerberos realm configuration doesn't map correctly between domains.

**Solution:**

1. Verify that a trust relationship exists between the Grafana server's domain and the SQL Server's domain.
1. Check that `krb5.conf` includes realm mappings for both domains:

   ```ini
   [realms]
   DOMAIN_A.COM = {
     kdc = kdc1.domain_a.com
   }
   DOMAIN_B.COM = {
     kdc = kdc1.domain_b.com
   }

   [domain_realm]
   .domain_a.com = DOMAIN_A.COM
   .domain_b.com = DOMAIN_B.COM
   ```

1. Ensure the SQL Server service account has delegation permissions if cross-domain access is needed.
1. As an alternative, use SQL Server Authentication or Azure Entra ID (App Registration), which don't require domain trust.

<!-- vale Grafana.Spelling = YES -->

### Azure Entra ID authentication errors

**Error message:** "AADSTS error codes" or "Azure AD authentication failed"

**Cause:** Azure Entra ID (formerly Azure AD) authentication is misconfigured.

**Solution:**

1. For **App Registration** authentication:
   - Verify the tenant ID, client ID, and client secret are correct.
   - Ensure the app registration has been added as a user in the Azure SQL database.
   - Check that the client secret hasn't expired.

1. For **Managed Identity** authentication:
   - Verify `managed_identity_enabled = true` is set in the Grafana server configuration.
   - Ensure the managed identity has been added to the Azure SQL database.
   - Confirm the Azure resource hosting Grafana has managed identity enabled.

1. For **Current User** authentication:
   - Ensure `user_identity_enabled = true` is set in the Grafana server configuration.
   - Verify the app registration is configured to issue both Access Tokens and ID Tokens.
   - Check that the required API permissions are configured (`user_impersonation` for Azure SQL).

For detailed Azure authentication configuration, refer to [Configure the Microsoft SQL Server data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/configure/).

## Query errors

These errors occur when there are issues with query syntax or configuration.

### Time column not found or invalid

**Error message:** "Could not find time column" or time series visualization shows no data

**Cause:** The query doesn't return a properly formatted `time` column for time series visualization.

**Solution:**

1. Ensure your query includes a column named `time` when using the **Time series** format.
1. Use the `$__time()` macro to rename your date column: `$__time(your_date_column)`.
1. Verify the time column is of a valid SQL date/time type (`datetime`, `datetime2`, `date`) or contains Unix epoch values.
1. Ensure the result set is sorted by the time column using `ORDER BY`.

### Macro expansion errors

**Error message:** "Error parsing query" or macros appear unexpanded in the query

**Cause:** Grafana macros are being used incorrectly.

**Solution:**

1. Verify macro syntax: use `$__timeFilter(column)` not `$_timeFilter(column)`.
1. Macros don't work inside stored procedures. Use explicit date parameters instead.
1. Check that the column name passed to macros exists in your table.
1. View the expanded query by clicking **Generated SQL** after running the query to debug macro expansion.

### Timezone and time shift issues

**Cause:** Time series data appears shifted or doesn't align with expected times.

**Solution:**

1. Store timestamps in UTC in your database to avoid timezone issues.
1. Time macros (`$__time`, `$__timeFilter`, etc.) always expand to UTC values.
1. If your timestamps are stored in local time, convert them to UTC in your query:

   ```sql
   SELECT
     your_datetime_column AT TIME ZONE 'Your Local Timezone' AT TIME ZONE 'UTC' AS time,
     value
   FROM your_table
   ```

1. Don't pass timezone parameters to time macros. They're not supported.

### Query returns too many rows

**Error message:** "Result set too large" or browser becomes unresponsive

**Cause:** The query returns more data than can be efficiently processed.

**Solution:**

1. Add time filters using `$__timeFilter(column)` to limit data to the dashboard time range.
1. Use aggregations (`AVG`, `SUM`, `COUNT`) with `GROUP BY` instead of returning raw rows.
1. Add a `TOP` clause to limit results: `SELECT TOP 1000 ...`.
1. Use the `$__timeGroup()` macro to aggregate data into time intervals.

### Stored procedure returns no data

**Cause:** Stored procedure output isn't being captured correctly.

**Solution:**

1. Ensure the stored procedure uses `SELECT` statements, not just variable assignments.
1. Remove `SET NOCOUNT ON` if present, or ensure it's followed by a `SELECT` statement.
1. Verify the stored procedure parameters are being passed correctly.
1. Test the stored procedure directly in SQL Server Management Studio with the same parameters.

For more information on using stored procedures, refer to the [query editor documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/query-editor/).

## Performance issues

These issues relate to slow queries or high resource usage.

### Slow query execution

**Cause:** Queries take a long time to execute.

**Solution:**

1. Reduce the dashboard time range to limit data volume.
1. Add indexes to columns used in `WHERE` clauses and time filters.
1. Use aggregations instead of returning individual rows.
1. Increase the **Min time interval** setting to reduce the number of data points.
1. Review the query execution plan in SQL Server Management Studio to identify bottlenecks.

### Connection pool exhaustion

**Error message:** "Too many connections", "Connection pool exhausted", or "failed to connect to server"

**Cause:** Too many concurrent connections to the database. This commonly manifests as intermittent alert failures while dashboards continue to work, because alert evaluations and dashboard queries compete for connections from the same pool.

**Solution:**

1. Increase the **Max open** connection limit in the data source configuration. A value of `50` to `100` handles most deployments with concurrent dashboards and alerts.
1. Enable **Auto max idle** to automatically manage idle connections.
1. Set **Max lifetime** to `14400` (4 hours) to ensure stale connections are recycled.
1. Reduce the number of panels querying the same data source simultaneously, or split heavy dashboards into multiple pages.
1. Check for long-running queries that might be holding connections open (review SQL Server's `sys.dm_exec_sessions`).
1. If alerts fail intermittently but dashboards work, this is a strong indicator of pool exhaustion under concurrent load.

### Slow dashboard load with read-only replicas

**Symptoms:** Dashboards take 5 to 6 minutes to load initially, or queries are significantly slower than expected.

**Cause:** The data source is configured with `ApplicationIntent=ReadOnly` in the **Host** field, which routes all queries to a read-only secondary replica. If the replica is undersized, under-indexed, or experiencing replication lag, queries run much slower than on the primary.

**Solution:**

1. Remove `ApplicationIntent=ReadOnly` from the Host field to route queries back to the primary instance.
1. If you need to use a read-only replica, ensure it has the same indexes and resources as the primary.
1. Check the replica's replication lag and resource utilization (CPU, I/O) during slow periods.
1. Consider creating a separate data source for replica queries and using it only for specific dashboards that don't require low latency.

## Alert errors

These errors occur when using Microsoft SQL Server queries in Grafana Alerting.

### Alerts fail intermittently while dashboards work

**Cause:** Connection pool exhaustion. Alert evaluations and dashboard queries share the same connection pool. Under concurrent load, alert evaluations may time out waiting for a free connection.

**Solution:**

Refer to [Connection pool exhaustion](#connection-pool-exhaustion) in the Performance issues section above.

### Alert queries return no data

**Cause:** Alert queries have different requirements than dashboard queries.

**Solution:**

1. Ensure the query format is set to **Time series** (Table format isn't supported for alerting).
1. Verify the query doesn't use template variables (alert queries can't resolve dashboard variables).
1. Check that the query returns data within the alert evaluation time range.
1. Test the query in Explore with a fixed time range to confirm it returns results.

For more information, refer to [Microsoft SQL Server alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/alerting/).

## Template variable errors

These errors occur when using template variables in queries.

### Template variable queries fail

**Cause:** Variable queries return unexpected results or errors.

**Solution:**

1. Verify the variable query syntax is valid SQL that returns a single column.
1. Check that the data source connection is working.
1. Ensure the user has permission to access the tables referenced in the variable query.
1. Test the query in the query editor before using it as a variable query.

### Double-quoting breaks queries with multi-value variables

**Cause:** Since Grafana v11.3, multi-value variables used with `IN` are automatically quoted. If you manually wrapped the variable in quotes (for example, `WHERE col IN ('${var}')`), values are now double-quoted (for example, `''value''`), causing query failures.

**Solution:**

1. Remove manual quotes around the variable: use `WHERE col IN ($var)` instead of `WHERE col IN ('${var}')`.
1. For single-value comparisons, use the `sqlstring` format: `WHERE col = ${var:sqlstring}`.
1. Refer to [Template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/template-variables/) for formatting options.

## Other common issues

The following issues don't produce specific error messages but are commonly encountered.

### System databases appear in queries

**Cause:** Queries accidentally access system databases.

**Solution:**

1. The query editor automatically excludes `tempdb`, `model`, `msdb`, and `master` from the database dropdown.
1. Always specify the database in your data source configuration to restrict access.
1. Ensure the database user only has permissions on the intended database.

### Data appears incorrect or misaligned

**Cause:** Data formatting or type conversion issues.

**Solution:**

1. Use explicit column aliases to ensure consistent naming: `SELECT value AS metric`.
1. Verify numeric columns are actually numeric types, not strings.
1. Check for `NULL` values that might affect aggregations.
1. Use the `FILL` option in `$__timeGroup()` macro to handle missing data points.

## Get additional help

If you continue to experience issues after following this troubleshooting guide:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs.
1. Enable debug logging in Grafana to capture detailed error information.
1. Check SQL Server logs for additional error details.
1. Contact [Grafana Support](https://grafana.com/contact/) if you're an Enterprise or Cloud customer.

When reporting issues, include:

- Grafana version
- Microsoft SQL Server data source plugin version, found on the **Administration > Plugins** page. The plugin updates independently of Grafana releases.
- SQL Server version
- Error messages (redact sensitive information)
- Steps to reproduce
- Relevant query examples (redact sensitive data)

# MySQL alerting

You can use Grafana Alerting with MySQL to create alerts based on your MySQL data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met.

For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/).

## Before you begin

Before creating alerts with MySQL, ensure you have:

- A MySQL data source configured in Grafana.
- Appropriate permissions to create alert rules.
- Understanding of the metrics you want to monitor.

## Supported query types

MySQL alerting works with **time series queries** that return numeric data over time. Table formatted queries are not supported in alert rule conditions.

To create a valid alert query:

- Include a `time` column that returns a SQL datetime or UNIX epoch timestamp
- Return numeric values for the metrics you want to alert on
- Sort results by the time column

For more information on writing time series queries, refer to [MySQL query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mysql/query-editor/).

### Query format requirements

| Query format | Alerting support | Notes                                    |
| ------------ | ---------------- | ---------------------------------------- |
| Time series  | Yes              | Required for alerting                    |
| Table        | No               | Convert to time series format for alerts |

## Create an alert rule

To create an alert rule using MySQL:

1. Navigate to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for the alert rule.
1. Select your **MySQL** data source.
1. Build your query using the query editor:
   - Set the **Format** to **Time series**
   - Include a time column using the `$__time()` or `$__timeGroup()` macro
   - Add numeric columns for the values to monitor
   - Use `$__timeFilter()` to filter data by the dashboard time range
1. Configure the alert condition (for example, when the average is above a threshold).
1. Set the evaluation interval and pending period.
1. Configure notifications and labels.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example alert queries

The following examples show common alerting scenarios with MySQL.

### Alert on high error count

Monitor the number of errors over time:

```sql
SELECT
  $__timeGroup(created_at, '1m') AS time,
  COUNT(*) AS error_count
FROM error_logs
WHERE $__timeFilter(created_at)
  AND level = 'error'
GROUP BY time
ORDER BY time
```

**Condition:** When error_count is above 100.

### Alert on average response time

Monitor API response times:

```sql
SELECT
  $__timeGroup(request_time, '5m') AS time,
  AVG(response_time_ms) AS avg_response_time
FROM api_requests
WHERE $__timeFilter(request_time)
GROUP BY time
ORDER BY time
```

**Condition:** When avg_response_time is above 500 (milliseconds).

### Alert on low order volume

Detect drops in order activity:

```sql
SELECT
  $__timeGroup(order_date, '1h') AS time,
  COUNT(*) AS order_count
FROM orders
WHERE $__timeFilter(order_date)
GROUP BY time
ORDER BY time
```

**Condition:** When order_count is below 10.

### Alert on disk usage percentage

Monitor database storage metrics:

```sql
SELECT
  $__timeGroup(recorded_at, '5m') AS time,
  AVG(disk_used_percent) AS disk_usage
FROM system_metrics
WHERE $__timeFilter(recorded_at)
  AND metric_type = 'disk'
GROUP BY time
ORDER BY time
```

**Condition:** When disk_usage is above 85.

## Limitations

When using MySQL with Grafana Alerting, be aware of the following limitations:

### Template variables not supported

Alert queries cannot contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$hostname` or `$environment` won't be resolved.

If your dashboard query uses template variables, create a separate query for alerting with hard coded values.

### Table format not supported

Queries using the **Table** format cannot be used for alerting. Set the query format to **Time series** and ensure your query returns a time column.

### Query timeout

Complex queries with large datasets may timeout during alert evaluation. Optimize queries for alerting by:

- Adding appropriate `WHERE` clauses to limit data
- Using indexes on time and filter columns
- Reducing the time range evaluated

## Best practices

Follow these best practices when creating MySQL alerts:

- **Use time series format:** Always set the query format to Time series for alert queries.
- **Include time filters:** Use the `$__timeFilter()` macro to limit data to the evaluation window.
- **Optimize queries:** Add indexes on columns used in `WHERE` clauses and `GROUP BY`.
- **Test queries first:** Verify your query returns expected results in Explore before creating an alert.
- **Set realistic thresholds:** Base alert thresholds on historical data patterns.
- **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor.

# MySQL annotations

Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics.
You can use MySQL as a data source for annotations to display events such as deployments, alerts, or other significant occurrences on your visualizations.

For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/).

## Before you begin

Before creating MySQL annotations, ensure you have:

- A MySQL data source configured in Grafana.
- Tables containing event data with timestamp fields.
- Read access to the tables containing your events.

## Create an annotation query

To add a MySQL annotation to your dashboard:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Annotation query**.
1. Enter a name for the annotation query.
1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox.
1. Select a color for the annotation event markers.
1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed.
1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed.
1. Click **Open query editor** to open the **Annotation Query** dialog box.
1. Select your **MySQL** data source from the **Data source** drop-down list.
1. Configure the annotation query and field mappings.
1. (Optional) Click **Test annotation query** to ensure that the query is working properly.
1. Click **Close** when you've completed the query setup.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## Query columns

Your annotation query must return a `time` column and can optionally include `timeend`, `text`, and `tags` columns.

| Column    | Required | Description                                                                                   |
| --------- | -------- | --------------------------------------------------------------------------------------------- |
| `time`    | Yes      | The timestamp for the annotation. Can be a SQL datetime or UNIX epoch value.                  |
| `timeend` | No       | The end timestamp for range annotations. Creates a shaded region instead of a vertical line.  |
| `text`    | No       | The annotation description displayed when you hover over the annotation.                      |
| `tags`    | No       | Tags for the annotation as a comma-separated string. Helps categorize and filter annotations. |

## Example queries

The following examples show common annotation query patterns.

### Basic annotation with epoch time

Display events using UNIX epoch timestamps:

```sql
SELECT
  epoch_time as time,
  description as text,
  CONCAT(tag1, ',', tag2) as tags
FROM events
WHERE $__unixEpochFilter(epoch_time)
```

### Annotation with a single tag

Display events with a single tag value:

```sql
SELECT
  epoch_time as time,
  message as text,
  category as tags
FROM event_log
WHERE $__unixEpochFilter(epoch_time)
```

### Range annotation with start and end time

Display events with duration as shaded regions:

```sql
SELECT
  start_time as time,
  end_time as timeend,
  description as text,
  CONCAT(type, ',', severity) as tags
FROM incidents
WHERE $__unixEpochFilter(start_time)
```

### Annotation with native SQL datetime

Display events using native MySQL datetime columns:

```sql
SELECT
  event_date as time,
  message as text,
  CONCAT(category, ',', priority) as tags
FROM system_events
WHERE $__timeFilter(event_date)
```

### Deployment annotations

Display deployment events:

```sql
SELECT
  deployed_at as time,
  CONCAT('Deployed ', version, ' to ', environment) as text,
  environment as tags
FROM deployments
WHERE $__timeFilter(deployed_at)
```

### Maintenance window annotations

Display maintenance windows as range annotations:

```sql
SELECT
  start_time as time,
  end_time as timeend,
  CONCAT('Maintenance: ', description) as text,
  'maintenance' as tags
FROM maintenance_windows
WHERE $__timeFilter(start_time)
```

## Macros

Use these macros in your annotation queries to filter by the dashboard time range:

| Macro                        | Description                                                      |
| ---------------------------- | ---------------------------------------------------------------- |
| `$__timeFilter(column)`      | Filters by time range using a native SQL datetime column.        |
| `$__unixEpochFilter(column)` | Filters by time range using a column with UNIX epoch timestamps. |

## Best practices

Follow these best practices when creating MySQL annotations:

- **Use time filters:** Always include `$__timeFilter()` or `$__unixEpochFilter()` to limit results to the dashboard time range.
- **Keep queries efficient:** Add indexes on time columns and filter columns to improve query performance.
- **Use meaningful text:** Include descriptive information in the `text` column to make annotations useful.
- **Organize with tags:** Use consistent tag values to categorize annotations and enable filtering.
- **Test queries first:** Verify your query returns expected results in Explore before adding it as an annotation.

# MySQL template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
Grafana displays these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
Grafana refers to such variables as **template variables**.

For an introduction to templating and template variables, refer to [Templating](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/).

## Query variable

A query variable in Grafana dynamically retrieves values from your data source using a query. With a query variable, you can write a SQL query that returns values such as measurement names, key names, or key values that are shown in a drop-down select box.

The variable query editor supports the same **Builder** and **Code** modes as the main query editor. Use Builder mode to construct queries visually by selecting a dataset, table, columns, and filters. Use Code mode to write SQL directly.

For example, the following query returns all values from the `hostname` column:

```sql
SELECT hostname FROM my_host
```

A query can return multiple columns, and Grafana automatically generates a list using the values from those columns. For example, the following query returns values from both the `hostname` and `hostname2` columns, which are included in the variable's drop-down list.

```sql
SELECT my_host.hostname, my_other_host.hostname2 FROM my_host JOIN my_other_host ON my_host.city = my_other_host.city
```

To use time range dependent macros like `$__timeFilter(column)` in your query, you must set the template variable's refresh mode to **On Time Range Change**.

```sql
SELECT event_name FROM event_log WHERE $__timeFilter(time_column)
```

### Key/value variables

You can create a key/value variable so the drop-down shows a user-friendly label (for example, hostname) while panel queries use a different value (for example, ID). Use the variable editor's **Value field** and **Text field** at the bottom of the query section to specify which query columns supply the value and the label. Your query can use any column names; you don't need `__value` or `__text` in the SQL.

Example: run a query that returns `hostname` and `id`, then set **Text field** to `hostname` and **Value field** to `id`.

```sql
SELECT hostname, id FROM my_host
```

Note that the values in the text column should be unique. If there are duplicates, Grafana uses only the first matching entry.

Alternatively, you can use the legacy approach: return columns named `__text` and `__value` in your query (for example, `SELECT hostname AS __text, id AS __value FROM my_host`).

### Nested variables

You can create nested variables, where one variable depends on the value of another. For example, if you have a variable named `region`, you can configure a `hosts` variable to only show hosts from the selected region. If `region` is a multi-value variable, use the `IN` operator instead of `=` to match against multiple selected values.

```sql
SELECT hostname FROM my_host WHERE region IN($region)
```

### Filter results with `__searchFilter`

Using `__searchFilter` in the query field allows the query results to be filtered based on the user's input in the drop-down selection box. If you don't enter anything, the default value for `__searchFilter` is `%`.

Note that you must enclose the `__searchFilter` expression in quotes as Grafana doesn't add them automatically.

The following example demonstrates how to use `__searchFilter` in the query field to enable real-time searching for `hostname` as the user types in the drop-down selection box.

```sql
SELECT hostname FROM my_host WHERE hostname LIKE '$__searchFilter'
```

## Use variables in queries

Grafana automatically quotes template variable values only when the template variable is a `multi-value`.

When using a multi-value variable, use the `IN` comparison operator instead of `=` to match against multiple values.

Grafana supports two syntaxes for using variables in queries:

- **`$<varname>` syntax**

Example with a template variable named `hostname`:

```sql
SELECT
  UNIX_TIMESTAMP(atimestamp) as time,
  aint as value,
  avarchar as metric
FROM my_table
WHERE $__timeFilter(atimestamp) and hostname in($hostname)
ORDER BY atimestamp ASC
```

- **`[[varname]]` syntax**

Example with a template variable named `hostname`:

```sql
SELECT
  UNIX_TIMESTAMP(atimestamp) as time,
  aint as value,
  avarchar as metric
FROM my_table
WHERE $__timeFilter(atimestamp) and hostname in([[hostname]])
ORDER BY atimestamp ASC
```

### Disable quoting for multi-value variables

By default, Grafana formats multi-value variables as a quoted, comma-separated string. For example, if `server01` and `server02` are selected, the result will be `'server01'`, `'server02'`. To disable quoting, use the `csv` formatting option for variables:

```text
${servers:csv}
```

This outputs the values as an unquoted comma-separated list.

### Use explicit quoting for string variables

Don't rely on implicit quoting behavior for string variables in SQL queries. Quoting behavior can vary depending on panel type and context:

- **Repeat panels** may not quote single-value variables at all, causing bare values to appear in the SQL and breaking the query.
- If you manually wrap a variable in quotes (for example, `WHERE name = '$myvar'`) and Grafana also applies its own quoting, the value gets double-quoted (for example, `''30''` instead of `'30'`). This intentional quoting behavior was restored in Grafana 11.3.

To avoid both problems, use the `sqlstring` format option. It handles escaping and quoting in a single step, so you don't add your own quotes around the variable:

```sql
SELECT *
FROM my_table
WHERE hostname = ${hostname:sqlstring}
```

This produces a safely quoted string (for example, `'server01'`) regardless of panel type or context. For multi-value variables, use `IN` with `${var:sqlstring}`:

```sql
SELECT *
FROM my_table
WHERE hostname IN (${hostname:sqlstring})
```

{{< admonition type="caution" >}}
Don't wrap `${var:sqlstring}` in additional quotes. The `sqlstring` formatter already produces a quoted value. Writing `'${var:sqlstring}'` results in double quoting.
{{< /admonition >}}

Refer to [Advanced variable format options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/#advanced-variable-format-options) for additional information.

# Troubleshoot MySQL data source issues

This document provides solutions to common issues you may encounter when configuring or using the MySQL data source in Grafana.

## Connection errors

These errors occur when Grafana cannot establish or maintain a connection to the MySQL server.

### Unable to connect to the server

**Error message:** "dial tcp: connection refused" or "Could not connect to MySQL"

**Cause:** Grafana cannot establish a network connection to the MySQL server.

**Solution:**

1. Verify that the MySQL server is running and accessible.
1. Check that the host and port are correct in the data source configuration. The default MySQL port is `3306`.
1. Ensure there are no firewall rules blocking the connection between Grafana and MySQL.
1. Verify that MySQL is configured to allow remote connections by checking the `bind-address` setting in your MySQL configuration.
1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your MySQL instance isn't publicly accessible.

### Connection timeout

**Error message:** "Connection timed out" or "I/O timeout"

**Cause:** The connection to MySQL timed out before receiving a response.

**Solution:**

1. Check the network latency between Grafana and MySQL.
1. Verify that MySQL isn't overloaded or experiencing performance issues.
1. Check if any network devices (load balancers, proxies) are timing out the connection.
1. Increase the `wait_timeout` setting in MySQL if connections are timing out during idle periods.

### TLS/SSL connection failures

TLS errors occur when the MySQL server requires or expects encrypted connections but the Grafana data source isn't configured to match. Common error messages include:

- "TLS handshake failed" or "x509: certificate verify failed"
- "Connection refused" or connection drops immediately
- "Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON"

**Possible causes and solutions:**

| Cause                                                                      | Solution                                                                                                                                                                                     |
| -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| MySQL has `require_secure_transport` enabled or AWS RDS Proxy enforces TLS | Enable **With CA Cert** or **Skip TLS Verification** in the data source configuration. Either option establishes a TLS connection.                                                           |
| Self-signed or private CA certificate isn't trusted                        | Enable **With CA Cert** and provide the root certificate under **TLS/SSL Root Certificate**.                                                                                                 |
| Certificate has expired or doesn't match the hostname                      | Renew the certificate, or enable **Skip TLS Verification** for development environments only.                                                                                                |
| TLS handshake fails against AWS RDS or RDS Proxy                           | Provide the [Amazon RDS root CA certificate](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html) under **TLS/SSL Root Certificate** with **With CA Cert** enabled. |
| Organization requires mutual TLS (mTLS)                                    | Enable **Use TLS Client Auth** and provide both a client certificate and key in addition to the CA certificate.                                                                              |

{{< admonition type="note" >}}
**Skip TLS Verification** and **With CA Cert** both establish encrypted connections. You don't need **Use TLS Client Auth** unless your setup requires mutual TLS. For most AWS RDS and RDS Proxy connections, **With CA Cert** with the Amazon root CA is sufficient.
{{< /admonition >}}

For provisioned data sources, enable TLS in `jsonData`:

```yaml
jsonData:
  tlsAuthWithCACert: true
secureJsonData:
  tlsCACert: ${CA_CERT}
```

To skip certificate validation instead, use:

```yaml
jsonData:
  tlsSkipVerify: true
```

For all available TLS provisioning options, refer to the [TLS provisioning examples](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mysql/configure/#mysql-provisioning-examples).

### Connection reset by peer

**Error message:** "Connection reset by peer" or "EOF"

**Cause:** The MySQL server closed the connection unexpectedly.

**Solution:**

1. Check the `max_connections` setting on the MySQL server to ensure it isn't being exceeded.
1. Verify the `wait_timeout` and `interactive_timeout` settings in MySQL aren't set too low.
1. Increase the **Max lifetime** setting in Grafana's data source configuration to be lower than the MySQL `wait_timeout`.
1. Check MySQL server logs for any errors or connection-related messages.

## Authentication errors

These errors occur when there are issues with authentication credentials or permissions.

### Access denied for user

**Error message:** "Access denied for user 'username'@'host'" or "Authentication failed"

**Cause:** The authentication credentials are invalid or the user doesn't have permission to connect from the Grafana server's host.

**Solution:**

1. Verify that the username and password are correct.
1. Check that the user exists in MySQL and is enabled.
1. Ensure the user has permission to connect from the Grafana server's IP address. MySQL restricts access based on the connecting host:

   ```sql
   SELECT user, host FROM mysql.user WHERE user = 'your_user';
   ```

1. If necessary, create a user that can connect from the Grafana server:

   ```sql
   CREATE USER 'grafana'@'grafana_server_ip' IDENTIFIED BY 'password';
   ```

1. If using the `mysql_native_password` authentication plugin, ensure it's enabled on the server.

### Cannot access database

**Error message:** "Access denied for user 'username'@'host' to database 'dbname'"

**Cause:** The authenticated user doesn't have permission to access the specified database.

**Solution:**

1. Verify that the database name is correct in the data source configuration.
1. Ensure the user has the required permissions on the database:

   ```sql
   GRANT SELECT ON your_database.* TO 'grafana'@'grafana_server_ip';
   FLUSH PRIVILEGES;
   ```

1. For production environments, grant permissions only on specific tables:

   ```sql
   GRANT SELECT ON your_database.your_table TO 'grafana'@'grafana_server_ip';
   ```

### PAM authentication issues

**Error message:** "Authentication plugin 'auth_pam' cannot be loaded" or cleartext password errors

**Cause:** PAM (Pluggable Authentication Modules) authentication requires cleartext password transmission.

**Solution:**

1. Enable **Allow Cleartext Passwords** in the data source configuration if using PAM authentication.
1. Ensure TLS is enabled to protect password transmission when using cleartext passwords.
1. Verify that the PAM plugin is correctly installed and configured on the MySQL server.

## Query errors

These errors occur when there are issues with query syntax or configuration.

### Time column not found or invalid

**Error message:** "Could not find time column" or time series visualization shows no data

**Cause:** The query doesn't return a properly formatted `time` column for time series visualization.

**Solution:**

1. Ensure your query includes a column named `time` when using the **Time series** format.
1. Use the `$__time()` macro to convert your date column: `$__time(your_date_column)`.
1. Verify the time column is of a valid MySQL date/time type (`DATETIME`, `TIMESTAMP`, `DATE`) or contains Unix epoch values.
1. Ensure the result set is sorted by the time column using `ORDER BY`.

### Macro expansion errors

**Error message:** "Error parsing query" or macros appear unexpanded in the query

**Cause:** Grafana macros are being used incorrectly.

**Solution:**

1. Verify macro syntax: use `$__timeFilter(column)` not `$_timeFilter(column)`.
1. Check that the column name passed to macros exists in your table.
1. View the expanded query by clicking **Generated SQL** after running the query to debug macro expansion.
1. Ensure backticks are used for reserved words or special characters in column names: `$__timeFilter(\`time-column\`)`.

### Hash character in quoted strings causes query errors

**Error message:** Query returns unexpected results or errors when string values contain `#`

**Cause:** In versions before Grafana 13.0, the SQL comment stripping logic incorrectly treated `#` inside quoted strings (for example, `WHERE color = '#FF0000'`) as a comment delimiter, which stripped the rest of the line.

**Solution:**

1. Upgrade to Grafana 13.0 or later, which preserves `#` inside quoted strings.
1. As a workaround on older versions, use the `CONCAT` function or a hexadecimal literal to avoid the `#` character in string literals.

### Timezone and time shift issues

**Cause:** Time series data appears shifted or doesn't align with expected times.

**Solution:**

1. Store timestamps in UTC in your database to avoid timezone issues.
1. Time macros (`$__time`, `$__timeFilter`, etc.) always expand to UTC values.
1. Set the **Session Timezone** in the data source configuration to match your data's timezone, or use `+00:00` for UTC.
1. If your timestamps are stored in local time, convert them to UTC in your query:

   ```sql
   SELECT
     CONVERT_TZ(your_datetime_column, 'Your/Timezone', 'UTC') AS time,
     value
   FROM your_table
   ```

### Query returns too many rows

**Error message:** "Result set too large" or browser becomes unresponsive

**Cause:** The query returns more data than can be efficiently processed.

**Solution:**

1. Add time filters using `$__timeFilter(column)` to limit data to the dashboard time range.
1. Use aggregations (`AVG`, `SUM`, `COUNT`) with `GROUP BY` instead of returning raw rows.
1. Add a `LIMIT` clause to restrict results: `SELECT ... LIMIT 1000`.
1. Use the `$__timeGroup()` macro to aggregate data into time intervals.

### Syntax error in SQL statement

**Error message:** "You have an error in your SQL syntax" followed by specific error details

**Cause:** The SQL query contains invalid syntax.

**Solution:**

1. Check for missing or extra commas, parentheses, or quotes.
1. Ensure reserved words used as identifiers are enclosed in backticks: `` `table` ``, `` `select` ``.
1. Verify that template variable syntax is correct: `$variable` or `${variable}`.
1. Test the query directly in a MySQL client to isolate Grafana-specific issues.

### Unknown column in field list

**Error message:** "Unknown column 'column_name' in 'field list'"

**Cause:** The specified column doesn't exist in the table or is misspelled.

**Solution:**

1. Verify the column name is spelled correctly.
1. Check that the column exists in the specified table.
1. If the column name contains special characters or spaces, enclose it in backticks: `` `column-name` ``.
1. Ensure the correct database is selected if you're referencing columns without the full table path.

### `CASE` expression fails on non-standard string column types

**Error message:** Query with `CASE WHEN` returns an error or unexpected results in Grafana, but works in other MySQL clients

**Cause:** The Grafana MySQL driver doesn't handle all MySQL string subtypes (such as `ENUM`, `SET`, `TINYTEXT`, or `MEDIUMTEXT`) in `CASE` expressions. The driver may fail to infer the result type correctly.

**Solution:**

1. Use `CAST()` to normalize the column to `CHAR` before using it in a `CASE` expression:

   ```sql
   SELECT
     CASE WHEN CAST(status_column AS CHAR) = 'active' THEN 1 ELSE 0 END AS is_active
   FROM my_table
   ```

1. This is a known limitation of the Go MySQL driver used by Grafana. Explicitly casting to `CHAR` ensures consistent type handling.

## Performance issues

These issues relate to slow queries or high resource usage.

### Slow query execution

**Cause:** Queries take a long time to execute.

**Solution:**

1. Reduce the dashboard time range to limit data volume.
1. Add indexes to columns used in `WHERE` clauses and time filters:

   ```sql
   CREATE INDEX idx_time ON your_table(time_column);
   ```

1. Use aggregations instead of returning individual rows.
1. Increase the **Min time interval** setting to reduce the number of data points.
1. Review the query execution plan using `EXPLAIN` to identify bottlenecks:

   ```sql
   EXPLAIN SELECT * FROM your_table WHERE time_column > NOW() - INTERVAL 1 HOUR;
   ```

### Connection pool exhaustion

**Error message:** "Too many connections" or "Connection pool exhausted"

**Cause:** Too many concurrent connections to the database.

**Solution:**

1. Increase the **Max open** connection limit in the data source configuration.
1. Enable **Auto (max idle)** to automatically manage idle connections.
1. Reduce the number of panels querying the same data source simultaneously.
1. Check for long-running queries that might be holding connections.
1. Increase the `max_connections` setting in MySQL if necessary:

   ```sql
   SHOW VARIABLES LIKE 'max_connections';
   SET GLOBAL max_connections = 200;
   ```

### Query timeout

**Error message:** "Query execution was interrupted" or "Lock wait timeout exceeded"

**Cause:** The query takes too long and exceeds the configured timeout.

**Solution:**

1. Optimize the query by adding appropriate indexes.
1. Reduce the amount of data being queried by narrowing the time range.
1. Use aggregations to reduce the result set size.
1. Check for table locks that might be blocking the query.

## Other common issues

The following issues don't produce specific error messages but are commonly encountered.

### Template variable queries fail

**Cause:** Variable queries return unexpected results or errors.

**Solution:**

1. Verify the variable query syntax is valid SQL that returns a single column.
1. Check that the data source connection is working.
1. Ensure the user has permission to access the tables referenced in the variable query.
1. Test the query in the query editor before using it as a variable query.

### Data appears incorrect or misaligned

**Cause:** Data formatting or type conversion issues.

**Solution:**

1. Use explicit column aliases to ensure consistent naming: `SELECT value AS metric`.
1. Verify numeric columns are actually numeric types, not strings.
1. Check for `NULL` values that might affect aggregations.
1. Use the `FILL` option in `$__timeGroup()` macro to handle missing data points.

### Special characters in database or table names

**Cause:** Queries fail when tables or databases contain reserved words or special characters.

**Solution:**

1. Enclose identifiers with special characters in backticks: `` `my-database`.`my-table` ``.
1. The query editor automatically handles this for selections, but manual queries require backticks.
1. Avoid using reserved words as identifiers when possible.

### Query caching can't be enabled on a provisioned data source

**Cause:** When a MySQL data source is provisioned via YAML, the query caching toggle in the UI is greyed out. YAML provisioning doesn't currently support setting caching configuration.

**Solution:**

1. Use the Grafana HTTP API to enable query caching on the provisioned data source. Send a `PUT` request to update the data source with caching enabled:

   ```sh
   curl -X PUT -H "Content-Type: application/json" \
     -H "Authorization: Bearer <API_TOKEN>" \
     "https://<GRAFANA_URL>/api/datasources/<DATASOURCE_ID>" \
     -d '{
       "jsonData": {
         "queryCachingTTL": 300
       }
     }'
   ```

   Replace `<API_TOKEN>` with a valid Grafana API token, `<GRAFANA_URL>` with your Grafana instance URL, and `<DATASOURCE_ID>` with the numeric ID of the data source. Set `queryCachingTTL` to the desired cache duration in seconds.

1. Retrieve the data source ID by calling `GET /api/datasources/name/<DATA_SOURCE_NAME>`.
1. YAML-based caching configuration is an open feature request. Check [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for current status.

{{< admonition type="note" >}}
Query caching is available in Grafana Enterprise and Grafana Cloud. For more information, refer to [Query and resource caching](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/#query-and-resource-caching).
{{< /admonition >}}

### An unexpected error happened

**Error message:** "An unexpected error happened"

**Cause:** A general error occurred that doesn't have a specific error message.

**Solution:**

1. Check the Grafana server logs for more details about the error.
1. Verify all data source configuration settings are correct.
1. Test the connection using the **Save & test** button.
1. Ensure the MySQL server is accessible and responding to queries.
1. For Grafana Cloud customers, contact support for assistance.

## Get additional help

If you continue to experience issues after following this troubleshooting guide:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs.
1. Enable debug logging in Grafana to capture detailed error information.
1. Check MySQL error logs for additional details.
1. Contact Grafana Support if you're an Enterprise or Cloud customer.

When reporting issues, include:

- Grafana version
- MySQL data source plugin version, found on the **Administration > Plugins** page. The plugin updates independently of Grafana releases.
- MySQL version
- Error messages (redact sensitive information)
- Steps to reproduce
- Relevant query examples (redact sensitive data)

# OpenTSDB alerting

You can use Grafana Alerting with OpenTSDB to create alerts based on your time series data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met.

For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/).

## Before you begin

Before creating alerts with OpenTSDB, ensure you have:

- An OpenTSDB data source configured in Grafana. Refer to [Configure the OpenTSDB data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/configure/).
- Appropriate permissions to create alert rules.
- Understanding of the metrics you want to monitor.

## Supported features

OpenTSDB alerting works with standard metric queries that return time series data. The following table summarizes alerting compatibility:

| Query type                    | Alerting support | Notes                                |
| ----------------------------- | ---------------- | ------------------------------------ |
| Metrics with aggregation      | Yes              | Recommended for alerting             |
| Metrics with downsampling     | Yes              | Use appropriate intervals            |
| Metrics with rate calculation | Yes              | Useful for counter metrics           |
| Metrics with filters/tags     | Yes              | Filter to specific hosts or services |

## Create an alert rule

To create an alert rule using OpenTSDB:

1. Navigate to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for the alert rule.
1. Select your **OpenTSDB** data source.
1. Build your query:
   - Select the metric to monitor.
   - Choose an appropriate aggregator (for example, `avg`, `sum`, `max`).
   - Add tag filters to target specific resources.
   - Enable downsampling with an interval matching your evaluation frequency.
1. Configure the alert condition (for example, when the value is above a threshold).
1. Set the evaluation interval and pending period.
1. Configure notifications and labels.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example alert queries

The following examples show common alerting scenarios with OpenTSDB.

### Alert on high CPU usage

Monitor CPU usage and alert when it exceeds 90%:

| Field                 | Value          |
| --------------------- | -------------- |
| Metric                | `sys.cpu.user` |
| Aggregator            | `avg`          |
| Tags                  | `host=*`       |
| Downsample Interval   | `1m`           |
| Downsample Aggregator | `avg`          |

**Condition:** When average is above `90`

### Alert on low disk space

Monitor available disk space and alert when it drops below a threshold:

| Field                 | Value               |
| --------------------- | ------------------- |
| Metric                | `sys.disk.free`     |
| Aggregator            | `min`               |
| Tags                  | `host=*`, `mount=/` |
| Downsample Interval   | `5m`                |
| Downsample Aggregator | `min`               |

**Condition:** When minimum is below `10737418240` (10 GB in bytes)

### Alert on high network traffic rate

Monitor network bytes received and alert on high traffic:

| Field                 | Value                |
| --------------------- | -------------------- |
| Metric                | `net.bytes.received` |
| Aggregator            | `sum`                |
| Tags                  | `host=webserver01`   |
| Rate                  | enabled              |
| Counter               | enabled              |
| Downsample Interval   | `1m`                 |
| Downsample Aggregator | `avg`                |

**Condition:** When sum is above `104857600` (100 MB/s in bytes)

### Alert on error count spike

Monitor application error counts:

| Field                 | Value                           |
| --------------------- | ------------------------------- |
| Metric                | `app.errors.count`              |
| Aggregator            | `sum`                           |
| Tags                  | `service=api`, `env=production` |
| Downsample Interval   | `5m`                            |
| Downsample Aggregator | `sum`                           |

**Condition:** When sum is above `100`

## Limitations

When using OpenTSDB with Grafana Alerting, be aware of the following limitations.

### Template variables not supported

Alert queries can't contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$hostname` or `$environment` aren't resolved.

If your dashboard query uses template variables, create a separate query for alerting with hard-coded values.

### Query complexity

Complex queries with many tags or long time ranges may timeout or fail to evaluate. Simplify queries for alerting by:

- Using specific tag filters instead of wildcards where possible.
- Enabling downsampling with appropriate intervals.
- Reducing the evaluation time range.

### OpenTSDB 2.4 behavior

When using OpenTSDB 2.4 with alerting, Grafana executes queries with the parameter `arrays=true`. This causes OpenTSDB to return data points as an array of arrays instead of a map of key-value pairs. Grafana automatically converts this data to the appropriate format.

## Best practices

Follow these best practices when creating OpenTSDB alerts:

- **Use specific tag filters:** Add tag filters to focus on relevant resources and improve query performance.
- **Match downsample interval to evaluation:** Set the downsample interval to match or be slightly smaller than your alert evaluation interval.
- **Test queries first:** Verify your query returns expected results in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/) before creating an alert.
- **Set realistic thresholds:** Base alert thresholds on historical data patterns to avoid false positives.
- **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor.
- **Enable downsampling:** Always enable downsampling for alerting queries to reduce data volume and improve reliability.
- **Consider counter resets:** For counter metrics, enable the Counter option and set appropriate max values to handle resets correctly.

## Next steps

- [Build queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/query-editor/) to explore your metrics before creating alerts.
- [Troubleshoot issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/troubleshooting/) if alerts aren't firing as expected.

# OpenTSDB annotations

Annotations allow you to overlay event information on graphs, providing context for metric changes. The OpenTSDB data source supports both metric-specific annotations and global annotations stored in OpenTSDB.

For general information about annotations in Grafana, refer to [Annotate visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/).

## Annotation types

OpenTSDB supports two types of annotations:

| Type                   | Description                                                                                          | Use case                                                               |
| ---------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| **Metric annotations** | Annotations attached to a specific time series (TSUID). Retrieved by querying the associated metric. | Track events affecting a specific host or service.                     |
| **Global annotations** | Annotations not tied to any time series. Apply system-wide.                                          | Track deployments, maintenance windows, or infrastructure-wide events. |

## How Grafana retrieves annotations

When you configure an annotation query, Grafana queries OpenTSDB for the specified metric and retrieves any annotations associated with that metric's time series. The query includes the `globalAnnotations=true` parameter, which allows Grafana to also retrieve global annotations when enabled.

Grafana displays the `description` field from each annotation as the annotation text.

## Configure an annotation query

To add OpenTSDB annotations to a dashboard:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Annotation query**.
1. Enter a name for the annotation query.
1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox.
1. Select a color for the annotation event markers.
1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed.
1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed.
1. Click **Open query editor** to open the **Annotation Query** dialog box.
1. Select the **OpenTSDB** data source from the **Data source** drop-down list.
1. Configure the annotation query and field mappings.
1. (Optional) Click **Test annotation query** to ensure that the query is working properly.
1. Click **Close** when you've completed the query setup.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## Annotation query fields

| Field                       | Description                                                                      |
| --------------------------- | -------------------------------------------------------------------------------- |
| **Name**                    | A descriptive name for this annotation query. Appears in the annotation legend.  |
| **Data source**             | Select the OpenTSDB data source.                                                 |
| **Enabled**                 | Toggle to enable or disable this annotation query.                               |
| **OpenTSDB metrics query**  | The metric name to query for annotations (for example, `events.deployment`).     |
| **Show Global Annotations** | Toggle to include global annotations that aren't tied to a specific time series. |

## Example annotation queries

The following examples demonstrate common annotation use cases.

### Track application deployments

Monitor when deployments occur for a specific application:

| Field                   | Value           |
| ----------------------- | --------------- |
| Name                    | App Deployments |
| OpenTSDB metrics query  | `deploy.myapp`  |
| Show Global Annotations | disabled        |

This query retrieves annotations attached to the `deploy.myapp` metric, showing deployment events for that specific application.

### Monitor infrastructure-wide events

Capture system-wide events such as network changes or data center maintenance:

| Field                   | Value                   |
| ----------------------- | ----------------------- |
| Name                    | Infrastructure Events   |
| OpenTSDB metrics query  | `events.infrastructure` |
| Show Global Annotations | enabled                 |

This query retrieves both metric-specific and global annotations, providing a complete picture of infrastructure events.

### Track incidents and outages

Mark incident start and resolution times:

| Field                   | Value             |
| ----------------------- | ----------------- |
| Name                    | Incidents         |
| OpenTSDB metrics query  | `events.incident` |
| Show Global Annotations | enabled           |

### Monitor configuration changes

Track when configuration changes are applied:

| Field                   | Value                 |
| ----------------------- | --------------------- |
| Name                    | Configuration changes |
| OpenTSDB metrics query  | `events.config`       |
| Show Global Annotations | disabled              |

### Correlate multiple event types

You can add multiple annotation queries to a single dashboard to correlate different event types. For example:

1. Add a "Deployments" annotation query for `deploy.*` metrics.
1. Add an "Incidents" annotation query for `events.incident`.
1. Add a "Maintenance" annotation query with global annotations enabled.

This allows you to see how deployments, incidents, and maintenance windows relate to your metric data.

## How annotations appear

Annotations appear as vertical lines on time series panels at the timestamps where events occurred. Hover over an annotation marker to view:

- The annotation name (from your query configuration)
- The event description (from the OpenTSDB annotation's `description` field)
- The timestamp

Different annotation queries can be assigned different colors in the annotation settings to distinguish between event types.

## Create annotations in OpenTSDB

To display annotations in Grafana, you must first create them in OpenTSDB. OpenTSDB provides an HTTP API for managing annotations.

### Annotation data structure

OpenTSDB annotations have the following fields:

| Field         | Required | Description                                                                                |
| ------------- | -------- | ------------------------------------------------------------------------------------------ |
| `startTime`   | Yes      | Unix epoch timestamp in seconds when the event started.                                    |
| `endTime`     | No       | Unix epoch timestamp in seconds when the event ended. Useful for duration-based events.    |
| `tsuid`       | No       | The time series UID to associate this annotation with. If empty, the annotation is global. |
| `description` | No       | Brief description of the event. This text displays in Grafana.                             |
| `notes`       | No       | Detailed notes about the event.                                                            |
| `custom`      | No       | A map of custom key-value pairs for additional metadata.                                   |

### Create a global annotation

Use the OpenTSDB API to create a global annotation:

```sh
curl -X POST http://<OPENTSDB_HOST>:4242/api/annotation \
  -H "Content-Type: application/json" \
  -d '{
    "startTime": 1609459200,
    "description": "Production deployment v2.5.0",
    "notes": "Deployed new feature flags and performance improvements",
    "custom": {
      "version": "2.5.0",
      "environment": "production",
      "deployer": "jenkins"
    }
  }'
```

### Create a metric-specific annotation

To attach an annotation to a specific time series, include the `tsuid`:

```sh
curl -X POST http://<OPENTSDB_HOST>:4242/api/annotation \
  -H "Content-Type: application/json" \
  -d '{
    "startTime": 1609459200,
    "endTime": 1609462800,
    "tsuid": "000001000001000001",
    "description": "Server maintenance",
    "notes": "Scheduled maintenance window for hardware upgrade"
  }'
```

To find the TSUID for a metric, use the OpenTSDB `/api/uid/tsmeta` endpoint.

### Create annotations programmatically

Integrate annotation creation into your deployment pipelines or monitoring systems:

**Deployment script example:**

```sh
#!/bin/bash
VERSION=$1
TIMESTAMP=$(date +%s)

curl -X POST http://opentsdb.example.com:4242/api/annotation \
  -H "Content-Type: application/json" \
  -d "{
    \"startTime\": $TIMESTAMP,
    \"description\": \"Deployed version $VERSION\",
    \"custom\": {
      \"version\": \"$VERSION\",
      \"environment\": \"production\"
    }
  }"
```

For more details on the annotation API, refer to the [OpenTSDB annotation API documentation](http://opentsdb.net/docs/build/html/api_http/annotation/index.html).

## Troubleshoot annotation issues

The following section addresses common issues you may encounter when using OpenTSDB annotations.

### Annotations don't appear

**Possible causes and solutions:**

| Cause                                    | Solution                                                                                                                              |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| Time range doesn't include annotations   | Expand the dashboard time range to include the annotation timestamps.                                                                 |
| Wrong metric name                        | Verify the metric name in your annotation query matches the metric associated with the annotations in OpenTSDB.                       |
| Annotations are global but toggle is off | Enable **Show Global Annotations** if your annotations don't have a TSUID.                                                            |
| No annotations exist                     | Verify annotations exist in OpenTSDB using the API: `curl http://<OPENTSDB_HOST>:4242/api/annotation?startTime=<START>&endTime=<END>` |

### Annotation text is empty

The annotation displays but has no description text.

**Solution:** Ensure the `description` field is populated when creating annotations in OpenTSDB. Grafana displays the `description` field as the annotation text.

## Next steps

- [Build queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/query-editor/) to visualize metrics alongside annotations.
- [Use template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/template-variables/) to create dynamic dashboards.
- [Set up alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/alerting/) to get notified when metrics cross thresholds.

# Configure the OpenTSDB data source

This document explains how to configure the OpenTSDB data source in Grafana.

## Before you begin

Before configuring the OpenTSDB data source, ensure you have:

- **Grafana permissions:** Organization administrator role to add data sources.
- **OpenTSDB instance:** A running OpenTSDB server (version 2.1 or later recommended).
- **Network access:** The Grafana server can reach the OpenTSDB HTTP API endpoint (default port 4242).
- **Metrics in OpenTSDB:** For autocomplete to work, metrics must exist in your OpenTSDB database.

## Add the data source

To add and configure the OpenTSDB data source:

1. Click **Connections** in the left-side menu.
1. Click **Add new connection**.
1. Type `OpenTSDB` in the search bar.
1. Select **OpenTSDB**.
1. Click **Add new data source** in the upper right.
1. Configure the data source settings as described in the following sections.

## Configuration options

The following table describes the available configuration options:

| Setting             | Description                                                                                                                    |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| **Name**            | The data source name. This is how you refer to the data source in panels and queries.                                          |
| **Default**         | Toggle to make this the default data source for new panels.                                                                    |
| **URL**             | The HTTP protocol, IP address, and port of your OpenTSDB server. The default port is `4242`. Example: `http://localhost:4242`. |
| **Allowed cookies** | Cookies to forward to the data source. Use this when your OpenTSDB server requires specific cookies for authentication.        |
| **Timeout**         | HTTP request timeout in seconds. Increase this value for slow networks or complex queries.                                     |

## Auth settings

Configure authentication if your OpenTSDB server requires it:

| Setting                       | Description                                                                                                                                |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Basic auth**                | Enable to authenticate with a username and password. When enabled, enter the username and password in the fields that appear.              |
| **With Credentials**          | Enable to send cookies or auth headers with cross-site requests. Use this when OpenTSDB is on a different domain and requires credentials. |
| **TLS Client Authentication** | Enable to use client certificates for authentication. Requires configuring client certificate and key.                                     |
| **Skip TLS Verify**           | Enable to skip verification of the OpenTSDB server's TLS certificate. Only use this in development environments.                           |
| **Forward OAuth Identity**    | Enable to forward the user's OAuth token to the data source. Useful when OpenTSDB is behind an OAuth-protected proxy.                      |
| **Custom HTTP Headers**       | Add custom headers to all requests sent to OpenTSDB. Useful for API keys or custom authentication schemes.                                 |

## OpenTSDB settings

Configure these settings based on your OpenTSDB server version and configuration:

| Setting          | Description                                                                                                                                  |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| **Version**      | Select your OpenTSDB version. This affects available query features. Refer to the following section for version-specific features.           |
| **Resolution**   | The resolution of your metric data. Select `second` for second-precision timestamps or `millisecond` for millisecond-precision timestamps.   |
| **Lookup limit** | Maximum number of results returned by suggest and lookup API calls. Default is `1000`. Increase this if you have many metrics or tag values. |

### Version-specific features

The version setting enables different query features in Grafana:

| Version   | Available features                                                                                                           |
| --------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **<=2.1** | Basic queries with tags. Uses legacy tag-based filtering.                                                                    |
| **==2.2** | Adds filter support (`literal_or`, `wildcard`, `regexp`, and more). Filters replace tags for more flexible queries.          |
| **==2.3** | Adds explicit tags support for rate calculations and additional filter types.                                                |
| **==2.4** | Adds fill policy support for downsampling (`none`, `null`, `zero`, `nan`). Enables `arrays=true` for alerting compatibility. |

Select the version that matches your OpenTSDB server. If you're unsure, check your OpenTSDB version with the `/api/version` endpoint.

## Secure SOCKS proxy

The OpenTSDB data source supports connecting through a secure SOCKS5 proxy. Use this to reach an OpenTSDB server hosted in a different network than Grafana without exposing it to the public internet. You configure the proxy at the Grafana instance level. For setup instructions, refer to [Configure a data source SOCKS5 connection proxy](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/proxy/).

## Verify the connection

Click **Save & test** to verify that Grafana can connect to your OpenTSDB server. A successful test confirms that the URL is correct and the server is responding.

If the test fails, refer to [Troubleshooting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/troubleshooting/) for common issues and solutions.

## Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

### YAML example

The following example provisions an OpenTSDB data source:

```yaml
apiVersion: 1

datasources:
  - name: OpenTSDB
    type: opentsdb
    access: proxy
    url: http://localhost:4242
    jsonData:
      # OpenTSDB version: 1 = <=2.1, 2 = 2.2, 3 = 2.3, 4 = 2.4
      tsdbVersion: 3
      # Resolution: 1 = second, 2 = millisecond
      tsdbResolution: 1
      # Maximum results for suggest/lookup API calls
      lookupLimit: 1000
```

### YAML example with basic authentication

The following example provisions an OpenTSDB data source with basic authentication:

```yaml
apiVersion: 1

datasources:
  - name: OpenTSDB
    type: opentsdb
    access: proxy
    url: http://localhost:4242
    basicAuth: true
    basicAuthUser: <USERNAME>
    jsonData:
      tsdbVersion: 3
      tsdbResolution: 1
      lookupLimit: 1000
    secureJsonData:
      basicAuthPassword: <PASSWORD>
```

### YAML example with custom headers

The following example provisions an OpenTSDB data source with custom HTTP headers:

```yaml
apiVersion: 1

datasources:
  - name: OpenTSDB
    type: opentsdb
    access: proxy
    url: http://localhost:4242
    jsonData:
      tsdbVersion: 3
      tsdbResolution: 1
      lookupLimit: 1000
      httpHeaderName1: X-Custom-Header
    secureJsonData:
      httpHeaderValue1: <HEADER_VALUE>
```

The following table describes the available fields:

| Field                              | Type    | Description                                                                  |
| ---------------------------------- | ------- | ---------------------------------------------------------------------------- |
| `basicAuth`                        | boolean | Enable basic authentication.                                                 |
| `basicAuthUser`                    | string  | Username for basic authentication.                                           |
| `jsonData.tsdbVersion`             | number  | OpenTSDB version: `1` (<=2.1), `2` (2.2), `3` (2.3), `4` (2.4).              |
| `jsonData.tsdbResolution`          | number  | Timestamp resolution: `1` (second), `2` (millisecond).                       |
| `jsonData.lookupLimit`             | number  | Maximum results for suggest and lookup API calls. Default: `1000`.           |
| `jsonData.httpHeaderName1`         | string  | Name of a custom HTTP header. Use incrementing numbers for multiple headers. |
| `secureJsonData.basicAuthPassword` | string  | Password for basic authentication.                                           |
| `secureJsonData.httpHeaderValue1`  | string  | Value for the custom HTTP header.                                            |

## Provision with Terraform

You can provision the OpenTSDB data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation.

### Terraform example

The following example provisions an OpenTSDB data source:

```hcl
terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 2.0.0"
    }
  }
}

provider "grafana" {
  url  = "<YOUR_GRAFANA_URL>"
  auth = "<YOUR_SERVICE_ACCOUNT_TOKEN>"
}

resource "grafana_data_source" "opentsdb" {
  type = "opentsdb"
  name = "OpenTSDB"
  url  = "http://localhost:4242"

  json_data_encoded = jsonencode({
    # OpenTSDB version: 1 = <=2.1, 2 = 2.2, 3 = 2.3, 4 = 2.4
    tsdbVersion = 3
    # Resolution: 1 = second, 2 = millisecond
    tsdbResolution = 1
    # Maximum results for suggest/lookup API calls
    lookupLimit = 1000
  })
}
```

### Terraform example with basic authentication

The following example provisions an OpenTSDB data source with basic authentication:

```hcl
resource "grafana_data_source" "opentsdb_auth" {
  type                = "opentsdb"
  name                = "OpenTSDB"
  url                 = "http://localhost:4242"
  basic_auth_enabled  = true
  basic_auth_username = "<USERNAME>"

  json_data_encoded = jsonencode({
    tsdbVersion    = 3
    tsdbResolution = 1
    lookupLimit    = 1000
  })

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = "<PASSWORD>"
  })
}
```

Replace the following placeholders:

- _`<YOUR_GRAFANA_URL>`_: Your Grafana instance URL (for example, `https://your-org.grafana.net` for Grafana Cloud)
- _`<YOUR_SERVICE_ACCOUNT_TOKEN>`_: A service account token with data source permissions
- _`<USERNAME>`_: The username for basic authentication
- _`<PASSWORD>`_: The password for basic authentication

## Next steps

Now that you've configured OpenTSDB, you can:

- [Query OpenTSDB data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/query-editor/) to build dashboards and visualizations
- [Use template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/template-variables/) to create dynamic, reusable dashboards
- [Add annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/annotations/) to overlay events on your graphs
- [Set up alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/alerting/) to get notified when metrics cross thresholds
- [Troubleshoot issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/troubleshooting/) if you encounter problems

# OpenTSDB query editor

The query editor allows you to build OpenTSDB queries visually. The available options depend on the OpenTSDB version you configured for the data source.

## Access the query editor

The OpenTSDB query editor is located on the [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/) page. You can also access the OpenTSDB query editor from a dashboard panel. Click the ellipsis in the upper right of the panel and select **Edit**.

## Create a query

To create a query:

1. Select the **OpenTSDB** data source in a panel.
1. Configure the query using the sections described in the following documentation.

## Metric section

The Metric section contains the core query configuration:

| Field          | Description                                                                                       |
| -------------- | ------------------------------------------------------------------------------------------------- |
| **Metric**     | The metric name to query. Start typing to see autocomplete suggestions from your OpenTSDB server. |
| **Aggregator** | The aggregation function to combine multiple time series. Default: `sum`.                         |
| **Alias**      | Custom display name for the series. Use `$tag_<tagname>` to include tag values in the alias.      |

### Alias patterns

The alias field supports dynamic substitution using tag values. Use the pattern `$tag_<tagname>` where `<tagname>` is the name of a tag on your metric.

| Pattern                | Description                         | Example output             |
| ---------------------- | ----------------------------------- | -------------------------- |
| `$tag_host`            | Inserts the value of the `host` tag | `webserver01`              |
| `$tag_env`             | Inserts the value of the `env` tag  | `production`               |
| `$tag_host - CPU`      | Combines tag value with static text | `webserver01 - CPU`        |
| `$tag_host ($tag_env)` | Multiple tag substitutions          | `webserver01 (production)` |

## Downsample section

Downsampling reduces the number of data points returned by aggregating values over time intervals. This improves query performance and reduces the amount of data transferred.

| Field                    | Description                                                                                                                                                                                  |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Down sample**          | The time interval for downsampling. Enter a value in the input next to the **Down sample** label, or leave it blank to use the automatic interval based on the panel's time range and width. |
| **Aggregator**           | The aggregation function for downsampling. Default: `avg`.                                                                                                                                   |
| **Fill**                 | (Version 2.2+) The fill policy for missing data points. Default: `none`.                                                                                                                     |
| **Disable downsampling** | Toggle to disable downsampling entirely. Use this when you need raw data points.                                                                                                             |

### Interval format

The interval field accepts time duration strings:

| Format | Description | Example |
| ------ | ----------- | ------- |
| `s`    | Seconds     | `30s`   |
| `m`    | Minutes     | `5m`    |
| `h`    | Hours       | `1h`    |
| `d`    | Days        | `1d`    |
| `w`    | Weeks       | `1w`    |

When the interval is left blank, Grafana automatically calculates an appropriate interval based on the panel's time range and pixel width. This ensures optimal data density for visualization.

## Filters section

Filters (available in OpenTSDB 2.2+) provide advanced filtering capabilities that replace the legacy tag-based filtering.

| Field        | Description                                                                                                     |
| ------------ | --------------------------------------------------------------------------------------------------------------- |
| **Key**      | The tag key to filter on. Select from autocomplete suggestions or type a custom value.                          |
| **Type**     | The filter type. Determines how the filter value is matched. Default: `iliteral_or`.                            |
| **Filter**   | The filter value or pattern. Supports autocomplete for tag values.                                              |
| **Group by** | Toggle to group results by this tag key. When enabled, separate time series are returned for each unique value. |

### Add, edit, and remove filters

To manage filters:

1. Click the **+** button next to "Filters" to add a new filter.
1. Configure the filter fields (Key, Type, Filter, Group by).
1. Click **add filter** to apply the filter.
1. To edit an existing filter, click the **pencil** icon next to it.
1. To remove a filter, click the **x** icon next to it.

You can add multiple filters to a single query. All filters are combined with AND logic.

### Filter types

| Type              | Description                                                | Example               |
| ----------------- | ---------------------------------------------------------- | --------------------- |
| `literal_or`      | Matches exact values. Use `\|` to specify multiple values. | `web01\|web02\|web03` |
| `iliteral_or`     | Case-insensitive literal match.                            | `WEB01\|web02`        |
| `wildcard`        | Matches using `*` as a wildcard character.                 | `web-*-prod`          |
| `iwildcard`       | Case-insensitive wildcard match.                           | `WEB-*`               |
| `regexp`          | Matches using regular expressions.                         | `web-[0-9]+`          |
| `not_literal_or`  | Excludes exact values.                                     | `web01\|web02`        |
| `not_iliteral_or` | Case-insensitive exclusion.                                | `TEST\|DEV`           |

### Group by behavior

When **Group by** is enabled for a filter:

- Results are split into separate time series for each unique value of the filtered tag.
- Each time series is labeled with its tag value.
- This is useful for comparing values across hosts, environments, or other dimensions.

When **Group by** is disabled:

- All matching time series are combined using the selected aggregator.
- A single aggregated time series is returned.

## Tags section

Tags filter metrics by key-value pairs. This is the legacy filtering method for OpenTSDB versions prior to 2.2.

| Field     | Description                                                       |
| --------- | ----------------------------------------------------------------- |
| **Key**   | The tag key to filter on. Select from autocomplete suggestions.   |
| **Value** | The tag value to match. Use `*` to match all values for this key. |

### Add, edit, and remove tags

To manage tags:

1. Click the **+** button next to "Tags" to add a new tag.
1. Select or type a tag key.
1. Select or type a tag value (use `*` for wildcard).
1. Click **add tag** to apply the tag filter.
1. To edit an existing tag, click the **pencil** icon next to it.
1. To remove a tag, click the **x** icon next to it.

{{< admonition type="note" >}}
Tags are deprecated in OpenTSDB 2.2 and later. Use Filters instead for more powerful filtering options including wildcards, regular expressions, and exclusion patterns.
{{< /admonition >}}

{{< admonition type="caution" >}}
Tags and Filters are mutually exclusive. If you have filters defined, you cannot add tags, and vice versa. The query editor displays a warning if you attempt to use both.
{{< /admonition >}}

## Rate section

The Rate section computes the rate of change, which is essential for counter metrics that continuously increment.

| Field             | Description                                                                                                |
| ----------------- | ---------------------------------------------------------------------------------------------------------- |
| **Rate**          | Toggle to enable rate calculation. Computes the per-second rate of change between consecutive values.      |
| **Counter**       | (When Rate is enabled) Toggle to indicate the metric is a monotonically increasing counter that may reset. |
| **Counter max**   | (When Counter is enabled) The maximum value before the counter wraps around.                               |
| **Reset value**   | (When Counter is enabled) The value the counter resets to after wrapping. Default: `0`.                    |
| **Explicit tags** | (Version 2.3+) Toggle to require all specified tags to exist on matching time series.                      |

### When to use rate calculation

Enable **Rate** when your metric is a continuously increasing counter, such as:

- Network bytes sent/received
- Request counts
- Error counts
- Disk I/O operations

The rate calculation converts cumulative values into per-second rates, making the data more meaningful for visualization.

### Counter settings

Enable **Counter** when your metric can reset to zero (for example, after a service restart). The counter settings help OpenTSDB calculate correct rates across resets:

- **Counter max**: Set this to the maximum value your counter can reach before wrapping. For 64-bit counters, use `18446744073709551615`. For 32-bit counters, use `4294967295`.
- **Reset value**: The value the counter resets to, typically `0`.

### Explicit tags

When **Explicit tags** is enabled (version 2.3+), OpenTSDB only returns time series that have all the tags specified in your query. This prevents unexpected results when some time series are missing tags that others have.

## Aggregators

The aggregator function combines multiple time series into one. The data source includes a built-in default list (`avg`, `sum`, `min`, `max`, `dev`, `zimsum`, `mimmin`, and `mimmax`), then replaces it with the list fetched from your OpenTSDB server. As a result, you may see additional aggregators, such as `count`, beyond the built-in defaults.

### Common aggregators

| Aggregator | Description                               | Use case                               |
| ---------- | ----------------------------------------- | -------------------------------------- |
| `sum`      | Sum all values at each timestamp.         | Total requests across all servers.     |
| `avg`      | Average all values at each timestamp.     | Average CPU usage across hosts.        |
| `min`      | Take the minimum value at each timestamp. | Lowest response time.                  |
| `max`      | Take the maximum value at each timestamp. | Peak memory usage.                     |
| `dev`      | Calculate the standard deviation.         | Measure variability in response times. |
| `count`    | Count the number of data points.          | Number of reporting hosts.             |

### Interpolation aggregators

These aggregators handle missing data points differently:

| Aggregator | Description                                          |
| ---------- | ---------------------------------------------------- |
| `zimsum`   | Sum values, treating missing data as zero.           |
| `mimmin`   | Minimum value, ignoring missing (interpolated) data. |
| `mimmax`   | Maximum value, ignoring missing (interpolated) data. |

{{< admonition type="note" >}}
The available aggregators depend on your OpenTSDB server version and configuration. The aggregator dropdown is populated dynamically from the `/api/aggregators` endpoint on your OpenTSDB server.
{{< /admonition >}}

## Fill policies

Fill policies (available in OpenTSDB 2.2+) determine how to handle missing data points during downsampling. This is important when your data has gaps or irregular collection intervals.

| Policy | Description                                         | Use case                                                    |
| ------ | --------------------------------------------------- | ----------------------------------------------------------- |
| `none` | Don't fill missing values. Gaps remain in the data. | Default behavior; preserves data fidelity.                  |
| `nan`  | Fill missing values with NaN (Not a Number).        | Useful for calculations that should propagate missing data. |
| `null` | Fill missing values with null.                      | Visualizations show gaps at null points.                    |
| `zero` | Fill missing values with zero.                      | Treat missing data as zero values; useful for counters.     |

### Choose the right fill policy

- Use `none` (default) when you want to see actual data gaps in your visualizations.
- Use `null` when you want graphs to show breaks at missing data points.
- Use `zero` when missing data should be interpreted as zero (for example, no requests during a period).
- Use `nan` when you need missing values to propagate through calculations.

## Autocomplete suggestions

The query editor provides autocomplete suggestions to help you build queries quickly and accurately.

### What autocomplete provides

| Field           | Source                      | Description                                     |
| --------------- | --------------------------- | ----------------------------------------------- |
| **Metric**      | `/api/suggest?type=metrics` | Suggests metric names as you type.              |
| **Tag keys**    | Previous query results      | Suggests tag keys based on the selected metric. |
| **Tag values**  | `/api/suggest?type=tagv`    | Suggests tag values as you type.                |
| **Filter keys** | Previous query results      | Suggests tag keys for filter configuration.     |

### Autocomplete requirements

For autocomplete to work:

- The OpenTSDB suggest API must be enabled on your server.
- Metrics must exist in your OpenTSDB database.
- The **Lookup limit** setting in your data source configuration controls the maximum number of suggestions returned.

If autocomplete isn't working, refer to [Troubleshooting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/troubleshooting/).

## Use template variables

You can use template variables in any text field in the query editor. Template variables are replaced with their current values when the query executes.

Common uses include:

- **Metric field**: `$metric` to dynamically select metrics.
- **Filter values**: `$host` to filter by a variable-selected host.
- **Tag values**: `$environment` to filter by environment.

For more information about creating and using template variables, refer to [Template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/template-variables/).

## Query examples

The following examples demonstrate common query patterns.

### Basic metric query with tag filtering

| Field      | Value              |
| ---------- | ------------------ |
| Metric     | `sys.cpu.user`     |
| Aggregator | `avg`              |
| Tags       | `host=webserver01` |

This query returns the average CPU usage for the host `webserver01`.

### Query with wildcard filter (OpenTSDB 2.2+)

| Field        | Value                 |
| ------------ | --------------------- |
| Metric       | `http.requests.count` |
| Aggregator   | `sum`                 |
| Filter Key   | `host`                |
| Filter Type  | `wildcard`            |
| Filter Value | `web-*`               |
| Group by     | enabled               |

This query sums HTTP request counts across all hosts matching `web-*` and groups results by host.

### Rate calculation for network counters

| Field       | Value                  |
| ----------- | ---------------------- |
| Metric      | `net.bytes.received`   |
| Aggregator  | `sum`                  |
| Rate        | enabled                |
| Counter     | enabled                |
| Counter max | `18446744073709551615` |

This query calculates the rate of bytes received per second. The counter max is set to the 64-bit unsigned integer maximum to handle counter wraps correctly.

### Use alias patterns

| Field      | Value                       |
| ---------- | --------------------------- |
| Metric     | `app.response.time`         |
| Aggregator | `avg`                       |
| Tags       | `host=*`, `env=production`  |
| Alias      | `$tag_host - Response Time` |

This query uses the alias pattern to create readable legend labels like `webserver01 - Response Time`.

### Downsample with a custom interval

| Field                 | Value               |
| --------------------- | ------------------- |
| Metric                | `sys.disk.io.bytes` |
| Aggregator            | `sum`               |
| Downsample Interval   | `5m`                |
| Downsample Aggregator | `avg`               |
| Fill                  | `zero`              |

This query reduces disk I/O data to 5-minute averages, filling gaps with zero values.

### Compare environments with filters

| Field        | Value                 |
| ------------ | --------------------- |
| Metric       | `app.errors.count`    |
| Aggregator   | `sum`                 |
| Filter Key   | `env`                 |
| Filter Type  | `literal_or`          |
| Filter Value | `staging\|production` |
| Group by     | enabled               |

This query shows error counts for both staging and production environments as separate time series for comparison.

### Exclude specific hosts

| Field        | Value                     |
| ------------ | ------------------------- |
| Metric       | `sys.cpu.user`            |
| Aggregator   | `avg`                     |
| Filter Key   | `host`                    |
| Filter Type  | `not_literal_or`          |
| Filter Value | `test-server\|dev-server` |
| Group by     | enabled                   |

This query shows CPU usage for all hosts except test-server and dev-server.

### Query with explicit tags (version 2.3+)

| Field         | Value                 |
| ------------- | --------------------- |
| Metric        | `app.request.latency` |
| Aggregator    | `avg`                 |
| Filter Key    | `host`                |
| Filter Type   | `wildcard`            |
| Filter Value  | `*`                   |
| Group by      | enabled               |
| Explicit tags | enabled               |

This query only returns time series that have the `host` tag defined, excluding any time series that are missing this tag.

## Next steps

- [Use template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/template-variables/) to create dynamic, reusable dashboards.
- [Set up alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/alerting/) to get notified when metrics cross thresholds.
- [Troubleshoot issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/troubleshooting/) if you encounter problems with queries.

# OpenTSDB template variables

Instead of hard-coding server, application, and sensor names in your metric queries, you can use template variables. Variables appear as drop-down menus at the top of the dashboard, so you can change the data being displayed without editing queries.

For an introduction to template variables, refer to the [Variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) documentation.

## Query variable

The OpenTSDB data source supports query-type template variables that fetch values directly from OpenTSDB. These variables dynamically populate based on data in your OpenTSDB database.

### Supported query functions

| Query                        | Description                                                                      | API used                    |
| ---------------------------- | -------------------------------------------------------------------------------- | --------------------------- |
| `metrics(prefix)`            | Returns metric names matching the prefix. Use empty parentheses for all metrics. | `/api/suggest?type=metrics` |
| `tag_names(metric)`          | Returns tag keys (names) that exist for a specific metric.                       | `/api/search/lookup`        |
| `tag_values(metric, tagkey)` | Returns tag values for a specific metric and tag key.                            | `/api/search/lookup`        |
| `suggest_tagk(prefix)`       | Returns tag keys matching the prefix across all metrics.                         | `/api/suggest?type=tagk`    |
| `suggest_tagv(prefix)`       | Returns tag values matching the prefix across all metrics.                       | `/api/suggest?type=tagv`    |

{{< admonition type="note" >}}
The `tag_names` and `tag_values` functions use the OpenTSDB lookup API, which requires metrics to exist in your database. The `suggest_tagk` and `suggest_tagv` functions use the suggest API, which searches across all metrics.
{{< /admonition >}}

### Create a query variable

To create a query variable:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Variable**.
1. Select **Query** as the variable type.
1. Enter a **Name** for your variable (for example, `host`).
1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed.
1. Click **Open variable editor** to open the **Query Variable** dialog box.
1. Select the **OpenTSDB** data source.
1. Enter your query using one of the supported query functions.
1. Click **Close** when you've completed the query setup.
1. (Optional) Configure **Multi-value** to allow selecting multiple values.
1. (Optional) Configure **Include All value** to add an "All" option.
1. (Optional) Turn on the **Allow custom values** switch to allow users to enter custom variable values.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

### Query variable examples

**List all metrics:**

```
metrics()
```

Returns all metric names in your OpenTSDB database. Useful for creating a metric selector.

**List metrics with a prefix:**

```
metrics(sys.cpu)
```

Returns metrics starting with `sys.cpu`, such as `sys.cpu.user`, `sys.cpu.system`, `sys.cpu.idle`.

**List tag keys for a metric:**

```
tag_names(sys.cpu.user)
```

Returns tag keys like `host`, `env`, `datacenter` that exist on the `sys.cpu.user` metric.

**List tag values for a metric and tag key:**

```
tag_values(sys.cpu.user, host)
```

Returns all host values for the `sys.cpu.user` metric, such as `webserver01`, `webserver02`, `dbserver01`.

**Search for tag keys by prefix:**

```
suggest_tagk(host)
```

Returns tag keys matching `host` across all metrics, such as `host`, `hostname`, `host_id`.

**Search for tag values by prefix:**

```
suggest_tagv(web)
```

Returns tag values matching `web` across all metrics, such as `webserver01`, `webserver02`, `web-prod-01`.

If template variables aren't populating in the **Preview of values** section, refer to [Troubleshooting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/troubleshooting/).

## Nested template variables

You can use one template variable to filter values for another. This creates cascading filters, such as selecting a data center first, then showing only hosts in that data center.

### Filter syntax

The `tag_values` function accepts additional tag filters after the tag key:

```
tag_values(metric, tagkey, tag1=value1, tag2=value2, ...)
```

Use template variables as filter values to create dynamic dependencies:

```
tag_values(metric, tagkey, tag1=$variable1, tag2=$variable2)
```

### Nested variable examples

| Query                                                      | Description                                                  |
| ---------------------------------------------------------- | ------------------------------------------------------------ |
| `tag_values(sys.cpu.user, host, env=$env)`                 | Returns host values filtered by the selected `env` value.    |
| `tag_values(sys.cpu.user, host, env=$env, datacenter=$dc)` | Returns host values filtered by both `env` and `datacenter`. |
| `tag_values(app.requests, endpoint, service=$service)`     | Returns endpoint values for the selected service.            |

### Create cascading filters

To create a hierarchy of dependent variables:

1. **Create the parent variable:**
   - Name: `datacenter`
   - Query: `tag_values(sys.cpu.user, datacenter)`

2. **Create the child variable:**
   - Name: `host`
   - Query: `tag_values(sys.cpu.user, host, datacenter=$datacenter)`

3. **Create additional levels as needed:**
   - Name: `cpu`
   - Query: `tag_values(sys.cpu.user, cpu, datacenter=$datacenter, host=$host)`

When users change the data center selection, the host variable automatically refreshes to show only hosts in that data center.

## Use variables in queries

Reference variables in your queries using the `$variablename` or `${variablename}` syntax. Grafana replaces the variable with its current value when the query executes.

### Where to use variables

Variables can be used in these query editor fields:

| Field                   | Example             | Description                                 |
| ----------------------- | ------------------- | ------------------------------------------- |
| **Metric**              | `$metric`           | Dynamically select which metric to query.   |
| **Tag value**           | `host=$host`        | Filter by a variable-selected tag value.    |
| **Filter value**        | `$host`             | Use in filter value field for filtering.    |
| **Alias**               | `$tag_host - $host` | Include variable values in legend labels.   |
| **Downsample interval** | `$interval`         | Use a variable for the downsample interval. |

### Variable syntax options

| Syntax                   | Description                                                                      |
| ------------------------ | -------------------------------------------------------------------------------- |
| `$variablename`          | Simple syntax for most cases.                                                    |
| `${variablename}`        | Use when the variable is adjacent to other text (for example, `${host}_suffix`). |
| `${variablename:format}` | Apply a specific format to the variable value.                                   |

## Multi-value variables

When you enable **Multi-value** for a variable, users can select multiple values simultaneously. The OpenTSDB data source handles multi-value variables using pipe (`|`) separation, which is compatible with OpenTSDB's `literal_or` filter type.

### Configure multi-value variables

1. When creating the variable, enable **Multi-value**.
1. Optionally enable **Include All option** to add an "All" selection.
1. Use the variable in a filter with the `literal_or` filter type.

### Multi-value example

With a `host` variable configured as multi-value:

| Field        | Value        |
| ------------ | ------------ |
| Filter Key   | `host`       |
| Filter Type  | `literal_or` |
| Filter Value | `$host`      |

If the user selects `webserver01`, `webserver02`, and `webserver03`, the filter value becomes `webserver01|webserver02|webserver03`.

### All value behavior

When the user selects "All", Grafana sends all available values pipe-separated. For large value sets, consider using a wildcard filter instead:

| Field        | Value      |
| ------------ | ---------- |
| Filter Key   | `host`     |
| Filter Type  | `wildcard` |
| Filter Value | `*`        |

## Interval and auto-interval variables

Grafana provides built-in interval variables that are useful with OpenTSDB downsampling:

| Variable         | Description                                                            |
| ---------------- | ---------------------------------------------------------------------- |
| `$__interval`    | Automatically calculated interval based on time range and panel width. |
| `$__interval_ms` | Same as `$__interval` but in milliseconds.                             |

Use these in the downsample interval field for automatic interval adjustment:

| Field               | Value         |
| ------------------- | ------------- |
| Downsample Interval | `$__interval` |

## Next steps

- [Build queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/query-editor/) using your template variables.
- [Set up alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/alerting/) with templated queries.
- [Troubleshoot issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/troubleshooting/) if variables aren't populating.

# Troubleshoot OpenTSDB data source issues

This document provides solutions to common issues you may encounter when configuring or using the OpenTSDB data source. For configuration instructions, refer to [Configure the OpenTSDB data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/configure/).

## Connection errors

These errors occur when Grafana can't connect to the OpenTSDB server.

### "Connection refused" or timeout errors

**Symptoms:**

- Save & test fails
- Queries return connection errors
- Intermittent timeouts

**Possible causes and solutions:**

| Cause                        | Solution                                                                                                          |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Wrong URL or port            | Verify the URL includes the correct protocol, IP address, and port. The default port is `4242`.                   |
| OpenTSDB not running         | Check that the OpenTSDB server is running and accessible.                                                         |
| Firewall blocking connection | Ensure firewall rules allow outbound connections from Grafana to the OpenTSDB server on the configured port.      |
| Network issues               | Verify network connectivity between Grafana and OpenTSDB. Try pinging the server or using `curl` to test the API. |

To test connectivity manually, run:

```sh
curl http://<OPENTSDB_HOST>:4242/api/version
```

## Authentication errors

These errors occur when credentials are invalid or misconfigured.

### "401 Unauthorized" or "403 Forbidden"

**Symptoms:**

- Save & test fails with authentication error
- Queries return authorization errors

**Solutions:**

1. Verify that basic authentication credentials are correct in the data source configuration.
1. Check that the OpenTSDB server is configured to accept the provided credentials.
1. If using cookies for authentication, ensure the required cookies are listed in **Allowed cookies**.

## Query errors

These errors occur when executing queries against OpenTSDB.

### No data returned

**Symptoms:**

- Query executes without error but returns no data
- Panels show "No data" message

**Possible causes and solutions:**

| Cause                           | Solution                                                                                                |
| ------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Time range doesn't contain data | Expand the dashboard time range. Verify data exists in OpenTSDB for the selected period.                |
| Wrong metric name               | Verify the metric name is correct. Use autocomplete to discover available metrics.                      |
| Incorrect tag filters           | Remove or adjust tag filters. Use `*` as a wildcard to match all values.                                |
| Version mismatch                | Ensure the configured OpenTSDB version matches your server. Filters are only available in version 2.2+. |
| Using both Filters and Tags     | Use either Filters or Tags, not both. They're mutually exclusive in OpenTSDB 2.2+.                      |

### Query timeout

**Symptoms:**

- Queries take a long time and then fail
- Error message mentions timeout

**Solutions:**

1. Reduce the time range of your query.
1. Add more specific tag filters to reduce the data volume.
1. Increase the **Timeout** setting in the data source configuration.
1. Enable downsampling to reduce the number of data points returned.
1. Check OpenTSDB server performance and HBase health.

## Autocomplete doesn't work

**Symptoms:**

- No suggestions appear when typing metric names, tag names, or tag values
- Drop-down menus are empty

**Solutions:**

1. Verify that the OpenTSDB `/api/suggest` endpoint is accessible. Test it manually with `curl http://<OPENTSDB_HOST>:4242/api/suggest?type=metrics`.
1. Increase the **Lookup limit** setting if you have many metrics or tags.
1. Verify that the data source connection is working by clicking **Save & test**.
1. Check that metrics exist in OpenTSDB. The suggest API only returns metrics that have been written to the database.

## Template variables don't populate

**Symptoms:**

- Template variable drop-down menus are empty
- **Preview of values** shows no results

**Solutions:**

1. Enable real-time metadata tracking in OpenTSDB by setting `tsd.core.meta.enable_realtime_ts` to `true` in your OpenTSDB configuration.
1. Sync existing metadata by running `tsdb uid metasync` on the OpenTSDB server.
1. Verify the variable query syntax is correct. Refer to [Template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/opentsdb/template-variables/) for the correct syntax.
1. Check that the data source connection is working.

## Performance issues

These issues relate to slow queries or high resource usage.

### Slow queries

**Symptoms:**

- Dashboards take a long time to load
- Queries are slow even for small time ranges

**Solutions:**

1. Enable downsampling in the query editor to reduce data volume.
1. Use more specific tag filters to limit the time series returned.
1. Reduce the time range.
1. Check OpenTSDB and HBase performance metrics.
1. Consider increasing OpenTSDB heap size if memory is constrained.

<!-- vale Grafana.Headings = NO -->

### HBase performance issues

<!-- vale Grafana.Headings = YES -->

OpenTSDB relies on HBase for data storage. Performance problems in HBase directly affect OpenTSDB query performance.

**Solutions:**

1. Monitor HBase region server health and compaction status.
1. Ensure sufficient heap memory is allocated to HBase region servers.
1. Check for region hot spots and redistribute regions if necessary.
1. Refer to the [OpenTSDB troubleshooting guide](http://opentsdb.net/docs/build/html/user_guide/troubleshooting.html) for HBase-specific issues.

## Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location).
1. Look for OpenTSDB-specific entries that include request and response details.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you've tried the solutions in this document and still encounter issues:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review [OpenTSDB issues on GitHub](https://github.com/grafana/grafana/issues?q=opentsdb) for known bugs.
1. Consult the [OpenTSDB documentation](http://opentsdb.net/docs/build/html/index.html) for server-specific guidance.
1. Contact Grafana Support if you're a Grafana Enterprise, Cloud Pro, or Cloud Contracted user.

When reporting issues, include:

- Grafana version
- OpenTSDB version
- Error messages (redact sensitive information)
- Steps to reproduce
- Data source configuration (redact credentials)

# Configure the Parca data source

{{< admonition type="warning" >}}
This plugin is deprecated and will only receive critical security updates. Support will end on January 2, 2027.
{{< /admonition >}}

This document explains how to configure the Parca data source in Grafana.

You can configure the data source using the Grafana UI, a YAML provisioning file, or Terraform.
If you make any changes in the UI, select **Save & test** to preserve those changes.

## Before you begin

Before configuring the data source, ensure you have:

- **Grafana permissions:** `Organization administrator` role.
- **Parca instance:** A running Parca instance (v0.19 or later) accessible from your Grafana server.

If you're provisioning the data source, you also need administrative rights on the server hosting your Grafana instance.

## Add the data source

To add the Parca data source:

1. Click **Connections** in the left-side menu.
1. Click **Add new connection**.
1. Type `Parca` in the search bar.
1. Select **Parca**.
1. Click **Add new data source**.

## Configure settings

The **Settings** tab contains the following configuration sections.

### Basic settings

| Setting     | Description                                                          |
| ----------- | -------------------------------------------------------------------- |
| **Name**    | A name to identify this data source in panels, queries, and Explore. |
| **Default** | Toggle to make this the default data source for new panels.          |

### Connection

| Setting | Description                                                                                                                                                         |
| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **URL** | The URL of your Parca instance. For example, `http://localhost:7070`. Grafana connects to Parca using gRPC-Web, so the URL should point to the Parca HTTP endpoint. |

### Authentication

Use this section to select an authentication method to access the data source. The available methods are:

| Method                     | Description                                                                                      |
| -------------------------- | ------------------------------------------------------------------------------------------------ |
| **Basic authentication**   | Authenticate with a username and password. Enter values in the **User** and **Password** fields. |
| **Forward OAuth Identity** | Forward the logged-in user's OAuth token to the data source.                                     |
| **No Authentication**      | Connect without credentials. Use this when the Parca instance doesn't require authentication.    |

### TLS settings

Configure TLS for secure communication with your Parca instance.

| Setting                             | Description                                                                                                         |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **Add self-signed certificate**     | Toggle to provide a CA certificate. When enabled, paste the CA certificate in the **CA Certificate** field.         |
| **TLS Client Authentication**       | Toggle to enable mutual TLS. When enabled, provide the **Server Name**, **Client Certificate**, and **Client Key**. |
| **Skip TLS certificate validation** | Toggle to skip server certificate verification. Use only for testing.                                               |

{{< admonition type="note" >}}
Use Transport Layer Security (TLS) for an additional layer of security when working with Parca.
{{< /admonition >}}

### Custom HTTP headers

Add custom HTTP headers to requests sent to the Parca instance. Click **Add header** to add a new row with **Header** (name) and **Value** (secret) fields.

Custom headers are useful for passing authentication tokens or routing information required by proxies between Grafana and Parca.

## Additional settings

Click the down arrow to expand the **Additional settings** section. These settings are optional.

### Advanced HTTP settings

| Setting             | Description                                                                                           |
| ------------------- | ----------------------------------------------------------------------------------------------------- |
| **Allowed cookies** | Cookies that should be forwarded to the data source. The Grafana proxy strips all cookies by default. |
| **Timeout**         | The HTTP request timeout in seconds.                                                                  |

### Secure SOCKS proxy

{{< admonition type="note" >}}
This section is only visible when the Grafana server has the secure SOCKS proxy feature enabled.
{{< /admonition >}}

| Setting     | Description                                                                  |
| ----------- | ---------------------------------------------------------------------------- |
| **Enabled** | Toggle to route requests to the Parca instance through a secure SOCKS proxy. |

## Verify the connection

Click **Save & test** to verify that Grafana can connect to your Parca instance. The health check queries the available profile types from the Parca server. A successful connection displays the message

**Data source is working**.

If the test fails, verify that the URL is correct and that your Parca instance is running and accessible from the Grafana server. For more help, refer to [Troubleshoot Parca data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/parca/troubleshooting/).

## Provision the data source

You can define the data source in YAML files as part of the Grafana provisioning system.
For more information, refer to [Provisioning Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

### YAML provisioning example

```yaml
apiVersion: 1

datasources:
  - name: Parca
    type: parca
    url: http://localhost:7070
```

To provision with basic authentication:

```yaml
apiVersion: 1

datasources:
  - name: Parca
    type: parca
    url: http://localhost:7070
    basicAuth: true
    basicAuthUser: <USERNAME>
    secureJsonData:
      basicAuthPassword: <PASSWORD>
```

Replace the following:

- `<USERNAME>`: Your Parca username.
- `<PASSWORD>`: Your Parca password.

### Terraform example

To provision the data source with Terraform, use the [`grafana_data_source` resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source):

```hcl
resource "grafana_data_source" "parca" {
  type = "parca"
  name = "Parca"
  url  = "http://localhost:7070"
}
```

To provision with basic authentication:

```hcl
resource "grafana_data_source" "parca" {
  type                = "parca"
  name                = "Parca"
  url                 = "http://localhost:7070"
  basic_auth_enabled  = true
  basic_auth_username = "<USERNAME>"

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = "<PASSWORD>"
  })
}
```

Replace the following:

- `<USERNAME>`: Your Parca username.
- `<PASSWORD>`: Your Parca password.

## Next steps

- [Query profiling data with the Parca query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/parca/query-editor/)

# Parca query editor

{{< admonition type="warning" >}}
This plugin is deprecated and will only receive critical security updates. Support will end on January 2, 2027.
{{< /admonition >}}

This document explains how to use the Parca query editor to query and visualize continuous profiling data.

The query editor gives you access to a profile type selector, a label selector with autocomplete, and collapsible options for controlling query behavior.

## Before you begin

Before using the query editor, ensure you have:

- [Configured the Parca data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/parca/configure/).
- Verified the connection using **Save & test**.

## Access the query editor

To access the query editor:

1. Sign in to Grafana.
1. Select **Explore** from the left-side menu.
1. Select your Parca data source from the data source drop-down.

You can also access the query editor from a dashboard panel by selecting the Parca data source when editing a panel.

## Profile type selector

Select a profile type from the cascading drop-down menu. Profile types are grouped by name at the first level, with a second level showing the sample type.

For example, a `process_cpu` profile might show sample types like `cpu` or `samples`.

You must select a profile type for the query to be valid. Grafana doesn't return any data if the profile type isn't selected when a query runs.

### Profile type ID format

Each profile type has an ID in the format `name:sampleType:sampleUnit:periodType:periodUnit`. Delta profile types include an additional `:delta` suffix. You don't need to construct these IDs manually -- the cascading drop-down handles selection for you.

Examples of profile type IDs:

| Profile type ID                                     | Description                                 |
| --------------------------------------------------- | ------------------------------------------- |
| `process_cpu:samples:count:cpu:nanoseconds`         | CPU profile measuring sample counts.        |
| `memory:alloc_objects:count:space:bytes`            | Memory profile measuring allocated objects. |
| `process_cpu:cpu:nanoseconds:cpu:nanoseconds:delta` | Delta CPU profile measuring nanoseconds.    |

## Label selector

Use the label selector to filter profiles by labels. The editor provides autocomplete for both label names and label values.

### Syntax

Parca uses a syntax similar to Prometheus for label filtering. Wrap label matchers in curly braces using the `{key="value"}` format.

The following operators are supported:

| Operator | Description                       | Example                 |
| -------- | --------------------------------- | ----------------------- |
| `=`      | Equals                            | `{job="my-service"}`    |
| `!=`     | Not equals                        | `{job!="test-service"}` |
| `=~`     | Regular expression match          | `{job=~"prod-.*"}`      |
| `!~`     | Negative regular expression match | `{job!~"test-.*"}`      |

You can combine multiple label matchers separated by commas:

```text
{job="my-service", instance=~"10.0.0.*"}
```

The label selector can be left empty to query all profiles of the selected type without filtering.

Refer to the [Parca documentation](https://www.parca.dev/docs) for additional syntax details.

### Autocomplete

The label selector provides autocomplete suggestions as you type. Autocomplete is triggered by the following characters: `{`, `,`, `[`, `(`, `=`, `~`, space, and `"`.

- **Label names:** Suggested when you start typing inside curly braces or after a comma.
- **Label values:** Suggested after typing a label name followed by `=` or `="`.

### Keyboard shortcuts

| Shortcut          | Action                            |
| ----------------- | --------------------------------- |
| **Shift + Enter** | Run the query with current input. |

## Query types

Expand the **Options** section to select a query type. The query type controls what data is returned and how it's visualized.

| Query type  | Description                                                      |
| ----------- | ---------------------------------------------------------------- |
| **Metric**  | Returns aggregated metric data visualized as a time series.      |
| **Profile** | Returns a merged profile visualized as a flame graph.            |
| **Both**    | Returns both metric and profile data. Only available in Explore. |

{{< admonition type="note" >}}
The **Both** option is only available in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/). In dashboard panels, you can select either **Metric** or **Profile** because panels support only one visualization type. If a query set to **Both** is moved from Explore to a dashboard, Grafana automatically changes it to **Profile**.
{{< /admonition >}}

## Use template variables in queries

The label selector field supports Grafana template variables. You can use variables like `$job` or `$instance` in the label selector to create dynamic, reusable dashboards.

For more information about supported variable types, examples, and limitations, refer to [Parca template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/parca/template-variables/).

## Query examples

The following examples show common profiling queries.

### Query CPU profiles for a specific service

1. Select **process_cpu** > **cpu** from the profile type drop-down.
1. Enter a label selector to filter by service:

   ```text
   {job="my-service"}
   ```

1. Set query type to **Profile** to view a flame graph, or **Metric** to view CPU usage over time.

### Compare profiles across instances

1. Select a profile type such as **process_cpu** > **samples**.
1. Filter to a specific service and instance:

   ```text
   {job="my-service", instance="10.0.0.1:7070"}
   ```

1. Add a second query (click **+ Add query**) with a different instance:

   ```text
   {job="my-service", instance="10.0.0.2:7070"}
   ```

1. Set both queries to **Metric** to compare time-series data across instances.

### Find memory allocation hot spots

1. Select **memory** > **alloc_objects** from the profile type drop-down.
1. Optionally filter by service:

   ```text
   {job="api-gateway"}
   ```

1. Set query type to **Profile** to view a flame graph showing which functions allocate the most objects.
1. Adjust the time range to narrow down to a specific incident window.

## Profile query results

Profiles are visualized as a flame graph. Refer to the [flame graph documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/flame-graph/) to learn about the visualization and its features.

Parca returns profiles aggregated over the selected time range. The absolute values in the flame graph grow as the time range gets bigger while keeping the relative values meaningful. You can zoom in on the time range to get a higher granularity profile, down to the point of a single Parca scrape interval.

The flame graph displays the following columns:

| Column    | Description                                     |
| --------- | ----------------------------------------------- |
| **level** | The depth of the function in the call stack.    |
| **value** | The cumulative value including child functions. |
| **self**  | The value attributable to this function only.   |
| **label** | The function name or symbol.                    |

Units are automatically mapped from Parca to Grafana formats: `nanoseconds` displays as `ns` and `count` displays as a short number.

## Metrics query results

Metrics results represent the aggregated value over time of the selected profile type.

Parca returns a separate time series for each label combination, allowing you to quickly identify spikes in the value of scraped profiles and zoom in to a particular time range for further investigation.

The time-series value field is named after the sample type from the profile type ID. For example, a query using `process_cpu:samples:count:cpu:nanoseconds` produces a value field named `samples`.

# Parca template variables

{{< admonition type="warning" >}}
This plugin is deprecated and will only receive critical security updates. Support will end on January 2, 2027.
{{< /admonition >}}

Instead of hard-coding label values in your profiling queries, you can use template variables to create dynamic, reusable dashboards. Variables appear as drop-down menus at the top of the dashboard, making it easy to switch between services, instances, or environments without editing queries.

For an introduction to template variables, refer to the [Variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) documentation.

## Supported variable types

| Variable type | Supported | Notes                                                                        |
| ------------- | --------- | ---------------------------------------------------------------------------- |
| Custom        | Yes       | Define a static list of values manually.                                     |
| Text box      | Yes       | Enter a free-form value at the top of the dashboard.                         |
| Constant      | Yes       | Define a hidden, fixed value.                                                |
| Data source   | Yes       | Select a data source instance by name.                                       |
| Query         | No        | Parca doesn't implement variable queries to dynamically populate drop-downs. |

{{< admonition type="note" >}}
Parca doesn't support query-type variables. You can't use the Parca data source to dynamically populate variable drop-downs. Use custom variables to define values manually, or use another data source to populate query variables.
{{< /admonition >}}

## Use variables in the label selector

The Parca query editor's label selector field supports template variable interpolation. Use the standard `$variablename` or `${variablename}` syntax. Variables are interpolated in the label selector only -- the profile type drop-down doesn't support them.

### Variable syntax options

| Syntax                   | Description                                                                      |
| ------------------------ | -------------------------------------------------------------------------------- |
| `$variablename`          | Simple syntax for most cases.                                                    |
| `${variablename}`        | Use when the variable is adjacent to other text (for example, `${host}_suffix`). |
| `${variablename:format}` | Apply a specific format to the variable value.                                   |

## Example: Filter profiles by service

Create a custom variable to switch between services in your profiling dashboard.

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Variable**.
1. Select **Custom** as the variable type.
1. Enter a **Name**, for example `service`.
1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed.
1. Click **Open variable editor** to open the **Custom options** dialog box.
1. Enter comma-separated values, for example `frontend, backend, api-gateway`.
1. Click **Apply**.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

In your Parca query's label selector, reference the variable:

```text
{job="$service"}
```

When you change the drop-down selection, the panel refreshes to show profiles for the selected service.

## Example: Filter by service and instance

Create two custom variables to filter by multiple dimensions.

1. Create a variable named `service` with custom values `frontend, backend, api-gateway`.
1. Create a second variable named `instance` with custom values for your instance addresses, for example `10.0.0.1:7070, 10.0.0.2:7070, 10.0.0.3:7070`.

In your Parca query's label selector, reference both variables:

```text
{job="$service", instance="$instance"}
```

## Example: Use a text box variable for free-form filtering

Create a text box variable for free-form label filtering.

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Variable**.
1. Select **Textbox** as the variable type.
1. Enter a **Name**, for example `label_filter`.
1. Optionally set a default **Value**, for example `job="my-service"`.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

In your Parca query's label selector, reference the variable:

```text
{$label_filter}
```

This lets you type any label matcher directly into the variable input at the top of the dashboard without editing the panel query.

## Limitations

Parca template variable support has the following limitations:

- **No query variable support:** You can't use Parca as a data source for populating variable options. Define values manually with custom variables or use another data source.
- **Label selector only:** Variables are interpolated in the label selector field. The profile type drop-down doesn't support variables.
- **No free-form filters:** Parca doesn't support free-form [filters](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/filter-group-by/).

## Next steps

- [Build queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/parca/query-editor/) using your template variables.
- [Troubleshoot issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/parca/troubleshooting/) if queries aren't returning expected data.

# Troubleshoot Parca data source issues

{{< admonition type="warning" >}}
This plugin is deprecated and will only receive critical security updates. Support will end on January 2, 2027.
{{< /admonition >}}

This page provides solutions to common issues you might encounter when configuring or using the Parca data source. For configuration instructions, refer to [Configure the Parca data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/parca/configure/).

## Connection errors

These errors occur when Grafana can't reach the Parca instance.

### "Save & test" fails

**Symptoms:**

- Data source test times out or returns an error.
- Unable to connect to the Parca server.

**Possible causes and solutions:**

| Cause                | Solution                                                                                                                   |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Incorrect URL        | Verify the URL points to your Parca HTTP endpoint (for example, `http://localhost:7070`). Grafana connects using gRPC-Web. |
| Parca isn't running  | Verify that your Parca instance is running and accessible from the Grafana server.                                         |
| Firewall blocking    | Check that firewall rules allow outbound traffic from Grafana to the Parca server on the configured port.                  |
| TLS misconfiguration | If using HTTPS, verify that TLS certificates are correctly configured on both Grafana and Parca.                           |

When the connection succeeds, the health check displays **"Data source is working"**. The health check queries the available profile types from the Parca server, so any connectivity or authentication issue causes it to fail.

### Connection refused or timeout errors

**Symptoms:**

- Queries fail with network errors.
- Intermittent connection issues.

**Solutions:**

1. Verify network connectivity from the Grafana server to the Parca endpoint.
1. Check that the Parca server is healthy and responding to requests.
1. For Grafana Cloud, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if accessing a private Parca instance.

## Query errors

These errors occur when running queries against the Parca data source.

### "Invalid report type" or "try updating Parca to v0.19+"

**Symptoms:**

- Queries fail with an error containing "invalid report type."
- The error message suggests updating Parca to v0.19+.
- Flame graph data doesn't load.

**Solutions:**

1. Upgrade your Parca server to v0.19 or later. This error occurs because older versions of Parca don't support the flame graph Arrow format that Grafana requires.

### `Unknown report type returned from query. update parca`

**Symptoms:**

- Profile queries fail with this exact error message.
- Metric queries may still succeed.

**Solutions:**

1. Upgrade your Parca server. This error occurs when the Parca server returns a response format that Grafana doesn't recognize. Updating to the latest Parca version resolves the issue.

### "No data" or empty results

**Symptoms:**

- Query runs without error but returns no data.
- Flame graph or metrics panel shows "No data."

**Possible causes and solutions:**

| Cause                           | Solution                                                                                                                                                                                                                                |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No profile type selected        | Select a profile type from the drop-down menu. This field is required -- queries silently return empty data without it.                                                                                                                 |
| Time range doesn't contain data | Expand the dashboard time range or verify that profiling data exists for the selected period.                                                                                                                                           |
| Label selector too restrictive  | Remove or broaden label filters to verify that matching profiles exist.                                                                                                                                                                 |
| Label selector syntax error     | Verify the label selector uses valid syntax (for example, `{job="my-service"}`). Refer to the [query editor documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/parca/query-editor/) for supported operators. |
| Parca isn't scraping targets    | Check your Parca configuration to verify that targets are being scraped and profiles collected.                                                                                                                                         |

## Query editor issues

These issues relate to the query editor interface.

### Profile type drop-down is empty

**Symptoms:**

- The profile type selector shows no options.
- The button text reads **Select a profile type** with nothing to choose.

**Possible causes and solutions:**

| Cause                      | Solution                                                                                         |
| -------------------------- | ------------------------------------------------------------------------------------------------ |
| Connection to Parca failed | Verify the data source connection using **Save & test** in the data source settings.             |
| Parca has no scraped data  | Verify that Parca is scraping targets and has collected profile data.                            |
| Network or auth error      | Check the browser developer console for failed requests to the `profileTypes` resource endpoint. |

### "Both" query type is missing

**Symptoms:**

- Only **Metric** and **Profile** appear in the query type options.
- The **Both** option isn't available.

**Solution:**

This is expected behavior. The **Both** query type is only available in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/). Dashboard panels support only one visualization type, so Grafana limits the options to **Metric** or **Profile**. If a query set to **Both** in Explore is used in a dashboard, Grafana automatically changes it to **Profile**.

### Autocomplete suggestions don't appear

**Symptoms:**

- No label name or value suggestions appear when typing in the label selector.
- Autocomplete works for some labels but not others.

**Possible causes and solutions:**

| Cause                          | Solution                                                                                                                                                              |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Label names haven't loaded     | Wait a moment after opening the query editor. Autocomplete loads label names from Parca on initialization.                                                            |
| Connection to Parca failed     | Verify the data source connection. Autocomplete queries the `labelNames` and `labelValues` resource endpoints.                                                        |
| Cursor position not recognized | Place your cursor inside curly braces `{}` and after an `=` sign for value suggestions. Autocomplete triggers on specific characters: `{`, `,`, `=`, `~`, `"`, space. |

## Template variable issues

These issues relate to using template variables with the Parca data source.

### Variables don't resolve in queries

**Symptoms:**

- Variable syntax like `$variable` appears literally in query results instead of being replaced.
- Queries return no data when using variables.

**Possible causes and solutions:**

| Cause                         | Solution                                                                                                                                  |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Variable used in profile type | Template variables are only supported in the **label selector** field. The profile type drop-down doesn't support variable interpolation. |
| Variable not defined          | Verify the variable exists in **Dashboard options** > **Variables** and has valid values.                                                 |
| Wrong variable syntax         | Use `$variablename` or `${variablename}`. Verify there are no typos in the variable name.                                                 |

### Can't use Parca to populate variable options

**Symptoms:**

- When creating a query-type variable with the Parca data source, no values are returned.

**Solution:**

Parca doesn't support query-type variables. You can't use the Parca data source to dynamically populate variable drop-downs. Use **Custom** variables with manually defined values, **Text box** variables for free-form input, or a different data source for query-type variables. Refer to [Parca template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/parca/template-variables/) for supported variable types and examples.

## Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location).
1. Look for Parca-specific entries that include request and response details. Common log messages include:
   - `"Failed to get profile types"` -- profile type loading failed.
   - `"Failed to get label names"` -- label autocomplete data failed to load.
   - `"Failed to process query"` -- a query to Parca returned an error.
   - `"Failed to unmarshall query"` -- the query JSON couldn't be parsed.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you've tried the solutions on this page and still encounter issues:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs related to the Parca data source.
1. Consult the [Parca documentation](https://www.parca.dev/docs) for service-specific guidance.
1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user.
1. When reporting issues, include:
   - Grafana version
   - Parca server version
   - Error messages (redact sensitive information)
   - Steps to reproduce
   - Relevant configuration (redact credentials)

# PostgreSQL alerting

The PostgreSQL data source supports [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/). You can create alert rules that evaluate time series queries against your PostgreSQL database and send notifications when conditions are met.

## Before you begin

- Ensure your [PostgreSQL data source is configured](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/postgres/configure/).
- Familiarize yourself with [Alert rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/) and [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Supported query format

Only **time series** queries can be used in alert rule conditions.

- Your query must return a column named `time` (native SQL date/time or UNIX epoch) and one or more numeric value columns.
- **Table** formatted queries aren't supported in alert rule conditions.

For details on writing time series queries, refer to [Time series queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/postgres/query-editor/#time-series-queries) in the PostgreSQL query editor.

## Limitations

Keep the following limitations in mind when using PostgreSQL with Grafana Alerting:

- **Time series format only:** Alert rules require the query **Format** set to **Time series**. Table format queries can't be used as alert conditions.
- **No fill mode in alerts:** The fill parameter in `$__timeGroup` (for example, `$__timeGroup(col, '5m', 0)`) isn't applied during alert evaluation. Missing data points remain as gaps.
- **Single query per condition:** Each alert condition evaluates a single query. If you need to combine results from multiple queries, use [math expressions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/#add-expressions) in the alert rule.
- **Template variables aren't supported:** Alert queries don't support dashboard template variables because alerts evaluate independently of any dashboard context. Hard-code values or use [alert rule labels](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/) instead.

## Create an alert rule

To create an alert rule that uses PostgreSQL:

1. Go to **Alerting** in the left menu and select **Alert rules**.
1. Click **New alert rule**.
1. In the query section, select your **PostgreSQL** data source.
1. Set the query **Format** to **Time series**.
1. Write a SQL query that returns a `time` column and one or more numeric values.
1. Configure the condition, evaluation group, and notification settings.
1. Save the rule.

For step-by-step guidance, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example queries for alerting

The following examples show common PostgreSQL alerting patterns.

### Threshold alert

The following query returns a time series suitable for a threshold alert (for example, alert when average value exceeds a limit):

```sql
SELECT
  $__timeGroupAlias("time_date_time", '5m'),
  avg("value_double") AS value
FROM test_data
WHERE $__timeFilter("time_date_time")
GROUP BY time
ORDER BY time
```

Use condition types such as **Is above** or **Is below** in the alert rule to evaluate the series.

### Multi-dimensional alert

To create separate alert instances for each value of a label column, include a string column in your query. Grafana creates one alert instance per unique label value, so you can monitor each host, service, or region independently.

```sql
SELECT
  $__timeGroupAlias("time_date_time", '5m'),
  avg("value_double") AS value,
  hostname
FROM test_data
WHERE $__timeFilter("time_date_time")
GROUP BY time, hostname
ORDER BY time
```

This query produces a separate time series for each `hostname`. The alert rule evaluates the condition against each series independently, so you receive individual notifications per host.

### Alert on row count

To alert when the number of rows in a time window exceeds a threshold (for example, error count):

```sql
SELECT
  $__timeGroupAlias("created_at", '5m'),
  count(*) AS error_count
FROM error_logs
WHERE $__timeFilter("created_at")
  AND severity = 'ERROR'
GROUP BY time
ORDER BY time
```

## Template annotations and labels

You can use [template annotations and labels](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/) to include query results or metadata in alert notifications and labels.

## Best practices

Follow these best practices when using PostgreSQL for alerting:

- **Keep queries fast:** Alert rules evaluate on a schedule (for example, every minute). Avoid expensive joins, subqueries, or scanning large tables without indexes. Add indexes on time columns and any columns used in `WHERE` clauses.
- **Use `$__timeFilter`:** Always include a `$__timeFilter` or `$__unixEpochFilter` macro to limit data to the evaluation window. Without it, the query scans the entire table on every evaluation.
- **Handle `NULL` values:** If your data contains `NULL` values, Grafana treats them as "no data." Configure the alert rule's **No data** behavior to match your expectations (for example, **Alerting**, **No data**, or **OK**).
- **Test in Explore first:** Verify your query returns the expected time series in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/) before using it in an alert rule. Check the **Format** is set to **Time series** and results are sorted by time.
- **Set realistic evaluation intervals:** Choose an evaluation interval that gives your query enough time to complete. If your query takes 10 seconds, don't evaluate every 10 seconds—allow for headroom.

# PostgreSQL annotations

Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics.
You can use PostgreSQL as a data source for annotations to display events such as deployments, alerts, or other significant occurrences on your visualizations.

For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/).

## Before you begin

Before creating PostgreSQL annotations, ensure you have:

- [A configured PostgreSQL data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/postgres/configure/).
- Tables containing event data with timestamp fields.
- Read access to the tables containing your events.

## Create an annotation query

To add a PostgreSQL annotation to your dashboard:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Annotation query**.
1. Enter a name for the annotation query.
1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox.
1. Select a color for the annotation event markers.
1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed.
1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed.
1. Click **Open query editor** to open the **Annotation Query** dialog box.
1. Select your **PostgreSQL** data source from the **Data source** drop-down list.
1. Configure the annotation query and field mappings.
1. (Optional) Click **Test annotation query** to ensure that the query is working properly.
1. Click **Close** when you've completed the query setup.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## Query columns

Your annotation query must return a `time` column and can optionally include `timeend`, `text`, and `tags` columns.

| Column    | Required | Description                                                                                   |
| --------- | -------- | --------------------------------------------------------------------------------------------- |
| `time`    | Yes      | The timestamp for the annotation. Can be a native SQL date/time type or UNIX epoch value.     |
| `timeend` | No       | The end timestamp for range annotations. Creates a shaded region instead of a vertical line.  |
| `text`    | No       | The annotation description displayed when you hover over the annotation.                      |
| `tags`    | No       | Tags for the annotation as a comma-separated string. Helps categorize and filter annotations. |

## Example queries

The following examples show common annotation query patterns.

### Basic annotation with epoch time

Display events using UNIX epoch timestamps:

```sql
SELECT
  epoch_time as time,
  metric1 as text,
  concat_ws(', ', metric1::text, metric2::text) as tags
FROM public.test_data
WHERE $__unixEpochFilter(epoch_time)
```

### Range annotation with start and end time

Display events with duration as shaded regions:

```sql
SELECT
  epoch_time as time,
  epoch_time_end as timeend,
  metric1 as text,
  concat_ws(', ', metric1::text, metric2::text) as tags
FROM public.test_data
WHERE $__unixEpochFilter(epoch_time)
```

### Annotation with native SQL date/time

Display events using native PostgreSQL date/time columns:

```sql
SELECT
  native_date_time as time,
  metric1 as text,
  concat_ws(', ', metric1::text, metric2::text) as tags
FROM public.test_data
WHERE $__timeFilter(native_date_time)
```

### Deployment annotations

Display deployment events:

```sql
SELECT
  deployed_at as time,
  concat('Deployed ', version, ' to ', environment) as text,
  environment as tags
FROM deployments
WHERE $__timeFilter(deployed_at)
```

### Range annotation for maintenance windows

Display maintenance windows as shaded regions:

```sql
SELECT
  start_time as time,
  end_time as timeend,
  concat('Maintenance: ', description) as text,
  'maintenance' as tags
FROM maintenance_windows
WHERE $__timeFilter(start_time)
```

## Macros

Use these macros in your annotation queries to filter by the dashboard time range:

| Macro                        | Description                                                      |
| ---------------------------- | ---------------------------------------------------------------- |
| `$__timeFilter(column)`      | Filters by time range using a native SQL date/time column.       |
| `$__unixEpochFilter(column)` | Filters by time range using a column with UNIX epoch timestamps. |

## Best practices

Follow these best practices when creating PostgreSQL annotations:

- **Use time filters:** Always include `$__timeFilter()` or `$__unixEpochFilter()` to limit results to the dashboard time range.
- **Keep queries efficient:** Add indexes on time columns and filter columns to improve query performance.
- **Use meaningful text:** Include descriptive information in the `text` column to make annotations useful.
- **Organize with tags:** Use consistent tag values to categorize annotations and enable filtering.
- **Test queries first:** Verify your query returns expected results in Explore before adding it as an annotation.

# PostgreSQL template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
Grafana displays these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
Grafana refers to such variables as **template variables**.

For an introduction to templating and template variables, refer to [Templating](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/).

## Query variable

A query variable in Grafana dynamically retrieves values from your data source using a query. With a query variable, you can write a PostgreSQL query that returns values such as measurement names, key names, or key values that are shown in a drop-down select box.

The PostgreSQL data source includes a dedicated variable query editor with the full SQL editor (Code mode with auto-completion and syntax highlighting) and **Value Field** / **Text Field** drop-downs. After you write and run a query, the editor automatically detects the returned columns and populates the field drop-downs so you can map which column provides the variable value and which provides the display label.

To create a query variable:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Variable**.
1. Select **Query** as the variable type.
1. Enter a **Name** for your variable (for example, `host`).
1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed.
1. Click **Open variable editor** to open the **Query Variable** dialog box.
1. Select your PostgreSQL data source.
1. Write a SQL query in the editor. Macros like `$__timeFilter` are supported.
1. Optionally set **Value Field** and **Text Field** to control which columns map to the variable value and display label.

For example, the following query returns all values from the `hostname` column:

```sql
SELECT hostname FROM host
```

A query can return multiple columns, and Grafana automatically generates a list using the values from those columns. For example, the following query returns values from both the `hostname` and `hostname2` columns, which are included in the variable's drop-down list.

```sql
SELECT host.hostname, other_host.hostname2 FROM host JOIN other_host ON host.city = other_host.city
```

To use time range dependent macros like `$__timeFilter(column)` in your query, you must set the template variable's refresh mode to **On Time Range Change**.

```sql
SELECT event_name FROM event_log WHERE $__timeFilter(time_column)
```

### Key/value variables

You can create a key/value variable so the drop-down shows a user-friendly label (for example, hostname) while panel queries use a different value (for example, ID). Use the variable editor’s **Value field** and **Text field** at the bottom of the query section to specify which query columns supply the value and the label. Your query can use any column names; you do not need `__value` or `__text` in the SQL.

Example: run a query that returns `hostname` and `id`, then set **Text field** to `hostname` and **Value field** to `id`.

```sql
SELECT hostname, id FROM host
```

Note that the values in the text column should be unique. If there are duplicates, Grafana uses only the first matching entry.

Alternatively, you can use the legacy approach: return columns named `__text` and `__value` in your query (for example, `SELECT hostname AS __text, id AS __value FROM host`).

### Nested variables

You can create nested variables, where one variable depends on the value of another. For example, if you have a variable named `region`, you can configure a `hosts` variable to only show hosts from the selected region. If `region` is a multi-value variable, use the `IN` operator instead of `=` to match against multiple selected values.

```sql
SELECT hostname FROM host WHERE region IN($region)
```

{{< admonition type="note" >}}
When using a multi-value PostgreSQL query variable with **Include All option** enabled and **Display** set to **Hidden**, the variable defaults to the first returned value, not **All**.
This is because Grafana persists the active selection at dashboard save time, and a hidden variable has no UI for users to change it.
To default a hidden variable to **All**, temporarily show the variable, select **All** in the drop-down, save the dashboard, then hide the variable and save again.
{{< /admonition >}}

### Filter results with `__searchFilter`

Using `__searchFilter` in the query field filters the query results based on user input in the drop-down selection box. If nothing is entered, the default value for `__searchFilter` is `%`.

You must enclose the `__searchFilter` expression in quotes because Grafana doesn't add them automatically.

The following example enables real-time searching for `hostname` as the user types in the drop-down selection box:

```sql
SELECT hostname FROM host WHERE hostname LIKE '$__searchFilter'
```

For tables with a large number of rows, combine `__searchFilter` with `LIMIT` to keep the drop-down responsive:

```sql
SELECT DISTINCT hostname
FROM host
WHERE hostname LIKE '$__searchFilter'
ORDER BY hostname
LIMIT 100
```

### Regex variable

You can use a **Regex** type variable to filter or transform values from another variable. For example, if you have a variable named `server` that returns values like `web-prod-01`, `web-staging-01`, and `db-prod-01`, you can create a regular expression variable to extract only the environment:

1. Create a new variable with **Type: Query**.
1. Set the query to return the raw values.
1. In the **Regex** field, enter a pattern to extract or filter. For example, `/^web-(.+)-\d+$/` extracts `prod` and `staging` from web server names.

You can also use regular expression to filter a variable's options. For example, setting the **Regex** field to `/prod/` on a variable that returns all server names limits the drop-down to only production servers.

## Multi-property variables

The PostgreSQL data source supports **multi-property variables**. Use them when the same logical concept has different identifiers in different contexts (for example, an environment called `dev` in one system and `development` in another). Instead of maintaining several variables in sync, you can map all of those values to one variable and reference the property you need in each panel or query.

{{< figure alt="PostgreSQL multi-property variable example"  src="/media/docs/postgres/postgreSQL-multi-prop-variable-v12.3.png" >}}

You can create a multi-property variable with either **Type: Custom** or **Type: Query**:

- **Type: Custom** – In **Custom options** > **JSON**, paste your own JSON array with the mapping. Each object in the array can have any number of properties; use `text` and `value` for the label and value shown in the drop-down, and add additional properties as needed. For the JSON format and examples, refer to [Multi-property custom variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#multi-property-custom-variables) in Add and manage variables.

- **Type: Query** – Write a SQL query that returns multiple columns. In the variable editor, set **Value field** and **Text field** to the columns that supply the value and the label for the drop-down. Add one column per property you want to reference; each column name becomes a property name. In panels and queries, reference a property with `${varName.columnName}`.

**Example (Type: Query):** A variable named `env` that lists environments with different identifiers per cloud. In the variable editor, set **Text field** to `name` and **Value field** to `id`.

```sql
SELECT
  name,
  id,
  aws_identifier AS env_aws,
  azure_identifier AS env_azure
FROM environments
```

In a panel query you might use `$env.env_aws` for an AWS-related query and `$env.env_azure` for an Azure-related query. For more on the concept, refer to [Configure multi-property variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#configure-multi-property-variables) in [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/).

## Use variables in queries

Grafana automatically quotes template variable values only when the template variable is a **multi-value** variable.

When using a multi-value variable, use the `IN` comparison operator instead of `=` to match against multiple values.

Grafana supports two syntaxes for using variables in queries:

- **`$<varname>` syntax**

Example with a template variable named `hostname`:

```sql
SELECT
  atimestamp AS time,
  aint AS value
FROM table
WHERE $__timeFilter(atimestamp) AND hostname IN($hostname)
ORDER BY atimestamp ASC
```

- **`[[varname]]` syntax**

Example with a template variable named `hostname`:

```sql
SELECT
  atimestamp AS time,
  aint AS value
FROM table
WHERE $__timeFilter(atimestamp) AND hostname IN([[hostname]])
ORDER BY atimestamp ASC
```

### Multi-value variable with IN clause

When a variable has **Multi-value** enabled, Grafana automatically adds quotes around each selected value. For example, if the user selects `web01` and `web02`, `$hostname` expands to `'web01','web02'`. Use the `IN` operator to match:

```sql
SELECT
  $__timeGroupAlias("created_at", '5m'),
  count(*) AS requests
FROM access_log
WHERE $__timeFilter("created_at")
  AND hostname IN($hostname)
GROUP BY time
ORDER BY time
```

If only a single value is selected, the `IN` clause still works correctly—`IN('web01')` is equivalent to `= 'web01'`.

### Multi-value variable with numeric columns

For numeric columns (such as IDs), disable quoting with the `csv` format option so values aren't wrapped in quotes:

```sql
SELECT name FROM host WHERE id IN(${host_id:csv})
```

If `host_id` has values `1`, `2`, and `3` selected, this expands to `IN(1,2,3)` instead of `IN('1','2','3')`.

### Disable quoting for multi-value variables

By default, Grafana formats multi-value variables as a quoted, comma-separated string. For example, if `server01` and `server02` are selected, the result is `'server01','server02'`. To disable quoting, use the `csv` formatting option:

```text
${servers:csv}
```

This outputs the values as an unquoted comma-separated list.

Refer to [Advanced variable format options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/#advanced-variable-format-options) for additional information.

# Troubleshoot PostgreSQL data source issues

This document provides troubleshooting information for common errors you may encounter when using the PostgreSQL data source in Grafana.

## Connection errors

The following errors occur when Grafana cannot establish or maintain a connection to PostgreSQL.

### Failed to connect to PostgreSQL

**Error message:** `failed to connect to ... : connect: connection refused` or `dial tcp: connect: connection refused`

**Cause:** Grafana cannot establish a network connection to the PostgreSQL server.

**Solution:**

1. Verify that the Host URL is correct in the data source configuration.
1. Check that PostgreSQL is running and accessible from the Grafana server.
1. Verify the port is correct (the PostgreSQL default port is `5432`).
1. Ensure there are no firewall rules blocking the connection.
1. Check that PostgreSQL is configured to accept connections from the Grafana server in `pg_hba.conf`.
1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your PostgreSQL instance is not publicly accessible.

### Grafana Cloud can't reach a private database

**Error message:** `dial tcp: connect: connection refused`, `i/o timeout`, or `context deadline exceeded` when using Grafana Cloud with a database on a private network.

**Cause:** Grafana Cloud runs in a hosted environment and can't directly reach databases on `localhost`, `127.0.0.1`, or private IP ranges (`10.x`, `172.16.x`, `192.168.x`). This is the most common issue when migrating from self-managed Grafana to Grafana Cloud.

**Solution:**

1. Set up [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) to create a secure tunnel between Grafana Cloud and your private network.
1. Install the PDC agent on a machine that has network access to your PostgreSQL instance.
1. If you experience intermittent connection drops with the Docker-based PDC agent, try the binary-based agent instead. This has resolved stability issues in some environments.
1. Update the **Host URL** in the data source settings to use the hostname as seen from the PDC agent's network (not `localhost`).

### Request timed out

**Error message:** "context deadline exceeded" or "i/o timeout"

**Cause:** The connection to PostgreSQL timed out before receiving a response.

**Solution:**

1. Check the network latency between Grafana and PostgreSQL.
1. Verify that PostgreSQL is not overloaded or experiencing performance issues.
1. Increase the **Max lifetime** setting in the data source configuration under **Connection limits**.
1. Reduce the time range or complexity of your query.
1. Check if any network devices (load balancers, proxies) are timing out the connection.

### Host not found

**Error message:** `failed to connect to ... : hostname resolving error` or `lookup hostname: no such host`

**Cause:** The hostname specified in the data source configuration can't be resolved.

**Solution:**

1. Verify the hostname is spelled correctly.
1. Check that DNS resolution is working on the Grafana server.
1. Try using the database's public IP address instead of a hostname. This is a useful diagnostic step and can serve as a workaround if DNS resolution is the issue.
1. Ensure the PostgreSQL server is accessible from the Grafana server's network.
1. For Grafana Cloud, DNS resolution behavior can differ between stack regions and cloud providers. If a hostname resolves from one stack but not another, try the direct IP address and contact [Grafana Support](https://grafana.com/docs/grafana-cloud/account-management/support/).

## Authentication errors

The following errors occur when there are issues with authentication credentials or permissions.

### No PostgreSQL user name specified

**Error message:** `FATAL: no PostgreSQL user name specified in startup packet (SQLSTATE 28000)`

**Cause:** The connection to PostgreSQL was attempted without a username. This typically means the **Username** field in the data source configuration is empty or was cleared.

**Solution:**

1. Open the data source settings and verify the **Username** field contains a valid PostgreSQL user.
1. Click **Save & test** and confirm the connection succeeds.
1. If the username disappears after saving, check your Grafana version. A bug in Grafana v13.1 on the fast release channel caused the username field to be cleared on save. Upgrading to a patched release or switching to the steady release channel resolves this issue.

### Password authentication failed

**Error message:** `failed to connect to ... : server error: FATAL: password authentication failed for user "username" (SQLSTATE 28P01)`

**Cause:** The username or password is incorrect.

**Solution:**

1. Verify that the username and password are correct in the data source configuration.
1. Check that the user exists in PostgreSQL.
1. Verify the password has not expired.
1. If no password is specified, ensure a [PostgreSQL password file](https://www.postgresql.org/docs/current/static/libpq-pgpass.html) is configured.

### Permission denied

**Error message:** `ERROR: permission denied for table table_name (SQLSTATE 42501)` or `ERROR: permission denied for schema schema_name (SQLSTATE 42501)`

**Cause:** The database user does not have permission to access the requested table or schema.

**Solution:**

1. Verify the user has `SELECT` permissions on the required tables.
1. Grant the necessary permissions:

   ```sql
   GRANT USAGE ON SCHEMA schema_name TO grafanareader;
   GRANT SELECT ON schema_name.table_name TO grafanareader;
   ```

1. Check that the user has access to the correct database.
1. Verify the search path includes the schema containing your tables.

### No pg_hba.conf entry

**Error message:** `failed to connect to ... : server error: FATAL: no pg_hba.conf entry for host "ip_address", user "username", database "database_name" (SQLSTATE 28000)`

**Cause:** PostgreSQL is not configured to accept connections from the Grafana server.

**Solution:**

1. Edit the `pg_hba.conf` file on the PostgreSQL server.
1. Add an entry to allow connections from the Grafana server:

   ```text
   host    database_name    username    grafana_ip/32    md5
   ```

1. Reload PostgreSQL configuration: `SELECT pg_reload_conf();`
1. If using SSL, ensure the correct authentication method is specified (for example, `hostssl` instead of `host`).

## TLS and certificate errors

The following errors occur when there are issues with TLS configuration.

### Certificate verification failed

**Error message:** "x509: certificate signed by unknown authority" or "certificate verify failed"

**Cause:** Grafana cannot verify the TLS certificate presented by PostgreSQL.

**Solution:**

1. Set the **TLS/SSL Mode** to the appropriate level (`require`, `verify-ca`, or `verify-full`).
1. If using a self-signed certificate, add the CA certificate in **TLS/SSL Auth Details**.
1. Verify the certificate chain is complete and valid.
1. Ensure the certificate has not expired.
1. For testing only, set **TLS/SSL Mode** to `disable` (not recommended for production).

### SSL not supported

**Error message:** `failed to connect to ... : server refused TLS connection` or `server does not support SSL`

**Cause:** The PostgreSQL server is not configured for SSL connections, but the data source requires SSL.

**Solution:**

1. Set **TLS/SSL Mode** to `disable` if SSL is not required.
1. Alternatively, enable SSL on the PostgreSQL server by configuring `ssl = on` in `postgresql.conf`.
1. Ensure the server has valid SSL certificates configured.

### Client certificate error

**Error message:** "TLS: failed to find any PEM data in certificate input" or "could not load client certificate"

**Cause:** The client certificate or key is invalid or incorrectly formatted.

**Solution:**

1. Verify the certificate and key are in PEM format.
1. Ensure the certificate file path is correct and readable by the Grafana process.
1. Check that the certificate and key match (belong to the same key pair).
1. If using certificate content, ensure you've pasted the complete certificate including headers.

## Database errors

The following errors occur when there are issues with the database configuration.

### Database does not exist

**Error message:** `failed to connect to ... : server error: FATAL: database "database_name" does not exist (SQLSTATE 3D000)`

**Cause:** The specified database name is incorrect or the database doesn't exist.

**Solution:**

1. Verify the database name in the data source configuration.
1. Check that the database exists: `\l` in `psql` or `SELECT datname FROM pg_database;`
1. Ensure the database name is case-sensitive and matches exactly.
1. Verify the user has permission to connect to the database.

### Relation does not exist

**Error message:** `ERROR: relation "table_name" does not exist (SQLSTATE 42P01)`

**Cause:** The specified table or view does not exist, or the user cannot access it.

**Solution:**

1. Verify the table name is correct and exists in the database.
1. Check the schema name if the table is not in the public schema.
1. Use fully qualified names: `schema_name.table_name`.
1. Verify the user has `SELECT` permission on the table.
1. Check the search path: `SHOW search_path;`

## Query errors

The following errors occur when there are issues with SQL syntax or query execution.

### Query truncated by double-dash in string literals

**Error message:** Unexpected syntax errors or truncated results when string values contain `--` (double dash).

**Cause:** In older Grafana versions, the SQL comment-stripping parser didn't correctly handle `--` inside single-quoted strings. A query like `WHERE name = 'value--suffix'` would be truncated at the `--`, causing the rest of the query to be silently dropped. This also affected strings with consecutive hyphens (for example, `'10YDE-VE-------2'` would be truncated to `'10YDE-VE`).

This was fixed with a quote-aware comment-stripping parser (PR #121772). The fix also handles PostgreSQL dollar-quoted strings (`$$...$$`). It is available in Grafana 13.1 and later, Grafana 13.0.2 and later, and these 12.x patch releases: 12.1.11, 12.2.9, 12.3.7, and 12.4.3.

**Solution:**

1. Upgrade to a Grafana release that includes the fix (13.1+, 13.0.2+, or one of the 12.x patch releases listed above).
1. If you can't upgrade immediately, work around the issue by using PostgreSQL string concatenation to avoid literal `--` in your queries:

   ```sql
   WHERE name = 'value' || '--' || 'suffix'
   ```

1. Alternatively, use a parameterized approach with a template variable for the value.

### Query syntax error

**Error message:** `ERROR: syntax error at or near "keyword" (SQLSTATE 42601)`

**Cause:** The SQL query contains invalid syntax.

**Solution:**

1. Check your query syntax for typos or invalid keywords.
1. Verify column and table names are correctly quoted if they contain special characters or are reserved words.
1. Use double quotes for identifiers: `"column_name"`.
1. Test the query directly in a PostgreSQL client such as `psql`.

### Column does not exist

**Error message:** `ERROR: column "column_name" does not exist (SQLSTATE 42703)`

**Cause:** The specified column name is incorrect or doesn't exist in the table.

**Solution:**

1. Verify the column name is spelled correctly.
1. Check that column names are case-sensitive in PostgreSQL when quoted.
1. Use the correct quoting for column names: `"Column_Name"` for case-sensitive names.
1. Verify the column exists in the table: `\d table_name` in `psql`.

### No time column found

**Error message:** "no time column found" or time series visualization shows no data

**Cause:** The query result does not include a properly formatted time column.

**Solution:**

1. Ensure your query includes a column named `time` that returns a timestamp or epoch value.
1. Use an alias to rename your time column: `SELECT created_at AS time`.
1. Ensure the time column is of type `timestamp`, `timestamptz`, or a numeric epoch value.
1. Order results by the time column: `ORDER BY time ASC`.

### Macro expansion error

**Error message:** "macro '$\_\_timeFilter' not found" or incorrect query results with macros

**Cause:** Grafana macros aren't being properly expanded.

**Solution:**

1. Verify the macro syntax is correct, for example `$\_\_timeFilter(time_column)`.
1. Ensure the column name passed to the macro exists in your table.
1. Check that the macro isn't inside a SQL comment (`--` or `/* */`). Grafana strips comments before expanding macros, so macros inside comments are silently ignored.
1. Use the **Preview** toggle in Builder mode to see the expanded query.
1. Open the [Query inspector](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/#query-inspector) to view the exact SQL sent to PostgreSQL after macro expansion.
1. For time-based macros, ensure the column contains timestamp data.

## Performance issues

The following issues relate to slow query execution or resource constraints.

### Query timeout

**Error message:** "canceling statement due to statement timeout" or "query timeout"

**Cause:** The query took longer than the configured timeout.

**Solution:**

1. Reduce the time range of your query.
1. Add indexes to columns used in WHERE clauses and joins.
1. Use the `$\_\_timeFilter` macro to limit data to the dashboard time range.
1. Increase the statement timeout in PostgreSQL if you have admin access.
1. Optimize your query to reduce complexity.

### Too many connections

**Error message:** `failed to connect to ... : server error: FATAL: too many connections for role "username" (SQLSTATE 53300)` or `connection pool exhausted`

**Cause:** The maximum number of connections to PostgreSQL has been reached.

**Solution:**

1. Reduce the **Max open** connections setting in the data source configuration.
1. Increase `max_connections` in PostgreSQL's `postgresql.conf` if you have admin access.
1. Check for connection leaks in other applications connecting to the same database.
1. Enable **Auto max idle** to automatically manage idle connections.

### Slow query performance

**Cause:** Queries take a long time to execute.

**Solution:**

1. Reduce the time range of your query.
1. Add appropriate indexes to your tables.
1. Use the `$\_\_timeFilter` macro to limit the data scanned.
1. Increase the **Min time interval** setting to reduce the number of data points.
1. Use `EXPLAIN ANALYZE` in PostgreSQL to identify query bottlenecks.
1. Consider using materialized views for complex aggregations.

## Provisioning errors

The following errors occur when provisioning the data source via YAML.

### Invalid provisioning configuration

**Error message:** "metric request error" or data source test fails after provisioning

**Cause:** The provisioning YAML file contains incorrect configuration.

**Solution:**

1. Ensure parameter names match the expected format exactly.
1. Verify the database name is **not** included in the URL.
1. Use the correct format for the URL: `hostname:port`.
1. Check that string values are properly quoted in the YAML file.
1. Refer to the [provisioning example](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/postgres/configure/#provision-the-data-source) for the correct format.

Example correct configuration:

```yaml
datasources:
  - name: Postgres
    type: postgres
    url: localhost:5432
    user: grafana
    secureJsonData:
      password: 'Password!'
    jsonData:
      database: grafana
      sslmode: 'disable'
```

## Other common issues

The following issues don't produce specific error messages but are commonly encountered.

### Empty query results

**Cause:** The query returns no data.

**Solution:**

1. Verify the time range includes data in your database.
1. Check that table and column names are correct.
1. Test the query directly in PostgreSQL.
1. Ensure filters are not excluding all data.
1. Verify the `$\_\_timeFilter` macro is using the correct time column.

### TimescaleDB functions not available

**Cause:** TimescaleDB-specific functions like `time_bucket` are not available in the query builder.

**Solution:**

1. Enable the **TimescaleDB** toggle in the data source configuration under **PostgreSQL Options**.
1. Verify TimescaleDB is installed and enabled in your PostgreSQL database.
1. Check that the `timescaledb` extension is created: `CREATE EXTENSION IF NOT EXISTS timescaledb;`

### Data appears delayed or missing recent points

**Cause:** The visualization doesn't show the most recent data.

**Solution:**

1. Check the dashboard time range and refresh settings.
1. Verify the **Min time interval** is not set too high.
1. Ensure data has been committed to the database (not in an uncommitted transaction).
1. Check for clock synchronization issues between Grafana and PostgreSQL.

## Get additional help

If you continue to experience issues after following this troubleshooting guide:

1. Check the [PostgreSQL documentation](https://www.postgresql.org/docs/) for database-specific guidance.
1. Review the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Contact Grafana Support if you are a Cloud Pro, Cloud Contracted, or Enterprise user.
1. When reporting issues, include:
   - Grafana version
   - PostgreSQL version
   - Error messages (redact sensitive information)
   - Steps to reproduce
   - Relevant configuration such as data source settings, TLS mode, and connection limits (redact passwords and other credentials)

# Prometheus alerting

You can use Grafana Alerting with Prometheus to create alerts based on your time-series data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met.

For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/).

## Before you begin

Before creating alerts with Prometheus, ensure you have:

- A Prometheus data source configured in Grafana
- Appropriate permissions to create alert rules
- Understanding of the PromQL metrics you want to monitor

## Alert rule types

Prometheus supports two alerting workflows in Grafana:

| Type                            | Description                                                                                                                                                                                                                                                                      |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Grafana-managed alert rules** | Alert rules defined and evaluated within Grafana, using Prometheus as the query data source. You create and manage these entirely in the Grafana Alerting UI.                                                                                                                    |
| **Data source-managed rules**   | Alerting rules defined in Prometheus itself (in `prometheus.yml` or rule files). When **Manage alerts via Alerting UI** is enabled in the data source configuration, Grafana displays these existing rules in the Alerting UI. For Prometheus (unlike Mimir), this is read-only. |

## Create a Grafana-managed alert rule

To create a Grafana-managed alert rule using Prometheus:

1. Navigate to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for the alert rule.
1. Select your **Prometheus** data source.
1. Write a PromQL query in the query editor.
1. Configure the alert condition (for example, when the last value is above a threshold).
1. Set the evaluation interval and pending period.
1. Configure notifications and labels.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## View data source-managed rules

When **Manage alerts via Alerting UI** is enabled in the Prometheus data source configuration, Grafana fetches and displays alerting rules defined in Prometheus. These appear in the Alerting UI alongside Grafana-managed rules but are marked as data source-managed.

For Prometheus data sources, this view is read-only. To modify these rules, update your Prometheus rule files directly.

{{< admonition type="note" >}}
For Mimir and Cortex data sources, the Alerting UI supports both viewing and creating data source-managed rules. Prometheus only supports viewing.
{{< /admonition >}}

## Evaluation groups and intervals

Alert rules are organized into evaluation groups. Each group has an evaluation interval that determines how frequently the rules in that group are evaluated. For example, an evaluation interval of `1m` means the alert query runs every 60 seconds.

The **pending period** determines how long a condition must be continuously true before the alert fires. For example, with a 1-minute evaluation interval and a 5-minute pending period, the condition must be true for 5 consecutive evaluations before firing.

Choose evaluation intervals based on your use case:

- **15s–30s:** Critical infrastructure alerts where fast detection matters.
- **1m:** Standard monitoring alerts (recommended default).
- **5m:** Non-urgent or noisy metrics where you want to reduce evaluation load.

## Example alert queries

The following examples show common alerting scenarios with Prometheus. Each example shows the PromQL query and how to configure the alert condition.

### Alert on high CPU usage

Monitor CPU usage and alert when it exceeds 90%:

**Query A:**

```promql
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[$__rate_interval])) * 100)
```

**Condition:** Set the threshold to alert when the last value of **A** is above `90`.

This approach separates the metric query from the threshold, making it easier to adjust the threshold later without editing the PromQL.

### Alert on high memory usage

Monitor memory usage across nodes:

**Query A:**

```promql
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
```

**Condition:** Alert when the last value of **A** exceeds `85`.

### Alert on high error rate

Monitor HTTP error rates per service:

**Query A:**

```promql
sum(rate(http_requests_total{status=~"5.."}[$__rate_interval])) by (job)
  /
sum(rate(http_requests_total[$__rate_interval])) by (job)
  * 100
```

**Condition:** Alert when the last value of **A** exceeds `5` (meaning error rate above 5%).

### Alert on target down

Monitor whether Prometheus scrape targets are reachable:

**Query A:**

```promql
up{job="myservice"}
```

**Condition:** Alert when the last value of **A** is below `1`.

### Alert when a metric disappears

Use `absent()` to detect when a metric stops being scraped entirely, for example, when a service crashes and no longer reports metrics:

**Query A:**

```promql
absent(up{job="myservice"})
```

**Condition:** Alert when the last value of **A** equals `1` (the `absent()` function returns 1 when the metric is missing, and nothing when the metric exists).

For detecting staleness over a time window (metric exists but hasn't reported recently):

```promql
absent_over_time(up{job="myservice"}[5m])
```

### Multi-condition alert (high latency AND high traffic)

Use multiple queries and expressions to alert only when multiple conditions are true simultaneously. This reduces noise by avoiding alerts during low-traffic periods.

**Query A** (P95 latency):

```promql
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="api"}[$__rate_interval])) by (le))
```

**Query B** (request rate):

```promql
sum(rate(http_requests_total{job="api"}[$__rate_interval]))
```

**Expression C** (Math, combining both conditions):

```
$A > 2 && $B > 100
```

**Condition:** Alert when **C** has a value (it only returns data when both latency exceeds 2 seconds AND request rate exceeds 100 req/s).

## Recording rules

[Grafana-managed recording rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/) pre-compute expensive PromQL expressions on a schedule and write the results back to a Prometheus-compatible data source as a new metric. This lets you query the pre-aggregated metric instead of re-evaluating the expensive expression on every dashboard load or alert evaluation.

### When to use recording rules

- **Dashboard panels that query the same expensive expression repeatedly:** Pre-compute it once and query the result.
- **Alert rules on complex expressions:** Simplify the alert query by alerting on the pre-aggregated metric.
- **High-cardinality aggregations:** Reduce thousands of series to a handful of pre-computed series.

### Set up Grafana-managed recording rules for Prometheus

1. **Enable the data source as a recording rules target:** In the Prometheus data source configuration, verify that **Allow as recording rules target** is toggled on (it's on by default). This allows Grafana to write recording rule results back to this instance.

1. **Verify write access:** The Prometheus-compatible backend must support remote write. Grafana Cloud Metrics (Mimir) and self-managed Mimir support this natively. Standard Prometheus requires the `--web.enable-remote-write-receiver` flag (Prometheus 2.33+).

1. **Create a recording rule:**
   1. Navigate to **Alerting** > **Alert rules**.
   1. Click **New alert rule**.
   1. Select **Recording rule** as the rule type (under the Grafana-managed section).
   1. Enter the PromQL expression you want to pre-compute:

      ```promql
      sum(rate(http_requests_total[$__rate_interval])) by (service)
      ```

   1. Enter a metric name for the result (for example, `service:http_requests:rate5m`). Follow the Prometheus [recording rule naming convention](https://prometheus.io/docs/practices/rules/#naming-and-aggregation): `level:metric:operations`.
   1. Select the **Target data source**, which is the Prometheus instance where results are written.
   1. Set the evaluation interval (for example, every 1 minute).
   1. Click **Save rule**.

1. **Query the recorded metric:** After the first evaluation, the new metric is available for dashboards and alerts:

   ```promql
   service:http_requests:rate5m{service="api"}
   ```

### Recording rules with PDC

If your Prometheus instance is behind [Private data source connect (PDC)](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/configure/#private-data-source-connect), Grafana can write recording rule results through the PDC tunnel. No additional configuration is needed, because PDC supports both reads and writes.

### Limitations

- Recording rules require the target data source to support remote write. Standard Prometheus needs the `--web.enable-remote-write-receiver` flag.
- Thanos does not support recording rules as a write target (refer to the [Prometheus type comparison](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/configure/#performance)).
- Recording rule evaluation uses the configured evaluation interval, not dashboard time ranges. Use a fixed range vector (for example, `[5m]`) rather than `$__rate_interval`.

For more information, refer to [Create Grafana-managed recording rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/).

## Limitations

When using Prometheus with Grafana Alerting, be aware of the following limitations.

### Template variables not supported

Alert queries don't support template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$instance` or `$job` aren't resolved.

If your dashboard query uses template variables, create a separate query for alerting with hard-coded values or use label matchers directly.

### Query complexity

Complex queries with many nested functions or large result sets may timeout or fail to evaluate. Simplify queries for alerting by:

- Reducing the time range used in range vectors
- Using appropriate aggregation to limit the number of returned series
- Adding label filters to narrow the data scanned
- Using recording rules to pre-compute expensive expressions

### OAuth token handling differs between Explore and Alerting

When using OAuth-authenticated Prometheus endpoints (Google Managed Prometheus, Azure Managed Prometheus), queries may succeed in Explore and dashboards but fail intermittently during alert evaluation. This happens because the alerting backend handles token refresh differently from the interactive query path.

If you're using GCP, consider the **datasource-syncer** pattern, a sidecar process that refreshes OAuth tokens and updates the data source credentials on a schedule shorter than the token lifetime.

For detailed troubleshooting steps, refer to [OAuth token expiration errors](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/troubleshooting/#oauth-token-expiration-errors-gcp-and-azure).

### Data source-managed rules are read-only

Grafana can display Prometheus alerting rules but can't create or modify them through the UI. To manage Prometheus-native alerting rules, edit your Prometheus rule files directly and reload the configuration.

## Configure alert state for execution errors

By default, when a Grafana-managed alert rule encounters an execution error or timeout (such as a network blip, i/o timeout, or a transient 502 from Prometheus), the rule enters an **Error** state, which fires the alert. This can cause false alarms and spam on-call teams when the underlying issue is a brief connectivity interruption rather than a genuine threshold breach.

To prevent false positives from transient errors, configure the **Alert state if execution error or timeout** setting on each alert rule:

1. Open the alert rule for editing.
1. In the alert conditions section, locate **Alert state if execution error or timeout**.
1. Change the value from **Alerting** (default) to one of:
   - **Keep Last State:** The alert retains its previous state (firing or normal) until a successful evaluation occurs. This is the recommended setting for most Prometheus alert rules.
   - **OK:** The alert is set to normal during the error, preventing it from firing.
1. Click **Save rule**.

{{< admonition type="note" >}}
If your alert rules frequently enter an error state, investigate the root cause (network stability, Prometheus resource limits, query timeout settings) rather than relying solely on this setting to suppress notifications.
{{< /admonition >}}

Common transient errors that trigger this behavior include:

- `sse.dependencyError` or `sse.dataQueryError` in alert state history
- "context deadline exceeded" or "i/o timeout" messages
- HTTP 502 or 500 responses from the Prometheus server

For more details on troubleshooting these errors, refer to [Troubleshoot Prometheus data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/troubleshooting/#transient-alert-errors-triggering-false-alarms).

## Best practices

Follow these best practices when creating Prometheus alerts:

- **Configure error state handling:** Set **Alert state if execution error or timeout** to **Keep Last State** to prevent transient backend errors from triggering false alarms.
- **Use `$__rate_interval`:** When using `rate()` or `increase()` in alert queries, use `$__rate_interval` to ensure the range window is always large enough relative to the scrape interval. Grafana resolves this variable based on the evaluation interval and scrape interval configuration.
- **Add label filters:** Include specific label matchers to focus on relevant data and improve query performance.
- **Set realistic pending periods:** Use the pending period to avoid alerting on brief spikes. For example, set a 5-minute pending period so the condition must persist before firing.
- **Test queries first:** Verify your query returns expected results in Explore before creating an alert.
- **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor and the severity.
- **Pre-aggregate with recording rules:** For complex or frequently evaluated expressions, create recording rules and alert on the pre-aggregated metric.
- **Use `absent()` for availability monitoring:** Detect when metrics stop being reported, which often indicates a crashed or unresponsive service.

If you encounter errors when creating or evaluating alert rules, refer to [Troubleshoot Prometheus data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/troubleshooting/#alerting-errors).

# Prometheus annotations

Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics. You can use Prometheus as a data source for annotations to display events such as deployments, alerts, or threshold crossings on your visualizations.

For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/).

## Before you begin

Before creating Prometheus annotations, ensure you have:

- A configured Prometheus data source in Grafana
- Metrics in Prometheus that represent the events you want to annotate
- Read access to the Prometheus instance

## Create an annotation query

To add a Prometheus annotation to your dashboard:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Annotation query**.
1. Enter a name for the annotation query.
1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox.
1. Select a color for the annotation event markers.
1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed.
1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed.
1. Click **Open query editor** to open the **Annotation Query** dialog box.
1. Select your **Prometheus** data source from the **Data source** drop-down list.
1. Enter a PromQL expression in the query field.
1. Set the **Min step** to control annotation density (a larger step means fewer annotations).
1. Configure the field mappings to control what appears in the annotation tooltip.
1. (Optional) Click **Test annotation query** to ensure that the query is working properly.
1. Click **Close** when you've completed the query setup.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## How Prometheus annotations work

Prometheus annotations work differently from SQL-based annotations. Instead of querying a table of events, you write a PromQL expression that returns time-series data. Grafana converts the query results into annotation events using these rules:

- Grafana executes the PromQL query as a range query over the dashboard's time window.
- **Every data point returned creates an annotation.** There is no automatic filtering of zero values. If you only want annotations at specific moments, your PromQL expression must filter the results (for example, using `> 0` or the `ALERTS` metric).
- Grafana uses the field mapping configuration to determine what text, title, and tags to display for each annotation.
- If the query returns multiple time series, each series produces its own set of annotations.

{{< admonition type="note" >}}
Because every returned data point creates an annotation, queries that return continuous data (like `node_cpu_seconds_total`) produce an annotation at every step interval, flooding your dashboard. Always use expressions that return data only at the moments you want annotated.
{{< /admonition >}}

## Field mappings

After entering your PromQL expression, use the field mapping drop-downs to control how query results are displayed as annotations. Grafana shows mapping options for:

| Field       | Description                                                                                          | Default behavior                                                                  |
| ----------- | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| **Time**    | The timestamp for the annotation.                                                                    | Uses the first time-type field (always present).                                  |
| **TimeEnd** | An end timestamp for range annotations, which display as a shaded region instead of a vertical line. | Not set (produces point annotations).                                             |
| **Title**   | Short label displayed on the annotation marker.                                                      | Not set.                                                                          |
| **Text**    | The annotation description displayed when you hover over it.                                         | Uses the first string-type field, or the metric/label display name if configured. |
| **Tags**    | Comma-separated tags for the annotation. Helps categorize and filter annotations.                    | Not set.                                                                          |

To configure field mappings, select the appropriate field name from each drop-down, or enter a fixed text value.

## Example annotation queries

The following examples show common annotation patterns with Prometheus.

### Alert-based annotations using `ALERTS`

The most common and reliable way to create Prometheus annotations. Prometheus automatically generates an `ALERTS` metric for all configured alerting rules.

```promql
ALERTS{alertstate="firing"}
```

This creates an annotation at every step interval where an alert is firing. The `ALERTS` metric includes labels such as:

- `alertname`: The name of the alerting rule
- `alertstate`: Either `firing` or `pending`
- Any labels defined on the alerting rule

To limit to specific alerts or severity levels:

```promql
ALERTS{alertname="HighCPUUsage", severity="critical"}
```

Configure the field mappings:

- **Text:** `alertname` (displays the alert name on hover)
- **Tags:** `severity` (allows filtering by severity)

### Service restart annotations

Display annotations when a process restarts. The `changes()` function detects when a value changes, and `> 0` ensures annotations only appear at the moment of change:

```promql
changes(process_start_time_seconds{job="myservice"}[5m]) > 0
```

### Deployment annotations

If you track deployments by pushing a timestamp metric via `Pushgateway` or a recording rule:

```promql
changes(deployment_timestamp_seconds{environment="production"}[10m]) > 0
```

Configure the field mappings:

- **Text:** `environment` (shows which environment was deployed)
- **Tags:** `environment`

### Threshold crossing annotations

Display annotations when available memory drops below 10%:

```promql
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1
```

{{< admonition type="note" >}}
Comparison operators in PromQL act as filters. They only return data points where the condition is true. This means the expression above only returns data when memory is below 10%, and annotations only appear at those times. There's no need to add an outer `> 0` wrapper.
{{< /admonition >}}

### Scaling event annotations

Display annotations when the number of running pods changes:

```promql
changes(kube_deployment_status_replicas{deployment="my-app"}[5m]) > 0
```

### Error spike annotations

Display annotations when the error rate exceeds a threshold:

```promql
sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job) > 0.05
```

This creates annotations when the error rate exceeds 5%.

### Version or build change annotations

If you expose build metadata using a metric like `build_info` (common in Go services), annotate when the version changes:

```promql
changes(build_info{job="myservice"}[10m]) > 0
```

Configure the field mappings:

- **Text:** `version` (shows the new version in the annotation tooltip)
- **Tags:** `job`

If your build info metric uses a `version` label (for example, `build_info{version="1.2.3"}`), the annotation tooltip displays the version that was deployed.

## Control annotation density

The **Min step** setting controls how many data points the query returns, which directly affects how many annotations appear. A larger step means fewer annotations:

- **Min step `1m`:** Up to one annotation per minute (good for short time ranges).
- **Min step `5m`:** Up to one annotation per 5 minutes (good for day-range dashboards).
- **Min step `1h`:** Up to one annotation per hour (good for week-range dashboards).

If your dashboard shows too many annotation markers, increase the Min step or add more specific filters to your query.

## Use template variables in annotations

You can use template variables in your annotation queries to filter annotations based on dashboard variable selections:

```promql
ALERTS{alertstate="firing", instance=~"$instance"}
```

```promql
changes(process_start_time_seconds{job="$job"}[5m]) > 0
```

Template variables in annotations are resolved at query time using the current dashboard variable values.

## Range annotations

Range annotations display as shaded regions rather than vertical lines, representing events with duration. To create range annotations, the query result must include a **TimeEnd** field mapping.

Because PromQL doesn't natively return start/end time pairs, range annotations with Prometheus are limited to scenarios where you have separate metrics for start and end times (for example, via recording rules or custom metrics pushed to `Pushgateway`). For most use cases, point annotations (using the examples in this page) are the practical approach.

If your annotations aren't appearing or you encounter errors, refer to [Troubleshoot Prometheus data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/troubleshooting/#annotation-errors).

# Migrate from Prometheus SigV4 to Amazon Managed Service for Prometheus

{{< admonition type="warning" >}}
Using SigV4 authentication with the core Prometheus data source for Amazon Managed Service for Prometheus is **deprecated**. In Grafana 13, the migration to the dedicated [Amazon Managed Service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/) is automatic. Existing data sources using SigV4 are migrated on startup.
{{< /admonition >}}

For background on this change, refer to [Prometheus data source update: Redefining our big tent philosophy](https://grafana.com/blog/2025/06/16/prometheus-data-source-update-redefining-our-big-tent-philosophy/).

## What changed in Grafana 13

In Grafana 13, the `prometheusTypeMigration` feature toggle is enabled by default and deprecated. This means:

- Prometheus data sources configured with SigV4 authentication are **automatically migrated** to the dedicated Amazon Managed Service for Prometheus plugin on Grafana startup.
- You no longer need to manually enable the feature toggle.
- Grafana Cloud users are migrated automatically with no action required.
- Dashboards, alerts, and queries continue to work after migration without changes.

## Check migration status

To determine if your Prometheus data sources have been migrated:

1. Navigate to **Connections** > **Data sources**.
1. Select your Prometheus data source.
1. Look for a migration banner at the top of the configuration page.

The banner displays one of the following messages:

- **"Migration Notice":** The data source has been migrated to the Amazon Managed Service for Prometheus plugin.
- **"Deprecation Notice":** The data source hasn't been migrated yet.
- **No banner:** No migration is needed (the data source doesn't use SigV4).

## Configure the Amazon Managed Service for Prometheus data source

After migration (or for new setups), configure the dedicated plugin:

1. Navigate to **Connections** > **Data sources**.
1. Select your Amazon Managed Service for Prometheus data source.
1. In the **Auth** section, configure SigV4 authentication:

| Setting                     | Description                                    | Example                                                         |
| --------------------------- | ---------------------------------------------- | --------------------------------------------------------------- |
| **Authentication Provider** | Choose your auth method                        | `AWS SDK Default`, `Access & secret key`, or `Credentials file` |
| **Default Region**          | AWS region for your workspace                  | `us-west-2`                                                     |
| **Access Key ID**           | Your AWS access key (if using access key auth) | `AKIA...`                                                       |
| **Secret Access Key**       | Your AWS secret key (if using access key auth) | `wJalrXUtn...`                                                  |
| **Assume Role ARN**         | IAM role ARN (optional)                        | `arn:aws:iam::123456789:role/GrafanaRole`                       |

4. Set the **HTTP URL** to your Amazon Managed Service for Prometheus workspace endpoint:
   `https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-12345678-1234-1234-1234-123456789012/`

5. Click **Save & test** to verify the connection.

## Provision the data source

```yaml
apiVersion: 1
datasources:
  - name: Amazon Managed Prometheus
    type: grafana-amazonprometheus-datasource
    url: https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-12345678-1234-1234-1234-123456789012/
    jsonData:
      httpMethod: POST
      sigV4Auth: true
      sigV4AuthType: keys
      sigV4Region: us-east-2
    secureJsonData:
      sigV4AccessKey: <ACCESS_KEY>
      sigV4SecretKey: <SECRET_KEY>
```

Replace `<ACCESS_KEY>` and `<SECRET_KEY>` with your AWS credentials.

## Troubleshoot migration issues

The following sections cover common issues you may encounter during or after the migration and how to resolve them.

### Amazon Managed Service for Prometheus plugin not installed

**Symptom:** Migration doesn't occur or the data source type is missing.

**Solution:**

1. Navigate to **Connections** > **Add new connection** and search for "Amazon Managed Service for Prometheus".
1. Install the plugin if it isn't already installed.
1. For air-gapped environments, download the plugin from [the Grafana plugin catalog](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/) and install it manually.

### "401 Unauthorized" after migration

**Symptom:** The migrated data source returns authentication errors.

**Solution:**

1. **Self-managed Grafana:** Verify that `grafana-amazonprometheus-datasource` is included in `forward_settings_to_plugins` under the `[aws]` heading in your `.ini` configuration file.
1. **Grafana Cloud:** Contact [Grafana Support](https://grafana.com/profile/org#support).

### Rollback the migration

If you need to revert migrated data sources back to the core Prometheus type:

1. Set `prometheusTypeMigration` to `false` in your Grafana configuration feature toggles. For more information, refer to [Manage feature toggles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/#manage-feature-toggles).
1. Restart Grafana.
1. Obtain a bearer token with `read` and `write` permissions for the data source API. For more information, refer to [Data source API](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/data_source/).
1. Run the following rollback script, providing your Grafana URL and bearer token:

```sh
#!/bin/bash

GRAFANA_URL=""
BEARER_TOKEN=""
LOG_FILE="grafana_migration_rollback_$(date +%Y%m%d_%H%M%S).log"

log_message() {
    local message="$1"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$timestamp] $message" | tee -a "$LOG_FILE"
}

update_data_source() {
    local uid="$1"
    local data="$2"

    response=$(curl -s -w "\n%{http_code}" -X PUT \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $BEARER_TOKEN" \
        -d "$data" \
        "$GRAFANA_URL/api/datasources/uid/$uid")

    http_code=$(echo "$response" | tail -n1)
    response_body=$(echo "$response" | sed '$d')

    if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then
        log_message "$uid reverted successfully"
    else
        log_message "$uid error: HTTP $http_code - $response_body"
    fi
}

if ! command -v jq &> /dev/null; then
    echo "Error: jq is required but not installed."
    exit 1
fi

if [[ -z "$GRAFANA_URL" || -z "$BEARER_TOKEN" ]]; then
    echo "Error: Set GRAFANA_URL and BEARER_TOKEN variables at the top of the script."
    exit 1
fi

log_message "Starting AMP to Prometheus rollback"

response=$(curl -s -w "\n%{http_code}" -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $BEARER_TOKEN" \
    "$GRAFANA_URL/api/datasources/")

http_code=$(echo "$response" | tail -n1)
response_body=$(echo "$response" | sed '$d')

if [[ "$http_code" -lt 200 || "$http_code" -ge 300 ]]; then
    log_message "Error fetching data sources: HTTP $http_code"
    exit 1
fi

total=$(echo "$response_body" | jq '[.[] | select(.jsonData["prometheus-type-migration"] == true and .type == "grafana-amazonprometheus-datasource")] | length')
log_message "Found $total data sources to revert"

echo "$response_body" | jq -c '.[] | select(.jsonData["prometheus-type-migration"] == true and .type == "grafana-amazonprometheus-datasource")' | while read -r data; do
    uid=$(echo "$data" | jq -r '.uid')
    read_only=$(echo "$data" | jq -r '.readOnly // false')

    if [[ "$read_only" == "true" ]]; then
        log_message "$uid is readOnly; edit the type to 'prometheus' in the provisioning file instead."
        continue
    fi

    updated_data=$(echo "$data" | jq '.type = "prometheus" | .jsonData["prometheus-type-migration"] = false')
    update_data_source "$uid" "$updated_data"
done

log_message "Rollback complete. Log: $LOG_FILE"
```

{{< admonition type="note" >}}
Provisioned data sources (`readOnly`) can't be reverted via the API. Update the `type` field to `prometheus` in your provisioning YAML file instead.
{{< /admonition >}}

If you continue to experience issues, check the Grafana server logs for detailed error messages and contact [Grafana Support](https://grafana.com/help/).

# Migrate from Prometheus Azure AD to Azure Monitor Managed Service for Prometheus

{{< admonition type="warning" >}}
Using Azure AD authentication with the core Prometheus data source for Azure Monitor Managed Service for Prometheus is **deprecated**. In Grafana 13, the migration to the dedicated [Azure Monitor Managed Service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-azureprometheus-datasource/) is automatic. Existing data sources using Azure AD authentication are migrated on startup.
{{< /admonition >}}

For background on this change, refer to [Prometheus data source update: Redefining our big tent philosophy](https://grafana.com/blog/2025/06/16/prometheus-data-source-update-redefining-our-big-tent-philosophy/).

## What changed in Grafana 13

In Grafana 13, the `prometheusTypeMigration` feature toggle is enabled by default and deprecated. This means:

- Prometheus data sources configured with Azure AD authentication are **automatically migrated** to the dedicated Azure Monitor Managed Service for Prometheus plugin on Grafana startup.
- You no longer need to manually enable the feature toggle.
- Grafana Cloud users are migrated automatically with no action required.
- Dashboards, alerts, and queries continue to work after migration without changes.

## Check migration status

To determine if your Prometheus data sources have been migrated:

1. Navigate to **Connections** > **Data sources**.
1. Select your Prometheus data source.
1. Look for a migration banner at the top of the configuration page.

The banner displays one of the following messages:

- **"Migration Notice":** The data source has been migrated to the Azure Monitor Managed Service for Prometheus plugin.
- **"Deprecation Notice":** The data source hasn't been migrated yet.
- **No banner:** No migration is needed (the data source doesn't use Azure AD authentication).

## Configure the Azure Monitor Managed Service for Prometheus data source

After migration (or for new setups), configure the dedicated plugin:

1. Navigate to **Connections** > **Data sources**.
1. Select your Azure Monitor Managed Service for Prometheus data source.
1. In the **Authentication** section, select your authentication method:

| Method               | Use case                            | Additional configuration required                   |
| -------------------- | ----------------------------------- | --------------------------------------------------- |
| **Managed Identity** | Azure-hosted Grafana instances      | None (system-assigned) or Client ID (user-assigned) |
| **App Registration** | Service principal authentication    | Directory ID, Application ID, Client secret         |
| **Current User**     | Current user's Entra ID credentials | None                                                |

For Managed Identity authentication:

- No additional configuration is required if using a system-assigned identity.
- For a user-assigned identity, provide the **Client ID**.

For App Registration authentication:

| Setting                     | Description                     | Example                                |
| --------------------------- | ------------------------------- | -------------------------------------- |
| **Directory (tenant) ID**   | Your Entra ID tenant ID         | `12345678-1234-1234-1234-123456789012` |
| **Application (client) ID** | Your app registration client ID | `87654321-4321-4321-4321-210987654321` |
| **Client secret**           | Your app registration secret    | `your-client-secret`                   |

To learn more about Entra ID authentication for Grafana, refer to [Configure Entra ID OAuth authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/azuread/#configure-azure-adentra-id-oauth-authentication).

4. Set the **Prometheus server URL** to your Azure Monitor workspace endpoint:

   ```
   https://your-workspace.eastus2.prometheus.monitor.azure.com
   ```

5. Click **Save & test** to verify the connection.

## Provision the data source

```yaml
apiVersion: 1
datasources:
  - name: Azure Monitor Prometheus
    type: grafana-azureprometheus-datasource
    url: https://your-workspace.eastus2.prometheus.monitor.azure.com
    jsonData:
      azureCredentials:
        authType: clientsecret
        azureCloud: AzureCloud
        clientId: <CLIENT_ID>
        tenantId: <TENANT_ID>
      httpMethod: POST
    secureJsonData:
      azureClientSecret: <CLIENT_SECRET>
```

Replace `<CLIENT_ID>`, `<TENANT_ID>`, and `<CLIENT_SECRET>` with your Azure credentials.

## Troubleshoot migration issues

The following sections cover common issues you may encounter during or after the migration and how to resolve them.

### Azure Monitor Managed Service for Prometheus plugin not installed

**Symptom:** Migration doesn't occur or the data source type is missing.

**Solution:**

1. Navigate to **Connections** > **Add new connection** and search for "Azure Monitor Managed Service for Prometheus".
1. Install the plugin if it isn't already installed.
1. For air-gapped environments, download the plugin from [the Grafana plugin catalog](https://grafana.com/grafana/plugins/grafana-azureprometheus-datasource/) and install it manually.

### "401 Unauthorized" after migration

**Symptoms:**

- The migrated data source returns `401 Unauthorized` errors on **Save & test** or when running queries.
- The data source authenticates with **Managed Identity**, **Workload Identity**, or **Current User**.

**Cause:** The Azure Monitor Managed Service for Prometheus plugin doesn't receive the Azure authentication settings from the Grafana server. Either the `[azure]` section of your `.ini` configuration file is missing the options required by your authentication method, or `forward_settings_to_plugins` doesn't include the `grafana-azureprometheus-datasource` plugin.

**Solution for self-managed Grafana:**

1. Set the options required by your authentication method in the `[azure]` section of your `.ini` configuration file. For a full description of each option, refer to the [`[azure]` configuration reference](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#azure).
   - **Managed Identity:** Set `managed_identity_enabled = true`. For a user-assigned identity, also set `managed_identity_client_id` to the identity's client ID. Leave it empty for a system-assigned identity.
   - **Workload Identity:** Set `workload_identity_enabled = true`.
   - **Current User:** Set `user_identity_enabled = true`.

1. If you've customized `forward_settings_to_plugins`, make sure `grafana-azureprometheus-datasource` is listed. By default, `forward_settings_to_plugins` includes all Grafana Labs Azure plugins, so the Prometheus plugin already receives these settings. For example:

   ```ini
   [azure]
   managed_identity_enabled = true
   forward_settings_to_plugins = grafana-azureprometheus-datasource
   ```

1. Verify that the identity has the required Azure role, such as **Monitoring Data Reader**, on the target Azure Monitor workspace.

1. Restart Grafana after you change the server configuration.

**Solution for Grafana Cloud:** Contact [Grafana Support](https://grafana.com/profile/org#support).

### Rollback the migration

If you need to revert migrated data sources back to the core Prometheus type:

1. Set `prometheusTypeMigration` to `false` in your Grafana configuration feature toggles. For more information, refer to [Manage feature toggles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/#manage-feature-toggles).
1. Restart Grafana.
1. Obtain a bearer token with `read` and `write` permissions for the data source API. For more information, refer to [Data source API](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/data_source/).
1. Run the following rollback script, providing your Grafana URL and bearer token:

```sh
#!/bin/bash

GRAFANA_URL=""
BEARER_TOKEN=""
LOG_FILE="grafana_azure_migration_rollback_$(date +%Y%m%d_%H%M%S).log"

log_message() {
    local message="$1"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$timestamp] $message" | tee -a "$LOG_FILE"
}

update_data_source() {
    local uid="$1"
    local data="$2"

    response=$(curl -s -w "\n%{http_code}" -X PUT \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $BEARER_TOKEN" \
        -d "$data" \
        "$GRAFANA_URL/api/datasources/uid/$uid")

    http_code=$(echo "$response" | tail -n1)
    response_body=$(echo "$response" | sed '$d')

    if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then
        log_message "$uid reverted successfully"
    else
        log_message "$uid error: HTTP $http_code - $response_body"
    fi
}

if ! command -v jq &> /dev/null; then
    echo "Error: jq is required but not installed."
    exit 1
fi

if [[ -z "$GRAFANA_URL" || -z "$BEARER_TOKEN" ]]; then
    echo "Error: Set GRAFANA_URL and BEARER_TOKEN variables at the top of the script."
    exit 1
fi

log_message "Starting Azure Prometheus to core Prometheus rollback"

response=$(curl -s -w "\n%{http_code}" -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $BEARER_TOKEN" \
    "$GRAFANA_URL/api/datasources/")

http_code=$(echo "$response" | tail -n1)
response_body=$(echo "$response" | sed '$d')

if [[ "$http_code" -lt 200 || "$http_code" -ge 300 ]]; then
    log_message "Error fetching data sources: HTTP $http_code"
    exit 1
fi

total=$(echo "$response_body" | jq '[.[] | select(.jsonData["prometheus-type-migration"] == true and .type == "grafana-azureprometheus-datasource")] | length')
log_message "Found $total data sources to revert"

echo "$response_body" | jq -c '.[] | select(.jsonData["prometheus-type-migration"] == true and .type == "grafana-azureprometheus-datasource")' | while read -r data; do
    uid=$(echo "$data" | jq -r '.uid')
    read_only=$(echo "$data" | jq -r '.readOnly // false')

    if [[ "$read_only" == "true" ]]; then
        log_message "$uid is readOnly; edit the type to 'prometheus' in the provisioning file instead."
        continue
    fi

    updated_data=$(echo "$data" | jq '.type = "prometheus" | .jsonData["prometheus-type-migration"] = false')
    update_data_source "$uid" "$updated_data"
done

log_message "Rollback complete. Log: $LOG_FILE"
```

{{< admonition type="note" >}}
Provisioned data sources (`readOnly`) can't be reverted via the API. Update the `type` field to `prometheus` in your provisioning YAML file instead.
{{< /admonition >}}

If you continue to experience issues, check the Grafana server logs for detailed error messages and contact [Grafana Support](https://grafana.com/help/).

# Troubleshoot Prometheus data source issues

This document provides troubleshooting information for common errors you may encounter when using the Prometheus data source in Grafana.

## Connection errors

The following errors occur when Grafana cannot establish or maintain a connection to Prometheus.

### Failed to connect to Prometheus

**Error message:** "There was an error returned querying the Prometheus API"

**Cause:** Grafana cannot establish a network connection to the Prometheus server.

**Solution:**

1. Verify that the Prometheus server URL is correct in the data source configuration.
1. Check that Prometheus is running and accessible from the Grafana server.
1. Ensure the URL includes the protocol (`http://` or `https://`).
1. Verify the port is correct (the Prometheus default port is `9090`).
1. Ensure there are no firewall rules blocking the connection.
1. If Grafana and Prometheus are running in separate containers, use the container IP address or hostname instead of `localhost`.
1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your Prometheus instance is not publicly accessible.

### Request timed out

**Error message:** "context deadline exceeded" or "request timeout"

**Cause:** The connection to Prometheus timed out before receiving a response.

**Solution:**

1. Check the network latency between Grafana and Prometheus.
1. Verify that Prometheus is not overloaded or experiencing performance issues.
1. Increase the **Query timeout** setting in the data source configuration under **Interval behavior**.
1. Check the [Grafana server timeout configuration](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana#timeout) for server-level timeout settings.
1. Reduce the time range or complexity of your query.
1. Check if any network devices (load balancers, proxies) are timing out the connection.

### Failed to parse data source URL

**Error message:** "Failed to parse data source URL"

**Cause:** The URL entered in the data source configuration is not valid.

**Solution:**

1. Verify the URL format is correct (for example, `http://localhost:9090` or `https://prometheus.example.com:9090`).
1. Ensure the URL includes the protocol (`http://` or `https://`).
1. Remove any trailing slashes or invalid characters from the URL.

### Data doesn't appear in Metrics Drilldown or Explore

**Symptom:** You've successfully tested the data source connection, but no metric data appears in Explore or Metrics Drilldown.

**Cause:** The wrong data source is selected, or the data source name doesn't match expectations.

**Solution:**

1. Verify you've selected the correct data source from the data source drop-down menu.
1. When using `remote_write` to send metrics to Grafana Cloud, the data source name follows the convention `grafanacloud-<stackname>-prom`.
1. Check that metrics are being actively scraped by querying `up` in the Explore view.
1. If using Grafana Cloud, verify the `remote_write` endpoint URL and credentials are correct.

### PDC connectivity errors

**Error messages:** `host unreachable`, `EOF`, `network unreachable`, `connection reset by peer`, `dial tcp: lookup ... no such host`

**Symptom:** Prometheus queries fail intermittently or consistently when using Private data source connect (PDC) to reach a Prometheus instance behind a private network. The data source test may pass occasionally but queries fail under load.

**Cause:** PDC tunnels traffic through an SSH connection between Grafana Cloud and your PDC agent. Connectivity failures are most commonly caused by DNS resolution issues, network configuration on the customer side, or the PDC agent's default connection limits being too low for the query volume.

**Solutions:**

1. **Verify DNS resolution from the PDC agent host.** The PDC agent must be able to resolve the Prometheus hostname from the machine it runs on. Run `nslookup` or `dig` for the Prometheus URL from the agent host to confirm.
1. **Check network connectivity from the agent.** Ensure the PDC agent can reach the Prometheus endpoint directly (for example, `curl http://prometheus-host:9090/-/healthy` from the agent machine).
1. **Increase parallel SSH connections.** The PDC agent defaults to 1 parallel SSH connection, which can bottleneck under load from multiple alert evaluations or dashboard queries. Increase this by setting the `--ssh-connections` flag (or `PDC_SSH_CONNECTIONS` environment variable) to a higher value (for example, 4 or 8):

   ```sh
   pdc-agent --ssh-connections=4
   ```

1. **Check firewall rules.** Ensure the PDC agent's outbound SSH connection to Grafana Cloud isn't being interrupted by firewall rules, NAT gateways, or idle connection timeouts.
1. **Verify the PDC agent is running and healthy.** Check agent logs for connection errors or restarts. The agent must maintain a persistent connection to Grafana Cloud.
1. **Check for idle timeout issues.** If the connection drops after periods of inactivity, configure TCP keepalive on the host or add a `keepalive` setting to the PDC agent configuration.

{{< admonition type="note" >}}
PDC connectivity issues are almost always caused by networking on the customer side (DNS, firewall rules, routing), not by Grafana Cloud. The data source test passing doesn't guarantee sustained connectivity under load. It only verifies a single query succeeds.
{{< /admonition >}}

For general PDC setup and configuration, refer to [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) and [Configure PDC](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/).

### Certificate verification failed

**Error message:** "x509: certificate signed by unknown authority" or "certificate verify failed"

**Cause:** Grafana cannot verify the TLS certificate presented by Prometheus.

**Solution:**

1. If using a self-signed certificate, enable **Add self-signed certificate** in the TLS settings and add your CA certificate.
1. Verify the certificate chain is complete and valid.
1. Ensure the certificate has not expired.
1. As a temporary workaround for testing, enable **Skip TLS verify** (not recommended for production).

### TLS handshake error

**Error message:** "TLS: handshake failure" or "connection reset"

**Cause:** The TLS handshake between Grafana and Prometheus failed.

**Solution:**

1. Verify that Prometheus is configured to use TLS.
1. Check that the TLS version and cipher suites are compatible.
1. If using client certificates, ensure they are correctly configured in the **TLS client authentication** section.
1. Verify the server name matches the certificate's Common Name or Subject Alternative Name.

## Authentication errors

The following errors occur when there are issues with authentication credentials or permissions.

### Unauthorized (401)

**Error message:** "401 Unauthorized" or "Authorization failed"

**Cause:** The authentication credentials are invalid or missing.

**Solution:**

1. Verify that the username and password are correct if using basic authentication.
1. Check that the authentication method selected matches your Prometheus configuration.
1. If using a reverse proxy with authentication, verify the credentials are correct.
1. For AWS SigV4 authentication, verify the IAM credentials and permissions. Alternatively, consider using the [Amazon Managed Service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/) for simplified AWS authentication.

### OAuth token expiration errors (GCP and Azure)

**Error messages:** "ACCESS_TOKEN_EXPIRED", "401 Unauthorized" in alerting but not in Explore

**Symptom:** Queries in Explore and dashboards work correctly, but alert rule evaluations fail intermittently with 401 errors. This is most common with Google Managed Prometheus (GMP) and Azure-managed Prometheus endpoints using OAuth/OIDC authentication.

**Cause:** The Grafana alerting backend and the interactive query path (Explore, dashboards) handle credential refreshes differently. The alerting evaluator can use a cached OAuth token beyond its expiry window due to a token staleness check issue in the Prometheus data source. This causes alerting to fail with expired credentials while interactive queries succeed because they trigger a fresh token exchange.

**Solutions:**

For Google Managed Prometheus (GMP):

1. Use the **GCP datasource-syncer** pattern: run a sidecar process that refreshes OAuth tokens and updates the Grafana data source credentials on a schedule shorter than the token lifetime (typically every 45 minutes for a 60-minute token). This ensures the stored token is always valid when the alerting backend uses it.
1. Alternatively, use the [Google Cloud Monitor data source](https://grafana.com/grafana/plugins/stackdriver/) with its built-in GCP credential management rather than pointing the Prometheus data source at a GMP endpoint.
1. If running on GKE with Workload Identity, ensure the Kubernetes service account token refresh is functioning and the projected token volume is mounted correctly.

For Azure Managed Prometheus:

1. Verify the Azure AD app registration's client secret hasn't expired.
1. If using Managed Identity, ensure the Grafana instance's identity has the **Monitoring Data Reader** role on the Azure Monitor workspace.
1. Check that **Forward OAuth identity** is not enabled alongside Azure AD authentication. Both use the same HTTP authorization headers and conflict with each other.

General steps:

1. Check the Grafana server logs for token refresh errors around the time alerts fail.
1. Verify the data source test (**Save & test**) passes. This confirms current credentials are valid but doesn't guarantee the alerting backend has a fresh token.
1. If the issue persists, set **Alert state if execution error or timeout** to **Keep Last State** to prevent false alarms while investigating. Refer to [Configure alert state for execution errors](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/alerting/#configure-alert-state-for-execution-errors).

{{< admonition type="note" >}}
This token caching behavior is a known issue that has received code fixes in recent Grafana releases. If you're experiencing this on an older Grafana version, upgrading may resolve it. Check the [Grafana changelog](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/whatsnew/) for relevant fixes.
{{< /admonition >}}

### LBAC not restricting data on non-Mimir backends

**Symptom:** You've configured Team LBAC rules, but users can still see all metrics regardless of their team assignments.

**Cause:** Label-Based Access Control (LBAC) for the Prometheus data source only works when the backend is **Grafana Cloud Metrics (Mimir)** or **Grafana Enterprise Metrics (GEM)**. It doesn't work with Google Managed Prometheus, self-managed Prometheus, Thanos, or other Prometheus-compatible endpoints. The LBAC enforcement relies on Mimir-specific HTTP headers (`X-Scope-OrgID` and team-scoped label matchers) that other backends ignore.

**Solution:**

1. Verify your backend is Grafana Cloud Metrics or GEM. If you're using a different Prometheus-compatible backend, LBAC isn't supported.
1. For Google Managed Prometheus or other external endpoints, use [data source permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/#data-source-permissions) to control which teams can access the data source entirely, rather than per-label access control.
1. If you need per-label restrictions on a non-Mimir backend, consider proxying through a Mimir instance or using a separate data source per team with different credentials scoped to the appropriate data.

### Azure AD or SigV4 authentication options not available

**Symptom:** The Azure AD or SigV4 authentication options don't appear in the authentication drop-down when configuring the Prometheus data source.

**Cause:** These authentication methods require server-side feature flags that aren't enabled by default, particularly on Grafana Cloud.

**Solution:**

1. **Grafana Cloud:** Contact [Grafana Support](https://grafana.com/profile/org#support) to request the feature flag be enabled for your stack. This isn't a self-service setting.
1. **Self-managed Grafana:** Add the required setting to your Grafana configuration file:
   - For Azure AD: set `azure_auth_enabled = true` under `[auth]`
   - For SigV4: set `sigv4_auth_enabled = true` under `[auth]`
1. **For Amazon Managed Prometheus:** Consider using the dedicated [Amazon Managed Service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/) instead, which handles AWS authentication natively without needing the SigV4 flag.

### Forbidden (403)

**Error message:** "403 Forbidden" or "Access denied"

**Cause:** The authenticated user does not have permission to access the requested resource.

**Solution:**

1. Verify the user has read access to the Prometheus API.
1. Check Prometheus security settings and access control configuration.
1. If using a reverse proxy, verify the proxy is not blocking the request.
1. For AWS Managed Prometheus, verify the IAM policy grants the required permissions. Alternatively, consider using the [Amazon Managed Service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/) for simplified AWS authentication.

## Query errors

The following errors occur when there are issues with PromQL syntax or query execution.

### Query syntax error

**Error message:** `parse error: unexpected character` or `bad_data: 1:X: parse error`

**Cause:** The PromQL query contains invalid syntax.

**Alternative cause:** A proxy between Grafana and Prometheus requires authentication. When proxy authentication fails, the proxy redirects the request to an HTML authentication page. Grafana cannot parse the HTML response, which results in a parse error. This appears to be a query issue but is actually a proxy authentication issue.

**Solution:**

1. Check your query syntax for typos or invalid characters.
1. Verify that metric names and label names are valid identifiers.
1. Ensure string values in label matchers are enclosed in quotes.
1. Use the Prometheus expression browser to test your query directly.
1. Refer to the [Prometheus querying documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) for syntax guidance.
1. If you have a proxy between Grafana and Prometheus, verify that proxy authentication is correctly configured. Check your proxy logs for authentication failures or redirects.

### Query returns no data for a metric

**Symptom:** The query returns no data and the visualization is empty.

**Cause:** The specified metric does not exist in Prometheus, or there is no data for the selected time range.

**Solution:**

1. Verify the metric name is spelled correctly.
1. Check that the metric is being scraped by Prometheus.
1. Use the Prometheus API to browse available metrics at `/api/v1/label/__name__/values`.
1. Use the [target metadata API](https://prometheus.io/docs/prometheus/latest/querying/api#querying-target-metadata) to verify which metrics a target exposes.
1. Verify the time range includes data for the metric.

### Query timeout limit exceeded

**Error message:** "query timed out in expression evaluation" or "query processing would load too many samples"

**Cause:** The query took longer than the configured timeout limit or would return too many samples.

**Solution:**

1. Reduce the time range of your query.
1. Add more specific label filters to limit the data scanned.
1. Increase the **Query timeout** setting in the data source configuration.
1. Use aggregation functions like `sum()`, `avg()`, or `rate()` to reduce the number of time series.
1. Increase the `query.timeout` or `query.max-samples` settings in Prometheus if you have admin access.

### Too many time series

**Error message:** `exceeded maximum resolution of 11,000 points per timeseries` or `maximum number of series limit exceeded`

**Cause:** The query is returning more time series or data points than the configured limits allow.

**Solution:**

1. Reduce the time range of your query.
1. Add label filters to limit the number of time series returned.
1. Increase the **Min interval** or **Resolution** in the query options to reduce the number of data points.
1. Use aggregation functions to combine time series.
1. Adjust the **Series limit** setting in the data source configuration under **Other settings**.

### Memory limit exceeded for high-cardinality queries

**Error messages:** "max-estimated-memory-consumption-per-query limit exceeded", "query requires too much memory", "max samples limit reached"

**Symptom:** Queries against high-cardinality metrics (thousands of unique label combinations) over long time ranges (days or weeks) fail with memory or sample limit errors. Short time ranges may work but expanding the range causes the failure.

**Cause:** Prometheus and Mimir enforce per-query memory and sample limits to protect the system from resource exhaustion. High-cardinality metrics (for example, metrics with a `pod`, `request_id`, or `user_id` label) multiplied by long time ranges produce result sets that exceed these limits.

**Solutions:**

1. **Reduce the query scope:**
   - Shorten the time range.
   - Add label matchers to select fewer series (for example, filter to a specific `namespace` or `job`).
   - Increase the **Min step** or use a larger step interval to reduce the number of data points per series.

1. **Use recording rules to pre-aggregate:**
   Create [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) that pre-compute the aggregation you need. For example, if you commonly query `sum(rate(http_requests_total[5m])) by (service)`, create a recording rule for it and query the pre-aggregated metric instead.

   ```yaml
   groups:
     - name: aggregations
       rules:
         - record: service:http_requests_total:rate5m
           expr: sum(rate(http_requests_total[5m])) by (service)
   ```

1. **Use Adaptive Metrics (Grafana Cloud):**
   If you're on Grafana Cloud, [Adaptive Metrics](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/reduce-costs/metrics-costs/control-metrics-usage-via-adaptive-metrics/) automatically identifies and aggregates high-cardinality metrics that aren't being queried at full resolution, reducing storage and query costs.

1. **Restructure dashboards for high-cardinality data:**
   - Use template variables to scope queries to a subset of labels rather than querying all series at once.
   - Split long-range overviews (weekly/monthly) into separate panels that use pre-aggregated recording rules.
   - Use the **Max data points** setting in the query options to cap the resolution for overview panels.

1. **Increase limits (self-managed only):**
   If you have admin access to Prometheus or Mimir, you can increase limits in the server configuration:
   - Prometheus: `--query.max-samples` flag
   - Mimir: `-querier.max-samples`, `-querier.max-estimated-memory-consumption-per-query`

   {{< admonition type="note" >}}
   Increasing limits allows larger queries to succeed but also increases the risk of resource exhaustion. Prefer reducing query scope or using recording rules over raising limits.
   {{< /admonition >}}

### Invalid function or aggregation

**Error message:** "unknown function" or "parse error: unexpected aggregation"

**Cause:** The query uses an invalid or unsupported PromQL function.

**Solution:**

1. Verify the function name is spelled correctly and is a valid PromQL function.
1. Check that you are using the correct syntax for the function.
1. Ensure your Prometheus version supports the function you are using.
1. Refer to the [PromQL functions documentation](https://prometheus.io/docs/prometheus/latest/querying/functions/) for available functions.

### `rate()` or `increase()` returning unexpected values

**Symptom:** `increase()` returns fractional values on integer counters, `rate()` shows an ever-increasing value instead of a steady per-second rate, or counter resets cause large spikes in visualizations.

**Possible causes and solutions:**

| Cause                                         | Solution                                                                                                                                              |
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `increase()` fractional values                | Expected behavior. Prometheus uses linear interpolation. Use `ceil()` or `floor()` if you need integers.                                              |
| `rate()` grows over time                      | Multiple instances write to the same series without unique labels. Ensure each target has unique `instance`/`pod` labels and aggregate with `sum by`. |
| Counter reset spikes after pod restarts       | Use `$__rate_interval` or a longer range vector to smooth spikes. Investigate frequent restarts as the root cause.                                    |
| Values differ between edit mode and dashboard | Panel width affects `$__interval` which affects `rate()` window calculations. Set a **Min step** on the query.                                        |

For detailed explanations of these behaviors, refer to [Expected PromQL behaviors](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/query-editor/#expected-promql-behaviors).

### Aggregation by labels with dots

**Symptom:** Queries that aggregate by label names containing dots (for example, `container.name`) return incorrect or incomplete results.

**Cause:** Prior to Grafana 13, there was a bug where labels with dots in their names were not handled correctly during aggregation operations like `sum by` or `avg by`.

**Solution:**

1. Upgrade to Grafana 13 or later, which correctly handles labels with dots in aggregation queries.
1. Verify your query uses the correct label name with dots (for example, `sum by (container.name) (metric_name)`).

## Configuration errors

The following errors occur when the data source is not configured correctly.

### Invalid Prometheus type

**Error message:** Unexpected behavior when querying metrics or labels

**Cause:** The **Prometheus type** setting does not match your actual Prometheus-compatible database.

**Solution:**

1. Open the data source configuration in Grafana.
1. Under **Performance**, select the correct **Prometheus type** (Prometheus, Cortex, Mimir, or Thanos).
1. Different database types support different APIs, so setting this incorrectly may cause unexpected behavior.

### Scrape interval mismatch

**Symptom:** Data appears sparse, or `rate()` queries return no data or incomplete results.

**Cause:** The **Scrape interval** setting in Grafana does not match the actual scrape interval in Prometheus. This especially affects `rate()` queries, which require at least two data points within the specified time window. For example, if your actual scrape interval is 5 minutes but Grafana uses the default (15 seconds for OSS, 1 minute for Grafana Cloud), a query like `rate(http_requests_total[1m])` returns no data because there are no data points within that 1-minute window.

**Solution:**

1. Check your Prometheus configuration file for the `scrape_interval` setting.
1. Update the **Scrape interval** in the Grafana data source configuration under **Interval behavior** to match.
1. Use `$__rate_interval` instead of hard-coded time windows in `rate()` queries. This variable automatically adjusts based on your scrape interval.
1. For more information, refer to [$\_\_rate_interval for Prometheus rate queries that just work](https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/).

<!-- vale Grafana.Spelling = NO -->

### `$__rate_interval` returns no data or incorrect values

<!-- vale Grafana.Spelling = YES -->

**Symptom:** Queries using `$__rate_interval` return no data, return different values in edit mode versus the dashboard, or produce unexpected gaps.

**Cause:** `$__rate_interval` is calculated as `max($__interval + scrape_interval, 4 * scrape_interval)`. If any input to this formula is incorrect, the resulting window is wrong: either too small (no data) or inconsistent across contexts.

**Common causes and solutions:**

| Cause                                                                                                                    | Solution                                                                                                                                  |
| ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Data source scrape interval left at default `15s` while actual Prometheus scrape interval is longer (for example, `60s`) | Set the **Scrape interval** under **Interval behavior** in the data source configuration to match your Prometheus `scrape_interval`.      |
| Query works in edit mode but shows gaps on the dashboard                                                                 | Panel size affects `$__interval`. Smaller panels produce larger intervals. Set a **Min step** on the query to enforce a consistent floor. |
| LBAC-enabled data source doesn't inherit scrape interval                                                                 | Set the **Min step** explicitly on each query panel rather than relying on data source inheritance.                                       |
| Using `$__rate_interval` in recording rules or alerting                                                                  | Use a fixed interval (for example, `[5m]`) instead of `$__rate_interval` in contexts without a panel/dashboard.                           |

**To debug the current value:**

1. Open the query inspector in a panel (click the panel title, then **Inspect** > **Query**).
1. Look at the expanded query sent to Prometheus. The actual interval value replacing `$__rate_interval` is visible in the request.
1. Compare this value against your actual scrape interval. If it's smaller than your scrape interval, you need to configure the data source scrape interval or set a Min step.

For detailed documentation on how `$__rate_interval` works and how to configure it, refer to [Use `$__rate_interval`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/template-variables/#use-__rate_interval).

## Performance issues

The following issues affect query speed and data freshness.

### Slow query performance

**Symptom:** Queries take a long time to execute, dashboards are slow to load, or the loading spinner persists.

**Cause:** Queries scan too much data, the Prometheus server is overloaded, or the network connection is slow.

**Solution:**

1. Reduce the time range of your query.
1. Add more specific label filters to limit the data scanned.
1. Increase the **Min step** in the query options to reduce the number of data points.
1. Check Prometheus server performance and resource utilization.
1. Enable **Disable metrics lookup** in the data source configuration for large Prometheus instances.
1. Enable **Incremental querying (beta)** to cache query results.
1. Use recording rules to pre-aggregate frequently queried data.
1. For high-cardinality metrics, refer to [Memory limit exceeded for high-cardinality queries](#memory-limit-exceeded-for-high-cardinality-queries).

### Data appears delayed or missing recent points

**Symptom:** The visualization doesn't show the most recent data, even after refreshing.

**Cause:** Scrape timing, clock drift, or dashboard refresh settings.

**Solution:**

1. Check the dashboard time range and refresh settings.
1. Verify the **Scrape interval** is configured correctly in the data source settings.
1. Ensure Prometheus has finished scraping the target (there's a delay between the scrape interval and data availability).
1. Check for clock synchronization issues between Grafana and Prometheus (use NTP).
1. For `rate()` and similar functions, the most recent partial scrape interval won't have enough data points. This is expected.

### Exemplars not showing

**Symptom:** Exemplar data doesn't appear on graphs even though you expect it.

**Cause:** Exemplars require specific configuration in both the data source and the query editor.

**Solution:**

1. Verify that exemplars are configured in the data source settings under **Exemplars** (at least one exemplar link must be defined).
1. Check that your Prometheus version supports exemplars (2.26+).
1. Ensure your instrumented application is actually sending exemplar data with metrics.
1. Verify the tracing data source is correctly configured for the exemplar link.
1. Enable the **Exemplars** toggle in the query editor for the specific query.
1. Exemplars only appear with **Range** query type, not **Instant**.

## Annotation errors

The following issues occur when using Prometheus as a data source for annotations.

### Annotations not appearing

**Symptom:** You've configured a Prometheus annotation query, but no annotations appear on your dashboard.

**Possible causes and solutions:**

| Cause                                                | Solution                                                                                                                                                                                                                        |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Query returns no data in the current time range      | Verify the query returns results in Explore for the dashboard's time range.                                                                                                                                                     |
| Query returns continuous data (too many annotations) | Every returned data point creates an annotation. If the query returns hundreds of points, annotations may render but are too dense to see. Increase the **Min step** or refine your query to only return data at event moments. |
| Wrong data source selected                           | Verify the correct Prometheus data source is selected in the annotation configuration.                                                                                                                                          |
| Annotation is disabled                               | Check that the annotation toggle is enabled (eye icon) in the dashboard's annotation settings.                                                                                                                                  |
| Time range mismatch                                  | Expand the dashboard time range to include the events you expect to see.                                                                                                                                                        |

{{< admonition type="note" >}}
Prometheus annotations create a marker for **every data point** returned by the query. There's no automatic filtering of zero values. If you only want annotations at specific moments, your PromQL expression must return data only at those times (for example, using `> 0`, `changes() > 0`, or the `ALERTS` metric).
{{< /admonition >}}

For more information on configuring annotations, refer to [Prometheus annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/annotations/).

## Alerting errors

The following issues occur when using Prometheus with Grafana Alerting.

### Transient alert errors triggering false alarms

**Error messages:** `sse.dependencyError`, `sse.dataQueryError`, "context deadline exceeded", "i/o timeout"

**Symptom:** Alert rules intermittently fire due to execution errors rather than genuine threshold breaches. On-call teams receive false positive notifications. Alert state history shows error states caused by transient backend issues (network blips, HTTP 502/500 responses, timeouts) rather than actual metric conditions being met.

**Cause:** By default, when an alert rule encounters an execution error or timeout, Grafana sets the alert state to **Alerting**, which fires the alert. Transient connectivity issues between Grafana and Prometheus (i/o timeouts, deadline exceeded, brief outages) trigger this behavior even though the underlying metric hasn't crossed its threshold.

**Solution:**

1. Open each affected alert rule for editing.
1. In the alert conditions section, change **Alert state if execution error or timeout** from **Alerting** to **Keep Last State**.
1. Save the rule.

This ensures the alert retains its previous state during transient errors and only fires when a successful evaluation confirms the threshold is breached.

**If errors are frequent**, also investigate:

1. Network stability between Grafana and Prometheus.
1. Prometheus resource utilization (CPU, memory, disk I/O).
1. The **Query timeout** setting in the data source configuration. Increase it if complex queries regularly exceed the limit.
1. Query complexity. Simplify queries or use recording rules to pre-compute expensive expressions.

For configuration details, refer to [Configure alert state for execution errors](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/alerting/#configure-alert-state-for-execution-errors).

### Alert rule fails to evaluate

**Symptom:** An alert rule using a Prometheus query shows evaluation errors or remains in a "No Data" state.

**Possible causes and solutions:**

| Cause                       | Solution                                                                                            |
| --------------------------- | --------------------------------------------------------------------------------------------------- |
| Template variables in query | Alert queries don't support template variables. Replace variables with hard-coded values.           |
| Query timeout               | Simplify the query or increase the evaluation timeout. Use recording rules for complex expressions. |
| Data source unreachable     | Verify the Prometheus data source connection is working (test it in the data source settings).      |
| No data in range            | Ensure the metric has recent data. Check that Prometheus is actively scraping the target.           |

### Data source-managed rules not visible

**Symptom:** Prometheus alerting rules don't appear in the Grafana Alerting UI.

**Solution:**

1. Verify that **Manage alerts via Alerting UI** is enabled in the data source configuration.
1. Check that Prometheus has alerting rules configured in its rule files.
1. Ensure Grafana can access the Prometheus rules API endpoint (`/api/v1/rules`).
1. For Prometheus (unlike Mimir), the Alerting UI only supports viewing existing rules, not creating new ones.

For more information on alerting with Prometheus, refer to [Prometheus alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/alerting/).

## Get additional help

If you continue to experience issues after following this troubleshooting guide:

1. Check the [Prometheus documentation](https://prometheus.io/docs/) for API and PromQL guidance.
1. Review the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Contact Grafana Support if you are a Cloud Pro, Cloud Contracted, or Enterprise user.
1. When reporting issues, include:
   - Grafana version
   - Prometheus version and type (Prometheus, Mimir, Cortex, Thanos)
   - Error messages (redact sensitive information)
   - Steps to reproduce
   - Relevant configuration such as data source settings, query timeout, and TLS settings (redact tokens, passwords, and other credentials)

# Configure the Pyroscope data source

The Pyroscope data source sets how Grafana connects to your Pyroscope backend.

You can configure the data source using either the data source interface in Grafana or using a configuration file.
This page explains how to set up and enable the data source capabilities using Grafana.

If you make any changes, select **Save & test** to preserve those changes.

If you're using your own installation of Grafana, you can provision the Pyroscope data source using a YAML configuration file.
For more information about provisioning and available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

## Before you begin

To configure a Pyroscope data source, you need administrator rights to your Grafana instance and a running Pyroscope backend that's receiving profiling data.

If you're provisioning a Pyroscope data source, then you also need administrative rights on the server hosting your Grafana instance.

## Add or modify a data source

You can use these procedures to configure a new Pyroscope data source or to edit an existing one.

### Create a new data source

To configure basic settings for the data source, complete the following steps:

1. Select **Connections** in the main menu.
1. Enter `Grafana Pyroscope` in the search bar.
1. Select **Grafana Pyroscope**.
1. Select **Add new data source** in the top-right corner of the page.
1. On the **Settings** tab, complete the **Name**, **Connection**, and **Authentication** sections.
   - Use the **Name** field to specify the name used for the data source in panels, queries, and Explore. Toggle the **Default** switch for the data source to be pre-selected for new panels.
   - Under **Connection**, enter the **URL** of the Pyroscope instance. For example, `https://example.com:4040`. Refer to [Connection URL](#connection-url) for more information.
   - Complete the [**Authentication** section](#authentication).

1. Optional: Use **Additional settings** to configure other options.
1. Select **Save & test**.

### Update an existing data source

To modify an existing Pyroscope data source:

1. Select **Connections** in the main menu.
1. Select **Data sources** to view a list of configured data sources.
1. Select the Pyroscope data source you want to modify.
1. Optional: Use **Additional settings** to configure or modify other options.
1. After completing your updates, select **Save & test**.

### Connection URL

The data source connection URL should point to a location of a running Pyroscope backend.

**Grafana Cloud Profiles**

Your Grafana Cloud instance automatically includes a fully provisioned data source.

If you are running a self-managed Grafana instance or need to configure an additional Pyroscope data source pointing to Grafana Cloud Profiles, you can find the Pyroscope URL under the **Manage your stack** section for your organization.

To authenticate against Grafana Cloud Profiles, enable **Basic auth** and enter your numeric instance ID as the username and an access policy token as the password. The token must belong to an access policy that includes the `profiles:read` scope and targets the correct stack. For more information, refer to [Grafana Cloud access policies](https://grafana.com/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-policies/).

**Self-managed Pyroscope backend**

The connection URL for a self-managed Pyroscope backend depends on how Pyroscope is deployed.
Refer to the steps under [Query profiles in Grafana](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/deploy-kubernetes/helm/#query-profiles-in-grafana) for more information on how to configure the data source.

If you plan to use the [Profiles Drilldown](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/simplified-exploration/profiles/) application and you are running a self-managed Pyroscope backend in microservices mode, the data source connection URL should point to a gateway or proxy that routes requests to the corresponding Pyroscope service.
Refer to the [Helm ingress configuration](https://github.com/grafana/pyroscope/blob/main/operations/pyroscope/helm/pyroscope/templates/ingress.yaml) for specific routing requirements.

## Authentication

Use this section to select an authentication method to access the data source.

{{< admonition type="note" >}}
Use Transport Layer Security (TLS) for an additional layer of security when working with Pyroscope.
For additional information on setting up TLS encryption with Pyroscope, refer to [Pyroscope configuration](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/configure-server/reference-configuration-parameters/).
{{< /admonition >}}

[//]: # 'Shared content for authentication section procedure in data sources'

{{< docs/shared source="grafana" lookup="datasources/datasouce-authentication.md" leveloffset="+2" version="<GRAFANA_VERSION>" >}}

## Additional settings

**Additional settings** are optional settings that give you more control over your data source. Select the **Additional settings** section to expand it and configure these options.

### Advanced HTTP settings

The Grafana Proxy deletes forwarded cookies. Use the **Allowed cookies** field to specify cookies that should be forwarded to the data source by name.

The **Timeout** field sets the HTTP request timeout in seconds.

### Private data source connect

[//]: # 'Shared content for authentication section procedure in data sources'

{{< docs/shared source="grafana" lookup="datasources/datasouce-private-ds-connect.md" leveloffset="+2" version="<GRAFANA_VERSION>" >}}

### Querying

Use the **Minimal step** field to set the minimum step used for metric queries that return time-series data. The default value is 15 seconds.
Enter a value in Grafana duration format using a number and a time unit specifier, such as `15s`, `1m`, or `1h`. Set it to the same value as, or higher than, the scrape interval configured in your Pyroscope database.

Adjusting this option can help prevent gaps when you zoom in to profiling data.

## Verify the connection

After you configure the data source, select **Save & test** to verify the connection.
When the connection is successful, Grafana displays a **Data source is working** message.
If the test fails, Grafana displays the error returned by the Pyroscope backend. For help resolving connection errors, refer to [Troubleshoot the Pyroscope data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/pyroscope/troubleshooting/).

## Provision the data source

You can provision the Pyroscope data source using YAML configuration files or Terraform, as part of the Grafana provisioning system.
For more information about provisioning and available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

### YAML provisioning example

The following example provisions a Pyroscope data source:

```yaml
apiVersion: 1

datasources:
  - name: Grafana Pyroscope
    type: grafana-pyroscope-datasource
    url: http://localhost:4040
    jsonData:
      minStep: '15s'
```

To provision with basic authentication, for example when connecting to Grafana Cloud Profiles:

```yaml
apiVersion: 1

datasources:
  - name: Grafana Pyroscope
    type: grafana-pyroscope-datasource
    url: https://example.grafana.net
    basicAuth: true
    basicAuthUser: '<INSTANCE_ID>'
    jsonData:
      minStep: '15s'
    secureJsonData:
      basicAuthPassword: <ACCESS_POLICY_TOKEN>
```

Replace the following:

- `<INSTANCE_ID>`: Your numeric instance ID, found under **Manage your stack**.
- `<ACCESS_POLICY_TOKEN>`: An access policy token with the `profiles:read` scope.

### Terraform example

To provision the data source with Terraform, use the [`grafana_data_source` resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source):

```hcl
resource "grafana_data_source" "pyroscope" {
  type = "grafana-pyroscope-datasource"
  name = "Grafana Pyroscope"
  url  = "http://localhost:4040"

  json_data_encoded = jsonencode({
    minStep = "15s"
  })
}
```

To provision with basic authentication:

```hcl
resource "grafana_data_source" "pyroscope" {
  type                = "grafana-pyroscope-datasource"
  name                = "Grafana Pyroscope"
  url                 = "http://localhost:4040"
  basic_auth_enabled  = true
  basic_auth_username = "<USERNAME>"

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = "<PASSWORD>"
  })
}
```

Replace the following:

- `<USERNAME>`: Your Pyroscope username.
- `<PASSWORD>`: Your Pyroscope password.

# Configure Trace to profiles

Trace to profiles lets you navigate from a trace span directly to the profiling data for that span. You configure it in the Tempo data source, which connects your tracing data in Tempo with your profiling data in Pyroscope.

{{< admonition type="note" >}}
Your application must be instrumented for profiles and traces. For more information, refer to [Link traces to profiles](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/configure-client/trace-span-profiles/).
{{< /admonition >}}

[//]: # 'Shared content for Trace to profiles in the Tempo data source'

{{< docs/shared source="grafana" lookup="datasources/tempo-traces-to-profiles.md" version="<GRAFANA_VERSION>" >}}

## Verify the connection

After you configure the Tempo data source and instrument your application, verify that Trace to profiles works.

1. Open a trace in **Explore** using your Tempo data source.
1. Expand a span to view its details.
1. Confirm that the **Profiles for this span** button appears in the span details.
1. Confirm that an embedded flame graph appears in the span details section.

If the span attribute `pyroscope.profile.id` isn't present on the span, the OpenTelemetry bridge package isn't configured correctly. Refer to [Link tracing and profiling with span profiles](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/configure-client/trace-span-profiles/) for per-language setup instructions.

## Troubleshoot trace to profiles

Use the following checklist to diagnose common issues with Trace to profiles.

### No "Profiles for this span" button appears

1. Verify that you installed and configured the OTel bridge package for your language. This is a separate package from the Pyroscope SDK and OpenTelemetry SDK. Refer to [Link tracing and profiling with span profiles](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/configure-client/trace-span-profiles/) for per-language instructions.
1. Check that the `pyroscope.profile.id` attribute exists on the span in Tempo. If it's missing, the bridge package isn't tagging spans correctly.
1. Check that the `span_name` label exists on the profiling data in Pyroscope. If it's missing, the bridge package isn't labeling profiling samples correctly.
1. Verify that the tags you configured in the Tempo data source are present in the span's attributes or resources. If the tags don't match, the span link doesn't appear.

### Button appears but no profile data is shown

1. The span might be shorter than the sampling interval of the profiler. Span profiling is most effective on spans longer than 20ms.
1. CPU profiling only captures time spent actively executing on the CPU. If the span is mostly waiting on I/O, network calls, or database queries, the CPU profile for that span may be sparse or empty.
1. Verify that the profile type selected in the Tempo data source configuration matches the profile type your application sends.

### Tags don't match

Tags you configure in the Tempo data source must be present in the span's attributes or resources. If a tag has dots in its name, for example, `service.name`, and the Pyroscope data source doesn't allow dots in labels, remap it to a name without dots, for example, `service_name`.

# Query profile data

You can query your profile data using the open source Grafana Profiles Drilldown app or the built-in Pyroscope data source query editor.

- Profiles Drilldown provides a queryless experience for inspecting your profiling data with opinionated workflows to assist your investigation.
- Data source query editor provides complete control over your data exploration and is recommended for experienced users.

In addition, you can embed flame graph panels in Grafana dashboards.
Refer to the [Flame graph panel](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/flame-graph/) documentation for details.

## Profiles Drilldown

[Grafana Profiles Drilldown](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/simplified-exploration/profiles/) is a native Grafana application designed to integrate seamlessly with Pyroscope, the open source continuous profiling platform, providing a smooth, queryless experience for browsing and analyzing profiling data.

You can use Profiles Drilldown in Grafana Cloud or in your own Grafana instance.
For more information, refer to [Access or install Profiles Drilldown](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/simplified-exploration/profiles/access/).

![Profiles Drilldown home screen](/media/docs/explore-profiles/explore-profiles-homescreen-v0.1.17.png)

### Use cases

There are several different modes for viewing, analyzing, and comparing profiling data.

The main use cases are the following:

- Proactive: Cutting costs, addressing latency issues, or optimizing memory usage for applications
- Reactive: Resolving incidents with line-level accuracy or debugging active latency/memory issues

Profiles Drilldown provides an intuitive interface to specifically support proactive and reactive use cases.
You get a holistic view of all of your services and how they're functioning, but also the ability to drill down for more targeted root cause analysis.

Profiles Drilldown offers a convenient platform to analyze profiles and get insights that are impossible to get from using other traditional signals like logs, metrics, or tracing.

{{< youtube id="x9aPw_CbIQc" >}}

{{< docs/play title="the Grafana Play site" url="https://play.grafana.org/a/grafana-pyroscope-app/profiles-explorer" >}}

## Pyroscope query editor

The Pyroscope data source query editor gives you access to a profile type selector, a label selector, and collapsible options.

Like Profiles Drilldown, the query editor also provides a flame graph to visualize data.

![Query editor](/media/docs/pyroscope/query-editor/query-editor.png 'Query editor')

To access the query editor:

1. Sign into Grafana or Grafana Cloud.
1. Select your Pyroscope data source.
1. From the menu, choose **Explore**.

1. Select a profile type from the drop-down menu.

   {{< figure src="/media/docs/pyroscope/query-editor/select-profile.png" class="docs-image--no-shadow" max-width="450px" caption="Profile selector" >}}

1. Use the labels selector input to filter by labels. Pyroscope uses similar syntax to Prometheus to filter labels.
   Refer to [Label selector syntax](#label-selector-syntax) for available operators and syntax.

   While the label selector can be left empty to query all profiles without filtering by labels, you must select a profile type or app for the query to be valid.

   Grafana doesn't show any data if the profile type or app isn't selected when a query runs.

   ![Labels selector](/media/docs/pyroscope/query-editor/labels-selector.png 'Labels selector')

1. Expand the **Options** section to view the available query options.
   ![Options section](/media/docs/pyroscope/query-editor/options-section.png 'Options section')

### Label selector syntax

The label selector uses a syntax similar to Prometheus. Enclose one or more label matchers in curly braces, separated by commas. Each matcher combines a label name, an operator, and a value in double quotes.

The label selector supports the following operators.

| Operator | Description                                             |
| -------- | ------------------------------------------------------- |
| `=`      | Selects labels that exactly match the value.            |
| `!=`     | Selects labels that don't match the value.              |
| `=~`     | Selects labels that match the regular expression.       |
| `!~`     | Selects labels that don't match the regular expression. |

The following examples show common label selectors:

```
{service_name="my-service"}
{service_name="my-service", namespace="production"}
{service_name=~"cart.*"}
{service_name="checkout", namespace!="staging"}
```

Leave the label selector empty, or set it to `{}`, to query all profiles for the selected profile type without filtering by labels.

### Query options

The **Options** section provides the following settings.

| Option          | Description                                                                                                                                                                                                                |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Query Type**  | Sets the type of data the query returns: **Profile** for flame graph data, **Metric** for time-series data, or **Both**. You can only select **Both** in Explore. Panels on dashboards allow only one visualization.       |
| **Group by**    | Groups metric data by one or more labels. Without a **Group by** label, metric data aggregates over all labels into a single time series. **Group by** applies only to metric data and doesn't change the profile results. |
| **Limit**       | Sets the maximum number of time series returned when **Group by** is set. The series returned are always ordered by descending value for the total aggregated data over the time period. Doesn't apply to profile queries. |
| **Span ID**     | Sets the span ID from which to search for profiles, so you can view the profile associated with a specific trace span. This option supports Trace to profiles, for example, `64f170a95f537095`.                            |
| **Max Nodes**   | Sets the maximum number of nodes rendered in the flame graph. Lower values improve performance for large profiles by aggregating smaller nodes. The field placeholder shows the default value, `16384`.                    |
| **Annotations** | Includes profiling annotations, such as sampling and throttling events, on time-series results. Use these annotations to identify when Pyroscope reduced or limited ingestion for a service.                               |

Select a query type to return the profile data. Data is shown in the [Flame Graph](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/flame-graph/), metric data visualized in a graph, or both.

Using **Group by**, you can group metric data by a specified label.
You can use multiple labels to group by.

In conjunction with **Group by**, you can set a positive number in the **Limit** input to limit the maximum number of time series returned by the data source.

#### Exemplars

The **Exemplars** toggle in the **Options** section links individual profile samples to points on a metrics graph, so you can jump from a spike in a time series to the specific profile that produced it.

#### Heatmap

When the `profilesHeatmap` feature is enabled, a **Heatmap** toggle appears in the **Options** section.
The heatmap visualizes the distribution of profile samples over time, which helps you spot patterns and outliers across many profiles.

When you enable **Heatmap**, use the **Heatmap Type** option to choose how samples are aggregated:

- **Individual**: Shows individual profile samples.
- **Span**: Aggregates samples by span duration, for use with Trace to profiles.

{{< admonition type="note" >}}
The `profilesHeatmap` feature is experimental. Experimental features are subject to change and may be removed in a future release. Enable it using the `profilesHeatmap` feature toggle.
{{< /admonition >}}

#### UTF-8 label names

When the `pyroscopeUTF8LabelNames` feature is enabled, the label selector supports UTF-8 label names.
Enclose label names that contain characters such as dots in double quotes, for example, `{"k8s.namespace"="production"}`.

The `pyroscopeUTF8LabelNames` feature is in public preview and enabled by default.

### Query examples

The following examples show common ways to build queries in the query editor. Each example lists the profile type to select and the label selector to enter.

**Investigate high CPU usage for a service:**

Select the `process_cpu:cpu:nanoseconds:cpu:nanoseconds` profile type and filter to the affected service:

```
{service_name="checkout"}
```

Set **Query Type** to **Both** in Explore to view the CPU flame graph alongside a time series of CPU usage.

**Compare memory allocations across environments:**

Select the `memory:alloc_space:bytes:space:bytes` profile type, set **Query Type** to **Metric**, and set **Group by** to `namespace`:

```
{service_name="cart"}
```

The metrics graph shows a separate time series for each namespace, which lets you compare allocation behavior between environments such as `production` and `staging`.

**Analyze goroutine growth:**

Select the `goroutine:goroutine:count:goroutine:count` profile type to inspect the number of goroutines and where they're created:

```
{service_name=~"api.*"}
```

**View the profile for a specific trace span:**

Select a profile type, set **Query Type** to **Profile**, and enter the span ID in the **Span ID** field, for example, `64f170a95f537095`. The flame graph then shows only the profile data associated with that span. This example supports [Trace to profiles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/pyroscope/configure-traces-to-profiles/).

### Profiles query results

Profiles can be visualized in a flame graph.
Refer to the [Flame Graph documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/flame-graph/) to learn about the visualization and its features.

![Flame graph](/media/docs/pyroscope/query-editor/flame-graph.png 'Flame graph')

Pyroscope returns profiles aggregated over a selected time range.
The absolute values in the flame graph grow as the time range gets bigger while keeping the relative values meaningful.
You can zoom in on the time range to get a higher granularity profile up to the point of a single scrape interval.

### Metrics query results

Metrics results represent the aggregated sum value over time of the selected profile type.

![Metrics graph](/media/docs/pyroscope/query-editor/metric-graph.png 'Metrics graph')

This allows you to quickly see any spikes in the value of the scraped profiles and zoom in to a particular time range.

## Next steps

- [Use template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/pyroscope/template-variables/)
- [Troubleshoot the Pyroscope data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/pyroscope/troubleshooting/)

# Pyroscope template variables

Template variables let you create dynamic, reusable dashboards by replacing hard-coded values, such as service names or profile types, with selectable variables. Grafana displays these variables as drop-down menus at the top of the dashboard, letting viewers change the displayed data without editing queries.

For an introduction to templating and template variables, refer to [Templating](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/).

## Query variable types

Query variables query the Pyroscope data source to populate drop-down values. When you create a query variable, select a Pyroscope data source and choose a query type.

| Query type       | Required inputs     | Description                                                                                                                                                                                                     |
| ---------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Profile type** | None                | Returns the profile types available from the data source. The drop-down displays a readable label, while the variable value is the full profile type ID, such as `process_cpu:cpu:nanoseconds:cpu:nanoseconds`. |
| **Label**        | Profile type        | Returns the label names available for the selected profile type, such as `service_name`, `namespace`, and `region`.                                                                                             |
| **Label value**  | Profile type, label | Returns the values for the selected label, scoped to the selected profile type.                                                                                                                                 |

Because the **Label** and **Label value** query types require a profile type, and **Label value** also requires a label, you can chain these variables so that each one depends on the previous selection.

## Create a query variable

To create a query variable:

1. Navigate to **Dashboard settings** > **Variables**.
1. Select **Add variable**.
1. Set **Select variable type** to **Query**.
1. Select the Pyroscope data source.
1. Select a **Query type** of **Profile type**, **Label**, or **Label value**.
1. For a **Label** or **Label value** query, select a **Profile type**.
1. For a **Label value** query, select a **Label**.
1. Select **Apply** to save the variable.

## Query variable examples

The following examples show common ways to use Pyroscope query variables.

**Select a profile type from a drop-down:**

Create a variable named `profile_type` to let viewers switch the profile type without editing the query:

1. Set **Query type** to **Profile type**.
1. Save the variable. The drop-down displays readable labels, while each value is the full profile type ID, such as `process_cpu:cpu:nanoseconds:cpu:nanoseconds`.

You can then reference `$profile_type` in the query editor's profile type field.

**Populate a drop-down with services:**

Create a variable named `service` that lists the services sending profiles:

1. Set **Query type** to **Label value**.
1. Set **Profile type** to a profile type, such as `process_cpu:cpu:nanoseconds:cpu:nanoseconds`.
1. Set **Label** to `service_name`.
1. Save the variable. The drop-down now lists the values of the `service_name` label.

**Chain dependent variables:**

Combine variables so each one narrows the next. This example builds a profile type selector, a label selector, and a label value selector that all update together:

1. Create a `profile_type` variable with **Query type** set to **Profile type**.
1. Create a `label` variable with **Query type** set to **Label** and **Profile type** set to `$profile_type`.
1. Create a `label_value` variable with **Query type** set to **Label value**, **Profile type** set to `$profile_type`, and **Label** set to `$label`.

When a viewer changes `profile_type`, the `label` drop-down refreshes with the labels for that profile type, and `label_value` refreshes with the values for the selected label.

## Use variables in queries

After you create a variable, reference it in the query editor using the `$variable` or `${variable}` syntax. You can reference variables in both the profile type field and the label selector.

For example, if you have a variable named `service` created with the **Label value** query type on the `service_name` label, reference it in the label selector:

```
{service_name="$service"}
```

To let viewers switch the profile type, enter `$profile_type` in the query editor's profile type field.

When **Multi-value** or **Include All** is enabled for the variable, the value becomes a regular expression pattern, such as `value1|value2`. Use the `=~` operator instead of `=` in your label selector:

```
{service_name=~"$service"}
```

## Where you can use template variables

The Pyroscope data source interpolates template variables in only some parts of the query editor. The following table lists which fields support variables.

| Query editor field | Supports template variables |
| ------------------ | --------------------------- |
| Profile type       | Yes                         |
| Label selector     | Yes                         |
| Group by           | No                          |
| Span ID            | No                          |
| Max Nodes          | No                          |
| Limit              | No                          |

Query variable definitions also support variables. You can reference one variable in another to chain **Label** and **Label value** queries, as shown in [Chain dependent variables](#query-variable-examples).

{{< admonition type="note" >}}
Fields other than the profile type and label selector don't interpolate template variables. If you enter a variable such as `$group` in the **Group by**, **Span ID**, **Max Nodes**, or **Limit** fields, Grafana passes the literal text to the backend instead of the variable's value.
{{< /admonition >}}

## Filters variable

The Pyroscope data source supports the [Filters](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#add-ad-hoc-filters) variable type, which lets dashboard viewers add label filters without editing queries.

To set up a Filters variable:

1. Navigate to **Dashboard settings** > **Variables**.
1. Select **Add variable**.
1. Set **Select variable type** to **Filters**.
1. Select the Pyroscope data source.
1. Select **Apply** to save the variable.

After you add the variable, a filter bar appears at the top of the dashboard. Viewers can add filters by selecting a label, an operator (`=`, `!=`, `=~`, `!~`), and a value. Grafana applies these filters to all Pyroscope queries on the dashboard.

## Related resources

- [Query profile data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/pyroscope/query-editor/): Use variables in the label selector.
- [Troubleshoot the Pyroscope data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/pyroscope/troubleshooting/): Resolve variable-related query issues.

# Troubleshoot Pyroscope data source issues

This document provides solutions to common issues you may encounter when configuring or using the Pyroscope data source. For configuration instructions, refer to [Configure the Pyroscope data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/pyroscope/configure/).

## Connection and configuration errors

These errors occur when setting up the data source or when connecting to the Pyroscope backend.

### "Data source is not working" or connection fails

**Symptoms:**

- Save & test fails with a connection error
- Error message indicates the Pyroscope backend is unreachable
- Timeout errors when testing the data source

**Possible causes and solutions:**

| Cause                         | Solution                                                                                                                                                                                  |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Incorrect URL                 | Verify the URL points to your Pyroscope instance. For self-managed deployments, the default port is `4040` (for example, `http://localhost:4040`).                                        |
| Pyroscope service not running | Check that the Pyroscope backend is running and accessible. Use `curl` or a browser to test connectivity to the Pyroscope URL.                                                            |
| Network connectivity issues   | Verify network connectivity from the Grafana server to the Pyroscope endpoint. Check firewall rules allow outbound connections on the required port.                                      |
| Microservices mode routing    | If running Pyroscope in microservices mode, ensure the URL points to a gateway or proxy that routes requests correctly. Refer to the Helm ingress configuration for routing requirements. |
| TLS/SSL certificate issues    | If using HTTPS, verify the certificate is valid and trusted by the Grafana server. Configure TLS settings in the data source if using self-signed certificates.                           |

### URL format issues

**Symptoms:**

- Data source test fails immediately
- Error indicates invalid URL

**Solutions:**

1. Ensure the URL includes the protocol (`http://` or `https://`).
1. Remove any trailing slashes from the URL.
1. For Grafana Cloud Profiles, find the correct URL under **Manage your stack** in your organization settings.
1. Verify the port number is correct for your deployment.

### "Connection refused" or timeout errors

**Symptoms:**

- Data source test fails with network errors
- Queries fail with connection errors
- Intermittent connectivity issues

**Solutions:**

1. Verify network connectivity from the Grafana server to the Pyroscope endpoint.
1. Check firewall rules allow outbound connections on the required port (default: 4040).
1. For Kubernetes deployments, verify the service is exposed correctly.
1. Check if a proxy is required and configure it in the data source settings.

### SSL/TLS certificate errors

**Symptoms:**

- Certificate validation failures
- SSL handshake errors
- "Certificate not trusted" messages

**Solutions:**

1. Ensure the system time is correct on the Grafana server.
1. Verify the certificate is valid and not expired.
1. For self-signed certificates, enable **Skip TLS Verify** in the data source settings (not recommended for production).
1. Add the CA certificate to the Grafana server's trusted certificates.
1. Check that intermediate certificates are included in the certificate chain.

### Private connectivity or reverse proxy issues

**Symptoms:**

- The data source works from some networks but not others.
- Connecting through a reverse proxy, load balancer, or private link fails.

**Solutions:**

1. Include the scheme in the **Connection URL**. When connecting through a private endpoint or reverse proxy, use `https://` explicitly rather than a bare host name.
1. Ensure the reverse proxy or gateway forwards the Pyroscope API paths unchanged. Rewriting or stripping the request path prevents the data source from reaching the backend.
1. For Grafana Cloud connecting to a Pyroscope backend on a private network, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/).
1. Verify that any firewall or private link allows outbound HTTPS from Grafana to the Pyroscope endpoint.

## Authentication errors

These errors occur when the data source cannot authenticate with the Pyroscope backend.

### "Authentication failed" or "Unauthorized"

**Symptoms:**

- Save & test fails with authentication errors
- Queries return 401 or 403 status codes
- Profile types don't load

**Possible causes and solutions:**

| Cause                          | Solution                                                                                                                                             |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| Invalid credentials            | Verify the username and password or API key are correct.                                                                                             |
| Expired credentials            | Generate new credentials and update the data source configuration.                                                                                   |
| Incorrect authentication type  | Ensure you've selected the correct authentication method for your Pyroscope deployment (Basic auth, API key, or no authentication for local setups). |
| Missing authentication headers | For custom authentication, verify the required headers are configured correctly in the data source settings.                                         |

### Basic authentication not working

**Symptoms:**

- Basic auth credentials are rejected
- Works in browser but fails in Grafana

**Solutions:**

1. Verify Basic authentication is enabled in the data source settings.
1. Ensure the username and password are entered correctly (no extra spaces).
1. Check that the Pyroscope backend is configured to accept Basic authentication.
1. For Grafana Cloud Profiles, use Basic authentication with your access policy token as the password. Refer to [Grafana Cloud access policy and token issues](#grafana-cloud-access-policy-and-token-issues).

### API key authentication issues

**Symptoms:**

- API key is rejected
- Error indicates invalid token

**Solutions:**

1. Verify you're using the correct API key format for your Pyroscope deployment.
1. For Grafana Cloud, generate a new access policy token with the `profiles:read` scope.
1. Ensure the credentials have the required permissions to read profiling data.
1. Check if the token has expired and generate a new one if necessary.

### Grafana Cloud access policy and token issues

**Symptoms:**

- Authentication fails when connecting a data source to Grafana Cloud Profiles.
- Queries return 401 or 403 after a backend migration or stack change.
- The data source reads from a different stack or tenant than expected.

**Cause:**

Grafana Cloud Profiles authenticates with an access policy token rather than a user password. The token must belong to an access policy that includes the `profiles:read` scope and targets the correct stack. A token with the wrong scope, or one created for a different stack, fails authentication or returns data from the wrong tenant. Tokens and connection URLs are stack-specific and don't carry over during a migration.

**Solutions:**

1. Create or select an access policy that includes the `profiles:read` scope and whose realm targets the correct stack, then generate a token for that policy.
1. In the data source, enable **Basic auth**. Enter the numeric instance ID as the username and the access policy token as the password. Find the instance ID and endpoint under **Manage your stack**.
1. Confirm the **Connection URL** points to the Pyroscope endpoint for the same stack as the token.
1. After a backend migration or stack change, regenerate the token, then update both the URL and credentials.

## Query errors

These errors occur when executing queries against the Pyroscope data source.

### "No data" or empty results

**Symptoms:**

- Query executes without error but returns no data
- Flame graph shows "No data" message
- Metrics graph is empty

**Possible causes and solutions:**

| Cause                                 | Solution                                                                                                             |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Time range doesn't contain data       | Expand the dashboard time range. Profiling data may not exist for the selected period.                               |
| Application not sending profiles      | Verify your application is configured to send profiles to Pyroscope. Check the Pyroscope agent or SDK configuration. |
| Incorrect profile type selected       | Select a different profile type from the drop-down menu. Not all profile types are available for all applications.   |
| Label selector too restrictive        | Remove or modify label filters to broaden the query. Start without filters to verify data exists.                    |
| Wrong service or application selected | Verify the `service_name` label matches your application's configuration.                                            |
| Profile type or app not selected      | Ensure you've selected a profile type or app in the query editor. The query returns no data if neither is selected.  |

{{< admonition type="note" >}}
If the data source connection tests successfully but no profiles exist for any query, the problem is usually on the ingestion side rather than in the data source. The data source only reads profiles from Pyroscope; it doesn't collect or send them. Verify that your application, SDK, or collector, such as Grafana Alloy or an OpenTelemetry Collector, is sending profiles to your Pyroscope backend. For ingestion setup, refer to [Configure the client to send profiles](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/configure-client/).
{{< /admonition >}}

### "Profile type not found" or empty profile type selector

**Symptoms:**

- Profile type drop-down is empty
- Error indicates profile type doesn't exist

**Solutions:**

1. Verify the Pyroscope backend is receiving profiles from your applications.
1. Check that the time range includes periods when profiles were collected.
1. Test the data source connection to ensure it's working correctly.
1. For new deployments, wait a few minutes for initial profile data to be ingested.

### Label names or values not loading

**Symptoms:**

- Label selector doesn't populate
- No labels appear in the drop-down menus
- Autocomplete doesn't work

**Solutions:**

1. Verify the data source connection is working by testing it in the settings.
1. Select a profile type first, as labels are loaded based on the selected profile type.
1. Expand the time range to ensure label data is available.
1. Check the browser console for JavaScript errors that might indicate issues.

### Query syntax errors

**Symptoms:**

- Error message indicates invalid query syntax
- Label selector is rejected

**Solutions:**

1. Use the correct label selector syntax. Pyroscope uses a syntax similar to Prometheus.
1. Enclose label values in double quotes: `{service_name="my-service"}`.
1. Use valid operators: `=`, `!=`, `=~`, `!~`.
1. Escape special characters in regular expression patterns when using `=~` or `!~`.

Example valid queries:

```
{service_name="my-app"}
{service_name="my-app", env="production"}
{service_name=~"my-app.*"}
```

### Queries return only recent data even though older data is retained

If a query over a long time range returns data only for the most recent days, the query time range likely exceeds a server-side query limit. This is most commonly reported in Grafana Cloud Profiles. Refer to [Query time range and retention limits](#query-time-range-and-retention-limits).

## Flame graph issues

These errors are specific to the flame graph visualization.

### Flame graph not rendering

**Symptoms:**

- Query returns data but flame graph doesn't display
- Flame graph area is blank
- Loading indicator never completes

**Solutions:**

1. Check that the Query Type is set to include profile data (not metrics only).
1. Verify the response contains valid profile data using the Query Inspector.
1. Try refreshing the page or re-running the query.
1. For large profiles, increase the browser memory limits or reduce the time range.

### Flame graph times out or fails to load on large time ranges

**Symptoms:**

- A flame graph renders for a short time range, such as five minutes, but times out for a longer range, such as 30 minutes or more.
- Queries over large time ranges are slow or fail intermittently.

**Cause:**

Rendering a flame graph over a large time range requires the Pyroscope backend to merge many profiles, which increases query time and memory use. Large or high-cardinality ranges can exceed request timeouts before a result returns.

**Solutions:**

1. Narrow the time range. Start with a short window, such as five minutes, then widen it gradually to find the largest range that renders reliably.
1. Add label filters to reduce the volume of profiles the backend must merge.
1. Confirm the range is within the server-side query limits described in [Query time range and retention limits](#query-time-range-and-retention-limits).
1. For self-managed deployments, review backend resources and query timeout settings if large ranges consistently fail.

### Flame graph shows aggregated data without detail

**Symptoms:**

- Flame graph lacks expected granularity
- Function names are missing or truncated
- Stack traces appear incomplete

**Solutions:**

1. Zoom in to a smaller time range to get more detailed profiles.
1. Verify your application is instrumented to capture full stack traces.
1. Check if the profiling agent is configured with sufficient sampling rate.
1. For compiled languages, ensure debug symbols are available.

### Unable to interact with flame graph

**Symptoms:**

- Clicking on flame graph elements doesn't work
- Tooltip doesn't appear
- Zoom and filter controls are unresponsive

**Solutions:**

1. Verify JavaScript is enabled in your browser.
1. Check the browser console for JavaScript errors.
1. Try using a different browser or clearing the browser cache.
1. Update Grafana to the latest version.

## Template variable errors

These errors occur when using template variables with the Pyroscope data source.

### Variables return no values

**Symptoms:**

- The variable drop-down is empty
- Dependent variables don't populate

**Solutions:**

1. Verify the data source connection is working by testing it in the settings.
1. For **Label** and **Label value** variables, verify that a profile type is selected. Labels and label values are scoped to the selected profile type.
1. Expand the dashboard time range to ensure the variable query covers a period when profiles were collected.
1. Confirm the Pyroscope backend is receiving profiles from your applications.

### Variables don't filter queries as expected

**Symptoms:**

- Query results don't change when the variable value changes
- Queries return no data after selecting a variable value

**Solutions:**

1. Verify the variable is referenced correctly in the label selector, for example, `{service_name="$service"}`.
1. When **Multi-value** or **Include All** is enabled, the variable value becomes a regular expression pattern. Use the `=~` operator instead of `=`, for example, `{service_name=~"$service"}`.
1. Confirm the label used in the variable exists on the profiling data for the selected profile type.

For more information, refer to [Pyroscope template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/pyroscope/template-variables/).

## Profiles Drilldown issues

These errors are specific to the [Profiles Drilldown](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/simplified-exploration/profiles/) application.

### Profiles Drilldown not available

**Symptoms:**

- Profiles Drilldown option doesn't appear in the menu
- Error when trying to access Profiles Drilldown

**Solutions:**

1. Verify the Profiles Drilldown plugin is installed and enabled.
1. For self-managed Grafana, install the plugin from the Grafana plugin catalog.
1. Restart Grafana after installing the plugin.
1. Check that you have the required permissions to access the application.

### Services or profiles not appearing in Profiles Drilldown

**Symptoms:**

- Profiles Drilldown shows no services
- Expected applications are missing

**Solutions:**

1. Verify the Pyroscope data source is configured and working.
1. Check that profiles are being ingested with the correct `service_name` label.
1. Ensure the time range includes periods when profiles were collected.
1. For microservices mode, verify the data source URL points to the correct gateway.

## Trace to profiles issues

These errors occur when linking tracing and profiling data.

Trace to profiles is the most configuration-sensitive feature of the data source. Most issues come from an unmet prerequisite rather than a data source misconfiguration.

**Prerequisites checklist:**

Confirm all of the following before troubleshooting further:

1. Your application is instrumented with a Pyroscope language SDK for profiling and with OpenTelemetry for tracing.
1. You installed the OpenTelemetry span-profiling bridge package for your language. This is a separate package from the Pyroscope SDK and the OpenTelemetry SDK.
1. Profiling uses SDK-based instrumentation. Span profiles aren't available with eBPF-based collection.
1. The Tempo data source has Trace to profiles configured and points to your Pyroscope data source.
1. The tags configured in the Tempo data source are present on the span's attributes or resources.

For per-language setup instructions, refer to [Link traces to profiles](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/configure-client/trace-span-profiles/).

### Span profiles not appearing

**Symptoms:**

- Trace spans don't show profile links
- "Profiles for this span" option is missing

**Solutions:**

1. Verify the Tempo data source is configured with Trace to profiles enabled and that it points to your Pyroscope data source.
1. Confirm the `pyroscope.profile.id` attribute exists on the span in Tempo. If it's missing, the bridge package isn't tagging spans correctly.
1. Confirm the `span_name` label exists on the profiling data in Pyroscope. If it's missing, the bridge package isn't labeling samples correctly.
1. Verify that the tags you configured in the Tempo data source match the span's attributes or resources. If they don't match, the link doesn't appear.
1. Verify profile data exists for the time range of the trace.

For more information, refer to [Configure Trace to profiles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/pyroscope/configure-traces-to-profiles/).

### Profile doesn't match the trace span

**Symptoms:**

- Profile data doesn't correlate with the trace
- Wrong time range in profile

**Solutions:**

1. Verify the trace and profile timestamps are synchronized.
1. Check that the application's clock is accurate.
1. Ensure the `span_id` and `trace_id` labels are correctly propagated to profiles.

### Span profiles aren't available with eBPF-based collection

**Symptoms:**

- Traces and profiles both exist, but no span links appear.
- Profiling is collected with eBPF, for example through Grafana Alloy or Beyla.

**Cause:**

Span profiling correlates profiling samples with the active trace span, which requires a language SDK and the OpenTelemetry span-profiling bridge. eBPF-based collection can't associate samples with individual spans, so it doesn't produce span profiles.

**Solutions:**

1. Instrument the application with a supported Pyroscope language SDK and the matching bridge package to enable span profiles.
1. If you must use eBPF profiling, view profiles by service instead. Per-span correlation isn't currently supported for eBPF-collected data.

### Filter span profiles by span name or trace ID, not span ID

**Symptoms:**

- Queries filtered by `span_id` return no data, while `span_name` works.
- Correlating profiles to a specific span by `span_id` is unreliable.

**Cause:**

The span-profiling bridge labels profiling samples with `span_name` and `trace_id`. Depending on the SDK, profiling context may be attached only to the root span of a trace, with child spans inheriting it. `span_id` is high cardinality and isn't a dependable correlation label.

**Solutions:**

1. Filter profiling data by `span_name` or `trace_id` rather than `span_id`.
1. Use the **Profiles for this span** link, which correlates using the `pyroscope.profile.id` span attribute rather than a label query.

### .NET profiler conflict with OpenTelemetry auto-instrumentation

**Symptoms:**

- On .NET, span profiles don't work when the Pyroscope .NET profiler and OpenTelemetry auto-instrumentation are both enabled.
- One profiler fails to attach.

**Cause:**

The .NET CLR allows only one profiler to attach to a process at a time. The Pyroscope .NET profiler and OpenTelemetry auto-instrumentation each rely on a separate CLR profiler, so they conflict.

**Solutions:**

1. Use OpenTelemetry manual instrumentation instead of auto-instrumentation.
1. Add the `Pyroscope.OpenTelemetry` package and register the `PyroscopeSpanProcessor` in your tracing pipeline.
1. For setup instructions, refer to [Span profiles with Traces to profiles for .NET](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/configure-client/trace-span-profiles/dotnet-span-profiles/).

### Correlate logs with profiles

**Symptoms:**

- You want to navigate from a log line directly to profiling data, but no link exists.

**Cause:**

Grafana supports Trace to profiles, but there's no direct logs-to-profiles correlation.

**Solutions:**

1. Correlate through shared labels, such as `service_name`, to move between logs and profiles for the same service and time range.
1. If your logs contain a `trace_id`, use a Loki derived field to open the trace, then use **Profiles for this span** from the trace to reach the profiling data.

## Performance issues

These issues relate to slow queries or high resource usage.

### Slow queries or timeouts

**Symptoms:**

- Queries take a long time to complete
- Requests time out before returning results
- Dashboard panels fail to load

**Solutions:**

1. Reduce the time range to decrease the amount of data processed.
1. Add label filters to narrow the query scope.
1. Increase the **Timeout** setting in the data source's **Additional settings**.
1. Adjust the **Minimal step** setting to reduce the number of data points.
1. For large deployments, ensure the Pyroscope backend has sufficient resources.

### High memory usage in browser

**Symptoms:**

- Browser becomes unresponsive when viewing profiles
- Memory warnings appear
- Page crashes when loading flame graphs

**Solutions:**

1. Reduce the time range to load smaller profiles.
1. Close unnecessary browser tabs to free up memory.
1. Use a smaller aggregation window for very large profiles.
1. Consider using Profiles Drilldown for large-scale analysis.

## Grafana Cloud-specific issues

These issues are most commonly reported in Grafana Cloud Profiles. Some limits also apply to self-managed Pyroscope deployments, where you can adjust them directly in the Pyroscope configuration.

### Query time range and retention limits

**Symptoms:**

- A query over a long time range, such as 30 days, returns data only for the most recent days.
- Older profiles appear to be missing even though the retention period hasn't elapsed.
- Users interpret the missing data as data loss or a bug.

**Cause:**

Pyroscope enforces query limits on the server side that are separate from, and often shorter than, the data retention period. These limits are enforced by the Pyroscope query frontend, not by the Grafana data source, so the data source can't change or override them.

| Limit                        | Default | Description                                                                                                                                                                                              |
| ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `querier.max-query-lookback` | `7d`    | How far back from the current time a query can read. Requests for data older than this window don't fail. Instead, the range is silently truncated to the allowed window, which looks like missing data. |
| `querier.max-query-length`   | `24h`   | The maximum time span of a single query. Requests that exceed this span are rejected.                                                                                                                    |

Because retention is typically longer than the query window, profiles can be retained but not queryable. For example, Grafana Cloud Profiles retains data for 14 days on free plans and 30 days on paid plans, while the default query window is 7 days.

{{< admonition type="note" >}}
The default values shown are the Pyroscope OSS defaults. Grafana Cloud and self-managed deployments may configure different values.
{{< /admonition >}}

**Solutions:**

1. Confirm this is a limit rather than a gap in ingestion by narrowing the time range to the most recent period and verifying that data appears.
1. To query further back than the query window, raise the limit rather than splitting the query into smaller ranges. Because the window is measured from the current time, splitting a long range into 7-day increments doesn't return older data.
1. For self-managed deployments, increase `querier.max-query-lookback` and, if needed, `querier.max-query-length` in the Pyroscope limits configuration. Set a value to `0` to disable that limit.
1. For Grafana Cloud Profiles, contact Grafana Support to adjust these limits for your tenant.
1. If a single query exceeds `querier.max-query-length`, reduce the query's time span so it stays within the configured limit.

## Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location).
1. Look for Pyroscope-specific entries that include request and response details.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you've tried the solutions in this guide and still encounter issues:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review the [Pyroscope GitHub issues](https://github.com/grafana/pyroscope/issues) for known bugs.
1. Refer to [Pyroscope documentation](https://grafana.com/docs/pyroscope/latest/) for additional guidance.
1. Contact [Grafana Support](https://grafana.com/contact/) if you're an Enterprise, Cloud Pro, or Cloud Contracted user.
1. When reporting issues, include:
   - Grafana version
   - Pyroscope version (if self-managed)
   - Error messages (redact sensitive information)
   - Steps to reproduce
   - Data source configuration (redact credentials)

# Service Graph and Service Graph view

The Service Graph is a visual representation of the relationships between services in a distributed system.
Each node represents a service such as an API or database, and each edge represents requests between services.

Use the Service Graph to detect performance issues, track error and latency trends across services, and navigate to the traces that caused them.

The Service Graph is built on RED metrics, a monitoring methodology that focuses on three signals:

- **Rate:** requests per second
- **Errors:** failed requests per second
- **Duration:** latency distribution (for example, p50, p95, p99)

Service graph data flows from traces through a metrics pipeline into Grafana:

1. Your application emits traces to Tempo.
1. The [Tempo metrics generator](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/) or [Grafana Alloy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/enable-service-graphs/) processes the traces and writes service graph metrics to a Prometheus-compatible backend.
1. Grafana reads those metrics from your linked Prometheus data source and renders the graph.

{{< figure src="/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-node-graph.png" class="docs-image--no-shadow" max-width="500px" alt="Node graph showing service dependencies with color-coded circles representing request success and failure rates" >}}

## Before you begin

To use the Service Graph, you need:

- **Service graph metrics generation** configured in [Grafana Alloy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/enable-service-graphs/) or [the Tempo metrics-generator](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/). Without this, no service graph data exists.
- **A Prometheus data source** in Grafana that receives the generated metrics.
- **The Prometheus data source linked** in the Tempo data source settings under [Service graph](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/additional-settings/#service-graph). Refer to [Additional settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/additional-settings/#service-graph) for details, or [Provision the data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/) for YAML configuration.

If you run a Service Graph query and see an empty result, refer to [Troubleshoot](#troubleshoot) on this page.

## How it works

The metrics generator or Alloy inspects trace spans for parent-child relationships that represent a request, using [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/general/trace/). It writes the following metrics to your Prometheus backend:

- `traces_service_graph_request_total`: Total requests between two services (counter).
- `traces_service_graph_request_failed_total`: Failed requests between two services (counter).
- `traces_service_graph_request_server_seconds`: Response time as observed by the server (histogram).
- `traces_service_graph_request_client_seconds`: Response time as observed by the client (histogram).

Grafana reads these metrics from the linked Prometheus data source to render the node graph and calculate request rates, error rates, and response times.

For the full metrics reference, including labels, virtual nodes, and additional metrics, refer to [Service graphs](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/) in the Tempo documentation.

## Display the Service Graph

To display the Service Graph:

1. Navigate to [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/).
1. Select the Tempo data source.
1. Select the **Service Graph** query type.
1. Run a query.
1. _(Optional)_ Filter by service name.

Each circle in the graph represents a service.
Numbers inside the circles indicate the response time and requests per second.

Each circle's color shows the proportion of successful and failed requests:

| Color      | Meaning             |
| ---------- | ------------------- |
| **Green**  | Success             |
| **Red**    | Fault               |
| **Yellow** | Errors              |
| **Purple** | Throttled responses |

The color mix of each circle shows the proportion of each state. A mostly green circle indicates healthy traffic; visible red, yellow, or purple arcs indicate faults, errors, or throttling respectively.

To open a context menu with links for quick navigation to traces, metrics, and related views, click a service.

For details on navigating, zooming, and interacting with the graph layout, refer to the [Node graph panel](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/node-graph/) documentation.

To investigate a service with a high error rate, use TraceQL:

```traceql
{ resource.service.name = "<SERVICE_NAME>" && status = error }
```

Replace _`<SERVICE_NAME>`_ with the name shown in the Service Graph node.
Refer to [TraceQL query examples](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-query-examples/) for more queries you can run from Service Graph observations.

## Service Graph view

The Service Graph view displays a table of RED metrics (rate, error rate, duration) alongside the node graph.
The table uses a different set of metrics from the node graph:

- `traces_spanmetrics_calls_total`: Rate and error rate columns.
- `traces_spanmetrics_latency_bucket`, or `traces_spanmetrics_latency` for native histograms: Duration column.

These span metrics must be present in your linked Prometheus data source.
Span metrics generation must be enabled in your Tempo or Alloy configuration. Refer to [Enable service graphs](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/enable-service-graphs/) in the Tempo documentation.

{{< figure src="/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph.png" class="docs-image--no-shadow" max-width="500px" alt="Screenshot of the Service Graph view showing a table of RED metrics alongside the node graph" >}}

For details on the Service Graph view, refer to the [Service Graph view documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-generator/service-graph-view/).

To open the Service Graph view, follow the same steps as [Display the Service Graph](#display-the-service-graph). The table appears alongside the node graph after you run the query.

To open a query in Prometheus with the span name of that row automatically set in the query, click a row in the **rate**, **error rate**, or **duration** columns.

![Linked Prometheus query opened from the Service Graph view rate column](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph-prom.png)

To open a query in Tempo with the span name of that row automatically set in the query, click a row in the **links** column.

You can also use TraceQL to query the same data programmatically. Refer to [TraceQL query examples](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-query-examples/) to translate Service Graph observations into queries.

## Troubleshoot

If the Service Graph isn't displaying data, the table is empty, or you see high cardinality warnings, refer to [Service graph issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/troubleshooting/#service-graph-issues) in the troubleshooting guide.

## Next steps

- [Enable service graphs](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/enable-service-graphs/) - Set up metric generation in Tempo or Alloy.
- [Node graph panel](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/node-graph/) - Navigate and customize the graph layout.

# Span filters

When working with traces, especially those comprising a vast number of spans, pinpointing specific spans of interest can be a daunting task.
This is where span filtering comes in.
Located above the trace timeline viewer, the **Filters** bar lets you refine the spans displayed based on specific criteria.
Whether you're looking to identify spans from a certain service, those exceeding a particular duration, or spans tagged with specific attributes, span filtering streamlines the process.

The more filters you add, the more specific the results become.

{{< admonition type="tip" >}}
If you arrived from the [Service Graph](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/), use **service.name** and **status = error** filters to drill into the specific service nodes showing errors.
{{< /admonition >}}

![The Filters section in the trace timeline viewer showing the filter drop-down, quick filter pills for Critical path, Errors, and High latency, navigation arrows, match count, and Show all spans toggle.](/media/docs/grafana/data-sources/tempo/screenshot-tempo-datasource-span-filters.png)

## Filter types

The **Filters** text box lets you pick a key, choose an operator, and enter or select a value.

Available filter keys include:

- **Text search**: free-text search across span content
- **duration**: filter by span duration (operators: `=`, `>=`, `<=`, `>`, `<`)
- **service.name**: the service that produced the span
- **span.name**: the operation name of the span
- **status**: span status (`ok`, `error`, `unset`)
- **kind**: span kind (for example, `server`, `client`, `internal`)
- Any span tag, process tag, or log field key present in the trace

For most attribute keys, the available operators are `=`, `!=`, `=~` (regular expression match), and `!~` (regular expression not match).

{{< admonition type="note" >}}
The same attributes available as filter keys (for example, `span.name`, `service.name`, `duration`) can be used directly in [TraceQL queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-editor/).
{{< /admonition >}}

## Quick filter pills

Below the text box, three quick-filter pills let you apply common filters with a single click:

| Pill              | Effect                                                                                                                       |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **Critical path** | Highlights spans in the critical path. The longest sequence of dependent tasks that determines the trace's minimum duration. |
| **Errors**        | Selects spans where `status = error`.                                                                                        |
| **High latency**  | Selects the 10% longest spans in the trace (p90 threshold).                                                                  |

Click a pill to toggle the filter on or off.

## Use span filters

The **Filters** bar is always visible above the trace timeline viewer.
To add a filter:

1. Click the **Filters** text box (placeholder text: _Filter by attribute or text_).
1. Select a key from the dropdown (for example, **service.name** or **duration**).
1. Choose an operator (for example, `=` or `>=`).
1. Select or type a value. The text box suggests values extracted from the current trace.

The trace timeline viewer updates as you add filters, highlighting the matching spans.

### Show all spans

By default, the **Show all spans** toggle is enabled, meaning all spans remain visible with matching spans highlighted.
Turn off **Show all spans** to hide non-matching spans and display only the spans that meet your filter criteria.
This is particularly useful when sifting through thousands of spans, allowing you to zero in on those that truly matter.

The toggle is disabled when no filters are active.

### Navigate between matches

Use the up and down arrow buttons next to the match count to jump between matching spans in the trace timeline.
The match count displays the current position and total matches (for example, _2/5 matches_).

### Remove a filter

To remove a filter, click the **×** on the filter chip below the **Filters** text box.

## Common use cases

To find all errors from a specific service, select `service.name` = `checkout`, then click the **Errors** pill.

To isolate slow database calls, select `span.name` =~ `db\..*`, then select `duration` > `200ms`.

To narrow to the critical path only, click the **Critical path** pill, then turn off **Show all spans**.

## Next steps

- [Service Graph](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/): Visualize service relationships and identify error-producing nodes.
- [Query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/): Build TraceQL queries using the same attributes available as span filter keys.
- [Configure the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/): Set up your Tempo data source and trace correlations.

# Additional settings

The **Additional settings** section of the Tempo data source settings contains options that most users don't need to change. These settings are collapsed by default.

Use the down arrow to expand the **Additional settings** section to view these options.

This page covers the following settings:

- **[Advanced HTTP settings](#advanced-http-settings)**: Configure cookie forwarding and request timeout
- **[Service graph](#service-graph)**: Link a Prometheus data source for Service Graph visualization
- **[Node graph](#node-graph)**: Enable the node graph above the trace view
- **[Tempo search](#tempo-search)**: Show or hide the search option in Explore
- **[TraceID query](#traceid-query)**: Narrow TraceID lookups to a time range for performance
- **[Tags time range](#tags-time-range)**: Control the time window for tag and tag-value queries
- **[Tag limit](#tag-limit)**: Set the maximum number of tags retrieved from Tempo
- **[Span bar](#span-bar)**: Display duration or a custom tag in the span bar
- **[Private data source connect](#private-data-source-connect)**: Connect to data sources in private networks

## Advanced HTTP settings

The Grafana Proxy deletes forwarded cookies. Use the **Allowed cookies** field to specify cookies by name that should be forwarded to the data source.

The **Timeout** field sets the HTTP request timeout in seconds.

## Service graph

The **Service graph** setting configures the [Service Graph](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/enable-service-graphs/) data.

You must link a Prometheus data source in the **Data source** setting for the [Service Graph](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/) to display data. Without this, Service Graph queries return empty results.

The linked Prometheus data source must contain metrics generated by [Grafana Alloy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/grafana-alloy/) or the [Tempo metrics generator](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-generator/service_graphs/#tempo). For the full list of prerequisites and troubleshooting steps, refer to [Service Graph and Service Graph view](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/).

{{< admonition type="note" >}}
If you cloned a provisioned data source (for example, the Grafana Cloud Traces data source) and the **Service graph** section doesn't appear in the cloned data source's settings, verify that the cloned data source has the same Tempo backend URL and authentication configured. The Service graph section only appears when the data source can connect to Tempo successfully. Refer to [Clone a provisioned data source for Grafana Cloud](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/#clone-a-provisioned-data-source-for-grafana-cloud) for setup steps.
{{< /admonition >}}

## Node graph

The **Node graph** setting enables the [node graph visualization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/node-graph/), which isn't activated by default.

After it's activated, Grafana displays the node graph above the trace view.

## Tempo search

The **Search** setting configures [Tempo search](https://grafana.com/docs/tempo/latest/configuration/#search).

You can configure the **Hide search** setting to hide the search query option in **Explore** if search is not configured in the Tempo instance.

## TraceID query

The **TraceID query** setting modifies how TraceID queries are run.
The time range can be used when there are performance issues or timeouts since it narrows down the search to the defined range.
This setting is disabled by default.

You can configure this setting as follows:

| Name                               | Description                                                 |
| ---------------------------------- | ----------------------------------------------------------- |
| **Use time range in query**        | Use a time range in the TraceID query. Default: `disabled`. |
| **Time shift for start of search** | Time shift for start of search. Default: `30m`.             |
| **Time shift for end of search**   | Time shift for end of search. Default: `30m`.               |

## Tags time range

The **Tags time range** feature controls how tag and tag-value queries are executed by specifying the time window applied to these requests. You can select one of the following options to constrain your queries:

| Name                | Description                        |
| ------------------- | ---------------------------------- |
| **Last 30 minutes** | Last 30 minutes of selected range. |
| **Last 3 hours**    | Last 3 hours of selected range.    |
| **Last 24 hours**   | Last 24 hours of selected range.   |
| **Last 3 days**     | Last 3 days of selected range.     |
| **Last 7 days**     | Last 7 days of selected range.     |

## Tag limit

The **Tag limit** setting modifies the max number of tags and tag values to retrieve from Tempo. Default: 5000

## Span bar

The **Span bar** setting helps you display additional information in the span bar row.

You can choose one of three options:

| Name         | Description                                                                                                                      |
| ------------ | -------------------------------------------------------------------------------------------------------------------------------- |
| **None**     | Adds nothing to the span bar row.                                                                                                |
| **Duration** | _(Default)_ Displays the span duration on the span bar row.                                                                      |
| **Tag**      | Displays the span tag on the span bar row. You must also specify which tag key to use to get the tag value, such as `component`. |

## Private data source connect

[//]: # 'Shared content for authentication section procedure in data sources'

{{< docs/shared source="grafana" lookup="datasources/datasouce-private-ds-connect.md" leveloffset="+1" version="<GRAFANA_VERSION>" >}}

## Next steps

- [Configure the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/): Return to connection, authentication, and streaming settings.
- [Provision the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/): Configure the Tempo data source using a YAML file.
- [Service Graph and Service Graph view](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/): Visualize service dependencies and monitor RED metrics.

# Configure trace to logs correlation

Trace to logs correlation lets you navigate from a span in Tempo directly to the matching logs in Loki, and from a log line in Loki back to its trace in Tempo.
When configured, a **Logs for this span** link appears in the trace view and a link to the configured tracing data source appears on log lines that contain a trace ID.

This is a two data source configuration:

- **Tempo data source:** Controls how Grafana queries logs when you click a span, including the time window, query, and tags to match.
- **Loki data source:** Controls how trace IDs are extracted from log lines and linked back to traces.

Both sides must be configured for the correlation to work in either direction.

![Trace to logs settings](/media/docs/grafana/data-sources/tempo/tempo-data-source-trace-to-logs.png)

## Before you begin

To configure trace to logs correlation, you need:

- A [Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/) configured in Grafana
- A [Loki data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/) configured in Grafana
- Applications that emit traces and logs with a shared identifier such as a trace ID, service name, or both
- Editor or Admin permissions in Grafana

{{< admonition type="note" >}}
You can't modify a provisioned data source from the Grafana UI. If you're using Grafana Cloud Traces (the pre-configured tracing data source in Grafana Cloud), its settings are read-only.
To configure trace to logs, [clone the data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/#clone-a-provisioned-data-source-for-grafana-cloud) to create an editable copy, or update the provisioning file for self-managed instances.
Refer to [Provision trace to logs settings](#provision-trace-to-logs-settings) for details.
{{< /admonition >}}

## Configure the Tempo data source

These settings control how Grafana queries your log data source when you click a span in the trace view.
This guide uses Loki, but Trace to logs also supports Elasticsearch, Splunk, OpenSearch, Falcon LogScale, Google Cloud Logging, and VictoriaMetrics Logs.

{{< admonition type="note" >}}
To change the target log data source, for example from Elasticsearch to Loki, select the new data source from the **Data source** drop-down in the following steps and update the tags and custom query to match its query language. The reverse direction, from a log line back to a trace, is configured on the log data source itself, and the mechanism differs by type: Loki uses [Derived fields](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/#derived-fields), while Elasticsearch and Splunk use [Data links](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/elasticsearch/configure/#data-links).
{{< /admonition >}}

1. Go to **Connections** > **Data sources** and select your Tempo data source.
1. Scroll to the **Trace to logs** section.
1. Select your Loki data source from the **Data source** drop-down list.
1. Configure the **Span start time shift** and **Span end time shift** fields.

   These fields expand the log query time window around the span's start and end timestamps.
   A span's timestamps are precise to the millisecond, but log lines are often written slightly before or after the span boundary.
   The default value of `0` can return no logs if timestamps don't align exactly.
   A common starting point is `-2s` for start shift and `2s` for end shift.

   You can increase this value for spans that represent slow or batch operations, for example `5s`.

1. Configure **Tags** to map span attributes to Loki label names.

   Grafana uses these tags to filter logs to the correct service, Pod, or cluster.
   The Loki label name you enter must exactly match the label name on your Loki streams.
   A mismatch silently breaks the correlation with no error shown.

   If you don't configure any tags, Grafana uses the following defaults: `cluster`, `hostname`, `namespace`, `pod`, `service.name` (remapped to `service_name`), and `service.namespace` (remapped to `service_namespace`).
   The automatic dot-to-underscore remapping only applies to these default tags. For custom tags, you must specify the remapped name explicitly.

   To customize the mapping, use the **Add tag** button to add each row.
   The following table shows common tags and their typical Loki label equivalents:

   | Span attribute      | Loki label          |
   | ------------------- | ------------------- |
   | `service.name`      | `service_name`      |
   | `service.namespace` | `service_namespace` |
   | `cluster`           | `cluster`           |
   | `namespace`         | `namespace`         |
   | `pod`               | `pod`               |
   | `hostname`          | `hostname`          |

   Choose labels with low cardinality.
   In Loki, every unique combination of label values creates a separate stream.
   Labels that take on many distinct values, such as `pod`, `host`, `thread`, `duration`, `traceId`, or `spanId`, can create hundreds of thousands or even millions of streams, causing slow queries, high memory usage, and log loss at ingest.
   The labels listed above (`service_name`, `namespace`, `cluster`) are good choices because they have a bounded, small set of values.
   If you need to query high-cardinality values like trace IDs or Pod names, store them as [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) instead. They are queryable without creating additional streams.
   For more information, refer to [Cardinality](https://grafana.com/docs/loki/latest/get-started/labels/cardinality/).

   {{< admonition type="note" >}}
   When using Loki, the **Logs for this span** link only appears on a span if at least one of the configured tags exists on that span.
   If none of the tags match, the link doesn't appear and no error is shown.
   {{< /admonition >}}

1. Configure the **Filter by trace ID** and **Filter by span ID** toggles.
   - **Filter by trace ID:** Filters Loki results to all logs for the entire trace, not just the selected span. This works well for most use cases.
   - **Filter by span ID:** Filters to logs for a single span. This only works if your logs include a span ID field.

   These toggles are disabled when **Use custom query** is enabled because the custom query controls filtering directly.

   Clicking **Logs for this span** filters by trace ID, not span ID, so results include all logs for the entire trace that contain a matching trace ID, not only logs emitted during that specific span.
   If your results look broader than expected, this is expected behavior.
   To narrow results to a specific span's activity, write a custom LogQL query that filters on span ID or span-specific attributes if your application logs them, for example `{service_name="my-service"} | json | spanId=""`.
   For this feature to work at all, your application must inject trace IDs into log lines, for example, using OpenTelemetry SDK structured logging.
   Without trace IDs in logs, the query falls back to a time-range filter.

1. Optional: Enable **Use custom query** and write a LogQL expression to replace the auto-generated query.

   Use `${__tags}` to inject the mapped tag filters automatically:

   ```logql
   {${__tags}} | logfmt | trace_id=`${__trace.traceId}`
   ```

   If a tag like `pod` is stored as Loki [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) instead of an indexed label, it can't appear in the stream selector `{}`. Move it to a pipeline filter instead:

   ```logql
   {${__tags}} | pod=`${__span.tags["k8s.pod.name"]}` |= `${__trace.traceId}`
   ```

   Refer to [Custom query variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#custom-query-variables) for the full list of available variables.

1. Click **Save & test**.

## Configure the Loki data source

These settings enable the **Tempo** link on Loki log lines, letting you navigate from a log entry to its trace.

{{< admonition type="note" >}}
Configuring the Tempo data source without also configuring the Loki data source is a common mistake.
If you skip this step, clicking a log line doesn't open a trace.
Both sides are required.
{{< /admonition >}}

1. Go to **Connections** > **Data sources** and select your Loki data source.
1. In the **Additional settings** section, scroll to **Derived fields**.
1. Click **Add** to add a new derived field.
1. Enter a **Name** for the field, for example `TraceID`.
1. Select the **Type** and enter a pattern to extract the trace ID from log lines.

   If you select **Regex in log line**, enter a regular expression with one capture group.
   The following table shows common patterns:

   | Log format                        | Regex               |
   | --------------------------------- | ------------------- |
   | `traceID=<VALUE>`                 | `traceID=(\w+)`     |
   | `trace_id=<VALUE>`                | `trace_id=(\w+)`    |
   | JSON field `"traceId": "<VALUE>"` | `"traceId":"(\w+)"` |

   If you select **Label**, enter a regular expression that matches the label key, for example `trace[_]?id` to match both `traceid` and `trace_id`.

   {{< admonition type="tip" >}}
   If your applications use different trace ID field names, you need a separate derived field entry for each format.
   To simplify this, standardize on OpenTelemetry structured metadata so trace IDs are stored as a consistent label.
   {{< /admonition >}}

1. Enable the **Internal link** toggle and select your Tempo data source from the drop-down list.
1. In the **Query** field, enter `${__value.raw}`.

   This passes the extracted trace ID to Tempo when the link is clicked.
   The field shows `${__value.raw}` as placeholder text, but you must enter it as the actual value.

1. Click **Save & test**.

For more details about derived field options, refer to [Derived fields](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/#derived-fields) in the Loki data source documentation.

## Provision trace to logs settings

You can provision the trace to logs configuration using the `tracesToLogsV2` block in your data source YAML file.
For the full provisioning YAML example including all Tempo settings, refer to [Provision the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/).
For general provisioning instructions, refer to [Provisioning data sources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

## Example: NGINX service

This example assumes:

- Applications running in Kubernetes that emit OpenTelemetry traces to Tempo
- Logs sent to Loki with labels `service_name`, `namespace`, and `pod`
- Trace IDs written to logs as `trace_id=<VALUE>`

The Tempo data source settings are:

| Field                     | Value                                                       |
| ------------------------- | ----------------------------------------------------------- |
| **Data source**           | Loki                                                        |
| **Span start time shift** | `-2s`                                                       |
| **Span end time shift**   | `2s`                                                        |
| **Tags**                  | `service.name` mapped to `service_name`, `namespace`, `pod` |
| **Filter by trace ID**    | Enabled                                                     |
| **Filter by span ID**     | Disabled                                                    |

The Loki data source derived field settings are:

| Field             | Value                 |
| ----------------- | --------------------- |
| **Name**          | TraceID               |
| **Type**          | Regex in log line     |
| **Regex**         | `trace_id=(\w+)`      |
| **Internal link** | Enabled, select Tempo |
| **Query**         | `${__value.raw}`      |

### Expected result

After configuring both data sources:

- In **Explore** with the Tempo data source selected, click any span and then click **Logs for this span**. Loki opens filtered to `service_name=nginx` and `namespace=production` within the configured time window around the span.
- In **Explore** with the Loki data source selected, any log line containing `trace_id=...` shows a link to your Tempo data source. Clicking it opens the full trace in Tempo.

## Troubleshooting

If trace to logs links aren't appearing, return no data, or only work for some services, refer to [Trace to logs/metrics/profiles issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/troubleshooting/#trace-to-logsmetricsprofiles-issues) in the troubleshooting guide.

If the configuration fields are greyed out, your data source is provisioned. Refer to [Provision trace to logs settings](#provision-trace-to-logs-settings) for options.

## Next steps

- [Configure trace to metrics correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/): Link spans to metrics queries in Prometheus.
- [Configure trace to profiles correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/): Link spans to profiling data in Grafana Pyroscope.
- [Configure the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/): Return to connection, authentication, and streaming settings.

# Configure trace to metrics correlation

Trace to metrics lets you navigate from a span in a trace to metrics in a Prometheus or other metrics data source.
When configured, a **Metrics for this span** link appears in the trace view.

Trace to metrics works with any Prometheus-compatible data source and doesn't require the Tempo metrics generator.
The metrics generator creates _new_ metrics from trace data; trace to metrics links to metrics that _already exist_ in your metrics data source.

{{< youtube id="TkapvLeMMpc" >}}

## Before you begin

To configure trace to metrics, you need:

- A [Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/) configured in Grafana
- A Prometheus-compatible metrics data source configured in Grafana
- Metrics that correspond to your traced services (for example, `requests_total`, `request_duration_seconds`)
- Editor or Admin permissions in Grafana

{{< admonition type="note" >}}
You can't modify a provisioned data source from the Grafana UI. If you're using Grafana Cloud Traces (the pre-configured tracing data source in Grafana Cloud), its settings are read-only.
To configure trace to metrics, [clone the data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/#clone-a-provisioned-data-source-for-grafana-cloud) to create an editable copy, or update the provisioning file for self-managed instances.
Refer to the [Provisioning](#provisioning) section for details.
{{< /admonition >}}

There are two ways to configure the trace to metrics feature:

- Use a basic configuration with a default query.
- Configure one or more custom queries where you can use a [template language](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/) to interpolate variables from the trace or span.

![Trace to metrics settings in the Tempo data source](/media/docs/grafana/data-sources/tempo/tempo-data-source-trace-to-metrics.png)

## Set up a basic configuration

To use a basic configuration, follow these steps:

1. Select a metrics data source from the **Data source** drop-down.
1. Optional: Change **Span start time shift** and **Span end time shift**. The placeholders show `-2m` (start) and `2m` (end), which are applied if you leave the fields empty.
1. Optional: Choose any tags to use in the query. Click **Add tag** to add a tag mapping.

   The tags you configure must be present in the spans attributes or resources for a trace to metrics span link to appear. You can optionally configure a new name for the tag. This is useful if the tag has dots in the name and the target data source doesn't allow using dots in labels. For example, you can remap `service.name` to `service_name`.

1. Don't select **Add query**.
1. Select **Save & test**.

## Set up custom queries

To use custom queries, you need to configure the tags you'd like to include in the linked queries.
For each tag, the key is the span attribute name.
In cases where the attribute name would result in an invalid metrics query or doesn't exactly match the desired label name, you can enter the label name as the second value.
For example, you could map the attribute `k8s.pod` to the label `pod`.

You can interpolate the configured tags using the `$__tags` keyword.
For example, when you configure the query `requests_total{$__tags}` with the tags `k8s.pod=pod` and `cluster`, it results in `requests_total{pod="nginx-554b9", cluster="us-east-1"}`.
The label values are dynamically inserted based on the span attributes' values.

You can link to any metric you'd like, but metrics for span durations, counts, and errors filtered by service or span are a great starting point.

To use custom queries with the configuration, follow these steps:

1. Select a metrics data source from the **Data source** drop-down.
1. Optional: Choose any tags to use in the query. Click **Add tag** to add a tag mapping.

   These tags can be used in the custom query with the `${__tags}` variable. This variable interpolates the mapped tags as a list in an appropriate syntax for the data source and only includes the tags that were present in the span, omitting those that weren't present. You can optionally configure a new name for the tag. This is useful in cases where the tag has dots in the name and the target data source doesn't allow using dots in labels. For example, you can remap `service.name` to `service_name`. If you don't map any tags here, you can still use any tag in the query like this: `method="${__span.tags.method}"`. Refer to [Custom query variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#custom-query-variables) for the full list of available variables.

1. Click **Add query** to add a custom query.
1. Specify a custom query to be used to query metrics data.

   Each linked query consists of:
   - **Link Label:** _(Optional)_ Descriptive label for the linked query.
   - **Query:** The query ran when navigating from a trace to the metrics data source.
     Interpolate tags using the `$__tags` keyword.
     For example, when you configure the query `requests_total{$__tags}` with the tags `k8s.pod=pod` and `cluster`, the result looks like `requests_total{pod="nginx-554b9", cluster="us-east-1"}`.

1. Select **Save & test**.

## Configuration options

The following table describes options for configuring the **Trace to metrics** settings:

| Setting name              | Description                                                                                                                                                                                                                                                     |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Data source**           | Defines the target data source.                                                                                                                                                                                                                                 |
| **Span start time shift** | Shifts the start time for the metrics query, based on the span's start time. You can use time units, such as `5s`, `1m`, `3h`. To extend the time to the past, use a negative value. Default: `-2m`.                                                            |
| **Span end time shift**   | Shifts the end time for the metrics query, based on the span's end time. You can use time units. Default: `2m`.                                                                                                                                                 |
| **Tags**                  | Defines the tags used in linked queries. The key sets the span attribute name, and the optional value sets the corresponding metric label name. For example, you can map `k8s.pod` to `pod`. To interpolate these tags into queries, use the `$__tags` keyword. |
| **Link Label**            | _(Optional)_ Descriptive label for the linked query.                                                                                                                                                                                                            |
| **Query**                 | Input to write a custom query. Use variable interpolation to customize it with variables from span.                                                                                                                                                             |

## Provisioning

You can provision the trace to metrics configuration using the `tracesToMetrics` block in your data source YAML file.
For the full provisioning YAML example including all Tempo settings, refer to [Provision the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/).

## Link from metrics to traces

To navigate in the reverse direction, from a metric to its associated trace, configure exemplars in your Prometheus data source. Refer to [Exemplars](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/) for setup instructions.

## Troubleshooting

If trace to metrics links aren't appearing or return no data, refer to [Trace to logs/metrics/profiles issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/troubleshooting/#trace-to-logsmetricsprofiles-issues) in the troubleshooting guide.

If the configuration fields are greyed out, your data source is provisioned. Refer to the [Provisioning](#provisioning) section for how to update the configuration via YAML.

## Next steps

- [Configure trace to logs correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/): Navigate from spans to related logs in Loki.
- [Configure trace to profiles correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/): Link spans to profiling data in Grafana Pyroscope.
- [Provision the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/): Configure the Tempo data source using a YAML file.

# Configure trace to profiles correlation

Trace to profiles lets you navigate from a span in a trace directly to profiling data in [Grafana Pyroscope](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/). When configured, a **Profiles for this span** button and an embedded flame graph appear in the span details view.

## Before you begin

To configure trace to profiles, you need:

- A [Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/) configured in Grafana
- A [Grafana Pyroscope data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/grafana-pyroscope/) configured in Grafana
- An application instrumented with the [OpenTelemetry span profiling bridge](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/configure-client/trace-span-profiles/) so that spans include the `pyroscope.profile.id` attribute
- Editor or Admin permissions in Grafana

{{< admonition type="note" >}}
You can't modify a provisioned data source from the Grafana UI. If you're using Grafana Cloud Traces (the pre-configured tracing data source in Grafana Cloud), its settings are read-only.
To configure trace to profiles, [clone the data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/#clone-a-provisioned-data-source-for-grafana-cloud) to create an editable copy, or update the provisioning file for self-managed instances.
Refer to the [Provisioning](#provisioning) section for details.
{{< /admonition >}}

[//]: # 'Shared content for Trace to profiles in the Tempo data source'

{{< docs/shared source="grafana" lookup="datasources/tempo-traces-to-profiles.md" leveloffset="+1" version="<GRAFANA_VERSION>" >}}

## Provisioning

You can provision the trace to profiles configuration using the `tracesToProfiles` block in your data source YAML file.
For the full provisioning YAML example including all Tempo settings, refer to [Provision the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/).

## Verify the integration

After configuring trace to profiles, verify the integration:

1. Open **Explore** and select your Tempo data source.
1. Run a query to load a trace.
1. Expand a span and confirm that the **Profiles for this span** button appears in the span details.
1. Click the button and verify that an embedded flame graph loads with profiling data.

## Troubleshooting

If trace to profiles links aren't appearing or the flame graph is empty, refer to [Trace to logs/metrics/profiles issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/troubleshooting/#trace-to-logsmetricsprofiles-issues) in the troubleshooting guide. For additional diagnosis steps, refer to [Troubleshoot trace to profiles](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/configure-client/trace-span-profiles/) in the Pyroscope documentation.

If the configuration fields are greyed out, your data source is provisioned. Refer to the [Provisioning](#provisioning) section for how to update the configuration via YAML.

## Next steps

- [Configure trace to logs correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/): Navigate from spans to related logs in Loki.
- [Configure trace to metrics correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/): Link spans to metrics queries in Prometheus.
- [Provision the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/): Configure the Tempo data source using a YAML file.

# Provision the Tempo data source

You can define and configure the Tempo data source in YAML files as part of the Grafana provisioning system.

You can use version control, like Git, to track and manage file changes.
Changes can be updated or rolled back as needed.

{{< admonition type="note" >}}
Provisioning via YAML is primarily used for self-managed Grafana instances.
{{< /admonition >}}

For more information about provisioning and available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

## Provisioned data sources

You can't modify a provisioned data source from the Grafana UI. The settings form is read-only and the **Save & test** button is replaced by **Test** (you can test the connection but not save changes).

To make changes, you can either:

- **Clone the data source:** Create a new data source of the same type and copy the settings from the provisioned data source. Refer to [Clone a provisioned data source for Grafana Cloud](#clone-a-provisioned-data-source-for-grafana-cloud) for detailed steps.
- **Update the provisioning file:** Edit the YAML configuration file and restart Grafana, or wait for the provisioning system to pick up the change. Any setting, including span time shifts on trace to logs, can be changed by editing the YAML.

## Example file

This example provisioning YAML file sets up the equivalents of the options available in the Tempo data source UI.
Replace `grafana-pyroscope-datasource` with the actual UID of your Pyroscope data source, and verify the other `datasource Uid` values match what's actually provisioned.

```yaml
apiVersion: 1

datasources:
  - name: Tempo
    type: tempo
    uid: EbPG8fYoz
    url: http://localhost:3200
    access: proxy
    basicAuth: false
    jsonData:
      tracesToLogsV2:
        # Field with an internal link pointing to a logs data source in Grafana.
        # datasourceUid value must match the uid value of the logs data source.
        datasourceUid: 'loki'
        spanStartTimeShift: '-1h'
        spanEndTimeShift: '1h'
        tags: [{ key: 'job' }, { key: 'instance' }, { key: 'pod' }, { key: 'namespace' }]
        filterByTraceID: false
        filterBySpanID: false
        customQuery: true
        query: 'method="$${__span.tags.method}"'
      tracesToMetrics:
        datasourceUid: 'prom'
        spanStartTimeShift: '-1h'
        spanEndTimeShift: '1h'
        tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }]
        queries:
          - name: 'Sample query'
            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))'
      tracesToProfiles:
        datasourceUid: 'grafana-pyroscope-datasource'
        tags: [{ key: 'job' }, { key: 'instance' }, { key: 'pod' }, { key: 'namespace' }]
        profileTypeId: 'process_cpu:cpu:nanoseconds:cpu:nanoseconds'
        customQuery: true
        query: 'method="$${__span.tags.method}"'
      serviceMap:
        datasourceUid: 'prometheus'
      nodeGraph:
        enabled: true
      search:
        hide: false
      traceQuery:
        timeShiftEnabled: true
        spanStartTimeShift: '-1h'
        spanEndTimeShift: '1h'
      spanBar:
        type: 'Tag'
        tag: 'http.path'
      streamingEnabled:
        search: true
        metrics: true
```

For details about individual settings, refer to:

- [Configure trace to logs correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/): `tracesToLogsV2` block
- [Configure trace to metrics correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/): `tracesToMetrics` block
- [Configure trace to profiles correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/): `tracesToProfiles` block
- [Additional settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/additional-settings/): `serviceMap`, `nodeGraph`, `search`, `traceQuery`, `spanBar` blocks

## Clone a provisioned data source for Grafana Cloud

If you have a data source that is provisioned by a configuration file in Grafana Cloud, you can clone that provisioned data source and then edit the new data source in the Grafana UI.

For example, if you want to edit the trace to logs settings in your Tempo data source that is provisioned on Grafana Cloud, you can enable traceID and spanID filtering by cloning the data source.

To clone a provisioned data source, follow these steps:

1. Create a viewer [Cloud Access Policy token](https://grafana.com/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-policies/) in the Grafana Cloud Portal, making sure it has read permissions at least for the data types you are trying to clone.
1. [Create a new data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#add-a-new-data-source) of the same type you want to clone.
1. Copy all of the settings from the existing provisioned data source into the new data source while replacing the password with the API key you created.

   The easiest way to do this is to open separate browser windows with the provisioned data source in one and the newly created data source in another.

After copying the HTTP and Auth section details, pasting the Cloud Access Policy token into the Password field, and changing any of the other options that you want, you can save and test the data source.

## Next steps

- [Configure the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/): Return to connection, authentication, and streaming settings.
- [Configure trace to logs correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/): Set up span-to-log navigation.
- [Additional settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/additional-settings/): Configure Service graph, node graph, search, and other settings.

# Trace correlations

You can use Grafana [correlations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/correlations/) to embed interactive correlation links in your trace view to jump from spans to related logs, metrics, profiles, or external systems. This guide explains how to configure and manage Trace correlations in Grafana.

## What are trace correlations?

Trace correlations let you define rules that inject context-sensitive links into your trace spans. When viewing traces in Explore or the Traces panel, users can click these links to navigate directly to relevant queries or URLs. Correlations are similar but more flexible to the [trace to logs, metrics, and profiles links you can configure for the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/).

{{< figure src="/media/docs/tempo/screenshot-trace-view-correlations.png" max-width="900px" class="docs-image--no-shadow" alt="Using correlations for a trace" >}}

## Before you begin

To use trace correlations, you need:

- Grafana 12 or later
- A [Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/) configured in Grafana
- Admin access to configuration settings or provisioning files in Grafana

## Set up a trace correlation

1. Log in to Grafana with an administrator account.

1. Go to **Administration** > **Plugins and data** > **Correlations**.

1. Select **Add correlation** or **Add new**.

1. On step 1, provide a **label** for the correlation, and an optional **description**.

1. On step 2, configure the correlation **target**.
   - Select the **Type** drop-down list and choose **Query** to link to another data source or choose **External** for a custom URL.

   - For a query **Target**, select the target drop-down list and select the data source that should be queried when the link is clicked. Define the target query.

   - For an external **Target**, enter the **External URL**.

   - For both query and external targets, you can use the following variables based on trace data. Object variables must be parsed into a value variable with a regular expression transformation.

   | Variable       | Type   | Description            |
   | -------------- | ------ | ---------------------- |
   | `traceId`      | String | Trace identifier       |
   | `spanID`       | String | Span identifier        |
   | `parentSpanID` | String | Parent span identifier |
   | `serviceName`  | String | Service name           |
   | `serviceTags`  | Object | Resource attributes    |
   | `tags`         | Object | Span attributes        |
   | `logs`         | Object | Trace events           |
   | `references`   | Object | Trace links            |

   {{< figure src="/media/docs/tempo/screenshot-grafana-trace-correlations-loki-step-2.png" max-width="900px" class="docs-image--no-shadow" alt="Setting up a correlation for a Loki target using trace variables" >}}

1. On step 3, configure the correlation data source:
   - Select your Tempo data source in the **Source** drop-down list.

   - Enter the trace data variable you use for the correlation in the **Results field**.

   - Optionally, add one or more **Transformations** to parse the trace data into additional variables. You can use these variables to configure the correlation **Target**.

   {{< figure src="/media/docs/tempo/screenshot-grafana-trace-correlations-loki-step-3.png" max-width="900px" class="docs-image--no-shadow" alt="Setting up a correlation for a Loki data source" >}}

1. Select **Save** to save the correlation.

## Verify correlations in Explore

1. Open **Explore** and select your Tempo tracing source.

1. Run a query to load spans.

1. Hover over the span links menu or open the span details to reveal the correlation link buttons.

   {{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-correlations.png" max-width="900px" class="docs-image--no-shadow" alt="Using correlations for a trace" >}}

1. Click a correlation link to open a split view or navigate to your target system or query.

## Examples

Below are several practical correlation configurations to get you started.

### Example 1: Trace to logs by service name and trace identifier

In this example, you configure trace to logs by service name and a trace identifier.

1. On step 1, add a new correlation with the label **Logs for this service and trace** and an optional description.

   {{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-correlations-example-1-step-1.png" max-width="900px" class="docs-image--no-shadow" alt="Using correlations for a trace" >}}

1. On step 2, configure the correlation target:
   - Select the target type **Query** and select your Loki data source as **Target**.

   - Define the Loki query, using `serviceName` and `traceID` as variables derived from the span data:

     ```
     {service_name="$serviceName"} | trace_id=`$traceID` |= ``
     ```

     In this query, `service_name` is the only [stream label](https://grafana.com/docs/loki/latest/get-started/labels/) used inside `{}`. Stream labels should be low-cardinality values that describe the source of your logs.
     The `trace_id` field appears after the `|` pipe as a [pipeline filter](https://grafana.com/docs/loki/latest/query/log_queries/#line-filter-expression), which searches log content or [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) without creating additional streams.
     Don't use trace IDs or other high-cardinality values as stream labels because each unique value creates a separate stream, which degrades Loki performance.
     For more information, refer to [Label best practices](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/) and [Cardinality](https://grafana.com/docs/loki/latest/get-started/labels/cardinality/).

     {{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-correlations-example-1-step-2.png" max-width="900px" class="docs-image--no-shadow" alt="Using correlations for a trace" >}}

1. On step 3, configure the correlation source:
   - Select your Tempo data source as **Source**.

   - Use `traceID` as **Results field**.

   - Add a new transformation to extract the `serviceName` from the span `serviceTags` using the regular expression:

     ```
     {(?=[^\}]*\bkey":"service.name")[^\}]*\bvalue":"(.*?)".*}
     ```

   {{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-correlations-example-1-step-3.png" max-width="900px" class="docs-image--no-shadow" alt="Using correlations for a trace" >}}

1. Save the correlation.

### Example 2: Trace to custom URL

In this example, you configure trace correlations with a custom URL.

1. On step 1, add a new correlation with the label **Open custom URL** and an optional description.

1. On step 2, configure the correlation target:
   - Select the target type **External**.

   - Define your target URL, using variables derived from the span data. In this example, we are using `serviceName` and `traceID`.

     ```
     https://my-server.example.com/service=$serviceName&trace=$traceID
     ```

1. On step 3, configure the correlation source:
   - Select your Tempo data source as **Source**.

   - Use `traceID` as **Results field**.

   - Add a new transformation to extract the `serviceName` from the span `serviceTags` using the regular expression:

     ```
     {(?=[^\}]*\bkey":"service.name")[^\}]*\bvalue":"(.*?)".*}
     ```

1. Save the correlation.

## Best practices

- **Name clearly:** Use descriptive names indicating source and target. For example: **Trace to errors in logs**.

- **Use low-cardinality stream labels:** When targeting Loki, use only low-cardinality values like `service_name`, `namespace`, or `cluster` inside the stream selector `{}`. Place high-cardinality values like trace IDs in [pipeline filters](https://grafana.com/docs/loki/latest/query/log_queries/#line-filter-expression) (after `|`) or store them as [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/). Using trace IDs as stream labels creates excessive streams and degrades Loki performance. For more information, refer to [Cardinality](https://grafana.com/docs/loki/latest/get-started/labels/cardinality/) and [Label best practices](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/). For detailed guidance on mapping span attributes to Loki labels, refer to [Configure trace to logs correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/).

- **Template wisely:** Use multiple `$variable` tokens if you need to inject more than one field.

## Next steps

- [Configure trace to logs correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/): Link spans to log queries in Loki with tag mapping and cardinality guidance.
- [Configure trace to metrics correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/): Link spans to metrics queries in Prometheus.
- [Configure trace to profiles correlation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/): Link spans to profiling data in Grafana Pyroscope.
- [Configure the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/): Return to connection, authentication, and streaming settings.

# Write TraceQL queries with the editor

Use the **TraceQL** editor when you need structural queries across parent and child spans, aggregations, or other features that the [Search query builder](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-search/) doesn't support.
TraceQL queries follow the pattern `{ conditions } | pipeline`.
Refer to [Construct a TraceQL query](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/construct-traceql-queries/) for the full syntax.

To get started, paste this query into the editor and select **Run query**:

```traceql
{ resource.service.name = "frontend" && span:status = error }
```

This returns all error spans from the `frontend` service.
Replace `frontend` with your service name.
For more examples, refer to [TraceQL query examples](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-query-examples/).

If queries return no results, check that your [Tempo data source is configured and connected](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/).

[//]: # 'Shared content for the TraceQL query editor'
[//]: # 'This content is located in /docs/sources/shared/datasources/tempo-editor-traceql.md'

{{< docs/shared source="grafana" lookup="datasources/tempo-editor-traceql.md" version="<GRAFANA_VERSION>" >}}

## Next steps

- [TraceQL query examples](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-query-examples/): Copy-paste query examples for common use cases
- [Search traces using the query builder](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-search/): Build queries visually
- [Construct a TraceQL query](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/construct-traceql-queries/): Full TraceQL syntax reference
- [Service Graph and Service Graph view](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/): Visualize service dependencies
- [Span filters](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/span-filters/): Refine results in the trace detail view

# TraceQL query examples

Use these TraceQL queries in Grafana **Explore** with a Tempo data source.
Adjust service names, routes, and attribute values to match your environment.

These examples use [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/general/trace/).
If your instrumentation uses different attribute names, substitute them in the queries.

For the full TraceQL syntax, refer to [Construct a TraceQL query](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/construct-traceql-queries/).
For an extended recipe collection, refer to the [TraceQL cookbook for Grafana Cloud Traces](https://grafana.com/docs/grafana-cloud/send-data/traces/traces-query-editor/traceql-cookbook/).

{{< admonition type="tip" >}}
If a query returns no results, widen the time range and confirm the correct Tempo data source is selected.
Prefer trace-level intrinsic fields like `trace:duration` and `trace:rootService` for faster queries.
{{< /admonition >}}

## Find error spans

Filter by error status or HTTP error codes:

```traceql
{ status = error }
```

```traceql
{ span.http.status_code >= 500 }
```

```traceql
{ span.http.status_code > 399 }
```

## Find slow requests

Filter by span duration.
Common thresholds are 1 second and 5 seconds:

```traceql
{ duration > 1s }
```

```traceql
{ duration > 5s }
```

Filter a specific endpoint by duration:

```traceql
{ span.http.url = "/api/checkout" && duration > 2s }
```

## Find exceptions by type

Query exception events recorded on spans.
Exception data uses the `event` scope because OpenTelemetry records exceptions as span events:

```traceql
{ event.exception.message =~ "context cancelled" }
```

```traceql
{ event.exception.type = "NotFoundException" }
```

Find error spans that have an exception message:

```traceql
{ status = error && event.exception.message != "" }
```

## Filter by service

Select spans from a specific service or match services by pattern.
You can use the [Service Graph view](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/) to identify which services have high error rates or latency, then query those services with TraceQL.

```traceql
{ resource.service.name = "checkout" }
```

```traceql
{ resource.service.name =~ "payment.*" && status = error }
```

## Find errors and slow spans together

Combine error status and duration filters to identify likely root causes:

```traceql
{ resource.service.name = "api" && status = error && duration > 1s }
```

## Analyze service dependencies

Find errors on a specific API path:

```traceql
{ span.http.url =~ ".*/api/.*" && span.http.status_code >= 500 }
```

Use the descendant operator (`>>`) to find traces where a frontend service calls a downstream service that errors:

```traceql
{ resource.service.name = "frontend" } >> { resource.service.name = "database" && status = error }
```

## Query rate and count metrics

Run these in **Metrics** mode in Explore.
TraceQL metrics queries have a default 24-hour time-range limit:

```traceql
{ } | rate()
```

```traceql
{ status = error } | rate()
```

```traceql
{ resource.service.name = "api" } | count_over_time()
```

```traceql
{ duration > 1s } | rate()
```

## Filter by region or environment

Narrow results to a specific cloud region or deployment environment:

```traceql
{ resource.cloud.region = "us-east-1" && status = error }
```

```traceql
{ resource.deployment.environment = "production" && duration > 2s }
```

## Use dashboard variables

Replace hard-coded values with [dashboard variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) when using these queries in panels.
For example, using a `$service` variable:

```traceql
{ resource.service.name = "$service" && status = error }
```

## More resources

- [TraceQL cookbook for Grafana Cloud Traces](https://grafana.com/docs/grafana-cloud/send-data/traces/traces-query-editor/traceql-cookbook/) - Extended recipe collection with additional examples
- [Construct a TraceQL query](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/construct-traceql-queries/) - Full TraceQL syntax, scopes, and operators
- [Query tracing data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/) - Query editor modes and options
- [Service Graph and Service Graph view](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/) - Visualize service dependencies

# Search traces using the query builder

The **Search** query builder lets you create TraceQL queries using drop-down lists and text fields instead of writing syntax directly.
Each selection you make generates a TraceQL query behind the scenes.
You can view the generated query and copy it to the [TraceQL editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-editor/) at any time.

Use Search when you're exploring data or learning TraceQL patterns.
For complex queries involving structural operators, aggregations, or features that Search doesn't support, use the [TraceQL editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-editor/).
To learn the full query syntax, refer to [Construct a TraceQL query](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/construct-traceql-queries/).

If queries return no results, check that your [Tempo data source is configured and connected](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/).

## Availability

The Search query builder is available by default in Grafana and Grafana Cloud. No feature toggle or additional configuration is required.

[//]: # 'Shared content for the Search - TraceQL query builder'

{{< docs/shared source="grafana" lookup="datasources/tempo-search-traceql.md" leveloffset="+1" version="<GRAFANA_VERSION>" >}}

## Try these common searches

The following examples show what to select in the Search builder and the TraceQL query each selection generates.
Adjust service names and values to match your environment.

**Find error spans for a service:**

1. Set **Service Name** to your service (for example, `frontend`).
1. Set **Status** to `error`.

Generated query:

```traceql
{ resource.service.name = "frontend" && status = error }
```

**Find slow API calls:**

1. Set **Span Name** to the operation (for example, `POST /api/orders`).
1. Set **Duration** to `>` `500ms`.

Generated query:

```traceql
{ name = "POST /api/orders" && duration > 500ms }
```

**Filter by HTTP status code:**

1. Select **Add tag**, choose **span**, and select `http.response.status_code`.
1. Set the operator to `>=` and the value to `500`.

Generated query:

```traceql
{ span.http.response.status_code >= 500 }
```

For more query examples, refer to [TraceQL query examples](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-query-examples/).

## Next steps

- [TraceQL query examples](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-query-examples/): Copy-paste query examples for common use cases
- [Write TraceQL queries with the editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-editor/): For complex queries the Search builder doesn't support
- [Construct a TraceQL query](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/construct-traceql-queries/): Full TraceQL syntax reference
- [Service Graph and Service Graph view](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/): Visualize service dependencies
- [Span filters](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/span-filters/): Refine results in the trace detail view

# Troubleshoot Tempo data source issues

This document provides solutions to common issues you may encounter when configuring or using the Tempo data source in Grafana.

## Scope of this guide

This guide covers issues related to connecting Grafana to Tempo and using the data source features. It applies to the following setups:

- **Self-managed Grafana + self-managed Tempo:** you manage both Grafana (OSS or Enterprise) and Tempo.
- **Grafana Cloud + Cloud Traces:** you use Grafana Cloud with the managed Tempo backend (Grafana Cloud Traces). Some configuration (streaming, metrics generator) is handled automatically.
- **Grafana Cloud + self-managed Tempo via PDC:** you use Grafana Cloud but connect to your own Tempo instance through [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/).

Where troubleshooting steps differ between these setups, the guide calls it out. Sections labeled _Grafana Cloud only_ or _self-managed Tempo_ apply only to those environments.

{{< admonition type="note" >}}
The error messages in this guide are representative examples. Recent versions of the Tempo data source return clearer, more descriptive messages that add context, such as the query time range or the specific reason a request failed, so the exact wording you see might differ from these examples. Match on the general error rather than the exact string.
{{< /admonition >}}

### Resources for troubleshooting Tempo

For issues with Tempo itself (not the data source), refer to the Tempo product documentation:

- [Troubleshoot Tempo](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/) - General Tempo troubleshooting, including ingestion failures and server-side errors.
- [Unable to find traces](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/querying/unable-to-see-trace/) - Traces missing due to ingestion, sampling, or retention issues.
- [Too many requests error](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/querying/too-many-requests-error/) - Query capacity limits and 429 errors.
- [Query issues](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/querying/) - Server-side query failures, bad blocks, and performance tuning.

### Resources for troubleshooting traces in Grafana Cloud

Additional resources for Grafana Cloud:

- [Troubleshoot Grafana Cloud Traces](https://grafana.com/docs/grafana-cloud/send-data/traces/troubleshoot/), which covers quick checks, ingestion issues, TraceQL and search, service graph, exemplars, and rate limiting and retry.
- [Investigate traces with Grafana Assistant](https://grafana.com/docs/grafana-cloud/send-data/traces/investigate-traces-with-assistant/) - Use Grafana Assistant to help troubleshoot any issues.
- [Troubleshoot traces collection with Alloy](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/traces-with-alloy/#troubleshoot)
- [Troubleshoot errors with metrics-generator in Cloud Traces](https://grafana.com/docs/grafana-cloud/send-data/traces/configure/metrics-generator/#troubleshoot-errors)

## Connection errors

These errors occur when Grafana cannot establish or maintain a connection to the Tempo instance.

### Failed to connect to Tempo

**Error message:** `Failed to connect to Tempo` or `dial tcp: connection refused`

**Cause:** Grafana can't reach the Tempo instance. This typically means the Tempo process isn't running, the URL in the data source settings is wrong, or a network-level rule is blocking the connection.

**Solution:**

1. Verify that the Tempo instance is running and accessible from the Grafana server.
1. Check that the URL is correct in the [data source configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/). The default Tempo HTTP port is `3200`. Common URL mistakes include:
   - Adding trailing path segments, for example, `/api` or `/tempo`, to self-managed Tempo URLs. The correct format is `http://<HOST>:3200` with no path. The `/tempo` suffix is only used for Grafana Cloud Traces URLs.
   - Using the gRPC port (`9095`) instead of the HTTP port (`3200`). The Tempo data source URL must be the HTTP endpoint. Grafana derives the gRPC connection for streaming from the same URL.
   - Refer to the [Connection](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/) section for URL format examples by environment.
1. Ensure there are no firewall rules blocking the connection between Grafana and Tempo.
1. For Grafana Cloud users connecting to a self-managed Tempo instance, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if Tempo isn't publicly accessible.
1. Verify the protocol (HTTP or HTTPS) matches your Tempo configuration.

### Connection timeout

**Error message:** `Connection timed out` or `context deadline exceeded`

**Cause:** The connection to Tempo timed out before receiving a response. This can happen when network latency is high, Tempo is overloaded, or an intermediary device (load balancer, proxy) terminates the connection before Tempo responds.

**Solution:**

1. Check the network latency between Grafana and Tempo.
1. Verify that Tempo isn't overloaded or experiencing performance issues. If you have [Tempo mixin dashboards](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/monitor/) installed, check them for resource saturation or elevated error rates.
1. Increase the **Timeout** setting in the [data source configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/additional-settings/) under **Additional settings**.
1. Check if any network devices (load balancers, proxies) are timing out the connection.
1. For large trace queries, consider reducing the time range or adding more specific filters.

### TLS/SSL connection failures

**Error message:** `TLS handshake failed` or `x509: certificate signed by unknown authority`

**Cause:** There's a mismatch between the TLS settings in Grafana and what the Tempo instance supports or requires. For example, Tempo may present a self-signed certificate that the Grafana server doesn't trust, or the certificate's hostname doesn't match the URL.

**Solution:**

1. Verify that Tempo has a [valid TLS certificate if HTTPS is enabled](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#authentication).
1. Check that the certificate is trusted by the Grafana server.
1. If using a self-signed certificate, configure the **TLS/SSL Auth Details** in the [data source settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#authentication).
1. Ensure the certificate's Common Name (CN) or Subject Alternative Name (SAN) matches the hostname used in the URL.

For additional information on setting up TLS encryption with Tempo, refer to [Configure TLS communication](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/network/tls/).

## Authentication errors

These errors occur when there are issues with authentication credentials or permissions.

### Authentication failed

**Error message:** `401 Unauthorized` or `403 Forbidden`

**Cause:** The authentication credentials are invalid or the user doesn't have permission to access Tempo. This error appears when you select **Save & test** or when running a query.

**Solution:**

1. Verify that the authentication credentials (username/password or API key) are correct in the [data source settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#authentication).
1. Check that the user or service account has the required permissions.
1. Ensure the authentication method selected in the data source matches what Tempo expects.
1. For Grafana Cloud, verify that your Cloud Access Policy token has the `traces:read` scope. Refer to [Create a Cloud Access Policy token](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/add-access-policy/) for setup instructions. To find your stack URL and credentials, refer to [Locate your stack URL, user, and password](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/locate-url-user-password/).
1. For self-managed Tempo, ensure the credentials match those configured in the Tempo `server` block or reverse proxy.

### Save & test returns 404

**Error message:** `Tempo echo endpoint returned status 404` when selecting **Save & test**

**Cause:** The Grafana health check reaches the server but the `api/echo` endpoint returns a 404. The most common cause is incorrect path segments in the URL (such as `/api` or `/tempo` on a self-managed instance) that shift the health check request to a nonexistent path. A 404 can also occur when a reverse proxy in front of Tempo rejects requests that include unexpected basic authentication headers.

**Solution:**

1. Verify that the URL doesn't include trailing path segments. For self-managed Tempo, the URL should be `http://<HOST>:3200` with no path. Refer to the [Connection](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/) section for the correct format.
1. If a reverse proxy sits between Grafana and Tempo, check whether the proxy is returning the 404. Tempo itself ignores unexpected auth headers, but a proxy may reject them.
1. If Tempo doesn't require authentication, select **No authentication** in the [data source settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#authentication).
1. Check that the authentication credentials are entered in the correct fields. For Grafana Cloud, the **User** field is the instance ID (not your Grafana login), and the **Password** is a Cloud Access Policy token.

### Multi-tenant authentication issues

**Error message:** `Unauthorized` when using multi-tenant Tempo

**Cause:** The tenant ID (`X-Scope-OrgID` header) is missing or incorrect. This header must be set on both the write path (in your Alloy or OpenTelemetry Collector configuration) and the read path (in the Grafana data source settings) to route data to the correct tenant.

**Solution:**

1. In the Tempo data source settings, verify that the `X-Scope-OrgID` header is configured under **Authentication > HTTP Headers**. Refer to [Multi-tenancy](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/) for data source configuration steps.
1. On the write path, verify that the same `X-Scope-OrgID` header is set in your Alloy or OpenTelemetry Collector configuration so traces are ingested under the correct tenant.
1. Check that the authenticated user has access to the specified tenant.
1. For cross-tenant queries, ensure all specified tenants are accessible.

For more information, refer to [Enable multi-tenancy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/multitenancy/) and [Tenant IDs](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/tenant-ids/).

## Configuration errors

These issues occur when configuring the Tempo data source itself, before you run queries or set up correlations.

### Can't edit a provisioned data source (read-only)

**Symptoms:**

- The data source settings form is read-only and the button reads **Test** instead of **Save & test**.
- You get a read-only error when you try to configure trace to logs correlation, enable **Filter by trace ID**, or change service graph or node graph settings.
- Changes you make in the UI silently revert after Grafana restarts.

**Cause:** The data source is provisioned through a YAML configuration file, Helm chart, or Terraform. By default, provisioned data sources are read-only in the Grafana UI, so the UI can't save changes. If the provisioning file sets `editable: true`, you can save changes in the UI, but provisioning overwrites them on the next restart or provisioning reload.

**Solutions:**

1. **Edit the provisioning file to change settings permanently.** This is the recommended fix for self-managed Grafana, where you have access to the provisioning file. Any setting available in the UI can be set in the YAML file, including trace to logs, trace to metrics, **Filter by trace ID**, and service graph configuration. Edit the file, then restart Grafana or wait for the provisioning system to reload. Refer to [Provision the Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/) for the full YAML reference.
1. **Clone the data source to get an editable copy.** On Grafana Cloud, where you might not have direct access to the provisioning file, clone the provisioned data source to configure correlation, **Filter by trace ID**, or service graph settings through the UI. The clone isn't provisioned, so you can edit and save it normally. Refer to [Clone a provisioned data source for Grafana Cloud](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/provision/#clone-a-provisioned-data-source-for-grafana-cloud) for the steps.

## Query errors

These errors occur when there are issues with TraceQL queries or trace lookups.

### Trace not found

**Error message:** `Trace not found` or `trace ID not found`

**Cause:** The specified trace ID doesn't exist in Tempo or has aged out of the configured retention period. Common reasons include:

- **Sampling:** Head or tail sampling in your instrumentation pipeline (Alloy, OpenTelemetry Collector) may have dropped the trace before it reached Tempo. This is the most common cause of missing traces.
- **Retention:** Traces are only available for the duration set in the Tempo retention configuration. For Grafana Cloud users, default retention periods apply unless you contact Support to change them.
- **Ingestion failure:** The trace may not have been successfully ingested due to rate limits, errors, or misconfigured pipelines.

**Solution:**

1. Verify the trace ID is correct and complete.
1. Check your sampling configuration in Alloy or OpenTelemetry Collector. If head or tail sampling is enabled, the trace may have been intentionally dropped. Refer to [Sampling strategies](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/tail-sampling/) for guidance.
1. Check that the trace is within the configured retention period for Tempo.
1. If using time range restrictions, expand the time range in the [**TraceID query** settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/additional-settings/#traceid-query).
1. Enable **Use time range in query** in the [TraceID query settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/additional-settings/#traceid-query) and adjust the time shift values to search a broader range.
1. Verify the trace was successfully ingested by Tempo. You can run a broad query to check for recent traces:

   ```traceql
   { } | count() > 0
   ```

   If this returns results, traces are being ingested but your specific trace ID may have been dropped by sampling or aged out. For more query examples, refer to the [TraceQL cookbook](https://grafana.com/docs/grafana-cloud/send-data/traces/traces-query-editor/traceql-cookbook/).

1. For Grafana Cloud users, refer to [Troubleshoot TraceQL and search](https://grafana.com/docs/grafana-cloud/send-data/traces/troubleshoot/#traceql-and-search) for TraceQL queries that can help investigate missing traces.

### TraceQL syntax errors

**Error message:** `parse error` or `unexpected token`

**Cause:** The TraceQL query contains syntax errors.

**Solution:**

1. Verify the query follows [TraceQL syntax](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/construct-traceql-queries/).
1. Check that all braces, parentheses, and quotes are balanced.
1. Ensure attribute names are correctly formatted with the correct scope. Use dot notation for span or resource attributes, for example, `span.http.status_code` or `resource.service.name`.
1. Use the **Search** query builder to generate valid TraceQL queries if you're unfamiliar with the syntax.
1. Verify operators are used correctly. Supported operators include `=`, `!=`, `=~`, `!~`, `>`, `<`, `>=`, `<=`.

### Query returns no results

**Cause:** The query is valid but doesn't match any traces.

**Solution:**

1. Expand the time range to search a broader period.
1. Verify the attribute names and values exist in your traces.
1. Check that the attribute names match exactly, as they're case-sensitive.
1. Use the **Search** query builder to explore available attributes and values.
1. Start with a broader query and progressively add filters to narrow results.
1. Try [Grafana Traces Drilldown](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/simplified-exploration/traces/), a queryless app that lets you explore tracing data using RED metrics without writing TraceQL.
1. Note that Tempo search is non-deterministic. Identical queries can return different results because Tempo scans in parallel and returns the first matching traces. Refer to [Understand search behavior](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/) for details.
1. Verify traces are being ingested into Tempo by querying for a known trace ID.

### Traces not appearing after successful connection

**Cause:** **Save & test** succeeds, but running queries in Explore returns no traces. The data source connection is valid, but no trace data is reaching Tempo, or the queries don't match any ingested data.

**Solution:**

1. Verify that your applications are instrumented and that your collector (Alloy, OpenTelemetry Collector) is forwarding spans to Tempo. Check collector logs for send errors, rejected spans, or drops.
1. For Grafana Cloud users who send traces with Alloy, refer to [Troubleshoot sending traces with Alloy](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/traces-with-alloy/#troubleshoot) for ingestion-side issues.
1. Confirm that the query time range covers the period when traces were generated. Trace data only appears for the window in which spans were ingested.
1. For multi-tenant setups, verify that the `X-Scope-OrgID` header is the same on both the write path (collector) and the read path (Grafana data source). A mismatch causes traces to be ingested under one tenant and queried under another.
1. Try a broad query, for example, `{}`, with a short, recent time range to confirm that any traces exist in Tempo. If this returns results, your original query filters are too narrow.

### Query timeout

**Error message:** `context deadline exceeded` or `query timeout`

**Cause:** The query took too long to execute. Trace searches scan raw trace data across the selected time range, so broad queries over long periods can exceed timeout limits.

**Solution:**

1. Reduce the time range to limit the amount of data scanned.
1. Add more specific filters to narrow the search scope.
1. Use indexed attributes in your queries when possible for faster lookups.
1. For self-managed Tempo, increase the query timeout in the Tempo server configuration if appropriate.
1. If you need aggregate data (rates, counts, durations) rather than individual traces, consider using [TraceQL metrics queries](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/metrics-queries/) (for example, `{ } | rate()`) or pre-computed metrics from the [metrics generator](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-generator/), which are optimized for aggregate workloads.

### TraceQL metrics query fails

**Error message:** `localblocks processor not found` or `metrics-generator not configured`

**Cause:** TraceQL metrics queries (such as `{ } | rate()` or `{ } | count_over_time()`) require the `local-blocks` processor to be active in the Tempo metrics-generator configuration. This processor is separate from the `service-graphs` and `span-metrics` processors and must be enabled independently.

**Solution:**

1. For self-managed Tempo, activate the `local-blocks` processor in the overrides configuration. The configuration path depends on your Tempo version and deployment method:
   - Standard: `overrides.metrics_generator_processors: ["local-blocks"]`
   - Newer versions and Helm: `overrides.defaults.metrics_generator.processors: [local-blocks]`

   Refer to [Configure TraceQL metrics](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/configure-traceql-metrics/) for setup steps and required configuration.

1. For Grafana Cloud, the `local-blocks` processor is enabled by default. If you still see this error, contact [Grafana Support](https://grafana.com/contact/).
1. Verify that the metrics generator is running and healthy by checking the Tempo metrics-generator logs for errors.

### TraceQL metrics query exceeds the maximum time range

**Error message:** One of the following, depending on whether streaming is enabled for metrics queries:

- `The selected time range exceeds the maximum allowed duration. Please select a shorter time range.`
- `metrics query time range exceeds the maximum allowed duration of <DURATION>`

**Symptoms:**

- A TraceQL metrics query, such as `{ } | rate()`, fails when the selected time range is too wide.
- Shorter time ranges run successfully, but range queries over long periods return the error.

**Cause:** Tempo limits the time range for TraceQL metrics queries. The default maximum is 24 hours. Tempo enforces this limit, not the data source, so you can't change it from the Grafana UI.

**Solutions:**

1. Reduce the query time range to within the configured maximum, which is 24 hours by default.
1. For self-managed Tempo, raise `query_frontend.metrics.max_duration` (default `24h`) in your Tempo configuration and restart Tempo, or set the per-tenant `max_metrics_duration` override. Refer to [Configure Tempo](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/) in the Tempo documentation.
1. For Grafana Cloud Traces, contact [Grafana Support](https://grafana.com/contact/) to request a higher limit. You can't change this limit yourself.

## Streaming issues

Streaming displays TraceQL query results as they become available. Without streaming, you don't see results until the query completes. For Grafana Cloud users, search streaming is enabled by default and metrics streaming is off by default. For self-managed Tempo, search streaming requires Tempo v2.2 or later, and metrics streaming requires Tempo v2.7 or later. Both require `stream_over_http_enabled: true` in the Tempo configuration.

For more information, refer to [Streaming](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#streaming).

### Streaming not working

**Cause:** Results don't appear incrementally and only display after the full query completes.

**Solution:**

1. For self-managed Tempo, verify that you're using the required Tempo version (v2.2 or later for search streaming, v2.7 or later for metrics streaming) and that streaming is enabled in your configuration (`stream_over_http_enabled: true`). Refer to [Tempo gRPC API](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#tempo-grpc-api) in the Tempo documentation. For Grafana Cloud users, search streaming is enabled by default and metrics streaming is off by default.
1. Verify that **Streaming** is enabled in your [Tempo data source settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#streaming). The **Search queries** and **Metrics queries** toggles are independent. Check that the toggle for the query type you're using is enabled.
1. If your Tempo instance is behind a load balancer or proxy that doesn't support gRPC or HTTP2, streaming may not work. Disable streaming and use standard HTTP queries instead.
1. If you're using [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) to reach Tempo, verify that the PDC tunnel is running and the Tempo instance is reachable from the PDC agent. PDC connectivity issues can present as streaming timeouts.

### gRPC transport security error

**Error message:** `rpc error: code = Unavailable desc = credentials require transport level security`

**Cause:** Grafana tries to use gRPC over an insecure (non-TLS) connection. Streaming uses gRPC, which may require TLS depending on your Tempo configuration.

**Solution:**

1. Configure TLS between Grafana and Tempo. Refer to [Configure TLS communication](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/network/tls/).
1. If TLS isn't an option, disable streaming in the [Tempo data source settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#streaming) and use standard HTTP queries instead.

### Partial results or streaming errors

**Error message:** `stream error` or incomplete results during streaming

**Cause:** The streaming connection was interrupted before all results were delivered.

**Solution:**

1. Check network stability between the browser and Grafana.
1. Verify that proxies or load balancers aren't terminating long-lived connections.
1. Disable streaming and use standard queries if issues persist.
1. Check Tempo server logs for any errors related to streaming.

## Service graph issues

The Service Graph visualizes service dependencies and highlights request rate, error rate, and duration (RED metrics) across connections. It requires a linked Prometheus data source with service graph metrics generated by the Tempo metrics generator or Grafana Alloy. For more information, refer to [Service Graph and Service Graph view](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/).

### Service graph not displaying

**Cause:** The Service Graph view shows no data or an empty graph.

**Solution:**

1. Verify that a Prometheus data source is linked in the Tempo data source **Service Graph** settings under [Additional settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/additional-settings/).
1. For self-managed Tempo, check that the metrics generator is configured to produce service graph metrics. Refer to [Service graph metrics](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-generator/service-graph-view/) in the Tempo documentation.
1. Ensure the following service graph metrics exist in Prometheus:
   - `traces_service_graph_request_total`
   - `traces_service_graph_request_failed_total`
   - `traces_service_graph_request_server_seconds_sum`
1. Verify that Grafana Alloy or the Tempo metrics generator is configured to produce these metrics. These are distinct from span metrics (`traces_spanmetrics_*`), which power the Service Graph view table.
1. To verify the metrics exist, run the following query in Explore with your Prometheus data source selected:

   ```promql
   {__name__=~"traces_service_graph_request.*"}
   ```

1. Check the Prometheus data source connection is working.
1. For Grafana Cloud Traces users, refer to [Troubleshoot service graph and RED metrics](https://grafana.com/docs/grafana-cloud/send-data/traces/troubleshoot/#troubleshoot-service-graph-and-red-metrics).

### Service Graph view table is empty

**Cause:** The Service Graph node graph renders correctly, but the RED metrics table alongside it shows no data.

**Solution:**

1. The Service Graph view table uses a different set of metrics from the node graph. The Grafana frontend queries `traces_spanmetrics_calls_total` and `traces_spanmetrics_latency_bucket` (or `traces_spanmetrics_latency` for native histograms). Confirm that these metrics exist in your linked Prometheus data source.
1. Verify that span metrics generation is enabled in your Tempo or Alloy configuration. Refer to [Enable service graphs](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/enable-service-graphs/) in the Tempo documentation.

### High cardinality warnings

**Cause:** Service graph or span metrics queries produce high-cardinality warnings or slow queries.

**Solution:**

1. Review the label cardinality of your service graph and span metrics. High-cardinality labels (such as unique Pod names or request IDs) create many time series and degrade query performance.
1. Refer to [Cardinality](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-generator/cardinality/) in the Tempo documentation for guidance on managing label cardinality.

### Service graph shows incomplete data

**Cause:** Some services or connections are missing from the graph.

**Solution:**

1. Verify that all services are instrumented and sending spans.
1. Check that span names and service names are consistent.
1. Ensure the time range includes data from all expected services.
1. Verify the Prometheus data source has metrics for all services.

## Trace to logs/metrics/profiles issues

These issues relate to the correlation features that link traces to other telemetry signals. For configuration details, refer to [Trace to logs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/), [Trace to metrics](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/), and [Trace to profiles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/).

### Trace to logs link not appearing

**Cause:** Links to logs don't appear when viewing a trace.

**Solution:**

1. Verify that a Loki or other log data source is configured in the [Trace to logs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/) settings.
1. Check that the configured tags exist in your spans.
1. Ensure at least one mapped tag has a non-empty value in the span.
1. If using a custom query, verify the query syntax is valid for the target data source.
1. Check that **Filter by trace ID** or **Filter by span ID** aren't enabled when using a custom query.
1. If using **Filter by trace ID** with Loki derived fields, verify that the derived field regular expression matches the trace ID field name used in your logs. Different instrumentation libraries use different field names (`traceID`, `trace_id`, or `traceId`). Refer to [Trace to logs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/) for common regular expression patterns by instrumentation format.

### Loki log lines don't show a trace link

**Cause:** The reverse direction, navigating from a Loki log line to a trace, isn't working. No trace link appears on log lines.

**Solution:**

1. Verify that a derived field is configured in the Loki data source under **Additional settings > Derived fields**. The derived field extracts the trace ID from log lines and creates a link to Tempo.
1. Check that the derived field regular expression matches the trace ID format in your logs. Use **Show example log message** in the derived field settings to test the regular expression against a sample log line.
1. Confirm the **Internal link** toggle is enabled and points to the correct Tempo data source.
1. If different services use different trace ID field names, you need a separate derived field entry for each format. Refer to [Trace to logs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/) for regular expression patterns by format.

### Trace to metrics link not appearing

**Cause:** Links to metrics don't appear when viewing a trace.

**Solution:**

1. Verify that a Prometheus or other metrics data source is configured in the [Trace to metrics](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/) settings.
1. Check that the configured tags exist in your spans.
1. If using custom queries, verify they're valid PromQL.
1. Ensure the `$__tags` variable is correctly placed in custom queries.

### Trace to profiles link not appearing

**Cause:** Links to profiling data don't appear when viewing a trace.

**Solution:**

1. Verify that a Grafana Pyroscope data source is configured in the [Trace to profiles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/) settings.
1. Check that the configured tags exist in your spans.
1. Verify that the configured profile type matches what Pyroscope has data for (for example, `process_cpu:cpu:nanoseconds:cpu:nanoseconds`). A mismatch between the configured profile type and the available data causes the link to not appear.
1. Tighten the **Span start time shift** and **Span end time shift** values. Profiles are sampled at intervals, so a narrow time window around the span is more likely to match profile data than a wide one.
1. Verify that the profiling data source is connected and returning data.

### Links appear but return no data

**Cause:** The generated query doesn't match any data in the target data source.

**Solution:**

1. Adjust the **Span start time shift** and **Span end time shift** to widen the time range.
1. Verify that log or metric labels match the span attributes.
1. Check that the tag mappings correctly translate attribute names between data sources. Span attributes use dots (for example, `service.name`) but Loki labels use underscores (for example, `service_name`). Ensure tag mappings account for this difference. Refer to [Trace to logs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/) for tag mapping configuration.
1. If a tag like `pod` is stored as Loki [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) rather than an indexed label, the auto-generated stream selector `{pod="..."}` returns no results. Enable **Use custom query** and move the tag to a pipeline filter. Refer to [Trace to logs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/) for a custom query example.
1. Run the generated query directly in the target data source's Explore view to confirm it returns data outside of the trace context.
1. Use the Query Inspector to view the generated query and verify it's correct.

### Trace links break after renaming a provisioned data source

**Symptoms:**

- Log-to-trace or trace-to-log links stop working after you rename a provisioned Tempo or Loki data source.
- Correlations or derived fields that previously resolved now point to a data source that no longer exists.

**Cause:** Correlations, derived fields, and trace to logs, metrics, and profiles settings reference the target data source by its `uid`, not its name. When a provisioned data source doesn't set an explicit `uid`, Grafana derives one from the data source name. Renaming the data source in the provisioning file changes the derived `uid`, so every link that referenced the old `uid` breaks.

**Solutions:**

1. Set an explicit, stable `uid` for each provisioned data source in the YAML file so the `uid` doesn't change when you rename the data source:

   ```yaml
   datasources:
     - name: Tempo
       type: tempo
       uid: tempo-prod
   ```

1. After you add or correct the `uid`, update the correlations, derived fields, and trace to logs, metrics, and profiles settings that reference it to use the current `uid`.
1. To confirm a data source's `uid`, open its settings page. The `uid` appears in the page URL.

## Performance issues

These issues relate to slow queries or high resource usage.

### Slow trace queries

**Cause:** Queries take a long time to return results.

**Solution:**

1. Reduce the dashboard or Explore time range.
1. Add more specific filters to narrow the search scope.
1. Use indexed attributes in TraceQL queries when possible.
1. Enable streaming to see results as they become available.
1. Consider using the **Search** query builder which can help optimize queries.

### High memory usage during queries

**Cause:** Large traces or result sets consume excessive memory.

**Solution:**

1. Reduce the **Limit** setting in query options to return fewer traces.
1. Reduce the **Span Limit** setting to return fewer spans per trace.
1. Add filters to reduce the result set size.
1. Avoid querying very large time ranges.

## Other common issues

The following issues don't produce specific error messages but are commonly encountered.

### Node graph not displaying

**Cause:** The node graph visualization doesn't appear above the trace view.

**Solution:**

1. Verify that **Node graph** is enabled in the Tempo data source settings under [Additional settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/additional-settings/).
1. Check that the trace contains multiple spans with parent-child relationships. Spans must include `traceID` and `spanID`, and non-root spans must include `parentSpanID` to establish hierarchy. If these identifiers are missing, the trace data is incomplete at the instrumentation level and the graph can't be rendered.

### Autocomplete not working

**Cause:** The TraceQL editor doesn't show suggestions for attributes or values. Autocomplete requires Tempo to enumerate tags, which requires a supported storage backend (`vParquet` or later).

**Solution:**

1. For self-managed Tempo, verify that the storage backend supports tag search. Tempo requires `vParquet` format or later for tag-based search and autocomplete. Refer to [Parquet configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/parquet/) for details. Grafana Cloud Traces supports autocomplete by default.
1. Check the **Tags time range** setting in the [data source settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/additional-settings/) to ensure it covers a period with recent trace data.
1. Increase the **Tag limit** setting if you have many unique attributes.
1. Verify the Tempo data source connection is working by selecting **Save & test** in the data source settings.

### TraceQL alerting not available

**Cause:** You can select the Tempo data source in the Grafana Alerting rule builder, but standard TraceQL search queries return trace data (spans and trace IDs), not numeric time series. Alert rules require numeric data to evaluate thresholds, so trace search queries don't produce usable alert conditions.

**Solution:**

TraceQL metrics queries (such as `{ } | rate()` or `{ status = error } | count_over_time()`) return numeric time series and can drive alert thresholds directly. However, TraceQL alerting requires the experimental `tempoAlerting` feature flag, which is disabled by default. To enable it, set the [`tempoAlerting` feature flag](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#feature_toggles) in your Grafana configuration. For Grafana Cloud, contact Support to enable it. For step-by-step setup, refer to [Trace-based alerts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/examples/trace-based-alerts/).

{{< admonition type="caution" >}}
The `tempoAlerting` feature flag is experimental. For production alerting, consider the Prometheus-based approach described below.
{{< /admonition >}}

Alternatively, use the Tempo metrics generator to produce Prometheus metrics from trace data, then create alert rules against the Prometheus data source that stores those metrics.

1. Configure the Tempo [metrics generator](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-generator/) (using the `service-graphs`, `span-metrics`, or `local-blocks` processor) to produce metrics from traces and remote-write them to a Prometheus-compatible backend.
1. Create alert rules against the Prometheus data source that receives those metrics. For example, alert on high error rates using `traces_spanmetrics_calls_total` with a `status_code="STATUS_CODE_ERROR"` filter.
1. Refer to the feature support table in the [Tempo data source overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/) for details on alerting support.

### Switching between multiple Tempo instances

**Cause:** You have multiple Tempo data sources configured (for example, one per environment or per tenant), but there's no built-in UI dropdown in Explore or dashboards to switch between them dynamically.

**Workaround:**

1. In dashboards, create a [data source template variable](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/#add-a-data-source-variable) filtered to `tempo` type. This adds a dropdown that lets viewers switch between Tempo instances without editing the dashboard.
1. In Explore, use the data source picker in the top bar to switch between configured Tempo data sources.
1. For multi-tenant setups on a single Tempo instance, consider using the `X-Scope-OrgID` header instead of separate data sources. Refer to [Multi-tenancy](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/) for configuration details.

## Private data source connect issues (_Grafana Cloud only_)

These issues relate to connecting to a private Tempo instance using Private data source connect (PDC). PDC is only available in Grafana Cloud.

{{< admonition type="note" >}}
Private data source connect allows Grafana Cloud to securely connect to data sources in your private network without exposing them to the public internet. For setup instructions, refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/).
{{< /admonition >}}

### PDC agent not connected

**Error message:** `No healthy PDC agents available` or `Private data source connect agent not connected`

**Cause:** The PDC agent installed in your private network isn't running or can't connect to Grafana Cloud.

**Solution:**

1. Verify the PDC agent is running in your private network.
1. Check the PDC agent logs for connection errors.
1. Ensure the PDC agent has outbound network access to Grafana Cloud endpoints.
1. Verify the PDC agent is using the correct token for your Grafana Cloud instance.
1. Check that firewall rules allow outbound HTTPS (port 443) connections from the PDC agent.
1. Restart the PDC agent if it appears stuck or unresponsive.

For more information, refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/).

### Connection through PDC failing

**Error message:** `Failed to connect through private data source` or `Connection refused via PDC`

**Cause:** The PDC agent is connected to Grafana Cloud but cannot reach the Tempo instance in your private network.

**Solution:**

1. Verify the Tempo URL is correct and accessible from the machine running the PDC agent.
1. Check that the PDC agent can resolve the Tempo hostname (DNS resolution).
1. Ensure there are no firewall rules blocking connections from the PDC agent to Tempo.
1. Verify the Tempo port (default `3200`) is open and accessible.
1. Test connectivity from the PDC agent host to Tempo using `curl` or `telnet`.

### PDC timeout errors

**Error message:** `Request timed out via private data source connect`

**Cause:** Queries through the PDC tunnel are timing out.

**Solution:**

1. Check network latency between the PDC agent and Tempo.
1. Verify the PDC agent host has sufficient resources (CPU, memory, network bandwidth).
1. Ensure no network devices between the PDC agent and Tempo are dropping long-running connections.
1. For large queries, consider reducing the time range or adding more specific filters.
1. Check if multiple data sources are sharing the same PDC agent, which may cause resource contention.

### PDC authentication mismatch

**Error message:** `Unauthorized` or `403 Forbidden` when using PDC

**Cause:** The authentication configured in the data source doesn't work with the PDC connection.

**Solution:**

1. Verify that authentication credentials are configured correctly in the Grafana Cloud data source settings.
1. Check that the Tempo instance accepts the authentication method configured.
1. Ensure any required headers (such as `X-Scope-OrgID` for multi-tenant Tempo) are properly forwarded through PDC.
1. Test the connection directly from the PDC agent host to verify credentials work outside of PDC.

## Get additional help

If you continue to experience issues after following this troubleshooting guide:

1. For Grafana Cloud users, try [Grafana Assistant](https://grafana.com/docs/grafana-cloud/send-data/traces/investigate-traces-with-assistant/) to investigate trace issues interactively (if enabled on your account).
1. For Grafana Cloud users who send traces with Alloy, refer to the [Alloy troubleshooting steps](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/traces-with-alloy/#troubleshoot) for ingestion-side issues.
1. Ask questions in the [Grafana community forums](https://community.grafana.com/) or the [Grafana Community Slack](https://slack.grafana.com/).
1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs.
1. Review the [Tempo GitHub issues](https://github.com/grafana/tempo/issues) for Tempo-specific bugs.
1. Enable [debug logging](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#log) in Grafana to capture detailed error information.
1. Check [Tempo server logs](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/monitor/) for additional error details.
1. Contact [Grafana Support](https://grafana.com/contact/) if you're an Enterprise, Cloud Pro, or Cloud Contracted user.

When reporting issues, include:

- Grafana version
- Tempo version
- Error messages (redact sensitive information)
- Steps to reproduce
- TraceQL query examples (redact sensitive data)
- Relevant configuration settings

# TestData alerting

The TestData data source supports [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/). You can use it to prototype alert rules, test threshold conditions, and verify notification pipelines without connecting to an external data source.

## Before you begin

- [Configure the TestData data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/configure/).
- Familiarize yourself with [alert rules](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/) and how to [create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Choose a scenario for alerting

TestData scenarios that return time-series data work with alert rule conditions. Choose a scenario based on the behavior you want to test.

| Scenario                     | Alerting use case                                                                                |
| ---------------------------- | ------------------------------------------------------------------------------------------------ |
| **Predictable Pulse**        | Deterministic on/off pattern. Produces repeatable alert firing and resolving on a fixed cycle.   |
| **Random Walk**              | Non-deterministic time series. Useful for quick prototyping and load testing alert evaluation.   |
| **CSV Metric Values**        | Controlled, fixed values. Test exact threshold boundaries.                                       |
| **Predictable CSV Wave**     | Custom repeating waveforms. Test complex threshold patterns with precise control.                |
| **Random Walk (with error)** | Returns both data and an error. Test how alerts handle partial failures.                         |
| **Error with source**        | Returns a plugin or downstream error. Test alert evaluation error handling.                      |
| **Flaky Query**              | Returns errors intermittently. Test how alerts handle a data source that fails part of the time. |
| **No Data Points**           | Returns empty results. Test no-data alert conditions.                                            |
| **Slow Query**               | Introduces a configurable delay. Test alert evaluation timeouts.                                 |

Scenarios that produce logs, traces, or streaming data aren't suitable for threshold-based alert conditions.

## Create an alert rule

To create an alert rule using TestData:

1. Navigate to **Alerting** > **Alert rules**.
1. Click **New alert rule**.
1. Enter a name for the rule.
1. Select the **TestData** data source.
1. Choose a scenario from the **Scenario** drop-down (for example, **Predictable Pulse**).
1. Configure the scenario options.
1. Add a **Reduce** expression to aggregate the series (for example, **Last** value).
1. Add a **Threshold** expression to define the condition (for example, **Is above** `0.5`).
1. Set the evaluation interval and pending period.
1. Configure notifications and labels.
1. Click **Save rule**.

For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/).

## Example: Predictable Pulse alert

The Predictable Pulse scenario produces a repeating on/off wave that's ideal for testing alert rules because the firing pattern is deterministic and reproducible.

To create a Predictable Pulse alert:

1. Select the **Predictable Pulse** scenario.
1. Configure the pulse options:

   | Field         | Value | Effect                                      |
   | ------------- | ----- | ------------------------------------------- |
   | **Step**      | `60`  | One data point every 60 seconds.            |
   | **On Count**  | `3`   | Three consecutive points at the "on" value. |
   | **Off Count** | `6`   | Six consecutive points at the "off" value.  |
   | **On Value**  | `1`   | Value during the "on" phase.                |
   | **Off Value** | `0`   | Value during the "off" phase.               |

1. Add a **Reduce** expression with function **Last**.
1. Add a **Threshold** expression: **Is above** `0.5`.

With this configuration, the alert fires for 3 minutes (3 x 60s on-points) and then resolves for 6 minutes (6 x 60s off-points), repeating indefinitely. Adjust the step, on count, and off count to control the timing.

## Example: CSV Metric Values threshold alert

The CSV Metric Values scenario lets you define an exact sequence of data points, making it useful for testing precise threshold boundaries.

To create a threshold alert with fixed values:

1. Select the **CSV Metric Values** scenario.
1. Enter a comma-separated list of values in the **String Input** field. For example:

   ```
   10,45,70,95,50,20
   ```

1. Add a **Reduce** expression with function **Last**.
1. Add a **Threshold** expression: **Is above** `80`.

With this input, the series cycles through the values `10, 45, 70, 95, 50, 20`. The alert fires when the reduced value exceeds `80` (the `95` data point) and resolves when it drops back below the threshold.

This approach is helpful for validating that a threshold condition triggers at the exact boundary you expect.

## Test error and no-data conditions

TestData includes scenarios designed for testing how alerts respond to failures. Use these to verify that your alert rules handle edge cases correctly before connecting to production data sources.

### No data

Select the **No Data Points** scenario. The backend returns an empty result, which triggers the alert rule's **no data** state. Use this to verify that your rule is configured to alert, keep the last state, or resolve when no data is received. Configure this behavior in the alert rule's **No data and error handling** section.

### Errors

Select **Error with source** and choose the error type:

- **Plugin**: Simulates a failure in the plugin itself. The alert enters an error state.
- **Downstream**: Simulates a failure in the data source or network. The alert enters an error state with a downstream classification.

You can also use **Random Walk (with error)**, which returns valid data alongside an error, to test how the alerting engine handles partial failures.

### Intermittent errors

Select **Flaky Query** to simulate a data source that fails part of the time. Set the **Error rate** to control how often requests fail, and use **Status code** and **Error source** to shape the error. Use this to verify how an alert rule behaves when evaluations succeed and fail across consecutive intervals, and how the **No data and error handling** settings respond to intermittent failures.

### Timeouts

Select **Slow Query** and set the **String Input** field to a duration longer than your evaluation interval (for example, `30s` for a rule evaluated every `10s`). The default delay is `5s`. Use this to verify timeout behavior and confirm that alerts transition to the expected state when queries take too long.

## Limitations

When using TestData with Grafana Alerting, be aware of the following limitations.

### Template variables aren't supported in alert queries

Grafana evaluates alert rules on the backend without dashboard context. Template variables like `$varname` aren't resolved during alert evaluation. Use fixed values in the scenario options instead.

### Only time-series scenarios work with alert conditions

Alert conditions require numeric time-series data. Scenarios that produce logs (Logs), traces (Trace), graphs (Node Graph, Flame Graph), annotations (Annotations), or tables (Table Static) can't be used as alert conditions.

### Streaming scenarios aren't evaluated by the alerting engine

The alerting engine evaluates queries at fixed intervals and doesn't consume streaming data. Avoid using Streaming Client, Grafana Live, or Simulation with streaming enabled for alert rules.

# TestData annotations

The TestData data source can generate simulated [annotation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/) events. Use it to test how dashboards and panels render annotation markers and overlays without connecting to an external source. The generated events are created in the browser and aren't persisted.

## Before you begin

- [Configure the TestData data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/configure/).
- Familiarize yourself with [annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/) in Grafana.

## Add an annotation query

To add a dashboard annotation query that uses TestData:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click **Settings**.
1. Select the **Annotations** tab.
1. Click **Add annotation query**.
1. Enter a name for the annotation.
1. From the **Data source** drop-down, select **TestData**. The drop-down only lists data sources that support annotations.
1. Set the **Count** field to the number of events you want to generate. Default: `10`.
1. Click **Save dashboard**.

TestData distributes the generated events evenly across the current dashboard time range. Each event includes sample text with a link and the tags `text` and `server`.

## Use the Annotations scenario in a panel

You can also generate annotation data as a panel query using the **Annotations** scenario. This scenario returns data on the annotations data topic, so Grafana renders the events as markers overlaid on the panel rather than plotting them as a series.

Because the Annotations scenario doesn't return a visible series on its own, add a second query that returns data to give the markers something to overlay:

1. Add or edit a panel and select the **TestData** data source.
1. On the first query, choose a scenario that returns a series, such as **Random Walk**, from the **Scenario** drop-down.
1. Add a second query, then choose the **Annotations** scenario from its **Scenario** drop-down.
1. Set the **Count** field on the Annotations query to the number of events you want to generate. Default: `10`.
1. Click **Run queries**.

Grafana plots the series from the first query and overlays the annotation markers on top. If you run the Annotations scenario by itself, the query returns annotation events but the panel doesn't display a visible series.

For a reference of all TestData scenarios, refer to the [TestData query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/query-editor/).

## Example: Overlay annotations on a time series panel

This example creates a time series panel with simulated data, then adds a TestData annotation query so you can verify how the panel renders annotation markers.

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add** icon and select **Visualization**.
1. Select the **TestData** data source.
1. Choose the **Random Walk** scenario from the **Scenario** drop-down.
1. Click **Run queries** to render the time series.
1. Click **Back to dashboard**, then click **Settings**.
1. Select the **Annotations** tab.
1. Click **Add annotation query**.
1. Enter a name for the annotation, for example `Test events`.
1. From the **Data source** drop-down, select **TestData**.
1. Set the **Count** field to `5`.
1. Click **Save dashboard**.

Grafana draws five vertical annotation markers evenly across the time range on the time series panel. Hover over a marker to see the sample text and tags. Adjust the **Count** field to change how many markers appear.

{{< admonition type="tip" >}}
To test how a panel handles crowded overlays, increase the **Count** field to a high value, such as `100`. This produces many closely spaced markers, which is useful for checking marker rendering and tooltip behavior at high density.
{{< /admonition >}}

## Limitations

- **Events are synthetic.** TestData generates fixed sample text and tags. You can't customize the event content.
- **Events aren't persisted.** The data source builds events in the browser each time the query runs. TestData doesn't store annotations or query an external source.
- **Not evaluated by the alerting engine.** The Annotations scenario runs in the browser, so it can't be used with [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/alerting/) or in any context that requires server-side evaluation.

# Configure the TestData data source

This document explains how to configure the TestData data source. TestData requires no external connection or authentication, so configuration is minimal.

For general information on managing data sources, refer to [Data source management](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/).

## Before you begin

You must have the Organization administrator role to add and configure data sources.

## Add the data source

To add the TestData data source:

1. Click **Connections** in the left-side menu.
1. Click **Add new connection**.
1. Type `TestData` in the search bar.
1. Select **TestData**.
1. Click **Add new data source**.

Grafana takes you to the **Settings** tab.

## Configuration options

TestData doesn't require any settings beyond the standard options common to all data sources.

| Setting     | Description                                                              |
| ----------- | ------------------------------------------------------------------------ |
| **Name**    | Sets the name you use to refer to the data source in panels and queries. |
| **Default** | Defines whether this data source is pre-selected for new panels.         |

## Verify the connection

Click **Save & test** to verify the data source. A successful test displays the message:

**Data source is working**

If the test fails, refer to [Troubleshoot TestData](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/troubleshooting/) for common issues and solutions.

## Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, refer to [Provisioning data sources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

TestData doesn't require any `jsonData` or `secureJsonData` fields.

**YAML example**

```yaml
apiVersion: 1

datasources:
  - name: TestData
    type: grafana-testdata-datasource
    access: proxy
```

## Provision with Terraform

To provision the data source with Terraform, use the [`grafana_data_source` resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source):

```hcl
resource "grafana_data_source" "testdata" {
  name = "TestData"
  type = "grafana-testdata-datasource"
}
```

For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source).

## Next steps

After configuring your TestData data source, you can:

- [Write queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/query-editor/) using the 33 available scenarios to generate simulated data.
- [Use template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/template-variables/) to create dynamic, reusable dashboards.
- [Set up alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/alerting/) to prototype and test alert rules with simulated data.
- [Troubleshoot issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/troubleshooting/) if you encounter problems with your data source.

# TestData query editor

Instead of a traditional query language, the TestData data source uses **scenarios** to generate simulated data. Each scenario produces a different type of data suited for testing specific visualizations, behaviors, or edge cases. TestData includes 33 scenarios covering time series, logs, traces, graphs, streaming, and error simulation.

Use scenarios to:

- **Prototype dashboards** without connecting to a real data source.
- **Reproduce bugs** with controlled, deterministic data that other developers can replicate.
- **Test panel behavior** with edge cases like empty results, timestamps outside the visible range, or mixed data and errors.
- **Validate alerting pipelines** using predictable patterns that fire and resolve on a known schedule.
- **Simulate streaming** to verify how panels handle real-time data updates.

To build a query, select a scenario from the **Scenario** drop-down. The query editor updates to show fields specific to that scenario. Click **Run queries** or use the keyboard shortcut to execute.

{{< admonition type="note" >}}
Some scenarios run entirely in the browser: Streaming Client, Grafana Live, Grafana API, Steps, Annotations, Node Graph, Flame Graph, Trace, and Raw Frames. These scenarios don't send queries to the backend, which means they can't be used with [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/alerting/) or in any context that requires server-side evaluation.
{{< /admonition >}}

## Scenario reference

The scenarios are organized into the following categories. Use the table below to find the right scenario for your use case.

<!-- vale Grafana.Spelling = NO -->

| Scenario                                                            | Category              | Purpose                                            |
| ------------------------------------------------------------------- | --------------------- | -------------------------------------------------- |
| [Random Walk](#random-walk)                                         | Data generation       | Random walk time series (default scenario).        |
| [Random Walk Table](#random-walk-table)                             | Data generation       | Random walk in table format with `Enum` state.     |
| [Random Walk (with error)](#random-walk-with-error)                 | Data generation       | Random walk that also returns an error.            |
| [Predictable Pulse](#predictable-pulse)                             | Data generation       | Repeating on/off wave based on absolute time.      |
| [Predictable CSV Wave](#predictable-csv-wave)                       | Data generation       | Custom repeating waveforms from CSV values.        |
| [Simulation](#simulation)                                           | Data generation       | Continuous simulation engine (flight, sine, tank). |
| [USA generated data](#usa-generated-data)                           | Data generation       | Multi-dimensional data with US state dimensions.   |
| [CSV Content](#csv-content)                                         | Manual input          | Paste or type CSV data directly.                   |
| [Steps](#steps)                                                     | Manual input          | Step-function data from CSV input.                 |
| [CSV File](#csv-file)                                               | Manual input          | Select from built-in CSV data files.               |
| [CSV Metric Values](#csv-metric-values)                             | Manual input          | Time series from comma-separated values.           |
| [Raw Frames](#raw-frames)                                           | Manual input          | Define data frames in JSON format.                 |
| [Load Apache Arrow Data](#load-apache-arrow-data)                   | Manual input          | Render base64-encoded Arrow payloads.              |
| [Table Static](#table-static)                                       | Manual input          | Static table with predefined columns.              |
| [Logs](#logs)                                                       | Visualization testing | Simulated log data with random levels.             |
| [Node Graph](#node-graph)                                           | Visualization testing | Data for node graph visualizations.                |
| [Flame Graph](#flame-graph)                                         | Visualization testing | Data for flame graph visualizations.               |
| [Trace](#trace)                                                     | Visualization testing | Simulated distributed trace data.                  |
| [Annotations](#annotations)                                         | Visualization testing | Annotation data points.                            |
| [Exponential heatmap bucket data](#exponential-heatmap-bucket-data) | Visualization testing | Heatmap data with exponential buckets.             |
| [Linear heatmap bucket data](#linear-heatmap-bucket-data)           | Visualization testing | Heatmap data with linear buckets.                  |
| [Exemplars](#exemplars)                                             | Visualization testing | Exemplar markers to overlay on a sibling series.   |
| [Streaming Client](#streaming-client)                               | Streaming             | Browser-side streaming (signal, logs, traces).     |
| [Grafana Live](#grafana-live)                                       | Streaming             | Server-side streaming via live channels.           |
| [Grafana API](#grafana-api)                                         | Data retrieval        | Fetch data from internal Grafana endpoints.        |
| [Conditional Error](#conditional-error)                             | Error testing         | Configurable error or CSV data.                    |
| [Error with source](#error-with-source)                             | Error testing         | Error with plugin or downstream classification.    |
| [Flaky Query](#flaky-query)                                         | Error testing         | Intermittent errors with configurable rate.        |
| [Errors and notices](#errors-and-notices)                           | Error testing         | Series with info, warning, and error notices.      |
| [No Data Points](#no-data-points)                                   | Error testing         | Empty result with no data.                         |
| [Datapoints Outside Range](#datapoints-outside-range)               | Error testing         | Data point outside the visible time range.         |
| [Slow Query](#slow-query)                                           | Error testing         | Configurable delay before returning data.          |
| [Query Metadata](#query-metadata)                                   | Metadata              | Returns query context metadata.                    |

<!-- vale Grafana.Spelling = YES -->

## Shared query options

Every scenario displays the **Scenario** drop-down. Most scenarios also show an **Alias** field and, when the scenario defines a default, a **String Input** field. Some scenarios expose additional shared controls. Scenario-specific options appear below these fields.

| Field            | Description                                                                                                                                                                                |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Scenario**     | Drop-down to select the data generation type. Scenarios are listed alphabetically.                                                                                                         |
| **Alias**        | Optional display name for the series. Hidden for the Simulation and Annotations scenarios. Supports the special values `__server_names` and `__house_locations` to use built-in name sets. |
| **String Input** | Text field shown when the selected scenario provides a default string value. Used differently by each scenario.                                                                            |
| **Labels**       | Available for Random Walk and Predictable Pulse. Set labels using key=value syntax, for example `key="value", key2="value2"`. Use `$seriesIndex` in a value to insert the series index.    |
| **Drop percent** | Available for CSV Content and CSV File. Drops a random percentage (0-100) of data points from the result.                                                                                  |

## Data generation scenarios

These scenarios produce time-series data with configurable parameters.

### Random Walk

Generates random walk time-series data. This is the default scenario.

| Field            | Description                                             |
| ---------------- | ------------------------------------------------------- |
| **Series count** | Number of series to generate. Default: `1`.             |
| **Start value**  | Initial value for the walk. Default: auto (random).     |
| **Min**          | Minimum value the walk can reach. Default: none.        |
| **Max**          | Maximum value the walk can reach. Default: none.        |
| **Spread**       | Controls how far each step can deviate. Default: `1`.   |
| **Noise**        | Adds noise to each data point. Default: `0`.            |
| **Drop (%)**     | Percentage of points to randomly exclude. Default: `0`. |

### Random Walk Table

Generates random walk data in table format with columns for Time, Value, Min, Max, Info, and State. The State column uses an `enum` field with values Unknown, Up, and Down. Optionally includes null values when **withNil** is enabled.

### Random Walk (with error)

Generates random walk time-series data and also returns an error in the response. Use this to test how panels handle data responses that contain both data and errors.

### Predictable Pulse

Generates a predictable pulse wave based on absolute time from the epoch, making it reproducible across runs.

| Field         | Description                                                                           |
| ------------- | ------------------------------------------------------------------------------------- |
| **Step**      | Seconds between data points. Default: `60`.                                           |
| **On Count**  | Number of data points at the start of each cycle that use the on value. Default: `3`. |
| **Off Count** | Number of data points in each cycle that use the off value. Default: `3`.             |
| **On Value**  | The value for "on" data points. Can be a number, `null`, or `nan`. Default: `2`.      |
| **Off Value** | The value for "off" data points. Can be a number, `null`, or `nan`. Default: `1`.     |

The wave cycles at `Step * (On Count + Off Count)` seconds. Timestamps align evenly on the step interval.

With the defaults (Step=60, On Count=3, Off Count=3), the wave completes a full cycle every 6 minutes. The on value (`2`) holds for the first 3 data points, then the off value (`1`) holds for the next 3.

### Predictable CSV Wave

Generates one or more predictable waves from CSV-defined values. Each wave cycles through its comma-separated values at a fixed time step.

| Field      | Description                                                                                     |
| ---------- | ----------------------------------------------------------------------------------------------- |
| **Values** | Comma-separated numeric values for the wave. Supports `null` and `nan`. Default: `0,0,2,2,1,1`. |
| **Step**   | Seconds between data points. Default: `60`.                                                     |
| **Name**   | Optional name for the series.                                                                   |
| **Labels** | Optional labels in key=value format.                                                            |

Click the **+** button on the last wave row to add another wave. Click the **-** button on any other row to remove it.

For example, to create two overlapping waves for comparison:

- Wave 1: Values `0,1,2,3,4,5`, Step `60`, Name `rising`
- Wave 2: Values `5,4,3,2,1,0`, Step `60`, Name `falling`

### Simulation

Runs a simulation engine that generates data continuously. Simulation supports streaming data through Grafana Live.

| Field          | Description                                                                                                    |
| -------------- | -------------------------------------------------------------------------------------------------------------- |
| **Simulation** | Simulation type: `Flight` (circling flight path), `Sine` (sine wave), or `Tank` (fill and drain a water tank). |
| **Stream**     | Toggle to stream data through Grafana Live instead of returning a static result.                               |
| **Interval**   | Tick frequency in Hz. Default: `10`.                                                                           |
| **Last**       | Toggle to return only the most recent value instead of the full time range.                                    |
| **UID**        | Optional unique identifier. Allows multiple instances of the same simulation type to run concurrently.         |

### USA generated data

Generates data with US state dimensions. Useful for testing geo-map visualizations and multi-dimensional data.

| Field      | Description                                                                                                        |
| ---------- | ------------------------------------------------------------------------------------------------------------------ |
| **Mode**   | Data format: `values-as-rows`, `values-as-fields`, `values-as-labeled-fields`, `timeseries`, or `timeseries-wide`. |
| **Period** | Time period for data generation. Default: `30m`.                                                                   |
| **Fields** | Fields to include: `foo`, `bar`, `baz`. Default: all.                                                              |
| **States** | US state codes to include (for example, `CA`, `NY`, `TX`). Default: all 50 states plus DC.                         |

## Manual input scenarios

These scenarios let you provide your own data directly instead of generating it.

### CSV Content

Provides a text editor where you paste or type CSV data directly. The first row is treated as headers. Use the **Drop percent** field to randomly exclude a percentage of data points.

For example, enter the following to create a time series with two value columns:

```
Time,Temperature,Humidity
2024-01-01 00:00:00,22.5,45
2024-01-01 01:00:00,21.8,48
2024-01-01 02:00:00,20.1,52
```

### Steps

Provides a CSV text area (the same editor as CSV Content) for defining step-function data. The default content is `a`, `b`, `c`. This scenario is handled entirely in the browser.

### CSV File

Selects from a set of built-in CSV data files. Use the **Drop percent** field to randomly exclude a percentage of data points.

Available files:

- `flight_info_by_state.csv`
- `population_by_state.csv`
- `gdp_per_capita.csv`
- `js_libraries.csv`
- `ohlc_dogecoin.csv`
- `weight_height.csv`
- `browser_marketshare.csv`
- `automobiles.csv`

### CSV Metric Values

Generates time-series data from comma-separated values entered in the **String Input** field. Values are evenly distributed across the selected time range. Default: `1,20,90,30,5,0`.

For example, with the default values and a 6-hour time range, the six values (`1`, `20`, `90`, `30`, `5`, `0`) are spread across the range at equal intervals.

### Raw Frames

Provides a JSON editor for defining data frames directly. The editor accepts the Grafana data frame JSON format and includes paste helpers for panel JSON and raw query results.

### Load Apache Arrow Data

Loads data from a base64-encoded Apache Arrow payload entered in the **String Input** text area. Use this to test Arrow format rendering.

### Table Static

Generates a static table with predefined columns: Time, Message, Description, and Value.

## Visualization test scenarios

These scenarios generate data for testing specific visualization types.

### Logs

Generates simulated log data with random log levels, container IDs, and hostnames.

| Field     | Description                                                                                |
| --------- | ------------------------------------------------------------------------------------------ |
| **Lines** | Number of log lines to generate. Default: `10`. Maximum: `10000`.                          |
| **Level** | Toggle to include a separate level column in the data instead of embedding it in messages. |

### Node Graph

Generates data for the [Node Graph visualization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/node-graph/).

| Field         | Description                                                                                               |
| ------------- | --------------------------------------------------------------------------------------------------------- |
| **Data type** | Type of graph data: `random`, `response_small`, `response_medium`, `random edges`, or `feature_showcase`. |
| **Count**     | Number of nodes to generate. Available for `random` and `random edges` types.                             |
| **Seed**      | Seed value for reproducible random generation. Available for `random` and `random edges` types.           |

### Flame Graph

Generates data for the [Flame Graph visualization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/flame-graph/).

| Field            | Description                                                           |
| ---------------- | --------------------------------------------------------------------- |
| **Diff profile** | Toggle to generate a differential flame graph comparing two profiles. |

### Trace

Generates simulated distributed trace data.

| Field          | Description                             |
| -------------- | --------------------------------------- |
| **Span count** | Number of spans in the generated trace. |

### Annotations

Generates synthetic annotation data points. These annotations are created entirely in the browser and are useful for testing how panels display annotation markers and overlays. They don't query an external source or persist data.

For more information about using annotations with TestData, refer to [TestData annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/annotations/). For more information about annotations in Grafana, refer to [Annotate visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/).

| Field     | Description                                       |
| --------- | ------------------------------------------------- |
| **Count** | Number of annotations to generate. Default: `10`. |

### Exponential heatmap bucket data

Generates heatmap data with exponentially distributed bucket boundaries (1, 2, 4, 8, 16, ...). Use this to test heatmap panels with exponential distributions.

### Linear heatmap bucket data

Generates heatmap data with linearly distributed bucket boundaries (0, 10, 20, 30, ...). Use this to test heatmap panels with linear distributions.

### Exemplars

Generates only exemplars, the diamond markers a panel draws on top of a series, and no series of its own. Add a second query in the same panel, such as Random Walk, to give the exemplars a series to annotate. Values and timestamps are seeded from the dashboard time range, so the same range always returns the same exemplars.

| Field      | Description                                                                                                                             |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **Count**  | Number of exemplars to generate.                                                                                                        |
| **Min**    | Minimum value for the exemplars. When empty, the range is derived from a reference random walk over the same time range, padded by 10%. |
| **Max**    | Maximum value for the exemplars. When empty, the range is derived as described for **Min**.                                             |
| **Labels** | Exemplar labels shown in the tooltip. Each label defines a **Name**, a **Length**, and an optional **Link**.                            |

To keep the exemplars aligned with a sibling series, pin **Min** and **Max**, or give the sibling query an explicit start value and spread. The legend filter matches on exemplar labels, so hiding a series in the legend also hides its markers.

## Streaming scenarios

These scenarios produce real-time streaming data. Data updates continuously in the browser without requiring manual query execution.

### Streaming Client

Generates streaming data directly from the browser client.

| Field    | Description                                                   |
| -------- | ------------------------------------------------------------- |
| **Type** | Stream type: `Signal`, `Logs`, `Fetch`, `Traces`, or `Watch`. |

Additional fields depend on the selected type:

- **Signal:** Speed (ms), Spread, Noise, Bands.
- **Logs, Traces, Watch:** Speed (ms) only.
- **Fetch:** URL of a remote CSV endpoint to stream incrementally.

### Grafana Live

Connects to a Grafana Live channel that streams random data from the server.

| Field       | Description                                |
| ----------- | ------------------------------------------ |
| **Channel** | The live channel to subscribe to. Options: |

| Channel                 | Description                                   |
| ----------------------- | --------------------------------------------- |
| `random-2s-stream`      | Random stream with points every 2s.           |
| `random-flakey-stream`  | Stream that returns data at random intervals. |
| `random-labeled-stream` | Value with moving labels.                     |
| `random-20Hz-stream`    | Random stream with points at 20 Hz.           |

## Data retrieval scenarios

These scenarios fetch data from internal Grafana endpoints.

### Grafana API

Fetches data from internal Grafana API endpoints and returns the result as a data frame. This scenario runs in the browser.

| Field        | Description                         |
| ------------ | ----------------------------------- |
| **Endpoint** | The API endpoint to query. Options: |

| Endpoint         | Description                       |
| ---------------- | --------------------------------- |
| **Data Sources** | Lists configured data sources.    |
| **Search**       | Returns dashboard search results. |
| **Annotations**  | Returns annotations.              |

## Error and edge-case testing scenarios

These scenarios help test how Grafana handles errors, empty data, and slow responses.

### Conditional Error

Produces an error or data depending on the **String Input** and **Error type** fields. When the **String Input** field is empty, the scenario triggers the selected error type. When the field contains CSV values (default: `1,20,90,30,5,0`), it returns time-series data like the CSV Metric Values scenario, regardless of the error type.

| Field          | Description                                                                                                                                                                             |
| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Error type** | Type of error to simulate: `Server panic` (backend panic), `Frontend exception` (uncaught error in the browser), or `Frontend observable` (error emitted through the query observable). |

To trigger an error, clear the **String Input** field and select the error type you want to test.

### Error with source

Returns an error with a configurable source classification.

| Field            | Description                                                                              |
| ---------------- | ---------------------------------------------------------------------------------------- |
| **Error source** | Source of the error: `Plugin` (plugin error) or `Downstream` (downstream service error). |

Use this to test how Grafana differentiates between plugin errors and downstream errors in alerting and error handling.

### Flaky Query

Returns errors intermittently based on a configurable error rate, with optional delays. Use this to test how panels, alert rules, and error handling behave when a data source fails part of the time.

| Field                 | Description                                                                                                                     |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| **Error rate**        | Percentage of requests that return an error. Default: `50`.                                                                     |
| **Status code**       | HTTP status code returned with the error. Default: `400`.                                                                       |
| **Error source**      | Source of the error: `Plugin` (plugin error) or `Downstream` (downstream service error).                                        |
| **Query delay**       | Base delay applied to each request in Go duration syntax, for example `1s` or `500ms`. Default: `5s`.                           |
| **Delay variability** | Randomizes the query delay by plus or minus this percentage. For example, `100` on a `1s` delay sleeps between 0 and 2 seconds. |
| **Error message**     | Message returned with the error. Default: `Flaky query error`.                                                                  |

### Errors and notices

Returns a random walk series with one query-result notice of each severity (info, warning, and error) attached to the frame metadata. Use this to preview how a panel displays query errors and notices.

### No Data Points

Returns an empty result with no data points. Use this to test how panels display when there's no data.

<!-- vale Grafana.Spelling = NO -->

### Datapoints Outside Range

<!-- vale Grafana.Spelling = YES -->

Returns a single data point with a timestamp one hour before the query time range. Use this to test how panels handle data outside the visible range.

### Slow Query

Introduces a configurable delay before returning random walk data. Set the delay duration in the **String Input** field using Go duration syntax (for example, `5s`, `1m`, `500ms`). Default: `5s`.

## Metadata scenarios

These scenarios return metadata rather than time-series data.

### Query Metadata

Returns a table with query metadata including the current user's username. Use this to verify that query context information is available.

# TestData template variables

Use template variables with the TestData data source to test dynamic, reusable dashboards. TestData provides a hierarchical metric tree that you can query to populate variable drop-downs with simulated values.

For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/add-template-variables/) documentation.

## Supported variable types

| Variable type | Supported |
| ------------- | --------- |
| Query         | Yes       |
| Custom        | Yes       |
| Data source   | Yes       |

## Create a query variable

To create a query variable using TestData:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Variable**.
1. Select **Query** as the variable type.
1. Enter a **Name** for your variable.
1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed.
1. Click **Open variable editor** to open the **Query Variable** dialog box.
1. Select the **TestData** data source.
1. Enter your query in the **Query** field.

## Metric tree queries

TestData generates a hierarchical metric tree with nodes named using letter combinations. The tree has 6 levels of depth, with each level containing 8 nodes labeled A through H. The query string navigates this tree using dot-separated path segments.

### Query syntax

Use `*` as a wildcard to return all children at a given level:

| Query    | Result                                                                      |
| -------- | --------------------------------------------------------------------------- |
| `*`      | Returns the top-level nodes: `A`, `B`, `C`, `D`, `E`, `F`, `G`, `H`.        |
| `A.*`    | Returns children of node A: `AA`, `AB`, `AC`, `AD`, `AE`, `AF`, `AG`, `AH`. |
| `A.AA.*` | Returns children of node AA: `AAA`, `AAB`, `AAC`, etc.                      |
| `A.AB`   | Returns children of the AB node: `ABA`, `ABB`, `ABC`, etc.                  |

Use glob patterns with curly braces to match multiple nodes:

| Query         | Result                                                               |
| ------------- | -------------------------------------------------------------------- |
| `{A,B}.*`     | Returns children of both A and B: `AA`, `AB`, ..., `BA`, `BB`, etc.  |
| `A.{AA,AB}.*` | Returns children of AA and AB: `AAA`, `AAB`, ..., `ABA`, `ABB`, etc. |

Use `*` after a prefix to match nodes that start with the given string:

| Query | Result                                                                   |
| ----- | ------------------------------------------------------------------------ |
| `A*`  | Returns all top-level nodes starting with A (only `A` at the top level). |

### Value queries

If your query string starts with `value`, TestData returns the query string itself as the result. This is useful for creating variables with specific values for testing. For example, a query of `value_production` returns `value_production`.

## Example: Create a region variable

This example walks through creating a query variable that populates a drop-down with simulated region values, then using it in a panel query.

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Variable**.
1. Select **Query** as the variable type.
1. Enter a **Name** for your variable (for example, `host`).
1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed.
1. Click **Open variable editor** to open the **Query Variable** dialog box.
1. Select the **TestData** data source.
1. Enter `*` in the **Query** field. This populates the variable with the top-level nodes (A through H).
1. Name the variable `region`.
1. Click **Close** when you've completed the query setup.
1. (Optional) Configure **Multi-value** to allow selecting multiple values.
1. (Optional) Configure **Include All value** to add an "All" option.
1. (Optional) Turn on the **Allow custom values** switch to allow users to enter custom variable values.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

To use the variable in a panel:

1. Add a new panel and select the **TestData** data source.
1. Choose the **Random Walk** scenario.
1. In the **Labels** field, enter `region=$region`.
1. The panel updates when you select a different value from the **region** drop-down.

## Example: Chain variables for cascading drop-downs

You can create dependent variables where one variable's query references another, producing cascading drop-downs.

1. Create a variable named `region` with the query `*`. This returns the top-level nodes (A through H).
1. Create a second variable named `host` with the query `$region.*`. When the user selects `A` as the region, the host drop-down populates with `AA`, `AB`, `AC`, etc.

Changing the region selection automatically refreshes the host drop-down.

## Use variables in queries

TestData interpolates template variables in the following query fields:

- **Labels**
- **Alias**
- **Scenario**
- **String Input**
- **CSV Content**
- **Raw Frame Content**

Use the standard Grafana variable syntax (`$varname` or `${varname}`) in any of these fields to insert variable values.

### Example

With a variable named `env` set to `production`, entering `env=$env` in the **Labels** field produces a series labeled `env=production`.

## Limitations

TestData doesn't support the use of [filters](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/filter-group-by/). Filters require the data source to implement tag key and value lookups, which TestData doesn't provide.

# Troubleshoot TestData data source issues

This document provides solutions to common issues you may encounter when using the TestData data source. TestData is a built-in data source with no external dependencies, so most issues relate to scenario configuration or data formatting.

## Query errors

These errors occur when running TestData scenarios.

### "No data" or empty panels

**Symptoms:**

- Panel displays "No data" message.
- Query executes without error but returns nothing.

**Possible causes and solutions:**

<!-- vale Grafana.Spelling = NO -->

| Cause                                | Solution                                                                                                                       |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
| No Data Points scenario selected     | The No Data Points scenario returns empty results by design. Select a different scenario such as Random Walk.                  |
| Datapoints Outside Range selected    | This scenario returns a data point one hour before the query time range. Expand the time range or select a different scenario. |
| Empty String Input for CSV scenarios | CSV Metric Values and CSV Content require input data. Add comma-separated values or CSV content.                               |
| Time range doesn't contain data      | Some scenarios generate data relative to the query time range. Expand the dashboard time range.                                |

<!-- vale Grafana.Spelling = YES -->

### Exemplars don't appear

**Symptoms:**

- The Exemplars scenario runs without error but no diamond markers appear on the panel.
- Markers disappear when you interact with the legend.

**Possible causes and solutions:**

| Cause                                      | Solution                                                                                                                                                                                           |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No series query in the panel               | Exemplars returns only markers and no series of its own. Add a second query, such as Random Walk, for the exemplars to annotate.                                                                   |
| Series hidden in the legend                | The legend filter matches on exemplar labels, so hiding a series in the legend also hides its markers. Show the series to restore the markers.                                                     |
| Markers fall outside the panel value range | With **Min** and **Max** empty, the value range is derived from a reference random walk and markers can land off-panel. Set **Min** and **Max** explicitly, or align them with the sibling series. |

### Wrong data type for the visualization

**Symptoms:**

- Panel shows "Data does not have a time field" or similar type mismatch errors.
- Visualization renders but looks wrong (for example, a table when you expected a graph).

**Solutions:**

1. Match the scenario to the visualization type. For example, use the Logs scenario with the Logs panel, Node Graph with the Node Graph panel, and Trace with the Traces panel.
1. Time-series panels require scenarios that produce time-series data (Random Walk, Predictable Pulse, CSV Metric Values, Predictable CSV Wave).
1. For table panels, use Table Static, Random Walk Table, or CSV File.

### Predictable Pulse pattern doesn't match expectations

**Symptoms:**

- The on/off pattern doesn't align with expected times.
- Values appear at unexpected intervals.

**Solutions:**

1. Predictable Pulse aligns timestamps to the step interval based on absolute time from the epoch, not from the start of the time range. This means the pattern starts at the same point regardless of when you load the dashboard.
1. Verify the cycle duration matches your intent: the full cycle is `Step * (On Count + Off Count)` seconds.
1. Check that **On Value** and **Off Value** are set correctly. The defaults are `2` and `1`, not `1` and `0`.

### Slow Query appears stuck

**Symptoms:**

- Query runs for an extended period.
- Panel shows a loading spinner indefinitely.

**Solutions:**

1. Check the **String Input** field, which controls the delay duration. The default is `5s`.
1. Reduce the value to a shorter duration (for example, `1s` or `500ms`).
1. The field accepts Go duration syntax: `5s` (seconds), `1m` (minutes), `500ms` (milliseconds).

### Unexpected error from Conditional Error

**Symptoms:**

- Query returns a server error or panic message.
- Error appears immediately when running the query.

**Solutions:**

1. The Conditional Error scenario triggers a server panic when the **String Input** field is empty. This is the intended behavior for error testing.
1. To return data instead of an error, populate the **String Input** field with comma-separated values (for example, `1,20,90,30,5,0`).
1. To change the error type, use the **Error type** drop-down to select between Server panic, Frontend exception, and Frontend observable.

### Error with source returns an error

The Error with source scenario intentionally returns errors for testing how Grafana handles different error sources. This isn't a misconfiguration.

- **Plugin** error source simulates an error originating from the plugin itself.
- **Downstream** error source simulates an error from a downstream service.

Use this scenario to test alerting rules, error displays, and error handling in custom panels.

### Flaky Query returns errors intermittently

The Flaky Query scenario returns errors for a configurable percentage of requests. Intermittent errors are the intended behavior, not a misconfiguration.

**Solutions:**

1. Adjust the **Error rate** field to control how often requests fail. Set it to `0` for no errors or `100` to fail every request.
1. Use the **Status code** and **Error source** fields to shape the returned error.
1. If you're using Flaky Query with an alert rule, be aware that intermittent failures can flip the rule between states across evaluations. For alerting behavior, refer to [TestData alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/alerting/).

## CSV and data input errors

These errors occur with scenarios that accept user-provided data.

### CSV Content or CSV Metric Values return unexpected results

**Symptoms:**

- Data doesn't match expectations.
- Fewer data points than expected.
- Parsing errors.

**Solutions:**

1. Verify CSV formatting: headers in the first row, commas as delimiters, numeric values unquoted.
1. For CSV Metric Values, enter only comma-separated numbers in the **String Input** field (for example, `1,20,90,30,5,0`).
1. The special values `null` and `nan` are supported in Predictable CSV Wave values. Use `null` for missing data points and `nan` for Not-a-Number values.
1. Check for trailing commas or extra whitespace in your input.

### Raw Frames JSON parse errors

**Symptoms:**

- Error message about invalid JSON.
- Panel fails to render.

**Solutions:**

1. Validate that the JSON matches the Grafana data frame format.
1. Use the paste helpers in the editor to import data from panel JSON or raw query results.
1. Verify that field types are consistent within each field array.

## Streaming errors

These errors occur with the Streaming Client and Grafana Live scenarios.

### Streaming Client doesn't update

**Symptoms:**

- Panel shows initial data but doesn't update.
- No new data points appear over time.

**Solutions:**

1. Verify the **Speed** field is set to a reasonable value (in milliseconds). Lower values produce faster updates.
1. For the **Fetch** type, verify the URL is accessible from the browser.
1. Check the browser console for WebSocket or network errors.

### Grafana Live channel shows no data

**Symptoms:**

- Panel remains empty after selecting a channel.
- No streaming data appears.

**Solutions:**

1. Confirm the selected channel matches one of the available options: `random-2s-stream`, `random-flakey-stream`, `random-labeled-stream`, or `random-20Hz-stream`.
1. Verify that Grafana Live is enabled in your Grafana instance.
1. Check the Grafana server logs for Live connection errors.

## Template variable errors

These errors occur when using TestData with [dashboard template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/template-variables/).

### Variable drop-down is empty

**Symptoms:**

- Variable drop-down shows no options.
- Preview in the variable editor returns zero values.

**Solutions:**

1. Verify the query string navigates valid nodes in the metric tree. Use `*` to return all top-level nodes.
1. Check that dot-separated path segments match the tree structure (for example, `A.*` returns children of node A). An invalid path like `Z.*` returns nothing because `Z` doesn't exist in the tree.
1. Verify the TestData data source is selected in the variable editor.
1. Click **Run query** in the variable editor preview to test.

### Chained variable doesn't update

**Symptoms:**

- A dependent variable (for example, `$region.*`) doesn't refresh when the parent variable changes.
- The drop-down shows stale values from a previous selection.

**Solutions:**

1. Set the dependent variable's **Refresh** option to **On dashboard load** or **On time range change** in the variable editor.
1. Verify the query uses the correct parent variable syntax (for example, `$region.*` not `region.*`).

### Variable value not interpolated in queries

**Symptoms:**

- Panel shows the literal string `$varname` instead of the variable value.
- Labels or data don't change when the variable selection changes.

**Solutions:**

1. Verify you're using the correct variable syntax: `$varname` or `${varname}`.
1. TestData interpolates variables in these fields only: **Labels**, **Alias**, **Scenario**, **String Input**, **CSV Content**, and **Raw Frame Content**. Other fields don't support variable substitution.

## Alerting errors

These errors occur when using TestData with [Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/alerting/).

### Alert rule shows "no data" unexpectedly

**Symptoms:**

- Alert rule status shows **No Data** even though the scenario works in a panel.
- Alert never transitions to firing or normal.

**Solutions:**

1. Verify you're using a backend-evaluated scenario. Browser-only scenarios (Streaming Client, Grafana Live, Grafana API, Steps, Annotations, Node Graph, Flame Graph, Trace, and Raw Frames) return empty results when evaluated by the alerting engine. Use a backend scenario like Random Walk or Predictable Pulse instead.
1. Confirm the scenario is returning time-series data. Scenarios that produce logs (Logs), traces (Trace), graphs (Node Graph, Flame Graph), or tables (Table Static) can't be used as alert conditions.
1. Check that the **String Input** field isn't empty for scenarios that require it (for example, CSV Metric Values).

### Alert rule never fires

**Symptoms:**

- Alert rule stays in **Normal** state.
- Data is visible in the panel but the alert condition is never met.

**Solutions:**

1. If using Random Walk, the values are non-deterministic and may not exceed your threshold. Use Predictable Pulse or CSV Metric Values with values that reliably cross the threshold.
1. Verify the **Reduce** expression is aggregating correctly. For example, **Last** returns the most recent value, while **Mean** averages the series.
1. Check the **Threshold** expression. Click **Preview** in the alert rule editor to see the evaluated value.
1. Ensure the evaluation interval is long enough for the scenario to produce data points. A Predictable Pulse with Step=60 needs at least a 60-second evaluation window.

### Template variables don't resolve in alert rules

**Symptoms:**

- Alert rule error mentions unresolved variables.
- Alert query uses `$varname` syntax but the variable isn't replaced.

**Solutions:**

Grafana evaluates alert rules on the backend without dashboard context. Template variables like `$varname` aren't resolved during alert evaluation. Replace variable references with fixed values in the scenario options.

For more information, refer to the [alerting limitations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/alerting/#template-variables-arent-supported-in-alert-queries).

### Alert evaluation timeout

**Symptoms:**

- Alert rule transitions to **Error** state.
- Logs show a timeout during query evaluation.

**Solutions:**

1. If you're using the Slow Query scenario, reduce the delay in the **String Input** field to a value shorter than the evaluation interval.
1. For other scenarios, verify that the query isn't producing an excessive number of data points. Reduce the time range or increase the step interval.

## Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location).
1. Look for entries containing `testdata` or `testdatasource` for scenario-specific details.
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you've tried the solutions in this document and still encounter issues:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs.
1. When reporting issues, include:
   - Grafana version
   - Selected scenario and its configuration
   - Error messages (redact sensitive information)
   - Steps to reproduce

# Configure the Zipkin data source

This document explains how to configure the Zipkin data source in Grafana, including connection settings, authentication, trace-to-logs and trace-to-metrics integrations, and provisioning with YAML or Terraform.

## Before you begin

Before configuring the data source, ensure you have:

- **Grafana permissions:** Organization administrator role
- **Zipkin instance:** A running Zipkin instance accessible from your Grafana server

## Add the data source

To add the Zipkin data source:

1. Click **Connections** in the left-side menu.
1. Click **Add new connection**.
1. Type `Zipkin` in the search bar.
1. Select **Zipkin**.
1. Click **Add new data source**.

## Configure settings

The following table describes the basic configuration settings for the Zipkin data source.

| Setting     | Description                                                      |
| ----------- | ---------------------------------------------------------------- |
| **Name**    | The name used to refer to the data source in panels and queries. |
| **Default** | Toggle to make this the default data source for new panels.      |
| **URL**     | The URL of the Zipkin instance, such as `http://localhost:9411`. |

## Authentication

The Zipkin data source supports the following authentication methods:

- **Basic authentication:** Provide a username and password to authenticate with the Zipkin instance.
- **TLS client authentication:** Configure client certificates for mutual TLS.
- **Forward OAuth identity:** Forward the user's OAuth token to the Zipkin instance.
- **With credentials:** Send credentials (cookies, TLS client certificates) with cross-site requests.

You can also configure custom headers and TLS settings in the **Advanced HTTP settings** section, which is located inside **Additional settings**.

## Trace to logs

{{< figure src="/media/docs/tempo/tempo-trace-to-logs-9-4.png" max-width="800px" class="docs-image--no-shadow" caption="Trace to logs settings" >}}

{{< admonition type="note" >}}
If you use Grafana Cloud, open a [support ticket in the Cloud Portal](/profile/org#support) to access this feature.
{{< /admonition >}}

The **Trace to logs** setting configures the [trace to logs feature](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/trace-integration/) that is available when you integrate Grafana with Zipkin. This feature lets you navigate from a span in a trace view directly to the relevant logs.

There are two ways to configure the trace to logs feature:

- Use a simplified configuration with a default query, or
- Configure a custom query where you can use a [template language](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/) to interpolate variables from the trace or span.

### Use a simple configuration

1. Select the target data source from the drop-down list.

   You can also click **Open advanced data source picker** to see more options, including adding a data source.

1. Set start and end time shift. Since the log timestamps may not exactly match the timestamps of the spans in the trace, you may need to widen or shift the time range to find the desired logs.
1. Select which tags to use in the logs query. The tags you configure must be present in the span attributes or resources for a trace to logs span link to appear. You can optionally configure a new name for the tag. This is useful if the tag has dots in the name and the target data source doesn't allow dots in labels. For example, you can remap `http.status` to `http_status`.
1. Optionally, switch on the **Filter by trace ID** and/or **Filter by span ID** setting to further filter the logs if your logs consistently contain trace or span IDs.

### Configure a custom query

1. Select the target data source from the drop-down list.

   You can also click **Open advanced data source picker** to see more options, including adding a data source.

1. Set start and end time shift. Since the log timestamps may not exactly match the timestamps of the spans in the trace, you may need to widen or shift the time range to find the desired logs.
1. Optionally, select tags to map. These tags can be used in the custom query with the `${__tags}` variable. This variable interpolates the mapped tags as a list in an appropriate syntax for the data source and only includes tags that are present in the span. You can optionally configure a new name for the tag. This is useful when the tag has dots in the name and the target data source doesn't allow dots in labels. For example, you can remap `http.status` to `http_status`. If you don't map any tags here, you can still use any tag in the query like this: `method="${__span.tags.method}"`.
1. Skip **Filter by trace ID** and **Filter by span ID** settings as these can't be used with a custom query.
1. Switch on **Use custom query**.
1. Specify a custom query to be used to query the logs. You can use various variables to make the query relevant for the current span. The link only appears if all the variables are interpolated with non-empty values to prevent creating an invalid query.

### Variables for custom queries

To use a variable, wrap it in `${}`. For example: `${__span.name}`.

| Variable name          | Description                                                                                                                                                                                                                                                                                                                             |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **\_\_tags**           | This variable uses the tag mapping from the UI to create a label matcher string in the specific data source syntax. The variable only uses tags that are present in the span. The link is still created even if only one of those tags is present in the span. You can use this if all tags aren't required for the query to be useful. |
| **\_\_span.spanId**    | The ID of the span.                                                                                                                                                                                                                                                                                                                     |
| **\_\_span.traceId**   | The ID of the trace.                                                                                                                                                                                                                                                                                                                    |
| **\_\_span.duration**  | The duration of the span.                                                                                                                                                                                                                                                                                                               |
| **\_\_span.name**      | The name of the span.                                                                                                                                                                                                                                                                                                                   |
| **\_\_span.tags**      | Namespace for the tags in the span. To access a specific tag named `version`, you would use `${__span.tags.version}`. If the tag contains a dot, access it as `${__span.tags["http.status"]}`.                                                                                                                                          |
| **\_\_trace.traceId**  | The ID of the trace.                                                                                                                                                                                                                                                                                                                    |
| **\_\_trace.duration** | The duration of the trace.                                                                                                                                                                                                                                                                                                              |
| **\_\_trace.name**     | The name of the trace.                                                                                                                                                                                                                                                                                                                  |

### Trace to logs settings

The following table describes the trace to logs configuration options.

| Setting name              | Description                                                                                                                                                                                                                                                                                     |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Data source**           | Defines the target data source. You can select Loki, Elasticsearch, Splunk, OpenSearch, FalconLogScale, Google Cloud Logging, and VictoriaMetrics Logs data sources.                                                                                                                            |
| **Span start time shift** | Shifts the start time for the logs query, based on the span's start time. You can use time units, such as `5s`, `1m`, `3h`. To extend the time to the past, use a negative value. Default: `0`.                                                                                                 |
| **Span end time shift**   | Shifts the end time for the logs query, based on the span's end time. You can use time units. Default: `0`.                                                                                                                                                                                     |
| **Tags**                  | Defines the tags to use in the logs query. Default: `cluster`, `hostname`, `namespace`, `pod`, `service.name`, `service.namespace`. You can change the tag name to remove dots from the name if they aren't allowed in the target data source. For example, map `http.status` to `http_status`. |
| **Filter by trace ID**    | Toggles whether to append the trace ID to the logs query.                                                                                                                                                                                                                                       |
| **Filter by span ID**     | Toggles whether to append the span ID to the logs query.                                                                                                                                                                                                                                        |
| **Use custom query**      | Toggles use of custom query with interpolation.                                                                                                                                                                                                                                                 |
| **Query**                 | Input to write a custom query. Use variable interpolation to customize it with variables from the span.                                                                                                                                                                                         |

## Trace to metrics

The **Trace to metrics** setting lets you navigate from a span in a trace view to a metrics query in a configured metrics data source.

To configure trace to metrics:

1. Select the target data source from the drop-down list.

   You can also click **Open advanced data source picker** to see more options, including adding a data source.

1. Create any desired linked queries.

| Setting name    | Description                                                                                                                                                                                                                                                     |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Data source** | Defines the target data source.                                                                                                                                                                                                                                 |
| **Tags**        | Defines the tags used in linked queries. The key sets the span attribute name, and the optional value sets the corresponding metric label name. For example, you can map `k8s.pod` to `pod`. To interpolate these tags into queries, use the `$__tags` keyword. |

Each linked query consists of:

- **Link Label:** _(Optional)_ Descriptive label for the linked query.
- **Query:** The query run when navigating from a trace to the metrics data source. Interpolate tags using the `$__tags` keyword. For example, when you configure the query `requests_total{$__tags}` with the tags `k8s.pod=pod` and `cluster`, the result looks like `requests_total{pod="nginx-554b9", cluster="us-east-1"}`.

## Additional settings

The **Additional settings** section is collapsible and contains optional settings for the node graph, span bar, advanced HTTP configuration, and secure SOCKS proxy (when enabled in your Grafana configuration).

### Node graph

The **Enable node graph** toggle enables the [Node graph visualization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/node-graph/), which is disabled by default.

After you enable it, Grafana displays the node graph above the trace view.

### Span bar

The **Span bar** setting lets you display additional information in the span bar row.

The **Label** drop-down has three options:

| Label        | Description                                                                                                                  |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| **None**     | Adds nothing to the span bar row.                                                                                            |
| **Duration** | _(Default)_ Displays the span duration on the span bar row.                                                                  |
| **Tag**      | Displays the span tag on the span bar row. Enter the **Tag key** to specify which tag value to display, such as `component`. |

## Verify the connection

Click **Save & test** to verify the connection. A successful connection displays the message **Data source is working**.

If you encounter errors, refer to [Troubleshoot Zipkin data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/zipkin/troubleshooting/).

## Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system.
For more information about provisioning and available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources).

### Provisioning example

```yaml
apiVersion: 1

datasources:
  - name: Zipkin
    type: zipkin
    uid: zipkin-ds
    url: http://localhost:9411
    access: proxy
    basicAuth: true
    basicAuthUser: my_user
    readOnly: false
    isDefault: false
    jsonData:
      tracesToLogsV2:
        datasourceUid: 'loki'
        spanStartTimeShift: '1h'
        spanEndTimeShift: '-1h'
        tags: ['job', 'instance', 'pod', 'namespace']
        filterByTraceID: false
        filterBySpanID: false
        customQuery: true
        query: 'method="$${__span.tags.method}"'
      tracesToMetrics:
        datasourceUid: 'prom'
        spanStartTimeShift: '1h'
        spanEndTimeShift: '-1h'
        tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }]
        queries:
          - name: 'Sample query'
            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))'
      nodeGraph:
        enabled: true
      spanBar:
        type: 'None'
    secureJsonData:
      basicAuthPassword: my_password
```

## Configure with Terraform

You can configure the Zipkin data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation.

### Basic Terraform example

The following example creates a basic Zipkin data source:

```hcl
resource "grafana_data_source" "zipkin" {
  name = "Zipkin"
  type = "zipkin"
  url  = "http://localhost:9411"
}
```

### Terraform example with trace to logs

The following example includes a trace to logs configuration that links traces to a Loki data source:

```hcl
resource "grafana_data_source" "zipkin" {
  name = "Zipkin"
  type = "zipkin"
  url  = "http://localhost:9411"

  json_data_encoded = jsonencode({
    tracesToLogsV2 = {
      datasourceUid    = grafana_data_source.loki.uid
      spanStartTimeShift = "1h"
      spanEndTimeShift   = "-1h"
      filterByTraceID  = true
      filterBySpanID   = false
      tags = [
        { key = "service.name", value = "service" },
        { key = "namespace" }
      ]
    }
    nodeGraph = {
      enabled = true
    }
  })
}
```

### Terraform example with basic authentication

The following example includes basic authentication:

```hcl
resource "grafana_data_source" "zipkin" {
  name = "Zipkin"
  type = "zipkin"
  url  = "http://localhost:9411"

  basic_auth_enabled  = true
  basic_auth_username = "zipkin_user"

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = var.zipkin_password
  })

  json_data_encoded = jsonencode({
    nodeGraph = {
      enabled = true
    }
  })
}
```

For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source).

# Zipkin query editor

This document explains how to use the Zipkin query editor to query and visualize traces. You can look up traces by ID, browse available traces using the cascading service and span selector, or upload a JSON trace file for visualization.

For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/).

## Query by trace ID

To query a specific trace by its ID:

1. Select the **TraceID** query type.
1. Enter the trace ID into the **Trace ID** field.
1. Press **Shift+Enter** to run the query.

Zipkin trace IDs are 16 or 32 character hexadecimal strings, for example `efe9cb8857f68c8f` or `463ac35c9f6413ad48485a3953bb6124`.

You can also use [template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/zipkin/template-variables/) in the trace ID field. For example, `${traceId}` is replaced with the variable's current value when the query runs.

## Query by trace selector

You can browse and select traces using the **Traces** cascading selector instead of entering a trace ID manually. This is useful when you want to explore traces without knowing a specific trace ID.

The trace selector has three levels:

1. **Service:** Select the service you're interested in, such as `frontend` or `api-gateway`.
1. **Span:** Select a specific operation within the selected service, such as `GET /api/users`.
1. **Trace:** Select a specific trace in which the selected operation occurred, represented by the root operation name and trace duration (for example, `get /api [23 ms]`).

When you select a trace at the third level, the trace ID is automatically populated and the query runs.

{{< admonition type="note" >}}
The trace selector lists traces within the currently selected Explore time range. If you don't see expected traces, try expanding the time range.
{{< /admonition >}}

## Upload a JSON trace file

You can upload a JSON file that contains a single trace and visualize it. If the file has multiple traces, Grafana visualizes the first trace.

To upload a trace file:

1. Click **Import trace**.
1. Drag and drop a JSON file or click to browse for a file.

The file must contain a JSON array of spans in the [Zipkin v2 span format](https://zipkin.io/zipkin-api/#/).

### Single-span trace example

```json
[
  {
    "traceId": "efe9cb8857f68c8f",
    "id": "efe9cb8857f68c8f",
    "kind": "SERVER",
    "name": "get /api",
    "timestamp": 1627975249601797,
    "duration": 23457,
    "localEndpoint": { "serviceName": "backend", "ipv4": "127.0.0.1", "port": 9000 },
    "tags": {
      "http.method": "GET",
      "http.path": "/api"
    }
  }
]
```

### Multi-span trace example

The following example shows a trace with a parent span and a child span, representing a frontend service calling a backend service:

```json
[
  {
    "traceId": "efe9cb8857f68c8f",
    "id": "efe9cb8857f68c8f",
    "kind": "SERVER",
    "name": "get /api",
    "timestamp": 1627975249601797,
    "duration": 23457,
    "localEndpoint": { "serviceName": "frontend", "ipv4": "127.0.0.1", "port": 8080 },
    "tags": {
      "http.method": "GET",
      "http.path": "/api",
      "http.status_code": "200"
    }
  },
  {
    "traceId": "efe9cb8857f68c8f",
    "parentId": "efe9cb8857f68c8f",
    "id": "8608dc6ce5cafe8e",
    "kind": "CLIENT",
    "name": "get /api/data",
    "timestamp": 1627975249602000,
    "duration": 18200,
    "localEndpoint": { "serviceName": "backend", "ipv4": "127.0.0.1", "port": 9000 },
    "tags": {
      "http.method": "GET",
      "http.path": "/api/data",
      "http.status_code": "200"
    }
  }
]
```

## View data mapping in the trace UI

Zipkin annotations appear in the trace view as logs with the annotation value displayed under the annotation key. This lets you view timestamped events that occurred during a span, such as retry attempts or cache misses.

## Span filters

{{< figure src="/media/docs/tempo/screenshot-grafana-tempo-span-filters-v10-1.png" max-width="800px" class="docs-image--no-shadow" caption="Span filters in the trace timeline viewer" >}}

Using span filters, you can filter spans in the trace timeline viewer. The more filters you add, the more specific the filtered spans are.

You can add one or more of the following filters:

| Filter           | Description                                       | Example                   |
| ---------------- | ------------------------------------------------- | ------------------------- |
| **Service name** | Filter by the service that produced the span.     | `frontend`                |
| **Span name**    | Filter by the operation name.                     | `GET /api/users`          |
| **Duration**     | Filter by minimum and/or maximum span duration.   | Min: `10ms`, Max: `500ms` |
| **Tags**         | Filter by span tags, process tags, or log fields. | `http.status_code=500`    |

To only show the spans you have matched, toggle on **Show matches only**.

### Span filter examples

- **Find slow database calls:** Set **Service name** to your database service and **Duration** minimum to `100ms`.
- **Find errors:** Add a **Tags** filter for `error=true` or `http.status_code=500`.
- **Isolate a specific service:** Set **Service name** and toggle on **Show matches only** to hide all other spans.

## Link to a trace ID from logs

You can link to Zipkin traces from logs in Loki, Elasticsearch, Splunk, and other logs data sources by configuring an internal link.

To configure this feature, refer to the [Derived fields](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure-loki-data-source/#derived-fields) section of the Loki data source docs or the [Data links](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/elasticsearch/configure/#data-links) section of the Elasticsearch or Splunk data source docs.

## Link to a trace ID from metrics

You can link to Zipkin traces from metrics in Prometheus data sources by configuring an exemplar.

To configure this feature, refer to the [introduction to exemplars](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/) documentation.

## Troubleshoot query issues

If you encounter issues with queries, refer to [Troubleshoot Zipkin data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/zipkin/troubleshooting/).

# Zipkin template variables

Use template variables to create dynamic, reusable dashboards. Instead of hard-coding trace IDs, you can use variables to let dashboard viewers select or input trace IDs at view time, making it easier to share dashboards across teams.

For an introduction to Grafana template variables, refer to [Variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/).

## Supported variable types

| Variable type | Description                                       | Supported |
| ------------- | ------------------------------------------------- | --------- |
| Custom        | Define a fixed list of trace IDs or other values. | Yes       |
| Text box      | Let users type a trace ID at dashboard view time. | Yes       |
| Data source   | Let users switch between Zipkin instances.        | Yes       |
| Query         | Populate options from a data source query.        | No        |

The Zipkin data source doesn't support query-based variables, but you can use custom or text box variables to parameterize trace ID queries.

## Create a text box variable

A text box variable lets dashboard viewers enter a trace ID directly. This is the most common variable type for Zipkin.

To create a text box variable:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Variable**.
1. Select **Textbox** as the variable type.
1. Set the **Name** to `traceId`.
1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed.
1. (Optional) Set a default **Value** with a known trace ID for initial display.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

The dashboard displays a text input where viewers can paste a trace ID.

## Create a custom variable

A custom variable provides a predefined list of values. This is useful when you want to give viewers a set of known trace IDs to choose from.

To create a custom variable:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Variable**.
1. Select **Custom** as the variable type.
1. Set the **Name** to `traceId`.
1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed.
1. Click **Open variable editor** to open the **Custom options** dialog box.
1. Enter trace IDs in the **Values separated by comma** field:

   ```
   efe9cb8857f68c8f,463ac35c9f6413ad48485a3953bb6124
   ```

1. Click **Apply**.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

The dashboard displays a drop-down where viewers can select from the predefined trace IDs.

## Create a data source variable

A data source variable lets viewers switch between multiple Zipkin instances. This is useful when you have separate Zipkin deployments for different environments.

To create a data source variable:

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Variable**.
1. Select **Data source** as the variable type.
1. Set the **Name** to `zipkinDs`.
1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed.
1. Select **Zipkin** as the **Type**.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

Then, in your panel's query editor, select `${zipkinDs}` as the data source instead of a specific Zipkin instance.

## Use variables in queries

You can use template variables in the **Trace ID** field of the query editor. Grafana replaces the variable with its current value when the query runs.

For example, if you create a text box variable named `traceId`, enter `${traceId}` in the trace ID field. When the viewer enters a new trace ID, the query automatically runs with the new value.

For more information about variable syntax, refer to [Variable syntax](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/).

## Troubleshoot template variable issues

If template variables aren't working as expected, refer to [Troubleshoot Zipkin data source issues](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/zipkin/troubleshooting/).

# Troubleshoot Zipkin data source issues

This document provides solutions to common issues you may encounter when configuring or using the Zipkin data source, including connection failures, query errors, JSON upload issues, and configuration problems. For configuration instructions, refer to [Configure Zipkin](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/zipkin/configure/).

## Connection errors

These errors occur when Grafana can't connect to the Zipkin instance.

### `error reading settings: url is empty`

**Symptoms:**

- **Save & test** fails immediately
- Health check returns an error

**Solutions:**

1. Open the data source settings and verify the **URL** field isn't empty.
1. Enter the full URL of your Zipkin instance, such as `http://localhost:9411`.

### "request failed: 404 Not Found" or other HTTP status errors

**Symptoms:**

- **Save & test** fails with an HTTP status error
- Trace queries return errors

**Possible causes and solutions:**

| Cause                          | Solution                                                                                                                             |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ |
| Incorrect URL or port          | Verify the URL and port. The default Zipkin port is `9411`. Ensure you haven't accidentally used the Jaeger port (`16686`).          |
| Missing `/api/v2` path         | The Zipkin data source calls `/api/v2/services`, `/api/v2/traces`, etc. Verify these endpoints are accessible at the configured URL. |
| Reverse proxy misconfiguration | If using a reverse proxy, ensure it correctly forwards requests to the Zipkin API.                                                   |
| Authentication failure         | If the Zipkin instance requires authentication, verify your credentials are correct.                                                 |

### `error creating http client` or TLS errors

**Symptoms:**

- **Save & test** fails with a connection or TLS error
- HTTPS connections are refused

**Solutions:**

1. If using HTTPS, verify the TLS certificate is valid and trusted by the Grafana server.
1. Configure TLS settings in **Additional settings** > **Advanced HTTP settings**.
1. For self-signed certificates, add the CA certificate to the Grafana server trust store or toggle **Skip TLS Verify** (not recommended for production).

### Save & test times out

**Symptoms:**

- **Save & test** hangs and eventually times out
- No error message is returned

**Possible causes and solutions:**

| Cause                     | Solution                                                                                                                          |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Zipkin instance is down   | Confirm the Zipkin instance is running and accessible from the Grafana server.                                                    |
| Firewall or network rules | Ensure the Grafana server can reach the Zipkin instance. Check that firewall rules allow outbound traffic on the configured port. |
| DNS resolution failure    | Verify the hostname in the URL resolves correctly from the Grafana server.                                                        |

For Grafana Cloud, if you're accessing a private Zipkin instance, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/).

## Query editor errors

These errors occur when using the query editor in Explore or dashboards.

### "Failed to load spans from Zipkin"

**Symptoms:**

- Error message appears in the query editor
- The **Traces** cascading selector doesn't populate services, spans, or traces

This error occurs when the frontend can't reach the Zipkin API endpoints (`/api/v2/services`, `/api/v2/spans`, or `/api/v2/traces`).

**Solutions:**

1. Verify the Zipkin instance is running and accessible.
1. Test the endpoint directly: `curl http://<ZIPKIN_URL>/api/v2/services`.
1. Check that the Grafana server can reach the Zipkin API.
1. Review the Grafana server logs for more details. Look for `An error occurred while doing a resource call` in the logs.

### "An error occurred within the plugin"

**Symptoms:**

- A generic error appears when querying or browsing traces
- No specific error details are shown in the UI

This is a generic error returned by the Zipkin plugin backend when an internal error occurs. The actual error details are only available in the Grafana server logs.

**Solutions:**

1. Enable [debug logging](#enable-debug-logging) to capture detailed error information.
1. Check the Grafana server logs for messages like `An error occurred while doing a resource call` or `An error occurred while processing response from resource call`.
1. Verify the Zipkin instance is healthy and responding to API requests.

### "No data" or empty results

**Symptoms:**

- Query runs without error but returns no data
- Trace view shows "No data"
- The cascading selector shows "No traces found" or "[No traces in time range]"

**Possible causes and solutions:**

| Cause                           | Solution                                                                                                                                   |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Invalid trace ID                | Verify the trace ID is correct. Zipkin trace IDs are 16 or 32 character hex strings.                                                       |
| Empty trace ID                  | Ensure the **Trace ID** field isn't empty. The backend rejects queries with `invalid/empty traceId`.                                       |
| Time range doesn't contain data | Expand the dashboard or Explore time range. Traces must fall within the selected time range for the cascading selector to find them.       |
| Trace has expired               | Zipkin may have purged old trace data based on its retention settings. Check your Zipkin storage configuration.                            |
| No services registered          | If the cascading selector shows "No traces found" at the top level, your Zipkin instance may have no data. Verify data is being collected. |

## Upload errors

These errors occur when importing a JSON trace file.

### "JSON is not valid Zipkin format"

**Symptoms:**

- Error appears after uploading a JSON trace file
- The trace doesn't render

**Possible causes and solutions:**

| Cause                | Solution                                                                                                                                                      |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Invalid JSON syntax  | Validate the file with a JSON linter. Common issues include trailing commas, missing quotes, or unclosed brackets.                                            |
| Wrong JSON structure | The file must be a JSON array of span objects (`[{...}, {...}]`), not a single object. Refer to the [Zipkin v2 span format](https://zipkin.io/zipkin-api/#/). |
| Wrong file type      | Ensure you're uploading a `.json` file, not a different format.                                                                                               |

### `unsupported query type upload. only available in frontend mode`

**Symptoms:**

- Error when trying to use uploaded trace data in a dashboard panel or alert

Upload queries only work in **Explore**. They can't be used in dashboard panels or alerting rules because the upload is processed in the browser, not the backend.

**Solution:**

Use the **TraceID** query type with the trace ID from your uploaded trace if you need to display it in a dashboard. This requires the trace to exist in your Zipkin instance.

## Configuration errors

These errors relate to data source configuration settings.

### "Invalid time shift. See tooltip for examples."

**Symptoms:**

- Error appears in the trace to logs or trace to metrics configuration
- The **Span start time shift** or **Span end time shift** field is highlighted

**Solutions:**

1. Use valid time unit formats: `5s`, `1m`, `3h`, `-30m`.
1. Use a negative value to shift the time to the past (for example, `-1h`).
1. The value must be a number followed by a time unit (`s`, `m`, `h`).

### Trace to logs links don't appear

**Symptoms:**

- No log links appear in the trace view
- Links appear for some spans but not others

**Possible causes and solutions:**

| Cause                               | Solution                                                                                                                                                               |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Tags not present in span            | The tags configured in trace to logs settings must exist in the span attributes for the link to appear. Verify the span contains the expected tags.                    |
| Logs data source not configured     | Ensure the target logs data source (Loki, Elasticsearch, Splunk, OpenSearch, FalconLogScale, Google Cloud Logging, or VictoriaMetrics Logs) is configured and working. |
| Custom query variables not resolved | If using a custom query, the link only appears when all variables resolve to non-empty values. Verify the span contains the referenced tags.                           |
| Wrong data source type selected     | The trace to logs data source must be a supported logs data source. Other data source types don't appear in the drop-down.                                             |

### Trace to metrics links don't appear

**Symptoms:**

- No metrics links appear in the trace view

**Solutions:**

1. Verify the target metrics data source is configured and working.
1. Ensure linked queries are defined with valid query syntax.
1. Check that the `$__tags` keyword in queries maps to tags that exist in the span.

## Template variable errors

These errors relate to using template variables with the Zipkin data source.

### Variable value not substituted in query

**Symptoms:**

- The trace ID field shows the variable syntax (for example, `${traceId}`) instead of the actual value
- Query returns no results when using a variable

**Solutions:**

1. Navigate to the dashboard you want to update and click **Edit**.
1. Click the **Dashboard options** icon to open the sidebar.
1. Expand the **Variables** section.
1. Verify the variable is defined.
1. Check the variable name matches the syntax used in the query. Variable names are case-sensitive.
1. Ensure the variable has a value selected or entered.
1. For text box variables, verify the viewer has entered a value.

## Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to `debug` in the configuration file:

   ```ini
   [log]
   level = debug
   ```

1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location).
1. Look for Zipkin-specific entries that include request and response details, such as:
   - `Failed to close response body`
   - `An error occurred while doing a resource call`
   - `An error occurred while processing response from resource call`
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.

## Get additional help

If you've tried these solutions and still encounter issues:

1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs.
1. Consult the [Zipkin documentation](https://zipkin.io/) for service-specific guidance.
1. Contact Grafana Support if you have a paid plan. [Open a support ticket](https://grafana.com/profile/org#support) for Grafana Cloud, or [contact support](https://grafana.com/contact/) for Grafana Enterprise.
1. When reporting issues, include:
   - Grafana version
   - Error messages (redact sensitive information)
   - Steps to reproduce
   - Relevant configuration (redact credentials)

# Developer Resources

# Migrate to the new Grafana APIs

{{< admonition type="note" >}}
New APIs are available in Grafana 12 and later.
Legacy APIs are deprecated starting in Grafana 13.
{{< /admonition >}}

Grafana is migrating existing APIs to the new `/apis` model, a Kubernetes-style API layer which follows a standardized API structure alongside consistent API versioning. Refer to the [New API structure in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis) documentation for more details.

**Legacy APIs are not being disabled for the moment**. Removal of legacy APIs is planned for a future major release, and any breaking changes will be announced well in advance to avoid disruptions.

## Deprecation notes

The API migration process is underway and there may not be an exact `/apis` match to the legacy API you're using. Some legacy APIs may not be migrated at all. For details, refer to [List of available HTTP APIs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api#list-of-available-http-apis) table.

### Query history API

The [Query History API](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/query_history) will not be migrated.

This functionality is being deprecated. Grafana will revert to using local on-device storage for this functionality, since this approach reduces the amount of traffic to the backend with minimal change in functionality. If you're using this API, consider using a similar approach.

# The new HTTP API structure in Grafana

{{< admonition type="note" >}}
Available in Grafana 12 and later.
{{< /admonition >}}

This document explains how Grafana structures the `/apis` HTTP APIs, which use a Kubernetes-style API layer with a standardized structure and consistent versioning. Read on to understand request paths, versions, namespaces, and common response fields when you migrate from legacy `/api` endpoints or work with new APIs.

For information about a specific API, refer to its doc. You can find them in [List of available HTTP APIs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api#list-of-available-http-apis).

## Before you begin

Before you begin, ensure you have the following:

- **Grafana version:** Use Grafana 12 or later.
- **API access:** Make sure you can send HTTP requests to your Grafana instance.
- **Legacy API context:** Know which legacy `/api` endpoints you want to replace if you're planning a migration.

## Migrate from legacy `api` endpoints

Grafana 13 deprecates legacy API endpoints (`/api`) in favor of a new generation of improved APIs (`/apis`). **Legacy APIs are not being disabled for the moment**. Removal of legacy APIs is planned for a future major release, and any breaking changes will be announced well in advance to avoid disruptions.

For more information, refer to [Migrate to the new APIs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis-migration).

## API structure

Grafana APIs use a common path format and a common response shape.

### API path

All Grafana APIs follow this standardized format:

```text
/apis/<GROUP>/<VERSION>/namespaces/<NAMESPACE>/<RESOURCE>[/<NAME>]
```

Replace the placeholders with values for your API group, version, namespace, resource, and optional resource name. Use the final `/<NAME>` segment for operations on individual resources such as get, update, or delete. Omit it for collection operations such as list or create.

### API response format

All Grafana API responses follow this structure:

```json
{
  "kind": "<KIND>",
  "apiVersion": "<GROUP>/<VERSION>",
  "metadata": {
    "name": "<NAME>",
    "namespace": "<NAMESPACE>",
    "uid": "db323171-c78a-42fa-be98-16a3d799a779",
    "resourceVersion": "1758777451428472",
    "generation": 10,
    "creationTimestamp": "2026-01-23T22:06:40Z",
    "annotations": {}
  },
  "spec": {
    // resource-specific fields
  }
}
```

Replace the placeholders with the values returned by the resource you request.

## Understand the components

Each part of the path and response has a specific purpose.

### Group (`<group>`)

Groups organize related functionality into logical collections. For example, `dashboard.grafana.app` is used for dashboard-related operations.

### Version (`<version>`)

Grafana APIs use semantic versioning with three stability levels:

| Level | Format     | Description                                                                 | Use Case                 | Enabled By Default? |
| ----- | ---------- | --------------------------------------------------------------------------- | ------------------------ | ------------------- |
| Alpha | `v1alpha1` | Early development stage. Unstable, may contain bugs, and subject to removal | For testing new features | No                  |
| Beta  | `v1beta1`  | More stable than alpha, but may still have some changes                     | For non-critical use     | No                  |
| GA    | `v1`       | Generally Available. Stable with backward compatibility guarantees          | For production use       | Yes                 |

#### Alpha

Alpha versions should not be served unless explicitly enabled by a feature flag, and should be considered completely experimental and subject to major changes.
An alpha version may undergo breaking changes without adding an additional version, and should not be relied upon by production workflows. Alpha versions may be removed completely, even without being promoted to a more stable level (e.g. an experimental API may be introduced as alpha for a new feature and subsequently removed completely, in case that feature gets canceled).

#### Beta

Beta versions should not contain breaking changes in the schema, but still may be subject to changes in handling logic or semantics.
Breaking schema changes require a new published beta version (such as publishing `v1beta2` for breaking changes to the `v1beta1` schema).
While beta versions are no longer considered experimental like alpha versions, they should still be disabled by default.

#### GA

GA versions are enabled by default, and can be treated as completely stable. The only changes that can be made to these APIs are bug fixes, and any other changes should instead result in a new published version of the API.

### Namespace (`<namespace>`)

Namespaces isolate resources within your Grafana instance. The format varies by deployment type:

#### OSS and on-premise Grafana

- **Default organization:** Use `default` for organization 1.
- **Additional organizations:** Use `org-<ORG_ID>`.

#### Grafana Cloud

- **Format:** Use `stacks-<STACK_ID>`.
- **Instance ID:** Your instance ID is the `STACK_ID`.

You can find the instance ID in the following places:

- **Grafana Cloud portal:** Go to `grafana.com`, open your stack, and select **Details** for your Grafana instance.
- **Swagger UI:** Open the `/swagger` page in your Grafana Cloud instance, where the namespace is automatically populated for the relevant endpoints.

### Resource (`<resource>`)

The resource is the type you want to interact with. Common examples include the following:

- **Dashboards:** Use `dashboards`.
- **Playlists:** Use `playlists`.
- **Folders:** Use `folders`.

### Kind (`<kind>`)

The kind identifies the resource type in API responses and corresponds to the singular form of the resource.
For example, the `dashboards` resource has a kind of `Dashboard`.

### Name (`<name>`)

The `<name>` is the unique identifier for a specific instance of a resource within its namespace and resource type. `<name>` is distinct from the `metadata.uid` field. The URL path always uses `metadata.name`.

For example, to get a dashboard defined as:

```json
{
  "kind": "Dashboard",
  "apiVersion": "dashboard.grafana.app/v1",
  "metadata": {
    "name": "production-overview", // This value IS used in the URL path
    "namespace": "default",
    "uid": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8" // This value is NOT used in the URL path
    // ... other metadata
  },
  "spec": {
    // ... dashboard spec
  }
}
```

You would use the following API call:

`GET /apis/dashboard.grafana.app/v1/namespaces/default/dashboards/production-overview`

### Metadata

The metadata section contains information about the resource instance.
This section includes [name](#name-name) and [namespace](#namespace-namespace), which are described earlier in this document, along with the following fields:

#### UID

An internal identifier that can be ignored for most use cases. Use the `name` field as the unique identifier instead. This value is _not_ the same as the Grafana UID.

#### ResourceVersion

A value that changes whenever any part of the resource changes, including metadata or status.

Use this field for:

- **Change detection:** Track whether the resource has changed.
- **Optimistic concurrency control:** Prevent overwriting a newer version of the resource.

#### Generation

A monotonically increasing number that increments only when the spec changes.
Updates to metadata or status don't affect this value.

#### CreationTimestamp

The time the object was created, formatted as an RFC 3339 UTC timestamp (for example, `2026-01-23T22:06:40Z`).

#### Annotations

A map of key-value pairs.

Common annotations include:

- **`grafana.app/createdBy` and `grafana.app/updatedBy`:** Identify who created or last updated the resource. Use the format `<USER_TYPE>:<UID>`, for example `user:u000000839`.
- **`grafana.app/folder`:** If the resource supports folders, this contains the folder UID that the object belongs to.
- **`grafana.app/updatedTimestamp`:** Stores the last update time as an RFC 3339 UTC timestamp, for example `2026-01-23T05:17:31Z`.

#### Labels

An optional map of key-value pairs for organizing and selecting resources.

### Spec

The spec field describes the desired state of the resource. Its structure depends on the resource type and API version. Refer to the Swagger or OpenAPI documentation for the exact schema for a resource's spec.

## Related resources

Use the following resources to keep working with Grafana APIs:

- **Migration guide:** Refer to [Migrate to the new APIs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis-migration).
- **HTTP API reference:** Refer to [HTTP API reference](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/).
- **Swagger UI:** Open the `/swagger` page in your Grafana instance to inspect endpoint schemas and try requests.

# Authentication options for the HTTP APIs

Authentication options depend on whether you're using [Grafana on prem](#authentication-options-in-grafana-ossenterprise) or [Grafana Cloud](#authentication-options-in-grafana-cloud).

To set up organizations, refer to the [X-Grafana-Org-Id header](#the-x-grafana-org-id-header) section.

## Authentication options in Grafana OSS/Enterprise

You can authenticate HTTP API requests using basic authentication or a service account token.

### Basic auth

This option is available in Grafana on prem only.

Basic auth is enabled by default and allows you authenticate your HTTP request via standard basic auth. Basic auth also authenticates LDAP users.

For example:

```bash
curl http://admin:admin@localhost:3000/api/org
{"id":1,"name":"Main Org."}
```

### Service account token

To create a service account token:

1. Go to **Administration** in the left-side menu
1. Click **Users and access > Service Accounts**.

For more information on how to use service account tokens, refer to the [Service Accounts](/docs/grafana/<GRAFANA_VERSION>/administration/service-accounts/) documentation.

You use the token in all requests in the `Authorization` header, for example:

```http
GET http://your.grafana.com/api/dashboards/db/mydash HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

The `Authorization` header value should be _`Bearer <YOUR_SERVICE_ACCOUNT_TOKEN>`_.

## Authentication options in Grafana Cloud

To use the HTTP API provided by a Grafana Cloud instance, authenticate requests with a service account token.

To access or create your service account tokens:

1. Go to **Administration** in the left-side menu
1. Click **Users and access > Service Accounts**.

For details on creating service accounts, assigning permissions, and adding tokens, refer to [Service Accounts](/docs/grafana-cloud/account-management/authentication-and-permissions/service-accounts/).

Include the service account token in the `Authorization` header for all requests to your Grafana instance:

```http
GET http://your.grafana.com/api/dashboards/db/mydash HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

Requests to the HTTP API are authenticated using the `Authorization` header:

```bash
Authorization: Bearer <SERVICE ACCOUNT TOKEN>
```

## The X-Grafana-Org-Id header

**X-Grafana-Org-Id** is an optional property that specifies the organization to which the action is applied. If not set, the created key belongs to the current context org. Use this header in all requests except those regarding admin.

**Example Request**:

```http
GET /api/org/ HTTP/1.1
Accept: application/json
Content-Type: application/json
X-Grafana-Org-Id: 2
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

# Dashboard API

{{< admonition type="note" >}}
Available in Grafana 12 and later.

This API complies with the new Grafana API structure. To learn more refer to documentation about the [API structure in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis).

{{< /admonition >}}

The Dashboard API (`/apis/dashboard.grafana.app/*`) replaces [`/api/dashboards/*`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/dashboard_public/), although there may not be an exact match for all legacy endpoints.

## Requirements

If you're running Grafana Enterprise, you'll need to have specific permissions for some endpoints. Refer to [Role-based access control permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## Endpoints

This document may not contain the latest version of the API. For the most up-to-date list of available endpoints, refer to [dashboard.grafana.app/v2](https://play.grafana.org/swagger?api=dashboard.grafana.app-v2) in Swagger.

| Method | Summary                                      | URI                                                                      |
| ------ | -------------------------------------------- | ------------------------------------------------------------------------ |
| POST   | [Create Dashboard](#create-dashboard)        | /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards          |
| PUT    | [Update Dashboard](#update-dashboard)        | /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards/:uid     |
| GET    | [Get Dashboard](#get-dashboard)              | /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards/:uid     |
| GET    | [Get Dashboard (DTO format)](#get-dashboard) | /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards/:uid/dto |
| GET    | [List Dashboards](#list-dashboards)          | /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards          |
| DELETE | [Delete Dashboard](#delete-dashboard)        | /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards/:uid     |

## Create Dashboard

`POST /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards`

Creates a new dashboard.

- namespace: to read more about the namespace to use, see the [API overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/apis/).

**Required permissions**

See note in the [introduction]({{< ref "#dashboard-api" >}}) for an explanation.

<!-- prettier-ignore-start -->
| Action              | Scope                                                                                                   |
| ------------------- | ------------------------------------------------------------------------------------------------------- |
| `dashboards:create` | <ul><li>`folders:*`</li><li>`folders:uid:*`</li></ul>                                                   |
| `dashboards:write`  | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example Create Request**:

```http
POST /apis/dashboard.grafana.app/v1/namespaces/default/dashboards HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "metadata": {
    "name": "gdxccn",
    "annotations": {
      "grafana.app/folder": "fef30w4jaxla8b"
    },
  },
  "spec": {
    "annotations": {
    "list": [
      {
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "enable": true,
        "hide": false,
        "iconColor": "red",
        "name": "Example annotation",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        }
      }]
    },
    "editable": true,
    "fiscalYearStartMonth": 0,
    "graphTooltip": 0,
    "links": [
      {
        "asDropdown": false,
        "icon": "external link",
        "includeVars": false,
        "keepTime": false,
        "tags": [],
        "targetBlank": false,
        "title": "Example Link",
        "tooltip": "",
        "type": "dashboards",
        "url": ""
      }
    ],
    "panels": [
      {
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "description": "With a description",
        "fieldConfig": {
          "defaults": {
            "color": {
              "mode": "palette-classic"
            },
            "custom": {
              "axisBorderShow": false,
              "axisCenteredZero": false,
              "axisColorMode": "text",
              "axisLabel": "",
              "axisPlacement": "auto",
              "barAlignment": 0,
              "barWidthFactor": 0.6,
              "drawStyle": "line",
              "fillOpacity": 0,
              "gradientMode": "none",
              "hideFrom": {
                "legend": false,
                "tooltip": false,
                "viz": false
              },
              "insertNulls": false,
              "lineInterpolation": "linear",
              "lineWidth": 1,
              "pointSize": 5,
              "scaleDistribution": {
                "type": "linear"
              },
              "showPoints": "auto",
              "spanNulls": false,
              "stacking": {
                "group": "A",
                "mode": "none"
              },
              "thresholdsStyle": {
                "mode": "off"
              }
            },
            "mappings": [],
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green"
                },
                {
                  "color": "red",
                  "value": 80
                }
              ]
            }
          },
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 0,
          "y": 0
        },
        "id": 1,
        "options": {
          "legend": {
            "calcs": [],
            "displayMode": "list",
            "placement": "bottom",
            "showLegend": true
          },
          "tooltip": {
            "hideZeros": false,
            "mode": "single",
            "sort": "none"
          }
        },
        "pluginVersion": "12.0.0",
        "targets": [
          {
            "datasource": {
              "type": "datasource",
              "uid": "grafana"
            },
            "refId": "A"
          }
        ],
        "title": "Example panel",
        "type": "timeseries"
      }
    ],
    "preload": false,
    "schemaVersion": 41,
    "tags": ["example"],
    "templating": {
      "list": [
        {
          "current": {
            "text": "",
            "value": ""
          },
          "definition": "",
          "description": "example description",
          "label": "ExampleLabel",
          "name": "ExampleVariable",
          "options": [],
          "query": "",
          "refresh": 1,
          "regex": "cluster",
          "type": "query"
        }
      ]
    },
    "time": {
      "from": "now-6h",
      "to": "now"
    },
    "timepicker": {},
    "timezone": "browser",
    "title": "Example Dashboard",
    "version": 0
  }
}
```

JSON Body schema:

- **metadata.name** – The Grafana [unique identifier]({{< ref "#identifier-id-vs-unique-identifier-uid" >}}). If you do not want to provide this, set metadata.generateName instead to the prefix you would like for the randomly generated uid (cannot be an empty string).
- **metadata.annotations.grafana.app/folder** - Optional field, the unique identifier of the folder under which the dashboard should be created.
- **spec** – The dashboard json.

{{< admonition type="note" >}}
Custom labels and annotations in the metadata field are supported on some instances, with full support planned for all instances when these APIs reach general availability. If they are not yet supported on your instance, they will be ignored.
{{< /admonition >}}

**Example Response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 485

{
  "kind": "Dashboard",
  "apiVersion": "dashboard.grafana.app/v1",
  "metadata": {
    "name": "gdxccn",
    "namespace": "default",
    "uid": "Cc7fA5ffHY94NnHZyMxXvFlpFtOmkK3qkBcVZPKSPXcX",
    "resourceVersion": "1",
    "generation": 1,
    "creationTimestamp": "2025-04-24T20:35:29Z",
    "labels": {
      "grafana.app/deprecatedInternalID": "11"
    },
    "annotations": {
      "grafana.app/createdBy": "service-account:dejwtrofg77y8d",
      "grafana.app/folder": "fef30w4jaxla8b"
    },
    "managedFields": [
      {
        "manager": "curl",
        "operation": "Update",
        "apiVersion": "dashboard.grafana.app/v0alpha1",
        "time": "2025-04-24T20:35:29Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:spec": {
            "f:annotations": {
              ".": {},
              "f:list": {}
            },
            "f:editable": {},
            "f:fiscalYearStartMonth": {},
            "f:graphTooltip": {},
            "f:links": {},
            "f:panels": {},
            "f:preload": {},
            "f:schemaVersion": {},
            "f:tags": {},
            "f:templating": {
              ".": {},
              "f:list": {}
            },
            "f:time": {
              ".": {},
              "f:from": {},
              "f:to": {}
            },
            "f:timepicker": {},
            "f:timezone": {},
            "f:title": {},
            "f:version": {}
          }
        }
      }
    ]
  },
  "spec": {
    "annotations": {
      "list": [
        {
          "datasource": {
            "type": "datasource",
            "uid": "grafana"
          },
          "enable": true,
          "hide": false,
          "iconColor": "red",
          "name": "Example annotation",
          "target": {
            "limit": 100,
            "matchAny": false,
            "tags": [],
            "type": "dashboard"
          }
        }
      ]
    },
    "editable": true,
    "fiscalYearStartMonth": 0,
    "graphTooltip": 0,
    "links": [
      {
        "asDropdown": false,
        "icon": "external link",
        "includeVars": false,
        "keepTime": false,
        "tags": [],
        "targetBlank": false,
        "title": "Example Link",
        "tooltip": "",
        "type": "dashboards",
        "url": ""
      }
    ],
    "panels": [
      {
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "description": "With a description",
        "fieldConfig": {
          "defaults": {
            "color": {
              "mode": "palette-classic"
            },
            "custom": {
              "axisBorderShow": false,
              "axisCenteredZero": false,
              "axisColorMode": "text",
              "axisLabel": "",
              "axisPlacement": "auto",
              "barAlignment": 0,
              "barWidthFactor": 0.6,
              "drawStyle": "line",
              "fillOpacity": 0,
              "gradientMode": "none",
              "hideFrom": {
                "legend": false,
                "tooltip": false,
                "viz": false
              },
              "insertNulls": false,
              "lineInterpolation": "linear",
              "lineWidth": 1,
              "pointSize": 5,
              "scaleDistribution": {
                "type": "linear"
              },
              "showPoints": "auto",
              "spanNulls": false,
              "stacking": {
                "group": "A",
                "mode": "none"
              },
              "thresholdsStyle": {
                "mode": "off"
              }
            },
            "mappings": [],
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green"
                },
                {
                  "color": "red",
                  "value": 80
                }
              ]
            }
          },
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 0,
          "y": 0
        },
        "id": 1,
        "options": {
          "legend": {
            "calcs": [],
            "displayMode": "list",
            "placement": "bottom",
            "showLegend": true
          },
          "tooltip": {
            "hideZeros": false,
            "mode": "single",
            "sort": "none"
          }
        },
        "pluginVersion": "12.0.0",
        "targets": [
          {
            "datasource": {
              "type": "datasource",
              "uid": "grafana"
            },
            "refId": "A"
          }
        ],
        "title": "Example panel",
        "type": "timeseries"
      }
    ],
    "preload": false,
    "schemaVersion": 41,
    "tags": [
      "example"
    ],
    "templating": {
      "list": [
        {
          "current": {
            "text": "",
            "value": ""
          },
          "definition": "",
          "description": "example description",
          "label": "ExampleLabel",
          "name": "ExampleVariable",
          "options": [],
          "query": "",
          "refresh": 1,
          "regex": "cluster",
          "type": "query"
        }
      ]
    },
    "time": {
      "from": "now-6h",
      "to": "now"
    },
    "timepicker": {},
    "timezone": "browser",
    "title": "Example Dashboard"
  },
  "status": {}
```

Status Codes:

- **201** – Created
- **400** – Errors (invalid json, missing or invalid fields, etc)
- **401** – Unauthorized
- **403** – Access denied
- **409** – Conflict (dashboard with the same uid already exists)

## Update Dashboard

`PUT /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards/:uid`

Updates an existing dashboard via the dashboard uid.

- namespace: to read more about the namespace to use, see the [API overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/apis/).
- uid: the unique identifier of the dashboard to update. this will be the _name_ in the dashboard response

**Required permissions**

See note in the [introduction]({{< ref "#dashboard-api" >}}) for an explanation.

<!-- prettier-ignore-start -->
| Action              | Scope                                                                                                   |
| ------------------- | ------------------------------------------------------------------------------------------------------- |
| `dashboards:write`  | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example Update Request**:

```http
POST /apis/dashboard.grafana.app/v1/namespaces/default/dashboards/gdxccn HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "metadata": {
    "name": "gdxccn",
    "annotations": {
      "grafana.app/folder": "fef30w4jaxla8b",
      "grafana.app/message": "commit message"
    },
  },
  "spec": {
    "title": "New dashboard - updated",
    "schemaVersion": 41,
    ...
  }
}
```

JSON Body schema:

- **metadata.name** – The [unique identifier]({{< ref "#identifier-id-vs-unique-identifier-uid" >}}).
- **metadata.annotations.grafana.app/folder** - Optional field, the unique identifier of the folder under which the dashboard should be created.
- **metadata.annotations.grafana.app/message** - Optional field, to set a commit message for the version history.
- **spec** – The dashboard json.

{{< admonition type="note" >}}
Custom labels and annotations in the metadata field are supported on some instances, with full support planned for all instances when these APIs reach general availability. If they are not yet supported on your instance, they will be ignored.
{{< /admonition >}}

**Example Response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 485

{
  "kind": "Dashboard",
  "apiVersion": "dashboard.grafana.app/v1",
  "metadata": {
    "name": "gdxccn",
    "namespace": "default",
    "uid": "Cc7fA5ffHY94NnHZyMxXvFlpFtOmkK3qkBcVZPKSPXcX",
    "resourceVersion": "2",
    "generation": 2,
    "creationTimestamp": "2025-03-06T19:57:18Z",
    "annotations": {
      "grafana.app/folder": "fef30w4jaxla8b",
      "grafana.app/createdBy": "service-account:cef2t2rfm73lsb",
      "grafana.app/updatedBy": "service-account:cef2t2rfm73lsb",
      "grafana.app/updatedTimestamp": "2025-03-07T02:58:36Z"
    }
  },
  "spec": {
    "schemaVersion": 41,
    "title": "New dashboard - updated",
    ...
  }
}
```

Status Codes:

- **200** – OK
- **400** – Errors (invalid json, missing or invalid fields, etc)
- **401** – Unauthorized
- **403** – Access denied
- **409** – Conflict (dashboard with the same version already exists)

## Get Dashboard

`GET /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards/:uid`

Gets a dashboard via the dashboard uid.

- namespace: to read more about the namespace to use, see the [API overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/apis/).
- uid: the unique identifier of the dashboard to update. this will be the _name_ in the dashboard response

**Required permissions**

See note in the [introduction]({{< ref "#dashboard-api" >}}) for an explanation.

<!-- prettier-ignore-start -->
| Action            | Scope                                                                                                   |
| ----------------- | ------------------------------------------------------------------------------------------------------- |
| `dashboards:read` | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example Get Request**:

```http
GET /apis/dashboard.grafana.app/v1/namespaces/default/dashboards/gdxccn HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 485

{
  "kind": "Dashboard",
  "apiVersion": "dashboard.grafana.app/v1",
  "metadata": {
    "name": "gdxccn",
    "namespace": "default",
    "uid": "Cc7fA5ffHY94NnHZyMxXvFlpFtOmkK3qkBcVZPKSPXcX",
    "resourceVersion": "2",
    "generation": 2,
    "creationTimestamp": "2025-03-06T19:57:18Z",
    "annotations": {
      "grafana.app/createdBy": "service-account:cef2t2rfm73lsb",
      "grafana.app/updatedBy": "service-account:cef2t2rfm73lsb",
      "grafana.app/updatedTimestamp": "2025-03-07T02:58:36Z"
    }
  },
  "spec": {
    "schemaVersion": 41,
    "title": "New dashboard - updated",
    ...
  }
}
```

Status Codes:

- **200** – OK
- **401** – Unauthorized
- **403** – Access denied
- **404** – Not Found

### Retrieve additional access information

`GET /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards/:uid/dto`

Retrieves a dashboard with additional access information.

The `GET` response includes an additional `access` section with data such as if it's a public dashboard, or the dashboard permissions (admin, editor) of the user who made the request.

## List Dashboards

`GET /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards`

Lists all dashboards in the given organization. You can control the maximum number of dashboards returned through the `limit` query parameter. You can then use the `continue` token returned to fetch the next page of dashboards.

- namespace: to read more about the namespace to use, see the [API overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/apis/).

**Query parameters**:

- **`limit`** (optional): Maximum number of dashboards to return
- **`continue`** (optional): Continue token from a previous response to fetch the next page

**Required permissions**

See note in the [introduction]({{< ref "#dashboard-api" >}}) for an explanation.

<!-- prettier-ignore-start -->
| Action            | Scope                                                                                                   |
| ----------------- | ------------------------------------------------------------------------------------------------------- |
| `dashboards:read` | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example Get Request**:

```http
GET /apis/dashboard.grafana.app/v1/namespaces/default/dashboards?limit=1 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 644

{
  "kind": "DashboardList",
  "apiVersion": "dashboard.grafana.app/v1alpha1",
  "metadata": {
    "resourceVersion": "1741315830000",
    "continue": "eyJvIjoxNTIsInYiOjE3NjE3MDQyMjQyMDcxODksInMiOmZhbHNlfQ=="
  },
  "items": [
    {
      "kind": "Dashboard",
      "apiVersion": "dashboard.grafana.app/v1alpha1",
      "metadata": {
        "name": "gpqcmf",
        "namespace": "default",
        "uid": "VQyL7pNTpfGPNlPM6HRJSePrBg5dXmxr4iPQL7txLtwX",
        "resourceVersion": "1",
        "generation": 1,
        "creationTimestamp": "2025-03-06T19:50:30Z",
        "annotations": {
          "grafana.app/createdBy": "service-account:cef2t2rfm73lsb",
          "grafana.app/updatedBy": "service-account:cef2t2rfm73lsb",
          "grafana.app/updatedTimestamp": "2025-03-06T19:50:30Z"
        }
      },
      "spec": {
        "schemaVersion": 41,
        "title": "New dashboard",
        "uid": "gpqcmf",
        "version": 1,
        ...
      }
    }
  ]
}
```

The `metadata.continue` field contains a token to fetch the next page.

**Example subsequent request using continue token**:

```http
GET /apis/dashboard.grafana.app/v1/namespaces/default/dashboards?limit=1&continue=eyJvIjoxNTIsInYiOjE3NjE3MDQyMjQyMDcxODksInMiOmZhbHNlfQ== HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example subsequent response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
  "kind": "DashboardList",
  "apiVersion": "dashboard.grafana.app/v1alpha1",
  "items": [
    {
      "kind": "Dashboard",
      "apiVersion": "dashboard.grafana.app/v1alpha1",
      "metadata": {
        "name": "hpqcmg",
        "namespace": "default",
        "uid": "WQyL7pNTpfGPNlPM6HRJSePrBg5dXmxr4iPQL7txLtwY",
        "resourceVersion": "1",
        "generation": 1,
        "creationTimestamp": "2025-03-06T19:51:31Z",
        "annotations": {
          "grafana.app/createdBy": "service-account:cef2t2rfm73lsb",
          "grafana.app/updatedBy": "service-account:cef2t2rfm73lsb",
          "grafana.app/updatedTimestamp": "2025-03-06T19:51:31Z"
        }
      },
      "spec": {
        "schemaVersion": 41,
        "title": "Another dashboard",
        "uid": "hpqcmg",
        "version": 1,
        ...
      }
    }
  ]
}
```

Continue making requests with the updated `continue` token until you receive a response without a `continue` field in the metadata, indicating you've reached the last page.

Status Codes:

- **200** – OK
- **401** – Unauthorized
- **403** – Access denied

## List dashboard history

You can retrieve the full version history of a dashboard using the List endpoint with specific query parameters. For details and examples, refer to [Resource history HTTP API](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/resource-history/).

## Delete Dashboard

`DELETE /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards/:uid`

Deletes a dashboard via the dashboard uid.

- **`namespace`**: To read more about the namespace to use, see the [API overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/apis/).
- **`uid`**: The unique identifier of the dashboard to update. This is the `metadata.name` field in the dashboard response and _not_ the `metadata.uid` field.

**Required permissions**

See note in the [introduction](#new-dashboard-apis) for an explanation.

<!-- prettier-ignore-start -->
| Action              | Scope                                                                                                   |
| ------------------- | ------------------------------------------------------------------------------------------------------- |
| `dashboards:delete` | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example Delete Request**:

```http
DELETE /apis/dashboard.grafana.app/v1/namespaces/default/dashboards/gdxccn HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 78

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Success",
  "details": {
    "name": "gdxccn",
    "group": "dashboard.grafana.app",
    "kind": "dashboards",
    "uid": "Cc7fA5ffHY94NnHZyMxXvFlpFtOmkK3qkBcVZPKSPXcX"
  }
}
```

Status Codes:

- **200** – OK
- **401** – Unauthorized
- **403** – Access denied
- **404** – Not found

# Folder APIs

{{< admonition type="note" >}}
Available in Grafana 12 and later.

This API complies with the new Grafana API structure. To learn more refer to documentation about the [API structure in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis/).

**This document may not contain the latest version of the API. For the most up-to-date list of available endpoints, refer to [folder.grafana.app/v1](https://play.grafana.org/swagger?api=folder.grafana.app-v1) in Swagger.**

{{< /admonition >}}

## Requirements

If you're running Grafana Enterprise, you'll need to have specific permissions for some endpoints. Refer to [Role-based access control permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## Endpoints

| Method | URI                                                            | Summary                                 |
| ------ | -------------------------------------------------------------- | --------------------------------------- |
| GET    | /apis/folder.grafana.app/v1/namespaces/:namespace/folders      | [Get all folders](#get-all-folders)     |
| GET    | /apis/folder.grafana.app/v1/namespaces/:namespace/folders/:uid | [Get folder by uid](#get-folder-by-uid) |
| POST   | /apis/folder.grafana.app/v1/namespaces/:namespace/folders      | [Create folder](#create-folder)         |
| PUT    | /apis/folder.grafana.app/v1/namespaces/:namespace/folders/:uid | [Update folder](#update-folder)         |
| DELETE | /apis/folder.grafana.app/v1/namespaces/:namespace/folders/:uid | [Delete folder](#delete-folder)         |

## Get all folders

`GET /apis/folder.grafana.app/v1/namespaces/:namespace/folders`

Returns all folders that the authenticated user has permission to view within the given organization. Use the `limit` query parameter to control the maximum number of dashboards returned. To retrieve additional dashboards, utilize the `continue` token provided in the response to fetch the next page.

- namespace: to read more about the namespace to use, see the [API overview](ref:apis).

**Query parameters**:

- **`limit`** (optional): Maximum number of folders to return
- **`continue`** (optional): Continue token from a previous response to fetch the next page

**Required permissions**

See note in the [introduction]({{< ref "#folder-api" >}}) for an explanation.

| Action         | Scope       |
| -------------- | ----------- |
| `folders:read` | `folders:*` |

**Example Request**:

```http
GET /apis/folder.grafana.app/v1/namespaces/default/folders?limit=1 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json
{
  "kind": "FolderList",
  "apiVersion": "folder.grafana.app/v1",
  "metadata": {
    "continue": "eyJvIjoxNTIsInYiOjE3NjE3MDQyMjQyMDcxODksInMiOmZhbHNlfQ=="
  },
  "items": [
    {
      "kind": "Folder",
      "apiVersion": "folder.grafana.app/v1",
      "metadata": {
        "name": "aef30vrzxs3y8d",
        "namespace": "default",
        "uid": "KCtv1FXDsJmTYQoTgcPnfuwZhDZge3uMpXOefaOHjb4X",
        "resourceVersion": "1741343686000",
        "creationTimestamp": "2025-03-07T10:34:46Z",
        "annotations": {
          "grafana.app/createdBy": "service-account:cef2t2rfm73lsb",
          "grafana.app/updatedBy": "service-account:cef2t2rfm73lsb",
          "grafana.app/updatedTimestamp": "2025-03-07T10:34:46Z"
        }
      },
      "spec": {
        "title": "example"
      }
    }
  ]
}
```

The `metadata.continue` field contains a token to fetch the next page.

**Example subsequent request using continue token**:

```http
GET /apis/folder.grafana.app/v1/namespaces/default/folders?limit=1&continue=eyJvIjoxNTIsInYiOjE3NjE3MDQyMjQyMDcxODksInMiOmZhbHNlfQ== HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example subsequent response**:

```http
HTTP/1.1 200
Content-Type: application/json
{
  "kind": "FolderList",
  "apiVersion": "folder.grafana.app/v1",
  "items": [
    {
      "kind": "Folder",
      "apiVersion": "folder.grafana.app/v1",
      "metadata": {
        "name": "bef30vrzxs3y8e",
        "namespace": "default",
        "uid": "YCtv1FXDsJmTYQoTgcPnfuwZhDZge3uMpXOefaOHjb5Y",
        "resourceVersion": "1741343687000",
        "creationTimestamp": "2025-03-07T10:35:47Z",
        "annotations": {
          "grafana.app/createdBy": "service-account:cef2t2rfm73lsb",
          "grafana.app/updatedBy": "service-account:cef2t2rfm73lsb",
          "grafana.app/updatedTimestamp": "2025-03-07T10:35:47Z"
        }
      },
      "spec": {
        "title": "another folder"
      }
    }
  ]
}
```

Continue making requests with the updated `continue` token until you receive a response without a `continue` field in the metadata, indicating you've reached the last page.

Status Codes:

- **200** – OK
- **401** – Unauthorized
- **403** – Access Denied

## Get folder by uid

`GET /apis/folder.grafana.app/v1/namespaces/:namespace/folders/:uid`

Will return the folder given the folder uid.

- namespace: to read more about the namespace to use, see the [API overview](ref:apis).
- uid: the unique identifier of the folder to update. this will be the _name_ in the folder response

**Required permissions**

See note in the [introduction]({{< ref "#folder-api" >}}) for an explanation.

| Action         | Scope       |
| -------------- | ----------- |
| `folders:read` | `folders:*` |

**Example Request**:

```http
GET /apis/folder.grafana.app/v1/namespaces/default/folders/aef30vrzxs3y8d HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json
{
  "kind": "Folder",
  "apiVersion": "folder.grafana.app/v1",
  "metadata": {
    "name": "aef30vrzxs3y8d",
    "namespace": "default",
    "uid": "KCtv1FXDsJmTYQoTgcPnfuwZhDZge3uMpXOefaOHjb4X",
    "resourceVersion": "1741343686000",
    "creationTimestamp": "2025-03-07T10:34:46Z",
    "annotations": {
      "grafana.app/createdBy": "service-account:cef2t2rfm73lsb",
      "grafana.app/updatedBy": "service-account:cef2t2rfm73lsb",
      "grafana.app/updatedTimestamp": "2025-03-07T10:34:46Z",
      "grafana.app/folder": "fef30w4jaxla8b"
    }
  },
  "spec": {
    "title": "test"
  }
}
```

Note the annotation `grafana.app/folder` which contains the uid of the parent folder.

Status Codes:

- **200** – Found
- **401** – Unauthorized
- **403** – Access Denied
- **404** – Folder not found

## Create folder

`POST /apis/folder.grafana.app/v1/namespaces/:namespace/folders`

Creates a new folder.

- namespace: to read more about the namespace to use, see the [API overview](ref:apis).

**Required permissions**

See note in the [introduction]({{< ref "#folder-api" >}}) for an explanation.

`folders:create` allows creating folders and subfolders. If granted with scope `folders:uid:general`, allows creating root level folders. Otherwise, allows creating subfolders under the specified folders.

| Action           | Scope       |
| ---------------- | ----------- |
| `folders:create` | `folders:*` |
| `folders:write`  | `folders:*` |

**Example Request**:

```http
POST /apis/folder.grafana.app/v1/namespaces/default/folders HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "metadata": {
    "name": "aef30vrzxs3y8d",
    "annotations": {
      "grafana.app/folder": "fef30w4jaxla8b"
    }
  },
  "spec": {
    "title": "child-folder"
  }
}
```

JSON Body schema:

- **metadata.name** – The Grafana [unique identifier]({{< ref "#identifier-id-vs-unique-identifier-uid" >}}). If you do not want to provide this, set metadata.generateName to the prefix you would like for the uid.
- **metadata.annotations.grafana.app/folder** - Optional field, the unique identifier of the parent folder under which the folder should be created. Requires nested folders to be enabled.
- **spec.title** – The title of the folder.

{{< admonition type="note" >}}
Custom labels and annotations in the metadata field are supported on some instances, with full support planned for all instances when these APIs reach general availability. If they are not yet supported on your instance, they will be ignored.
{{< /admonition >}}

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json
{
  "kind": "Folder",
  "apiVersion": "folder.grafana.app/v1",
  "metadata": {
    "name": "eef33r1fprd34d",
    "namespace": "default",
    "uid": "X8momvVZnsXdOqvLD9I4ngqLVif2CgRWXHy9xb2UgjQX",
    "resourceVersion": "1741320415009",
    "creationTimestamp": "2025-03-07T04:06:55Z",
    "labels": {
      "grafana.app/deprecatedInternalID": "1159"
    },
    "annotations": {
      "grafana.app/folder": "fef30w4jaxla8b",
      "grafana.app/createdBy": "service-account:cef2t2rfm73lsb"
    }
  },
  "spec": {
    "title": "child-folder"
  }
}
```

Status Codes:

- **201** – Created
- **400** – Errors (invalid json, missing or invalid fields, etc)
- **401** – Unauthorized
- **403** – Access denied
- **409** – Conflict (folder with the same uid already exists)

## Update folder

`PUT /apis/folder.grafana.app/v1/namespaces/:namespace/folders/:uid`

Updates an existing folder identified by uid.

- namespace: to read more about the namespace to use, see the [API overview](ref:apis).
- uid: the unique identifier of the folder to update. this will be the _name_ in the folder response

**Required permissions**

See note in the [introduction]({{< ref "#folder-api" >}}) for an explanation.

| Action          | Scope       |
| --------------- | ----------- |
| `folders:write` | `folders:*` |

**Example Request**:

```http
PUT /apis/folder.grafana.app/v1/namespaces/default/folders/fef30w4jaxla8b HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "metadata": {
    "name": "aef30vrzxs3y8d",
    "annotations": {
      "grafana.app/folder": "xkj92m5pqw3vn4"
    }
  },
  "spec": {
    "title": "updated title"
  }
}
```

JSON Body schema:

- **metadata.name** – The [unique identifier]({{< ref "#identifier-id-vs-unique-identifier-uid" >}}) of the folder.
- **metadata.annotations.grafana.app/folder** - Optional field, the unique identifier of the parent folder under which the folder should be - update this to move the folder under a different parent folder. Requires nested folders to be enabled.
- **spec.title** – The title of the folder.

{{< admonition type="note" >}}
Custom labels and annotations in the metadata field are supported on some instances, with full support planned for all instances when these APIs reach general availability. If they are not yet supported on your instance, they will be ignored.
{{< /admonition >}}

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "kind": "Folder",
  "apiVersion": "folder.grafana.app/v1",
  "metadata": {
    "name": "fef30w4jaxla8b",
    "namespace": "default",
    "uid": "YaWLsFrMwEaTlIQwX2iMnhHlJuZHtZugps50BQoyjXEX",
    "resourceVersion": "1741345736000",
    "creationTimestamp": "2025-03-07T11:08:56Z",
    "annotations": {
      "grafana.app/folder": "xkj92m5pqw3vn4",
      "grafana.app/createdBy": "service-account:cef2t2rfm73lsb",
      "grafana.app/updatedBy": "service-account:cef2t2rfm73lsb",
      "grafana.app/updatedTimestamp": "2025-03-07T11:08:56Z"
    }
  },
  "spec": {
    "title": "updated title"
  }
}
```

Status Codes:

- **200** – Updated
- **400** – Errors (invalid json, missing or invalid fields, etc)
- **401** – Unauthorized
- **403** – Access Denied
- **404** – Folder not found
- **412** – Precondition failed (the folder has been changed by someone else). With this status code, the response body will have the following properties:

```http
HTTP/1.1 412 Precondition Failed
Content-Type: application/json; charset=UTF-8
Content-Length: 97

{
  "message": "The folder has been changed by someone else",
  "status": "version-mismatch"
}
```

## Delete folder

`DELETE /apis/folder.grafana.app/v1/namespaces/:namespace/folders/:uid`

Deletes an existing folder identified by UID along with all dashboards (and their alerts) stored in the folder. This operation cannot be reverted.

If [Grafana Alerting](ref:alerting) is enabled, you can set an optional query parameter `forceDeleteRules=false` so that requests will fail with 400 (Bad Request) error if the folder contains any Grafana alerts. However, if this parameter is set to `true` then it will delete any Grafana alerts under this folder.

- namespace: to read more about the namespace to use, see the [API overview](ref:apis).
- uid: the unique identifier of the folder to delete. this will be the _name_ in the folder response

**Required permissions**

See note in the [introduction]({{< ref "#folder-api" >}}) for an explanation.

| Action           | Scope       |
| ---------------- | ----------- |
| `folders:delete` | `folders:*` |

**Example Request**:

```http
DELETE /apis/folder.grafana.app/v1/namespaces/default/folders/fef30w4jaxla8b HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "kind": "Folder",
  "apiVersion": "folder.grafana.app/v1",
  "metadata": {
    "name": "fef30w4jaxla8b",
    "namespace": "default",
    "uid": "YaWLsFrMwEaTlIQwX2iMnhHlJuZHtZugps50BQoyjXEX",
    "resourceVersion": "1741345736000",
    "creationTimestamp": "2025-03-07T11:08:56Z",
    "annotations": {
      "grafana.app/folder": "xkj92m5pqw3vn4",
      "grafana.app/createdBy": "service-account:cef2t2rfm73lsb",
      "grafana.app/updatedBy": "service-account:cef2t2rfm73lsb",
      "grafana.app/updatedTimestamp": "2025-03-07T11:08:56Z"
    }
  },
  "spec": {
    "title": "updated title"
  }
}
```

Status Codes:

- **200** – Deleted
- **401** – Unauthorized
- **400** – Bad Request
- **403** – Access Denied
- **404** – Folder not found

# Playlist API

{{< admonition type="note" >}}
Available in Grafana 12 and later.

This API complies with the new Grafana API structure. To learn more refer to documentation about the [API structure in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis).

**This document may not contain the latest version of the API. For the most up-to-date list of available endpoints, refer to [playlist.grafana.app/v1](https://play.grafana.org/swagger?api=playlist.grafana.app-v1) in Swagger.**

{{< /admonition >}}

## List Playlists

`GET /apis/playlist.grafana.app/v1/namespaces/:namespace/playlists`

Lists all playlists in the specified namespace.

- `namespace`: To learn more about which namespace to use, refer to the [API overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/apis/).

**Example Request**:

```http
GET /apis/playlist.grafana.app/v1/namespaces/default/playlists HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "kind": "PlaylistList",
  "apiVersion": "playlist.grafana.app/v1",
  "metadata": {},
  "items": [
    {
      "kind": "Playlist",
      "apiVersion": "playlist.grafana.app/v1",
      "metadata": {
        "name": "my-playlist-uid",
        "namespace": "default",
        "resourceVersion": "1234567890",
        "creationTimestamp": "2024-01-15T10:30:00Z"
      },
      "spec": {
        "title": "My Playlist",
        "interval": "5m",
        "items": [
          {
            "type": "dashboard_by_uid",
            "value": "dashboard-uid-1"
          },
          {
            "type": "dashboard_by_tag",
            "value": "important"
          }
        ]
      }
    }
  ]
}
```

## Get a Playlist

`GET /apis/playlist.grafana.app/v1/namespaces/:namespace/playlists/:name`

Retrieves a specific playlist by name.

- `namespace`: To learn more about which namespace to use, refer to the [API overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/apis/).
- `name`: The UID of the playlist.

**Example Request**:

```http
GET /apis/playlist.grafana.app/v1/namespaces/default/playlists/my-playlist-uid HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "kind": "Playlist",
  "apiVersion": "playlist.grafana.app/v1",
  "metadata": {
    "name": "my-playlist-uid",
    "namespace": "default",
    "resourceVersion": "1234567890",
    "creationTimestamp": "2024-01-15T10:30:00Z"
  },
  "spec": {
    "title": "My Playlist",
    "interval": "5m",
    "items": [
      {
        "type": "dashboard_by_uid",
        "value": "dashboard-uid-1"
      },
      {
        "type": "dashboard_by_tag",
        "value": "important"
      }
    ]
  }
}
```

## Create a Playlist

`POST /apis/playlist.grafana.app/v1/namespaces/:namespace/playlists`

Creates a new playlist.

- `namespace`: To learn more about which namespace to use, refer to the [API overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/apis/).

**Example Request**:

```http
POST /apis/playlist.grafana.app/v1/namespaces/default/playlists HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "kind": "Playlist",
  "apiVersion": "playlist.grafana.app/v1",
  "metadata": {
    "name": "my-new-playlist-uid"
  },
  "spec": {
    "title": "My New Playlist",
    "interval": "5m",
    "items": [
      {
        "type": "dashboard_by_uid",
        "value": "dashboard-uid-1"
      },
      {
        "type": "dashboard_by_tag",
        "value": "monitoring"
      }
    ]
  }
}
```

**Example Response**:

```http
HTTP/1.1 201 Created
Content-Type: application/json

{
  "kind": "Playlist",
  "apiVersion": "playlist.grafana.app/v1",
  "metadata": {
    "name": "my-new-playlist-uid",
    "namespace": "default",
    "resourceVersion": "1234567891",
    "creationTimestamp": "2024-01-15T10:35:00Z"
  },
  "spec": {
    "title": "My New Playlist",
    "interval": "5m",
    "items": [
      {
        "type": "dashboard_by_uid",
        "value": "dashboard-uid-1"
      },
      {
        "type": "dashboard_by_tag",
        "value": "monitoring"
      }
    ]
  }
}
```

## Update a Playlist

`PUT /apis/playlist.grafana.app/v1/namespaces/:namespace/playlists/:name`

Updates an existing playlist. The entire playlist spec must be provided.

- `namespace`: To learn more about which namespace to use, refer to the [API overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/apis/).
- `name`: The UID of the playlist.

**Example Request**:

```http
PUT /apis/playlist.grafana.app/v1/namespaces/default/playlists/my-playlist-uid HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "kind": "Playlist",
  "apiVersion": "playlist.grafana.app/v1",
  "metadata": {
    "name": "my-playlist-uid",
    "namespace": "default",
    "resourceVersion": "1234567890"
  },
  "spec": {
    "title": "My Updated Playlist",
    "interval": "10m",
    "items": [
      {
        "type": "dashboard_by_uid",
        "value": "dashboard-uid-1"
      },
      {
        "type": "dashboard_by_uid",
        "value": "dashboard-uid-2"
      },
      {
        "type": "dashboard_by_tag",
        "value": "updated-tag"
      }
    ]
  }
}
```

**Example Response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json

{
  "kind": "Playlist",
  "apiVersion": "playlist.grafana.app/v1",
  "metadata": {
    "name": "my-playlist-uid",
    "namespace": "default",
    "resourceVersion": "1234567892",
    "creationTimestamp": "2024-01-15T10:30:00Z"
  },
  "spec": {
    "title": "My Updated Playlist",
    "interval": "10m",
    "items": [
      {
        "type": "dashboard_by_uid",
        "value": "dashboard-uid-1"
      },
      {
        "type": "dashboard_by_uid",
        "value": "dashboard-uid-2"
      },
      {
        "type": "dashboard_by_tag",
        "value": "updated-tag"
      }
    ]
  }
}
```

## Delete a Playlist

`DELETE /apis/playlist.grafana.app/v1/namespaces/:namespace/playlists/:name`

Deletes a playlist.

- `namespace`: To learn more about which namespace to use, refer to the [API overview](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/apis/).
- `name`: The UID of the playlist.

**Example Request**:

```http
DELETE /apis/playlist.grafana.app/v1/namespaces/default/playlists/my-playlist-uid HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Success",
  "code": 200
}
```

## Playlist Item Types

Playlist items support three types:

- `dashboard_by_uid`: Include a specific dashboard by its UID
- `dashboard_by_tag`: Include all dashboards with a specific tag
- `dashboard_by_id`: (Deprecated) Include a dashboard by internal ID

# Resource History HTTP API

{{< admonition type="note" >}}
Available in Grafana 12 and later.

This API complies with the new Grafana API structure. To learn more refer to documentation about the [API structure in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis).

{{< /admonition >}}

The new Grafana APIs track version history for resources. You can retrieve the history of any resource by using the standard List endpoint with specific query parameters. This page documents how to list resource history, using dashboards as an example. The same pattern applies to any resource.

**Resource history is available on all stable (GA) API versions (for example, `v1`, `v2`)**. Alpha or beta versions may not support it.

## List resource history

`GET /apis/<group>/<version>/namespaces/<namespace>/<resource>?labelSelector=grafana.app/get-history=true&fieldSelector=metadata.name=<NAME>`

Lists the version history of a specific resource. This uses the standard List endpoint with two required query parameters:

- **`labelSelector`**: Must be set to `grafana.app/get-history=true` to request history instead of a normal list.
- **`fieldSelector`**: Must be set to `metadata.name=<NAME>` to identify the specific resource. `<NAME>` is the `metadata.name` field of the resource (the Grafana UID).

You can control pagination through additional query parameters:

- **`limit`** (optional): Maximum number of history entries to return per page.
- **`continue`** (optional): Token from a previous response to fetch the next page.

History entries are returned in reverse chronological order (newest first).

### Dashboard example

The following request retrieves the version history for a dashboard with `metadata.name` of `production-overview` in the `default` namespace:

**Example request**:

```http
GET /apis/dashboard.grafana.app/v1/namespaces/default/dashboards?labelSelector=grafana.app/get-history=true&fieldSelector=metadata.name=production-overview HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json

{
  "kind": "DashboardList",
  "apiVersion": "dashboard.grafana.app/v1",
  "metadata": {
    "resourceVersion": "1758777451428472",
    "continue": "eyJvIjoxNTIsInYiOjE3NjE3MDQyMjQyMDcxODksInMiOmZhbHNlfQ=="
  },
  "items": [
    {
      "kind": "Dashboard",
      "apiVersion": "dashboard.grafana.app/v1",
      "metadata": {
        "name": "production-overview",
        "namespace": "default",
        "uid": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
        "resourceVersion": "1758777451428472",
        "generation": 3,
        "creationTimestamp": "2026-03-15T14:22:10Z",
        "annotations": {
          "grafana.app/updatedBy": "user:u000000001",
          "grafana.app/message": "Added latency panel"
        }
      },
      "spec": {
        "title": "Production Overview",
        "schemaVersion": 41,
        ...
      }
    },
    {
      "kind": "Dashboard",
      "apiVersion": "dashboard.grafana.app/v1",
      "metadata": {
        "name": "production-overview",
        "namespace": "default",
        "uid": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
        "resourceVersion": "1758777451428100",
        "generation": 2,
        "creationTimestamp": "2026-03-10T09:15:30Z",
        "annotations": {
          "grafana.app/updatedBy": "user:u000000001",
          "grafana.app/message": "Updated thresholds"
        }
      },
      "spec": {
        "title": "Production Overview",
        "schemaVersion": 41,
        ...
      }
    }
  ]
}
```

Each item in the `items` array represents a historical version of the resource. The following metadata fields are especially relevant for history:

- **`metadata.generation`**: The version number. This increments each time the resource's spec changes.
- **`metadata.resourceVersion`**: A unique identifier for this specific version. Use this for pagination and change detection.
- **`metadata.creationTimestamp`**: When this specific version was saved.
- **`metadata.annotations.grafana.app/updatedBy`**: The user who saved this version, in the format `<user-type>:<uid>`.
- **`metadata.annotations.grafana.app/message`**: An optional commit message set during the update. You can set this annotation when [updating a resource](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/dashboard/#update-dashboard).
- **`spec`**: The full resource spec as it existed at that version.

## Pagination

You can paginate through history using `limit` and `continue`, the same way you paginate a standard List request.

**Example request with pagination**:

```http
GET /apis/dashboard.grafana.app/v1/namespaces/default/dashboards?labelSelector=grafana.app/get-history=true&fieldSelector=metadata.name=production-overview&limit=2 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

If the response includes a `metadata.continue` field, pass it as the `continue` query parameter in the next request to fetch the next page. When the response doesn't include a `continue` field, you've reached the last page.

**Example request for the next page**:

```http
GET /apis/dashboard.grafana.app/v1/namespaces/default/dashboards?labelSelector=grafana.app/get-history=true&fieldSelector=metadata.name=production-overview&limit=2&continue=eyJvIjoxNTIsInYiOjE3NjE3MDQyMjQyMDcxODksInMiOmZhbHNlfQ== HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

## Retrieve a specific version

To retrieve the full resource at a specific historical version, use the standard Get endpoint with the `resourceVersion` query parameter set to the `metadata.resourceVersion` from the history list.

**Example request**:

```http
GET /apis/dashboard.grafana.app/v1/namespaces/default/dashboards/production-overview?resourceVersion=1758777451428100 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

This returns the complete resource as it existed at that version.

## Status codes

- **200**: OK
- **400**: Bad request (missing `fieldSelector`, invalid `labelSelector`, etc.)
- **401**: Unauthorized
- **403**: Access denied

# Secrets Management API

{{< admonition type="note" >}}
Available in Grafana 12 and later.

This API complies with the new Grafana API structure. To learn more refer to documentation about the [API structure in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis).

**This document may not contain the latest version of the API. For the most up-to-date list of available endpoints, refer to [secret.grafana.app/v1](https://play.grafana.org/swagger?api=secret.grafana.app-v1beta1) in Swagger.**

{{< /admonition >}}

The Grafana Secrets Management API allows you to manage secrets that are used by other services and applications within your Grafana instance.

{{< admonition type="caution" >}}
The API is currently in [public preview](https://grafana.com/docs/release-life-cycle/#public-preview) and might be subject to changes.
{{< /admonition >}}

## Requirements

If you're running Grafana Enterprise, you'll need to have specific permissions for some endpoints. Refer to [Role-based access control permissions](ref:rbac-permissions) for more information.

## Decrypters

The decrypters field is an allowlist that lets the secure value know which services and apps can decrypt the secret value.

Currently available decrypters:

- `k6-cloud` (for Grafana Cloud k6)
- `provisioning.grafana.app` (for GitSync/Provisioning)
- `synthetic-monitoring` (for Synthetic Monitoring checks)

## Create a secure value

`POST /apis/secret.grafana.app/v1beta1/namespaces/:namespace/securevalues`

Creates a new secure value.

**URL parameters**

- `namespace`: To read more about which namespace to use, see the [API overview](ref:api-overview).

**Request body**

- `metadata.name`: The Grafana unique identifier. If you do not want to provide this, set `metadata.generateName` instead to the prefix you would like for the randomly generated uid (can't be an empty string).
- `spec.description`: Short description that explains the purpose of this secure value. Required. Up to 25 characters long.
- `spec.value`: The secret value to store. Required. Up to 24576 bytes long.
- `spec.decrypters`: List of services allowed to decrypt this secure value. Up to 64 items, see note in [decrypters](#decrypters) for a list of supported values.

**Required permissions**

See note in the [introduction](#secrets-management-api) for an explanation.

<!-- prettier-ignore-start -->
| Action                       | Scope                                     |
| ---------------------------- | ----------------------------------------- |
| `secret.securevalues:create` | <ul><li>`secret.securevalues:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example create request**:

```http
POST /apis/secret.grafana.app/v1beta1/namespaces/default/securevalues HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "metadata": {
    "name": "api-key"
  },
  "spec": {
    "description": "External API Key",
    "value": "secret-api-key-12345",
    "decrypters": ["synthetic-monitoring"]
  }
}
```

**Example response**:

```http
HTTP/1.1 201 Created
Content-Type: application/json; charset=UTF-8
Content-Length: 343

{
  "apiVersion": "secret.grafana.app/v1beta1",
  "kind": "SecureValue",
  "metadata": {
    "name": "api-key",
    "namespace": "default",
    "uid": "f1e2d3c4-b5a6-7890-abcd-ef1234567890",
    "creationTimestamp": "2024-01-15T10:35:00Z"
  },
  "spec": {
    "description": "External API Key",
    "decrypters": ["synthetic-monitoring"]
  },
  "status": {}
}
```

Status Codes:

- **201** – Created
- **400** – Errors (invalid JSON, missing or invalid fields, etc)
- **401** – Unauthorized
- **403** – Access denied
- **409** – Conflict (secure value with the same name already exists)

{{< admonition type="note">}}
The `spec.value` field is never returned by API endpoints. Users cannot not decrypt secrets.
{{< /admonition >}}

## List secure values

`GET /apis/secret.grafana.app/v1beta1/namespaces/:namespace/securevalues`

List all secure values in a namespace.

**URL parameters**

- `namespace`: To read more about which namespace to use, see the [API overview](ref:api-overview).

**Query parameters**

- `labelSelector`: Filter secure values by labels.

**Required permissions**

See note in the [introduction](#secrets-management-api) for an explanation.

<!-- prettier-ignore-start -->
| Action                       | Scope                                     |
| ---------------------------- | ----------------------------------------- |
| `secret.securevalues:read`   | <ul><li>`secret.securevalues:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example list request**:

```http
GET /apis/secret.grafana.app/v1beta1/namespaces/default/securevalues HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 383

{
  "apiVersion": "secret.grafana.app/v1beta1",
  "kind": "SecureValueList",
  "metadata": {
    "resourceVersion": "12345"
  },
  "items": [
    {
      "apiVersion": "secret.grafana.app/v1beta1",
      "kind": "SecureValue",
      "metadata": {
        "name": "database-password",
        "namespace": "default",
        "creationTimestamp": "2024-01-15T10:30:00Z"
      },
      "spec": {
        "description": "Production DB Password",
        "decrypters": ["synthetic-monitoring"]
      },
      "status": {}
    }
  ]
}
```

Status Codes:

- **200** – OK
- **401** – Unauthorized
- **403** – Access denied

## Get a secure value

`GET /apis/secret.grafana.app/v1beta1/namespaces/:namespace/securevalues/:name`

Get the details of a specific secure value. It will not return the secret value.

**URL parameters**

- `namespace`: To read more about which namespace to use, see the [API overview](ref:api-overview).
- `name`: The name of the secure value.

**Required permissions**

See note in the [introduction](#secrets-management-api) for an explanation.

<!-- prettier-ignore-start -->
| Action                       | Scope                                     |
| ---------------------------- | ----------------------------------------- |
| `secret.securevalues:read`   | <ul><li>`secret.securevalues:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example get request**:

```http
GET /apis/secret.grafana.app/v1beta1/namespaces/default/securevalues/api-key HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 294

{
  "apiVersion": "secret.grafana.app/v1beta1",
  "kind": "SecureValue",
  "metadata": {
    "name": "api-key",
    "namespace": "default",
    "uid": "f1e2d3c4-b5a6-7890-abcd-ef1234567890",
    "creationTimestamp": "2024-01-15T10:35:00Z"
  },
  "spec": {
    "description": "External API Key",
    "decrypters": ["synthetic-monitoring"]
  },
  "status": {}
}
```

Status Codes:

- **200** – OK
- **401** – Unauthorized
- **403** – Access denied
- **404** – Not found

## Update a secure value

`PUT /apis/secret.grafana.app/v1beta1/namespaces/:namespace/securevalues/:name`

Replace an existing secure value with a new specification.

**URL parameters**

- `namespace`: To read more about which namespace to use, see the [API overview](ref:api-overview).
- `name`: The name of the secure value.

**Request body**

- `spec.description`: Short description that explains the purpose of this secure value. Required. Up to 25 characters long.
- `spec.value`: The secret value to store. Required. Up to 24576 bytes long.
- `spec.decrypters`: List of services allowed to decrypt this secure value. Up to 64 items, see note in [decrypters](#decrypters) for a list of supported values.

**Required permissions**

See note in the [introduction](#secrets-management-api) for an explanation.

<!-- prettier-ignore-start -->
| Action                      | Scope                                     |
| --------------------------- | ----------------------------------------- |
| `secret.securevalues:write` | <ul><li>`secret.securevalues:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example update request**:

```http
PUT /apis/secret.grafana.app/v1beta1/namespaces/default/securevalues/api-key HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "metadata": {
    "name": "api-key"
  },
  "spec": {
    "description": "External API Key",
    "value": "new-value-12345",
    "decrypters": ["synthetic-monitoring"]
  }
}
```

**Example response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 282

{
  "apiVersion": "secret.grafana.app/v1beta1",
  "kind": "SecureValue",
  "metadata": {
    "name": "api-key",
    "namespace": "default",
    "uid": "f1e2d3c4-b5a6-7890-abcd-ef1234567890",
    "creationTimestamp": "2024-01-15T10:35:00Z"
  },
  "spec": {
    "description": "External API Key",
    "decrypters": ["synthetic-monitoring"]
  }
}
```

Status Codes:

- **200** – OK
- **400** – Errors (invalid JSON, missing or invalid fields, etc)
- **401** – Unauthorized
- **403** – Access denied
- **404** – Not found

## Delete a secure value

`DELETE /apis/secret.grafana.app/v1beta1/namespaces/:namespace/securevalues/:name`

Permanently delete a secure value. This also deletes the underlying stored secret value.

**URL parameters**

- `namespace`: To read more about the namespace to use, see the [API overview](ref:api-overview).
- `name`: The name of the secure value.

**Required permissions**

See note in the [introduction](#secrets-management-api) for an explanation.

<!-- prettier-ignore-start -->
| Action                       | Scope                                     |
| ---------------------------- | ----------------------------------------- |
| `secret.securevalues:delete` | <ul><li>`secret.securevalues:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example delete request**:

```http
DELETE /apis/secret.grafana.app/v1beta1/namespaces/default/securevalues/api-key HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 65

{
  "apiVersion": "v1",
  "kind": "Status",
  "status": "Success",
  "code": 200
}
```

Status Codes:

- **200** – OK
- **401** – Unauthorized
- **403** – Access denied
- **404** – Not found

# Stars HTTP API

{{< admonition type="note" >}}
Available in Grafana 13 and later.

This API complies with the new Grafana API structure. To learn more refer to documentation about the [API structure in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis).

**This document may not contain the latest version of the API. For the most up-to-date list of available endpoints, refer to [collections.grafana.app/v1](https://play.grafana.org/swagger?api=collections.grafana.app.grafana.app-v1) in Swagger.**

{{< /admonition >}}

Also known as the `collections` API, `/apis/collections.grafana.app/*` allows you to star and unstar your resources in Grafana.

This API replaces `POST` and `DELETE` `/api/user/stars/dashboard/uid/{dashboard_uid}`, which are now are deprecated, although they keep working with no functional change. **Use the generic Collections Stars API for any new integrations**.

## Endpoints

These are the available endpoints:

| Method   | Summary                                 | URI                                                                                                            | Replaced endpoint                             |
| -------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------- | --------------------------------------------- |
| PUT/POST | [Star a resource](#star-a-resource)     | /apis/collections.grafana.app/v1alpha1/namespaces/{namespace}/stars/user-{user_uid}/update/{group}/{kind}/{id} | /api/user/stars/dashboard/uid/{dashboard_uid} |
| DELETE   | [Unstar a resource](#unstar-a-resource) | /apis/collections.grafana.app/v1alpha1/namespaces/{namespace}/stars/user-{user_uid}/update/{group}/{kind}/{id} | /api/user/stars/dashboard/uid/{dashboard_uid} |

### Path parameters

Parameters include:

- {namespace} is the same namespace you already use for other Grafana App Platform APIs.
  - In Grafana Cloud: your stack slug. For example, `my-stack`
  - In Grafana OSS/Enterprise: your organization's ID. For example. `org-1`

- {user_uid}: the signed-in user's UID, not their username or numeric ID. Look this up from the UID field returned by GET /api/user. Refer to [Migration steps](#migration-steps) for more details.
  - The resource name is always `user-{user_uid}`, for example `user-a1b2c3d4`.

One endpoint shape now covers every starrable resource instead of one endpoint per kind. For a given resource, {group} / {kind} / {id}: identify the resource being starred/unstarred:

- {group} = resource.grafana.app. For example `dashboard.grafana.app` or `folder.grafana.app`
- {kind} = resource. For example: `Dashboard` or `Folder`
- {id} = the resource UID, which has the same value used in the legacy path

## Star a resource

`PUT /apis/collections.grafana.app/v1alpha1/namespaces/{namespace}/stars/user-{user_uid}/update/{group}/{kind}/{id}`

Stars a resource for the signed-in user.

**Example request**:

None. The star target is fully encoded in the URL path; no JSON body is required or read.

**Example response**:

Status Codes:

- **200** - OK on a successful star (or unstar)
- **204** - No Content when unstarring a resource that has no stars recorded and therefore there's nothing to remove. It's not treated as an error
- **Standard 4xx errors** apply for auth/authorization failures, since you can only manage your own stars

## Unstar a resource

`DELETE /apis/collections.grafana.app/v1alpha1/namespaces/{namespace}/stars/user-{user_uid}/update/{group}/{kind}/{id}`

Unstars a resource for the signed-in user.

Refer to [Star a resource](#star-a-resource) for more details.

## Migration steps

To migrate from `/api/user/stars/*` to `/apis/collections.grafana.app/*` follow these steps:

1. Identify every direct caller of `POST`/`PUT` or `DELETE` `/api/user/stars/dashboard/uid/{uid}` in your integration. Skip the Grafana UI itself, which already calls the new API.
1. Use `GET /api/user` to resolve the signed-in user's UID and cache it for the session.
1. Determine the namespace your Grafana instance or stack uses for App Platform API calls.
1. Replace the call with `PUT` (star) or `DELETE` (unstar) against `
/apis/collections.grafana.app/v1alpha1/namespaces/{namespace}/stars/user-{uid}/update/dashboard.grafana.app/Dashboard/{dashboard_uid}`, **with no request body**.
1. Update response handling. Treat `200` as success for both star and unstar, and `204` as a no-op success when unstarring something that wasn't starred.
1. Leave any code that reads `GET /api/user/stars` as-is; it's unaffected by this change.

### Migration example

The following call in the legacy API:

```
  curl -X POST \
    -H "Authorization: Bearer $TOKEN" \
    "https://<your-grafana>/api/user/stars/dashboard/uid/abc123"
```

Becomes:

```
  USER_UID=$(curl -s -H "Authorization: Bearer $TOKEN" \
    "https://<your-grafana>/api/user" | jq -r .uid)

  curl -X PUT \
    -H "Authorization: Bearer $TOKEN" \
    "https://<your-grafana>/apis/collections.grafana.app/v1alpha1/namespaces/default/stars/user-${USER_UID}/update/dashboard.grafana.app/Dashboard/abc123"
```

Unstarring has an identical behavior, with `DELETE` instead of `PUT`/`POST`.

# RBAC API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

Use this API to create, update, delete, get, and list roles.

To check which basic or fixed roles have the required permissions, refer to [RBAC role definitions](/docs/grafana/latest/administration/roles-and-permissions/access-control/rbac-fixed-basic-role-definitions).

## Requirements

Role-based access control API is only available in Grafana Cloud or Grafana Enterprise. Read more about [Grafana Enterprise](/docs/grafana/latest/introduction/grafana-enterprise/).

## Get status

`GET /api/access-control/status`

Returns an indicator to check if role-based access control is enabled or not.

#### Required permissions

| Action               | Scope                  |
| -------------------- | ---------------------- |
| status:accesscontrol | services:accesscontrol |

#### Example request

```http
GET /api/access-control/status
Accept: application/json
Content-Type: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
  "enabled": true
}
```

#### Status codes

| Code | Description                                                                      |
| ---- | -------------------------------------------------------------------------------- |
| 200  | Returned a flag indicating if the role-based access control is enabled or no.    |
| 403  | Access denied                                                                    |
| 404  | Not found, an indication that role-based access control is not available at all. |
| 500  | Unexpected error. Refer to body and/or server logs for more details.             |

## Create and manage custom roles

### Get all roles

`GET /api/access-control/roles`

Gets all existing roles. The response contains all global and organization local roles, for the organization which user is signed in.

Query Parameters:

- `includeHidden`: Optional. Set to `true` to include roles that are `hidden`.

#### Required permissions

| Action     | Scope    |
| ---------- | -------- |
| roles:read | roles:\* |

#### Example request

```http
GET /api/access-control/roles
Accept: application/json
Content-Type: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

[
    {
        "version": 3,
        "uid": "XvHQJq57z",
        "name": "fixed:reports:reader",
        "displayName": "Report reader",
        "description": "Read all reports and shared report settings.",
        "group": "Reports",
        "updated": "2021-11-19T10:48:00+01:00",
        "created": "2021-11-19T10:48:00+01:00",
        "global": false
    },
    {
        "version": 5,
        "uid": "vi9mlLjGz",
        "name": "fixed:datasources.permissions:writer",
        "description: "Create, read or delete data source permissions.",
        "global": true,
        "updated": "2021-05-13T22:41:49+02:00",
        "created": "2021-05-13T16:24:26+02:00"
    }
]
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Global and organization local roles are returned.                    |
| 403  | Access denied                                                        |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

### Get a role

`GET /api/access-control/roles/:uid`

Get a role for the given UID.

#### Required permissions

| Action     | Scope    |
| ---------- | -------- |
| roles:read | roles:\* |

#### Example request

```http
GET /api/access-control/roles/PYnDO3rMk
Accept: application/json
Content-Type: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "version": 4,
    "uid": "6dNwJq57z",
    "name": "fixed:reports:writer",
    "displayName": "Report writer",
    "description": "Create, read, update, or delete all reports and shared report settings.",
    "group": "Reports",
    "permissions": [
        {
            "action": "reports:delete",
            "scope": "reports:*",
            "updated": "2021-11-19T10:48:00+01:00",
            "created": "2021-11-19T10:48:00+01:00"
        },
        {
            "action": "reports:read",
            "scope": "reports:*",
            "updated": "2021-11-19T10:48:00+01:00",
            "created": "2021-11-19T10:48:00+01:00"
        },
        {
            "action": "reports:send",
            "scope": "reports:*",
            "updated": "2021-11-19T10:48:00+01:00",
            "created": "2021-11-19T10:48:00+01:00"
        },
        {
            "action": "reports:create",
            "scope": "",
            "updated": "2021-11-19T10:48:00+01:00",
            "created": "2021-11-19T10:48:00+01:00"
        },
        {
            "action": "reports:write",
            "scope": "reports:*",
            "updated": "2021-11-19T10:48:00+01:00",
            "created": "2021-11-19T10:48:00+01:00"
        },
        {
            "action": "reports.settings:read",
            "scope": "",
            "updated": "2021-11-19T10:48:00+01:00",
            "created": "2021-11-19T10:48:00+01:00"
        },
        {
            "action": "reports.settings:write",
            "scope": "",
            "updated": "2021-11-19T10:48:00+01:00",
            "created": "2021-11-19T10:48:00+01:00"
        }
    ],
    "updated": "2021-11-19T10:48:00+01:00",
    "created": "2021-11-19T10:48:00+01:00",
    "global": false
}
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Role is returned.                                                    |
| 403  | Access denied.                                                       |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

### Create a new custom role

`POST /api/access-control/roles`

Creates a new custom role and maps given permissions to that role. Note that roles with the same prefix as [Fixed roles](/docs/grafana/latest/administration/roles-and-permissions/access-control/#fixed-roles) can't be created.

#### Required permissions

`permissions:type:delegate` scope ensures that users can only create custom roles with the same, or a subset of permissions which the user has.
For example, if a user does not have required permissions for creating users, they won't be able to create a custom role which allows to do that. This is done to prevent escalation of privileges.

| Action      | Scope                     |
| ----------- | ------------------------- |
| roles:write | permissions:type:delegate |

#### Example request

```http
POST /api/access-control/roles
Accept: application/json
Content-Type: application/json

{
    "uid": "jZrmlLCGka",
    "name": "custom:delete:roles",
    "displayName": "custom delete roles",
    "description": "My custom role which gives users permissions to delete roles",
    "group":"My Group",
    "displayName": "My Custom Role",
    "global": false,
    "permissions": [
        {
            "action": "roles:delete",
            "scope": "permissions:type:delegate"
        }
    ]
}
```

#### JSON body schema

| Field Name  | Date Type  | Required | Description                                                                                                                                                                                                                               |
| ----------- | ---------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| uid         | string     | No       | UID of the role. If not present, the UID is automatically created for you and returned in response. Refer to [Custom roles](/docs/grafana/latest/administration/roles-and-permissions/access-control/#custom-roles) for more information. |
| global      | boolean    | No       | A flag indicating if the role is global or not. If set to `false`, the default org ID of the authenticated user will be used from the request.                                                                                            |
| version     | number     | No       | **Deprecated.** Ignored on create; the server always assigns version 1 to new roles. The response returns the assigned version.                                                                                                           |
| name        | string     | Yes      | Name of the role. Refer to [Custom roles](/docs/grafana/latest/administration/roles-and-permissions/access-control/#custom-roles) for more information.                                                                                   |
| description | string     | No       | Description of the role.                                                                                                                                                                                                                  |
| displayName | string     | No       | Display name of the role, visible in the UI.                                                                                                                                                                                              |
| group       | string     | No       | The group name the role belongs to.                                                                                                                                                                                                       |
| hidden      | boolean    | No       | Specify whether the role is hidden or not. If set to `true`, then the role does not show in the role picker. It will not be listed by API endpoints unless explicitly specified.                                                          |
| permissions | Permission | No       | If not present, the role will be created without any permissions.                                                                                                                                                                         |

**Permission**

| Field Name | Data Type | Required | Description                                                                                                                                                                                                                                   |
| ---------- | --------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| action     | string    | Yes      | Refer to [Custom role actions and scopes](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for full list of available actions.                                                           |
| scope      | string    | No       | If not present, no scope will be mapped to the permission. Refer to [Custom role actions and scopes](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for full list of available scopes. |

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "version": 2,
    "uid": "jZrmlLCGka",
    "name": "custom:delete:create:roles",
    "displayName": "custom delete create roles",
    "description": "My custom role which gives users permissions to delete and create roles",
    "group":"My Group",
    "displayName": "My Custom Role",
    "global": false,
    "permissions": [
        {
            "action": "roles:delete",
            "scope": "permissions:type:delegate",
            "updated": "2021-05-13T23:19:46+02:00",
            "created": "2021-05-13T23:19:46+02:00"
        }
    ],
    "updated": "2021-05-13T23:20:51.416518+02:00",
    "created": "2021-05-13T23:19:46+02:00"
}
```

#### Create role validation errors

Permission validation only occurs when permission validation is enabled (`rbac.permission_validation_enabled = true`).

> It has been enabled by default since Grafana 10.2.

##### Invalid action

The following example shows a request with an invalid action. The action `serviceaccounts.permissions:reader` is not a valid action. The valid action should be `serviceaccounts.permissions:read`.

```http
POST /api/access-control/roles HTTP/1.1
Content-Type: application/json
{
	"Name": "Read Service Account with id 6",
	"Permissions": [
			{
			"action": "serviceaccounts.permissions:reader",
			"scope": "serviceaccounts:uid:6"
		}
	]
}
```

```http
HTTP/1.1 400 Bad Request
Content-Type: application/json
{
	"extra": {
		"validationError": "the provided action was not found in the list of valid actions: serviceaccounts.permissions:reader"
	},
	"message": "Permission contains an invalid action",
	"messageId": "accesscontrol.permission-invalid-action",
	"statusCode": 400,
	"traceID": ""
}
```

##### Invalid scope

The following example shows a request with an invalid scope. The scope `serviceaccounts:serviceaccount6` is not a valid scope for the action `serviceaccounts.permissions:read`. The valid scopes for this action are `*`, `serviceaccounts:*` and `serviceaccounts:id:*`.

```http
POST /api/access-control/roles HTTP/1.1
Content-Type: application/json
{
	"Name": "Read Service Account with id 6",
	"Permissions": [
			{
			"action": "serviceaccounts.permissions:read",
			"scope": "serviceaccounts:serviceaccount6"
		}
	]
}
```

```http
HTTP/1.1 400 Bad Request
Content-Type: application/json
{
	"extra": {
		"validationError": "unknown scope: serviceaccounts:serviceaccount6 for action: serviceaccounts.permissions:read provided, expected prefixes are [* serviceaccounts:* serviceaccounts:id:*]"
	},
	"message": "Invalid scope",
	"messageId": "accesscontrol.permission-invalid-scope",
	"statusCode": 400,
	"traceID": ""
}
```

#### Status codes

| Code | Description                                                                        |
| ---- | ---------------------------------------------------------------------------------- |
| 200  | Role is updated.                                                                   |
| 400  | Bad request (invalid json, missing content-type, missing or invalid fields, etc.). |
| 403  | Access denied (one of the specified permissions is not assigned to the requester)  |
| 500  | Unexpected error. Refer to body and/or server logs for more details.               |

### Update a role

`PUT /api/access-control/roles/:uid`

Update the role with the given UID, and its permissions. The operation is idempotent and all permissions of the role will be replaced based on the request content. Send the current `version` of the role (from the last GET or list response) for optimistic locking; the request fails if the role was updated elsewhere. The server auto-increments the stored version on each update.

You can update `custom` roles and `basic` roles permissions. However `fixed` roles cannot be updated.

#### Required permissions

`permissions:type:delegate` scope ensures that users can only update custom roles with the same, or a subset of permissions which the user has.
For example, if a user does not have required permissions for creating users, they won't be able to update a custom role which allows to do that. This is done to prevent escalation of privileges.

| Action      | Scope                     |
| ----------- | ------------------------- |
| roles:write | permissions:type:delegate |

#### Example request

```http
PUT /api/access-control/roles/jZrmlLCGka
Accept: application/json
Content-Type: application/json

{
    "version": 3,
    "name": "custom:delete:write:roles",
    "displayName": "custom delete write roles",
    "description": "My custom role which gives users permissions to delete and write roles",
    "group":"My Group",
    "displayName": "My Custom Role",
    "global": false,
    "permissions": [
        {
            "action": "roles:delete",
            "scope": "permissions:type:delegate"
        },
        {
            "action": "roles:write",
            "scope": "permissions:type:delegate"
        }
    ]
}
```

#### JSON body schema

| Field Name  | Data Type           | Required | Description                                                                                                                                                                                                                                      |
| ----------- | ------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| version     | number              | Yes      | Current version of the role (from the last GET or list). Required for optimistic locking; the request fails if the stored version is greater. **Deprecated** as an input to set a value—the server auto-increments the stored version on update. |
| name        | string              | Yes      | Name of the role.                                                                                                                                                                                                                                |
| description | string              | No       | Description of the role.                                                                                                                                                                                                                         |
| displayName | string              | No       | Display name of the role, visible in the UI.                                                                                                                                                                                                     |
| group       | string              | No       | The group name the role belongs to.                                                                                                                                                                                                              |
| hidden      | boolean             | No       | Specifies whether the role is hidden or not. If set to `true`, then the role does not show in the role picker. It will not be listed by API endpoints unless explicitly specified.                                                               |
| permissions | List of Permissions | No       | The full list of permissions for the role after the update.                                                                                                                                                                                      |

**Permission**

| Field Name | Data Type | Required | Description                                                                                                                                                                                                                                   |
| ---------- | --------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| action     | string    | Yes      | Refer to [Custom role actions and scopes](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for full list of available actions.                                                           |
| scope      | string    | No       | If not present, no scope will be mapped to the permission. Refer to [Custom role actions and scopes](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for full list of available scopes. |

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "version":3,
    "uid":"jZrmlLCGka",
    "name":"custom:delete:write:roles",
    "displayName":"custom delete write roles",
    "description":"My custom role which gives users permissions to delete and write roles",
    "group":"My Group",
    "displayName": "My Custom Role",
    "permissions":[
        {
            "action":"roles:delete",
            "scope":"permissions:type:delegate",
            "updated":"2021-08-06T18:27:40+02:00",
            "created":"2021-08-06T18:27:40+02:00"
        },
        {
            "action":"roles:write",
            "scope":"permissions:type:delegate",
            "updated":"2021-08-06T18:27:41+02:00",
            "created":"2021-08-06T18:27:41+02:00"
        }
    ],
    "updated":"2021-08-06T18:27:41+02:00",
    "created":"2021-08-06T18:27:40+02:00",
    "global":false
}
```

#### Update role validation errors

Permission validation only occurs when permission validation is enabled (`rbac.permission_validation_enabled = true`).

> It has been enabled by default since Grafana 10.2.

For more information, refer to [Create role validation errors](#create-role-validation-errors).

#### Status codes

| Code | Description                                                                        |
| ---- | ---------------------------------------------------------------------------------- |
| 200  | Role is updated.                                                                   |
| 400  | Bad request (invalid json, missing content-type, missing or invalid fields, etc.). |
| 403  | Access denied (one of the specified permissions is not assigned to the requester)  |
| 404  | Role was not found to update.                                                      |
| 500  | Unexpected error. Refer to body and/or server logs for more details.               |

### Delete a custom role

`DELETE /api/access-control/roles/:uid?force=false`

Delete a role with the given UID, and it's permissions. If the role is assigned, the deletion operation will fail, unless the `force` query param is set to `true`, and in that case all assignments will also be deleted.

#### Required permissions

`permissions:type:delegate` scope ensures that users can only delete a custom role with the same, or a subset of permissions which the user has.
For example, if a user does not have required permissions for creating users, they won't be able to delete a custom role which allows to do that.

| Action       | Scope                     |
| ------------ | ------------------------- |
| roles:delete | permissions:type:delegate |

#### Example request

```http
DELETE /api/access-control/roles/jZrmlLCGka?force=true&global=false
Accept: application/json
```

#### Query parameters

| Param  | Type    | Required | Description                                                                                                                                                                                                                                                             |
| ------ | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| force  | boolean | No       | When set to `true`, the role will be deleted with all its assignments.                                                                                                                                                                                                  |
| global | boolean | No       | A flag indicating if the role is global or not. If set to false, the default org ID of the authenticated user will be used from the request. Refer to the [About RBAC](/docs/grafana/latest/administration/roles-and-permissions/access-control/) for more information. |

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "message": "Role deleted"
}
```

#### Status codes

| Code | Description                                                                        |
| ---- | ---------------------------------------------------------------------------------- |
| 200  | Role is deleted.                                                                   |
| 400  | Bad request (invalid json, missing content-type, missing or invalid fields, etc.). |
| 403  | Access denied                                                                      |
| 500  | Unexpected error. Refer to body and/or server logs for more details.               |

## Create and remove user role assignments

### List roles assigned to a user

`GET /api/access-control/users/:userId/roles`

Lists the roles that have been directly assigned to a given user. The list does not include basic roles (Viewer, Editor, Admin or Grafana Admin), and it does not include roles that have been inherited from a team.

Query Parameters:

- `includeHidden`: Optional. Set to `true` to include roles that are `hidden`.
- `includeMapped`: Optional. Set to `true` to include roles that have been mapped through the group attribute sync feature.

#### Required permissions

| Action           | Scope                |
| ---------------- | -------------------- |
| users.roles:read | users:id:`<user ID>` |

#### Example request

```http
GET /api/access-control/users/1/roles
Accept: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

[
    {
        "version": 4,
        "uid": "6dNwJq57z",
        "name": "fixed:reports:writer",
        "displayName": "Report writer",
        "description": "Create, read, update, or delete all reports and shared report settings.",
        "group": "Reports",
        "updated": "2021-11-19T10:48:00+01:00",
        "created": "2021-11-19T10:48:00+01:00",
        "global": false
    }
]
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Set of assigned roles is returned.                                   |
| 403  | Access denied.                                                       |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

### List your permissions

`GET /api/access-control/user/permissions`

Lists the permissions granted to the signed in user.

#### Required permissions

No permission is required.

#### Query parameters

| Param       | Type    | Required | Description                            |
| ----------- | ------- | -------- | -------------------------------------- |
| reloadcache | boolean | No       | A flag to reload the permission cache. |

#### Example request

```http
GET /api/access-control/user/permissions
Accept: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
  "dashboards:read": ["dashboards:uid:70KrY6IVz"],
  "dashboards:write": ["dashboards:uid:70KrY6IVz"],
  "datasources.id:read": ["datasources:*"],
  "datasources:read": ["datasources:*"],
  "datasources:explore": [""],
  "datasources:query": ["datasources:uid:grafana"],
  "datasources:read": ["datasources:uid:grafana"],
  "orgs:read": [""]
}
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Set of assigned permissions is returned.                             |
| 403  | Access denied.                                                       |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

### List permissions assigned to a user

`GET /api/access-control/users/:userId/permissions`

Lists the permissions granted to a given user.

#### Required permissions

| Action                 | Scope                |
| ---------------------- | -------------------- |
| users.permissions:read | users:id:`<user ID>` |

#### Example request

```http
GET /api/access-control/users/1/permissions
Accept: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

[
    {
        "action": "ldap.status:read",
        "scope": ""
    },
    {
        "action": "ldap.user:read",
        "scope": ""
    }
]
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Set of assigned permissions is returned.                             |
| 403  | Access denied.                                                       |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

### Add a user role assignment

`POST /api/access-control/users/:userId/roles`

Assign a role to a specific user.

For bulk updates consider
[Set user role assignments](#set-user-role-assignments).

#### Required permissions

`permissions:type:delegate` scope ensures that users can only assign roles which have same, or a subset of permissions which the user has.
For example, if a user does not have required permissions for creating users, they won't be able to assign a role which will allow to do that. This is done to prevent escalation of privileges.

| Action          | Scope                     |
| --------------- | ------------------------- |
| users.roles:add | permissions:type:delegate |

#### Example request

```http
POST /api/access-control/users/1/roles
Accept: application/json
Content-Type: application/json

{
    "global": false,
    "roleUid": "XvHQJq57z"
}
```

#### JSON body schema

| Field Name | Data Type | Required | Description                                                                                                                                                                                  |
| ---------- | --------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| roleUid    | string    | Yes      | UID of the role.                                                                                                                                                                             |
| global     | boolean   | No       | A flag indicating if the assignment is global or not. If set to `false`, the default org ID of the authenticated user will be used from the request to create organization local assignment. |

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "message": "Role added to the user."
}
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Role is assigned to a user.                                          |
| 403  | Access denied.                                                       |
| 404  | Role not found.                                                      |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

## Remove a user role assignment

`DELETE /api/access-control/users/:userId/roles/:roleUID`

Revoke a role from a user.

For bulk updates consider
[Set user role assignments](#set-user-role-assignments).

#### Required permissions

`permissions:type:delegate` scope ensures that users can only unassign roles which have same, or a subset of permissions which the user has.
For example, if a user does not have required permissions for creating users, they won't be able to unassign a role which will allow to do that. This is done to prevent escalation of privileges.

| Action             | Scope                     |
| ------------------ | ------------------------- |
| users.roles:remove | permissions:type:delegate |

#### Query parameters

| Param  | Type    | Required | Description                                                                                                                                                               |
| ------ | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| global | boolean | No       | A flag indicating if the assignment is global or not. If set to `false`, the default org ID of the authenticated user will be used from the request to remove assignment. |

#### Example request

```http
DELETE /api/access-control/users/1/roles/AFUXBHKnk
Accept: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "message": "Role removed from user."
}
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Role is unassigned.                                                  |
| 403  | Access denied.                                                       |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

### Set user role assignments

`PUT /api/access-control/users/:userId/roles`

Update the user's role assignments to match the provided set of UIDs.
This will remove any assigned roles that aren't in the request and add
roles that are in the set but are not already assigned to the user.

If you want to add or remove a single role, consider using
[Add a user role assignment](#add-a-user-role-assignment) or
[Remove a user role assignment](#remove-a-user-role-assignment)
instead.

#### Required permissions

`permissions:type:delegate` scope ensures that users can only assign or unassign roles which have same, or a subset of permissions which the user has.
For example, if a user does not have required permissions for creating users, they won't be able to assign or unassign a role which will allow to do that. This is done to prevent escalation of privileges.

| Action             | Scope                     |
| ------------------ | ------------------------- |
| users.roles:add    | permissions:type:delegate |
| users.roles:remove | permissions:type:delegate |

#### Example request

```http
PUT /api/access-control/users/1/roles
Accept: application/json
Content-Type: application/json

{
    "global": false,
    "roleUids": [
        "ZiHQJq5nk",
        "GzNQ1357k"
    ]
}
```

#### JSON body schema

| Field Name    | Date Type | Required | Description                                                                                                                                          |
| ------------- | --------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| global        | boolean   | No       | A flag indicating if the assignment is global or not. If set to `false`, the default org ID of the authenticated user will be used from the request. |
| roleUids      | list      | Yes      | List of role UIDs.                                                                                                                                   |
| includeHidden | boolean   | No       | Specify whether the hidden role assignments should be updated.                                                                                       |

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "message": "User roles have been updated."
}
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Roles have been assigned.                                            |
| 403  | Access denied.                                                       |
| 404  | Role not found.                                                      |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

## Create and remove service account role assignments

### List roles assigned to a service account

`GET /api/access-control/users/:serviceAccountId/roles`

Lists the roles that have been directly assigned to a given service account. The list does not include basic roles (Viewer, Editor, Admin or Grafana Admin).

Query Parameters:

- `includeHidden`: Optional. Set to `true` to include roles that are `hidden`.

#### Required permissions

| Action           | Scope                           |
| ---------------- | ------------------------------- |
| users.roles:read | users:id:`<service account ID>` |

#### Example request

```http
GET /api/access-control/users/1/roles
Accept: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

[
    {
        "version": 4,
        "uid": "6dNwJq57z",
        "name": "fixed:reports:writer",
        "displayName": "Report writer",
        "description": "Create, read, update, or delete all reports and shared report settings.",
        "group": "Reports",
        "updated": "2021-11-19T10:48:00+01:00",
        "created": "2021-11-19T10:48:00+01:00",
        "global": false
    }
]
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Set of assigned roles is returned.                                   |
| 403  | Access denied.                                                       |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

### List permissions assigned to a service account

`GET /api/access-control/users/:serviceAccountId/permissions`

Lists the permissions that a given service account has.

#### Required permissions

| Action                 | Scope                           |
| ---------------------- | ------------------------------- |
| users.permissions:read | users:id:`<service account ID>` |

#### Example request

```http
GET /api/access-control/users/1/permissions
Accept: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

[
    {
        "action": "ldap.status:read",
        "scope": ""
    },
    {
        "action": "ldap.user:read",
        "scope": ""
    }
]
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Set of assigned permissions is returned.                             |
| 403  | Access denied.                                                       |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

### Add a service account role assignment

`POST /api/access-control/users/:serviceAccountId/roles`

Assign a role to a specific service account.

For bulk updates consider
[Set service account role assignments](#set-service-account-role-assignments).

#### Required permissions

`permissions:type:delegate` scope ensures that users can only assign roles which have same, or a subset of permissions which the user has.
For example, if a user does not have required permissions for creating users, they won't be able to assign a role which will allow to do that. This is done to prevent escalation of privileges.

| Action          | Scope                     |
| --------------- | ------------------------- |
| users.roles:add | permissions:type:delegate |

#### Example request

```http
POST /api/access-control/users/1/roles
Accept: application/json
Content-Type: application/json

{
    "global": false,
    "roleUid": "XvHQJq57z"
}
```

#### JSON body schema

| Field Name | Data Type | Required | Description                                                                                                                                                                                  |
| ---------- | --------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| roleUid    | string    | Yes      | UID of the role.                                                                                                                                                                             |
| global     | boolean   | No       | A flag indicating if the assignment is global or not. If set to `false`, the default org ID of the authenticated user will be used from the request to create organization local assignment. |

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "message": "Role added to the user."
}
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Role is assigned to a user.                                          |
| 403  | Access denied.                                                       |
| 404  | Role not found.                                                      |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

## Remove a service account role assignment

`DELETE /api/access-control/users/:serviceAccountId/roles/:roleUID`

Revoke a role from a service account.

For bulk updates consider
[Set service account role assignments](#set-service-account-role-assignments).

#### Required permissions

`permissions:type:delegate` scope ensures that users can only unassign roles which have same, or a subset of permissions which the user has.
For example, if a user does not have required permissions for creating users, they won't be able to unassign a role which will allow to do that. This is done to prevent escalation of privileges.

| Action             | Scope                     |
| ------------------ | ------------------------- |
| users.roles:remove | permissions:type:delegate |

#### Query parameters

| Param  | Type    | Required | Description                                                                                                                                                               |
| ------ | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| global | boolean | No       | A flag indicating if the assignment is global or not. If set to `false`, the default org ID of the authenticated user will be used from the request to remove assignment. |

#### Example request

```http
DELETE /api/access-control/users/1/roles/AFUXBHKnk
Accept: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "message": "Role removed from user."
}
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Role is unassigned.                                                  |
| 403  | Access denied.                                                       |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

### Set service account role assignments

`PUT /api/access-control/users/:serviceAccountId/roles`

Update the service accounts's role assignments to match the provided set of UIDs.
This will remove any assigned roles that aren't in the request and add
roles that are in the set but are not already assigned to the service account.

If you want to add or remove a single role, consider using
[Add a service account role assignment](#add-a-service-account-role-assignment) or
[Remove a service account role assignment](#remove-a-service-account-role-assignment)
instead.

#### Required permissions

`permissions:type:delegate` scope ensures that users can only assign or unassign roles which have same, or a subset of permissions which the user has.
For example, if a user does not have required permissions for creating users, they won't be able to assign or unassign a role which will allow to do that. This is done to prevent escalation of privileges.

| Action             | Scope                     |
| ------------------ | ------------------------- |
| users.roles:add    | permissions:type:delegate |
| users.roles:remove | permissions:type:delegate |

#### Example request

```http
PUT /api/access-control/users/1/roles
Accept: application/json
Content-Type: application/json

{
    "global": false,
    "roleUids": [
        "ZiHQJq5nk",
        "GzNQ1357k"
    ]
}
```

#### JSON body schema

| Field Name    | Date Type | Required | Description                                                                                                                                          |
| ------------- | --------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| global        | boolean   | No       | A flag indicating if the assignment is global or not. If set to `false`, the default org ID of the authenticated user will be used from the request. |
| roleUids      | list      | Yes      | List of role UIDs.                                                                                                                                   |
| includeHidden | boolean   | No       | Specify whether the hidden role assignments should be updated.                                                                                       |

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "message": "User roles have been updated."
}
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Roles have been assigned.                                            |
| 403  | Access denied.                                                       |
| 404  | Role not found.                                                      |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

## Create and remove team role assignments

### List roles assigned to a team

`GET /api/access-control/teams/:teamId/roles`

Lists the roles that have been directly assigned to a given team.

Query Parameters:

- `includeHidden`: Optional. Set to `true` to include roles that are `hidden`.

#### Required permissions

| Action           | Scope                |
| ---------------- | -------------------- |
| teams.roles:read | teams:id:`<team ID>` |

#### Example request

```http
GET /api/access-control/teams/1/roles
Accept: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

[
    {
        "version": 4,
        "uid": "j08ZBi-nk",
        "name": "fixed:licensing:reader",
        "displayName": "Licensing reader",
        "description": "Read licensing information and licensing reports.",
        "group": "Licenses",
        "updated": "2022-02-03T14:19:50+01:00",
        "created": "0001-01-01T00:00:00Z",
        "global": false
    }
]
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Set of assigned roles is returned.                                   |
| 403  | Access denied.                                                       |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

### Add a team role assignment

`POST /api/access-control/teams/:teamId/roles`

Assign a role to a specific team.

For bulk updates consider [Set team role assignments](#set-team-role-assignments).

#### Required permissions

`permissions:type:delegate` scope ensures that users can only assign roles which have same, or a subset of permissions which the user has.
For example, if a user does not have the permissions required to create users, they won't be able to assign a role that contains these permissions. This is done to prevent escalation of privileges.

| Action          | Scope                     |
| --------------- | ------------------------- |
| teams.roles:add | permissions:type:delegate |

#### Example request

```http
POST /api/access-control/teams/1/roles
Accept: application/json
Content-Type: application/json

{
    "roleUid": "XvHQJq57z"
}
```

#### JSON body schema

| Field Name | Data Type | Required | Description      |
| ---------- | --------- | -------- | ---------------- |
| roleUid    | string    | Yes      | UID of the role. |

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "message": "Role added to the team."
}
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Role is assigned to a team.                                          |
| 403  | Access denied.                                                       |
| 404  | Role not found.                                                      |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

## Remove a team role assignment

`DELETE /api/access-control/teams/:teams/roles/:roleUID`

Revoke a role from a team.

For bulk updates consider [Set team role assignments](#set-team-role-assignments).

#### Required permissions

`permissions:type:delegate` scope ensures that users can only unassign roles which have same, or a subset of permissions which the user has.
For example, if a user does not have the permissions required to create users, they won't be able to assign a role that contains these permissions. This is done to prevent escalation of privileges.```

| Action             | Scope                     |
| ------------------ | ------------------------- |
| teams.roles:remove | permissions:type:delegate |

#### Example request

```http
DELETE /api/access-control/teams/1/roles/AFUXBHKnk
Accept: application/json
```

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "message": "Role removed from team."
}
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Role is unassigned.                                                  |
| 403  | Access denied.                                                       |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

### Set team role assignments

`PUT /api/access-control/teams/:teamId/roles`

Update the team's role assignments to match the provided set of UIDs.
This will remove any assigned roles that aren't in the request and add
roles that are in the set but are not already assigned to the user.

If you want to add or remove a single role, consider using
[Add a team role assignment](#add-a-team-role-assignment) or
[Remove a team role assignment](#remove-a-team-role-assignment)
instead.

#### Required permissions

`permissions:type:delegate` scope ensures that users can only assign or unassign roles which have same, or a subset of permissions which the user has.
For example, if a user does not have required permissions for creating users, they won't be able to assign or unassign a role to a team which will allow to do that. This is done to prevent escalation of privileges.

| Action             | Scope                     |
| ------------------ | ------------------------- |
| teams.roles:add    | permissions:type:delegate |
| teams.roles:remove | permissions:type:delegate |

#### Example request

```http
PUT /api/access-control/teams/1/roles
Accept: application/json
Content-Type: application/json

{
    "roleUids": [
        "ZiHQJq5nk",
        "GzNQ1357k"
    ]
}
```

#### JSON body schema

| Field Name    | Date Type | Required | Description                                                    |
| ------------- | --------- | -------- | -------------------------------------------------------------- |
| roleUids      | list      | Yes      | List of role UIDs.                                             |
| includeHidden | boolean   | No       | Specify whether the hidden role assignments should be updated. |

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "message": "Team roles have been updated."
}
```

#### Status codes

| Code | Description                                                          |
| ---- | -------------------------------------------------------------------- |
| 200  | Roles have been assigned.                                            |
| 403  | Access denied.                                                       |
| 404  | Role not found.                                                      |
| 500  | Unexpected error. Refer to body and/or server logs for more details. |

## Reset basic roles to their default

`POST /api/access-control/roles/hard-reset`

`permissions:type:escalate` scope enables users to reset basic roles permissions.
This could result in basic roles having permissions exceedind those of callers.

Reset basic roles permissions to their default.

#### Required permissions

| Action      | Scope                     |
| ----------- | ------------------------- |
| roles:write | permissions:type:escalate |

#### Example request

```http
POST /api/access-control/roles/hard-reset
Accept: application/json
Content-Type: application/json

{
    "BasicRoles": true
}
```

#### JSON body schema

| Field Name | Data Type | Required | Description                              |
| ---------- | --------- | -------- | ---------------------------------------- |
| BasicRoles | boolean   | No       | Option to reset basic roles permissions. |

#### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "message": "Reset performed"
}
```

#### Status codes

| Code | Description                 |
| ---- | --------------------------- |
| 200  | Reset performed             |
| 500  | Failed to reset basic roles |

# Admin API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Requirements

**To use the Admin API endpoints you have to use Basic authentication, and the Grafana user must have the Grafana server administrator permission**. The `admin` user that Grafana is provisioned with by default has permissions to use these API endpoints.

You can't authenticate to the Admin HTTP API with service account tokens. Service accounts are limited to an organization and an organization role, and can't be granted [Grafana server administrator permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/#grafana-server-administrators).

If you're running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## Fetch settings

`GET /api/admin/settings`

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

| Action        | Scope                                                                               |
| ------------- | ----------------------------------------------------------------------------------- |
| settings:read | settings:\*_<br>settings:auth.saml:_<br>settings:auth.saml:enabled (property level) |

**Example Request**:

```http
GET /api/admin/settings
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "DEFAULT": {
    "app_mode":"production"
  },
  "analytics": {
    "google_analytics_ua_id":"",
    "reporting_enabled":"false"
  },
  "auth.anonymous":{
    "enabled":"true",
    "org_name":"Main Org.",
    "org_role":"Viewer"
  },
  "auth.basic":{
    "enabled":"false"
  },
  "auth.github":{
    "allow_sign_up":"false",
    "allowed_domains":"",
    "allowed_organizations":"",
    "api_url":"https://api.github.com/user",
    "auth_url":"https://github.com/login/oauth/authorize",
    "client_id":"some_id",
    "client_secret":"************",
    "enabled":"false",
    "scopes":"user:email,read:org",
    "team_ids":"",
    "token_url":"https://github.com/login/oauth/access_token"
  },
  "auth.google":{
    "allow_sign_up":"false","allowed_domains":"",
    "api_url":"https://www.googleapis.com/oauth2/v1/userinfo",
    "auth_url":"https://accounts.google.com/o/oauth2/auth",
    "client_id":"some_client_id",
    "client_secret":"************",
    "enabled":"false",
    "scopes":"https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email",
    "token_url":"https://accounts.google.com/o/oauth2/token"
  },
  "auth.ldap":{
    "config_file":"/etc/grafana/ldap.toml",
    "enabled":"false"
  },
  "auth.proxy":{
    "auto_sign_up":"true",
    "enabled":"false",
    "header_name":"X-WEBAUTH-USER",
    "header_property":"username"
  },
  "dashboards.json":{
    "enabled":"false",
    "path":"/var/lib/grafana/dashboards"
  },
  "database":{
    "host":"127.0.0.1:0000",
    "name":"grafana",
    "password":"************",
    "path":"grafana.db",
    "ssl_mode":"disable",
    "type":"sqlite3",
    "user":"root"
  },
  "emails":{
    "templates_pattern":"emails/*.html, emails/*.txt",
    "welcome_email_on_sign_up":"false",
    "content_types":"text/html"
  },
  "log":{
    "buffer_len":"10000",
    "level":"Info",
    "mode":"file"
  },
  "log.console":{
    "level":""
  },
  "log.file":{
    "daily_rotate":"true",
    "file_name":"",
    "level":"",
    "log_rotate":"true",
    "max_days":"7",
    "max_lines":"1000000",
    "max_lines_shift":"28",
    "max_size_shift":""
  },
  "paths":{
    "data":"/tsdb/grafana",
    "logs":"/logs/apps/grafana"},
    "security":{
    "admin_password":"************",
    "admin_user":"admin",
    "cookie_remember_name":"grafana_remember",
    "cookie_username":"grafana_user",
    "disable_gravatar":"false",
    "login_remember_days":"7",
    "secret_key":"************"
  },
  "server":{
    "cert_file":"",
    "cert_key":"",
    "certs_watch_interval": "0s",
    "domain":"mygraf.com",
    "enable_gzip":"true",
    "enforce_domain":"false",
    "http_addr":"127.0.0.1",
    "http_port":"0000",
    "protocol":"http",
    "root_url":"%(protocol)s://%(domain)s:%(http_port)s/",
    "router_logging":"true",
    "data_proxy_logging":"true",
    "static_root_path":"public"
  },
  "session":{
    "cookie_name":"grafana_sess",
    "cookie_secure":"false",
    "gc_interval_time":"",
    "provider":"file",
    "provider_config":"sessions",
    "session_life_time":"86400"
  },
  "smtp":{
    "cert_file":"",
    "enabled":"false",
    "from_address":"admin@grafana.localhost",
    "from_name":"Grafana",
    "ehlo_identity":"dashboard.example.com",
    "host":"localhost:25",
    "key_file":"",
    "password":"************",
    "skip_verify":"false",
    "user":""
  },
  "users":{
    "allow_org_create":"true",
    "allow_sign_up":"false",
    "auto_assign_org":"true",
    "auto_assign_org_role":"Viewer"
  }
}
```

## Update settings

`PUT /api/admin/settings`

{{< admonition type="note" >}}
Available in Grafana Enterprise v8.0+.
{{< /admonition >}}

Updates/removes and reloads database settings. You must provide either `updates`, `removals` or both.

This endpoint only supports changes to `auth.saml` configuration.

This endpoint writes to Grafana's settings table, where values override the configuration file, environment variables, and command line arguments until you remove them with `removals`. Grafana gives no indication that a stored value is in effect, so a credential rotation done by editing the configuration file silently keeps using the old value. The SSO Settings API, the user interfaces, and Terraform write a different table; for SAML, `DELETE /api/v1/sso-settings/saml` clears both. Refer to [Check for stored settings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/settings-updates-at-runtime/#check-for-stored-settings).

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

| Action         | Scope                                                                               |
| -------------- | ----------------------------------------------------------------------------------- |
| settings:write | settings:\*_<br>settings:auth.saml:_<br>settings:auth.saml:enabled (property level) |

**Example request:**

```http
PUT /api/admin/settings
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "updates": {
    "auth.saml": {
      "enabled": "true"
    }
  },
  "removals": {
    "auth.saml": ["single_logout"]
  },
}
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 32

{
  "message":"Settings updated"
}
```

Status codes:

- **200** - OK
- **400** - Bad Request
- **401** - Unauthorized
- **403** - Forbidden
- **500** - Internal Server Error

## Grafana Stats

`GET /api/admin/stats`

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

| Action            | Scope |
| ----------------- | ----- |
| server.stats:read | n/a   |

**Example Request**:

```http
GET /api/admin/stats
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "users":2,
  "orgs":1,
  "dashboards":4,
  "snapshots":2,
  "tags":6,
  "datasources":1,
  "playlists":1,
  "stars":2,
  "alerts":2,
  "activeUsers":1
}
```

## Grafana Usage Report preview

`GET /api/admin/usage-report-preview`

Previews usage report to be sent to vendor.

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

**Example Request**:

```http
GET /api/admin/usage-report-preview
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
	"version": "8_4_0",
	"metrics": {
		"stats.active_admins.count": 1,
		"stats.active_editors.count": 1,
		"stats.active_sessions.count": 0,
		"stats.active_users.count": 2,
		"stats.active_viewers.count": 0,
		"stats.admins.count": 1,
		"stats.alert_rules.count": 0,
		"stats.alerting.ds.other.count": 0,
		"stats.alerts.count": 5,
		"stats.annotations.count": 6,
		"stats.api_keys.count": 1
  }
}
```

## Global Users

`POST /api/admin/users`

Creates new user.

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

| Action       | Scope |
| ------------ | ----- |
| users:create | n/a   |

**Example Request**:

```http
POST /api/admin/users HTTP/1.1
Accept: application/json
Content-Type: application/json

{
  "name":"User",
  "email":"user@graf.com",
  "login":"user",
  "password":"userpassword",
  "OrgId": 1
}
```

Note that `OrgId` is an optional parameter that can be used to assign a new user to a different organization when [auto_assign_org](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#auto-assign-org) is set to `true`.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"id":5,"message":"User created"}
```

## Password for User

`PUT /api/admin/users/:id/password`

Changes password for a specific user.

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

| Action               | Scope           |
| -------------------- | --------------- |
| users.password:write | global.users:\* |

**Example Request**:

```http
PUT /api/admin/users/2/password HTTP/1.1
Accept: application/json
Content-Type: application/json

{"password":"userpassword"}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message": "User password updated"}
```

## Permissions

`PUT /api/admin/users/:id/permissions`

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

| Action                  | Scope           |
| ----------------------- | --------------- |
| users.permissions:write | global.users:\* |

**Example Request**:

```http
PUT /api/admin/users/2/permissions HTTP/1.1
Accept: application/json
Content-Type: application/json

{"isGrafanaAdmin": true}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message": "User permissions updated"}
```

## Delete global User

`DELETE /api/admin/users/:id`

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

| Action       | Scope           |
| ------------ | --------------- |
| users:delete | global.users:\* |

**Example Request**:

```http
DELETE /api/admin/users/2 HTTP/1.1
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message": "User deleted"}
```

## Auth tokens for User

`GET /api/admin/users/:id/auth-tokens`

Returns a list of all auth tokens (devices) that the user currently have logged in from.

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

| Action               | Scope           |
| -------------------- | --------------- |
| users.authtoken:read | global.users:\* |

**Example Request**:

```http
GET /api/admin/users/1/auth-tokens HTTP/1.1
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "id": 361,
    "isActive": false,
    "clientIp": "127.0.0.1",
    "browser": "Chrome",
    "browserVersion": "72.0",
    "os": "Linux",
    "osVersion": "",
    "device": "Other",
    "createdAt": "2019-03-05T21:22:54+01:00",
    "seenAt": "2019-03-06T19:41:06+01:00"
  },
  {
    "id": 364,
    "isActive": false,
    "clientIp": "127.0.0.1",
    "browser": "Mobile Safari",
    "browserVersion": "11.0",
    "os": "iOS",
    "osVersion": "11.0",
    "device": "iPhone",
    "createdAt": "2019-03-06T19:41:19+01:00",
    "seenAt": "2019-03-06T19:41:21+01:00"
  }
]
```

## Revoke auth token for User

`POST /api/admin/users/:id/revoke-auth-token`

Revokes the given auth token (device) for the user. User of issued auth token (device) will no longer be logged in
and will be required to authenticate again upon next activity.

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

| Action                | Scope           |
| --------------------- | --------------- |
| users.authtoken:write | global.users:\* |

**Example Request**:

```http
POST /api/admin/users/1/revoke-auth-token HTTP/1.1
Accept: application/json
Content-Type: application/json

{
  "authTokenId": 364
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "message": "User auth token revoked"
}
```

## Logout User

`POST /api/admin/users/:id/logout`

Logout user revokes all auth tokens (devices) for the user. User of issued auth tokens (devices) will no longer be logged in
and will be required to authenticate again upon next activity.

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

| Action       | Scope           |
| ------------ | --------------- |
| users.logout | global.users:\* |

**Example Request**:

```http
POST /api/admin/users/1/logout HTTP/1.1
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "message": "User auth token revoked"
}
```

## Reload provisioning configurations

`POST /api/admin/provisioning/dashboards/reload`

`POST /api/admin/provisioning/datasources/reload`

`POST /api/admin/provisioning/plugins/reload`

`POST /api/admin/provisioning/access-control/reload`

`POST /api/admin/provisioning/alerting/reload`

Reloads the provisioning config files for specified type and provision entities again. It won't return
until the new provisioned entities are already stored in the database. In case of dashboards, it will stop
polling for changes in dashboard files and then restart it with new configurations after returning.

**Required permissions**

Only works with Basic Authentication (username and password). Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/admin#requirements) for more information.

| Action              | Scope                      | Provision entity |
| ------------------- | -------------------------- | ---------------- |
| provisioning:reload | provisioners:accesscontrol | accesscontrol    |
| provisioning:reload | provisioners:dashboards    | dashboards       |
| provisioning:reload | provisioners:datasources   | datasources      |
| provisioning:reload | provisioners:plugins       | plugins          |
| provisioning:reload | provisioners:alerting      | alerting         |

**Example Request**:

```http
POST /api/admin/provisioning/dashboards/reload HTTP/1.1
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "message": "Dashboards config reloaded"
}
```

## Reload LDAP configuration

**This endpoint has been removed from the Admin API. Starting with Grafana 12.4.0, SSO settings automatically reload the LDAP settings from the database or from the .toml file.**

`POST /api/admin/ldap/reload`

**Example Response**:

```http
HTTP/1.1 410
Content-Type: application/json

{"message":"This endpoint has been removed. LDAP configuration is now loaded from SSO settings."}
```

## Rotate data encryption keys

`POST /api/admin/encryption/rotate-data-keys`

[Rotates](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/configure-database-encryption/#rotate-data-keys) the data encryption keys.

**Example Request**:

```http
POST /api/admin/encryption/rotate-data-keys HTTP/1.1
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 204
Content-Type: application/json
```

## Re-encrypt data encryption keys

`POST /api/admin/encryption/reencrypt-data-keys`

[Re-encrypts](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/configure-database-encryption/#re-encrypt-data-keys) the data encryption keys.

**Example Request**:

```http
POST /api/admin/encryption/reencrypt-data-keys HTTP/1.1
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 204
Content-Type: application/json
```

## Re-encrypt secrets

`POST /api/admin/encryption/reencrypt-secrets`

[Re-encrypts](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/configure-database-encryption/#re-encrypt-secrets) secrets.

**Example Request**:

```http
POST /api/admin/encryption/reencrypt-secrets HTTP/1.1
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 204
Content-Type: application/json
```

## Roll back secrets

`POST /api/admin/encryption/rollback-secrets`

[Rolls back](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/configure-database-encryption/#roll-back-secrets) secrets.

**Example Request**:

```http
POST /api/admin/encryption/rollback-secrets HTTP/1.1
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 204
Content-Type: application/json
```

# Alerting provisioning HTTP API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="alerts/alerting_provisioning.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Annotations API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

Annotations are saved in the Grafana database (sqlite, mysql or postgres). Annotations can be organization annotations that can be shown on any dashboard by configuring an annotation data source - they are filtered by tags. Or they can be tied to a panel on a dashboard and are then only shown on that panel.

## Requirements

If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## Find Annotations

`GET /api/annotations?from=1506676478816&to=1507281278816&tags=tag1&tags=tag2&limit=100`

**Required permissions**

See note in the [introduction](#annotations-api) for an explanation.

<!-- prettier-ignore-start -->
| Action             | Scope                                                                                                                                                        |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `annotations:read` | <ul><li>`annotations:*`</li><li>`annotations:type:*`</li><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example Request**:

```http
GET /api/annotations?from=1506676478816&to=1507281278816&tags=tag1&tags=tag2&limit=100 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

Query Parameters:

- `from`: epoch datetime in milliseconds. Optional.
- `to`: epoch datetime in milliseconds. Optional.
- `limit`: number. Optional - default is 100. Max limit for results returned.
- `alertId`: number. Optional. Find annotations for a specified alert.
- `dashboardId`: Deprecated. Use dashboardUID instead.
- `dashboardUID`: string. Optional. Find annotations that are scoped to a specific dashboard, when dashboardUID presents, dashboardId would be ignored.
- `panelId`: number. Optional. Find annotations that are scoped to a specific panel
- `userId`: number. Optional. Find annotations created by a specific user
- `type`: string. Optional. `alert`|`annotation` Return alerts or user created annotations
- `tags`: string. Optional. Use this to filter organization annotations. Organization annotations are annotations from an annotation data source that are not connected specifically to a dashboard or panel. To do an "AND" filtering with multiple tags, specify the tags parameter multiple times e.g. `tags=tag1&tags=tag2`.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json
[
    {
        "id": 1124,
        "dashboardUID": "uGlb_lG7z",
        "panelId": 2,
        "userId": 1,
        "userName": "",
        "time": 1507266395000,
        "timeEnd": 1507266395000,
        "text": "test",
        "tags": [
            "tag1",
            "tag2"
        ],
        "data": {}
    },
    {
        "id": 1123,
        "dashboardUID": "jcIIG-07z",
        "panelId": 2,
        "userId": 1,
        "userName": "",
        "time": 1507265111000,
        "text": "test",
        "tags": [
            "tag1",
            "tag2"
        ],
        "data": {}
    }
]
```

{{< admonition type="note" >}}
Alerting fields, `alertId`, `alertName`, `newState`, `prevState`, are omitted when the annotation isn't tied to an alert. A missing `alertId` means the same as `0`: not an alert annotation.
{{< /admonition >}}

> Starting in Grafana v6.4 regions annotations are now returned in one entity that now includes the timeEnd property.

## Create Annotation

Creates an annotation in the Grafana database. The `dashboardUid` and `panelId` fields are optional.
If they are not specified then an organization annotation is created and can be queried in any dashboard that adds
the Grafana annotations data source. When creating a region annotation include the timeEnd property.

The format for `time` and `timeEnd` should be epoch numbers in millisecond resolution.

`POST /api/annotations`

**Required permissions**

See note in the [introduction](#annotations-api) for an explanation.

<!-- prettier-ignore-start -->
| Action               | Scope                                                                                                                                                        |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `annotations:create` | <ul><li>`annotations:*`</li><li>`annotations:type:*`</li><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Required JSON Body Fields**

- `text`: description of the annotation.

**Example Request**:

```http
POST /api/annotations HTTP/1.1
Accept: application/json
Content-Type: application/json

{
  "dashboardUID":"jcIIG-07z",
  "time":1507037197339,
  "timeEnd":1507180805056,
  "tags":["tag1","tag2"],
  "text":"Annotation Description"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "message":"Annotation added",
    "id": 1,
}
```

> The response for this HTTP request is slightly different in versions prior to v6.4. In prior versions you would
> also get an endId if you where creating a region. But in 6.4 regions are represented using a single event with time and
> timeEnd properties.

## Create Annotation in Graphite format

Creates an annotation by using Graphite-compatible event format. The `when` and `data` fields are optional. If `when` is not specified then the current time will be used as annotation's timestamp. The `tags` field can also be in prior to Graphite `0.10.0`
format (string with multiple tags being separated by a space).

`POST /api/annotations/graphite`

**Required permissions**

See note in the [introduction](#annotations-api) for an explanation.

| Action               | Scope                           |
| -------------------- | ------------------------------- |
| `annotations:create` | `annotations:type:organization` |

**Example Request**:

```http
POST /api/annotations/graphite HTTP/1.1
Accept: application/json
Content-Type: application/json

{
  "what": "Event - deploy",
  "tags": ["deploy", "production"],
  "when": 1467844481,
  "data": "deploy of main branch happened at Wed Jul 6 22:34:41 UTC 2016"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "message":"Graphite annotation added",
    "id": 1
}
```

## Update Annotation

`PUT /api/annotations/:id`

Updates all properties of an annotation that matches the specified id. To only update certain property, consider using the [Patch Annotation](#patch-annotation) operation.

**Required permissions**

See note in the [introduction](#annotations-api) for an explanation.

<!-- prettier-ignore-start -->
| Action              | Scope                                                                                                                                                        |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `annotations:write` | <ul><li>`annotations:*`</li><li>`annotations:type:*`</li><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example Request**:

```http
PUT /api/annotations/1141 HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
Content-Type: application/json

{
  "time":1507037197339,
  "timeEnd":1507180805056,
  "text":"Annotation Description",
  "tags":["tag3","tag4","tag5"]
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "message":"Annotation updated"
}
```

## Patch Annotation

`PATCH /api/annotations/:id`

Updates one or more properties of an annotation that matches the specified id.

This operation currently supports updating of the `text`, `tags`, `time` and `timeEnd` properties.

**Required permissions**

See note in the [introduction](#annotations-api) for an explanation.

<!-- prettier-ignore-start -->
| Action              | Scope                                                                                                                                                        |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `annotations:write` | <ul><li>`annotations:*`</li><li>`annotations:type:*`</li><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example Request**:

```http
PATCH /api/annotations/1145 HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
Content-Type: application/json

{
  "text":"New Annotation Description",
  "tags":["tag6","tag7","tag8"]
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "message":"Annotation patched"
}
```

## Delete Annotation By Id

`DELETE /api/annotations/:id`

Deletes the annotation that matches the specified id.

**Required permissions**

See note in the [introduction](#annotations-api) for an explanation.

<!-- prettier-ignore-start -->
| Action               | Scope                                                                                                                                                        |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `annotations:delete` | <ul><li>`annotations:*`</li><li>`annotations:type:*`</li><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example Request**:

```http
DELETE /api/annotations/1 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "message":"Annotation deleted"
}
```

## Find Annotations Tags

`GET /api/annotations/tags`

Find all the event tags created in the annotations.

**Required permissions**

See note in the [introduction](#annotations-api) for an explanation.

| Action             | Scope |
| ------------------ | ----- |
| `annotations:read` | N/A   |

**Example Request**:

```http
GET /api/annotations/tags?tag=out HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

Query Parameters:

- `tag`: Optional. A string that you can use to filter tags.
- `limit`: Optional. A number, where the default is 100. Max limit for results returned.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "result": {
        "tags": [
            {
                "tag": "outage",
                "count": 1
            }
        ]
    }
}
```

# Correlations API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

This API can be used to define correlations between data sources.

## Create correlations

`POST /api/datasources/uid/:sourceUID/correlations`

Creates a correlation between two data sources - the source data source identified by `sourceUID` in the path, and the target data source which is specified in the body.

**Example request:**

```http
POST /api/datasources/uid/uyBf2637k/correlations HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
{
	"targetUID": "PDDA8E780A17E7EF1",
	"label": "My Label",
	"description": "Logs to Traces",
  "type": "query",
  "config": {
    "field": "message",
    "target": {},
  }
}
```

JSON body schema:

- **targetUID** – Target data source uid.
- **label** – A label for the correlation.
- **description** – A description for the correlation.

**Example response:**

```http
HTTP/1.1 200
Content-Type: application/json
{
  "message": "Correlation created",
  "result": {
    "description": "Logs to Traces",
    "label": "My Label",
    "sourceUID": "uyBf2637k",
    "targetUID": "PDDA8E780A17E7EF1",
    "uid": "50xhMlg9k",
    "type": "query",
    "config": {
      "field": "message",
      "target": {},
    }
  }
}
```

Status codes:

- **200** – OK
- **400** - Errors (invalid JSON, missing or invalid fields)
- **401** – Unauthorized
- **403** – Forbidden, source data source is read-only
- **404** – Not found, either source or target data source could not be found
- **500** – Internal error

## Delete correlations

`DELETE /api/datasources/uid/:sourceUID/correlations/:correlationUID`

Deletes a correlation.

**Example request:**

```http
DELETE /api/datasources/uid/uyBf2637k/correlations/J6gn7d31L HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response:**

```http
HTTP/1.1 200
Content-Type: application/json
{
  "message": "Correlation deleted"
}
```

Status codes:

- **200** – OK
- **401** – Unauthorized
- **403** – Forbidden, data source is read-only
- **404** – Correlation not found
- **500** – Internal error

## Update correlations

`PATCH /api/datasources/uid/:sourceUID/correlations/:correlationUID`

Updates a correlation.

**Example request:**

```http
POST /api/datasources/uid/uyBf2637k/correlations/J6gn7d31L HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
{
	"label": "My Label",
	"description": "Logs to Traces",
}
```

JSON body schema:

- **label** – A label for the correlation.
- **description** – A description for the correlation.

**Example response:**

```http
HTTP/1.1 200
Content-Type: application/json
{
  "message": "Correlation updated",
  "result": {
    "description": "Logs to Traces",
    "label": "My Label",
    "sourceUID": "uyBf2637k",
    "targetUID": "PDDA8E780A17E7EF1",
    "uid": "J6gn7d31L",
    "type": "query",
    "config": {
			"field": "message",
			"target": {}
		}
  }
}
```

Status codes:

- **200** – OK
- **400** – Bad request
- **401** – Unauthorized
- **403** – Forbidden, source data source is read-only
- **404** – Not found, either source or target data source could not be found
- **500** – Internal error

## Get single correlation

`GET /api/datasources/uid/:sourceUID/correlations/:correlationUID`

Gets a single correlation.

**Example request:**

```http
GET /api/datasources/uid/uyBf2637k/correlations/J6gn7d31L HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response:**

```http
HTTP/1.1 200
Content-Type: application/json
{
  "description": "Logs to Traces",
  "label": "My Label",
  "sourceUID": "uyBf2637k",
  "targetUID": "PDDA8E780A17E7EF1",
  "uid": "J6gn7d31L",
  "provisioned": false,
  "type": "query",
  "config": {
    "field": "message",
    "target": {},
  }
}
```

Status codes:

- **200** – OK
- **401** – Unauthorized
- **404** – Not found, either source data source or correlation were not found
- **500** – Internal error

## Get all correlations originating from a given data source

`GET /api/datasources/uid/:sourceUID/correlations`

Get all correlations originating from the data source identified by the given `sourceUID` in the path.

**Example request:**

```http
GET /api/datasources/uid/uyBf2637k/correlations HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response:**

```http
HTTP/1.1 200
Content-Type: application/json
[
  {
    "description": "Logs to Traces",
    "label": "My Label",
    "sourceUID": "uyBf2637k",
    "targetUID": "PDDA8E780A17E7EF1",
    "uid": "J6gn7d31L",
    "provisioned": false,
    "type": "query",
    "config": {
      "field": "message",
      "target": {},
    }
  },
  {
    "description": "Logs to Metrics",
    "label": "Another Label",
    "sourceUID": "uyBf2637k",
    "targetUID": "P15396BDD62B2BE29",
    "uid": "uWCpURgVk",
    "provisioned": false,
    "type": "query",
    "config": {
      "field": "message",
      "target": {},
    }
  }
]
```

Status codes:

- **200** – OK
- **401** – Unauthorized
- **404** – Not found, either source data source is not found or no correlation exists originating from the given data source
- **500** – Internal error

## Get all correlations

`GET /api/datasources/correlations`

Get all correlations.

Query parameters:

- **page** - Optional. Specify which page number to return. Use the limit parameter to specify the number of correlations per page. The default is page 1.
- **limit** - Optional. Limits the number of returned correlations per page. The default is 100 correlations per page. The maximum limit is 1000 correlations in a page.
- **sourceUID** - Optional. Specify a source datasource UID to filter by. This can be repeated to filter by multiple datasources.

**Example request:**

```http
GET /api/datasources/correlations HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response:**

```http
HTTP/1.1 200
Content-Type: application/json
[
  {
    "description": "Prometheus to Loki",
    "label": "My Label",
    "sourceUID": "uyBf2637k",
    "targetUID": "PDDA8E780A17E7EF1",
    "uid": "J6gn7d31L",
    "provisioned": false,
    "type": "query",
    "config": {
      "field": "message",
      "target": {},
    }
  },
  {
    "description": "Loki to Tempo",
    "label": "Another Label",
    "sourceUID": "PDDA8E780A17E7EF1",
    "targetUID": "P15396BDD62B2BE29",
    "uid": "uWCpURgVk",
    "provisioned": false,
    "type": "query",
    "config": {
      "field": "message",
      "target": {},
    }
  }
]
```

Status codes:

- **200** – OK
- **401** – Unauthorized
- **404** – Not found, no correlation is found
- **500** – Internal error

# Dashboard Permissions API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

This API can be used to update/get the permissions for a dashboard.

Permissions with `dashboardId=-1` are the default permissions for users with the Viewer and Editor roles. Permissions can be set for a user, a team or a role (Viewer or Editor). Permissions cannot be set for Admins - they always have access to everything.

The permission levels for the permission field:

- 1 = View
- 2 = Edit
- 4 = Admin

> If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## Get permissions for a dashboard

`GET /api/dashboards/uid/:uid/permissions`

Gets all existing permissions for the dashboard with the given `uid`.

**Required permissions**

See note in the [introduction](#dashboard-permission-api) for an explanation.

<!-- prettier-ignore-start -->
| Action                        | Scope                                                                                                   |
| ----------------------------- | ------------------------------------------------------------------------------------------------------- |
| `dashboards.permissions:read` | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example request**:

```http
GET /api/dashboards/uid/dHEquNzGz/permissions HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 551

[
  {
    "id": 1,
    "dashboardId": -1,
    "created": "2017-06-20T02:00:00+02:00",
    "updated": "2017-06-20T02:00:00+02:00",
    "userId": 0,
    "userLogin": "",
    "userEmail": "",
    "teamId": 0,
    "team": "",
    "role": "Viewer",
    "permission": 1,
    "permissionName": "View",
    "uid": "dHEquNzGz",
    "title": "",
    "slug": "",
    "isFolder": false,
    "url": ""
  },
  {
    "id": 2,
    "dashboardId": -1,
    "created": "2017-06-20T02:00:00+02:00",
    "updated": "2017-06-20T02:00:00+02:00",
    "userId": 0,
    "userLogin": "",
    "userEmail": "",
    "teamId": 0,
    "team": "",
    "role": "Editor",
    "permission": 2,
    "permissionName": "Edit",
    "uid": "dHEquNzGz",
    "title": "",
    "slug": "",
    "isFolder": false,
    "url": ""
  }
]
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Access denied
- **404** - Dashboard not found

## Update permissions for a dashboard

`POST /api/dashboards/uid/:uid/permissions`

Updates permissions for a dashboard. This operation will remove existing permissions if they're not included in the request.

**Required permissions**

See note in the [introduction](#dashboard-permission-api) for an explanation.

<!-- prettier-ignore-start -->
| Action                         | Scope                                                                                                   |
| ------------------------------ | ------------------------------------------------------------------------------------------------------- |
| `dashboards.permissions:write` | <ul><li>`dashboards:*`</li><li>`dashboards:uid:*`</li><li>`folders:*`</li><li>`folders:uid:*`</li></ul> |
{ .no-spacing-list }
<!-- prettier-ignore-end -->

**Example request**:

```http
POST /api/dashboards/uid/dHEquNzGz/permissions
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "items": [
    {
      "role": "Viewer",
      "permission": 1
    },
    {
      "role": "Editor",
      "permission": 2
    },
    {
      "teamId": 1,
      "permission": 1
    },
    {
      "userId": 11,
      "permission": 4
    }
  ]
}
```

JSON body schema:

- **items** - The permission items to add/update. Items that are omitted from the list will be removed.

**Example response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 35

{"message":"Dashboard permissions updated"}
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Access denied
- **404** - Dashboard not found

# Shared Dashboards API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Requirements

If you're running Grafana Enterprise, you'll need to have specific permissions for some endpoints. Refer to [Role-based access control permissions](ref:role-based-access-control-permissions) for more information.

## Create a shared dashboard

`POST /api/dashboards/uid/:uid/public-dashboards/`

Creates a new shared dashboard.

**Required permissions**

See note in the [introduction](#shared-dashboards-api) for an explanation.

| Action                    | Scope                            |
| ------------------------- | -------------------------------- |
| `dashboards.public:write` | `dashboards:uid:<dashboard UID>` |

**Example Request for new shared dashboard**:

```http
POST /api/dashboards/uid/xCpsVuc4z/public-dashboards/ HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
    "uid": "cd56d9fd-f3d4-486d-afba-a21760e2acbe",
    "accessToken": "5c948bf96e6a4b13bd91975f9a2028b7",
    "timeSelectionEnabled": false,
    "isEnabled": true,
    "annotationsEnabled": false,
    "share": "public"
}
```

JSON Body schema:

- **uid** – Optional. Unique identifier when creating a shared dashboard. If it's null, it will generate a new uid.
- **accessToken** – Optional. Unique access token. If it's null, it will generate a new access token.
- **timeSelectionEnabled** – Optional. Set to `true` to enable the time picker in the shared dashboard. The default value is `false`.
- **isEnabled** – Optional. Set to `true` to enable the shared dashboard. The default value is `false`.
- **annotationsEnabled** – Optional. Set to `true` to show annotations. The default value is `false`.
- **share** – Optional. Set the share mode. The default value is `public`.

**Example Response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 78

{
    "uid": "cd56d9fd-f3d4-486d-afba-a21760e2acbe",
    "dashboardUid": "xCpsVuc4z",
    "accessToken": "5c948bf96e6a4b13bd91975f9a2028b7",
    "createdBy": 1,
    "updatedBy": 1,
    "createdAt": "2023-09-05T15:48:21-03:00",
    "updatedAt": "2023-09-05T15:48:21-03:00",
    "timeSelectionEnabled": false,
    "isEnabled": false,
    "annotationsEnabled": false,
    "share": "public"
}
```

Status Codes:

- **200** – Created
- **400** – Errors (such as invalid json, missing or invalid fields, or dashboard is shared)
- **401** – Unauthorized
- **403** – Access denied
- **404** – Dashboard not found

The error response body will have the following properties:

```http
HTTP/1.1 400 Bad request
Content-Type: application/json; charset=UTF-8
Content-Length: 107

{
    "statusCode": 400,
    "messageId": "publicdashboards.dashboardIsPublic",
    "message": "Dashboard is already public"
}
```

## Update a shared dashboard

`PATCH /api/dashboards/uid/:uid/public-dashboards/:publicDashboardUid`

Will update the shared dashboard given the specified unique identifier (uid).

**Required permissions**

See note in the [introduction](#shared-dashboard-api) for an explanation.

| Action                    | Scope                            |
| ------------------------- | -------------------------------- |
| `dashboards.public:write` | `dashboards:uid:<dashboard UID>` |

**Example Request for updating a shared dashboard**:

```http
PATCH /api/dashboards/uid/xCpsVuc4z/public-dashboards/cd56d9fd-f3d4-486d-afba-a21760e2acbe HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
    "timeSelectionEnabled": false,
    "isEnabled": true,
    "annotationsEnabled": false,
    "share": "public"
}
```

JSON Body schema:

- **timeSelectionEnabled** – Optional. Set to `true` to enable the time picker in the shared dashboard. The default value is `false`.
- **isEnabled** – Optional. Set to `true` to enable the shared dashboard. The default value is `false`.
- **annotationsEnabled** – Optional. Set to `true` to show annotations. The default value is `false`.
- **share** – Optional. Set the share mode. The default value is `public`.

**Example Response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 78

{
    "uid": "cd56d9fd-f3d4-486d-afba-a21760e2acbe",
    "dashboardUid": "xCpsVuc4z",
    "accessToken": "5c948bf96e6a4b13bd91975f9a2028b7",
    "createdBy": 1,
    "updatedBy": 1,
    "createdAt": "2023-09-05T15:48:21-03:00",
    "updatedAt": "2023-09-05T15:48:21-03:00",
    "timeSelectionEnabled": false,
    "isEnabled": false,
    "annotationsEnabled": false,
    "share": "public"
}
```

Status Codes:

- **200** – Updated
- **400** – Errors (such as invalid json, missing or invalid fields)
- **401** – Unauthorized
- **403** – Access denied
- **404** – Dashboard not found

The error response body will have the following properties:

```http
HTTP/1.1 400 Bad request
Content-Type: application/json; charset=UTF-8
Content-Length: 107

{
    "statusCode": 400,
    "messageId": "publicdashboards.dashboardIsPublic",
    "message": "Dashboard is already public"
}
```

## Get shared dashboard by dashboard uid

`GET /api/dashboards/uid/:uid/public-dashboards/`

Will return the shared dashboard given the dashboard unique identifier (uid).

**Required permissions**

See note in the [introduction](#shared-dashboard-api) for an explanation.

| Action            | Scope                            |
| ----------------- | -------------------------------- |
| `dashboards:read` | `dashboards:uid:<dashboard UID>` |

**Example Request**:

```http
GET /api/dashboards/uid/xCpsVuc4z/public-dashboards/ HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "uid": "e71950f3-e7dd-4d1e-aa8a-a857bc5e7d64",
    "dashboardUid": "xCpsVuc4z",
    "accessToken": "dab10f3a4fbb4342a602b03079c7ed64",
    "createdBy": 1,
    "updatedBy": 1,
    "createdAt": "2023-09-05T15:48:21-03:00",
    "updatedAt": "2023-09-05T15:48:21-03:00",
    "timeSelectionEnabled": false,
    "isEnabled": false,
    "annotationsEnabled": false,
    "share": "public"
}
```

Status Codes:

- **200** – Found
- **401** – Unauthorized
- **403** – Access denied
- **404** – Dashboard not found

## Delete shared dashboard by dashboard uid and shared dashboard uid

`DELETE /api/dashboards/uid/:uid/public-dashboards/:publicDashboardUid`

Will delete the shared dashboard given the specified unique identifier (uid).

**Required permissions**

See note in the [introduction](#shared-dashboard-api) for an explanation.

| Action                    | Scope                            |
| ------------------------- | -------------------------------- |
| `dashboards.public:write` | `dashboards:uid:<dashboard UID>` |

**Example Request**:

```http
DELETE /api/dashboards/uid/xCpsVuc4z/public-dashboards/cd56d9fd-f3d4-486d-afba-a21760e2acbe HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

Status Codes:

- **200** – Deleted
- **401** – Unauthorized
- **403** – Access denied

## Get a list of all shared dashboards with pagination

`GET /api/dashboards/public-dashboards`

**Required permissions**

See note in the [introduction](#shared-dashboard-api) for an explanation.

| Action            | Scope                            |
| ----------------- | -------------------------------- |
| `dashboards:read` | `dashboards:uid:<dashboard UID>` |

**Example Request**:

```http
GET /api/dashboards/public-dashboards?perpage=2&page=3 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "publicDashboards": [
        {
            "uid": "e9f29a3c-fcc3-4fc5-a690-ae39c97d24ba",
            "accessToken": "6c13ec1997ba48c5af8c9c5079049692",
            "title": "Datasource Shared Queries",
            "dashboardUid": "d2f21d0a-76c7-47ec-b5f3-9dda16e5a996",
            "isEnabled": true
        },
        {
            "uid": "a174f604-6fe7-47de-97b4-48b7e401b540",
            "accessToken": "d1fcff345c0f45e8a78c096c9696034a",
            "title": "Datasource with template variables",
            "dashboardUid": "51DiOw0Vz",
            "isEnabled": true
        }
    ],
    "totalCount": 30,
    "page": 3,
    "perPage": 2
}
```

# Dashboard Versions

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Get all dashboard versions by dashboard UID

Query parameters:

- **limit** - Maximum number of results to return. Defaults to 1000 if not set, or if an invalid value is passed in.
- **start** - Version to start from when returning queries

`GET /api/dashboards/uid/:uid/versions`

Gets all existing dashboard versions for the dashboard with the given `uid`.

**Example request for getting all dashboard versions**:

```http
GET /api/dashboards/uid/QA7wKklGz/versions?limit=2?start=0 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 428

{
  "continueToken": "",
  "versions": [
    {
      "id": 2,
      "dashboardId": 1,
      "uid": "QA7wKklGz",
      "parentVersion": 1,
      "restoredFrom": 0,
      "version": 2,
      "created": "2017-06-08T17:24:33-04:00",
      "createdBy": "admin",
      "message": "Updated panel title"
    },
    {
      "id": 1,
      "dashboardId": 1,
      "uid": "QA7wKklGz",
      "parentVersion": 0,
      "restoredFrom": 0,
      "version": 1,
      "created": "2017-06-08T17:23:33-04:00",
      "createdBy": "admin",
      "message": "Initial save"
    }
  ]
}
```

Status Codes:

- **200** - Ok
- **400** - Errors
- **401** - Unauthorized
- **404** - Dashboard version not found

## Get dashboard version by dashboard UID

`GET /api/dashboards/uid/:uid/versions/:version`

Get the dashboard version with the given version, for the dashboard with the given UID.

**Example request for getting a dashboard version**:

```http
GET /api/dashboards/uid/QA7wKklGz/versions/1 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 1300

{
  "id": 1,
  "dashboardId": 1,
  "uid": "QA7wKklGz",
  "parentVersion": 0,
  "restoredFrom": 0,
  "version": 1,
  "created": "2017-04-26T17:18:38-04:00",
  "message": "Initial save",
  "data": {
    "annotations": {
      "list": [

      ]
    },
    "editable": true,
    "gnetId": null,
    "graphTooltip": 0,
    "id": 1,
    "links": [

    ],
    "rows": [
      {
        "collapse": false,
        "height": "250px",
        "panels": [

        ],
        "repeat": null,
        "repeatIteration": null,
        "repeatRowId": null,
        "showTitle": false,
        "title": "Dashboard Row",
        "titleSize": "h6"
      }
    ],
    "schemaVersion": 14,
      "tags": [

    ],
    "templating": {
      "list": [

      ]
    },
    "time": {
      "from": "now-6h",
      "to": "now"
    },
    "timepicker": {},
    "timezone": "browser",
    "title": "test",
    "version": 1
  },
  "createdBy": "admin"
}
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **404** - Dashboard version not found

# Data source API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Requirements

If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## Endpoints

| Method | URI                                    | Summary                                                                                        |
| ------ | -------------------------------------- | ---------------------------------------------------------------------------------------------- |
| GET    | /api/datasources                       | [Get all data sources](#get-all-data-sources)                                                  |
| GET    | /api/datasources/uid/:uid              | [Get a single data source by uid](#get-a-single-data-source-by-uid)                            |
| GET    | /api/datasources/name/:name            | [Get a single data source by name](#get-a-single-data-source-by-name) (deprecated)             |
| GET    | /api/datasources/id/:name              | [Get data source Id by name](#get-data-source-id-by-name) (deprecated)                         |
| POST   | /api/datasources                       | [Create a data source](#create-a-data-source)                                                  |
| PUT    | /api/datasources/uid/:uid              | [Update an existing data source](#update-an-existing-data-source)                              |
| DELETE | /api/datasources/uid/:uid              | [Delete an existing data source by uid](#delete-an-existing-data-source-by-uid)                |
| DELETE | /api/datasources/name/:datasourceName  | [Delete an existing data source by name](#delete-an-existing-data-source-by-name) (deprecated) |
| GET    | /api/datasources/proxy/uid/:uid/\*     | [Data source proxy calls](#data-source-proxy-calls)                                            |
| GET    | /api/datasources/uid/:uid/health       | [Check data source health](#check-data-source-health)                                          |
| GET    | /api/datasources/uid/:uid/resources/\* | [Fetch data source resources](#fetch-data-source-resources)                                    |
| POST   | /api/ds/query                          | [Query a data source](#query-a-data-source)                                                    |

## Get all data sources

`GET /api/datasources`

{{< admonition type="warning" >}}
This API currently doesn't handle pagination. The default maximum number of data sources returned is 5000. You can change this value in the default.ini file.
{{< /admonition >}}

**Required permissions**

See note in the [introduction](#data-source-api) for an explanation.

| Action           | Scope          |
| ---------------- | -------------- |
| datasources:read | datasources:\* |

### Examples

**Example Request**:

```http
GET /api/datasources HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
   {
     "id": 1,
     "orgId": 1,
     "uid": "H8joYFVGz"
     "name": "datasource_elastic",
     "type": "elasticsearch",
     "typeLogoUrl": "public/app/plugins/datasource/elasticsearch/img/elasticsearch.svg",
     "access": "proxy",
     "url": "http://mydatasource.com",
     "password": "",
     "user": "",
     "database": "grafana-dash",
     "basicAuth": false,
     "isDefault": false,
     "jsonData": {
         "logLevelField": "",
         "logMessageField": "",
         "maxConcurrentShardRequests": 256,
         "timeField": "@timestamp"
     },
     "readOnly": false
   }
]
```

## Get a single data source by uid

`GET /api/datasources/uid/:uid`

**Required permissions**

See note in the [introduction](#data-source-api) for an explanation.

| Action           | Scope                                                                                  |
| ---------------- | -------------------------------------------------------------------------------------- |
| datasources:read | datasources:\*<br>datasources:uid:\*<br>datasources:uid:kLtEtcRGk (single data source) |

### Examples

**Example request:**

```http
GET /api/datasources/uid/kLtEtcRGk HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "id": 1,
  "uid": "kLtEtcRGk",
  "orgId": 1,
  "name": "test_datasource",
  "type": "graphite",
  "typeLogoUrl": "",
  "access": "proxy",
  "url": "http://mydatasource.com",
  "password": "",
  "user": "",
  "database": "",
  "basicAuth": false,
  "basicAuthUser": "",
  "basicAuthPassword": "",
  "withCredentials": false,
  "isDefault": false,
  "jsonData": {
    "graphiteType": "default",
    "graphiteVersion": "1.1"
  },
  "secureJsonFields": {},
  "version": 1,
  "readOnly": false
}
```

## Get a single data source by name

`GET /api/datasources/name/:name`

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release.
{{< /admonition >}}

**Required permissions**

See note in the [introduction](#data-source-api) for an explanation.

| Action           | Scope                                                                                          |
| ---------------- | ---------------------------------------------------------------------------------------------- |
| datasources:read | datasources:\*<br>datasources:name:\*<br>datasources:name:test_datasource (single data source) |

### Examples

**Example Request**:

```http
GET /api/datasources/name/test_datasource HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "id": 1,
  "uid": "kLtEtcRGk",
  "orgId": 1,
  "name": "test_datasource",
  "type": "graphite",
  "typeLogoUrl": "",
  "access": "proxy",
  "url": "http://mydatasource.com",
  "password": "",
  "user": "",
  "database": "",
  "basicAuth": false,
  "basicAuthUser": "",
  "basicAuthPassword": "",
  "withCredentials": false,
  "isDefault": false,
  "jsonData": {
    "graphiteType": "default",
    "graphiteVersion": "1.1"
  },
  "secureJsonFields": {},
  "version": 1,
  "readOnly": false
}
```

## Get data source Id by name

`GET /api/datasources/id/:name`

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release.
{{< /admonition >}}

**Required permissions**

See note in the [introduction](#data-source-api) for an explanation.

| Action              | Scope                                                                                          |
| ------------------- | ---------------------------------------------------------------------------------------------- |
| datasources.id:read | datasources:\*<br>datasources:name:\*<br>datasources:name:test_datasource (single data source) |

### Examples

**Example Request**:

```http
GET /api/datasources/id/test_datasource HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "id":1
}
```

## Create a data source

`POST /api/datasources`

**Required permissions**

See note in the [introduction](#data-source-api) for an explanation.

| Action             | Scope |
| ------------------ | ----- |
| datasources:create | n/a   |

### Examples

**Example Graphite Request**:

```http
POST /api/datasources HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "name":"test_datasource",
  "type":"graphite",
  "url":"http://mydatasource.com",
  "access":"proxy",
  "basicAuth":false
}
```

**Example Graphite Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "datasource": {
    "id": 1,
    "orgId": 1,
    "name": "test_datasource",
    "type": "graphite",
    "typeLogoUrl": "",
    "access": "proxy",
    "url": "http://mydatasource.com",
    "password": "",
    "user": "",
    "database": "",
    "basicAuth": false,
    "basicAuthUser": "",
    "basicAuthPassword": "",
    "withCredentials": false,
    "isDefault": false,
    "jsonData": {},
    "secureJsonFields": {},
    "version": 1,
    "readOnly": false
  },
  "id": 1,
  "message": "Datasource added",
  "name": "test_datasource"
}
```

{{< admonition type="note" >}}
By defining `password` and `basicAuthPassword` under `secureJsonData` Grafana encrypts them securely as an encrypted blob in the database. The response then lists the encrypted fields under `secureJsonFields`.
{{< /admonition >}}

**Example Graphite Request with basic auth enabled**:

```http
POST /api/datasources HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "name": "test_datasource",
  "type": "graphite",
  "url": "http://mydatasource.com",
  "access": "proxy",
  "basicAuth": true,
  "basicAuthUser": "basicuser",
  "secureJsonData": {
    "basicAuthPassword": "basicpassword"
  }
}
```

**Example Response with basic auth enabled**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "datasource": {
    "id": 1,
    "orgId": 1,
    "name": "test_datasource",
    "type": "graphite",
    "typeLogoUrl": "",
    "access": "proxy",
    "url": "http://mydatasource.com",
    "password": "",
    "user": "",
    "database": "",
    "basicAuth": true,
    "basicAuthUser": "basicuser",
    "basicAuthPassword": "",
    "withCredentials": false,
    "isDefault": false,
    "jsonData": {},
    "secureJsonFields": {
      "basicAuthPassword": true
    },
    "version": 1,
    "readOnly": false
  },
  "id": 102,
  "message": "Datasource added",
  "name": "test_datasource"
}
```

**Example CloudWatch Request**:

```http
POST /api/datasources HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "name": "test_datasource",
  "type": "cloudwatch",
  "url": "http://monitoring.us-west-1.amazonaws.com",
  "access": "proxy",
  "jsonData": {
    "authType": "keys",
    "defaultRegion": "us-west-1"
  },
  "secureJsonData": {
    "accessKey": "Ol4pIDpeKSA6XikgOl4p",
    "secretKey": "dGVzdCBrZXkgYmxlYXNlIGRvbid0IHN0ZWFs"
  }
}
```

## Update an existing data source

`PUT /api/datasources/uid/:uid`

**Required permissions**

See note in the [introduction](#data-source-api) for an explanation.

| Action            | Scope                                                                                  |
| ----------------- | -------------------------------------------------------------------------------------- |
| datasources:write | datasources:\*<br>datasources:uid:\*<br>datasources:uid:kLtEtcRGk (single data source) |

### Examples

**Example Request**:

```http
PUT /api/datasources/uid/kLtEtcRGk HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "id":1,
  "uid": "uid",
  "orgId":1,
  "name":"test_datasource",
  "type":"graphite",
  "access":"proxy",
  "url":"http://mydatasource.com",
  "password":"",
  "user":"",
  "database":"",
  "basicAuth":true,
  "basicAuthUser":"basicuser",
  "secureJsonData": {
    "basicAuthPassword": "basicpassword"
  },
  "isDefault":false,
  "jsonData":null
}
```

Note that the UID cannot be modified.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "datasource": {
    "id": 1,
    "uid": "uid",
    "orgId": 1,
    "name": "test_datasource",
    "type": "graphite",
    "typeLogoUrl": "",
    "access": "proxy",
    "url": "http://mydatasource.com",
    "password": "",
    "user": "",
    "database": "",
    "basicAuth": true,
    "basicAuthUser": "basicuser",
    "basicAuthPassword": "",
    "withCredentials": false,
    "isDefault": false,
    "jsonData": {},
    "secureJsonFields": {
      "basicAuthPassword": true
    },
    "version": 1,
    "readOnly": false
  },
  "id": 102,
  "message": "Datasource updated",
  "name": "test_datasource"
}
```

{{< admonition type="note" >}}
Similar to [creating a data source](#create-a-data-source), `password` and `basicAuthPassword` should be defined under `secureJsonData` in order to be stored securely as an encrypted blob in the database. Then, the encrypted fields are listed under `secureJsonFields` section in the response.
{{< /admonition >}}

## Delete an existing data source by uid

`DELETE /api/datasources/uid/:uid`

**Required permissions**

See note in the [introduction](#data-source-api) for an explanation.

| Action             | Scope                                                                                  |
| ------------------ | -------------------------------------------------------------------------------------- |
| datasources:delete | datasources:\*<br>datasources:uid:\*<br>datasources:uid:kLtEtcRGk (single data source) |

### Examples

**Example request:**

```http
DELETE /api/datasources/uid/kLtEtcRGk HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "message": "Data source deleted",
    "id": 1
}
```

## Delete an existing data source by name

`DELETE /api/datasources/name/:datasourceName`

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release.
{{< /admonition >}}

**Required permissions**

See note in the [introduction](#data-source-api) for an explanation.

| Action             | Scope                                                                                          |
| ------------------ | ---------------------------------------------------------------------------------------------- |
| datasources:delete | datasources:\*<br>datasources:name:\*<br>datasources:name:test_datasource (single data source) |

### Examples

**Example Request**:

```http
DELETE /api/datasources/name/test_datasource HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "message":"Data source deleted",
  "id": 1
}
```

## Data source proxy calls

`GET /api/datasources/proxy/uid/:uid/*`

Proxies all calls to the actual data source identified by the `uid`.

## Check data source health

`GET /api/datasources/uid/:uid/health`

Makes a call to the health endpoint of data source identified by the given `uid`. This is not mandatory - every plugin author has to <a href="https://grafana.com/tutorials/build-a-data-source-backend-plugin/#add-support-for-health-checks" target="_blank">implement support for health checks</a> in their plugin themselves.

### Examples

**Example Request**:

```http
GET api/datasources/uid/P8045C56BDA891CB2/health HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "message": "1. Successfully queried the CloudWatch metrics API.\n2. Successfully queried the CloudWatch logs API.",
  "status": "OK"
}
```

## Fetch data source resources

`GET /api/datasources/uid/:uid/resources/*`

Makes a call to the resources endpoint of data source identified by the given `uid`.

### Examples

**Example Request**:

```http
GET api/datasources/uid/P8045C56BDA891CB2/resources/dimension-keys?region=us-east-2&namespace=AWS%2FEC2&dimensionFilters=%7B%7D&metricName=CPUUtilization HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
	{
		"text": "AutoScalingGroupName",
		"value": "AutoScalingGroupName",
		"label": "AutoScalingGroupName"
	},
	{
		"text": "ImageId",
		"value": "ImageId",
		"label": "ImageId"
	},
	{
		"text": "InstanceId",
		"value": "InstanceId",
		"label": "InstanceId"
	},
	{
		"text": "InstanceType",
		"value": "InstanceType",
		"label": "InstanceType"
	}
]
```

## Query a data source

Queries a data source having a backend implementation.

`POST /api/ds/query`

{{< admonition type="note" >}}
Grafana's built-in data sources usually have a backend implementation.
{{< /admonition >}}

**Example request for the Test data source**:

```http
POST /api/ds/query HTTP/1.1
Accept: application/json
Content-Type: application/json

{
   "queries":[
      {
         "refId":"A",
         "scenarioId":"csv_metric_values",
         "datasource":{
            "uid":"PD8C576611E62080A"
         },
         "format": "table",
         "maxDataPoints":1848,
         "intervalMs":200,
         "stringInput":"1,20,90,30,5,0"
      }
   ],
   "from":"now-5m",
   "to":"now"
}
```

JSON Body schema:

- **from/to** – Specifies the time range for the queries. The time can be either epoch timestamps in milliseconds or relative using Grafana time units. For example, `now-5m`.
- **queries** – Specifies one or more queries. Must contain at least 1.
- **queries.datasource.uid** – Specifies the UID of data source to be queried. Each query in the request must have a unique `datasource`.
- **queries.refId** – Specifies an identifier of the query. Defaults to "A".
- **queries.format** – Specifies the format the data should be returned in. Valid options are `time_series` or `table` depending on the data source.
- **queries.maxDataPoints** - Species the maximum amount of data points that a dashboard panel can render. Defaults to 100.
- **queries.intervalMs** - Specifies the time series time interval in milliseconds. Defaults to 1000.

In addition, specific properties of each data source should be added in a request (for example **queries.stringInput** as shown in the request above). To better understand how to form a query for a certain data source, use the Developer Tools in your browser of choice and inspect the HTTP requests being made to `/api/ds/query`.

**Example Test data source time series query response:**

```json
{
  "results": {
    "A": {
      "frames": [
        {
          "schema": {
            "refId": "A",
            "fields": [
              {
                "name": "time",
                "type": "time",
                "typeInfo": {
                  "frame": "time.Time"
                }
              },
              {
                "name": "A-series",
                "type": "number",
                "typeInfo": {
                  "frame": "int64",
                  "nullable": true
                }
              }
            ]
          },
          "data": {
            "values": [
              [1644488152084, 1644488212084, 1644488272084, 1644488332084, 1644488392084, 1644488452084],
              [1, 20, 90, 30, 5, 0]
            ]
          }
        }
      ]
    }
  }
}
```

#### Status codes

| Code | Description                                                                                                                                                                      |
| ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 200  | All data source queries returned a successful response.                                                                                                                          |
| 400  | Bad request due to invalid JSON, missing content type, missing or invalid fields, etc. Or one or more data source queries were unsuccessful. Refer to the body for more details. |
| 403  | Access denied.                                                                                                                                                                   |
| 404  | Either the data source or plugin required to fulfil the request could not be found.                                                                                              |
| 500  | Unexpected error. Refer to the body and/or server logs for more details.                                                                                                         |

# Data source LBAC rules API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

LBAC (Label-Based Access Control) rules can be set for teams.

## Get LBAC rules for a data source

`GET /api/datasources/uid/:uid/lbac/teams`

Gets all existing LBAC rules for the data source with the given `uid`.

**Required permissions**

| Action           | Scope                                                                                    |
| ---------------- | ---------------------------------------------------------------------------------------- |
| datasources:read | datasources:_<br>datasources:uid:_<br>datasources:uid:my_datasource (single data source) |

### Examples

**Example request:**

```
GET /api/datasources/uid/my_datasource/lbac/teams HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 131

{
  "rules": [
    {
      "teamUId": "fdnd1pf4m9sxvc",
      "rules": [
        "{ service_name=\"bigquery-sync-mysql\" }"
      ]
    },
    {
      "teamUid": "dfed1p2m9sxvfc",
      "rules": [
        "{ service_name=\"api\" }"
      ]
    }
  ]
}
```

## Update LBAC rules for a data source

`PUT /api/datasources/uid/:uid/lbac/teams`

Updates LBAC rules for teams associated with the data source with the given `uid`. Here you submit a list of teams and the rules for each team.
Deleting a team from the list will remove the team's LBAC rules. You have to submit all teams and their rules to be updated, to remove a team's rules, you have to submit the current list of rules without the team.

**Required permissions**

| Action                        | Scope                                                                                    |
| ----------------------------- | ---------------------------------------------------------------------------------------- |
| datasources:write             | datasources:_<br>datasources:uid:_<br>datasources:uid:my_datasource (single data source) |
| datasources.permissions:write | datasources:_<br>datasources:uid:_<br>datasources:uid:my_datasource (single data source) |

### Examples

**Example request:**

```http
PUT /api/datasources/uid/my_datasource/lbac/teams
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "rules": [
    {
      "teamUId": "fdnd1pf4m9sxvc",
      "rules": [
        "{ service_name=\"bigquery-sync-mysql\" }"
      ]
    },
    {
      "teamUid": "dfed1p2m9sxvfc",
      "rules": [
        "{ service_name=\"api\" }"
      ]
    }
  ]
}
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 35

{
  "id": 1,
  "message": "Data source LBAC rules updated",
  "name": "loki",
  "rules": [
    {
      "teamUId": "fdnd1pf4m9sxvc",
      "rules": [
        "{ service_name=\"bigquery-sync-mysql\" }"
      ]
    },
    {
      "teamUid": "dfed1p2m9sxvfc",
      "rules": [
        "{ service_name=\"api\" }"
      ]
    }
  ],
  "uid": "ee1nm1t7spog0e",
}
```

# Data source Permissions API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

> The Data Source Permissions is only available in Grafana Enterprise. Read more about [Grafana Enterprise](/docs/grafana/latest/introduction/grafana-enterprise/).

> If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

This API can be used to list, add and remove permissions for a data source.

Permissions can be set for a user, team, service account or a basic role (Admin, Editor, Viewer).

### Optional `ds_type` query parameter {#ds-type}

Every endpoint in this API accepts an optional query parameter `ds_type`. Set it to the data source **plugin type** (for example `prometheus` or `loki`). Use `ds_type` when more than one data source in the organization shares the same UID so Grafana can resolve the correct instance.

## Get permissions for a data source

`GET /api/access-control/datasources/:uid`

Gets all existing permissions for the data source with the given `uid`.

Append `?ds_type=<TYPE>` when you need to disambiguate the UID; refer to [Optional `ds_type` query parameter](#ds-type).

**Required permissions**

See note in the [introduction](#data-source-permissions-api) for an explanation.

| Action                       | Scope                                                                                      |
| ---------------------------- | ------------------------------------------------------------------------------------------ |
| datasources.permissions:read | datasources:\*<br>datasources:uid:\*<br>datasources:uid:my_datasource (single data source) |

### Examples

**Example request:**

```http
GET /api/access-control/datasources/my_datasource HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example request (with `ds_type` when the UID is not unique):**

```http
GET /api/access-control/datasources/my_datasource?ds_type=prometheus HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 551

[
    {
        "id": 1,
        "roleName": "fixed:datasources:reader",
        "isManaged": false,
        "isInherited": false,
        "isServiceAccount": false,
        "userId": 1,
        "userLogin": "admin_user",
        "userAvatarUrl": "/avatar/admin_user",
        "actions": [
            "datasources:read",
            "datasources:query",
            "datasources:read",
            "datasources:query",
            "datasources:write",
            "datasources:delete"
        ],
        "permission": "Edit"
    },
    {
        "id": 2,
        "roleName": "managed:teams:1:permissions",
        "isManaged": true,
        "isInherited": false,
        "isServiceAccount": false,
        "team": "A team",
        "teamId": 1,
        "teamAvatarUrl": "/avatar/523d70c8551046f441727d690431858c",
        "actions": [
            "datasources:read",
            "datasources:query"
        ],
        "permission": "Query"
    },
    {
        "id": 3,
        "roleName": "basic:admin",
        "isManaged": false,
        "isInherited": false,
        "isServiceAccount": false,
        "builtInRole": "Admin",
        "actions": [
            "datasources:query",
            "datasources:read",
            "datasources:write",
            "datasources:delete"
        ],
        "permission": "Edit"
    },
]
```

Status codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Access denied
- **500** - Internal error

## Add or revoke access to a data source for a user

`POST /api/access-control/datasources/:uid/users/:id`

Sets user permission for the data source with the given `uid`.

Append `?ds_type=<TYPE>` when you need to disambiguate the UID; refer to [Optional `ds_type` query parameter](#ds-type).

To add a permission, set the `permission` field to either `Query`, `Edit`, or `Admin`.
To remove a permission, set the `permission` field to an empty string.

**Required permissions**

See note in the [introduction](#data-source-permissions-api) for an explanation.

| Action                        | Scope                                                                                      |
| ----------------------------- | ------------------------------------------------------------------------------------------ |
| datasources.permissions:write | datasources:\*<br>datasources:uid:\*<br>datasources:uid:my_datasource (single data source) |

### Examples

**Example request:**

```http
POST /api/access-control/datasources/my_datasource/users/1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "permission": "Query",
}
```

When granting a `Query` permission, the user also has read access to the data source.

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 35

{"message": "Permission updated"}
```

**Example request:**

```http
POST /api/access-control/datasources/my_datasource/users/1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "permission": "",
}
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 35

{"message": "Permission removed"}
```

Status codes:

- **200** - Ok
- **400** - Permission cannot be added, see response body for details
- **401** - Unauthorized
- **403** - Access denied

## Add or revoke access to a data source for a team

`POST /api/access-control/datasources/:uid/teams/:id`

Sets team permission for the data source with the given `uid`.

Append `?ds_type=<TYPE>` when you need to disambiguate the UID; refer to [Optional `ds_type` query parameter](#ds-type).

To add a permission, set the `permission` field to either `Query`, `Edit`, or `Admin`.
To remove a permission, set the `permission` field to an empty string.

When adding `Query`, `Edit`, or `Admin` permissions, the user implicitly has `Read` access.

**Required permissions**

See note in the [introduction](#data-source-permissions-api) for an explanation.

| Action                        | Scope                                                                                      |
| ----------------------------- | ------------------------------------------------------------------------------------------ |
| datasources.permissions:write | datasources:\*<br>datasources:uid:\*<br>datasources:uid:my_datasource (single data source) |

### Examples

**Example request:**

```http
POST /api/access-control/datasources/my_datasource/teams/1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "permission": "Edit",
}
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 35

{"message": "Permission updated"}
```

**Example request:**

```http
POST /api/access-control/datasources/my_datasource/teams/1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "permission": "",
}
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 35

{"message": "Permission removed"}
```

Status codes:

- **200** - Ok
- **400** - Permission cannot be added, see response body for details
- **401** - Unauthorized
- **403** - Access denied

## Add or revoke access to a data source for a basic role

`POST /api/access-control/datasources/:uid/builtInRoles/:builtinRoleName`

Sets permission for the data source with the given `uid` to all users who have the specified basic role.

Append `?ds_type=<TYPE>` when you need to disambiguate the UID; refer to [Optional `ds_type` query parameter](#ds-type).

You can set permissions for the following basic roles: `Admin`, `Editor`, `Viewer`.

To add a permission, set the `permission` field to either `Query`, `Edit`, or `Admin`.
To remove a permission, set the `permission` field to an empty string.

**Required permissions**

See note in the [introduction](#data-source-permissions-api) for an explanation.

| Action                        | Scope                                                                                      |
| ----------------------------- | ------------------------------------------------------------------------------------------ |
| datasources.permissions:write | datasources:\*<br>datasources:uid:\*<br>datasources:uid:my_datasource (single data source) |

### Examples

**Example request:**

```http
POST /api/access-control/datasources/my_datasource/builtInRoles/Admin
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "permission": "Edit",
}
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 35

{"message": "Permission updated"}
```

**Example request:**

```http
POST /api/access-control/datasources/my_datasource/builtInRoles/Viewer
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "permission": "",
}
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 35

{"message": "Permission removed"}
```

Status codes:

- **200** - Ok
- **400** - Permission cannot be added, see response body for details
- **401** - Unauthorized
- **403** - Access denied

# Folder/Dashboard Search API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Search folders and dashboards

`GET /api/search/`

> Note: When using [Role-based access control](/docs/grafana/latest/administration/roles-and-permissions/access-control/), search results will contain only dashboards and folders which you have access to.

Query parameters:

- **query** – Search Query
- **tag** – List of tags to search for
- **type** – Type to search for, `dash-folder` or `dash-db`
- **dashboardIds** – List of dashboard id's to search for
- **dashboardUID** - List of dashboard uid's to search for, It is deprecated since Grafana v9.1, please use dashboardUIDs instead
- **dashboardUIDs** – List of dashboard uid's to search for
- **folderUIDs** – List of folder UIDs to search in
- **starred** – Flag indicating if only starred Dashboards should be returned
- **limit** – Limit the number of returned results (max is 5000; default is 1000). If an invalid value is provided (for example, strings or special characters), the parameter defaults to 1000.
- **page** – Use this parameter to access hits beyond limit. Numbering starts at 1. limit param acts as page size.

**Example request for retrieving folders and dashboards at the root level**:

```http
GET /api/search?query=&starred=false HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response for retrieving folders and dashboards at the root level**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "id": 163,
    "uid": "000000163",
    "orgId": 1,
    "title": "Folder",
    "url": "/dashboards/f/000000163/folder",
    "type": "dash-folder",
    "tags": [],
    "isStarred": false,
    "uri":"db/folder" // deprecated in Grafana v5.0
  },
  {
    "id":1,
    "uid": "cIBgcSjkk",
    "orgId": 1,
    "title":"Production Overview",
    "url": "/d/cIBgcSjkk/production-overview",
    "type":"dash-db",
    "tags":[prod],
    "isStarred":true,
    "uri":"db/production-overview" // deprecated in Grafana v5.0
  }
]
```

**Example request searching for dashboards**:

```http
GET /api/search?query=Production%20Overview&starred=true&tag=prod HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response searching for dashboards**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "id":1,
    "uid": "cIBgcSjkk",
    "orgId": 1,
    "title":"Production Overview",
    "url": "/d/cIBgcSjkk/production-overview",
    "type":"dash-db",
    "tags":[prod],
    "isStarred":true,
    "folderId": 2,
    "folderUid": "000000163",
    "folderTitle": "Folder",
    "folderUrl": "/dashboards/f/000000163/folder",
    "uri":"db/production-overview" // deprecated in Grafana v5.0
  }
]
```

# Folder Permissions API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

This API can be used to update/get the permissions for a folder.

Permissions with `folderId=-1` are the default permissions for users with the Viewer and Editor roles. Permissions can be set for a user, a team or a role (Viewer or Editor). Permissions cannot be set for Admins - they always have access to everything.

The permission levels for the permission field:

- 1 = View
- 2 = Edit
- 4 = Admin

> If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## Get permissions for a folder

`GET /api/folders/:uid/permissions`

Gets all existing permissions for the folder with the given `uid`.

**Required permissions**

See note in the [introduction](#folder-permission-api) for an explanation.

| Action                     | Scope       |
| -------------------------- | ----------- |
| `folders.permissions:read` | `folders:*` |

**Example request**:

```http
GET /api/folders/nErXDvCkzz/permissions HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 551

[
  {
    "id": 1,
    "folderId": -1,
    "created": "2017-06-20T02:00:00+02:00",
    "updated": "2017-06-20T02:00:00+02:00",
    "userId": 0,
    "userLogin": "",
    "userEmail": "",
    "teamId": 0,
    "team": "",
    "role": "Viewer",
    "permission": 1,
    "permissionName": "View",
    "uid": "nErXDvCkzz",
    "title": "",
    "slug": "",
    "isFolder": false,
    "url": ""
  },
  {
    "id": 2,
    "dashboardId": -1,
    "created": "2017-06-20T02:00:00+02:00",
    "updated": "2017-06-20T02:00:00+02:00",
    "userId": 0,
    "userLogin": "",
    "userEmail": "",
    "teamId": 0,
    "team": "",
    "role": "Editor",
    "permission": 2,
    "permissionName": "Edit",
    "uid": "",
    "title": "",
    "slug": "",
    "isFolder": false,
    "url": ""
  }
]
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Access denied
- **404** - Folder not found

## Update permissions for a folder

`POST /api/folders/:uid/permissions`

Updates permissions for a folder. This operation will remove existing permissions if they're not included in the request.

**Required permissions**

See note in the [introduction](#folder-permission-api) for an explanation.

| Action                      | Scope       |
| --------------------------- | ----------- |
| `folders.permissions:write` | `folders:*` |

**Example request**:

```http
POST /api/folders/nErXDvCkzz/permissions
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
{
  "items": [
    {
      "role": "Viewer",
      "permission": 1
    },
    {
      "role": "Editor",
      "permission": 2
    },
    {
      "teamId": 1,
      "permission": 1
    },
    {
      "userId": 11,
      "permission": 4
    }
  ]
}
```

JSON body schema:

- **items** - The permission items to add/update. Items that are omitted from the list will be removed.

**Example response**:

```http
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 35

{"message":"Folder permissions updated","id":1,"title":"Department ABC"}
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Access denied
- **404** - Dashboard not found

# Library Element API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Identifier (id) vs unique identifier (uid)

The identifier (ID) of a library element is an auto-incrementing numeric value that is unique per Grafana install.

The unique identifier (UID) of a library element uniquely identifies library elements between multiple Grafana installs. It's automatically generated unless you specify it during library element creation. The UID provides consistent URLs for accessing library elements and when syncing library elements between multiple Grafana installs.

The maximum length of a UID is 40 characters.

## Get all library elements

`GET /api/library-elements`

Returns a list of all library elements the authenticated user has permission to view. Use the `perPage` query parameter to control the maximum number of library elements returned; the default limit is 100. You can also use the `page` query parameter to fetch library elements from any page other than the first one.

Query parameters:

- `searchString`: Part of the name or description searched for.
- `kind`: Kind of element to search for. Use `1` for library panels.
- `sortDirection`: Sort order of elements. Use `alpha-asc` for ascending and `alpha-desc` for descending sort order.
- `typeFilter`: A comma separated list of types to filter the elements by.
- `excludeUid`: Element UID to exclude from search results.
- `folderFilter`: **Deprecated.** A comma separated list of folder IDs to filter the elements by. Use `folderFilterUIDs` instead.
- `folderFilterUIDs`: A comma separated list of folder UIDs to filter the elements by.
- `perPage`: The number of results per page; default is 100.
- `page`: The page for a set of records, given that only `perPage` records are returned at a time. Numbering starts at `1`.

**Example Request**:

```http
GET /api/library-elements?perPage=10 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "result": {
     "totalCount": 15,
     "page": 1,
     "perPage": 10
     "elements": [
        {
            "id": 25,
            "orgId": 1,
            "folderId": 0,
            "uid": "V--OrYHnz",
            "name": "API docs Example",
            "kind": 1,
            "type": "text",
            "description": "",
            "model": {...},
            "version": 1,
            "meta": {
                "folderName": "General",
                "folderUid": "",
                "connectedDashboards": 1,
                "created": "2021-09-27T09:56:17+02:00",
                "updated": "2021-09-27T09:56:17+02:00",
                "createdBy": {
                    "id": 1,
                    "name": "admin",
                    "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
                },
                "updatedBy": {
                    "id": 1,
                    "name": "admin",
                    "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
                }
            }
        },
        {...}
        {...}
     ],
  }
}
```

Status Codes:

- `200`: Found
- `401`: Unauthorized

## Get library element by uid

`GET /api/library-elements/:uid`

Returns a library element with the given UID.

**Example Request**:

```http
GET /api/library-elements/V--OrYHnz HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "result": {
      "id": 25,
      "orgId": 1,
      "folderId": 0,
      "uid": "V--OrYHnz",
      "name": "API docs Example",
      "kind": 1,
      "type": "text",
      "description": "",
      "model": {...},
      "version": 1,
      "meta": {
          "folderName": "General",
          "folderUid": "",
          "connectedDashboards": 1,
          "created": "2021-09-27T09:56:17+02:00",
          "updated": "2021-09-27T09:56:17+02:00",
          "createdBy": {
              "id": 1,
              "name": "admin",
              "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
          },
          "updatedBy": {
              "id": 1,
              "name": "admin",
              "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
          }
      }
   }
}
```

Status Codes:

- `200`: Found
- `401`: Unauthorized
- `404`: Library element not found

## Get library element by name

`GET /api/library-elements/name/:name`

Returns a library element with the given name

**Example Request**:

```http
GET /api/library-elements/name/API docs Example HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "result": [
        {
            "id": 25,
            "orgId": 1,
            "folderId": 0,
            "uid": "V--OrYHnz",
            "name": "API docs Example",
            "kind": 1,
            "type": "text",
            "description": "",
            "model": {...},
            "version": 1,
            "meta": {
                "folderName": "General",
                "folderUid": "",
                "connectedDashboards": 1,
                "created": "2021-09-27T09:56:17+02:00",
                "updated": "2021-09-27T09:56:17+02:00",
                "createdBy": {
                    "id": 1,
                    "name": "admin",
                    "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
                },
                "updatedBy": {
                    "id": 1,
                    "name": "admin",
                    "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
                }
            }
        }
    ]
}
```

Status Codes:

- `200`: Found
- `401`: Unauthorized
- `404`: Library element not found

## Get library element connections

`GET /api/library-elements/:uid/connections`

Returns a list of connections for a library element based on the UID specified.

**Example Request**:

```http
GET /api/library-elements/V--OrYHnz/connections HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "result": [
        {
            "id": 148, // Deprecated: will be removed in the future.
            "kind": 1,
            "elementId": 25,
            "connectionId": 527,
            "connectionUid": "dHEquNzGz",
            "created": "2021-09-27T10:00:07+02:00",
            "createdBy": {
                "id": 1,
                "name": "admin",
                "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
            }
        }
    ]
}
```

Status Codes:

- `200`: Found
- `401`: Unauthorized
- `404`: Library element not found

## Create library element

`POST /api/library-elements`

Creates a new library element.

JSON Body schema:

- `folderId`: ID of the folder where the library element is stored. It is deprecated since Grafana v9
- `folderUid`: Optional, the UID of the folder where the library element is stored, empty string when it is at the root level.
- `name`: Optional, the name of the library element.
- `model`: The JSON model for the library element.
- `kind`: Kind of element to create, Use `1` for library panels.
- `uid`: Optional, the [unique identifier](#identifier-id-vs-unique-identifier-uid).

**Example Request**:

```http
POST /api/library-elements HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "uid": "nErXDvCkzz",
  "folderUid": "",
  "name": "Example library panel",
  "model": {...},
  "kind": 1
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "result": {
        "id": 28,
        "orgId": 1,
        "folderId": 0,
        "folderUid": "",
        "uid": "nErXDvCkzz",
        "name": "Example library panel",
        "kind": 1,
        "type": "",
        "description": "",
        "model": {...},
        "version": 1,
        "meta": {
            "folderName": "General",
            "folderUid": "",
            "connectedDashboards": 0,
            "created": "2021-09-30T09:14:22.378307+02:00",
            "updated": "2021-09-30T09:14:22.378307+02:00",
            "createdBy": {
                "id": 1,
                "name": "admin",
                "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
            },
            "updatedBy": {
                "id": 1,
                "name": "admin",
                "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
            }
        }
    }
}
```

Status Codes:

- `200`: Created
- `400`: Errors (for example, name or UID already exists, invalid JSON, missing or invalid fields, and so on).
- `401`: Unauthorized
- `403`: Access denied

## Update library element

`PATCH /api/library-elements/:uid`

Updates an existing library element identified by uid.

JSON Body schema:

- `folderId`: ID of the folder where the library element is stored. It is deprecated since Grafana v9
- `folderUid`: UID of the folder where the library element is stored, empty string when it is at the root level.
- `name`: Name of the library element.
- `model`: The JSON model for the library element.
- `kind`: Kind of element to create. Use `1` for library panels.
- `version`: Version of the library element you are updating.
- `uid`: Optional, the [unique identifier](#identifier-id-vs-unique-identifier-uid).

**Example Request**:

```http
PATCH /api/library-elements/nErXDvCkzz HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "name": "Renamed library panel",
  "kind": 1,
  "version": 1
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "result": {
        "id": 28,
        "orgId": 1,
        "folderId": 0,
        "folderUid": "",
        "uid": "nErXDvCkzz",
        "name": "Renamed library panel",
        "kind": 1,
        "type": "",
        "description": "",
        "model": {
            "description": "",
            "type": ""
        },
        "version": 2,
        "meta": {
            "folderName": "General",
            "folderUid": "",
            "connectedDashboards": 0,
            "created": "2021-09-30T09:14:22+02:00",
            "updated": "2021-09-30T09:25:57.697214+02:00",
            "createdBy": {
                "id": 1,
                "name": "admin",
                "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
            },
            "updatedBy": {
                "id": 1,
                "name": "admin",
                "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
            }
        }
    }
}
```

Status Codes:

- `200`: Updated
- `400`: Errors (for example, name or UID already exists, invalid JSON, missing or invalid fields, and so on).
- `401`: Unauthorized
- `403`: Access denied
- `404`: Library element not found
- `412`: Version mismatch

## Delete library element

`DELETE /api/library-elements/:uid`

Deletes an existing library element as specified by the UID. This operation cannot be reverted.

{{< admonition type="note" >}}
You cannot delete a library element that is connected. This operation cannot be reverted.
{{< /admonition >}}

**Example Request**:

```http
DELETE /api/library-elements/nErXDvCkzz HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "message": "Library element deleted",
    "id": 28
}
```

Status Codes:

- `200`: Deleted
- `401`: Unauthorized
- `400`: Bad request
- `403`: Access denied
- `404`: Library element not found

# Enterprise License API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

Licensing is only available in Grafana Enterprise. Read more about [Grafana Enterprise](/docs/grafana/latest/introduction/grafana-enterprise/).

{{< admonition type="caution" >}}
You can't authenticate to the Licensing HTTP API with service account tokens.
Service accounts are limited to an organization and an organization role.
They can't be granted [Grafana server administrator permissions](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/#grafana-server-administrators).

To use these API endpoints you have to use Basic authentication and the Grafana user must have the Grafana server administrator permission.

The `admin` user that Grafana is provisioned with by default has permissions to use these API endpoints.
{{< /admonition >}}

> If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## Check license availability

> **Note:** Available in Grafana Enterprise v7.4+.

`GET /api/licensing/check`

Checks if a valid license is available.

**Required permissions**

See note in the [introduction](#enterprise-license-api) for an explanation.

| Action         | Scope |
| -------------- | ----- |
| licensing:read | n/a   |

### Examples

**Example request:**

```http
GET /api/licensing/check
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 4

true
```

Status codes:

- **200** - OK

## Add license

{{< admonition type="note" >}}
Available in Grafana Enterprise v7.4+.
{{< /admonition >}}

`POST /api/licensing/token`

Applies a license to a Grafana instance.

**Required permissions**

See note in the [introduction](#enterprise-license-api) for an explanation.

| Action          | Scope |
| --------------- | ----- |
| licensing:write | n/a   |

### Examples

**Example request:**

```http
POST /licensing/token
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0aGlzIjoiaXMiLCJub3QiOiJhIiwidmFsaWQiOiJsaWNlbnNlIn0.bxDzxIoJlYMwiEYKYT_l2s42z0Y30tY-6KKoyz9RuLE"}
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 357

{
"status":0,
"jti":"",
"iss":"",
"sub":"",
"iat":0,
"exp":0,
"nbf":0,
"lexp":0,
"lid":"",
"limit_by":"",
"included_users":0,
"lic_exp_warn_days":0,
"tok_exp_warn_days":0,
"update_days":0,
"prod":null,
"company":"",
"account":"",
"slug":"",
"usage_billing":false,
"max_concurrent_user_sessions":0,
"details_url":"",
"trial":false,
"trial_exp":0,
"anonymousRatio":0
}

```

The response is a JSON blob with specific values intentionally not shown. The
available fields may change at any time without any prior notice. Refer to [Check license availability](#check-license-availability) for information on using the API to check the status of your license.

Status Codes:

- **200** - OK
- **400** - Bad request
- **500** - Internal server error (refer to server logs for more details)

## Manually force license refresh

{{< admonition type="note" >}}
Available in Grafana Enterprise v7.4+.
{{< /admonition >}}

`POST /api/licensing/token/renew`

Manually ask license issuer for a new token.

**Required permissions**

See note in the [introduction](#enterprise-license-api) for an explanation.

| Action          | Scope |
| --------------- | ----- |
| licensing:write | n/a   |

### Examples

**Example request:**

```http
POST /api/licensing/token/renew
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{}
```

**Example response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 357

{
  "jti":"2",
  "iss":"https://grafana.com",
  "sub":"https://play.grafana.org/"
  "lid":"1",
  "included_users":15,
  "lic_exp_warn_days":30,
  "tok_exp_warn_days":2,
  "update_days":1,
  "prod":["grafana-enterprise"],
  "company":"Grafana Labs"
}
```

The response is a JSON blob available for debugging purposes. The
available fields may change at any time without any prior notice.

Status Codes:

- **200** - OK
- **401** - Unauthorized
- **403** - Access denied

## Remove license from database

{{< admonition type="note" >}}
Available in Grafana Enterprise v7.4+.
{{< /admonition >}}

`DELETE /api/licensing/token`

Removes the license stored in the Grafana database.

**Required permissions**

See note in the [introduction](#enterprise-license-api) for an explanation.

| Action           | Scope |
| ---------------- | ----- |
| licensing:delete | n/a   |

### Examples

**Example request:**

```http
DELETE /api/licensing/token
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{"instance": "http://play.grafana.org/"}
```

JSON Body schema:

- **instance** – Root URL for the instance for which the license should be deleted. Required.

**Example response:**

```http
HTTP/1.1 202 Accepted
Content-Type: application/json
Content-Length: 2

{}
```

Status codes:

- **202** - Accepted, license removed or did not exist.
- **401** - Unauthorized
- **403** - Access denied
- **422** - Unprocessable entity, incorrect instance name provided.

# Organization API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

The Organization HTTP API is divided in two resources, `/api/org` (current organization)
and `/api/orgs` (admin organizations). One big difference between these are that
the admin of all organizations API only works with basic authentication, see [Admin Organizations API](#admin-organizations-api) for more information.

> If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

### Current Organization endpoints

| Method | Summary                                                                                                         | URI                    |
| ------ | --------------------------------------------------------------------------------------------------------------- | ---------------------- |
| GET    | [Get current Organization](#get-current-organization)                                                           | /api/org/              |
| GET    | [Get all users within the current organization](#get-all-users-within-the-current-organization)                 | /api/org/users         |
| GET    | [Get all users within the current organization (lookup)](#get-all-users-within-the-current-organization-lookup) | /api/org/users/lookup  |
| PATCH  | [Updates the given user](#updates-the-given-user)                                                               | /api/org/users/:userId |
| DELETE | [Delete user in current organization](#delete-user-in-current-organization)                                     | /api/org/users/:userId |
| PUT    | [Update current Organization](#update-current-organization)                                                     | /api/org               |
| POST   | [Add a new user to the current organization](#add-a-new-user-to-the-current-organization)                       | /api/org/users         |

### Admin Organizations endpoints

| Method | Summary                                                       | URI                            |
| ------ | ------------------------------------------------------------- | ------------------------------ |
| GET    | [Get Organization by Id](#get-organization-by-id)             | /api/orgs/:orgId               |
| GET    | [Get Organization by Name](#get-organization-by-name)         | /api/orgs/name/:orgName        |
| POST   | [Create Organization](#create-organization)                   | /api/orgs                      |
| GET    | [Search all Organizations](#search-all-organizations)         | /api/orgs                      |
| PUT    | [Update Organization](#update-organization)                   | /api/orgs/:orgId               |
| DELETE | [Delete Organization](#delete-organization)                   | /api/orgs/:orgId               |
| GET    | [Get Users in Organization](#get-users-in-organization)       | /api/orgs/:orgId/users         |
| POST   | [Add User in Organization](#add-user-in-organization)         | /api/orgs/:orgId/users         |
| PATCH  | [Update Users in Organization](#update-users-in-organization) | /api/orgs/:orgId/users/:userId |
| DELETE | [Delete User in Organization](#delete-user-in-organization)   | /api/orgs/:orgId/users/:userId |

## Current Organization API

### Get current Organization

`GET /api/org/`

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action    | Scope |
| --------- | ----- |
| orgs:read | N/A   |

**Example Request**:

```http
GET /api/org/ HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "id":1,
  "name":"Main Org."
}
```

### Get all users within the current organization

`GET /api/org/users`

Returns all org users within the current organization.
Accessible to users with org admin role.

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action         | Scope    |
| -------------- | -------- |
| org.users:read | users:\* |

**Example Request**:

```http
GET /api/org/users HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "orgId": 1,
    "userId": 1,
    "email": "admin@localhost",
    "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56",
    "login": "admin",
    "role": "Admin",
    "lastSeenAt": "2019-08-09T11:02:49+02:00",
    "lastSeenAtAge": "< 1m"
  }
]
```

### Get all users within the current organization (lookup)

`GET /api/org/users/lookup`

Returns all org users within the current organization, but with less detailed information.
Accessible to users with org admin role, admin in any folder or admin of any team.
Mainly used by Grafana UI for providing list of users when adding team members and
when editing folder/dashboard permissions.

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action         | Scope    |
| -------------- | -------- |
| org.users:read | users:\* |

**Example Request**:

```http
GET /api/org/users/lookup HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "userId": 1,
    "login": "admin",
    "avatarUrl": "/avatar/46d229b033af06a191ff2267bca9ae56"
  }
]
```

### Updates the given user

`PATCH /api/org/users/:userId`

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action          | Scope    |
| --------------- | -------- |
| org.users:write | users:\* |

**Example Request**:

```http
PATCH /api/org/users/1 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "role": "Viewer",
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Organization user updated"}
```

### Delete user in current organization

`DELETE /api/org/users/:userId`

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action           | Scope    |
| ---------------- | -------- |
| org.users:remove | users:\* |

**Example Request**:

```http
DELETE /api/org/users/1 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"User removed from organization"}
```

### Update current Organization

`PUT /api/org`

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action     | Scope |
| ---------- | ----- |
| orgs:write | N/A   |

**Example Request**:

```http
PUT /api/org HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "name":"Main Org."
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Organization updated"}
```

### Add a new user to the current organization

`POST /api/org/users`

Adds a global user to the current organization.

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action        | Scope    |
| ------------- | -------- |
| org.users:add | users:\* |

**Example Request**:

```http
POST /api/org/users HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "role": "Admin",
  "loginOrEmail": "admin"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"User added to organization","userId":1}
```

## Admin Organizations API

{{< admonition type="caution" >}}
You can't authenticate to the Admin Organizations HTTP API with service account tokens.
Service accounts are limited to an organization and an organization role.
They can't be granted [Grafana server administrator permissions](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/#grafana-server-administrators).

To use these API endpoints you have to use Basic authentication and the Grafana user must have the Grafana server administrator permission.

The `admin` user that Grafana is provisioned with by default has permissions to use these API endpoints.
{{< /admonition >}}

### Get Organization by Id

`GET /api/orgs/:orgId`

Only works with Basic Authentication (username and password), see [introduction](#admin-organizations-api).

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action    | Scope |
| --------- | ----- |
| orgs:read | N/A   |

**Example Request**:

```http
GET /api/orgs/1 HTTP/1.1
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "id":1,
  "name":"Main Org.",
  "address":{
    "address1":"",
    "address2":"",
    "city":"",
    "zipCode":"",
    "state":"",
    "country":""
  }
}
```

### Get Organization by Name

`GET /api/orgs/name/:orgName`

Only works with Basic Authentication (username and password), see [introduction](#admin-organizations-api).

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action    | Scope | Note                           |
| --------- | ----- | ------------------------------ |
| orgs:read | N/A   | Needs to be assigned globally. |

**Example Request**:

```http
GET /api/orgs/name/Main%20Org%2E HTTP/1.1
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "id":1,
  "name":"Main Org.",
  "address":{
    "address1":"",
    "address2":"",
    "city":"",
    "zipCode":"",
    "state":"",
    "country":""
  }
}
```

### Create Organization

`POST /api/orgs`

Only works with Basic Authentication (username and password), see [introduction](#admin-organizations-api).

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action      | Scope | Note                           |
| ----------- | ----- | ------------------------------ |
| orgs:create | N/A   | Needs to be assigned globally. |

**Example Request**:

```http
POST /api/orgs HTTP/1.1
Accept: application/json
Content-Type: application/json

{
  "name":"New Org."
}
```

Note: The api will work in the following two ways

1. Need to set GF_USERS_ALLOW_ORG_CREATE=true
2. Set the config value users.allow_org_create to true in ini file

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "orgId":"1",
  "message":"Organization created"
}
```

### Search all Organizations

`GET /api/orgs?perpage=10&page=1`

Only works with Basic Authentication (username and password), see [introduction](#admin-organizations-api).

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action    | Scope | Note                           |
| --------- | ----- | ------------------------------ |
| orgs:read | N/A   | Needs to be assigned globally. |

**Example Request**:

```http
GET /api/orgs HTTP/1.1
Accept: application/json
Content-Type: application/json
```

Note: The api will only work when you pass the admin name and password
to the request HTTP URL, like http://admin:admin@localhost:3000/api/orgs

Default value for the `perpage` parameter is `1000` and for the `page` parameter is `0`.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "id":1,
    "name":"Main Org."
  }
]
```

### Update Organization

`PUT /api/orgs/:orgId`

Update Organization, fields _Address 1_, _Address 2_, _City_ are not implemented yet.
Only works with Basic Authentication (username and password), see [introduction](#admin-organizations-api).

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action     | Scope |
| ---------- | ----- |
| orgs:write | N/A   |

**Example Request**:

```http
PUT /api/orgs/1 HTTP/1.1
Accept: application/json
Content-Type: application/json

{
  "name":"Main Org 2."
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Organization updated"}
```

### Delete Organization

`DELETE /api/orgs/:orgId`

Only works with Basic Authentication (username and password), see [introduction](#admin-organizations-api).

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action      | Scope |
| ----------- | ----- |
| orgs:delete | N/A   |

**Example Request**:

```http
DELETE /api/orgs/1 HTTP/1.1
Accept: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Organization deleted"}
```

### Get Users in Organization

`GET /api/orgs/:orgId/users`

Only works with Basic Authentication (username and password), see [introduction](#admin-organizations-api).

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action         | Scope    |
| -------------- | -------- |
| org.users:read | users:\* |

**Example Request**:

```http
GET /api/orgs/1/users HTTP/1.1
Accept: application/json
Content-Type: application/json
```

Note: The api will only work when you pass the admin name and password
to the request HTTP URL, like http://admin:admin@localhost:3000/api/orgs/1/users

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json
[
  {
    "orgId":1,
    "userId":1,
    "email":"admin@mygraf.com",
    "login":"admin",
    "role":"Admin"
  }
]
```

### Add User in Organization

`POST /api/orgs/:orgId/users`

Only works with Basic Authentication (username and password), see [introduction](#admin-organizations-api).

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action        | Scope    |
| ------------- | -------- |
| org.users:add | users:\* |

**Example Request**:

```http
POST /api/orgs/1/users HTTP/1.1
Accept: application/json
Content-Type: application/json

{
  "loginOrEmail":"user",
  "role":"Viewer"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"User added to organization", "userId": 1}
```

### Update Users in Organization

`PATCH /api/orgs/:orgId/users/:userId`

Only works with Basic Authentication (username and password), see [introduction](#admin-organizations-api).

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action          | Scope    |
| --------------- | -------- |
| org.users:write | users:\* |

**Example Request**:

```http
PATCH /api/orgs/1/users/2 HTTP/1.1
Accept: application/json
Content-Type: application/json

{
  "role":"Admin"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Organization user updated"}
```

### Delete User in Organization

`DELETE /api/orgs/:orgId/users/:userId`

Only works with Basic Authentication (username and password), see [introduction](#admin-organizations-api).

**Required permissions**

See note in the [introduction](#organization-api) for an explanation.

| Action           | Scope    |
| ---------------- | -------- |
| org.users:remove | users:\* |

**Example Request**:

```http
DELETE /api/orgs/1/users/2 HTTP/1.1
Accept: application/json
Content-Type: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"User removed from organization"}
```

# Frontend Settings API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Get Settings

`GET /api/frontend/settings`

**Example Request**:

```http
GET /api/frontend/settings HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "allowOrgCreate":true,
  "appSubUrl":"",
  "buildInfo":{
    "buildstamp":xxxxxx,
    "commit":"vyyyy",
    "version":"zzzzz"
  },
  "datasources":{
    "datasourcename":{
      "index":"grafana-dash",
      "meta":{
        "annotations":true,
        "module":"plugins/datasource/grafana/datasource",
        "name":"Grafana",
        "partials":{
          "annotations":"app/plugins/datasource/grafana/partials/annotations.editor.html",
          "config":"app/plugins/datasource/grafana/partials/config.html"
        },
        "pluginType":"datasource",
        "serviceName":"Grafana",
        "type":"grafanasearch"
      }
    }
  },
  "defaultDatasource": "Grafana"
}
```

# Login API

## Renew session based on remember cookie

`GET /api/login/ping`

**Example Request**:

```http
GET /api/login/ping HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message": "Logged in"}
```

# Health API

## Returns health information about Grafana

`GET /api/health`

**Example Request**

```http
GET /api/health
Accept: application/json
```

**Example Response**:

```http
HTTP/1.1 200 OK

{
  "commit": "087143285",
  "database": "ok",
  "version": "5.1.3"
}
```

# User and Org Preferences API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

Keys:

- **theme** - One of: `light`, `dark`, or an empty string for the default theme
- **homeDashboardId** - Deprecated. Use `homeDashboardUID` instead.
- **homeDashboardUID** - The `:uid` of a dashboard, or the reserved value `global-home`, which shows the instance default home (`home_page`, `default_home_dashboard_path`, or the Grafana home page) instead of the dashboard set at the next level. An empty string clears the preference so the next level applies. `GET` returns the stored value as-is, including `global-home`; the resolved home that Grafana computes from all levels never contains it.
- **timezone** - Any valid IANA timezone string (e.g., `America/New_York`, `Europe/London`), `utc`, `browser`, or an empty string for the default.

Omitting a key will cause the current value to be replaced with the
system default value.

## Get Current User Prefs

`GET /api/user/preferences`

**Example Request**:

```http
GET /api/user/preferences HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "theme": "",
    "homeDashboardId": 217,
    "homeDashboardUID": "jcIIG-07z",
    "timezone": "utc",
    "weekStart": "",
    "navbar": {
        "bookmarkUrls": null
    },
    "queryHistory": {
        "homeTab": ""
    }
}
```

## Update Current User Prefs

`PUT /api/user/preferences`

**Example Request**:

```http
PUT /api/user/preferences HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "theme": "",
  "homeDashboardUID":"jcIIG-07z",
  "timezone":"utc"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: text/plain; charset=utf-8

{"message":"Preferences updated"}
```

## Patch Current User Prefs

Update one or more preferences without modifying the others.

`PATCH /api/user/preferences`

**Example Request**:

```http
PATCH /api/user/preferences HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "theme": "dark"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: text/plain; charset=utf-8

{"message":"Preferences updated"}
```

## Get Current Org Prefs

`GET /api/org/preferences`

**Example Request**:

```http
GET /api/org/preferences HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "theme": "",
    "homeDashboardId": 0,
    "homeDashboardUID": "",
    "timezone": "",
    "weekStart": "",
    "navbar": {
        "bookmarkUrls": null
    },
    "queryHistory": {
        "homeTab": ""
    }
}
```

## Update Current Org Prefs

`PUT /api/org/preferences`

**Example Request**:

```http
PUT /api/org/preferences HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "theme": "",
  "homeDashboardUID":"jcIIG-07z",
  "timezone":"utc"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: text/plain; charset=utf-8

{"message":"Preferences updated"}
```

## Patch Current Org Prefs

Update one or more preferences without modifying the others.

`PATCH /api/org/preferences`

**Example Request**:

```http
PATCH /api/org/preferences HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "theme": "dark"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: text/plain; charset=utf-8

{"message":"Preferences updated"}
```

# Query and resource caching API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Requirements

If you're running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## Enable caching for a data source

`POST /api/datasources/:dataSourceUID/cache/enable`

**Required permissions**

See note in the [introduction](#query-and-resource-caching-api) for an explanation.

| Action                    | Scope          |
| ------------------------- | -------------- |
| datasources.caching:write | datasources:\* |

### Examples

**Example Request**:

```http
POST /api/datasources/jZrmlLCGka/cache/enable HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
   "message": "Data source cache enabled",
   "dataSourceID": 1,
   "dataSourceUID": "jZrmlLCGka",
   "enabled": true,
   "ttlQueriesMs": 300000,
   "ttlResourcesMs": 300000,
   "useDefaultTTL": true,
   "defaultTTLMs": 300000,
   "created": "2023-04-21T11:49:22-04:00",
   "updated": "2023-04-24T16:30:29-04:00"
}
```

#### Status codes

| Code | Description                                                              |
| ---- | ------------------------------------------------------------------------ |
| 200  | Cache was successfully enabled for the data source                       |
| 500  | Unexpected error. Refer to the body and/or server logs for more details. |

## Disable caching for a data source

`POST /api/datasources/:dataSourceUID/cache/disable`

**Required permissions**

See note in the [introduction](#query-and-resource-caching-api) for an explanation.

| Action                    | Scope          |
| ------------------------- | -------------- |
| datasources.caching:write | datasources:\* |

### Examples

**Example Request**:

```http
POST /api/datasources/jZrmlLCGka/cache/disable HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
   "message": "Data source cache disabled",
   "dataSourceID": 1,
   "dataSourceUID": "jZrmlLCGka",
   "enabled": false,
   "ttlQueriesMs": 300000,
   "ttlResourcesMs": 300000,
   "useDefaultTTL": true,
   "defaultTTLMs": 0,
   "created": "2023-04-21T11:49:22-04:00",
   "updated": "2023-04-24T16:30:31-04:00"
}
```

#### Status codes

| Code | Description                                                              |
| ---- | ------------------------------------------------------------------------ |
| 200  | Cache was successfully enabled for the data source                       |
| 500  | Unexpected error. Refer to the body and/or server logs for more details. |

## Clean cache for all data sources

`POST /api/datasources/:dataSourceUID/cache/clean`

Will clean cached data for _all_ data sources with caching enabled. The `dataSourceUID` specified will only be used to return the configuration for that data source.

**Required permissions**

See note in the [introduction](#query-and-resource-caching-api) for an explanation.

| Action                    | Scope          |
| ------------------------- | -------------- |
| datasources.caching:write | datasources:\* |

### Examples

**Example Request**:

```http
POST /api/datasources/jZrmlLCGka/cache/clean HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
   "message": "Data source cache cleaned",
   "dataSourceID": 1,
   "dataSourceUID": "jZrmlLCGka",
   "enabled": false,
   "ttlQueriesMs": 300000,
   "ttlResourcesMs": 300000,
   "useDefaultTTL": true,
   "defaultTTLMs": 0,
   "created": "2023-04-21T11:49:22-04:00",
   "updated": "2023-04-24T16:30:31-04:00"
}
```

#### Status codes

| Code | Description                                                              |
| ---- | ------------------------------------------------------------------------ |
| 200  | Cache was successfully enabled for the data source                       |
| 500  | Unexpected error. Refer to the body and/or server logs for more details. |

## Update cache configuration for a data source

`POST /api/datasources/:dataSourceUID/cache`

**Required permissions**

See note in the [introduction](#query-and-resource-caching-api) for an explanation.

| Action                    | Scope          |
| ------------------------- | -------------- |
| datasources.caching:write | datasources:\* |

### Examples

**Example Request**:

```http
POST /api/datasources/jZrmlLCGka/cache HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
   "dataSourceID": 1,
   "dataSourceUID": "jZrmlLCGka",
   "enabled": true,
   "useDefaultTTL": false,
   "ttlQueriesMs": 60000,
   "ttlResourcesMs": 300000,
}
```

#### JSON Body Schema

| Field name     | Data type | Description                                                                                                                                 |
| -------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| dataSourceID   | number    | The ID of the data source to configure.                                                                                                     |
| dataSourceUID  | string    | The UID of the data source to configure.                                                                                                    |
| enabled        | boolean   | Whether or not to enable caching for this data source.                                                                                      |
| useDefaultTTL  | boolean   | Whether the configured default TTL (Time-To-Live) should be used for both query and resource caching, instead of the user-specified values. |
| ttlQueriesMs   | number    | The TTL to use for query caching, in milliseconds.                                                                                          |
| ttlResourcesMs | number    | The TTL to use for resource caching, in milliseconds.                                                                                       |

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
   "message": "Data source cache settings updated",
   "dataSourceID": 1,
   "dataSourceUID": "jZrmlLCGka",
   "enabled": true,
   "useDefaultTTL": false,
   "ttlQueriesMs": 60000,
   "ttlResourcesMs": 300000,
   "defaultTTLMs": 300000,
   "created": "2023-04-21T11:49:22-04:00",
   "updated": "2023-04-24T17:03:40-04:00"
}
```

#### Status codes

| Code | Description                                                              |
| ---- | ------------------------------------------------------------------------ |
| 200  | Cache was successfully enabled for the data source                       |
| 400  | Request errors (invalid json, missing or invalid fields, etc)            |
| 500  | Unexpected error. Refer to the body and/or server logs for more details. |

## Get cache configuration for a data source

`GET /api/datasources/:dataSourceUID/cache`

**Required permissions**

See note in the [introduction](#query-and-resource-caching-api) for an explanation.

| Action                   | Scope          |
| ------------------------ | -------------- |
| datasources.caching:read | datasources:\* |

### Examples

**Example Request**:

```http
GET /api/datasources/jZrmlLCGka/cache HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
   "message": "Data source cache settings loaded",
   "dataSourceID": 1,
   "dataSourceUID": "jZrmlLCGka",
   "enabled": true,
   "useDefaultTTL": false,
   "ttlQueriesMs": 60000,
   "ttlResourcesMs": 300000,
   "defaultTTLMs": 300000,
   "created": "2023-04-21T11:49:22-04:00",
   "updated": "2023-04-24T17:03:40-04:00"
}
```

#### Status codes

| Code | Description                                                              |
| ---- | ------------------------------------------------------------------------ |
| 200  | Cache was successfully enabled for the data source                       |
| 500  | Unexpected error. Refer to the body and/or server logs for more details. |

# Query history API

{{< docs/shared lookup="developers/deprecated-apis-nonmigrated.md" source="grafana" version="<GRAFANA_VERSION>" >}}

This API can be used to add queries to Query history. It requires that the user is logged in and that Query history feature is enabled in config file.

## Add query to Query history

`POST /api/query-history`

Adds query to query history.

**Example request:**

```http
POST /api/query-history HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
{
  "datasourceUid": "PE1C5CBDA0504A6A3",
  "queries": [
    {
        "refId": "A",
        "key": "Q-87fed8e3-62ba-4eb2-8d2a-4129979bb4de-0",
        "scenarioId": "csv_content",
        "datasource": {
            "type": "testdata",
            "uid": "PD8C576611E62080A"
        }
    }
]
}
```

JSON body schema:

- **datasourceUid** – Data source uid.
- **queries** – JSON of query or queries.

**Example response:**

```http
HTTP/1.1 200
Content-Type: application/json
{
  "result": {
    "uid": "Ahg678z",
    "datasourceUid": "PE1C5CBDA0504A6A3",
    "createdBy": 1,
    "createdAt": 1643630762,
    "starred": false,
    "comment": "",
    "queries": [
      {
        "refId": "A",
        "key": "Q-87fed8e3-62ba-4eb2-8d2a-4129979bb4de-0",
        "scenarioId": "csv_content",
        "datasource": {
            "type": "testdata",
            "uid": "PD8C576611E62080A"
        }
      }
    ]
  }
}
```

Status codes:

- **200** – OK
- **400** - Errors (invalid JSON, missing or invalid fields)
- **401** – Unauthorized
- **500** – Internal error

## Query history search

`GET /api/query-history`

Returns a list of queries in the query history that matches the search criteria. Query history search supports pagination. Use the `limit` parameter to control the maximum number of queries returned; the default limit is 100. You can also use the `page` query parameter to fetch queries from any page other than the first one.

Query parameters:

- **datasourceUid** - Filter the query history for the selected data source. To perform an "AND" filtering with multiple data sources, specify the data source parameter using the following format: `datasourceUid=uid1&datasourceUid=uid2`.
- **searchString** – Filter the query history based on the content.
- **sort** - Specify the sorting order. Sorting can be `time-asc` or `time-desc`. The default is `time-desc`.
- **onlyStarred** - Search for queries that are starred. Defaults to `false`.
- **page** - Search supports pagination. Specify which page number to return. Use the limit parameter to specify the number of queries per page.
- **limit** - Limits the number of returned query history items per page. The default is 100 queries per page.
- **from/to** - Specifies time range for the query history search. The time can be either epoch timestamps in milliseconds or relative using Grafana time units. For example, now-5m.

**Example request for query history search**:

```http
GET /api/query-history?datasourceUid="PE1C5CBDA0504A6A3"&datasourceUid="FG1C1CBDA0504A6EL"&searchString="ALERTS"&sort="time-asc" HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response for query history search**:

```http
HTTP/1.1 200
Content-Type: application/json
{
  "result": {
    "totalCount": 150,
    "page": 1,
    "perPage": 100
    "queryHistory":[{
    "uid": "Ahg678z",
    "datasourceUid": "PE1C5CBDA0504A6A3",
    "createdBy": 1,
    "createdAt": 1643630762,
    "starred": false,
    "comment": "",
    "queries": [
      {
        "refId": "A",
        "key": "Q-87fed8e3-62ba-4eb2-8d2a-4129979bb4de-0",
        "scenarioId": "csv_content",
        "datasource": {
            "type": "testdata",
            "uid": "PE1C5CBDA0504A6A3"
        }
      }
    ]
  }]
}
```

Status codes:

- **200** – OK
- **401** – Unauthorized
- **500** – Internal error

## Delete query from Query history by UID

`DELETE /api/query-history/:uid`

Deletes the query in query history that matches the specified uid. It requires that the user is logged in and that Query history feature is enabled in config file.

**Example Request**:

```http
DELETE /api/query-history/P8zM2I1nz HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
    "message": "Query deleted",
    "id": 28
}
```

Status codes:

- **200** – OK
- **401** – Unauthorized
- **500** – Internal error

## Update comment of query in Query history by UID

`PATCH /api/query-history/:uid`

Updates comment of a query with a specific uid that is stored in the query history.

Query parameters:

- **comment** – New comment that will be added to the specified query.

**Example Request**:

```http
PATCH /api/query-history/P8zM2I1nz HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
{
  "comment": "Debugging query",
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json
{
  "result": {
    "uid": "P8zM2I1nz",
    "datasourceUid": "PE1C5CBDA0504A6A3",
    "createdBy": 1,
    "createdAt": 1643630762,
    "starred": false,
    "comment": "Debugging query",
    "queries": [
      {
        "refId": "A",
        "key": "Q-87fed8e3-62ba-4eb2-8d2a-4129979bb4de-0",
        "scenarioId": "csv_content",
        "datasource": {
            "type": "testdata",
            "uid": "PD8C576611E62080A"
        }
      }
    ]
  }
}
```

Status codes:

- **200** – OK
- **400** - Errors (invalid JSON, missing or invalid fields)
- **401** – Unauthorized
- **500** – Internal error

## Star query in Query history

`POST /api/query-history/star/:uid`

Stars query in query history.

**Example request:**

```http
POST /api/query-history/star/P8zM2I1nz HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response:**

```http
HTTP/1.1 200
Content-Type: application/json
{
  "result": {
    "uid": "P8zM2I1nz",
    "datasourceUid": "PE1C5CBDA0504A6A3",
    "createdBy": 1,
    "createdAt": 1643630762,
    "starred": false,
    "comment": "Debugging query",
    "queries": [
      {
        "refId": "A",
        "key": "Q-87fed8e3-62ba-4eb2-8d2a-4129979bb4de-0",
        "scenarioId": "csv_content",
        "datasource": {
            "type": "testdata",
            "uid": "PD8C576611E62080A"
        }
      }
    ]
  }
}
```

Status codes:

- **200** – OK
- **401** – Unauthorized
- **500** – Internal error

## Unstar query in Query history

`DELETE /api/query-history/star/:uid`

Removes stars from query in query history.

**Example request:**

```http
DELETE /api/query-history/star/P8zM2I1nz  HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example response:**

```http
HTTP/1.1 200
Content-Type: application/json
{
  "result": {
    "uid": "P8zM2I1nz",
    "datasourceUid": "PE1C5CBDA0504A6A3",
    "createdBy": 1,
    "createdAt": 1643630762,
    "starred": false,
    "comment": "Debugging query",
    "queries": [
      {
        "refId": "A",
        "key": "Q-87fed8e3-62ba-4eb2-8d2a-4129979bb4de-0",
        "scenarioId": "csv_content",
        "datasource": {
            "type": "testdata",
            "uid": "PD8C576611E62080A"
        }
      }
    ]
  }
}
```

Status codes:

- **200** – OK
- **401** – Unauthorized
- **500** – Internal error

# Reporting API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

This API allows you to interact programmatically with the [Reporting](/docs/grafana/latest/dashboards/create-reports/) feature.

> The Reporting API is not stabilized yet, it is still in active development and may change without prior notice.

> Reporting is only available in Grafana Enterprise. Read more about [Grafana Enterprise](/docs/grafana/latest/introduction/grafana-enterprise/).

> If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## List all reports

`GET /api/reports`

#### Required permissions

See note in the [introduction](#reporting-api) for an explanation.

| Action       | Scope                       |
| ------------ | --------------------------- |
| reports:read | reports:\*<br>reports:id:\* |

### Example request

```http
GET /api/reports HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 1840

[
	{
		"id": 2,
		"userId": 1,
		"orgId": 1,
		"name": "Report 2",
		"recipients": "example-report@grafana.com",
		"replyTo": "",
		"message": "Hi, \nPlease find attached a PDF status report. If you have any questions, feel free to contact me!\nBest,",
		"schedule": {
			"startDate": "2022-10-02T00:00:00+02:00",
			"endDate": null,
			"frequency": "once",
			"intervalFrequency": "",
			"intervalAmount": 0,
			"workdaysOnly": false,
			"dayOfMonth": "2",
			"timeZone": "Europe/Warsaw"
		},
		"options": {
			"orientation": "landscape",
			"layout": "grid",
		},
		"enableDashboardUrl": true,
		"state": "scheduled",
		"dashboards": [
			{
				"dashboard": {
					"id": 463,
					"uid": "7MeksYbmk",
					"name": "Alerting with TestData"
				},
				"reportVariables": {
					"namefilter": "TestData"
				}
			}
		],
		"formats": [
			"pdf",
			"csv"
		],
		"created": "2022-09-19T11:44:42+02:00",
		"updated": "2022-09-19T11:44:42+02:00"
	}
]
```

### Status Codes

- **200** – OK
- **401** - Authentication failed, refer to [Authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/authentication/).
- **500** – Unexpected error or server misconfiguration. Refer to server logs for more details.

## Get a report

`GET /api/reports/:id`

#### Required permissions

See note in the [introduction](#reporting-api) for an explanation.

| Action       | Scope                                                      |
| ------------ | ---------------------------------------------------------- |
| reports:read | reports:\*<br>reports:id:\*<br>reports:id:1(single report) |

### Example request

```http
GET /api/reports/2 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 940

{
	"id": 2,
	"userId": 1,
	"orgId": 1,
	"name": "Report 2",
	"recipients": "example-report@grafana.com",
	"replyTo": "",
	"message": "Hi, \nPlease find attached a PDF status report. If you have any questions, feel free to contact me!\nBest,",
	"schedule": {
		"startDate": "2022-10-02T00:00:00+02:00",
		"endDate": null,
		"frequency": "once",
		"intervalFrequency": "",
		"intervalAmount": 0,
		"workdaysOnly": false,
		"dayOfMonth": "2",
		"timeZone": "Europe/Warsaw"
	},
	"options": {
		"orientation": "landscape",
		"layout": "grid",
	},
	"enableDashboardUrl": true,
	"state": "scheduled",
	"dashboards": [
		{
			"dashboard": {
				"id": 463,
				"uid": "7MeksYbmk",
				"name": "Alerting with TestData"
			},
			"timeRange": {
				"from": "",
				"to": ""
			},
			"reportVariables": {
				"namefilter": "TestData"
			}
		}
	],
	"formats": [
		"pdf",
		"csv"
	],
	"created": "2022-09-12T11:44:42+02:00",
	"updated": "2022-09-12T11:44:42+02:00"
}
```

### Status Codes

- **200** – OK
- **400** – Bad request (invalid report ID).
- **401** - Authentication failed, refer to [Authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/authentication/).
- **403** – Forbidden (access denied to a report or a dashboard used in the report).
- **404** – Not found (such report does not exist).
- **500** – Unexpected error or server misconfiguration. Refer to server logs for more details.

## Create a report

`POST /api/reports`

#### Required permissions

See note in the [introduction](#reporting-api) for an explanation.

| Action         | Scope |
| -------------- | ----- |
| reports:create | n/a   |

### Example request

```http
POST /api/reports HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
	"name": "Report 4",
	"recipients": "texample-report@grafana.com",
	"replyTo": "",
	"message": "Hello, please, find the report attached",
	"schedule": {
		"startDate": "2022-10-02T10:00:00+02:00",
		"endDate": "2022-11-02T20:00:00+02:00",
		"frequency": "daily",
		"intervalFrequency": "",
		"intervalAmount": 0,
		"workdaysOnly": true,
		"timeZone": "Europe/Warsaw"
	},
	"options": {
		"orientation": "landscape",
		"layout": "grid"
	},
	"enableDashboardUrl": true,
	"dashboards": [
		{
			"dashboard": {
				"uid": "7MeksYbmk",
			},
			"timeRange": {
				"from": "2022-08-08T15:00:00+02:00",
				"to": "2022-09-02T17:00:00+02:00"
			},
			"reportVariables": {
				"variable1": "Value1"
			}
		}
	],
	"formats": [
		"pdf",
		"csv"
	]
}
```

#### Config JSON Body Schema

| Field name         | Data type | Description                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ------------------ | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name               | string    | Name of the report that is used as an email subject.                                                                                                                                                                                                                                                                                                                                                                                              |
| recipients         | string    | Comma-separated list of emails to which to send the report to.                                                                                                                                                                                                                                                                                                                                                                                    |
| replyTo            | string    | Comma-separated list of emails used in a reply-to field of the report email.                                                                                                                                                                                                                                                                                                                                                                      |
| message            | string    | Text message used for the body of the report email.                                                                                                                                                                                                                                                                                                                                                                                               |
| startDate          | string    | Report distribution starts from this date.                                                                                                                                                                                                                                                                                                                                                                                                        |
| endDate            | string    | Report distribution ends on this date.                                                                                                                                                                                                                                                                                                                                                                                                            |
| frequency          | string    | Specifies how often the report should be sent. Can be `once`, `hourly`, `daily`, `weekly`, `monthly`, `last` or `custom`.<br/><br/>`last` - schedules the report for the last day of month.<br/><br/>`custom` - schedules the report to be sent on a custom interval.<br/>It requires `intervalFrequency` and `intervalAmount` to be specified: for example, every 2 weeks, where 2 is an `intervalAmount` and `weeks` is an `intervalFrequency`. |
| intervalFrequency  | string    | The type of the `custom` interval: `hours`, `days`, `weeks`, `months`.                                                                                                                                                                                                                                                                                                                                                                            |
| intervalAmount     | number    | `custom` interval amount.                                                                                                                                                                                                                                                                                                                                                                                                                         |
| workdaysOnly       | string    | Send the report only on Monday-Friday. Applicable to `hourly` and `daily` types of schedule.                                                                                                                                                                                                                                                                                                                                                      |
| timeZone           | string    | Time zone used to schedule report execution.                                                                                                                                                                                                                                                                                                                                                                                                      |
| orientation        | string    | Can be `portrait` or `landscape`.                                                                                                                                                                                                                                                                                                                                                                                                                 |
| layout             | string    | Can be `grid` or `simple`.                                                                                                                                                                                                                                                                                                                                                                                                                        |
| enableDashboardUrl | bool      | Adds a dashboard url to the bottom of the report email.                                                                                                                                                                                                                                                                                                                                                                                           |
| formats            | []string  | Specified what kind of attachment to generate for the report - `csv`, `pdf`, `image`.<br/>`pdf` is the default one.<br/>`csv` attaches a CSV file for each table panel.<br/>`image` embeds an image of a dashboard into the email's body.                                                                                                                                                                                                         |
| dashboards         | []object  | Dashboards to generate a report for.<br/> See "Report Dashboard Schema" section below.                                                                                                                                                                                                                                                                                                                                                            |

#### Report Dashboard Schema

| Field name                     | Data type | Description                                                                                                                                                   |
| ------------------------------ | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| dashboard.uid                  | string    | Dashboard [UID](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/api-legacy/dashboard_versions/).                |
| timeRange.from                 | string    | Dashboard time range from.                                                                                                                                    |
| timeRange.to                   | string    | Dashboard time range to.                                                                                                                                      |
| reportVariables.<variableName> | string    | Key-value pairs containing the template variables for this report, in JSON format. If empty, the template variables from the report's dashboard will be used. |

### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 35

{
	"id": 4,
	"message": "Report created"
}
```

### Status Codes

- **200** – OK
- **400** – Bad request (invalid json, missing or invalid fields values, etc.).
- **403** - Forbidden (access denied to a report or a dashboard used in the report).
- **500** - Unexpected error or server misconfiguration. Refer to server logs for more details

## Update a report

`PUT /api/reports/:id`

#### Required permissions

See note in the [introduction](#reporting-api) for an explanation.

| Action        | Scope                                                     |
| ------------- | --------------------------------------------------------- |
| reports:write | reports:\*</br>reports:id:\*</br>reports:1(single report) |

### Example request

See [JSON body schema](#config-json-body-schema) for fields description.

```http
GET /api/reports HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
	"name": "Updated Report",
	"recipients": "example-report@grafana.com",
	"replyTo": "",
	"message": "Hello, please, find the report attached",
	"schedule": {
		"frequency": "hourly",
		"timeZone": "Africa/Cairo",
		"workdaysOnly": true,
		"startDate": "2022-10-10T10:00:00+02:00",
		"endDate": "2022-11-20T19:00:00+02:00"
	},
	"options": {
		"orientation": "landscape",
		"layout": "grid",
	},
	"enableDashboardUrl": true,
	"state": "scheduled",
	"dashboards": [
		{
			"dashboard": {
				"id": 463,
				"uid": "7MeksYbmk",
				"name": "Alerting with TestData"
			},
			"timeRange": {
				"from": "2022-08-08T15:00:00+02:00",
				"to": "2022-09-02T17:00:00+02:00"
			},
			"reportVariables": {
				"variable1": "Value1"
			}
		}
	],
	"formats": [
		"pdf",
		"csv"
	]
}
```

### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 28

{
	"message": "Report updated"
}
```

### Status Codes

- **200** – OK
- **400** – Bad request (invalid json, missing or invalid fields values, etc.).
- **401** - Authentication failed, refer to [Authentication API](/docs/grafana/latest/developer-resources/api-reference/http-api/authentication/).
- **403** – Forbidden (access denied to a report or a dashboard used in the report).
- **404** – Not found (such report does not exist).
- **500** – Unexpected error or server misconfiguration. Refer to server logs for more details.

## Delete a report

`DELETE /api/reports/:id`

#### Required permissions

See note in the [introduction](#reporting-api) for an explanation.

| Action         | Scope                                                     |
| -------------- | --------------------------------------------------------- |
| reports:delete | reports:\*</br>reports:id:\*</br>reports:1(single report) |

### Example request

```http
DELETE /api/reports/6 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 39

{
	"message": "Report config was removed"
}
```

### Status Codes

- **200** – OK
- **400** – Bad request (invalid report ID).
- **401** - Authentication failed, refer to [Authentication API](/docs/grafana/latest/developer-resources/api-reference/http-api/authentication/).
- **404** - Not found (report with this ID does not exist).
- **500** - Unexpected error or server misconfiguration. Refer to server logs for more details

## Send a report

`POST /api/reports/email`

Generate and send a report. This API waits for the report to be generated before returning. We recommend that you set the client's timeout to at least 60 seconds.

#### Required permissions

See note in the [introduction](#reporting-api) for an explanation.

| Action       | Scope |
| ------------ | ----- |
| reports:send | n/a   |

### Example request

```http
POST /api/reports/email HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "id":"3",
  "useEmailsFromReport": true
}
```

#### JSON Body Schema

| Field name          | Data type | Description                                                                                                                                            |
| ------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| id                  | string    | ID of the report to send. It is the same as in the URL when editing a report, not to be confused with the ID of the dashboard. Required.               |
| emails              | string    | Comma-separated list of emails to which to send the report to. Overrides the emails from the report. Required if `useEmailsFromReport` is not present. |
| useEmailsFromReport | boolean   | Send the report to the emails specified in the report. Required if `emails` is not present.                                                            |

### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 29

{"message":"Report was sent"}
```

### Status Codes

- **200** – Report was sent.
- **400** – Bad request (invalid json, missing content-type, missing or invalid fields, etc.).
- **401** - Authentication failed, refer to [Authentication API](/docs/grafana/latest/developer-resources/api-reference/http-api/authentication/).
- **403** - Forbidden (access denied to a report or a dashboard used in the report).
- **404** - Report not found.
- **500** - Unexpected error or server misconfiguration. Refer to server logs for more details.

## Get reports branding settings

`GET /api/reports/settings`

Returns reports branding settings that are global and used across all the reports.

#### Required permissions

See note in the [introduction](#reporting-api) for an explanation.

| Action                | Scope |
| --------------------- | ----- |
| reports.settings:read | n/a   |

### Example request

```http
GET /api/reports/settings HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 181

{
	"id": 1,
	"userId": 1,
	"orgId": 1,
	"branding": {
		"reportLogoUrl": "",
		"emailLogoUrl": "",
		"emailFooterMode": "sent-by",
		"emailFooterText": "Grafana Labs",
		"emailFooterLink": "https://grafana.com/"
	}
}
```

### Status Codes

- **200** – OK
- **401** - Authentication failed, refer to [Authentication API](/docs/grafana/latest/developer-resources/api-reference/http-api/authentication/).
- **500** - Unexpected error or server misconfiguration. Refer to server logs for more detail

## Save reports branding settings

`POST /api/reports/settings`

Creates settings if they don't exist, otherwise updates them. These settings are global and used across all the reports.

#### Required permissions

See note in the [introduction](#reporting-api) for an explanation.

| Action                 | Scope |
| ---------------------- | ----- |
| reports.settings:write | n/a   |

### Example request

```http
POST /api/reports/settings HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
	"branding": {
		"reportLogoUrl": "https://grafana.com/reportLogo.jpg",
		"emailLogoUrl": "https://grafana.com/emailLogo.jpg",
		"emailFooterMode": "sent-by",
		"emailFooterText": "Grafana Labs",
		"emailFooterLink": "https://grafana.com/"
	}
}
```

#### JSON Body Schema

| Field name               | Data type | Description                                                                                                                                                                                                                                                                            |
| ------------------------ | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| branding.reportLogoUrl   | string    | URL of an image used as a logo on every page of the report.                                                                                                                                                                                                                            |
| branding.emailLogoUrl    | string    | URL of an image used as a logo in the email.                                                                                                                                                                                                                                           |
| branding.emailFooterMode | string    | Can be `sent-by` or `none`.<br/>`sent-by` adds a "Sent by `branding.emailFooterText`" footer link to the email. Requires specifying values in the `branding.emailFooterText` and `branding.emailFooterLink` fields.<br/>`none` suppresses adding a "Sent by" footer link to the email. |
| branding.emailFooterText | string    | Text of a URL added to the email "Sent by" footer.                                                                                                                                                                                                                                     |
| branding.emailFooterLink | string    | URL address value added to the email "Sent by" footer.                                                                                                                                                                                                                                 |

### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 35

{
	"message": "Report settings saved"
}
```

### Status Codes

- **200** – OK
- **400** – Bad request (invalid json, missing or invalid fields values, etc.).
- **401** - Authentication failed, refer to [Authentication API](/docs/grafana/latest/developer-resources/api-reference/http-api/authentication/).
- **500** - Unexpected error or server misconfiguration. Refer to server logs for more detail

## Send a test email

`POST /api/reports/test-email`

Sends a test email with a report without persisting it in the database.

#### Required permissions

See note in the [introduction](#reporting-api) for an explanation.

| Action       | Scope |
| ------------ | ----- |
| reports:send | n/a   |

### Example request

See [JSON body schema](#config-json-body-schema) for fields description.

```http
POST /api/reports/test-email HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{{
	"name": "Report 4",
	"recipients": "example-report@grafana.com",
	"replyTo": "",
	"message": "Hello, please, find the report attached",
	"schedule": {
		"startDate": "2022-10-02T10:00:00+02:00",
		"endDate": "2022-11-02T20:00:00+02:00",
		"frequency": "daily",
		"intervalFrequency": "",
		"intervalAmount": 0,
		"workdaysOnly": true,
		"timeZone": "Europe/Warsaw"
	},
	"options": {
		"orientation": "landscape",
		"layout": "grid"
	},
	"enableDashboardUrl": true,
	"dashboards": [
		{
			"dashboard": {
				"uid": "7MeksYbmk",
			},
			"timeRange": {
				"from": "2022-08-08T15:00:00+02:00",
				"to": "2022-09-02T17:00:00+02:00"
			},
			"reportVariables": {
				"variable1": "Value1"
			}
		}
	],
	"formats": [
		"pdf",
		"csv"
	]
}
```

### Example response

```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 29

{
	"message": "Test email sent"
}
```

### Status Codes

- **200** – OK
- **400** – Bad request (invalid json, missing or invalid fields values, etc.).
- **401** - Authentication failed, refer to [Authentication API](/docs/grafana/latest/developer-resources/api-reference/http-api/authentication/).
- **403** - Forbidden (access denied to a report or a dashboard used in the report).
- **500** - Unexpected error or server misconfiguration. Refer to server logs for more details

# Service account API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

> If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.
> For Grafana Cloud instances, please use a Bearer token to authenticate. The examples within this section reference Basic authentication which is for On-Prem Grafana instances.

## Search service accounts with Paging

`GET /api/serviceaccounts/search?perpage=10&page=1&query=myserviceaccount`

**Required permissions**

See note in the [introduction](#service-account-api) for an explanation.

| Action               | Scope |
| -------------------- | ----- |
| serviceaccounts:read | n/a   |

**Example Request**:

```http
GET /api/serviceaccounts/search?perpage=10&page=1&query=mygraf HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

Default value for the `perpage` parameter is `1000` and for the `page` parameter is `1`. The `totalCount` field in the response can be used for pagination of the user list E.g. if `totalCount` is equal to 100 users and the `perpage` parameter is set to 10 then there are 10 pages of users. The `query` parameter is optional and it will return results where the query value is contained in one of the `name`. Query values with spaces need to be URL encoded e.g. `query=Jane%20Doe`.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json
{
	"totalCount": 2,
	"serviceAccounts": [
		{
			"id": 1,
			"name": "grafana",
			"login": "sa-grafana",
			"orgId": 1,
			"isDisabled": false,
			"role": "Viewer",
			"tokens": 0,
			"avatarUrl": "/avatar/85ec38023d90823d3e5b43ef35646af9",
			"accessControl": {
				"serviceaccounts:delete": true,
				"serviceaccounts:read": true,
				"serviceaccounts:write": true
			}
		},
		{
			"id": 2,
			"name": "test",
			"login": "sa-test",
			"orgId": 1,
			"isDisabled": false,
			"role": "Viewer",
			"tokens": 0,
			"avatarUrl": "/avatar/8ea890a677d6a223c591a1beea6ea9d2",
			"accessControl": {
				"serviceaccounts:delete": true,
				"serviceaccounts:read": true,
				"serviceaccounts:write": true
			}
		}
	],
	"page": 1,
	"perPage": 10
}
```

## Create service account

`POST /api/serviceaccounts`

**Required permissions**

See note in the [introduction](#service-account-api) for an explanation.

| Action                 | Scope |
| ---------------------- | ----- |
| serviceaccounts:create | n/a   |

**Example Request**:

```http
POST /api/serviceaccounts HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=

{
  "name": "grafana",
  "role": "Viewer",
  "isDisabled": false
}
```

**Example Response**:

```http
HTTP/1.1 201
Content-Type: application/json

{
	"id": 1,
	"name": "test",
	"login": "sa-test",
	"orgId": 1,
	"isDisabled": false,
	"createdAt": "2022-03-21T14:35:33Z",
	"updatedAt": "2022-03-21T14:35:33Z",
	"avatarUrl": "/avatar/8ea890a677d6a223c591a1beea6ea9d2",
	"role": "Viewer",
	"teams": []
}
```

Fixed and custom roles can be set on service accounts using the [RBAC HTTP API](/docs/grafana/latest/developers/http_api/access_control/#set-user-role-assignments).

## Get a service account by ID

`GET /api/serviceaccounts/:id`

**Required permissions**

See note in the [introduction](#service-account-api) for an explanation.

| Action               | Scope                 |
| -------------------- | --------------------- |
| serviceaccounts:read | serviceaccounts:id:\* |

**Example Request**:

```http
GET /api/serviceaccounts/1 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
	"id": 1,
	"name": "test",
	"login": "sa-test",
	"orgId": 1,
	"isDisabled": false,
	"createdAt": "2022-03-21T14:35:33Z",
	"updatedAt": "2022-03-21T14:35:33Z",
	"avatarUrl": "/avatar/8ea890a677d6a223c591a1beea6ea9d2",
	"role": "Viewer",
	"teams": []
}
```

## Update service account

`PATCH /api/serviceaccounts/:id`

**Required permissions**

See note in the [introduction](#service-account-api) for an explanation.

| Action                | Scope                 |
| --------------------- | --------------------- |
| serviceaccounts:write | serviceaccounts:id:\* |

**Example Request**:

```http
PATCH /api/serviceaccounts/2 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=

{
  "name": "test",
	"role": "Editor"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
	"id": 2,
	"name": "test",
	"login": "sa-grafana",
	"orgId": 1,
	"isDisabled": false,
	"createdAt": "2022-03-21T14:35:44Z",
	"updatedAt": "2022-03-21T14:35:44Z",
	"avatarUrl": "/avatar/8ea890a677d6a223c591a1beea6ea9d2",
	"role": "Editor",
	"teams": []
}
```

Fixed and custom roles can be set on service accounts using the [RBAC HTTP API](/docs/grafana/latest/developers/http_api/access_control/#set-user-role-assignments).

## Delete service account

`DELETE /api/serviceaccounts/:id`

**Required permissions**

See note in the [introduction](#service-account-api) for an explanation.

| Action                 | Scope                 |
| ---------------------- | --------------------- |
| serviceaccounts:delete | serviceaccounts:id:\* |

**Example Request**:

```http
DELETE /api/serviceaccounts/2 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
	"message": "Service account deleted"
}
```

---

## Get service account tokens

`GET /api/serviceaccounts/:id/tokens`

**Required permissions**

See note in the [introduction](#service-account-api) for an explanation.

| Action               | Scope                 |
| -------------------- | --------------------- |
| serviceaccounts:read | serviceaccounts:id:\* |

**Example Request**:

```http
GET /api/serviceaccounts/2/tokens HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
	{
		"id": 1,
		"name": "grafana",
		"role": "Viewer",
		"created": "2022-03-23T10:31:02Z",
		"expiration": null,
		"secondsUntilExpiration": 0,
		"hasExpired": false
	}
]
```

## Create service account tokens

`POST /api/serviceaccounts/:id/tokens`

**Required permissions**

See note in the [introduction](#service-account-api) for an explanation.

| Action                | Scope                 |
| --------------------- | --------------------- |
| serviceaccounts:write | serviceaccounts:id:\* |

**Example Request**:

```http
POST /api/serviceaccounts/2/tokens HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=

{
	"name": "grafana",
	"secondsToLive": 604800
}
```

Default value for the `secondsToLive` is 0, which means that the service account token will never expire.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
	"id": 7,
	"name": "grafana",
	"key": "eyJrIjoiVjFxTHZ6dGdPSjg5Um92MjN1RlhjMkNqYkZUbm9jYkwiLCJuIjoiZ3JhZmFuYSIsImlkIjoxfQ=="
}
```

## Delete service account tokens

`DELETE /api/serviceaccounts/:id/tokens/:tokenId`

**Required permissions**

See note in the [introduction](#service-account-api) for an explanation.

| Action                | Scope                 |
| --------------------- | --------------------- |
| serviceaccounts:write | serviceaccounts:id:\* |

**Example Request**:

```http
DELETE /api/serviceaccounts/2/tokens/1 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
	"message": "API key deleted"
}
```

# Short URL API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< admonition type="note" >}}
Grafana announced the deprecation of this API on September 23, 2026. Use the `shorturl.grafana.app/v1beta1` API group instead:

- **Create a short URL:** `POST /apis/shorturl.grafana.app/v1beta1/namespaces/{namespace}/shorturls`
- **Retrieve a short URL:** `GET /apis/shorturl.grafana.app/v1beta1/namespaces/{namespace}/shorturls/{name}`

The new API uses different request and response shapes than the endpoints on this page, so changing the URL alone isn't enough. Refer to the [Short URL API Swagger documentation](https://play.grafana.org/swagger?api=shorturl.grafana.app-v1beta1) for the schemas, and to [API structure in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis/) for how to determine the `{namespace}` for your instance.
{{< /admonition >}}

Use this API to create shortened URLs. A short URL represents a longer URL containing complex query parameters in a smaller and simpler format.

## Create short URL

`POST /api/short-urls`

Creates a short URL.

**Example request:**

```http
POST /api/short-urls HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "path": "d/TxKARsmGz/new-dashboard?orgId=1&from=1599389322894&to=1599410922894"
}
```

JSON body schema:

- **path** – The path to shorten, relative to the Grafana [root_url](/docs/grafana/latest/setup-grafana/configure-grafana/#root_url).

**Example response:**

```http
HTTP/1.1 200
Content-Type: application/json

{
  "uid": AT76wBvGk,
  "url": http://localhost:3000/goto/AT76wBvGk?orgId=1
}

```

Status codes:

- **200** – Created
- **400** – Errors (invalid JSON, missing or invalid fields)

# Snapshot API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Create new snapshot

`POST /api/snapshots`

**Example Request**:

```http
    POST /api/snapshots HTTP/1.1
    Accept: application/json
    Content-Type: application/json
    Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

    {
      "dashboard": {
        "editable":false,
        "nav":[
        {
          "enable":false,
        "type":"timepicker"
        }
        ],
        "rows": [
          {

          }
        ],
        "style":"dark",
        "tags":[],
        "templating":{
          "list":[
          ]
        },
        "time":{
        },
        "timezone":"browser",
        "title":"Home",
        "version":5
        },
      "expires": 3600
    }
```

JSON Body schema:

- **dashboard** – Required. The complete dashboard model.
- **name** – Optional. snapshot name
- **expires** - Optional. When the snapshot should expire in seconds. 3600 is 1 hour, 86400 is 1 day. Default is never to expire.
- **external** - Optional. Save the snapshot on an external server rather than locally. Default is `false`.
- **key** - Optional. Define the unique key. Required if **external** is `true`.
- **deleteKey** - Optional. Unique key used to delete the snapshot. It is different from the **key** so that only the creator can delete the snapshot. Required if **external** is `true`.

{{< admonition type="note" >}}
When creating a snapshot using the API, you have to provide the full dashboard payload including the snapshot data. This endpoint is designed for the Grafana UI.
{{< /admonition >}}

**Example Response**:

```http
    HTTP/1.1 200
    Content-Type: application/json
    {
      "deleteKey":"XXXXXXX",
      "deleteUrl":"myurl/api/snapshots-delete/XXXXXXX",
      "key":"YYYYYYY",
      "url":"myurl/dashboard/snapshot/YYYYYYY",
      "id": 1
    }
```

Keys:

- **deleteKey** – Key generated to delete the snapshot
- **key** – Key generated to share the dashboard

## Get list of Snapshots

`GET /api/dashboard/snapshots`

Query parameters:

- **query** – Search Query
- **limit** – Limit the number of returned results. Default is 1000 is not set or provided an invalid value.

**Example Request**:

```http
GET /api/dashboard/snapshots HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "id":8,
    "name":"Home",
    "key":"YYYYYYY",
    "orgId":1,
    "userId":1,
    "external":false,
    "externalUrl":"",
    "expires":"2200-13-32T25:23:23+02:00",
    "created":"2200-13-32T28:24:23+02:00",
    "updated":"2200-13-32T28:24:23+02:00"
  }
]
```

## Get Snapshot by Key

`GET /api/snapshots/:key`

**Example Request**:

```http
GET /api/snapshots/YYYYYYY HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "meta":{
    "isSnapshot":true,
    "type":"snapshot",
    "canSave":false,
    "canEdit":false,
    "canStar":false,
    "slug":"",
    "expires":"2200-13-32T25:23:23+02:00",
    "created":"2200-13-32T28:24:23+02:00"
  },
  "dashboard": {
    "editable":false,
    "nav": [
      {
        "enable":false,
        "type":"timepicker"
      }
    ],
    "rows": [
      {

      }
    ],
    "style":"dark",
    "tags":[],
    "templating":{
      "list":[
      ]
    },
    "time":{
    },
    "timezone":"browser",
    "title":"Home",
    "version":5
  }
}
```

## Delete Snapshot by Key

`DELETE /api/snapshots/:key`

**Example Request**:

```http
DELETE /api/snapshots/YYYYYYY HTTP/1.1
Accept: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Snapshot deleted. It might take an hour before it's cleared from any CDN caches.", "id": 1}
```

## Delete Snapshot by deleteKey

This API call can be used without authentication by using the secret delete key for the snapshot.

`GET /api/snapshots-delete/:deleteKey`

**Example Request**:

```http
GET /api/snapshots-delete/XXXXXXX HTTP/1.1
Accept: application/json
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Snapshot deleted. It might take an hour before it's cleared from any CDN caches.", "id": 1}
```

# SSO Settings API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

> If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

The API can be used to create, update, delete, get, and list SSO Settings for OAuth2 and SAML.

The settings managed by this API are stored in the database and override
[settings from other sources](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/)
(arguments, environment variables, settings file, etc).
Therefore, every time settings for a specific provider are removed or reset to the default settings at runtime,
the settings are inherited from the other sources in the reverse order of precedence
(`arguments > environment variables > settings file`).

The `source` field on the `GET` endpoints is `database` when this API stores settings for the provider, and `system` otherwise. Stored settings keep overriding the configuration file until you remove them, which most often surprises people during credential rotation.

Use `DELETE` to discard a provider's stored settings. `PUT` replaces the whole record, so keys you leave out are deleted, and `PATCH` can't remove a key. A `system` source doesn't rule out a value stored by [settings updates at runtime](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/settings-updates-at-runtime/); refer to [Check for stored settings](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/settings-updates-at-runtime/#check-for-stored-settings).

## List SSO Settings

`GET /api/v1/sso-settings`

Lists the SSO Settings for all providers.

The providers or SSO keys that are not managed by this API are retrieved from the other sources (settings file, environment variables, default values).

**Required permissions**

See note in the [introduction](#sso-settings) for an explanation.

| Action          | Scope                        |
| --------------- | ---------------------------- |
| `settings:read` | `settings:auth.{provider}:*` |

**Example Request**:

```http
GET /api/v1/sso-settings HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json
[
  {
    "id":        "1",
    "provider":  "github",
    "settings": {
      "apiUrl": "https://api.github.com/user",
      "clientId": "my_github_client",
      "clientSecret": "*********",
      "enabled": true,
      "scopes": "user:email,read:org"
      // rest of the settings
    },
    "source":    "system",
  },
  {
    "id":        "2",
    "provider":  "azuread",
    "settings": {
      "authUrl": "https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/v2.0/authorize",
      "clientId": "my_azuread_client",
      "clientSecret": "*********",
      "enabled": true,
      "scopes": "openid,email,profile"
      // rest of the settings
    },
    "source":    "system",
  }
]
```

Status Codes:

- **200** – SSO Settings found
- **400** – Bad Request
- **401** – Unauthorized
- **403** – Access Denied

## Get SSO Settings

`GET /api/v1/sso-settings/:provider`

Gets the SSO Settings for a provider.

The SSO keys that are not managed by this API are retrieved from the other sources (settings file, environment variables, default values).

**Required permissions**

See note in the [introduction](#sso-settings) for an explanation.

| Action          | Scope                        |
| --------------- | ---------------------------- |
| `settings:read` | `settings:auth.{provider}:*` |

**Example Request**:

```http
GET /api/v1/sso-settings/github HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json
ETag: db87f729761898ee
{
  "id":        "1",
  "provider":  "github",
  "settings": {
    "apiUrl": "https://api.github.com/user",
    "clientId": "my_github_client",
    "clientSecret": "*********",
    "enabled": true,
    "scopes": "user:email,read:org"
    // rest of the settings
  },
  "source":    "system",
}
```

Status Codes:

- **200** – SSO Settings found
- **400** – Bad Request
- **401** – Unauthorized
- **403** – Access Denied
- **404** – SSO Settings not found

## Update SSO Settings

`PUT /api/v1/sso-settings/:provider`

Updates the SSO Settings for a provider.

When you submit new settings for a provider via API,
Grafana verifies whether the given settings are allowed and valid.
If they are, then Grafana stores the settings in the database and reloads
Grafana services with no need to restart the instance.

{{< admonition type="note" >}}
If you run Grafana in high availability mode, configuration changes
may not get applied to all Grafana instances immediately. You may need
to wait a few minutes for the configuration to propagate to all Grafana instances.
{{< /admonition >}}

**Required permissions**

See note in the [introduction](#sso-settings) for an explanation.

| Action           | Scope                        |
| ---------------- | ---------------------------- |
| `settings:write` | `settings:auth.{provider}:*` |

**Example Request**:

```http
PUT /api/v1/sso-settings/github HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "settings": {
    "apiUrl": "https://api.github.com/user",
    "clientId": "my_github_client",
    "clientSecret": "my_github_secret",
    "enabled": true,
    "scopes": "user:email,read:org"
  }
}
```

**Example Response**:

```http
HTTP/1.1 204
Content-Type: application/json
```

Status Codes:

- **204** – SSO Settings updated
- **400** – Bad Request
- **401** – Unauthorized
- **403** – Access Denied

## Delete SSO Settings

`DELETE /api/v1/sso-settings/:provider`

Deletes an existing SSO Settings entry for a provider.

**Required permissions**

See note in the [introduction](#sso-settings) for an explanation.

| Action           | Scope                        |
| ---------------- | ---------------------------- |
| `settings:write` | `settings:auth.{provider}:*` |

**Example Request**:

```http
DELETE /api/v1/sso-settings/azuread HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 204
Content-Type: application/json
```

Status Codes:

- **204** – SSO Settings deleted
- **400** – Bad Request
- **401** – Unauthorized
- **403** – Access Denied
- **404** – SSO Settings not found

# Team API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

This API can be used to manage Teams and Team Memberships.

Access to these API endpoints is restricted as follows:

- All authenticated users are able to view details of teams they are a member of.
- Organization Admins are able to manage all teams and team members.

> If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## Team Search With Paging

`GET /api/teams/search?perpage=50&page=1&query=myteam&sort=memberCount-desc`

or

`GET /api/teams/search?name=myteam`

**Required permissions**

See note in the [introduction](#team-api) for an explanation.

| Action     | Scope    |
| ---------- | -------- |
| teams:read | teams:\* |

**Example Request**:

```http
GET /api/teams/search?perpage=10&page=1&query=mytestteam HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "totalCount": 1,
  "teams": [
    {
      "id": 1,
      "orgId": 1,
      "name": "MyTestTeam",
      "email": "",
      "avatarUrl": "\/avatar\/3f49c15916554246daa714b9bd0ee398",
      "memberCount": 1
    }
  ],
  "page": 1,
  "perPage": 1000
}
```

### Using the query parameter

Default value for the `perpage` parameter is `1000` and for the `page` parameter is `1`.

The `totalCount` field in the response can be used for pagination of the teams list E.g. if `totalCount` is equal to 100 teams and the `perpage` parameter is set to 10 then there are 10 pages of teams.

The `query` parameter is optional and it will return results where the query value is contained in the `name` field. Query values with spaces need to be URL encoded e.g. `query=my%20team`.

The `sort` param is an optional comma separated list of options to order the search result. Accepted values for the sort filter are: ` name-asc`, `name-desc`, `email-asc`, `email-desc`, `memberCount-asc`, `memberCount-desc`. By default, if `sort` is not specified, the teams list will be ordered by `name` in ascending order.

### Using the name parameter

The `name` parameter returns a single team if the parameter matches the `name` field.

#### Status Codes:

- **200** - Ok
- **400** - Bad Request
- **401** - Unauthorized
- **403** - Permission denied
- **404** - Team not found (if searching by name)

## Get Team By Id

`GET /api/teams/:id`

**Required permissions**

See note in the [introduction](#team-api) for an explanation.

| Action     | Scope    |
| ---------- | -------- |
| teams:read | teams:\* |

**Example Request**:

```http
GET /api/teams/1 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "id": 1,
  "orgId": 1,
  "name": "MyTestTeam",
  "email": "",
  "created": "2017-12-15T10:40:45+01:00",
  "updated": "2017-12-15T10:40:45+01:00"
}
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Permission denied
- **404** - Team not found

## Add Team

The Team `name` needs to be unique. `name` is required and `email` is optional.

`POST /api/teams`

**Required permissions**

See note in the [introduction](#team-api) for an explanation.

| Action       | Scope |
| ------------ | ----- |
| teams:create | N/A   |

**Example Request**:

```http
POST /api/teams HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "name": "MyTestTeam",
  "email": "email@test.com",
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Team created","teamId":2,"uid":"ceaulqadfoav4e"}
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Permission denied
- **409** - Team name is taken

## Update Team

There are two fields that can be updated for a team: `name` and `email`.

`PUT /api/teams/:id`

**Required permissions**

See note in the [introduction](#team-api) for an explanation.

| Action      | Scope    |
| ----------- | -------- |
| teams:write | teams:\* |

**Example Request**:

```http
PUT /api/teams/2 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "name": "MyTestTeam",
  "email": "email@test.com"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Team updated"}
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Permission denied
- **404** - Team not found
- **409** - Team name is taken

## Delete Team By Id

`DELETE /api/teams/:id`

**Required permissions**

See note in the [introduction](#team-api) for an explanation.

| Action       | Scope    |
| ------------ | -------- |
| teams:delete | teams:\* |

**Example Request**:

```http
DELETE /api/teams/2 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Team deleted"}
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Permission denied
- **404** - Failed to delete Team. ID not found

## Get Team Members

`GET /api/teams/:teamId/members`

**Required permissions**

See note in the [introduction](#team-api) for an explanation.

| Action                 | Scope    |
| ---------------------- | -------- |
| teams.permissions:read | teams:\* |

**Example Request**:

```http
GET /api/teams/1/members HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "orgId": 1,
    "teamId": 1,
    "userId": 3,
    "email": "user1@example.com",
    "login": "user1",
    "avatarUrl": "\/avatar\/1b3c32f6386b0185c40d359cdc733a79"
  },
  {
    "orgId": 1,
    "teamId": 1,
    "userId": 2,
    "email": "user2@example.com",
    "login": "user2",
    "avatarUrl": "\/avatar\/cad3c68da76e45d10269e8ef02f8e73e"
  }
]
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Permission denied

## Add Team Member

`POST /api/teams/:teamId/members`

**Required permissions**

See note in the [introduction](#team-api) for an explanation.

| Action                  | Scope    |
| ----------------------- | -------- |
| teams.permissions:write | teams:\* |

**Example Request**:

```http
POST /api/teams/1/members HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "userId": 2
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Member added to Team"}
```

Status Codes:

- **200** - Ok
- **400** - User is already added to this team
- **401** - Unauthorized
- **403** - Permission denied
- **404** - Team not found

## Remove Member From Team

`DELETE /api/teams/:teamId/members/:userId`

**Required permissions**

See note in the [introduction](#team-api) for an explanation.

| Action                  | Scope    |
| ----------------------- | -------- |
| teams.permissions:write | teams:\* |

**Example Request**:

```http
DELETE /api/teams/2/members/3 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Team Member removed"}
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Permission denied
- **404** - Team not found/Team member not found

## Bulk Update Team Members

Allows bulk updating team members and administrators using user emails.
Will override all current members and administrators for the specified team.

`PUT /api/teams/:teamId/members

**Required permissions**

See note in the [introduction](#team-api) for an explanation.

| Action                  | Scope    |
| ----------------------- | -------- |
| teams.permissions:write | teams:\* |

**Example Request**:

```http
PUT /api/teams/1/members HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "members": ["user1@example.com", "user2@example.com"]
  "admins": ["user3@example.com"]
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Team memberships have been updated"}
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Permission denied
- **404** - Team not found/Team member not found
- **500** - Internal error

## Get Team Preferences

`GET /api/teams/:teamId/preferences`

**Required permissions**

See note in the [introduction](#team-api) for an explanation.

| Action     | Scope    |
| ---------- | -------- |
| teams:read | teams:\* |

**Example Request**:

```http
GET /api/teams/2/preferences HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "theme": "",
  "homeDashboardId": 0,
  "homeDashboardUID": "",
  "timezone": ""
}
```

## Update Team Preferences

`PUT /api/teams/:teamId/preferences`

**Required permissions**

See note in the [introduction](#team-api) for an explanation.

| Action      | Scope    |
| ----------- | -------- |
| teams:write | teams:\* |

**Example Request**:

```http
PUT /api/teams/2/preferences HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "theme": "dark",
  "homeDashboardId": 39,
  "homeDashboardUID": "jcIIG-07z",
  "timezone": "utc"
}
```

JSON Body Schema:

- **theme** - One of: `light`, `dark`, or an empty string for the default theme
- **homeDashboardId** - Deprecated. Use `homeDashboardUID` instead.
- **homeDashboardUID** - The `:uid` of a dashboard, or the reserved value `global-home` to show the instance default home instead of the organization's dashboard. An empty string clears the preference so the organization's dashboard applies.
- **timezone** - One of: `utc`, `browser`, or an empty string for the default

Omitting a key will cause the current value to be replaced with the system default value.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: text/plain; charset=utf-8

{
  "message":"Preferences updated"
}
```

# Team Sync API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

> Team Sync is only available in Grafana Enterprise. Read more about [Grafana Enterprise](/docs/grafana/latest/introduction/grafana-enterprise/).

> If you are running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](/docs/grafana/latest/administration/roles-and-permissions/access-control/custom-role-actions-scopes/) for more information.

## Get External Groups

`GET /api/teams/:teamId/groups`

**Required permissions**

See note in the [introduction](#external-group-synchronization-api) for an explanation.

| Action                 | Scope    |
| ---------------------- | -------- |
| teams.permissions:read | teams:\* |

**Example Request**:

```http
GET /api/teams/1/groups HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "orgId": 1,
    "teamId": 1,
    "groupId": "cn=editors,ou=groups,dc=grafana,dc=org"
  }
]
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Permission denied

## Add External Group

`POST /api/teams/:teamId/groups`

**Required permissions**

See note in the [introduction](#external-group-synchronization-api) for an explanation.

| Action                  | Scope    |
| ----------------------- | -------- |
| teams.permissions:write | teams:\* |

**Example Request**:

```http
POST /api/teams/1/groups HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer YWRtaW46YWRtaW4=

{
  "groupId": "cn=editors,ou=groups,dc=grafana,dc=org"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Group added to Team"}
```

Status Codes:

- **200** - Ok
- **400** - Group is already added to this team
- **401** - Unauthorized
- **403** - Permission denied
- **404** - Team not found

## Remove External Group

`DELETE /api/teams/:teamId/groups?groupId=external-group-id`

**Required permissions**

See note in the [introduction](#external-group-synchronization-api) for an explanation.

| Action                  | Scope    |
| ----------------------- | -------- |
| teams.permissions:write | teams:\* |

**Example Request**:

```http
DELETE /api/teams/1/groups?groupId=cn%3Deditors%2Cou%3Dgroups%2Cdc%3Dgrafana%2Cdc%3Dorg HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Team Group removed"}
```

Status Codes:

- **200** - Ok
- **401** - Unauthorized
- **403** - Permission denied
- **404** - Team not found/Group not found

## Search Team Groups

`GET /api/teams/:teamId/groups/search`

Search for team groups with pagination support.

**Required permissions**

| Action                 | Scope    |
| ---------------------- | -------- |
| teams.permissions:read | teams:\* |

**Example Request**:

```http
GET /api/teams/1/groups/search?name=editors&query=group&page=1&perpage=10 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "totalCount": 1,
    "teamGroups": [
      {
        "orgId": 1,
        "teamId": 1,
        "groupId": "cn=editors,ou=groups,dc=grafana,dc=org"
      }
    ],
    "page": 1,
    "perPage": 10
  }
]
```

Status Codes:

- **200** - Ok
- **400** - Bad Request (invalid team ID format or missing query parameter)
- **401** - Unauthorized
- **403** - Permission denied
- **500** - Internal Server Error

# User API

{{< docs/shared lookup="developers/deprecated-apis.md" source="grafana" version="<GRAFANA_VERSION>" >}}

The User API is a special, instance-wide endpoint that lists users across every organization on all its Grafana instances.

**This API does not work in Grafana Cloud**.

## Permissions

To use these API endpoints you must have the Grafana server administrator permissions.

In Grafana OSS, the default `admin` user has permissions to use these API endpoints. If you're running Grafana Enterprise, for some endpoints you'll need to have specific permissions. Refer to [Role-based access control permissions](ref:role-based-access-control-permissions) for more information.

Because Grafana Cloud stacks are multi-tenant, **no Grafana Cloud account is allowed to call these endpoints**. That level of access is reserved only for Grafana platform operators. If you're a Grafana Cloud user, refer to the [Organization HTTP API](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/http_api/api-legacy/org/) instead.

## Authentication

**You need basic authentication to use this API**, you can't authenticate with service account tokens. Service accounts are limited to an organization and an organization role, and can't be granted [Grafana server administrator permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/#grafana-server-administrators).

Alternatively, you can use the [Organization HTTP API](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/org/#current-organization-api) with service account tokens to manage users in a specific organization

## Search users

`GET /api/users?perpage=10&page=1&sort=login-asc,email-asc`

**Required permissions**

See note in the [introduction](#user-api) for an explanation.

| Action     | Scope           |
| ---------- | --------------- |
| users:read | global.users:\* |

**Example Request**:

```http
GET /api/users HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

Default value for the `perpage` parameter is `1000` and for the `page` parameter is `1`. Requires basic authentication and that the authenticated user is a Grafana Admin.

The `sort` param is an optional comma separated list of options to order the search result. Accepted values for the sort filter are: `login-asc`, `login-desc`, `email-asc`, `email-desc`, `name-asc`, `name-desc`, `lastSeenAtAge-asc`, `lastSeenAtAge-desc`. By default, if `sort` is not specified, the user list will be ordered by `login`, `email` in ascending order.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "id": 1,
    "name": "Admin",
    "login": "admin",
    "email": "admin@mygraf.com",
    "isAdmin": true,
    "isDisabled": false,
    "lastSeenAt": "2020-04-10T20:29:27+03:00",
    "lastSeenAtAge": "2m",
    "authLabels": ["OAuth"]
  },
  {
    "id": 2,
    "name": "User",
    "login": "user",
    "email": "user@mygraf.com",
    "isAdmin": false,
    "isDisabled": false,
    "lastSeenAt": "2020-01-24T12:38:47+02:00",
    "lastSeenAtAge": "2M",
    "authLabels": []
  }
]
```

## Search users with paging

`GET /api/users/search?perpage=10&page=1&query=mygraf&sort=login-asc,email-asc`

**Required permissions**

See note in the [introduction](#user-api) for an explanation.

| Action     | Scope           |
| ---------- | --------------- |
| users:read | global.users:\* |

**Example Request**:

```http
GET /api/users/search?perpage=10&page=1&query=mygraf HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

Default value for the `perpage` parameter is `1000` and for the `page` parameter is `1`. The `totalCount` field in the response can be used for pagination of the user list E.g. if `totalCount` is equal to 100 users and the `perpage` parameter is set to 10 then there are 10 pages of users. The `query` parameter is optional and it will return results where the query value is contained in one of the `name`, `login` or `email` fields. Query values with spaces need to be URL encoded e.g. `query=Jane%20Doe`.

The `sort` param is an optional comma separated list of options to order the search result. Accepted values for the sort filter are: `login-asc`, `login-desc`, `email-asc`, `email-desc`, `name-asc`, `name-desc`, `lastSeenAtAge-asc`, `lastSeenAtAge-desc`. By default, if `sort` is not specified, the user list will be ordered by `login`, `email` in ascending order.

Requires basic authentication and that the authenticated user is a Grafana Admin.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json
{
  "totalCount": 2,
  "users": [
    {
      "id": 1,
      "name": "Admin",
      "login": "admin",
      "email": "admin@mygraf.com",
      "isAdmin": true,
      "isDisabled": false,
      "lastSeenAt": "2020-04-10T20:29:27+03:00",
      "lastSeenAtAge': "2m",
      "authLabels": ["OAuth"]
    },
    {
      "id": 2,
      "name": "User",
      "login": "user",
      "email": "user@mygraf.com",
      "isAdmin": false,
      "isDisabled": false,
      "lastSeenAt": "2020-01-24T12:38:47+02:00",
      "lastSeenAtAge": "2M",
      "authLabels": []
    }
  ],
  "page": 1,
  "perPage": 10
}
```

## Get single user by Id

`GET /api/users/:id`

**Required permissions**

See note in the [introduction](#user-api) for an explanation.

| Action     | Scope           |
| ---------- | --------------- |
| users:read | global.users:\* |

**Example Request**:

```http
GET /api/users/1 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

Requires basic authentication and that the authenticated user is a Grafana Admin.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "id": "1",
  "email": "user@mygraf.com",
  "name": "admin",
  "login": "admin",
  "theme": "light",
  "orgId": 1,
  "isGrafanaAdmin": true,
  "isDisabled": true,
  "isExternal": false,
  "authLabels": [],
  "updatedAt": "2019-09-09T11:31:26+01:00",
  "createdAt": "2019-09-09T11:31:26+01:00",
  "avatarUrl": ""
}
```

## Get single user by username (login) or email

`GET /api/users/lookup?loginOrEmail=user@mygraf.com`

**Required permissions**

See note in the [introduction](#user-api) for an explanation.

| Action     | Scope           |
| ---------- | --------------- |
| users:read | global.users:\* |

**Example Request using the email as option**:

```http
GET /api/users/lookup?loginOrEmail=user@mygraf.com HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

**Example Request using the username as option**:

```http
GET /api/users/lookup?loginOrEmail=admin HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

Requires basic authentication and that the authenticated user is a Grafana Admin.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "id": 1,
  "email": "user@mygraf.com",
  "name": "admin",
  "login": "admin",
  "theme": "light",
  "orgId": 1,
  "isGrafanaAdmin": true,
  "isDisabled": false,
  "isExternal": false,
  "authLabels": null,
  "updatedAt": "2019-09-25T14:44:37+01:00",
  "createdAt": "2019-09-25T14:44:37+01:00",
  "avatarUrl":""
}
```

## User update

`PUT /api/users/:id`

**Required permissions**

See note in the [introduction](#user-api) for an explanation.

| Action      | Scope           |
| ----------- | --------------- |
| users:write | global.users:\* |

**Example Request**:

```http
PUT /api/users/2 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=

{
  "email":"user@mygraf.com",
  "name":"User2",
  "login":"user",
  "theme":"light"
}
```

Requires basic authentication and that the authenticated user is a Grafana Admin.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"User updated"}
```

## Get organizations for user

`GET /api/users/:id/orgs`

**Required permissions**

See note in the [introduction](#user-api) for an explanation.

| Action     | Scope           |
| ---------- | --------------- |
| users:read | global.users:\* |

**Example Request**:

```http
GET /api/users/1/orgs HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

Requires basic authentication and that the authenticated user is a Grafana Admin.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "orgId":1,
    "name":"Main Org.",
    "role":"Admin"
  }
]
```

## Get teams for user

`GET /api/users/:id/teams`

**Required permissions**

See note in the [introduction](#user-api) for an explanation.

| Action     | Scope           |
| ---------- | --------------- |
| users:read | global.users:\* |
| teams:read | teams:\*        |

**Example Request**:

```http
GET /api/users/1/teams HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

Requires basic authentication and that the authenticated user is a Grafana Admin.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "id":1,
    "orgId":1,
    "name":"team1",
    "email":"",
    "avatarUrl":"/avatar/3fcfe295eae3bcb67a49349377428a66",
    "memberCount":1
  }
]
```

## Actual user

`GET /api/user`

**Example Request**:

```http
GET /api/user HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

Requires basic authentication.

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "id":1,
  "email":"admin@mygraf.com",
  "name":"Admin",
  "login":"admin",
  "theme":"light",
  "orgId":1,
  "isGrafanaAdmin":true,
  "isDisabled":false
  "isExternal": false,
  "authLabels": [],
  "updatedAt": "2019-09-09T11:31:26+01:00",
  "createdAt": "2019-09-09T11:31:26+01:00",
  "avatarUrl": ""
}
```

## Change password

`PUT /api/user/password`

Changes the password for the user. Requires basic authentication.

**Example Request**:

```http
PUT /api/user/password HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=

{
  "oldPassword": "old_password",
  "newPassword": "new_password"
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"User password changed"}
```

**Change Password with a Script**

If you need to change a password with a script, here is an example of changing the Admin password using curl with basic auth:

```bash
curl -X PUT -H "Content-Type: application/json" -d '{
  "oldPassword": "oldpass",
  "newPassword": "newpass",
  "confirmNew": "newpass"
}' http://admin:oldpass@<your_grafana_host>:3000/api/user/password
```

## Switch user context for a specified user

`POST /api/users/:userId/using/:organizationId`

Switch user context to the given organization. Requires basic authentication and that the authenticated user is a Grafana Admin.

**Example Request**:

```http
POST /api/users/7/using/2 HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Active organization changed"}
```

## Switch user context for signed in user

`POST /api/user/using/:organizationId`

Switch user context to the given organization.

**Example Request**:

```http
POST /api/user/using/2 HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Active organization changed"}
```

## Organizations of the actual user

`GET /api/user/orgs`

Return a list of all organizations of the current user. Requires basic authentication.

**Example Request**:

```http
GET /api/user/orgs HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "orgId":1,
    "name":"Main Org.",
    "role":"Admin"
  }
]
```

## Teams that the actual user is member of

`GET /api/user/teams`

Return a list of all teams that the current user is member of.

**Example Request**:

```http
GET /api/user/teams HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "id": 1,
    "orgId": 1,
    "name": "MyTestTeam",
    "email": "",
    "avatarUrl": "\/avatar\/3f49c15916554246daa714b9bd0ee398",
    "memberCount": 1
  }
]
```

## Star a dashboard

`POST /api/user/stars/dashboard/uid/:uid`

Stars the given Dashboard for the actual user.

**Example Request**:

```http
POST /api/user/stars/dashboard/uid/BqokFhx7z HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Dashboard starred!"}
```

## Unstar a dashboard

`DELETE /api/user/stars/dashboard/uid/:uid`

Deletes the starring of the given Dashboard for the actual user.

**Example Request**:

```http
DELETE /api/user/stars/dashboard/uid/BqokFhx7z HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{"message":"Dashboard unstarred"}
```

## Auth tokens of the actual user

`GET /api/user/auth-tokens`

Return a list of all auth tokens (devices) that the actual user currently have logged in from.

**Example Request**:

```http
GET /api/user/auth-tokens HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

[
  {
    "id": 361,
    "isActive": true,
    "clientIp": "127.0.0.1",
    "browser": "Chrome",
    "browserVersion": "72.0",
    "os": "Linux",
    "osVersion": "",
    "device": "Other",
    "createdAt": "2019-03-05T21:22:54+01:00",
    "seenAt": "2019-03-06T19:41:06+01:00"
  },
  {
    "id": 364,
    "isActive": false,
    "clientIp": "127.0.0.1",
    "browser": "Mobile Safari",
    "browserVersion": "11.0",
    "os": "iOS",
    "osVersion": "11.0",
    "device": "iPhone",
    "createdAt": "2019-03-06T19:41:19+01:00",
    "seenAt": "2019-03-06T19:41:21+01:00"
  }
]
```

## Revoke an auth token of the actual user

`POST /api/user/revoke-auth-token`

Revokes the given auth token (device) for the actual user. User of issued auth token (device) will no longer be logged in
and will be required to authenticate again upon next activity.

**Example Request**:

```http
POST /api/user/revoke-auth-token HTTP/1.1
Accept: application/json
Content-Type: application/json
Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>

{
  "authTokenId": 364
}
```

**Example Response**:

```http
HTTP/1.1 200
Content-Type: application/json

{
  "message": "User auth token revoked"
}
```

# Create Service Account tokens and dashboards for an organization

Use the Grafana API to set up new Grafana organizations or to add dynamically generated dashboards to an existing organization.

## Authentication

There are two authentication methods to access the API:

- Basic authentication: A Grafana Admin user can access some parts of the Grafana API through basic authentication.
- Service Account tokens: All organization actions can be accessed through a Service Account token. A Service Account token is associated with an organization. It can be used to create dashboards and other components specific for that organization.

## How to create a new organization and a Service Account Token

The task is to create a new organization and then add a Token that can be used by other users. In the examples below which use basic auth, the user is `admin` and the password is `admin`.

1. [Create the org](/docs/grafana/<GRAFANA_VERSION>/http_api/org/#create-organization). Here is an example using curl:

   ```bash
   curl -X POST -H "Content-Type: application/json" -d '{"name":"apiorg"}' http://admin:admin@localhost:3000/api/orgs
   ```

   This should return a response: `{"message":"Organization created","orgId":6}`. Use the orgId for the next steps.

1. Optional step. If the org was created previously and/or step 3 fails then first [add your Admin user to the org](/docs/grafana/<GRAFANA_VERSION>/http_api/org/#add-user-in-organization):

   ```bash
   curl -X POST -H "Content-Type: application/json" -d '{"loginOrEmail":"admin", "role": "Admin"}' http://admin:admin@localhost:3000/api/orgs/<org id of new org>/users
   ```

1. [Switch the org context for the Admin user to the new org](/docs/grafana/<GRAFANA_VERSION>/http_api/user/#switch-user-context-for-signed-in-user):

   ```bash
   curl -X POST http://admin:admin@localhost:3000/api/user/using/<id of new org>
   ```

1. [Create a Service Account](../../serviceaccount/#create-service-account):

   ```bash
   curl -X POST -H "Content-Type: application/json" -d '{"name":"test", "role": "Admin"}' http://admin:admin@localhost:3000/api/serviceaccounts
   ```

1. [Create a Service Account token](../../serviceaccount/#create-service-account-tokens) for the service account created in the previous step:

   ```bash
   curl -X POST -H "Content-Type: application/json" -d '{"name":"test-token"}' http://admin:admin@localhost:3000/api/serviceaccounts/<service account id>/tokens
   ```

   This should return a response:

   ```http
   HTTP/1.1 200
   Content-Type: application/json

   {
     "id": 7,
     "name": "test-token",
     "key": "eyJrIjoiVjFxTHZ6dGdPSjg5Um92MjN1RlhjMkNqYkZUbm9jYkwiLCJuIjoiZ3JhZmFuYSIsImlkIjoxfQ=="
   }
   ```

   Save the key returned here in your password manager as it is not possible to fetch again it in the future.

## How to add a dashboard

Using the Token that was created in the previous step, you can create a dashboard or carry out other actions without having to switch organizations.

1. [Add a dashboard](/docs/grafana/<GRAFANA_VERSION>/http_api/dashboard/#create-update-dashboard) using the key (or bearer token as it is also called):

```bash
curl -X POST --insecure -H "Authorization: Bearer eyJrIjoiR0ZXZmt1UFc0OEpIOGN5RWdUalBJTllUTk83VlhtVGwiLCJuIjoiYXBpa2V5Y3VybCIsImlkIjo2fQ==" -H "Content-Type: application/json" -d '{
  "dashboard": {
    "id": null,
    "title": "Production Overview",
    "tags": [ "templated" ],
    "timezone": "browser",
    "rows": [
      {
      }
    ],
    "schemaVersion": 6,
    "version": 0
  },
  "overwrite": false
}' http://localhost:3000/api/dashboards/db
```

> **Note:** If you export a dashboard for sharing externally using the Share > Export menu in the Grafana UI, you cannot import that dashboard. Instead, click **View JSON** and save it to a file or fetch the JSON output through the API.

# cURL examples

This page provides examples of calls to the Grafana API using cURL.

The most basic example for a dashboard for which there is no authentication. You can test the following on your local machine, assuming a default installation and anonymous access enabled, required:

```
curl http://localhost:3000/api/search
```

Here's a cURL command that works for getting the home dashboard when you are running Grafana locally with [basic authentication](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/#basic-auth) enabled using the default admin credentials:

```
curl http://admin:admin@localhost:3000/api/search
```

To pass a username and password with [HTTP basic authorization](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/manage-rbac-roles/), encode them as base64.
You can't use authorization tokens in the request.

For example, to [list permissions associated with roles](/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/manage-rbac-roles/) given a username of `user` and password of `password`, use:

```
curl --location '<grafana_url>/api/access-control/builtin-roles' --user 'user:password'
```

# Grafana Labs Software Grant and Contributor License Agreement ("Agreement")

This agreement is based on the Apache Software Foundation Contributor License Agreement.
(v r190612)

Thank you for your interest in software projects stewarded by Raintank, Inc. dba Grafana Labs (“Grafana Labs”). In order to clarify the intellectual property license
granted with Contributions from any person or entity, Grafana Labs
must have a Contributor License Agreement (CLA) on file that has been
agreed to by each Contributor, indicating agreement to the license terms
below. This license is for your protection as a Contributor as well
as the protection of Grafana Labs and its users; it does not change
your rights to use your own Contributions for any other purpose.
This Agreement allows an individual to contribute to Grafana Labs on that individual’s own behalf, or an entity (the "Corporation") to
submit Contributions to Grafana Labs, to authorize Contributions
submitted by its designated employees to Grafana Labs, and to grant
copyright and patent licenses thereto.

You accept and agree to the following terms and conditions for Your
present and future Contributions submitted to Grafana Labs. Except
for the license granted herein to Grafana Labs and recipients of
software distributed by Grafana Labs, You reserve all right, title,
and interest in and to Your Contributions.

1. Definitions. "You" (or "Your") shall mean the copyright owner or legal entity
   authorized by the copyright owner that is making this Agreement
   with Grafana Labs. For legal entities, the entity making a
   Contribution and all other entities that control, are controlled by,
   or are under common control with that entity are considered to be a
   single Contributor. For the purposes of this definition, "control"
   means (i) the power, direct or indirect, to cause the direction or
   management of such entity, whether by contract or otherwise, or
   (ii) ownership of fifty percent (50%) or more of the outstanding
   shares, or (iii) beneficial ownership of such entity.
   "Contribution" shall mean any work, as well as
   any modifications or additions to an existing work, that is intentionally
   submitted by You to Grafana Labs for inclusion in, or
   documentation of, any of the products owned or managed by Grafana Labs (the "Work"). For the purposes of this definition,
   "submitted" means any form of electronic, verbal, or written
   communication sent to Grafana Labs or its representatives,
   including but not limited to communication on electronic mailing
   lists, source code control systems (such as GitHub), and issue tracking systems
   that are managed by, or on behalf of, Grafana Labs for the
   purpose of discussing and improving the Work, but excluding
   communication that is conspicuously marked or otherwise designated
   in writing by You as "Not a Contribution."

1. Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to Grafana Labs and to
   recipients of software distributed by Grafana Labs a perpetual,
   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
   copyright license to reproduce, prepare derivative works of,
   publicly display, publicly perform, sublicense, and distribute
   Your Contributions and such derivative works.

1. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to Grafana Labs and to recipients
   of software distributed by Grafana Labs a perpetual, worldwide,
   non-exclusive, no-charge, royalty-free, irrevocable (except as
   stated in this section) patent license to make, have made, use,
   offer to sell, sell, import, and otherwise transfer the Work,
   where such license applies only to those patent claims licensable
   by You that are necessarily infringed by Your Contribution(s)
   alone or by combination of Your Contribution(s) with the Work to
   which such Contribution(s) were submitted. If any entity institutes
   patent litigation against You or any other entity (including a
   cross-claim or counterclaim in a lawsuit) alleging that your
   Contribution, or the Work to which you have contributed, constitutes
   direct or contributory patent infringement, then any patent licenses
   granted to that entity under this Agreement for that Contribution or
   Work shall terminate as of the date such litigation is filed.

1. You represent that You are legally entitled to grant the above license. If You are an individual, and if Your employer(s) has rights to intellectual property
   that you create that includes Your Contributions, you represent
   that You have received permission to make Contributions on behalf
   of that employer, or that Your employer has waived such rights for
   your Contributions to Grafana Labs. If You are a Corporation, any individual who makes a contribution from an account associated with You will be considered authorized to Contribute on Your behalf.

1. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others).

1. You are not expected to provide support for Your Contributions,except to the extent You desire to provide support. You may provide
   support for free, for a fee, or not at all. Unless required by
   applicable law or agreed to in writing, You provide Your
   Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
   OF ANY KIND, either express or implied, including, without
   limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT,
   MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.

1. Should You wish to submit work that is not Your original creation, You may submit it to Grafana Labs separately from any
   Contribution, identifying the complete details of its source and
   of any license or other restriction (including, but not limited
   to, related patents, trademarks, and license agreements) of which
   you are personally aware, and conspicuously marking the work as
   "Submitted on behalf of a third-party: [named here]".

# Fundamentals

# Grafana dashboards overview

Have you ever wondered what a dashboard is? In the observability world, this term is frequently used, but what exactly does it mean? The concept is borrowed from automobiles, where a dashboard gives drivers access to the controls necessary to operate a vehicle. Similarly, digital dashboards help us comprehend and manage systems. This topic explains how Grafana dashboards function, enabling you to create your own with greater ease.

The following image illustrates a sample Grafana dashboard:

{{< figure src="/media/docs/grafana/dashboards-overview/complex-dashboard-example.png" max-width="750px" caption="Example Grafana dashboard" >}}

A Grafana dashboard consists of panels displaying data in beautiful graphs, charts, and other visualizations. These panels are created using components that transform raw data from a data source into visualizations. The process involves passing data through three gates: a plugin, a query, and an optional transformation.

The image below displays all the gates, followed by detailed explanations of their purpose, usage, and significance.

{{< figure src="/media/docs/grafana/dashboards-overview/dashboard-component-architecture.png" max-width="750px" caption="Dashboard component architecture" >}}

## Data sources

A data source refers to any entity that consists of data. It can be an SQL database, Grafana Loki, Grafana Mimir, or a JSON-based API. It can even be a basic CSV file. The first step in creating a dashboard visualization is selecting the data source that contains the data you need.

It can be difficult to understand the distinctions between different data sources as each possesses its own structure and requires different query methods. However, in dashboards, you can see different data sources visualized in one single view, making it easier to understand your data overall.

## Plugins

A Grafana plugin is software that adds new capabilities to Grafana. They come in many types, but right now we'll address _data source plugins_. The job of a Grafana data source plugin is to take a query you want answered, retrieve the data from the data source, and reconcile the differences between the data model of the data source and the data model of Grafana dashboards. It does this using a unified data structure called a [data frame](https://grafana.com/developers/plugin-tools/key-concepts/data-frames).

The data coming into the plugin from the data source might be in many different formats (such as JSON, rows and columns, or CSV), but when it leaves the plugin and moves through the rest of the gates toward a visualization, it's always in data frames.

Currently, Grafana offers a diverse range of 155 data sources that you can use. The most commonly used options are already pre-installed and accessible. Before exploring other options, look for an existing data source that matches your requirements. Grafana constantly updates the list, but if you don't find a suitable data source, you can browse through the [plugin catalog](/grafana/plugins/?type=datasource) or [create a plugin](/developers/plugin-tools).

## Queries

Queries allow you to reduce the entirety of your data to a specific dataset, providing a more manageable visualization. They help answer questions you have about system and operational processes. For instance, a company with an online store might want to determine the number of customers who add products to their shopping carts. This can be achieved through a query that aggregates access metrics for the shopping cart service, revealing the number of users accessing the service per second.

When working with data sources, it's crucial to recognize that each one has its own distinct query language. For example, Prometheus data sources utilize [PromQL](/blog/2020/02/04/introduction-to-promql-the-prometheus-query-language/), while [LogQL](/docs/loki/latest/logql/) is used for logs, and particular databases employ SQL. A query is the foundation of every visualization in Grafana, and a dashboard might use a range of query languages.

The following image shows the Query Editor associated with the Prometheus data source. The `node_cpu_seconds_total` query is written in PromQL and requests just one metric:

{{< figure src="/media/docs/grafana/dashboards-overview/example-query.png" max-width="750px" caption="Example query" >}}

## Transformations

When the data format in a visualization doesn’t meet your requirements, you can apply a [transformation](ref:transform-data) that manipulates the data returned by a query.
You might not need to transform data when you're first starting out, but they're powerful and worth mentioning.

Transforming data is useful in the following kinds of situations:

- You want to combine two fields together, for example, concatenating `Given Name` and `Family Name` into a `Full Name` field.
- You have CSV data (all text), and you want to convert a field type (such as parsing a date or a number out of a string).
- You want to filter, join, merge, or perform other SQL-like operations that might not be supported by the underlying data source or query language.

Transformations are located in the **Transform** tab in the edit dialog for a panel. Select the transformation you want, and define the transformation. The following image shows that you can have as many transformations as you want, just like queries. For example, you can chain together a series of transformations that make a change to a data type, filter results, organize columns, and sort the result into one data pipeline. Every time the dashboard is refreshed, the transformation applies to the latest data from the data source.

The following image shows the transformation dialog:

{{< figure src="/media/docs/grafana/dashboards-overview/example-transform-chain.png" max-width="750px" caption="Example chain of transformations" >}}

## Panels

After the data is sourced, queried, and transformed, it passes to a panel, which is the final gate in the journey to a Grafana visualization. A panel is a container that displays the visualization and provides you with various controls to manipulate it. The panel configuration is where you specify how you want to see the data. For example, you use a drop-down menu in the top-right of the panel to specify the type of visualization you want to see, such as a bar chart, pie chart, or histogram.

The panel options let you customize many aspects of the visualization and the options differ based on which visualization you select. Panels also contain queries that specify the data the panel is visualizing.

The following image shows a table panel being edited, the panel settings showing the query at the bottom, and the panel options on the right. In this image, you can see how the data source, plugin, query, and panel all come together.

{{< figure src="/media/docs/grafana/dashboards-overview/example-table-panel.png" max-width="750px" caption="Example table panel" >}}

Selecting the best visualization depends on the data and how you want it presented. To see examples of dashboards in one place that you can browse and inspect, refer to [Grafana Play](https://play.grafana.org/), which has feature showcases and a variety of examples.

## Conclusion

With the data source, plugin, query, transformation, and panel model in mind, you can now see right through any Grafana dashboard you encounter, and imagine how to build your own.

Building a Grafana dashboard is a process that starts with determining your dashboard requirements and identifying which data sources support those requirements. If you want to integrate a specialized database with a Grafana dashboard, you must ensure the correct plugin is installed so that you can add a data source to use with that plugin.

And with the data source identified and the plugin installed, you can write your query, transform the data, and format the visualization to meet your needs.

This component architecture is part of what makes Grafana so powerful and general. Given the data source plugin and data frame abstraction, any data source you can access can work with the same general approach.

# Introduction to exemplars

An exemplar is a specific trace representative of measurement taken in a given time interval.
While metrics excel at giving you an aggregated view of your system, traces give you a fine grained view of a single request; exemplars are a way to link the two.

Suppose your company website is experiencing a surge in traffic volumes.
While more than eighty percent of the users are able to access the website in under two seconds, some users are experiencing a higher than normal response time resulting in bad user experience.

To identify the factors that are contributing to the latency, you must compare a trace for a fast response against a trace for a slow response.
Given the vast amount of data in a typical production environment, it's an extremely laborious and time-consuming effort.

Use exemplars to help isolate problems within your data distribution by pinpointing query traces exhibiting high latency within a time interval.
After you localize the latency problem to a few exemplar traces, you can combine it with additional system based information or location properties to perform a root cause analysis faster, leading to quick resolutions to performance issues.

Support for exemplars is available for the Prometheus data source only.
After you enable the functionality, exemplar data is available by default.
For more information on exemplar configuration and how to enable exemplars, refer to the Exemplars section in [Prometheus configuration options](https://grafana.com/docs/grafana/latest/datasources/prometheus/configure/#configuration-options).

Grafana shows exemplars alongside a metric in the Explore view and in dashboards.
Each exemplar displays as a highlighted star.
You can hover your cursor over an exemplar to view the unique trace ID, which is a combination of a key value pair.
To investigate further, click the blue button next to the `traceID` property.

{{< figure src="/media/docs/grafana/exemplars/screenshot-exemplars.png" class="docs-image--no-shadow" max-width= "750px" caption="Screenshot showing the detail window of an exemplar" >}}

Refer to [View exemplar data](#view-exemplar-data) for instructions on how to drill down and view exemplar trace details from metrics and logs.
To know more about exemplars, refer to the blog post [Intro to exemplars, which enable Grafana Tempo’s distributed tracing at massive scale](/blog/2021/03/31/intro-to-exemplars-which-enable-grafana-tempos-distributed-tracing-at-massive-scale/).

## View exemplar data

When support for exemplar support is enabled for a Prometheus data source, you can view exemplar data either in the Explore view or from the Loki log details.

### In Explore

Explore visualizes exemplar traces as highlighted stars alongside metrics data.
For more information on how Explore visualizes trace data, refer to [Tracing in Explore](../../explore/trace-integration/).

To examine the details of an exemplar trace:

1. Place your cursor over an exemplar (highlighted star).
   Depending on the trace data source you are using, you may see a blue button with the label `Query with <DATA SOURCE NAME>`.
   In the following example, the tracing data source is Tempo.

   {{< figure src="/media/docs/grafana/exemplars/screenshot-exemplar-details.png" class="docs-image--no-shadow" max-width= "350px" caption="Screenshot showing exemplar details" >}}

1. Click the **Query with Tempo** option next to the `traceID` property.
   The trace details, including the spans within the trace are listed in a separate panel on the right.

   {{< figure src="/media/docs/grafana/exemplars/screenshot-exemplar-explore-view.png" class="docs-image--no-shadow" max-width= "900px" caption="Explorer view with panel showing trace details" >}}

For more information on how to drill down and analyze the trace and span details, refer to the [Analyze trace and span details](#analyze-trace-and-spans) section.

### In logs

You can also view exemplar trace details from the Loki logs in Explore.
Use regular expressions within the Derived fields links for Loki to extract the `traceID` information.
Now when you expand Loki logs, you can see a `traceID` property under the **Detected fields** section.
To learn more about how to extract a part of a log message into an internal or external link, refer to [using derived fields in Loki](../../explore/logs-integration/).

To view the details of an exemplar trace:

1. Expand a log line and scroll down to the `Fields` section.
   Depending on your backend trace data source, you may see a blue button with the label `<DATA SOURCE NAME>`.

1. Click the blue button next to the `traceID` property.
   Typically, it has the name of the backend data source.
   In the following example, the tracing data source is Tempo.
   The trace details, including the spans within the trace are listed in a separate panel on the right.

{{< figure src="/media/docs/grafana/exemplars/screenshot-exemplar-loki-logs.png" class="docs-image--no-shadow" max-width= "750px" caption="Explorer view with panel showing trace details" >}}

For more information on how to drill down and analyze the trace and span details, refer to the [Analyze trace and span details](#analyze-trace-and-spans) section.

### Analyze trace and spans

This panel shows the details of the trace in different segments.

- The top segment displays the trace ID to indicate that the query results correspond to the specific trace.

  You can add more traces to the results using the `Add query` button.

- The next segment shows the entire span for the specific trace as a narrow strip.
  All levels of the trace from the client all the way down to database query is displayed, which provides a bird's eye view of the time distribution across all layers over which the HTTP request was processed.
  1. You can click within this strip view to display a magnified view of a smaller time segment within the span. This magnified view shows up in the bottom segment of the panel.

  1. In the magnified view, you can expand or collapse the various levels of the trace to drill down to the specific span of interest.

     For example, if the strip view shows that most of the latency was within the app layer, you can expand the trace down the app layer to investigate the problem further.
     To expand a particular layer of span, click the left icon.
     The same button can collapse an expanded span.

- To see the details of the span at any level, click the span itself.

  This displays additional metadata associated with the span.
  The metadata itself is initially shown in a narrow strip but you can see more details by clicking the metadata strip.

  {{< figure src="/media/docs/grafana/exemplars/screenshot-exemplar-span-details.png" class="docs-image--no-shadow" max-width= "600px" caption="Span details" >}}

# Get started with Grafana and InfluxDB

{{< docs/shared lookup="influxdb/intro.md" source="grafana" version="<GRAFANA VERSION>" >}}

#### Get InfluxDB

You can [download InfluxDB](https://portal.influxdata.com/downloads/) and install it locally or you can sign up for [InfluxDB Cloud](https://www.influxdata.com/products/influxdb-cloud/). Windows installers are not available for some versions of InfluxDB.

#### Install other InfluxDB software

[Install Telegraf](https://docs.influxdata.com/telegraf/v1.18/introduction/installation/). This tool is an agent that helps you get metrics into InfluxDB. For more information, refer to [Telegraf documentation](https://docs.influxdata.com/telegraf/v1.18/).

If you chose to use InfluxDB Cloud, then you should [download and install the InfluxDB Cloud CLI](https://portal.influxdata.com/downloads/). This tool allows you to send command line instructions to your cloud account. For more information, refer to [Influx CLI documentation](https://docs.influxdata.com/influxdb/cloud/write-data/developer-tools/influx-cli/).

#### Get data into InfluxDB

If you downloaded and installed InfluxDB on your local machine, then use the [Quick Start](https://docs.influxdata.com/influxdb/v2.0/write-data/#quick-start-for-influxdb-oss) feature to visualize InfluxDB metrics.

If you are using the cloud account, then the wizards will guide you through the initial process. For more information, refer to [Configure Telegraf](https://docs.influxdata.com/influxdb/cloud/write-data/no-code/use-telegraf/#configure-telegraf).

##### Note for Windows users:

Windows users might need to make additional adjustments. Look for special instructions in the InfluxData documentation and [Using Telegraf on Windows](https://www.influxdata.com/blog/using-telegraf-on-windows/) blog post. The regular system monitoring template in InfluxDB Cloud is not compatible with Windows. Windows users who use InfluxDB Cloud to monitor their system will need to use the [Windows System Monitoring Template](https://github.com/influxdata/community-templates/tree/master/windows_system).

#### Add your InfluxDB data source to Grafana

You can have more than one InfluxDB data source defined in Grafana.

1. Follow the general instructions to [add a data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/#add-a-data-source).
1. Decide if you will use InfluxQL or Flux as your query language.
   - [Configure the data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/configure/) for your chosen query language.
     Each query language has its own unique data source settings.
   - For querying features specific to each language, see the data source's [query editor documentation](https://grafana.com/docs/grafana/latest/datasources/influxdb/query-editor/).

##### InfluxDB guides

InfluxDB publishes guidance for connecting different versions of their product to Grafana.

- **InfluxDB OSS or Enterprise 1.8+.** To turn on Flux, refer to [Configure InfluxDB](https://docs.influxdata.com/influxdb/v1.8/administration/config/#flux-enabled-false.). Select your InfluxDB version in the upper right corner.
- **InfluxDB OSS or Enterprise 2.x.** Refer to [Use Grafana with InfluxDB](https://docs.influxdata.com/influxdb/v2.0/tools/grafana/). Select your InfluxDB version in the upper right corner.
- **InfluxDB Cloud.** Refer to [Use Grafana with InfluxDB Cloud](https://docs.influxdata.com/influxdb/cloud/tools/grafana/).

##### Important tips

- Make sure your Grafana token has read access. If it doesn't, then you'll get an authentication error and be unable to connect Grafana to InfluxDB.
- Avoid apostrophes and other non-standard characters in bucket and token names.
- If the text name of the organization or bucket doesn't work, then try the ID number.
- If you change your bucket name in InfluxDB, then you must also change it in Grafana and your Telegraf .conf file as well.

#### Add a query

This step varies depending on the query language that you selected when you set up your data source in Grafana.

##### InfluxQL query language

In the query editor, click **select measurement**.

![InfluxQL query](/static/img/docs/influxdb/influxql-query-7-5.png)

Grafana displays a list of possible series. Click one to select it, and Grafana graphs any available data. If there is no data to display, then try another selection or check your data source.

##### Flux query language

Create a simple Flux query.

1. [Add a panel](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/).
1. In the query editor, select your InfluxDB-Flux data source. For more information, refer to [Queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/).
1. Select the **Table** visualization.
1. In the query editor text field, enter `buckets()` and then click outside of the query editor.

This generic query returns a list of buckets.

![Flux query](/static/img/docs/influxdb/flux-query-7-5.png)

You can also create Flux queries in the InfluxDB Explore view.

1. In your browser, log in to the InfluxDB native UI (OSS is typically something like http://localhost:8086 or for InfluxDB Cloud use: https://cloud2.influxdata.com).
1. Click **Explore** to open the Data Explorer.
1. The InfluxDB Data Explorer provides two mechanisms for creating Flux queries: a graphical query editor and a script editor. Using the graphical query editor, [create a query](https://docs.influxdata.com/influxdb/cloud/query-data/execute-queries/data-explorer/). It will look something like this:

   ![InfluxDB Explore query](/static/img/docs/influxdb/influx-explore-query-7-5.png)

1. Click **Script Editor** to view the text of the query, and then copy all the lines of your Flux code, which will look something like this:

   ![InfluxDB Explore Script Editor](/static/img/docs/influxdb/explore-query-text-7-5.png)

1. In Grafana, [add a panel](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/) and then paste your Flux code into the query editor.
1. Click **Apply**. Your new panel should be visible with data from your Flux query.

#### Check InfluxDB metrics in Grafana Explore

In your Grafana instance, go to the [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/explore/) view and build queries to experiment with the metrics you want to monitor. Here you can also debug issues related to collecting metrics.

#### Start building dashboards

There you go! Use Explore and Data Explorer to experiment with your data, and add the queries that you like to your dashboard as panels. Have fun!

Here are some resources to learn more:

- Grafana documentation: [InfluxDB data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/)
- InfluxDB documentation: [Comparison of Flux vs InfluxQL](https://docs.influxdata.com/influxdb/v1.8/flux/flux-vs-influxql/)

# Get started with Grafana and MS SQL Server

Microsoft SQL Server is a popular relational database management system that is widely used in development and production environments. This topic walks you through the steps to create a series of dashboards in Grafana to display metrics from a MS SQL Server database.

#### Download MS SQL Server

MS SQL Server can be installed on Windows or Linux operating systems and also on Docker containers. Refer to the [MS SQL Server downloads page](https://www.microsoft.com/en-us/sql-server/sql-server-downloads), for a complete list of all available options.

#### Install MS SQL Server

You can install MS SQL Server on the host running Grafana or on a remote server. To install the software from the [downloads page](https://www.microsoft.com/en-us/sql-server/sql-server-downloads), follow their setup prompts.

If you are on a Windows host but want to use Grafana and MS SQL data source on a Linux environment, refer to the [WSL to set up your Grafana development environment](/blog/2021/03/03/how-to-set-up-a-grafana-development-environment-on-a-windows-pc-using-wsl) blog post. This will allow you to leverage the resources available in [grafana/grafana](https://github.com/grafana/grafana) GitHub repository. Here you will find a collection of supported data sources, including MS SQL Server, along with test data and pre-configured dashboards for use.

#### Add the MS SQL data source

There are several ways to authenticate in MSSQL. Start by:

1. Click **Connections** in the left-side menu and filter by `mssql`.
1. Select the **Microsoft SQL Server** option.
1. Click **Create a Microsoft SQL Server data source** in the top right corner to open the configuration page.
1. Select the desired authentication method and fill in the right information as detailed below.
1. Click **Save & test**.

##### General configuration

| Name       | Description                                                                                                           |
| ---------- | --------------------------------------------------------------------------------------------------------------------- |
| `Name`     | The data source name. This is how you refer to the data source in panels and queries.                                 |
| `Host`     | The IP address/hostname and optional port of your MS SQL instance. If port is omitted, the default 1433 will be used. |
| `Database` | Name of your MS SQL database.                                                                                         |

##### SQL Server Authentication

| Name       | Description                     |
| ---------- | ------------------------------- |
| `User`     | Database user's login/username. |
| `Password` | Database user's password.       |

##### Windows Active Directory (Kerberos)

Below are the four possible ways to authenticate via Windows Active Directory/Kerberos.

{{< admonition type="note" >}}
Windows Active Directory (Kerberos) authentication is not supported in Grafana Cloud at the moment.
{{< /admonition >}}

| Method                    | Description                                                                                                                                                  |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Username + password**   | Enter the domain user and password                                                                                                                           |
| **Keytab file**           | Specify the path to a valid keytab file to use that for authentication.                                                                                      |
| **Credential cache**      | Log in on the host via `kinit` and pass the path to the credential cache. The cache path can be found by running `klist` on the host in question.            |
| **Credential cache file** | This option allows multiple valid configurations to be present and matching is performed on host, database, and user. See the example JSON below this table. |

```json
[
  {
    "user": "grot@GF.LAB",
    "database": "dbone",
    "address": "mysql1.mydomain.com:3306",
    "credentialCache": "/tmp/krb5cc_1000"
  },
  {
    "user": "grot@GF.LAB",
    "database": "dbtwo",
    "address": "mysql2.gf.lab",
    "credentialCache": "/tmp/krb5cc_1000"
  }
]
```

For installations from the [grafana/grafana](https://github.com/grafana/grafana/tree/main) repository, `gdev-mssql` data source is available. Once you add this data source, you can use the `Datasource tests - MSSQL` dashboard with three panels showing metrics generated from a test database.

![MS SQL Server dashboard](/static/img/docs/getting-started/gdev-sql-dashboard.png)

Optionally, play around this dashboard and customize it to:

- Create different panels.
- Change titles for panels.
- Change frequency of data polling.
- Change the period for which the data is displayed.
- Rearrange and resize panels.

#### Start building dashboards

Now that you have gained some idea of using the pre-packaged MS SQL data source and some test data, the next step is to setup your own instance of MS SQL Server database and data your development or sandbox area.

To fetch data from your own instance of MS SQL Server, add the data source using instructions in Step 4 of this topic. In Grafana [Explore](../../explore/) build queries to experiment with the metrics you want to monitor.

Once you have a curated list of queries, create [dashboards](../../dashboards/) to render metrics from the SQL Server database. For troubleshooting, user permissions, known issues, and query examples, refer to [Using Microsoft SQL Server in Grafana](../../datasources/mssql/).

# Get started with Grafana and Prometheus

Prometheus is an open source monitoring system for which Grafana provides out-of-the-box support. This topic walks you through the steps to create a series of dashboards in Grafana to display system metrics for a server monitored by Prometheus.

{{< admonition type="tip" >}}
Check out our Prometheus **Learning Journeys**.

- [Connect to a Prometheus data source in Grafana Cloud](https://www.grafana.com/docs/learning-journeys/prometheus/) to visualize your metrics directly from where they are stored.
- [Send metrics to Grafana Cloud using Prometheus remote write](https://www.grafana.com/docs/learning-journeys/prom-remote-write/) to explore Grafana Cloud without making significant changes to your existing configuration.
  {{< /admonition >}}

_Grafana and Prometheus_:

1. Download Prometheus and Node exporter
1. Install Prometheus Node exporter
1. Install and configure Prometheus
1. Configure Prometheus for Grafana
1. Check Prometheus metrics in Grafana Explore view
1. Start building dashboards

## Download Prometheus and Node exporter

Download the following components:

- [Prometheus](https://prometheus.io/download/#prometheus)
- [Node exporter](https://prometheus.io/download/#node_exporter)

Like Grafana, you can install Prometheus on many different operating systems. Refer to the [Prometheus download page](https://prometheus.io/download/) to see a list of stable versions of Prometheus components.

## Install Prometheus Node exporter

Install Node exporter on all hosts you want to monitor. This guide shows you how to install it locally.

Prometheus Node exporter is a widely used tool that exposes system metrics. For instructions on installing Node exporter, refer to the [Installing and running the Node exporter](https://prometheus.io/docs/guides/node-exporter/#installing-and-running-the-node-exporter) section in the Prometheus documentation.

When you run Node exporter locally, navigate to `http://localhost:9100/metrics` to check that it is exporting metrics.

{{< admonition type="note" >}}
The instructions in the referenced topic are intended for Linux users. You may have to alter the instructions slightly depending on your operating system. For example, if you are on Windows, use the [windows_exporter](https://github.com/prometheus-community/windows_exporter) instead.
{{< /admonition >}}

## Install and configure Prometheus

1. After [downloading Prometheus](https://prometheus.io/download/#prometheus), extract it and navigate to the directory.

   ```
   tar xvfz prometheus-*.tar.gz
   cd prometheus-*
   ```

1. Locate the `prometheus.yml` file in the directory.

1. Modify the Prometheus configuration file to monitor the hosts where you installed Node exporter.

By default, Prometheus looks for the file `prometheus.yml` in the current working directory. This behavior can be changed via the `--config.file` command line flag. For example, some Prometheus installers use it to set the configuration file to `/etc/prometheus/prometheus.yml`.

The following example shows you the code you should add. Notice that static configs targets are set to `['localhost:9100']` to target node-explorer when running it locally.

```
 # A scrape configuration containing exactly one endpoint to scrape from Node exporter running on a host:
 scrape_configs:
     # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
     - job_name: 'node'

     # metrics_path defaults to '/metrics'
     # scheme defaults to 'http'.

       static_configs:
       - targets: ['localhost:9100']
```

1. Start the Prometheus service:

   ```
    ./prometheus --config.file=./prometheus.yml
   ```

1. Confirm that Prometheus is running by navigating to `http://localhost:9090`.

You can see that the Node exporter metrics have been delivered to Prometheus. Next, the metrics will be sent to Grafana.

## Configure Prometheus for Grafana

When running Prometheus locally, there are two ways to configure Prometheus for Grafana. You can use a hosted Grafana instance at [Grafana Cloud](/) or run Grafana locally.

This guide describes configuring Prometheus in a hosted Grafana instance on Grafana Cloud.

1. Sign up for [https://grafana.com/](/auth/sign-up/create-user). Grafana gives you a Prometheus instance out of the box.

![Prometheus details in Grafana.com](/static/img/docs/getting-started/screenshot-grafana-prometheus-details.png)

1. Because you are running your own Prometheus instance locally, you must `remote_write` your metrics to the Grafana.com Prometheus instance. Grafana provides code to add to your `prometheus.yml` config file. This includes a remote write endpoint, your user name and password.

Add the following code to your prometheus.yml file to begin sending metrics to your hosted Grafana instance.

```
remote_write:
- url: <https://your-remote-write-endpoint>
  basic_auth:
    username: <your user name>
    password: <Your Grafana.com API Key>
```

{{< admonition type="note" >}}
To configure your Prometheus instance to work with Grafana locally instead of Grafana Cloud, install Grafana [here](/grafana/download) and follow the configuration steps listed [here](/docs/grafana/latest/datasources/prometheus/#configure-the-data-source).
{{< /admonition >}}

## Troubleshooting

These are some of the troubleshooting steps you can try if Prometheus isn’t running or functioning as expected. The steps provided have been selected based on the Learning Journeys we offer for Prometheus. If you’d like to explore further, check out the [Prometheus Learning Journey](https://grafana.com/docs/learning-journeys/prometheus/) if you want to visualize data in Grafana Cloud without sending or storing data in Grafana Cloud, such as for local retention needs. Alternatively, if you already have a Prometheus setup and want to explore Grafana Cloud without making significant changes, visit the [Prometheus remote write learning journey](https://grafana.com/docs/learning-journeys/prom-remote-write/).

### 1. Checking if Prometheus is running

If the Prometheus web UI is inaccessible (e.g., "Connection refused" error in the browser) or Prometheus queries fail (e.g., errors in Grafana like "Data source unavailable" or "No data points"), a good place to start is confirming that the Prometheus process and service are running.

You can do this by checking the system process or verifying the service status:

**Linux**

```bash
sudo systemctl status prometheus
```

- Shows whether the process is running and if the service is active.

**MacOS**

```bash
pgrep prometheus
```

- Returns the process ID (PID) if Prometheus is running.

**Windows** (`PowerShell`)

```powershell
Get-Process -Name prometheus -ErrorAction SilentlyContinue
```

- Checks if the Prometheus process is running by name.

### 2. If Prometheus is not running

Start by checking for common causes:

**Check for port conflicts**.

Prometheus runs on port 9090 by default. If another process is using this port, Prometheus may fail to start. You can check for port conflicts with:

**Linux & MacOS**

```bash
lsof -i :9090
```

**Windows** (`PowerShell`)

```powershell
netstat -ano | findstr :9090
```

- Shows if another process is using port **9090**.

**Verify the Prometheus binary placement**: ensure Prometheus binaries (`prometheus` and `promtool`) are correctly installed.

**Linux & MacOS**

```bash
ls /usr/local/bin/prometheus /usr/local/bin/promtool
```

- If missing, move them to `/usr/local/bin` or a directory in your system’s **PATH**.

**Check if Prometheus is in the path**.

```bash
which prometheus
which promtool
```

- If there’s **no output**, the binaries are not in the system **PATH**.

**Ensure configuration & data files are in place**.

**Linux & MacOs**

```bash
ls /etc/prometheus /var/lib/prometheus
ls /etc/prometheus/prometheus.yml
```

- Makes sure that Prometheus has its necessary configuration and data storage directories.

**Check permissions**: If Prometheus is running as a dedicated user, ensure the correct ownership:

```bash
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
```

(Optional) **Secure Prometheus by creating a dedicated user**

```bash
sudo useradd --no-create-home --shell /bin/false prometheus
```

- Recommended for security: runs Prometheus as a non-login user.

### 3. Checking if Prometheus is running as a service

If Prometheus is running as a process, check whether it is properly set up and managed as a service to ensure it restarts automatically after reboots or failures.

**Check Prometheus service status**

**Linux**

```bash
systemctl status prometheus.service
```

**Windows**

```powershell
sc query prometheus
```

**MacOs**

```bash
pgrep prometheus
```

- If the service is **inactive (dead) or stopped**, proceed to the next steps.

### 4. If Prometheus is not running as a service

If Prometheus is not running as a managed service, ensure it is correctly configured and can restart automatically.

**Verify service configuration** **(Linux & MacOs)**

Check the service unit file to ensure correct paths:

```bash
sudo nano /etc/systemd/system/prometheus.service
```

- Look for the `ExecStart` line:

```bash
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/
```

- Ensure:
  - The **binary path** (`/usr/local/bin/prometheus`) is correct.
  - The **configuration file** (`/etc/prometheus/prometheus.yml`) is in place.
  - The **storage path** (`/var/lib/prometheus/`) exists.

**Restart and enable Prometheus service (Linux & MacOs)**

```bash
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus
```

**Check Prometheus health status**

After restarting, verify if Prometheus is responsive:

```bash
curl -s http://localhost:9090/-/ready
```

- If successful, this confirms Prometheus is **ready to serve requests**.

**Restart Prometheus service (Windows)**

If running as a Windows service, restart it:

```powershell
net stop prometheus
net start prometheus
```

### 5. Checking if Prometheus is capturing metrics

If you installed [Node exporter](#install-prometheus-node-exporter) to expose your system metrics, you can check if Prometheus is capturing metrics by sending a request to the `/metrics` endpoint.

```bash
curl http://localhost:9090/metrics
```

- It should return a number of metrics and metadata about the metrics being exposed.

## Check Prometheus metrics in Grafana Metrics Drilldown

In your Grafana instance, go to the [Drilldown](https://www.grafana.com/docs/grafana/latest/explore/simplified-exploration/metrics/) view and experience query-less browsing of Prometheus-compatible metrics.

## Start building dashboards

Now that you have a curated list of queries, create [dashboards](../../dashboards/) to render system metrics monitored by Prometheus. When you install Prometheus and Node exporter or windows_exporter, you will find recommended dashboards for use.

The following image shows a dashboard with three panels showing some system metrics.

![Prometheus dashboards](/static/img/docs/getting-started/simple_grafana_prom_dashboard.png)

To learn more:

- Grafana documentation: [Prometheus data source](../../datasources/prometheus/)
- Prometheus documentation: [What is Prometheus?](https://prometheus.io/docs/introduction/overview/)

# Glossary

This topic lists words and abbreviations that are commonly used in the Grafana documentation and community.

<table>
  <tr>
    <td style="vertical-align: top">app plugin</td>
    <td>
      An extension of Grafana that allows users to provide additional functionality to enhance their experience by including a set of panel and data source plugins, as well as custom pages. See also <i>data source plugin</i>, <i>panel plugin</i>, and <i>plugin</i>.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">dashboard</td>
    <td>
      A set of one or more panels, organized and arranged into one or more rows, that provide an at-a-glance view of related information.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">data source</td>
    <td>
      A file, database, or service providing the data. Grafana supports several data sources by default, and can be extended to support additional data sources through plugins.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">data source plugin</td>
    <td>
      Extends Grafana with support for additional data sources. See also <i>data source</i>, <i>app plugin</i>, <i>panel plugin</i>, and <i>plugin</i>.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">exemplar</td>
    <td>
      An exemplar is any data that serves as a detailed example of one of the observations aggregated into a metric. An exemplar contains the observed value together with an optional timestamp and arbitrary labels, which are typically used to reference a trace.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">Explore</td>
    <td>
      Explore allows a user to focus on building a query. Users can refine the query to return the expected metrics before building a dashboard. For more information, refer to the <a href="https://grafana.com/docs/grafana/latest/explore">Explore</a> topic.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">export or import dashboard</td>
    <td>
      Grafana includes the ability to export your dashboards to a file containing JSON. Community members sometimes share their created dashboards on the <a href="https://grafana.com/grafana/dashboards">Grafana Dashboards page</a>. Dashboards previously exported or found on this site may be imported by other users.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">exporter</td>
    <td>
      An exporter translates data that comes out of a data source into a format that Prometheus can digest.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top"><code>gcx</code></td>
    <td>
      A command-line tool that enables users to authenticate, manage multiple environments, and perform administrative tasks through Grafana's REST API. It's compatible with AI agents.
  </tr>
  <tr>
    <td style="vertical-align: top">Integration (Grafana Cloud)</td>
    <td>
      Each Integration in Grafana Cloud uses the cloud agent to connect your data source to Grafana Cloud for visualizing. Note: Prometheus uses the word “integrations” to refer to software that exposes Prometheus metrics without needing an exporter, which is a different use of the same word we use here.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">graph</td>
    <td>
      A commonly-used visualization that displays data as points, lines, or bars.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top"><code>grafanactl</code></td>
    <td>
      A command-line tool that enables users to authenticate, manage multiple environments, and perform administrative tasks through Grafana's REST API. 
  </tr>
  <tr>
    <td style="vertical-align: top">mixin</td>
    <td>
      A mixin is a set of Grafana dashboards and Prometheus rules and alerts, written in Jsonnet and packaged together in a bundle.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">panel</td>
    <td>
      Basic building block in Grafana, composed by a query and a visualization. Can be moved and resized within a dashboard.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">panel plugin</td>
    <td>
      Extends Grafana with additional visualization options. See also <i>panel</i>, <i>plugin</i>, <i>app plugin</i>, and <i>data source plugin</i>.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">plugin</td>
    <td>
      An extension of Grafana that allows users to provide additional functionality to enhance their experience. See also <i>app plugin</i>, <i>data source plugin</i>, and <i>panel plugin</i>.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">query</td>
    <td>
      Used to request data from a data source. The structure and format of the query depend on the specific data source.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">time series</td>
    <td>
      A series of measurements, ordered by time. Time series are stored in data sources and returned as the result of a query.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">trace</td>
    <td>
      An observed execution path of a request through a distributed system. For more information, refer to <a href="https://opentracing.io/docs/overview/what-is-tracing/">What is Distributed Tracing?</a>
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">transformation</td>
    <td>
      Transformations process the result set of a query before it’s passed on for visualization. For more information, refer to the <a href="https://grafana.com/docs/grafana/latest/panels/transformations">Transformations overview</a> topic.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: top">visualization</td>
    <td>A graphical representation of query results.</td>
  </tr>
</table>

# Introduction to histograms and heatmaps

A histogram is a graphical representation of the distribution of numerical data. It groups values into buckets
(sometimes also called bins) and then counts how many values fall into each bucket.

Instead of graphing the actual values, histograms graph the buckets. Each bar represents a bucket,
and the bar height represents the frequency (such as count) of values that fell into that bucket's interval.

## Histogram example

This _histogram_ shows the value distribution of a couple of time series. You can easily see that
most values land between 240-300 with a peak between 260-280.

![Histogram example](/static/img/docs/v43/heatmap_histogram.png)

Here is an example showing height distribution of people.

{{< figure src="/static/img/docs/histogram-panel/histogram-example-v8-0.png" max-width="625px" caption="Bar chart example" >}}

For more information about histogram visualization options, refer to [Histogram](ref:histogram).

Histograms only look at _value distributions_ over a specific time range. The problem with histograms is that you cannot see any trends or changes in the distribution over time. This is where heatmaps become useful.

## Heatmaps

A _heatmap_ is like a histogram, but over time, where each time slice represents its own histogram. Instead of using bar height as a representation of frequency, it uses cells, and colors the cell proportional to the number of values in the bucket.

In this example, you can clearly see what values are more common and how they trend over time.

![Heatmap example](/static/img/docs/v43/heatmap_histogram_over_time.png)

For more information about heatmap visualization options, refer to [Heatmap](ref:heatmap).

## Pre-bucketed data

There are a number of data sources supporting histogram over time, like Elasticsearch (by using a Histogram bucket
aggregation) or Prometheus (with [histogram](https://prometheus.io/docs/concepts/metric_types/#histogram) metric type
and _Format as_ option set to Heatmap). But generally, any data source could be used as long as it meets the requirement
that it either returns series with names representing bucket bounds, or that it returns series sorted by the bounds
in ascending order.

## Raw data vs aggregated

If you use the heatmap with regular time series data (not pre-bucketed), then it's important to keep in mind that your data
is often already aggregated by your time series backend. Most time series queries do not return raw sample data,
but instead include a group by time interval or maxDataPoints limit coupled with an aggregation function (usually average).

This all depends on the time range of your query of course. But the important point is to know that the histogram bucketing
that Grafana performs might be done on already aggregated and averaged data. To get more accurate heatmaps, it is better
to do the bucketing during metric collection, or to store the data in Elasticsearch or any other data source which
supports doing histogram bucketing on the raw data.

If you remove or lower the group by time (or raise maxDataPoints) in your query to return more data points, your heatmap will be
more accurate, but this can also be very CPU and memory taxing for your browser, possibly causing hangs or crashes if the number of
data points becomes unreasonably large.

# What is Prometheus?

Observability focuses on understanding the internal state of your systems based on the data they produce, which helps determine if your infrastructure is healthy. Prometheus is a core technology for monitoring and observability of systems, but the term "Prometheus" can be confusing because it is used in different contexts. Understanding Prometheus basics, why it’s valuable for system observability, and how users use it in practice will both help you better understand it and help you use Grafana.

Prometheus began in 2012 at SoundCloud because existing technologies were insufficient for their observability needs. Prometheus offers both a robust data model and a query language. Prometheus is also simple and scalable. In 2018, Prometheus graduated from Cloud Native Computing Foundation (CNCF) incubation, and today has a thriving community.

## Prometheus as data

The following panel in a Grafana dashboard shows how much disk bandwidth on a Mac laptop is being used. The green line represents disk `reads`, and the yellow line represents `writes`.

{{< figure src="/media/docs/grafana/intro-prometheus/disk-io.png" max-width="750px" caption="Disk I/O dashboard" >}}

Data like these form _time series_. The X-axis is a moment in time and the Y-axis is a number or measurement; for example, 5 megabytes per second. This type of time series data appears everywhere in systems monitoring, as well as in places such as seasonal temperature charts and stock prices. This data is simply some measurement (such as a company stock price or Disk I/O) through a series of time instants.

Prometheus is a technology that collects and stores time series data. Time series are fundamental to Prometheus; its [data model](https://prometheus.io/docs/concepts/data_model/) is arranged into:

- _metrics_ that consist of a _timestamp_ and a _sample_, which is the numeric value, such as how many disk bytes have been read or a stock price
- a set of labels called _dimensions_, for example, `job` and `device`

You can store time series data in any relational database, however, these systems are not developed to store and query large volumes of time series data. Prometheus and similar software provide tools to compact and optimize time series data.

### Simple dashboard using PromQL

The following Grafana dashboard image shows a Disk I/O graph of raw data from Prometheus derived from a laptop.

The **Metrics browser** field contains the following query:

`node_disk_written_bytes_total{job="integrations/macos-node", device!=""}`

In this example, the Y-axis shows the total number of bytes written, and the X-axis shows dates and times. As the laptop runs, the number of bytes written increases over time. Below **Metrics browser** is a counter that counts the number of bytes written over time.

{{< figure src="/media/docs/grafana/intro-prometheus/dashboard-example.png" max-width="750px" caption="Metrics browser and counter" >}}

The query is a simple example of [PromQL](/blog/2020/02/04/introduction-to-promql-the-prometheus-query-language/), the Prometheus Query Language. The query identifies the metric of interest (`node_disk_written_bytes_total`) and provides two labels (`job` and `device`). The label selector `job="integrations/macos-node"` filters metrics. It both reduces the scope of the metrics to those coming from the MacOS integration job and specifies that the “device” label cannot be empty. The result of this query is the raw stream of numbers that the graph displays.

Although this view provides some insight into the performance of the system, it doesn’t provide the full story. A clearer picture of system performance requires understanding the rate of change that displays _how fast the data being written is changing_. To properly monitor disk performance, you need to also see spikes in activity that illustrate if and when the system is under load, and whether disk performance is at risk. PromQL includes a [rate()](https://prometheus.io/docs/prometheus/latest/querying/functions/#rate) function that shows the per-second average rate of increase over `5m` (5-minute) intervals. This view provides a much clearer picture of what’s happening with the system.

{{< figure src="/media/docs/grafana/intro-prometheus/rate-function.png" max-width="750px" caption="Prometheus rate function" >}}

A counter metric is just one type of metric; it is a number (such as total bytes written) that only increases. Prometheus [supports several others](https://prometheus.io/docs/concepts/metric_types/), such as the metric type `gauge`, which can increase or decrease.

The following gauge visualization displays the total RAM usage on a computer.

{{< figure src="/media/docs/grafana/intro-prometheus/gauge-example.png" max-width="750px" caption="Gauge visualization" >}}

The third metric type is called a `histogram`, which counts observations and organizes them into configurable groups. The following example displays floating-point numbers grouped into ranges that display how frequently each occurred.

{{< figure src="/media/docs/grafana/intro-prometheus/histogram-example.png" max-width="750px" caption="Histogram visualization" >}}

These core concepts of time series, metrics, labels, and aggregation functions are foundational to Grafana and observability.

## Why this is valuable

Software and systems are a difficult business. Sometimes things go wrong. Observability helps you understand a system’s state so that issues can be quickly identified and proactively addressed. And when problems do occur, you can be alerted to them to diagnose and solve them within your Service Level Objectives (SLOs).

The [three pillars of observability](https://www.oreilly.com/library/view/distributed-systems-observability/9781492033431/ch04.html) are metrics, logs, and traces. Prometheus supports the metrics pillar. When software on a computer runs slowly, observability can help you identify whether CPU is saturated, the system is out of memory, or if the disk is writing at maximum speed so you can proactively respond.

## Prometheus as software

Prometheus isn’t just a data format; it is also considered an [open source systems monitoring and alerting toolkit](https://prometheus.io/docs/introduction/overview/). That’s because Prometheus is software, not just data.

Prometheus can scrape metric data from software and infrastructure and store it. Scraping means that Prometheus software periodically revisits the same endpoint to check for new data. Prometheus scrapes data from a piece of software instrumented with a client library.

For example, a NodeJS application can configure the [prom-client](https://github.com/siimon/prom-client) to expose metrics easily at an endpoint, and Prometheus can regularly scrape that endpoint. Prometheus includes a number of other tools within the toolkit to instrument your applications.

## Prometheus as deployment

The first section of this document introduced the Prometheus as Data concept and how the Prometheus data model and metrics are organized. The second section introduced the concept of Prometheus as Software that is used to collect, process, and store metrics. This section describes how Prometheus as Data and Prometheus as Software come together.

Consider the following example. Suppose a 'MyApp' application uses a Prometheus client to expose metrics. One approach to collecting metrics data is to use a URL in the application that points to an endpoint `http://localhost:3000/metrics` that produces Prometheus metrics data.

The following image shows the two metrics associated with the endpoint. The HELP text explains what the metric means, and the TYPE text indicates what kind of metric it is (in this case, a gauge). `MyAppnodejs_active_request_total` indicates the number of requests (in this case, `1`). `MyAppnodejs_heap_size_total_bytes` indicates the heap size reported in bytes. There are only two numbers because this data shows the value at the moment the data was fetched.

{{< figure src="/media/docs/grafana/intro-prometheus/endpoint-data.png" max-width="750px" caption="Endpoint example" >}}

The 'MyApp' metrics are available in an HTTP endpoint, but how do they get to Grafana, and subsequently, into a dashboard? The process of recording and transmitting the readings of an application or piece of infrastructure is known as _telemetry_. Telemetry is critical to observability because it helps you understand exactly what's going on in your infrastructure. The metrics introduced previously, for example, `MyAppnodejs_active_requests_total`, are telemetry data.

To get metrics into Grafana, you can use either the Prometheus software or [Grafana Alloy](https://grafana.com/docs/alloy/latest/) to scrape metrics. Grafana Alloy collects and forwards the telemetry data to open-source deployments of the Grafana Stack, Grafana Cloud, or Grafana Enterprise, where your data can be analyzed. For example, you can configure Grafana Alloy to pull the data from 'MyApp' every five seconds and send the results to Grafana Cloud.

Metrics data is only one type of telemetry data; the other kinds are logs and traces. Using Grafana Alloy can be a great option to send telemetry data because as you scale your observability practices to include logs and traces, which Grafana Alloy also supports, you've got a solution already in place.

The following image illustrates how Grafana Alloy works as an intermediary between 'MyApp' and Grafana Cloud.

{{< figure src="/media/docs/alloy/flow-diagram-small-alloy.png" alt="Grafana Alloy" caption="Grafana Alloy" >}}

## Bringing it together

The combination of Prometheus and Grafana Alloy gives you control over the metrics you want to report, where they come from, and where they’re going. Once the data is in Grafana, it can be stored in a Grafana Mimir database. Grafana dashboards consist of visualizations populated by data queried from the Prometheus data source. The PromQL query filters and aggregates the data to provide you the insight you need. With those steps, we’ve gone from raw numbers, generated by software, into Prometheus, delivered to Grafana, queried by PromQL, and visualized by Grafana.

## What’s next?

Now that you understand how Prometheus metrics work, what will you build?

- One great next step is to [build a dashboard](ref:build-dashboards) in Grafana and start turning that raw Prometheus telemetry data into insights about what’s going with your services and infrastructure.
- Another great step is to learn about [Grafana Mimir](/oss/mimir/), which is essentially a database for Prometheus data. If you’re wondering how to make this work for a large volumes of metrics with a lot of data and fast querying, check out Grafana Mimir.
- If you’re interested in working with Prometheus data in Grafana directly, check out the [Prometheus data source](ref:prometheus) documentation, or check out [PromQL basics](https://prometheus.io/docs/prometheus/latest/querying/basics/).

# Introduction to time series

Imagine you wanted to know how the temperature outside changes throughout the day. Once every hour, you'd check the thermometer and write down the time along with the current temperature. After a while, you'd have something like this:

| Time  | Value |
| ----- | ----- |
| 09:00 | 24°C  |
| 10:00 | 26°C  |
| 11:00 | 27°C  |

Temperature data like this is one example of what we call a _time series_ — a sequence of measurements, ordered in time. Every row in the table represents one individual measurement at a specific time.

Tables are useful when you want to identify individual measurements, but they make it difficult to see the big picture. A more common visualization for time series is the _graph_, which instead places each measurement along a time axis. Visual representations like the graph make it easier to discover patterns and features of the data that otherwise would be difficult to see.

{{< figure src="/static/img/docs/example_graph.png" class="docs-image--no-shadow" max-width="850px" alt="Temperature data displayed on dashboard" >}}

Temperature data like the one in the example, is far from the only example of a time series. Other examples of time series are:

- CPU and memory usage
- Sensor data
- Stock market index

While each of these examples are sequences of chronologically ordered measurements, they also share other attributes:

- New data is appended at the end, at regular intervals — for example, hourly at 09:00, 10:00, 11:00, and so on.
- Measurements are seldom updated after they were added — for example, yesterday's temperature doesn't change.

Time series are powerful. They help you understand the past by letting you analyze the state of the system at any point in time. Time series could tell you that the server crashed moments after the free disk space went down to zero.

Time series can also help you predict the future, by uncovering trends in your data. If the number of registered users has been increasing monthly by 4% for the past few months, you can predict how big your user base is going to be at the end of the year.

Some time series have patterns that repeat themselves over a known period. For example, the temperature is typically higher during the day, before it dips down at night. By identifying these periodic, or _seasonal_, time series, you can make confident predictions about the next period. If you know that the system load peaks every day around 18:00, you can add more machines right before.

## Aggregating time series

Depending on what you're measuring, the data can vary greatly. What if you wanted to compare periods longer than the interval between measurements? If you'd measure the temperature once every hour, you'd end up with 24 data points per day. To compare the temperature in August over the years, you'd have to combine the 31 times 24 data points into one.

Combining a collection of measurements is called _aggregation_. There are several ways to aggregate time series data. Here are some common ones:

- **Average** returns the sum of all values divided by the total number of values.
- **Min** and **Max** return the smallest and largest value in the collection.
- **Sum** returns the sum of all values in the collection.
- **Count** returns the number of values in the collection.

For example, by aggregating the data in a month, you can determine that August 2017 was, on average, warmer than the year before. Instead, to see which month had the highest temperature, you'd compare the maximum temperature for each month.

How you choose to aggregate your time series data is an important decision and depends on the story you want to tell with your data. It's common to use different aggregations to visualize the same time series data in different ways.

## Time series and monitoring

In the IT industry, time series data is often collected to monitor things like infrastructure, hardware, or application events. Machine-generated time series data is typically collected with short intervals, which allows you to react to any unexpected changes, moments after they occur. As a consequence, data accumulates at a rapid pace, making it vital to have a way to store and query data efficiently. As a result, databases optimized for time series data have seen a rise in popularity in recent years.

### Time series databases

A time series database (TSDB) is a database explicitly designed for time series data. While it's possible to use any regular database to store measurements, a TSDB comes with some useful optimizations.

Modern time series databases take advantage of the fact that measurements are only ever appended, and rarely updated or removed. For example, the timestamps for each measurement change very little over time, which results in redundant data being stored.

Look at this sequence of Unix timestamps:

```
1572524345, 1572524375, 1572524404, 1572524434, 1572524464
```

Looking at these timestamps, they all start with `1572524`, leading to poor use of disk space. Instead, we could store each subsequent timestamp as the difference, or _delta_, from the first one:

```
1572524345, +30, +29, +30, +30
```

We could even take it a step further, by calculating the deltas of these deltas:

```
1572524345, +30, -1, +1, +0
```

If measurements are taken at regular intervals, most of these delta-of-deltas will be 0. Because of optimizations like these, TSDBs use drastically less space than other databases.

Another feature of a TSDB is the ability to filter measurements using _tags_. Each data point is labeled with a tag that adds context information, such as where the measurement was taken. Here's an example of the [InfluxDB data format](https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/#syntax) that demonstrates how each measurement is stored.

```
weather,location=us-midwest temperature=82 1465839830100400200
  |    -------------------- --------------  |
  |             |             |             |
  |             |             |             |
+-----------+--------+-+---------+-+---------+
|measurement|,tag_set| |field_set| |timestamp|
+-----------+--------+-+---------+-+---------+
```

Here are some of the TSDBs supported by Grafana:

- [Graphite](https://graphiteapp.org/)
- [InfluxDB](https://www.influxdata.com/products/influxdb-overview/)
- [Prometheus](https://prometheus.io/)

### Collecting time series data

Now that we have a place to store our time series, how do we actually gather the measurements? To collect time series data, you'd typically install a _collector_ on the device, machine, or instance you want to monitor. Some collectors are made with a specific database in mind, and some support different output destinations.

Here are some examples of collectors:

- [collectd](https://collectd.org/)
- [statsd](https://github.com/statsd/statsd)
- [Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/)
- [Telegraf](https://github.com/influxdata/telegraf)

A collector either _pushes_ data to a database or lets the database _pull_ the data from it. Both methods come with their own set of pros and cons:

|      | Pros                                                                      | Cons                                                                     |
| ---- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| Push | Easier to replicate data to multiple destinations.                        | The TSDB has no control over how much data gets sent.                    |
| Pull | Better control of how much data that gets ingested, and its authenticity. | Firewalls, VPNs or load balancers can make it hard to access the agents. |

Since it would be inefficient to write every measurement to the database, collectors pre-aggregate the data and write to the time series database at regular intervals.

# Time series dimensions

In [Introduction to time series](ref:time-series-databases), the concept of _labels_, also called _tags_, is introduced:

> Another feature of a TSDB is the ability to filter measurements using _tags_. Each data point is labeled with a tag that adds context information, such as where the measurement was taken.

With time series data, the data often contain more than a single series, and is a set of multiple time series. Many Grafana data sources support this type of data.

{{< figure src="/static/img/docs/example_graph_multi_dim.png" class="docs-image--no-shadow" max-width="850px" alt="Temperature by location" >}}

The common case is issuing a single query for a measurement with one or more additional properties as dimensions. For example, querying a temperature measurement along with a location property. In this case, multiple series are returned back from that single query and each series has unique location as a dimension.

To identify unique series within a set of time series, Grafana stores dimensions in _labels_.

## Labels

Each time series in Grafana optionally has labels. Labels are a set of key/value pairs for identifying dimensions. Example labels could be `{location=us}` or `{country=us,state=ma,city=boston}`. Within a set of time series, the combination of its name and labels identifies each series. For example, `temperature {country=us,state=ma,city=boston}` could identify the series of temperature values for the city of Boston in the US.

Different sources of time series data have dimensions stored natively, or common storage patterns that allow the data to be extracted into dimensions.

Time series databases (TSDBs) usually natively support dimensionality. Prometheus also stores dimensions in _labels_. In TSDBs such as Graphite or OpenTSDB the term _tags_ is used instead.

In table databases such SQL, these dimensions are generally the `GROUP BY` parameters of a query.

## Multiple dimensions in table format

In SQL or SQL-like databases that return table responses, additional dimensions are usually represented as columns in the query response table.

### Single dimension

For example, consider a query like:

```sql
SELECT BUCKET(StartTime, 1h), AVG(Temperature) AS Temp, Location FROM T
  GROUP BY BUCKET(StartTime, 1h), Location
  ORDER BY time asc
```

This query would return a table with three columns with data types time, number, and string respectively:

| StartTime | Temp | Location |
| --------- | ---- | -------- |
| 09:00     | 24   | LGA      |
| 09:00     | 20   | BOS      |
| 10:00     | 26   | LGA      |
| 10:00     | 22   | BOS      |

The table format is a _long_ formatted time series, also called _tall_. It has repeated time stamps, and repeated values in Location. In this case, we have two time series in the set that would be identified as `Temp {Location=LGA}` and `Temp {Location=BOS}`.

Individual time series from the set are extracted by using the time typed column `StartTime` as the time index of the time series, the numeric typed column `Temp` as the series name, and the name and values of the string typed `Location` column to build the labels, such as Location=LGA.

### Multiple dimensions

If the query is updated to select and group by more than just one string column, for example, `GROUP BY BUCKET(StartTime, 1h), Location, Sensor`, then an additional dimension is added:

| StartTime | Temp | Location | Sensor |
| --------- | ---- | -------- | ------ |
| 09:00     | 24   | LGA      | A      |
| 09:00     | 24.1 | LGA      | B      |
| 09:00     | 20   | BOS      | A      |
| 09:00     | 20.2 | BOS      | B      |
| 10:00     | 26   | LGA      | A      |
| 10:00     | 26.1 | LGA      | B      |
| 10:00     | 22   | BOS      | A      |
| 10:00     | 22.2 | BOS      | B      |

In this case the labels that represent the dimensions will have two keys based on the two string typed columns `Location` and `Sensor`. This data results four series: `Temp {Location=LGA,Sensor=A}`, `Temp {Location=LGA,Sensor=B}`, `Temp {Location=BOS,Sensor=A}`, and `Temp {Location=BOS,Sensor=B}`.

{{< admonition type="note" >}}
More than one dimension is currently only supported in the Logs queries within the Azure Monitor service as of version 7.1.
{{< /admonition >}}

{{< admonition type="note" >}}
Multiple dimensions are not supported in a way that maps to multiple alerts in Grafana, but rather they are treated as multiple conditions to a single alert.
For more information, see the documentation on [creating alerts with multiple series](ref:create-grafana-managed-rule).
{{< /admonition >}}

### Multiple values

In the case of SQL-like data sources, more than one numeric column can be selected, with or without additional string columns to be used as dimensions. For example, `AVG(Temperature) AS AvgTemp, MAX(Temperature) AS MaxTemp`. This, if combined with multiple dimensions, can result in a lot of series. Selecting multiple values is currently only designed to be used with visualization.

Additional technical information on tabular time series formats and how dimensions are extracted can be found in [the developer documentation on data frames as time series](https://grafana.com/developers/plugin-tools/key-concepts/data-frames#data-frames-as-time-series).

# Introduction

# Grafana Cloud

Grafana Cloud is a highly available, fast, fully-managed OpenSaaS logging and metrics platform. It is everything you love about Grafana, hosted by Grafana Labs.

Visit the [Grafana Cloud documentation](/docs/grafana-cloud) for more information.

# Grafana Enterprise

Grafana Enterprise is a commercial edition of Grafana that includes additional features not found in the open source version.

Building on everything you already know and love about Grafana open source, Grafana Enterprise includes [exclusive data source plugins](#enterprise-data-sources) and [additional features](#enterprise-features). You also get 24x7x365 support and training from the core Grafana team.

To learn more about Grafana Enterprise, refer to [our product page](/enterprise).

## Enterprise features in Grafana Cloud

Many Grafana Enterprise features are also available in paid [Grafana Cloud](/docs/grafana-cloud) accounts. For details, refer to [Grafana Cloud features](/docs/grafana-cloud/introduction/understand-grafana-cloud-features/). For pricing and plans, refer to [Grafana Cloud pricing](https://grafana.com/pricing/).

To migrate to Grafana Cloud, refer to [Migrate from Grafana Enterprise to Grafana Cloud](/docs/grafana/<GRAFANA_VERSION>/administration/migration-guide/)

## Authentication

Grafana Enterprise includes integrations with more ways to authenticate your users and enhanced authentication capabilities.

### Team sync

[Team sync](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/) allows you to set up synchronization between teams in Grafana and teams in your auth provider so that your users automatically end up in the right team.

Supported auth providers:

- [Auth Proxy](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/auth-proxy#team-sync-enterprise-only)
- [Entra ID OAuth](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/azuread/#team-sync-enterprise-only)
- [GitHub OAuth](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/github/#configure-team-synchronization)
- [Generic OAuth integration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/generic-oauth/#configure-team-synchronization)
- [GitLab OAuth](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/gitlab/#configure-team-synchronization)
- [Google OAuth](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/google/#configure-team-synchronization)
- [LDAP](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/enhanced-ldap/#ldap-group-synchronization-for-teams)
- [Okta](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/okta#configure-team-synchronization-enterprise-only)
- [SAML](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/saml#configure-team-sync)

### Enhanced LDAP integration

With [enhanced LDAP integration](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/enhanced-ldap/), you can set up active LDAP synchronization.

### SAML authentication

[SAML authentication](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/saml/) enables users to authenticate with single sign-on services that use Security Assertion Markup Language (SAML).

### Protected roles

With [protected roles](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/#protected-roles), you can define user roles that are exempt from being converted from one authentication type to another when changing auth providers.

## Enterprise features

Grafana Enterprise adds the following features:

- [Role-based access control](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/) to control access with role-based permissions.
- [Data source permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/#data-source-permissions) to restrict query access to specific teams and users.
- [Data source query and resource caching](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/#query-and-resource-caching) to temporarily store query results in Grafana to reduce data source load and rate limiting.
- [Reporting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/create-reports/) to generate a PDF report from any dashboard and set up a schedule to have it emailed to whomever you choose.
- [Export dashboard as PDF](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/share-dashboards-panels/#export-a-dashboard-as-pdf)
- [Custom branding](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/configure-custom-branding/) to customize Grafana from the brand and logo to the footer links.
- [Usage insights](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/assess-dashboard-usage/) to understand how your Grafana instance is used.
- [Recorded queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/recorded-queries/) to see trends over time for your data sources.
- [Vault integration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/configure-database-encryption/#encrypting-your-database-with-a-key-from-a-key-management-service-kms) to manage your configuration or provisioning secrets with Vault.
- [Auditing](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/audit-grafana/) tracks important changes to your Grafana instance to help you manage and mitigate suspicious activity and meet compliance requirements.
- [Request security](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/configure-request-security/) makes it possible to restrict outgoing requests from the Grafana server.
- [Settings updates at runtime](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/settings-updates-at-runtime/) allows you to update Grafana settings at runtime without requiring a restart.

## Enterprise data sources

With a Grafana Enterprise license, you also get access to premium data sources, including:

- [Adobe Analytics](/grafana/plugins/grafana-adobeanalytics-datasource)
- [Amazon Aurora](/grafana/plugins/grafana-aurora-datasource)
- [Amazon DynamoDB](/grafana/plugins/grafana-dynamodb-datasource/)
- [AppDynamics](/grafana/plugins/dlopes7-appdynamics-datasource)
- [Atlassian Statuspage](/grafana/plugins/grafana-atlassianstatuspage-datasource)
- [Azure Cosmos DB](/grafana/plugins/grafana-azurecosmosdb-datasource)
- [Azure DevOps](/grafana/plugins/grafana-azuredevops-datasource)
- [Catchpoint](/grafana/plugins/grafana-catchpoint-datasource)
- [Cloudflare](/grafana/plugins/grafana-cloudflare-datasource)
- [CockroachDB](/grafana/plugins/grafana-cockroachdb-datasource)
- [Databricks](/grafana/plugins/grafana-databricks-datasource)
- [Datadog](/grafana/plugins/grafana-datadog-datasource)
- [Drone CI](/grafana/plugins/grafana-drone-datasource)
- [Dynatrace](/grafana/plugins/grafana-dynatrace-datasource)
- [GitLab](/grafana/plugins/grafana-gitlab-datasource)
- [Grafana Enterprise Logs](/grafana/plugins/grafana-enterprise-logs-app/)
- [Grafana Enterprise Metrics](/grafana/plugins/grafana-metrics-enterprise-app/)
- [Grafana Enterprise Traces](/grafana/plugins/grafana-enterprise-traces-app/)
- [Honeycomb](/grafana/plugins/grafana-honeycomb-datasource)
- [IBM Db2](/grafana/plugins/grafana-ibmdb2-datasource)
- [Jenkins](/grafana/plugins/grafana-jenkins-datasource)
- [Jira](/grafana/plugins/grafana-jira-datasource)
- [LogicMonitor](/grafana/plugins/grafana-logicmonitor-datasource/)
- [Looker](/grafana/plugins/grafana-looker-datasource/)
- [MongoDB](/grafana/plugins/grafana-mongodb-datasource)
- [Netlify](/grafana/plugins/grafana-netlify-datasource)
- [New Relic](/grafana/plugins/grafana-newrelic-datasource)
- [Oracle Database](/grafana/plugins/grafana-oracle-datasource)
- [PagerDuty](/grafana/plugins/grafana-pagerduty-datasource)
- [Salesforce](/grafana/plugins/grafana-salesforce-datasource)
- [SAP HANA®](/grafana/plugins/grafana-saphana-datasource)
- [ServiceNow](/grafana/plugins/grafana-servicenow-datasource)
- [Snowflake](/grafana/plugins/grafana-snowflake-datasource)
- [SolarWinds](/grafana/plugins/grafana-solarwinds-datasource)
- [Splunk](/grafana/plugins/grafana-splunk-datasource)
- [Splunk Infrastructure Monitoring](/grafana/plugins/grafana-splunk-monitoring-datasource)
<!-- vale Grafana.Spelling = NO -->
- [Sqlyze](/grafana/plugins/grafana-odbc-datasource)
<!-- vale Grafana.Spelling = YES -->
- [Sumo Logic](/grafana/plugins/grafana-sumologic-datasource)
- [Vercel](/grafana/plugins/grafana-vercel-datasource)
- [Wavefront](/grafana/plugins/grafana-wavefront-datasource)
- [Zendesk](/grafana/plugins/grafana-zendesk-datasource)

## Try Grafana Enterprise

To purchase or obtain a trial license, contact the Grafana Labs [Sales Team](/contact?about=grafana-enterprise-stack).

# Setup Grafana

# Configure a Grafana Docker image

{{< admonition type="caution" >}}
Starting with Grafana release `12.4.0`, the `grafana/grafana-oss` Docker Hub repository will no longer be updated.
Instead, we encourage you to use the `grafana/grafana` Docker Hub repository. These two repositories have the same Grafana OSS docker images.
{{< /admonition >}}

# Configure a Grafana Docker image

This topic explains how to run Grafana on Docker in complex environments that require you to:

- Use different images
- Change logging levels
- Define secrets on the Cloud
- Configure plugins

> **Note:** The examples in this topic use the Grafana Enterprise Docker image. You can use the Grafana Open Source edition by changing the Docker image to `grafana/grafana`.

## Supported Docker image variants

You can install and run Grafana using the following official Docker images.

- **Grafana Enterprise**: `grafana/grafana-enterprise`

- **Grafana Open Source**: `grafana/grafana`

Each edition is available with an Alpine, Ubuntu, or Distroless base image. Each base image also has a slim variant.

Append the variant suffix to the Grafana version in the image tag:

| Base image | Standard tag           | Slim tag                    |
| ---------- | ---------------------- | --------------------------- |
| Alpine     | `<version>`            | `<version>-slim`            |
| Ubuntu     | `<version>-ubuntu`     | `<version>-ubuntu-slim`     |
| Distroless | `<version>-distroless` | `<version>-distroless-slim` |

## Alpine image (recommended)

[Alpine Linux](https://alpinelinux.org/about/) is a Linux distribution not affiliated with any commercial entity. It is a versatile operating system that caters to users who prioritize security, efficiency, and user-friendliness. Alpine Linux is much smaller than other distribution base images, allowing for slimmer and more secure images to be created.

By default, the images are built using the widely used [Alpine Linux project](http://alpinelinux.org/) base image, which can be found in the [Alpine docker repo](https://hub.docker.com/_/alpine).
If you prioritize security and want to minimize the size of your image, it is recommended that you use the Alpine variant. However, it's important to note that the Alpine variant uses [musl libc](http://www.musl-libc.org/) instead of [glibc and others](http://www.etalabs.net/compare_libcs.html). As a result, some software might encounter problems depending on their libc requirements. Nonetheless, most software should not experience any issues, so the Alpine variant is generally reliable.

## Ubuntu image

The Ubuntu-based Grafana Enterprise and OSS images are built using the [Ubuntu](https://ubuntu.com/) base image, which can be found in the [Ubuntu docker repo](https://hub.docker.com/_/ubuntu). An Ubuntu-based image can be a good option for users who prefer an Ubuntu-based image or require certain tools unavailable on Alpine.

- **Grafana Enterprise**: `grafana/grafana-enterprise:<version>-ubuntu`

- **Grafana Open Source**: `grafana/grafana:<version>-ubuntu`

## Distroless image

The Distroless-based Grafana Enterprise and OSS images use the [Distroless](https://github.com/GoogleContainerTools/distroless) base image. Distroless images contain fewer operating system packages than the Alpine and Ubuntu images. They don't include a shell, package manager, or other general-purpose operating system utilities, which results in a smaller footprint.

- **Grafana Enterprise**: `grafana/grafana-enterprise:<version>-distroless`

- **Grafana Open Source**: `grafana/grafana:<version>-distroless`

## Slim images

Slim images don't include the plugins that Grafana bundles with the standard images. You can still install plugins when the container starts by setting the `GF_PLUGINS_PREINSTALL` environment variable. For instructions, refer to [Install plugins in the Docker container](../installation/docker/#install-plugins-in-the-docker-container).

To use a slim image, add `-slim` to the base image suffix. For example, use `<version>-slim` for Alpine, `<version>-ubuntu-slim` for Ubuntu, or `<version>-distroless-slim` for Distroless.

## Run a specific version of Grafana

You can also run a specific version of Grafana or a beta version based on the main branch of the [`grafana/grafana` GitHub repository](https://github.com/grafana/grafana).

> **Note:** If you use a Linux operating system such as Debian or Ubuntu and encounter permission errors when running Docker commands, you might need to prefix the command with `sudo` or add your user to the `docker` group. The official Docker documentation provides instructions on how to [run Docker without a non-root user](https://docs.docker.com/engine/install/linux-postinstall/).

To run a specific version of Grafana, add it in the command <version number> section:

```bash
docker run -d -p 3000:3000 --name grafana grafana/grafana-enterprise:<version number>
```

Example:

The following command runs the Grafana Enterprise container and specifies version 9.4.7. If you want to run a different version, modify the version number section.

```bash
docker run -d -p 3000:3000 --name grafana grafana/grafana-enterprise:9.4.7
```

For recent releases of Grafana, there are also `minor` version tags in the `grafana/grafana` and `grafana/grafana-enterprise` docker repositories. For example, if you want to always have the latest `12.1` version, you can use `grafana/grafana-enterprise:12.1` or `grafana/grafana-enterprise:12.1-ubuntu`.

## Run the Grafana main branch

After every successful build of the main branch, two tags, `grafana/grafana:main` and `grafana/grafana:main-ubuntu`, are updated. Additionally, two new tags are created: `grafana/grafana-dev:<version>` and `grafana/grafana-dev:<version>-ubuntu`, where `version` is a prerelease version of Grafana. For example, if `1234` is the GitHub Run ID of the build, `12.2.0-1234`. These tags provide access to the most recent Grafana main builds. For more information, refer to [`grafana/grafana-dev`](https://hub.docker.com/r/grafana/grafana-dev/tags).

To ensure stability and consistency, we strongly recommend using the `grafana/grafana-dev:<version>` tag when running the Grafana main branch in a production environment. This tag ensures that you are using a specific version of Grafana instead of the most recent commit, which could potentially introduce bugs or issues. It also avoids polluting the tag namespace for the main Grafana images with thousands of pre-release tags.

For a list of available tags, refer to [`grafana/grafana`](https://hub.docker.com/r/grafana/grafana/tags/) and [`grafana/grafana-dev`](https://hub.docker.com/r/grafana/grafana-dev/tags/).

## Default paths

Grafana comes with default configuration parameters that remain the same among versions regardless of the operating system or the environment (for example, virtual machine, Docker, Kubernetes, etc.). You can refer to the [Configure Grafana](../configure-grafana/) documentation to view all the default configuration settings.

The following configurations are set by default when you start the Grafana Docker container. When running in Docker you cannot change the configurations by editing the `conf/grafana.ini` file. Instead, you can modify the configuration using [environment variables](../configure-grafana/#override-configuration-with-environment-variables).

| Setting               | Default value             |
| --------------------- | ------------------------- |
| GF_PATHS_CONFIG       | /etc/grafana/grafana.ini  |
| GF_PATHS_DATA         | /var/lib/grafana          |
| GF_PATHS_HOME         | /usr/share/grafana        |
| GF_PATHS_LOGS         | /var/log/grafana          |
| GF_PATHS_PLUGINS      | /var/lib/grafana/plugins  |
| GF_PATHS_PROVISIONING | /etc/grafana/provisioning |

## Install plugins in the Docker container

You can install publicly available plugins and plugins that are private or used internally in an organization. For plugin installation instructions, refer to [Install plugins in the Docker container](../installation/docker/#install-plugins-in-the-docker-container).

### Install plugins from other sources

To install plugins from other sources, you must define the custom URL and specify it immediately before the plugin name in the `GF_PLUGINS_PREINSTALL` environment variable: `GF_PLUGINS_PREINSTALL=<plugin ID>@[<plugin version>]@<url to plugin zip>`.

Example:

The following command runs Grafana Enterprise on **port 3000** in detached mode and installs the custom plugin, which is specified as a URL parameter in the `GF_PLUGINS_PREINSTALL` environment variable.

```bash
docker run -d -p 3000:3000 --name=grafana \
  -e "GF_PLUGINS_PREINSTALL=custom-plugin@@http://plugin-domain.com/my-custom-plugin.zip,grafana-clock-panel" \
  grafana/grafana-enterprise
```

## Build a custom Grafana Docker image

In the Grafana GitHub repository, the `packaging/docker/custom/` folder includes a `Dockerfile` that you can use to build a custom Grafana image. The `Dockerfile` accepts `GRAFANA_VERSION` and `GF_INSTALL_PLUGINS` as build arguments.

The `GRAFANA_VERSION` build argument must be a valid `grafana/grafana` Docker image tag. By default, Grafana builds an Alpine-based image. To build an Ubuntu-based image, append `-ubuntu` to the `GRAFANA_VERSION` build argument.

Example:

The following example shows you how to build and run a custom Grafana Docker image based on the latest official Ubuntu-based Grafana Docker image:

```bash
# go to the custom directory
cd packaging/docker/custom

# run the docker build command to build the image
docker build \
  --build-arg "GRAFANA_VERSION=latest-ubuntu" \
  -t grafana-custom .

# run the custom grafana container using docker run command
docker run -d -p 3000:3000 --name=grafana grafana-custom
```

### Build a Grafana Docker image with pre-installed plugins

If you run multiple Grafana installations with the same plugins, you can save time by building a customized image that includes plugins available on the [Grafana Plugin download page](/grafana/plugins). When you build a customized image, Grafana doesn't have to install the plugins each time it starts, making the startup process more efficient.

> **Note:** To specify the version of a plugin, you can use the `GF_INSTALL_PLUGINS` build argument and add the version number. The latest version is used if you don't specify a version number. For example, you can use `--build-arg "GF_INSTALL_PLUGINS=grafana-clock-panel 1.0.1,yesoreyeram-infinity-datasource 3.8.0"` to specify the versions of two plugins.

Example:

The following example shows how to build and run a custom Grafana Docker image with pre-installed plugins.

```bash
# go to the custom directory
cd packaging/docker/custom

# running the build command
# include the plugins you want e.g. clock planel etc
docker build \
  --build-arg "GRAFANA_VERSION=latest" \
  --build-arg "GF_INSTALL_PLUGINS=grafana-clock-panel,yesoreyeram-infinity-datasource" \
  -t grafana-custom .

# running the custom Grafana container using the docker run command
docker run -d -p 3000:3000 --name=grafana grafana-custom
```

### Build a Grafana Docker image with pre-installed plugins from other sources

You can create a Docker image containing a plugin that is exclusive to your organization, even if it is not accessible to the public. Simply use the `GF_INSTALL_PLUGINS` build argument to specify the plugin's URL and installation folder name, such as `GF_INSTALL_PLUGINS=<url to plugin zip>;<plugin install folder name>`.

The following example demonstrates creating a customized Grafana Docker image that includes a custom plugin from a URL link, the clock panel plugin, and the simple-json-datasource plugin. You can define these plugins in the build argument using the Grafana Plugin environment variable.

```bash
# go to the folder
cd packaging/docker/custom

# running the build command
docker build \
  --build-arg "GRAFANA_VERSION=latest" \
  --build-arg "GF_INSTALL_PLUGINS=http://plugin-domain.com/my-custom-plugin.zip;my-custom-plugin,grafana-clock-panel,yesoreyeram-infinity-datasource" \
  -t grafana-custom .

# running the docker run command
docker run -d -p 3000:3000 --name=grafana grafana-custom
```

## Logging

By default, Docker container logs are directed to `STDOUT`, a common practice in the Docker community. You can change this by setting a different [log mode](../configure-grafana/#mode) such as `console`, `file`, or `syslog`. You can use one or more modes by separating them with spaces, for example, `console file`. By default, both `console` and `file` modes are enabled.

Example:

The following example runs Grafana using the `console file` log mode that is set in the `GF_LOG_MODE` environment variable.

```bash
# Run Grafana while logging to both standard out
# and /var/log/grafana/grafana.log

docker run -p 3000:3000 -e "GF_LOG_MODE=console file" grafana/grafana-enterprise
```

## Configure Grafana with Docker Secrets

You can input confidential data like login credentials and secrets into Grafana using configuration files. This method works well with [Docker Secrets](https://docs.docker.com/engine/swarm/secrets/), as the secrets are automatically mapped to the `/run/secrets/` location within the container.

You can apply this technique to any configuration options in `conf/grafana.ini` by setting `GF_<SectionName>_<KeyName>__FILE` to the file path that contains the secret information. For more information about Docker secret command usage, refer to [docker secret](https://docs.docker.com/engine/reference/commandline/secret/).

The following example demonstrates how to set the admin password:

- Admin password secret: `/run/secrets/admin_password`
- Environment variable: `GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/admin_password`

### Configure Docker secrets credentials for AWS CloudWatch

Grafana ships with built-in support for the [Amazon CloudWatch datasource](../../datasources/aws-cloudwatch/). To configure the data source, you must provide information such as the AWS ID-Key, secret access key, region, and so on. You can use Docker secrets as a way to provide this information.

Example:

The example below shows how to use Grafana environment variables via Docker Secrets for the AWS ID-Key, secret access key, region, and profile.

The example uses the following values for the AWS Cloudwatch data source:

```bash
AWS_default_ACCESS_KEY_ID=aws01us02
AWS_default_SECRET_ACCESS_KEY=topsecret9b78c6
AWS_default_REGION=us-east-1
```

1. Create a Docker secret for each of the values noted above.

   ```bash
   echo "aws01us02" | docker secret create aws_access_key_id -
   ```

   ```bash
   echo "topsecret9b78c6" | docker secret create aws_secret_access_key -
   ```

   ```bash
   echo "us-east-1" | docker secret create aws_region -
   ```

1. Run the following command to determine that the secrets were created.

   ```bash
   $ docker secret ls
   ```

   The output from the command should look similar to the following:

   ```
   ID                          NAME           DRIVER    CREATED              UPDATED
   i4g62kyuy80lnti5d05oqzgwh   aws_access_key_id             5 minutes ago        5 minutes ago
   uegit5plcwodp57fxbqbnke7h   aws_secret_access_key         3 minutes ago        3 minutes ago
   fxbqbnke7hplcwodp57fuegit   aws_region                    About a minute ago   About a minute ago
   ```

   Where:

   ID = the secret unique ID that will be use in the docker run command

   NAME = the logical name defined for each secret

1. Add the secrets to the command line when you run Docker.

   ```bash
   docker run -d -p 3000:3000 --name grafana \
     -e "GF_DEFAULT_INSTANCE_NAME=my-grafana" \
     -e "GF_AWS_PROFILES=default" \
     -e "GF_AWS_default_ACCESS_KEY_ID__FILE=/run/secrets/aws_access_key_id" \
     -e "GF_AWS_default_SECRET_ACCESS_KEY__FILE=/run/secrets/aws_secret_access_key" \
     -e "GF_AWS_default_REGION__FILE=/run/secrets/aws_region" \
     -v grafana-data:/var/lib/grafana \
     grafana/grafana-enterprise
   ```

You can also specify multiple profiles to `GF_AWS_PROFILES` (for example, `GF_AWS_PROFILES=default another`).

The following list includes the supported environment variables:

- `GF_AWS_${profile}_ACCESS_KEY_ID`: AWS access key ID (required).
- `GF_AWS_${profile}_SECRET_ACCESS_KEY`: AWS secret access key (required).
- `GF_AWS_${profile}_REGION`: AWS region (optional).

## Troubleshoot a Docker deployment

By default, the Grafana log level is set to `INFO`, but you can increase the log level to `DEBUG` mode when you want to reproduce a problem.

For more information about logging, refer to [logs](../configure-grafana/#log).

### Increase log level using the Docker run (CLI) command

To increase the log level to `DEBUG` mode, add the environment variable `GF_LOG_LEVEL` to the command line.

```bash
docker run -d -p 3000:3000 --name=grafana \
  -e "GF_LOG_LEVEL=debug" \
  grafana/grafana-enterprise
```

### Increase log level using the Docker Compose

To increase the log level to `DEBUG` mode, add the environment variable `GF_LOG_LEVEL` to the `docker-compose.yaml` file.

```yaml
version: '3.8'
services:
  grafana:
    image: grafana/grafana-enterprise
    container_name: grafana
    restart: unless-stopped
    environment:
      # increases the log level from info to debug
      - GF_LOG_LEVEL=debug
    ports:
      - '3000:3000'
    volumes:
      - 'grafana_storage:/var/lib/grafana'
volumes:
  grafana_storage: {}
```

### Validate Docker Compose YAML file

The chance of syntax errors appearing in a YAML file increases as the file becomes more complex. You can use the following command to check for syntax errors.

```bash
# go to your docker-compose.yaml directory
cd /path-to/docker-compose/file

# run the validation command
docker compose config
```

If there are errors in the YAML file, the command output highlights the lines that contain errors. If there are no errors in the YAML file, the output includes the content of the `docker-compose.yaml` file in detailed YAML format.

# Set up Grafana for high availability

{{< admonition type="note" >}}
To prevent duplicate alerts in Grafana high availability, additional steps are required.

Please refer to [Alerting high availability](#alerting-high-availability) for more information.
{{< /admonition >}}

Grafana uses an embedded sqlite3 database to store users, dashboards, and other persistent data by default. For high availability, you must use a shared database to store this data. This shared database can be either MySQL or Postgres.

<div class="text-center">
  <img src="/static/img/docs/tutorials/grafana-high-availability.png"  max-width= "800px" class="center" />
</div>

## Architecture

Your Grafana high availability environment will consist of two or more Grafana servers (cluster nodes) served by a load balancing reverse proxy. The cluster uses an active-active architecture with the load balancer allocating traffic between nodes and re-allocating traffic to surviving nodes should there be failures. You need to configure your load balancer with a listener that responds to a shared cluster hostname. The shared name is the hostname your users use to access Grafana.

For ease of use, we recommend you configure your load balancer to provide SSL termination. The shared Grafana database tracks session information, so your load balancer won't need to provide session affinity services. See your load balancer's documentation for details on its configuration and operations.

## Before you begin

Before you complete the following tasks, configure a MySQL or Postgres database to be highly available. Configuring the MySQL or Postgres database for high availability is out of the scope of this guide, but you can find instructions online for each database.

## Configure multiple Grafana servers to use the same database

Once you have a Postgres or MySQL database available, you can configure your multiple Grafana instances to use a shared backend database. Grafana has default and custom configuration files, and you can update the database settings by updating your custom configuration file as described in the [[database]](../configure-grafana/#database). Once configured to use a shared database, your multiple Grafana instances will persist all long-term data in that database.

## Grafana Enterprise only: License your Grafana servers

If you're using Grafana Enterprise:

1. Get a license token in the name of your cluster's shared hostname.
1. Edit the [`root_url`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#root_url) setting in each node's `grafana.ini` configuration file to reflect the cluster's shared hostname.
1. Install the license key as normal. For more information on installing your license key, refer to [Add your license to a Grafana instance](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/enterprise-licensing/#step-2-add-your-license-to-a-grafana-instance).

## Alerting high availability

Grafana Alerting provides a high availability mode. It preserves the semantics of legacy dashboard alerting by executing all alerts on every server and by sending notifications only once per alert. Load distribution between servers is not supported at this time.

For further information and instructions on setting up alerting high availability, refer to [Enable alerting high availability](../../alerting/set-up/configure-high-availability/).

**Legacy dashboard alerts**

Legacy Grafana Alerting supports a limited form of high availability. In this model, alert notifications are deduplicated when running multiple servers. This means all alerts are executed on every server, but alert notifications are only sent once per alert. Grafana does not support load distribution between servers.

## Grafana Live

Grafana Live works with limitations in highly available setup. For details, refer to the [Configure Grafana Live HA setup](../set-up-grafana-live/#configure-grafana-live-ha-setup).

## User sessions

Grafana uses auth token strategy with database by default. This means that a load balancer can send a user to any Grafana server without having to log in on each server.

# Set up Grafana Live

Grafana Live is a real-time messaging engine you can use to push event data to a frontend as soon as an event occurs.

This could be notifications about dashboard changes, new frames for rendered data, and so on. Live features can help eliminate a page reload or polling in many places, it can stream Internet of things (IoT) sensors or any other real-time data to panels.

{{< admonition type="note" >}}
By `real-time`, we indicate a soft real-time. Due to network latencies, garbage collection cycles, and so on, the delay of a delivered message can be up to several hundred milliseconds or higher.
{{< /admonition >}}

## Concepts

Grafana Live sends data to clients over persistent WebSocket connections, based on a Pub/Sub model. The Grafana frontend subscribes on each channel to receive data that has been published in that channel. All subscriptions on a page are multiplexed inside a single WebSocket connection. There are some rules regarding Live channel names – see [Grafana Live channel](#grafana-live-channel).

Handling persistent connections like WebSocket in scale may require operating system and infrastructure tuning. That's why by default Grafana Live supports 100 simultaneous connections max. For more details on how to tune this limit, refer to [Live configuration section](#configure-grafana-live).

## Features

Having a way to send data to clients in real-time opens a road for new ways of data interaction and visualization. Below we describe Grafana Live features supported at the moment.

### Dashboard change notifications

As soon as there is a change to the dashboard layout, it is automatically reflected on other devices connected to Grafana Live.

### Data streaming from plugins

With Grafana Live, backend data source plugins can stream updates to frontend panels.

For data source plugin channels, Grafana uses `ds` scope. Namespace in the case of data source channels is a data source unique ID (UID) which is issued by Grafana at the moment of data source creation. The path is a custom string that plugin authors free to choose themselves (just make sure it consists of allowed symbols).

For example, a data source channel looks like this: `ds/<DATASOURCE_UID>/<CUSTOM_PATH>`.

Refer to the tutorial about [building a streaming data source backend plugin](/tutorials/build-a-streaming-data-source-plugin/) for more details.

The basic streaming example included in Grafana core streams frames with some generated data to a panel. To look at it create a new panel and point it to the `-- Grafana --` data source. Next, choose `Live Measurements` and select the `plugin/testdata/random-20Hz-stream` channel.

### Data streaming from Telegraf

A new API endpoint `/api/live/push/:streamId` allows accepting metrics data in InfluxDB line protocol format from Telegraf. These metrics are transformed into Grafana data frames and published to channels.

**Important:** When streaming data from Telegraf, ensure that:

- The data format is set to `influx` in your Telegraf configuration
- Timestamps are in nanoseconds (Unix epoch time)

Refer to the tutorial about [streaming metrics from Telegraf to Grafana](/tutorials/stream-metrics-from-telegraf-to-grafana/) for more information.

## Grafana Live channel

Grafana Live is a PUB/SUB server, clients subscribe to channels to receive real-time updates published to those channels.

### Channel structure

Channel is a string identifier. In Grafana channel consists of 3 parts delimited by `/`:

- Scope
- Namespace
- Path

For example, the channel `grafana/dashboard/xyz` has the scope `grafana`, namespace `dashboard`, and path `xyz`.

Scope, namespace and path can only have ASCII alphanumeric symbols (A-Z, a-z, 0-9), `_` (underscore) and `-` (dash) at the moment. The path part can additionally have `/`, `.` and `=` symbols. The meaning of scope, namespace and path is context-specific.

The maximum length of a channel is 160 symbols.

Scope determines the purpose of a channel in Grafana. For example, for data source plugin channels Grafana uses `ds` scope. For built-in features like dashboard edit notifications Grafana uses `grafana` scope.

Namespace has a different meaning depending on scope. For example, for `grafana` scope this could be a name of built-in real-time feature like `dashboard` (i.e. dashboards events).

The path, which is the final part of a channel, usually contains the identifier of some concrete resource such as the ID of a dashboard that a user is currently looking at. But a path can be anything.

Channels are lightweight and ephemeral - they are created automatically on user subscription and removed as soon as last user left a channel.

### Data format

Data sent over Live channels supports multiple formats depending on the endpoint:

#### WebSocket channels

Data transmitted through WebSocket channels for real-time streaming to frontend clients must be **JSON-encoded**. This applies to:

- Dashboard change notifications
- Data source plugin streaming (`ds` scope channels)
- Custom application channels

#### HTTP Push API

The `/api/live/push/:streamId` endpoint accepts metrics data in **InfluxDB line protocol format**. This is particularly useful for streaming metrics from tools like Telegraf.

The line protocol format follows this structure:

```
<measurement>[,<tag_key>=<tag_value>[,<tag_key>=<tag_value>]] <field_key>=<field_value>[,<field_key>=<field_value>] [<timestamp>]
```

**Important:** Timestamps must be encoded in **nanoseconds** (Unix epoch time in nanoseconds).

**Example:**

```
temperature,location=room1,sensor=A temp_celsius=23.5,humidity=45 1651498849000000000
```

In this example:

- `temperature` is the measurement name
- `location=room1` and `sensor=A` are tags
- `temp_celsius=23.5` and `humidity=45` are fields
- `1651498849000000000` is the timestamp in nanoseconds

When data is received via the HTTP Push API, it is automatically transformed into Grafana data frames and published to the appropriate channels for visualization.

For more information about the InfluxDB line protocol format, refer to the [InfluxDB line protocol documentation](https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/).

## Configure Grafana Live

Grafana Live is enabled by default. In Grafana v8.0, it has a strict default for a maximum number of connections per Grafana server instance.

### Max number of connections

Grafana Live uses persistent connections (WebSocket at the moment) to deliver real-time updates to clients.

WebSocket is a persistent connection that starts with an HTTP Upgrade request (using the same HTTP port as the rest of Grafana) and then switches to a TCP mode where WebSocket frames can travel in both directions between a client and a server. Each logged-in user opens a WebSocket connection – one per browser tab.

The number of maximum WebSocket connections users can establish with Grafana is limited to 100 by default. See [max_connections](../configure-grafana/#max_connections) option.

In case you want to increase this limit, ensure that your server and infrastructure allow handling more connections. The following sections discuss several common problems which could happen when managing persistent connections, in particular WebSocket connections.

### Request origin check

To avoid hijacking of WebSocket connection Grafana Live checks the Origin request header sent by a client in an HTTP Upgrade request. Requests without Origin header pass through without any origin check.

By default, Live accepts connections with Origin header that matches configured [root_url](../configure-grafana/#root_url) (which is a public Grafana URL).

It is possible to provide a list of additional origin patterns to allow WebSocket connections from. This can be achieved using the [allowed_origins](../configure-grafana/#allowed_origins) option of Grafana Live configuration.

#### Resource usage

Each persistent connection costs some memory on a server. Typically, this should be about 50 KB per connection at this moment. Thus a server with 1 GB RAM is expected to handle about 20k connections max. Each active connection consumes additional CPU resources since the client and server send PING/PONG frames to each other to maintain a connection.

Using the streaming functionality results in additional CPU usage. The exact CPU resource utilization can be hard to estimate as it heavily depends on the Grafana Live usage pattern.

#### Open file limit

Each WebSocket connection costs a file descriptor on a server machine where Grafana runs. Most operating systems have a quite low default limit for the maximum number of descriptors that process can open.

To look at the current limit on Unix run:

```
ulimit -n
```

On a Linux system, you can also check out the current limits for a running process with:

```
cat /proc/<PROCESS_PID>/limits
```

The open files limit shows approximately how many user connections your server can currently handle.

To increase this limit, refer to [these instructions](https://docs.riak.com/riak/kv/2.2.3/using/performance/open-files-limit.1.html) for popular operating systems.

#### Ephemeral port exhaustion

Ephemeral port exhaustion problem can happen between your load balancer (or reverse proxy) software and Grafana server. For example, when you load balance requests/connections between different Grafana instances. If you connect directly to a single Grafana server instance, then you should not come across this issue.

The problem arises because each TCP connection uniquely identified in the OS by the 4-part-tuple:

```
source ip | source port | destination ip | destination port
```

By default, on load balancer/server boundary you are limited to 65535 possible variants. But actually, due to some OS limits (for example on Unix available ports defined in `ip_local_port_range` sysctl parameter) and sockets in TIME_WAIT state, the number is even less.

In order to eliminate a problem you can:

- Increase the ephemeral port range by tuning `ip_local_port_range` kernel option.
- Deploy more Grafana server instances to load balance across.
- Deploy more load balancer instances.
- Use virtual network interfaces.

#### WebSocket and proxies

Not all proxies can transparently proxy WebSocket connections by default. For example, if you are using Nginx before Grafana you need to configure WebSocket proxy like this:

```
http {
    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

    upstream grafana {
        server 127.0.0.1:3000;
    }

    server {
        listen 8000;

        location / {
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $http_host;
            proxy_pass http://grafana;
        }
    }
}
```

See the [Nginx blog on their website](https://www.nginx.com/blog/websocket-nginx/) for more information. Also, refer to your load balancer/reverse proxy documentation to find out more information on dealing with WebSocket connections.

Some corporate proxies can remove headers required to properly establish a WebSocket connection. In this case, you should tune intermediate proxies to not remove required headers. However, the better option is to use Grafana with TLS. Now WebSocket connection will inherit TLS and thus must be handled transparently by proxies.

Proxies like Nginx and Envoy have default limits on maximum number of connections which can be established. Make sure you have a reasonable limit for max number of incoming and outgoing connections in your proxy configuration.

## Configure Grafana Live HA setup

By default, Grafana Live uses in-memory data structures and in-memory PUB/SUB hub for handling subscriptions.

In a high availability Grafana setup involving several Grafana server instances behind a load balancer, you can find the following limitations:

- Built-in features like dashboard change notifications will only be broadcasted to users connected to the same Grafana server process instance.
- Streaming from Telegraf will deliver data only to clients connected to the same instance which received Telegraf data, active stream cache is not shared between different Grafana instances.
- A separate unidirectional stream between Grafana and backend data source may be opened on different Grafana servers for the same channel.

To bypass these limitations, Grafana has a Live HA engine that requires Redis to work.

### Configure Redis Live engine

When the Redis engine is configured, Grafana Live keeps its state in Redis and uses Redis PUB/SUB functionality to deliver messages to all subscribers throughout all Grafana server nodes.

Here is an example configuration:

```
[live]
ha_engine = redis
ha_engine_address = 127.0.0.1:6379
```

For additional information, refer to the [ha_engine](../configure-grafana/#ha_engine) and [ha_engine_address](../configure-grafana/#ha_engine_address) options.

After running:

- All built-in real-time notifications like dashboard changes are delivered to all Grafana server instances and broadcasted to all subscribers.
- Streaming from Telegraf delivers messages to all subscribers.
- A separate unidirectional stream between Grafana and backend data source opens on different Grafana servers. Publishing data to a channel delivers messages to instance subscribers, as a result, publications from different instances on different machines do not produce duplicate data on panels.

{{< admonition type="note" >}}
Live currently does not support Redis Sentinel. We recommend using a Redis Cluster for high-availability via a k8s helm chart such as the Bitnami Redis chart which has values to provision a Redis Cluster. Grafana Live can then be pointed to the `redis-headless` service.

```
 live:
   ha_engine: redis
   ha_engine_address: redis-headless.grafana.svc.cluster.local:6379
   ha_engine_password: $__file{/your/redis/password/secret/mount}
```

{{< /admonition >}}

To connect to Redis over TLS, use a `rediss://` connection URL as the engine address:

```ini
[live]
ha_engine = redis
ha_engine_address = rediss://redis.example.com:6380
```

The `redis://` scheme is also supported. Both schemes accept credentials and a database number in the URL, for example `redis://username:password@localhost:6379/0`. A password set in the URL (even an explicitly empty one) takes precedence; `ha_engine_password` applies when the URL sets no password. Grafana verifies server certificates against the operating system's trusted certificate authorities; verification cannot be disabled through this setting.

# Set up Grafana monitoring

Grafana supports tracing.

Grafana can emit Jaeger or OpenTelemetry Protocol (OTLP) traces for its HTTP API endpoints and propagate Jaeger and [w3c Trace Context](https://www.w3.org/TR/trace-context/) trace information to compatible data sources.
All HTTP endpoints are logged evenly (annotations, dashboard, tags, and so on).
When a trace ID is propagated, it is reported with operation 'HTTP /datasources/proxy/:id/\*'.

Refer to [Configuration's OpenTelemetry section](../configure-grafana/#tracingopentelemetry) for a reference of tracing options available in Grafana.

## View Grafana internal metrics

Grafana collects some metrics about itself internally. Grafana supports pushing metrics to Graphite or exposing them to be scraped by Prometheus.

For more information about configuration options related to Grafana metrics, refer to [metrics](../configure-grafana/#metrics) and [metrics.graphite](../configure-grafana/#metricsgraphite) in [Configuration](../configure-grafana/).

### Available metrics

When enabled, Grafana exposes a number of metrics, including:

- Active Grafana instances
- Number of dashboards, users, and playlists
- HTTP status codes
- Requests by routing group
- Grafana active alerts
- Grafana performance

#### Native histogram format

Grafana exposes HTTP request metrics using native histograms for a more accurate representation of metric distributions. By default, both native histograms and classic histogram buckets are exposed for compatibility.

To reduce metric cardinality, you can disable classic histogram buckets and expose only native histograms by setting the following option in your configuration file:

```
[metrics]
# Enable classic HTTP histogram buckets alongside native histograms
# Set to false to only expose native histogram format (reduces cardinality)
classic_http_histogram_enabled = false
```

### Pull metrics from Grafana into Prometheus

These instructions assume you have already added Prometheus as a data source in Grafana.

1. Enable Prometheus to scrape metrics from Grafana. In your configuration file (`grafana.ini` or `custom.ini` depending on your operating system) remove the semicolon to enable the following configuration options:

   ```
   # Metrics available at HTTP URL /metrics and /metrics/plugins/:pluginId
   [metrics]
   # Disable / Enable internal metrics
   enabled           = true

   # Disable total stats (stat_totals_*) metrics to be generated
   disable_total_stats = false
   ```

1. (optional) If you want to require authorization to view the metrics endpoints, then uncomment and set the following options:

   ```
   basic_auth_username =
   basic_auth_password =
   ```

1. Restart Grafana. Grafana now exposes metrics at http://localhost:3000/metrics.
1. Add the job to your prometheus.yml file.
   Example:

   ```
   - job_name: 'grafana_metrics'

     scrape_interval: 15s
     scrape_timeout: 5s

     static_configs:
       - targets: ['localhost:3000']
   ```

1. Restart Prometheus. Your new job should appear on the Targets tab.
1. In Grafana, click **Connections** in the left-side menu.
1. Under your connections, click **Data Sources**.
1. Select the **Prometheus** data source.
1. Under the name of your data source, click **Dashboards**.
1. On the Dashboards tab, click **Import** in the _Grafana metrics_ row to import the Grafana metrics dashboard. All scraped Grafana metrics are available in the dashboard.

### View Grafana metrics in Graphite

These instructions assume you have already added Graphite as a data source in Grafana.

1. Enable sending metrics to Graphite. In your configuration file (`grafana.ini` or `custom.ini` depending on your operating system) remove the semicolon to enable the following configuration options:

   ```
   # Metrics available at HTTP API Url /metrics
   [metrics]
   # Disable / Enable internal metrics
   enabled           = true

   # Disable total stats (stat_totals_*) metrics to be generated
   disable_total_stats = false
   ```

1. Enable [metrics.graphite] options:

   ```
   # Send internal metrics to Graphite
   [metrics.graphite]
   # Enable by setting the address setting (ex localhost:2003)
   address = <hostname or ip>:<port#>
   prefix = prod.grafana.%(instance_name)s.
   ```

1. Restart Grafana. Grafana now exposes metrics at http://localhost:3000/metrics and sends them to the Graphite location you specified.

### Pull metrics from Grafana backend plugin into Prometheus

Any installed [backend plugin](https://grafana.com/developers/plugin-tools/key-concepts/backend-plugins/) exposes a metrics endpoint through Grafana that you can configure Prometheus to scrape.

These instructions assume you have already added Prometheus as a data source in Grafana.

1. Enable Prometheus to scrape backend plugin metrics from Grafana. In your configuration file (`grafana.ini` or `custom.ini` depending on your operating system) remove the semicolon to enable the following configuration options:

   ```
   # Metrics available at HTTP URL /metrics and /metrics/plugins/:pluginId
   [metrics]
   # Disable / Enable internal metrics
   enabled           = true

   # Disable total stats (stat_totals_*) metrics to be generated
   disable_total_stats = false
   ```

1. (optional) If you want to require authorization to view the metrics endpoints, then uncomment and set the following options:

   ```
   basic_auth_username =
   basic_auth_password =
   ```

1. Restart Grafana. Grafana now exposes metrics at `http://localhost:3000/metrics/plugins/<plugin id>`, e.g. http://localhost:3000/metrics/plugins/grafana-github-datasource if you have the [Grafana GitHub datasource](/grafana/plugins/grafana-github-datasource/) installed.
1. Add the job to your prometheus.yml file.
   Example:

   ```
   - job_name: 'grafana_github_datasource'

     scrape_interval: 15s
     scrape_timeout: 5s
     metrics_path: /metrics/plugins/grafana-test-datasource

     static_configs:
       - targets: ['localhost:3000']
   ```

1. Restart Prometheus. Your new job should appear on the Targets tab.
1. In Grafana, hover your mouse over the **Configuration** (gear) icon on the left sidebar and then click **Data Sources**.
1. Select the **Prometheus** data source.
1. Import a Golang application metrics dashboard - for example [Go Processes](/grafana/dashboards/6671).

# Set up Grafana HTTPS for secure web traffic

When accessing the Grafana UI through the web, it is important to set up HTTPS to ensure the communication between Grafana and the end user is encrypted, including login credentials and retrieved metric data.

In order to ensure secure traffic over the internet, Grafana must have a key for encryption and a [Secure Socket Layer (SSL) Certificate](https://www.kaspersky.com/resource-center/definitions/what-is-a-ssl-certificate) to verify the identity of the site.

The following image shows a browser lock icon which confirms the connection is safe.

{{< figure src="/media/docs/grafana/https-config/screenshot-secure-https.png" max-width="500px" caption="Secure HTTPS connection" >}}

This topic shows you how to:

1. Obtain a certificate and key
2. Configure Grafana HTTPS
3. Restart the Grafana server

## Before you begin

To follow these instructions, you need:

- You must have shell access to the system and `sudo` access to perform actions as root or administrator.
- For the CA-signed option, you need a domain name that you possess and that is associated with the machine you are using.

## Obtain a certificate and key

You can use one of two methods to obtain a certificate and a key. The faster and easier _self-signed_ option might show browser warnings to the user that they will have to accept each time they visit the site. Alternatively, the Certificate Authority (CA) signed option requires more steps to complete, but it enables full trust with the browser. To learn more about the difference between these options, refer to [Difference between self-signed CA and self-signed certificate](https://www.baeldung.com/cs/self-signed-ca-vs-certificate).

### Generate a self-signed certificate

This section shows you how to use `openssl` tooling to generate all necessary files from the command line.

1. Run the following command to generate a 2048-bit RSA private key, which is used to decrypt traffic:

   ```bash
   sudo openssl genrsa -out /etc/grafana/grafana.key 2048
   ```

1. Run the following command to generate a certificate, using the private key from the previous step.

   ```bash
   sudo openssl req -new -key /etc/grafana/grafana.key -out /etc/grafana/grafana.csr
   ```

   When prompted, answer the questions, which might include your fully-qualified domain name, email address, country code, and others. The following example is similar to the prompts you will see.

   ```
   You are about to be asked to enter information that will be incorporated
   into your certificate request.
   What you are about to enter is what is called a Distinguished Name or a DN.
   There are quite a few fields but you can leave some blank
   For some fields there will be a default value,
   If you enter '.', the field will be left blank.
   -----
   Country Name (2 letter code) [AU]:US
   State or Province Name (full name) [Some-State]:Virginia
   Locality Name (eg, city) []:Richmond
   Organization Name (eg, company) [Internet Pty Ltd]:
   Organizational Unit Name (eg, section) []:
   Common Name (e.g. server FQDN or YOUR name) []:subdomain.mysite.com
   Email Address []:me@mysite.com

   Please enter the following 'extra' attributes
   to be sent with your certificate request
   A challenge password []:
   An optional company name []:
   ```

1. Run the following command to self-sign the certificate with the private key, for a period of validity of 365 days:

   ```bash
   sudo openssl x509 -req -days 365 -in /etc/grafana/grafana.csr -signkey /etc/grafana/grafana.key -out /etc/grafana/grafana.crt
   ```

1. Run the following commands to set the appropriate permissions for the files:

   ```bash
   sudo chown grafana:grafana /etc/grafana/grafana.crt
   sudo chown grafana:grafana /etc/grafana/grafana.key
   sudo chmod 400 /etc/grafana/grafana.key /etc/grafana/grafana.crt
   ```

   **Note**: When using these files, browsers might provide warnings for the resulting website because a third-party source does not trust the certificate. Browsers will show trust warnings; however, the connection will remain encrypted.

   The following image shows an insecure HTTP connection.

   {{< figure src="/media/docs/grafana/https-config/screenshot-insecure-https.png" max-width="750px" caption="Insecure HTTPS connection" >}}

### Obtain a signed certificate from LetsEncrypt

[LetsEncrypt](https://letsencrypt.org/) is a nonprofit certificate authority that provides certificates without any charge. For signed certificates, there are multiple companies and certificate authorities (CAs) available. The principles for generating the certificates might vary slightly in accordance with the provider but will generally remain the same.

The examples in this section use LetsEncrypt because it is free.

{{< admonition type="note" >}}
The instructions provided in this section are for a Debian-based Linux system. For other distributions and operating systems, please refer to the [certbot instructions](https://certbot.eff.org/instructions). Also, these instructions require you to have a domain name that you are in control of. Dynamic domain names like those from Amazon EC2 or DynDNS providers will not function.
{{< /admonition >}}

#### Install `snapd` and `certbot`

`certbot` is an open-source program used to manage LetsEncrypt certificates, and `snapd` is a tool that assists in running `certbot` and installing the certificates.

1. To install `snapd`, run the following commands:

   ```bash
   sudo apt-get install snapd
   sudo snap install core; sudo snap refresh core
   ```

1. Run the following commands to install:

   ```bash
   sudo apt-get remove certbot
   sudo snap install --classic certbot
   sudo ln -s /snap/bin/certbot /usr/bin/certbot
   ```

   These commands:
   - Uninstall `certbot` from your system if it has been installed using a package manager
   - Install `certbot` using `snapd`

#### Generate certificates using `certbot`

The `sudo certbot certonly --standalone` command prompts you to answer questions before it generates a certificate. This process temporarily opens a service on port `80` that LetsEncrypt uses to verify communication with your host.

To generate certificates using `certbot`, complete the following steps:

1. Ensure that port `80` traffic is permitted by applicable firewall rules.

1. Run the following command to generate certificates:

   ```bash
   $ sudo certbot certonly --standalone

   Saving debug log to /var/log/letsencrypt/letsencrypt.log
   Enter email address (used for urgent renewal and security notices)
   (Enter 'c' to cancel): me@mysite.com

   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   Please read the Terms of Service at
   https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf. You must
   agree in order to register with the ACME server. Do you agree?
   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   (Y)es/(N)o: y

   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   Would you be willing, once your first certificate is successfully issued, to
   share your email address with the Electronic Frontier Foundation, a founding
   partner of the Let’s Encrypt project and the non-profit organization that
   develops Certbot? We’d like to send you email about our work encrypting the web,
   EFF news, campaigns, and ways to support digital freedom.
   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   (Y)es/(N)o: n
   Account registered.
   Please enter the domain name(s) you would like on your certificate (comma and/or
   space separated) (Enter 'c' to cancel): subdomain.mysite.com
   Requesting a certificate for subdomain.mysite.com

   Successfully received certificate.
   Certificate is saved at: /etc/letsencrypt/live/subdomain.mysite.com/fullchain.pem
   Key is saved at:         /etc/letsencrypt/live/subdomain.mysite.com/privkey.pem
   This certificate expires on 2023-06-20.
   These files will be updated when the certificate renews.
   Certbot has set up a scheduled task to automatically renew this certificate in the background.

   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   If you like Certbot, please consider supporting our work by:
   * Donating to ISRG / Let’s Encrypt:   https://letsencrypt.org/donate
   * Donating to EFF:                    https://eff.org/donate-le
   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   ```

#### Set up symlinks to Grafana

Symbolic links, also known as symlinks, enable you to create pointers to existing LetsEncrypt files in the `/etc/grafana` directory. By using symlinks rather than copying files, you can use `certbot` to refresh or request updated certificates from LetsEncrypt without the need to reconfigure the Grafana settings.

To set up symlinks to Grafana, run the following commands:

```bash
$ sudo ln -s /etc/letsencrypt/live/subdomain.mysite.com/privkey.pem /etc/grafana/grafana.key
$ sudo ln -s /etc/letsencrypt/live/subdomain.mysite.com/fullchain.pem /etc/grafana/grafana.crt
```

#### Adjust permissions

Grafana usually runs under the `grafana` Linux group, and you must ensure that the Grafana server process has permission to read the relevant files. Without read access, the HTTPS server fails to start properly.

To adjust permissions, perform the following steps:

1. Run the following commands to set the appropriate permissions and groups for the files:

   ```bash
   $ sudo chgrp -R grafana /etc/letsencrypt/*
   $ sudo chmod -R g+rx /etc/letsencrypt/*
   $ sudo chgrp -R grafana /etc/grafana/grafana.crt /etc/grafana/grafana.key
   $ sudo chmod 440 /etc/grafana/grafana.crt /etc/grafana/grafana.key
   ```

1. Run the following command to verify that the `grafana` group can read the symlinks:

   ```bash
   $ $ ls -l /etc/grafana/grafana.*

   lrwxrwxrwx 1 root grafana    67 Mar 22 14:15 /etc/grafana/grafana.crt -> /etc/letsencrypt/live/subdomain.mysite.com/fullchain.pem
   -rw-r----- 1 root grafana 54554 Mar 22 14:13 /etc/grafana/grafana.ini
   lrwxrwxrwx 1 root grafana    65 Mar 22 14:15 /etc/grafana/grafana.key -> /etc/letsencrypt/live/subdomain.mysite.com/privkey.pem
   ```

## Configure Grafana HTTPS and restart Grafana

In this section you edit the `grafana.ini` file so that it includes the certificate you created. If you need help identifying where to find this file, or what each key means, refer to [Configuration file location](../configure-grafana/#configuration-file-location).

To configure Grafana HTTPS and restart Grafana, complete the following steps.

1. Open the `grafana.ini` file and edit the following configuration parameters:

   ```
   [server]
   http_addr =
   http_port = 3000
   domain = mysite.com
   root_url = https://subdomain.mysite.com:3000
   cert_key = /etc/grafana/grafana.key
   cert_file = /etc/grafana/grafana.crt
   enforce_domain = False
   protocol = https
   ```

   > **Note**: The standard port for SSL traffic is 443, which you can use instead of Grafana's default port 3000. This change might require additional operating system privileges or configuration to bind to lower-numbered privileged ports.

1. Optional. From Grafana v11.2, edit the `cert_pass` configuration option with the decryption password if you are using encrypted certificates.

1. [Restart the Grafana server](../start-restart-grafana/#linux) using `systemd`, `init.d`, or the binary as appropriate for your environment.

## Troubleshooting

Refer to the following troubleshooting tips as required.

### Failure to obtain a certificate

The following reasons explain why the `certbot` process might fail:

- To make sure you can get a certificate from LetsEncrypt, you need to ensure that port 80 is open so that LetsEncrypt can communicate with your machine. If port 80 is blocked or firewall is enabled, the exchange will fail and you won't be able to receive a certificate.
- LetsEncrypt requires proof that you control the domain, so attempts to obtain certificates for domains you do not
  control might be rejected.

### Grafana starts, but HTTPS is unavailable

When you configure HTTPS, the following errors might appear in Grafana's logs.

#### Permission denied

```
level=error msg="Stopped background service" service=*api.HTTPServer reason="open /etc/grafana/grafana.crt: permission denied"
```

##### Resolution

To ensure secure HTTPS setup, it is essential that the cryptographic keys and certificates are as restricted as possible. However, if the file permissions are too restricted, the Grafana process may not have access to the necessary files, thus impeding a successful HTTPS setup. Please re-examine the listed instructions to double check the file permissions and try again.

#### Cannot assign requested address

```
listen tcp 34.148.30.243:3000: bind: cannot assign requested address
```

##### Resolution

Check the config to ensure the `http_addr` is left blank, allowing Grafana to bind to all interfaces. If you have set `http_addr` to a specific subdomain, such as `subdomain.mysite.com`, this might prevent the Grafana process from binding to an external address, due to network address translation layers being present.

# Sign in to Grafana

This topic describes how to sign in to Grafana.

## Before you begin

- [Install Grafana](../installation/)

## Steps

To sign in to Grafana for the first time, follow these steps:

1. Open your web browser and go to root URL specified in [Grafana configuration file](../configure-grafana/).

   Unless you have configured Grafana differently, it is set to use `http://localhost:3000` by default.

1. On the signin page, enter `admin` for username and password.
1. Click **Sign in**.

   If successful, you will see a prompt to change the password.

1. Click **OK** on the prompt and change your password.

> **Note:** We strongly recommend that you change the default administrator password.

# Start the Grafana server

This topic includes instructions for starting the Grafana server. For certain configuration changes, you might have to restart the Grafana server for them to take effect.

The following instructions start the `grafana-server` process as the `grafana` user, which was created during the package installation.

If you installed with the APT repository or `.deb` package, then you can start the server using `systemd` or `init.d`. If you installed a binary `.tar.gz` file, then you execute the binary.

## Linux

The following subsections describe three methods of starting and restarting the Grafana server: with systemd, initd, or by directly running the binary. You should follow only one set of instructions, depending on how your machine is configured.

### Start the Grafana server with systemd

Complete the following steps to start the Grafana server using systemd and verify that it is running.

1. To start the service, run the following commands:

   ```bash
   sudo systemctl daemon-reload
   sudo systemctl start grafana-server
   ```

1. To verify that the service is running, run the following command:

   ```bash
   sudo systemctl status grafana-server
   ```

### Configure the Grafana server to start at boot using systemd

To configure the Grafana server to start at boot, run the following command:

```bash
sudo systemctl enable grafana-server.service
```

#### Serve Grafana on a port < 1024

{{< docs/shared lookup="systemd/bind-net-capabilities.md" source="grafana" version="<GRAFANA VERSION>" >}}

### Restart the Grafana server using systemd

To restart the Grafana server, run the following command:

```bash
sudo systemctl restart grafana-server
```

{{< admonition type="note" >}}
SUSE or openSUSE users might need to start the server with the systemd method, then use the init.d method to configure Grafana to start at boot.
{{< /admonition >}}

### Start the Grafana server using init.d

Complete the following steps to start the Grafana server using init.d and verify that it is running:

1. To start the Grafana server, run the following command:

   ```bash
   sudo service grafana-server start
   ```

1. To verify that the service is running, run the following command:

   ```bash
   sudo service grafana-server status
   ```

### Configure the Grafana server to start at boot using init.d

To configure the Grafana server to start at boot, run the following command:

```bash
sudo update-rc.d grafana-server defaults
```

#### Restart the Grafana server using init.d

To restart the Grafana server, run the following command:

```bash
sudo service grafana-server restart
```

### Start the server using the binary

The `grafana` binary .tar.gz needs the working directory to be the root install directory where the binary and the `public` folder are located.

To start the Grafana server, run the following command:

```bash
./bin/grafana server
```

## Docker

To restart the Grafana service, use the `docker restart` command.

`docker restart grafana`

Alternatively, you can use the `docker compose restart` command to restart Grafana. For more information, refer to [docker compose documentation](https://docs.docker.com/compose/).

### Docker compose example

Configure your `docker-compose.yml` file. For example:

```yml
version: '3.8'
services:
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    environment:
      - TERM=linux
      - GF_PLUGINS_PREINSTALL=grafana-clock-panel,grafana-polystat-panel
    ports:
      - '3000:3000'
    volumes:
      - 'grafana_storage:/var/lib/grafana'
volumes:
  grafana_storage: {}
```

Start the Grafana server:

`docker compose up -d`

This starts the Grafana server container in detached mode along with the two plugins specified in the YAML file.

To restart the running container, use this command:

`docker compose restart grafana`

## Windows

Complete the following steps to start the Grafana server on Windows:

1. Execute `grafana.exe server`; the `grafana` binary is located in the `bin` directory.

   We recommend that you run `grafana.exe server` from the command line.

   If you want to run Grafana as a Windows service, you can download [NSSM](https://nssm.cc/).

1. To run Grafana, open your browser and go to the Grafana port (http://localhost:3000/ is default).

   > **Note:** The default Grafana port is `3000`. This port might require extra permissions on Windows. If it does not appear in the default port, you can try changing to a different port.

1. To change the port, complete the following steps:

   a. In the `conf` directory, copy `sample.ini` to `custom.ini`.

   > **Note:** You should edit `custom.ini`, never `defaults.ini`.

   b. Edit `custom.ini` and uncomment the `http_port` configuration option (`;` is the comment character in ini files) and change it to something similar to `8080`, which should not require extra Windows privileges.

To restart the Grafana server, complete the following steps:

1. Open the **Services** app.
1. Right-click on the **Grafana** service.
1. In the context menu, click **Restart**.

## macOS

Restart methods differ depending on whether you installed Grafana using Homebrew or as standalone macOS binaries.

### Start Grafana using Homebrew

To start Grafana using [Homebrew](http://brew.sh/), run the following start command:

```bash
brew services start grafana
```

### Restart Grafana using Homebrew

Use the [Homebrew](http://brew.sh/) restart command:

```bash
brew services restart grafana
```

### Restart standalone macOS binaries

To restart Grafana:

1. Open a terminal and go to the directory where you copied the install setup files.
1. Run the command:

```bash
./bin/grafana server
```

## Next steps

After the Grafana server is up and running, consider taking the next steps:

- Refer to [Get Started](../../getting-started/) to learn how to build your first dashboard.
- Refer to [Configuration](../configure-grafana/) to learn about how you can customize your environment.

# Configure Team Sync

Team sync lets you set up synchronization between your auth providers teams and teams in Grafana. This enables LDAP, OAuth, or SAML users who are members of certain teams or groups to automatically be added or removed as members of certain teams in Grafana.

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/docs/grafana-cloud/).
{{< /admonition >}}

Grafana keeps track of all synchronized users in teams, and you can see which users have been synchronized in the team members list, see `LDAP` label in screenshot.
This mechanism allows Grafana to remove an existing synchronized user from a team when its group membership changes. This mechanism also enables you to manually add a user as member of a team, and it will not be removed when the user signs in. This gives you flexibility to combine LDAP group memberships and Grafana team memberships.

> Currently the synchronization only happens when a user logs in, unless LDAP is used with the active background synchronization.

<div class="clearfix"></div>

## Supported providers

- [Auth Proxy](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/auth-proxy/#team-sync)
- [Entra ID](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/azuread/#team-sync)
- [Generic OAuth integration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/generic-oauth/#configure-team-synchronization)
- [GitHub OAuth](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/github/#configure-team-synchronization)
- [GitLab OAuth](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/gitlab/#configure-team-synchronization)
- [Google OAuth](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/google/#configure-team-synchronization)
- [LDAP](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/enhanced-ldap/)
- [Okta](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/okta/#configure-team-synchronization)
- [SAML](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/saml/)

## Synchronize a Grafana team with an external group

If you have already grouped some users into a team, then you can synchronize that team with an external group.

1. In Grafana, navigate to **Administration > Users and access > Teams**.
1. Select a team.
1. Go to the External group sync tab, and click **Add group**.

   ![External group sync](/static/img/docs/enterprise/team_add_external_group.png)

1. Insert the value of the group you want to sync with. This becomes the Grafana `GroupID`.
   Examples:
   - For LDAP, this is the LDAP distinguished name (DN) of LDAP group you want to synchronize with the team.
   - For Auth Proxy, this is the value we receive as part of the custom `Groups` header.

1. Click **Add group** to save.

> Group matching is case insensitive.

## LDAP specific: wildcard matching

When using LDAP, you can use a wildcard (\*) in the common name attribute (CN)
to match any group in the corresponding Organizational Unit (OU).

Ex: `cn=*,ou=groups,dc=grafana,dc=org` can be matched by `cn=users,ou=groups,dc=grafana,dc=org`

# Manage multi-team access in a single Grafana instance

If your organization has multiple teams using Grafana, you can use a single Grafana Enterprise deployment or a single Grafana Cloud stack to manage access across teams using roles and folders. This approach reduces complexity, simplifies identity and access management, and facilitates cross-team collaboration.

## Benefits

By using a single Grafana instance to manage access, you can:

- Implement a unified SSO, establishing clear permissions.
- Reduce setup and maintenance work, avoiding multi-stack complexity.
- Centralize plugin configuration and management.
- Ensure teams can access the right dashboards and data, avoiding stepping on or overwriting each other’s work.
- Enable collaboration across teams. Teams are not isolated in silos and can discover and collaborate with each other’s work.
- Optimize resource management. With shared spaces, like an “Everyone” folder, you can publish executive dashboards or cross-team metrics that all groups can benefit from, without duplicating it across stacks.

## Example: Three teams, one stack

Consider the following setup of three teams:

- Team A builds product features and needs autonomy with their own dashboards and data sources.
- Team B handles data engineering and needs autonomy with their own dashboards.
- Team C is the observability team and the admins of the Grafana stack.

Follow these suggested steps to structure, configure, and set permissions to access data in your Grafana instance:

1. [Before you begin](#before-you-begin)
1. [Create teams and configure user access](#create-teams-and-configure-user-access)
1. [Design a folder structure to match your access needs](#design-a-folder-structure-to-match-your-access-needs)
1. [Configure data access based on each team’s requirements](#configure-data-access-based-on-team-requirements)
1. [Scale access management with Terraform and SSO](#scale-access-management-with-terraform-and-sso)

### Before you begin

For more information on how to install a Grafana instance:

- If you’re using self-managed Grafana Enterprise, refer to [Configure Grafana](../../configure-grafana/).
- If you’re using Grafana Cloud, refer to [Your Grafana Cloud stack](https://grafana.com/docs/grafana-cloud/security-and-account-management/cloud-stacks).

{{< admonition type="note" >}}
For guidance on when to use one stack versus multiple, refer to [Stack architecture guidance](https://grafana.com/docs/grafana-cloud/security-and-account-management/cloud-stacks/stack-architecture-guidance/).
{{< /admonition >}}

### Create teams and configure user access

After you’ve deployed your Grafana instance:

- To follow the example in this doc, create three [Grafana Teams](../../../administration/team-management/configure-grafana-teams/#create-a-grafana-team) and add them to the Grafana instance.
- Determine the [RBAC](ref:rbac) strategy for your organization. RBAC extends default Grafana roles, provides more granular access rights, and simplifies how to grant, modify, or revoke user access to Grafana resources, such as users and reports.
- Assign each user to the [relevant team](../../../administration/user-management/manage-org-users/). By default [new users](../../configure-grafana/#auto_assign_org) are granted the **Viewer** role.
- Assign the [**Admin** role](ref:rbac-assign) to Team C so that they can manage all resources in the instance.

### Design a folder structure to match your access needs

To design a [folder](ref:create-folder) setup that helps users quickly understand where to go, what they can access, and what they can manage:

- Create an “Everyone” folder for shared items that all teams can manage, and grant teams Admin access to that folder.
- For each team, create a folder that they can manage and grant them the `fixed:teams:read` [fixed role](ref:rbac-assign). This means they can share items in their team folder with other teams, to encourage collaboration and learning from each other.
- For Team C, create an “Admins” folder for sensitive content only Admins can access.
- Optionally, create a personal folder for each team member so that they can work on draft content before moving it into their team folder when ready.

{{< figure src="/media/docs/grafana/oac/AccessTeams01.png" max-width="750px" alt="Teams and folders in the stack, and the related admin permissions Team A and Team B have been granted" >}}

### Configure data access based on team requirements

Next, focus on how teams interact with data to decide further access needs.

#### Shared baseline data access

Grant the `datasources:explorer` fixed role to all teams so they can use the [Drilldown apps](ref:drilldown) for easily exploring data sources.

However, you may need to protect data in shared resources. For example, all teams can be forwarding metrics to a shared [data source](ref:add-data-source), but not everyone needs to see all of the data. In this case, grant each team query access to the data relevant for them, based on [label based access controls (LBAC) per team](ref:lbac). This way, you’ll maintain a central observability pipeline but still preserve data separation.

#### Autonomous team data management

If any of your teams, Team A for example, need to build and manage their own data sources for product-specific use cases, grant the `datasources:creator` fixed role so they can create and manage their own data sources independently.

{{< figure src="/media/docs/grafana/oac/AccessTeams02.png" max-width="750px" alt="Teams and data sources in the stack, and the related permissions Team A and Team B have been granted" >}}

#### Resources at an instance level

Some Grafana resources, such as service accounts, alert contact points, [Fleet Management collectors](https://grafana.com/docs/grafana-cloud/send-data/fleet-management/), and other feature resources, are not linked to teams but are managed at the stack level. For these type of resources, assign fixed roles to teams carefully.

For example, users working in [Frontend Observability](https://grafana.com/docs/grafana-cloud/monitor-applications/frontend-observability/) need a writer fixed role so that they can create and manage services.

{{< figure src="/media/docs/grafana/oac/AccessTeams03.png" max-width="750px" alt="Grafana Cloud Frontend Observability resources in the stack, and the related permissions Team A have been granted" >}}

### Scale access management with Terraform and SSO

After you've made sure the model is working, you can codify it.

You can add any new users to your Grafana instance with an Identity Provider through [SCIM](../../configure-access/configure-authentication/). Use [role sync](../../../configure-access/configure-authentication/saml/configure-saml-team-role-mapping/#configure-role-sync-for-saml) to automatically assign users the correct basic role (Viewer, Editor, or Admin) based on their mapped attributes in the IdP..

You can also use Terraform to provision teams their folders, fixed roles, and shared data source LBAC rules. For example, if you need to add a new team (Team D), you only need to add the new team to Grafana and run the Terraform script, which will automatically set them up to start using Grafana.

{{< figure src="/media/docs/grafana/oac/AccessTeams04.png" max-width="750px" alt="Add new Team D from Okta and automate the rest of their IAM setup using Terraform" >}}

## Other resources

Read on to learn more about access management:

- The [Least privilege custom role explainer](https://grafana.com/blog/2024/09/10/grafana-access-management-how-to-use-teams-for-seamless-user-and-permission-management/) blog walks through how to design roles that keep things simple and safe, so your users have just the access they need.
- See the [LBAC for metrics data sources](https://www.youtube.com/watch?v=gj27qKPSVsM) demo to learn how you can give every team a clear view of their own data while still benefiting from a shared pipeline.
- The [Introducing SCIM](https://grafana.com/blog/2025/05/14/introducing-scim-provisioning-in-grafana-enterprise-grade-user-management-made-simple/) post covers how to connect Grafana to your identity provider, making it easy to bring new users on board and keep permissions in sync as your organization grows.

# Anonymous authentication

You can make Grafana accessible without any login required by enabling anonymous access in the configuration file.

{{< admonition type="note" >}}
Anonymous users are charged as active users in Grafana Enterprise
{{< /admonition >}}

## Before you begin

To see the devices, you need:

- Permissions `users:read` which is normally only granted to server admins, that allow you to read users and devices tab.

## Anonymous devices

The anonymous devices feature enhances the management and monitoring of anonymous access within your Grafana instance. This feature is part of ongoing efforts to provide more control and transparency over anonymous usage.

Users can now view anonymous usage statistics, including the count of devices and users over the last 30 days.

- Go to **Administration -> Users** to access the anonymous devices tab.
- A new stat for the usage stats page -> Usage & Stats page shows the active anonymous devices last 30 days.

The number of anonymous devices is not limited by default. The configuration option `device_limit` allows you to enforce a limit on the number of anonymous devices. This enables you to have greater control over the usage within your Grafana instance and keep the usage within the limits of your environment. Once the limit is reached, any new devices that try to access Grafana will be denied access.

## Configuration

Example:

```bash
[auth.anonymous]
enabled = true

# Organization name that should be used for unauthenticated users
org_name = Main Org.

# Role for unauthenticated users, other valid values are `Editor` and `Admin`
org_role = Viewer

# Hide the Grafana version text from the footer and help tooltip for unauthenticated users (default: false)
hide_version = true

# Setting this limits the number of anonymous devices in your instance. Any new anonymous devices added after the limit has been reached will be denied access.
device_limit =
```

If you change your organization name in the Grafana UI this setting needs to be updated to match the new name.

## Licensing for anonymous access

Grafana Enterprise (self-managed) licenses anonymous access as active users.

Anonymous access lets people use Grafana without login credentials. It was an early way to share dashboards, but Public dashboards gives you a more secure way to share dashboards.

### How anonymous usage is counted

Grafana estimates anonymous active users from anonymous devices:

- **Counting rule**: Grafana counts 1 anonymous user for every 3 anonymous devices detected.

# Configure auth proxy authentication

You can configure Grafana to let a HTTP reverse proxy handle authentication. Popular web servers have a very
extensive list of pluggable authentication modules, and any of them can be used with the AuthProxy feature.
Below we detail the configuration options for auth proxy.

```bash
[auth.proxy]
# Defaults to false, but set to true to enable this feature
enabled = true
# HTTP Header name that will contain the username or email
header_name = X-WEBAUTH-USER
# HTTP Header property, defaults to `username` but can also be `email`
header_property = username
# Set to `true` to enable auto sign up of users who do not exist in Grafana DB. Defaults to `true`.
auto_sign_up = true
# Define cache time to live in minutes
# If combined with Grafana LDAP integration it is also the sync interval
# Set to 0 to always fetch and sync the latest user data
sync_ttl = 15
# Limit where auth proxy requests come from by configuring a list of IP addresses.
# This can be used to prevent users spoofing the X-WEBAUTH-USER header.
# Example `whitelist = 192.168.1.1, 192.168.1.0/24, 2001::23, 2001::0/120`
whitelist =
# Optionally define more headers to sync other user attributes
# Example `headers = Name:X-WEBAUTH-NAME Role:X-WEBAUTH-ROLE Email:X-WEBAUTH-EMAIL Groups:X-WEBAUTH-GROUPS`
headers =
# Non-ASCII strings in header values are encoded using quoted-printable encoding
;headers_encoded = false
# Check out docs on this for more details on the below setting
enable_login_token = false
```

## Interacting with Grafana’s AuthProxy via curl

```bash
curl -H "X-WEBAUTH-USER: admin"  http://localhost:3000/api/users
[
    {
        "id":1,
        "name":"",
        "login":"admin",
        "email":"admin@localhost",
        "isAdmin":true
    }
]
```

We can then send a second request to the `/api/user` method which will return the details of the logged in user. We will use this request to show how Grafana automatically adds the new user we specify to the system. Here we create a new user called “anthony”.

```bash
curl -H "X-WEBAUTH-USER: anthony" http://localhost:3000/api/user
{
    "email":"anthony",
    "name":"",
    "login":"anthony",
    "theme":"",
    "orgId":1,
    "isGrafanaAdmin":false
}
```

## Making Apache’s auth work together with Grafana’s AuthProxy

I’ll demonstrate how to use Apache for authenticating users. In this example we use BasicAuth with Apache’s text file based authentication handler, i.e. htpasswd files. However, any available Apache authentication capabilities could be used.

### Apache BasicAuth

In this example we use Apache as a reverse proxy in front of Grafana. Apache handles the Authentication of users before forwarding requests to the Grafana backend service.

#### Apache configuration

```bash
    <VirtualHost *:80>
        ServerAdmin webmaster@authproxy
        ServerName authproxy
        ErrorLog "logs/authproxy-error_log"
        CustomLog "logs/authproxy-access_log" common

        <Proxy *>
            AuthType Basic
            AuthName GrafanaAuthProxy
            AuthBasicProvider file
            AuthUserFile /etc/apache2/grafana_htpasswd
            Require valid-user

            RewriteEngine On
            RewriteRule .* - [E=PROXY_USER:%{LA-U:REMOTE_USER},NS]
            RequestHeader set X-WEBAUTH-USER "%{PROXY_USER}e"
        </Proxy>

        RequestHeader unset Authorization

        ProxyRequests Off
        ProxyPass / http://localhost:3000/
        ProxyPassReverse / http://localhost:3000/
    </VirtualHost>
```

- The first four lines of the virtualhost configuration are standard, so we won’t go into detail on what they do.

- We use a **\<proxy>** configuration block for applying our authentication rules to every proxied request. These rules include requiring basic authentication where user:password credentials are stored in the **/etc/apache2/grafana_htpasswd** file. This file can be created with the `htpasswd` command.
  - The next part of the configuration is the tricky part. We use Apache’s rewrite engine to create our **X-WEBAUTH-USER header**, populated with the authenticated user.
    - **RewriteRule .\* - [E=PROXY_USER:%{LA-U:REMOTE_USER}, NS]**: This line is a little bit of magic. What it does, is for every request use the rewriteEngines look-ahead (LA-U) feature to determine what the REMOTE_USER variable would be set to after processing the request. Then assign the result to the variable PROXY_USER. This is necessary as the REMOTE_USER variable is not available to the RequestHeader function.

    - **RequestHeader set X-WEBAUTH-USER “%{PROXY_USER}e”**: With the authenticated username now stored in the PROXY_USER variable, we create a new HTTP request header that will be sent to our backend Grafana containing the username.

- The **RequestHeader unset Authorization** removes the Authorization header from the HTTP request before it is forwarded to Grafana. This ensures that Grafana does not try to authenticate the user using these credentials (BasicAuth is a supported authentication handler in Grafana).

- The last 3 lines are then just standard reverse proxy configuration to direct all authenticated requests to our Grafana server running on port 3000.

## Full walkthrough using Docker.

For this example, we use the official Grafana Docker image available at [Docker Hub](https://hub.docker.com/r/grafana/grafana/).

- Create a file `grafana.ini` with the following contents

```bash
[users]
allow_sign_up = false
auto_assign_org = true
auto_assign_org_role = Editor

[auth.proxy]
enabled = true
header_name = X-WEBAUTH-USER
header_property = username
auto_sign_up = true
```

Launch the Grafana container, using our custom `grafana.ini` to replace `/etc/grafana/grafana.ini`. We don't expose
any ports for this container as it will only be connected to by our Apache container.

```bash
docker run -i -v $(pwd)/grafana.ini:/etc/grafana/grafana.ini --name grafana grafana/grafana
```

### Apache Container

For this example we use the official Apache docker image available at [Docker Hub](https://hub.docker.com/_/httpd/)

- Create a file named `httpd.conf` with the following contents

```bash
ServerRoot "/usr/local/apache2"
Listen 80
LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule headers_module modules/mod_headers.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
<IfModule unixd_module>
User daemon
Group daemon
</IfModule>
ServerAdmin you@example.com
<Directory />
    AllowOverride none
    Require all denied
</Directory>
DocumentRoot "/usr/local/apache2/htdocs"
ErrorLog /proc/self/fd/2
LogLevel error
<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    <IfModule logio_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>
    CustomLog /proc/self/fd/1 common
</IfModule>
<Proxy *>
    AuthType Basic
    AuthName GrafanaAuthProxy
    AuthBasicProvider file
    AuthUserFile /tmp/htpasswd
    Require valid-user
    RewriteEngine On
    RewriteRule .* - [E=PROXY_USER:%{LA-U:REMOTE_USER},NS]
    RequestHeader set X-WEBAUTH-USER "%{PROXY_USER}e"
</Proxy>
RequestHeader unset Authorization
ProxyRequests Off
ProxyPass / http://grafana:3000/
ProxyPassReverse / http://grafana:3000/
```

- Create a `htpasswd` file. We create a new user **anthony** with the password **password**

  ```bash
  htpasswd -bc htpasswd anthony password
  ```

- Launch the Apache HTTP server container using our custom `httpd.conf` and our `htpasswd` file. The container will listen on port 80, and we create a link to the **grafana** container so that this container can resolve the hostname **grafana** to the Grafana container’s IP address.

  ```bash
  docker run -i -p 80:80 --link grafana:grafana -v $(pwd)/httpd.conf:/usr/local/apache2/conf/httpd.conf -v $(pwd)/htpasswd:/tmp/htpasswd httpd:2.4
  ```

### Use grafana.

With our Grafana and Apache containers running, you can now connect to http://localhost/ and log in using the username and password we created in the `htpasswd` file.

{{< admonition type="note" >}}
If the user is deleted from Grafana, the user will be not be able to login and resync until after the `sync_ttl` has expired.
{{< /admonition >}}

### Team Sync

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and to customers on select Grafana Cloud plans. For pricing information, visit [pricing](https://grafana.com/pricing/) or contact our sales team.

{{< /admonition >}}

With Team Sync, it's possible to set up synchronization between teams in your authentication provider and Grafana. You can send Grafana values as part of an HTTP header and have Grafana map them to your team structure. This allows you to put users into specific teams automatically.

To support the feature, auth proxy allows optional headers to map additional user attributes. The specific attribute to support team sync is `Groups`.

```bash
# Optionally define more headers to sync other user attributes
headers = "Groups:X-WEBAUTH-GROUPS"
```

You use the `X-WEBAUTH-GROUPS` header to send the team information for each user. Specifically, the set of Grafana's group IDs that the user belongs to.

First, we need to set up the mapping between your authentication provider and Grafana. Follow [these instructions](../../configure-team-sync/#synchronize-a-grafana-team-with-an-external-group) to add groups to a team within Grafana.

Once that's done. You can verify your mappings by querying the API.

```bash
# First, inspect your teams and obtain the corresponding ID of the team we want to inspect the groups for.
curl -H "X-WEBAUTH-USER: admin" -H "X-WEBAUTH-GROUPS: lokiteamOnExternalSystem" http://localhost:3000/api/teams/search
{
  "totalCount": 2,
  "teams": [
    {
      "id": 1,
      "orgId": 1,
      "name": "Core",
      "email": "core@grafana.com",
      "avatarUrl": "/avatar/327a5353552d2dc3966e2e646908f540",
      "memberCount": 1,
      "permission": 0
    },
    {
      "id": 2,
      "orgId": 1,
      "name": "Loki",
      "email": "loki@grafana.com",
      "avatarUrl": "/avatar/102f937d5344d33fdb37b65d430f36ef",
      "memberCount": 0,
      "permission": 0
    }
  ],
  "page": 1,
  "perPage": 1000
}

# Then, query the groups for that particular team. In our case, the Loki team which has an ID of "2".
curl -H "X-WEBAUTH-USER: admin" -H "X-WEBAUTH-GROUPS: lokiteamOnExternalSystem" http://localhost:3000/api/teams/2/groups
[
  {
    "orgId": 1,
    "teamId": 2,
    "groupId": "lokiTeamOnExternalSystem"
  }
]
```

Finally, whenever Grafana receives a request with a header of `X-WEBAUTH-GROUPS: lokiTeamOnExternalSystem`, the user under authentication will be placed into the specified team. Placement in multiple teams is supported by using comma-separated values e.g. `lokiTeamOnExternalSystem,CoreTeamOnExternalSystem`.

```bash
curl -H "X-WEBAUTH-USER: leonard" -H "X-WEBAUTH-GROUPS: lokiteamOnExternalSystem" http://localhost:3000/dashboards/home
{
  "meta": {
    "isHome": true,
    "canSave": false,
    ...
}
```

With this, the user `leonard` will be automatically placed into the Loki team as part of Grafana authentication.

{{< admonition type="note" >}}
An empty `X-WEBAUTH-GROUPS` or the absence of a groups header will remove the user from all teams.
{{< /admonition >}}

[Learn more about Team Sync](../../configure-team-sync/)

## Login token and session cookie

With `enable_login_token` set to `true` Grafana will, after successful auth proxy header validation, assign the user
a login token and cookie. You only have to configure your auth proxy to provide headers for the /login route.
Requests via other routes will be authenticated using the cookie.

Use the settings `login_maximum_inactive_lifetime_duration` and `login_maximum_lifetime_duration` under `[auth]` to control session
lifetime.

# Configure enhanced LDAP integration

The enhanced LDAP integration adds additional functionality on top of the [LDAP integration](../ldap/) available in the open source edition of Grafana.

{{< admonition type="note" >}}
Available in [Grafana Enterprise](../../../../introduction/grafana-enterprise/) and [Grafana Cloud](/docs/grafana-cloud).
If you are a Grafana Cloud customer, please [open a support ticket in the Cloud Portal](/profile/org#support) to request this feature.
{{< /admonition >}}

> To control user access with role-based permissions, refer to [role-based access control](../../../../administration/roles-and-permissions/access-control/).

## LDAP group synchronization for teams

With enhanced LDAP integration, you can set up synchronization between LDAP groups and teams. This enables LDAP users that are members
of certain LDAP groups to automatically be added or removed as members to certain teams in Grafana.

![LDAP group synchronization](/static/img/docs/enterprise/team_members_ldap.png)

Grafana keeps track of all synchronized users in teams, and you can see which users have been synchronized from LDAP in the team members list, see `LDAP` label in screenshot.
This mechanism allows Grafana to remove an existing synchronized user from a team when its LDAP group membership changes. This mechanism also allows you to manually add
a user as member of a team, and it will not be removed when the user signs in. This gives you flexibility to combine LDAP group memberships and Grafana team memberships.

[Learn more about team sync.](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync)

<div class="clearfix"></div>

## Active LDAP synchronization

In the open source version of Grafana, user data from LDAP is synchronized only during the login process when authenticating using LDAP.

With active LDAP synchronization, you can configure Grafana to actively sync users with LDAP servers in the background. Only users that have logged into Grafana at least once are synchronized.

Users with updated role and team membership will need to refresh the page to get access to the new features.

Removed users are automatically logged out and their account disabled. These accounts are displayed in the **Server Admin > Users** page with a `disabled` label. Disabled users keep their custom permissions on dashboards, folders, and data sources, so if you add them back in your LDAP database, they have access to the application with the same custom permissions as before.

```bash
[auth.ldap]
...

# You can use the Cron syntax or several predefined schedulers -
# @yearly (or @annually) | Run once a year, midnight, Jan. 1st        | 0 0 1 1 *
# @monthly               | Run once a month, midnight, first of month | 0 0 1 * *
# @weekly                | Run once a week, midnight between Sat/Sun  | 0 0 * * 0
# @daily (or @midnight)  | Run once a day, midnight                   | 0 0 * * *
# @hourly                | Run once an hour, beginning of hour        | 0 * * * *
sync_cron = "0 1 * * *" # This is default value (At 1 am every day)
# This cron expression format uses 5 space-separated fields, for example
# sync_cron = "*/10 * * * *"
# This will run the LDAP Synchronization every 10th minute, which is also the minimal interval between the Grafana sync times i.e. you cannot set it for every 9th minute

# You can also disable active LDAP synchronization
active_sync_enabled = true # enabled by default
```

Single bind configuration (as in the [Single bind example](../ldap/#single-bind-example)) is not supported with active LDAP synchronization because Grafana needs user information to perform LDAP searches.

For the synchronization to work, the `servers.search_filter` and `servers.attributes.username` in the `ldap.toml` configuration file must match. By default, the `servers.attributes.username` is `cn`, so if you use another attribute as the search filter, you must also update the username attribute.

For example:

```
[[servers]]
search_filter = "(sAMAccountName=%s)"

[servers.attributes]
username  = "sAMAccountName"
```

If the attributes aren't the same, the users' sessions will be terminated after each synchronization. That's because the search will be done using the username's value, and that value doesn't exist for the attribute used in the search filter.

# Configure Microsoft Entra ID OAuth authentication

The Microsoft Entra ID authentication allows you to use a Microsoft Entra ID (formerly known as Azure Active Directory) tenant as an identity provider for Grafana. You can use Entra ID application roles to assign users and groups to Grafana roles from the Azure Portal.

{{< admonition type="caution" >}}
If you use the same email address in Microsoft Entra ID and in other authentication providers (such as Grafana.com), you need to do additional configuration to ensure that your users are matched correctly.

Refer to [Using the same email address to login with different identity providers](../#using-the-same-email-address-to-login-with-different-identity-providers) for more information.
{{< /admonition >}}

## Register your application with Microsoft Entra ID

To enable the Entra ID OAuth, register your application with Entra ID.

1. Log in to the [Azure portal](https://portal.azure.com), then click **Microsoft Entra ID** in the side menu.

1. If you have access to more than one tenant, select your account in the upper right. Set your session to the Entra ID tenant you wish to use.

1. Under **Manage** in the side menu, click **App Registrations** > **New Registration**. Enter a descriptive name.

1. Under **Redirect URI**, select the app type **Web**.

1. Add the redirect URLs `https://<grafana domain>/login/azuread` and `https://<grafana domain>`, then click **Register**. The app's **Overview** page opens.

1. Note the **Application ID**. This is the OAuth client ID.

1. Click **Endpoints** from the top menu.
   - Note the **OAuth 2.0 authorization endpoint (v2)** URL. This is the authorization URL.
   - Note the **OAuth 2.0 token endpoint (v2)**. This is the token URL.

1. Click **Certificates & secrets** in the side menu, then add a new entry under the supported client authentication option you want to use. The following are the supported client authentication options with their respective configuration steps.
   - **Client secrets**
     1. Add a new entry under **Client secrets** with the following configuration.
        - Description: Grafana OAuth 2.0
        - Expires: Select an expiration period

     1. Click **Add** then copy the key **Value**. This is the OAuth 2.0 client secret.

     {{< admonition type="note" >}}
     Make sure that you copy the string in the **Value** field, rather than the one in the **Secret ID** field.
     {{< /admonition >}}
     1. You must have set `client_authentication` under `[auth.azuread]` to `client_secret_post` in the Grafana server configuration for this to work.

   - **Federated credentials**
     - **_Managed Identity_**
       1. Refer to [Configure an application to trust a managed identity (preview)](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation-config-app-trust-managed-identity?tabs=microsoft-entra-admin-center) for a complete guide on setting up a managed identity as a federated credential.
          Add a new entry under Federated credentials with the following configuration.
          - Federated credential scenario: Select **Other issuer**.
          - Issuer: The OAuth 2.0 / OIDC issuer URL of the Microsoft Entra ID authority. For example: `https://login.microsoftonline.com/{tenantID}/v2.0`.
          - Subject identifier: The Object (Principal) ID GUID of the Managed Identity.
          - Name: A unique descriptive name for the credential.
          - Description: Grafana OAuth.
          - Audience: The audience value that must appear in the external token. For Public cloud, it would be `api://AzureADTokenExchange`. See mentioned documentation for the full list of available audiences.

       1. Click **Add**, and then copy the Managed Identity Client ID and the federated credential Audience values. This is your OAuth 2.0 federated credential.

       1. You must have set `client_authentication` under `[auth.azuread]` to `managed_identity` in the Grafana server configuration for this to work.

       {{< admonition type="note" >}}
       Managed identities as federated credentials are only applicable to workloads hosted in Azure.

       You can only add user-assigned managed identities as federated credentials on Entra ID applications.
       {{< /admonition >}}

     - **_Workload Identity (K8s/AKS)_**
       1. Refer to [Federated identity credential for an Entra ID application](https://azure.github.io/azure-workload-identity/docs/topics/federated-identity-credential.html#azure-portal-ui) for a complete guide on setting up a federated credential for workload identity.
          Add a new entry under Federated credentials with the following configuration.
          - Federated credential scenario: Select **Kubernetes accessing Azure resources**.
          - [Cluster issuer URL](https://learn.microsoft.com/en-us/azure/aks/use-oidc-issuer#get-the-oidc-issuer-url): The OIDC issuer URL that your cluster is integrated with. For example: `https://{region}.oic.prod-aks.azure.com/{tenant_id}/{uuid}`.
          - Namespace: Namespace of your Grafana deployment. For example: `grafana`.
          - Service account name: Service account name of your Grafana deployment. For example: `grafana`.
          - Subject identifier: The expected identity (subject claim) from the OIDC token, which Azure uses to validate and authorize token issuance to the requesting workload. For example: `system:serviceaccount:grafana:grafana`.
          - Name: A unique descriptive name for the credential.
          - Description: Grafana OAuth.
          - Audience: The audience value that must appear in the external token. For Public cloud, it would be `api://AzureADTokenExchange`. See mentioned documentation for the full list of available audiences.

       1. You must have set `client_authentication` (env var `GF_AUTH_AZUREAD_CLIENT_AUTHENTICATION`) under `[auth.azuread]` to `workload_identity` in the Grafana server configuration for this to work.

       1. You may optionally set `workload_identity_token_file` under `[auth.azuread]` to `$__env{AZURE_FEDERATED_TOKEN_FILE}` in the Grafana server configuration for this to work. You may alternatively set it through the env var `GF_AUTH_AZUREAD_WORKLOAD_IDENTITY_TOKEN_FILE="${AZURE_FEDERATED_TOKEN_FILE}"`. When unset, Grafana defaults to the legacy path `/var/run/secrets/azure/tokens/azure-identity-token`.

       1. You must have set `client_id` (env var `GF_AUTH_AZUREAD_CLIENT_ID`) under `[auth.azuread]` in the Grafana server configuration for this to work. This must match the Entra ID/Entra ID App Registration Application (client) ID.

       1. You must have set `token_url` (env var `GF_AUTH_AZUREAD_TOKEN_URL`) under `[auth.azuread]` to `https://login.microsoftonline.com/{tenantID}/oauth2/v2.0/token` in the Grafana server configuration for this to work.

       1. You must have set `auth_url` (env var `GF_AUTH_AZUREAD_AUTH_URL`) under `[auth.azuread]` to `https://login.microsoftonline.com/{tenantID}/oauth2/v2.0/authorize` in the Grafana server configuration for this to work.

       1. You must have set `federated_credential_audience` (env var `GF_AUTH_AZUREAD_FEDERATED_CREDENTIAL_AUDIENCE`) under `[auth.azuread]` to `api://AzureADTokenExchange` in the Grafana server configuration for this to work.

     {{< admonition type="note" >}}
     Managed identities as federated credentials are only applicable to workloads hosted in Azure.

     You can only add user-assigned managed identities as federated credentials on Entra ID applications.
     {{< /admonition >}}

1. Define the required application roles for Grafana [using the Azure Portal](#configure-application-roles-for-grafana-in-the-azure-portal) or [using the manifest file](#configure-application-roles-for-grafana-in-the-manifest-file).

1. Go to **Microsoft Entra ID** and then to **Enterprise Applications**, under **Manage**.

1. Search for your application and click it.

1. Click **Users and Groups**.
1. Click **Add user/group** to add a user or group to the Grafana roles.

{{< admonition type="note" >}}
When assigning a group to a Grafana role, ensure that users are direct members of the group. Users in nested groups will not have access to Grafana due to limitations within Entra ID side. For more information, see [Microsoft Entra service limits and restrictions](https://learn.microsoft.com/en-us/entra/identity/users/directory-service-limits-restrictions).
{{< /admonition >}}

### Configure application roles for Grafana in the Microsoft Entra ID portal

This section describes setting up basic application roles for Grafana within the Entra ID Portal. For more information, see [Add app roles to your application and receive them in the token](https://learn.microsoft.com/en-us/entra/identity-platform/howto-add-app-roles-in-apps).

1. Go to **App Registrations**, search for your application, and click it.

1. Click **App roles** and then **Create app role**.

1. Define a role corresponding to each Grafana role: Viewer, Editor, and Admin.
   1. Choose a **Display name** for the role. For example, "Grafana Editor".

   1. Set the **Allowed member types** to **Users/Groups**.

   1. Ensure that the **Value** field matches the Grafana role name. For example, "Editor".

   1. Choose a **Description** for the role. For example, "Grafana Editor Users".

   1. Click **Apply**.

### Configure application roles for Grafana in the manifest file

If you prefer to configure the application roles for Grafana in the manifest file, complete the following steps:

1. Go to **App Registrations**, search for your application, and click it.

1. Click **Manifest**.

1. Add a Universally Unique Identifier to each role.

{{< admonition type="note" >}}
Every role requires a [Universally Unique Identifier](https://en.wikipedia.org/wiki/Universally_unique_identifier) which you can generate on Linux with `uuidgen`, and on Windows through Microsoft PowerShell with `New-Guid`.
{{< /admonition >}}

1. Replace each "SOME_UNIQUE_ID" with the generated ID in the manifest file:

   ```json
   	"appRoles": [
   			{
   				"allowedMemberTypes": [
   					"User"
   				],
   				"description": "Grafana org admin Users",
   				"displayName": "Grafana Org Admin",
   				"id": "SOME_UNIQUE_ID",
   				"isEnabled": true,
   				"lang": null,
   				"origin": "Application",
   				"value": "Admin"
   			},
   			{
   				"allowedMemberTypes": [
   					"User"
   				],
   				"description": "Grafana read only Users",
   				"displayName": "Grafana Viewer",
   				"id": "SOME_UNIQUE_ID",
   				"isEnabled": true,
   				"lang": null,
   				"origin": "Application",
   				"value": "Viewer"
   			},
   			{
   				"allowedMemberTypes": [
   					"User"
   				],
   				"description": "Grafana Editor Users",
   				"displayName": "Grafana Editor",
   				"id": "SOME_UNIQUE_ID",
   				"isEnabled": true,
   				"lang": null,
   				"origin": "Application",
   				"value": "Editor"
   			}
   		],
   ```

1. Click **Save**.

### Assign server administrator privileges

Use the application role `GrafanaAdmin` to grant users server administrator privileges. This is useful if you want to grant server administrator privileges to a subset of users.

Grafana also assigns the user the `Admin` role of the default organization.

Set the setting `allow_assign_grafana_admin` under `[auth.azuread]` to `true` for this to work.
If the setting is set to `false`, the user is assigned the role of `Admin` of the default organization, but not server administrator privileges.

```json
{
  "allowedMemberTypes": ["User"],
  "description": "Grafana server admin Users",
  "displayName": "Grafana Server Admin",
  "id": "SOME_UNIQUE_ID",
  "isEnabled": true,
  "lang": null,
  "origin": "Application",
  "value": "GrafanaAdmin"
}
```

## Configure the Entra ID authentication client using the Grafana UI

After you've registered your application with Entra Id, As a Grafana Admin, you can configure your Entra ID OAuth client from within Grafana using the Grafana UI. To do this, navigate to the **Administration > Authentication > Entra ID** page and fill in the form. If you have a current configuration in the Grafana configuration file, the form will be pre-populated with those values. Otherwise the form will contain default values.

After you have filled in the form, click **Save** to save the configuration. If the save was successful, Grafana will apply the new configurations.

If you need to reset changes you made in the UI back to the default values, click **Reset**. After you have reset the changes, Grafana will apply the configuration from the Grafana configuration file (if there is any configuration) or the default values.

{{< admonition type="note" >}}
If you run Grafana in high availability mode, configuration changes may not get applied to all Grafana instances immediately. You may need to wait a few minutes for the configuration to propagate to all Grafana instances.
{{< /admonition >}}

## Configure the Entra ID authentication client using the Terraform provider

```terraform
resource "grafana_sso_settings" "azuread_sso_settings" {
  provider_name = "azuread"
  oauth2_settings {
    name                          = "Entra ID"
    auth_url                      = "https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/authorize"
    token_url                     = "https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token"
    client_authentication         = "CLIENT_AUTHENTICATION_OPTION"
    client_id                     = "APPLICATION_ID"
    client_secret                 = "CLIENT_SECRET"
    managed_identity_client_id    = "MANAGED_IDENTITY_CLIENT_ID"
    federated_credential_audience = "FEDERATED_CREDENTIAL_AUDIENCE"
    allow_sign_up                 = true
    auto_login                    = false
    scopes                        = "openid email profile"
    allowed_organizations         = "TENANT_ID"
    role_attribute_strict         = false
    allow_assign_grafana_admin    = false
    skip_org_role_sync            = false
    use_pkce                      = true
    custom = {
      domain_hint = "contoso.com"
      force_use_graph_api = "true"
    }
  }
}
```

Refer to [Terraform Registry](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/sso_settings) for a complete reference on using the `grafana_sso_settings` resource.

## Configure the Entra ID authentication client using the Grafana configuration file

Ensure that you have access to the [Grafana configuration file](../../../configure-grafana/#configuration-file-location).

### Enable Entra ID OAuth in Grafana

Add the following to the [Grafana configuration file](../../../configure-grafana/#configuration-file-location):

```
[auth.azuread]
name = Entra ID
enabled = true
allow_sign_up = true
auto_login = false
client_authentication = CLIENT_AUTHENTICATION_OPTION
client_id = APPLICATION_ID
client_secret = CLIENT_SECRET
managed_identity_client_id = MANAGED_IDENTITY_CLIENT_ID
federated_credential_audience = FEDERATED_CREDENTIAL_AUDIENCE
scopes = openid email profile
auth_url = https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/authorize
token_url = https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token
allowed_domains =
allowed_groups =
allowed_organizations = TENANT_ID
role_attribute_strict = false
allow_assign_grafana_admin = false
skip_org_role_sync = false
use_pkce = true
```

You can also use these environment variables to configure `client_authentication`, `client_id`, `client_secret`, `managed_identity_client_id`, and `federated_credential_audience`:

```
GF_AUTH_AZUREAD_CLIENT_AUTHENTICATION
GF_AUTH_AZUREAD_CLIENT_ID
GF_AUTH_AZUREAD_CLIENT_SECRET
GF_AUTH_AZUREAD_MANAGED_IDENTITY_CLIENT_ID
GF_AUTH_AZUREAD_FEDERATED_CREDENTIAL_AUDIENCE
```

{{< admonition type="note" >}}
Verify that the Grafana [root_url](../../../configure-grafana/#root_url) is set in your Azure Application Redirect URLs.
{{< /admonition >}}

### Configure refresh token

When a user logs in using an OAuth provider, Grafana verifies that the access token has not expired. When an access token expires, Grafana uses the provided refresh token (if any exists) to obtain a new access token.

Grafana uses a refresh token to obtain a new access token without requiring the user to log in again. If a refresh token doesn't exist, Grafana logs the user out of the system after the access token has expired.

Refresh token fetching and access token expiration check is enabled by default for the Entra ID provider since Grafana v10.1.0. If you would like to disable access token expiration check then set the `use_refresh_token` configuration value to `false`.

{{< admonition type="note" >}}
The `accessTokenExpirationCheck` feature toggle has been removed in Grafana v10.3.0 and the `use_refresh_token` configuration value will be used instead for configuring refresh token fetching and access token expiration check.
{{< /admonition >}}

### Configure allowed tenants

To limit access to authenticated users who are members of one or more tenants, set `allowed_organizations`
to a _comma-_ or _space-separated_ list of tenant IDs. You can find tenant IDs on the Azure portal under **Microsoft Entra ID -> Overview**.

Make sure to include the tenant IDs of all the federated Users' root directory if your Entra ID contains external identities.

For example, if you want to only give access to members of the tenant `example` with an ID of `8bab1c86-8fba-33e5-2089-1d1c80ec267d`, then set the following:

```
allowed_organizations = 8bab1c86-8fba-33e5-2089-1d1c80ec267d
```

### Configure allowed groups

Microsoft Entra ID groups can be used to limit user access to Grafana. For more information about managing groups in Entra ID, refer to [Manage Microsoft Entra groups and group membership](https://learn.microsoft.com/en-us/entra/fundamentals/how-to-manage-groups).

To limit access to authenticated users who are members of one or more Entra ID groups, set `allowed_groups`
to a _comma-_ or _space-separated_ list of group object IDs.

1. To find object IDs for a specific group on the Azure portal, go to **Microsoft Entra ID > Manage > Groups**.

   You can find the Object Id of a group by clicking on the group and then clicking on **Properties**. The object ID is listed under **Object ID**. If you want to only give access to members of the group `example` with an Object Id of `8bab1c86-8fba-33e5-2089-1d1c80ec267d`, then set the following:

   ```
     allowed_groups = 8bab1c86-8fba-33e5-2089-1d1c80ec267d
   ```

1. You must enable adding the [group attribute](https://learn.microsoft.com/en-us/entra/identity-platform/optional-claims#configure-groups-optional-claims) to the tokens in your Entra ID App registration either [from the Azure Portal](#configure-group-membership-claims-on-the-azure-portal) or [from the manifest file](#configure-group-membership-claim-in-the-manifest-file).

#### Configure group membership claims on the Azure Portal

To ensure that the `groups` claim is included in the token, add the `groups` claim to the token configuration either through the Azure Portal UI or by editing the manifest file.

To configure group membership claims from the Azure Portal UI, complete the following steps:

1. Navigate to the **App Registrations** page and select your application.
1. Under **Manage** in the side menu, select **Token configuration**.
1. Click **Add groups claim** and select the relevant option for your use case (for example, **Security groups** and **Groups assigned to the application**).

For more information, see [Configure groups optional claims](https://learn.microsoft.com/en-us/entra/identity-platform/optional-claims#configure-groups-optional-claims).

{{< admonition type="note" >}}
If the user is a member of more than 200 groups, Entra ID does not emit the groups claim in the token and instead emits a group overage claim. To set up a group overage claim, see [Users with over 200 Group assignments](#users-with-over-200-group-assignments).
{{< /admonition >}}

#### Configure group membership claim in the manifest file

1. Go to **App Registrations**, search for your application, and click it.

1. Click **Manifest**.

1. Add the following to the root of the manifest file:

   ```
   "groupMembershipClaims": "ApplicationGroup, SecurityGroup"
   ```

### Configure allowed domains

The `allowed_domains` option limits access to users who belong to specific domains. Separate domains with space or comma. For example,

```
allowed_domains = mycompany.com mycompany.org
```

### PKCE

IETF's [RFC 7636](https://datatracker.ietf.org/doc/html/rfc7636)
introduces "proof key for code exchange" (PKCE) which provides
additional protection against some forms of authorization code
interception attacks. PKCE will be required in [OAuth 2.1](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1-03).

> You can disable PKCE in Grafana by setting `use_pkce` to `false` in the`[auth.azuread]` section.

### Configure automatic login

To bypass the login screen and log in automatically, enable the "auto_login" feature.
This setting is ignored if multiple auth providers are configured to use auto login.

```
auto_login = true
```

### Team Sync

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and to customers on select Grafana Cloud plans. For pricing information, visit [pricing](https://grafana.com/pricing/) or contact our sales team.
{{< /admonition >}}

With Team Sync you can map your Entra ID groups to teams in Grafana so that your users will automatically be added to
the correct teams.

You can reference Entra ID groups by group object ID, like `8bab1c86-8fba-33e5-2089-1d1c80ec267d`.

To learn more, refer to the [Team Sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync) documentation.

## Skip organization role sync

If you don't want Entra ID authentication to sync user roles and organization membership and prevent the sync of org roles from Entra ID, set `skip_org_role_sync` to `true`. Use this if you want to manage the organization roles for your users from within Grafana or that your organization roles are synced from another provider.
See [Configure Grafana](../../../configure-grafana/#authazuread) for more details.

```ini
[auth.azuread]
# ..
# prevents the sync of org roles from Entra ID
skip_org_role_sync = true
```

## Configuration options

The following table outlines the various Entra ID configuration options. You can apply these options as environment variables, similar to any other configuration within Grafana. For more information, refer to [Override configuration with environment variables](../../../configure-grafana/#override-configuration-with-environment-variables).

| Setting                         | Required | Supported on Cloud | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Default                                              |
| ------------------------------- | -------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| `enabled`                       | No       | Yes                | Enables Entra ID authentication.                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `false`                                              |
| `name`                          | No       | Yes                | Name that refers to the Entra ID authentication from the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                | `OAuth`                                              |
| `icon`                          | No       | Yes                | Icon used for the Entra ID authentication in the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                        | `signin`                                             |
| `client_authentication`         | Yes      | Yes                | Defines the client authentication method used to authenticate to the token endpoint. Supported values: `none`, `client_secret_post`, `managed_identity`, or `workload_identity`.                                                                                                                                                                                                                                                                                                                |                                                      |
| `workload_identity_token_file`  | No       | Yes                | The path to the token file used to authenticate to the OAuth2 provider. This is only required when `client_authentication` is set to `workload_identity`. The token file contains the service account token projected by Kubernetes. It should have the same value as the `AZURE_FEDERATED_TOKEN_FILE` env var.                                                                                                                                                                                 | `/var/run/secrets/azure/tokens/azure-identity-token` |
| `federated_credential_audience` | No       | Yes                | The audience of the federated identity credential of your OAuth2 app. Required when `client_authentication` is set to `managed_identity` or `workload_identity`. For public cloud, this is typically `api://AzureADTokenExchange`.                                                                                                                                                                                                                                                              | `api://AzureADTokenExchange`                         |
| `client_id`                     | Yes      | Yes                | Client ID of the App (`Application (client) ID` on the **App registration** dashboard).                                                                                                                                                                                                                                                                                                                                                                                                         |                                                      |
| `client_secret`                 | Yes      | Yes                | Client secret of the App.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |                                                      |
| `auth_url`                      | Yes      | Yes                | Authorization endpoint of the Entra ID OAuth2 provider.                                                                                                                                                                                                                                                                                                                                                                                                                                         |                                                      |
| `token_url`                     | Yes      | Yes                | Endpoint used to obtain the OAuth2 access token.                                                                                                                                                                                                                                                                                                                                                                                                                                                |                                                      |
| `token_exchange_timeout`        | No       | Yes                | Timeout in seconds for the OAuth token exchange request. If set to `0` or left unset, defaults to `15` seconds.                                                                                                                                                                                                                                                                                                                                                                                 | `15`                                                 |
| `auth_style`                    | No       | Yes                | Name of the [OAuth2 AuthStyle](https://pkg.go.dev/golang.org/x/oauth2#AuthStyle) to be used when ID token is requested from OAuth2 provider. It determines how `client_id` and `client_secret` are sent to Oauth2 provider. Available values are `AutoDetect`, `InParams` and `InHeader`.                                                                                                                                                                                                       | `AutoDetect`                                         |
| `scopes`                        | No       | Yes                | List of comma- or space-separated OAuth2 scopes.                                                                                                                                                                                                                                                                                                                                                                                                                                                | `openid email profile`                               |
| `allow_sign_up`                 | No       | Yes                | Controls Grafana user creation through the Entra ID login. Only existing Grafana users can log in with Entra ID if set to `false`.                                                                                                                                                                                                                                                                                                                                                              | `true`                                               |
| `auto_login`                    | No       | Yes                | Set to `true` to enable users to bypass the login screen and automatically log in. This setting is ignored if you configure multiple auth providers to use auto-login.                                                                                                                                                                                                                                                                                                                          | `false`                                              |
| `login_prompt`                  | No       | Yes                | Indicates the type of user interaction when the user logs in with Entra ID. Available values are `login`, `consent` and `select_account`.                                                                                                                                                                                                                                                                                                                                                       |                                                      |
| `role_attribute_strict`         | No       | Yes                | Set to `true` to deny user login if the Grafana org role cannot be extracted using `role_attribute_path` or `org_mapping`. For more information on user role mapping, refer to [Map roles](#map-roles).                                                                                                                                                                                                                                                                                         | `false`                                              |
| `org_attribute_path`            | No       | No                 | [JMESPath](http://jmespath.org/examples.html) expression to use for Grafana org to role lookup. Grafana will first evaluate the expression using the OAuth2 ID token. If no value is returned, the expression will be evaluated using the user information obtained from the UserInfo endpoint. The result of the evaluation will be mapped to org roles based on `org_mapping`. For more information on org to role mapping, refer to [Org roles mapping example](#org-roles-mapping-example). |                                                      |
| `org_mapping`                   | No       | No                 | List of comma- or space-separated `<ExternalOrgName>:<OrgIdOrName>:<Role>` mappings. Value can be `*` meaning "All users". Role is optional and can have the following values: `None`, `Viewer`, `Editor` or `Admin`. For more information on external organization to role mapping, refer to [Org roles mapping example](#org-roles-mapping-example).                                                                                                                                          |                                                      |
| `allow_assign_grafana_admin`    | No       | No                 | Set to `true` to automatically sync the Grafana server administrator role. When enabled, if the Entra ID user's App role is `GrafanaAdmin`, Grafana grants the user server administrator privileges and the organization administrator role. If disabled, the user will only receive the organization administrator role. For more details on user role mapping, refer to [Map roles](#map-roles).                                                                                              | `false`                                              |
| `skip_org_role_sync`            | No       | Yes                | Set to `true` to stop automatically syncing user roles. This will allow you to set organization roles for your users from within Grafana manually.                                                                                                                                                                                                                                                                                                                                              | `false`                                              |
| `allowed_groups`                | No       | Yes                | List of comma- or space-separated groups. The user should be a member of at least one group to log in. If you configure `allowed_groups`, you must also configure Entra ID to include the `groups` claim following [Configure group membership claims on the Azure Portal](#configure-group-membership-claims-on-the-azure-portal).                                                                                                                                                             |                                                      |
| `allowed_organizations`         | No       | Yes                | List of comma- or space-separated Azure tenant identifiers. The user should be a member of at least one tenant to log in.                                                                                                                                                                                                                                                                                                                                                                       |                                                      |
| `allowed_domains`               | No       | Yes                | List of comma- or space-separated domains. The user should belong to at least one domain to log in.                                                                                                                                                                                                                                                                                                                                                                                             |                                                      |
| `domain_hint`                   | No       | Yes                | The realm of the user in a federated directory. This skips the email-based discovery process that the user goes through on the Entra ID sign-in page, for a slightly more streamlined user experience. More info [here](https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols-oidc#send-the-sign-in-request).                                                                                                                                                                  |                                                      |
| `tls_skip_verify_insecure`      | No       | No                 | If set to `true`, the client accepts any certificate presented by the server and any host name in that certificate. _You should only use this for testing_, because this mode leaves SSL/TLS susceptible to man-in-the-middle attacks.                                                                                                                                                                                                                                                          | `false`                                              |
| `tls_client_cert`               | No       | No                 | The path to the certificate.                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |                                                      |
| `tls_client_key`                | No       | No                 | The path to the key.                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                      |
| `tls_client_ca`                 | No       | No                 | The path to the trusted certificate authority list.                                                                                                                                                                                                                                                                                                                                                                                                                                             |                                                      |
| `use_pkce`                      | No       | Yes                | Set to `true` to use [Proof Key for Code Exchange (PKCE)](https://datatracker.ietf.org/doc/html/rfc7636). Grafana uses the SHA256 based `S256` challenge method and a 128 bytes (base64url encoded) code verifier.                                                                                                                                                                                                                                                                              | `true`                                               |
| `use_refresh_token`             | No       | Yes                | Enables the use of refresh tokens and checks for access token expiration. When enabled, Grafana automatically adds the `offline_access` scope to the list of scopes.                                                                                                                                                                                                                                                                                                                            | `true`                                               |
| `force_use_graph_api`           | No       | Yes                | Set to `true` to always fetch groups from the Microsoft Graph API instead of the `id_token`. If a user belongs to more than 200 groups, the Microsoft Graph API will be used to retrieve the groups regardless of this setting.                                                                                                                                                                                                                                                                 | `false`                                              |
| `signout_redirect_url`          | No       | Yes                | URL to redirect to after the user logs out.                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

## Common troubleshooting

Here are some common issues and particulars you can run into when configuring Entra ID authentication in Grafana.

### Users with over 200 Group assignments

To ensure that the token size doesn't exceed HTTP header size limits,
Entra ID limits the number of object IDs that it includes in the groups claim.
If a user is member of more groups than the coverage limit (200), then Entra ID does not emit the groups claim in the token and emits a group overage claim instead.

> More information in [Groups overage claim](https://learn.microsoft.com/en-us/entra/identity-platform/id-token-claims-reference#groups-overage-claim)

If Grafana receives a token with a group overage claim instead of a groups claim,
Grafana attempts to retrieve the user's group membership by calling the included endpoint.

The Entra ID `App registration` must include the following API permissions for group overage claim calls to succeed:

| Permissions name       | Type      | Admin consent required | Status  |
| ---------------------- | --------- | ---------------------- | ------- |
| `GroupMember.Read.All` | Delegated | Yes                    | Granted |
| `User.Read`            | Delegated | No                     | Granted |

Admin consent is required for the `GroupMember.Read.All` permission. To grant admin consent, navigate to **API permissions** in the **App registration** and select **Grant admin consent for [your-organization]**.

{{< admonition type="note" >}}
You can make Grafana always get group information from the Microsoft Graph API by turning on the [`force_use_graph_api`](./#force-fetching-groups-from-microsoft-graph-api) setting in the configuration.
{{< /admonition >}}

#### Configure the required Graph API permissions

1. Navigate to **Microsoft Entra ID > Manage > App registrations** and select your application.
1. Select **API permissions** and then click on **Add a permission**.
1. Select **Microsoft Graph** from the list of APIs.
1. Select **Delegated permissions**.
1. Under the **GroupMember** section, select **GroupMember.Read.All**.
1. Click **Add permissions**.
1. Select **Microsoft Graph** from the list of APIs.
1. Select **Delegated permissions**.
1. In the **Select permissions** pane, under the **User** section, select **User.Read**.
1. Click the **Add permissions** button at the bottom of the page.

{{< admonition type="note" >}}
Admin consent may be required for this permission.
{{< /admonition >}}

### Force fetching groups from Microsoft Graph API

To force fetching groups from Microsoft Graph API instead of the `id_token`, you can use the `force_use_graph_api` configuration option.

```ini
[auth.azuread]
force_use_graph_api = true
```

### Map roles

By default, Entra ID authentication will map users to organization roles based on the most privileged application role assigned to the user in Entra ID.

If no application role is found, the user is assigned the role specified by
[the `auto_assign_org_role` option](../../../configure-grafana/#auto_assign_org_role).
You can disable this default role assignment by setting `role_attribute_strict = true`. This setting denies user access if no role or an invalid role is returned and the `org_mapping` expression evaluates to an empty mapping.

You can use the `org_mapping` configuration option to assign the user to multiple organizations and specify their role based on their Entra ID group membership. For more information, refer to [Org roles mapping example](#org-roles-mapping-example). If the org role mapping (`org_mapping`) is specified and Entra ID returns a valid role, then the user will get the highest of the two roles.

_On every login_ the user organization role will be reset to match Entra ID's application role and
their organization membership will be reset to the default organization.

#### Org roles mapping example

The Entra ID integration uses the external users' groups in the `org_mapping` configuration to map organizations and roles based on their Entra ID group membership.

In this example, the user has been granted the role of a `Viewer` in the `org_foo` organization, and the role of an `Editor` in the `org_bar` and `org_baz` orgs.

The external user is part of the following Entra ID groups: `032cb8e0-240f-4347-9120-6f33013e817a` and `bce1c492-0679-4989-941b-8de5e6789cb9`.

Config:

```ini
org_mapping = ["032cb8e0-240f-4347-9120-6f33013e817a:org_foo:Viewer", "bce1c492-0679-4989-941b-8de5e6789cb9:org_bar:Editor", "*:org_baz:Editor"]
```

# Configure Generic OAuth authentication

{{< docs/shared lookup="auth/intro.md" source="grafana" version="<GRAFANA VERSION>" >}}

Grafana provides OAuth2 integrations for the following auth providers:

- [Entra ID OAuth](../azuread/)
- [GitHub OAuth](../github/)
- [GitLab OAuth](../gitlab/)
- [Google OAuth](../google/)
- [Grafana Com OAuth](../grafana-cloud/)
- [Keycloak OAuth](../keycloak/)
- [Okta OAuth](../okta/)

If your OAuth2 provider is not listed, you can use Generic OAuth authentication.

This topic describes how to configure Generic OAuth authentication using different methods and includes [examples of setting up Generic OAuth](#examples-of-setting-up-generic-oauth) with specific OAuth2 providers.

## Before you begin

To follow this guide:

- Ensure you know how to create an OAuth2 application with your OAuth2 provider. Consult the documentation of your OAuth2 provider for more information.
- Ensure your identity provider returns OpenID UserInfo compatible information such as the `sub` claim.
- If you are using refresh tokens, ensure you know how to set them up with your OAuth2 provider. Consult the documentation of your OAuth2 provider for more information.

{{< admonition type="note" >}}
If Users use the same email address in Entra ID that they use with other authentication providers (such as Grafana.com), you need to do additional configuration to ensure that the users are matched correctly. Please refer to the [Using the same email address to login with different identity providers](../#using-the-same-email-address-to-login-with-different-identity-providers) documentation for more information.
{{< /admonition >}}

## Configure generic OAuth authentication client using the Grafana UI

As a Grafana Admin, you can configure Generic OAuth client from within Grafana using the Generic OAuth UI. To do this, navigate to **Administration > Authentication > Generic OAuth** page and fill in the form. If you have a current configuration in the Grafana configuration file then the form will be pre-populated with those values otherwise the form will contain default values.

After you have filled in the form, click **Save** to save the configuration. If the save was successful, Grafana will apply the new configurations.

If you need to reset changes you made in the UI back to the default values, click **Reset**. After you have reset the changes, Grafana will apply the configuration from the Grafana configuration file (if there is any configuration) or the default values.

{{< admonition type="note" >}}
If you run Grafana in high availability mode, configuration changes may not get applied to all Grafana instances immediately. You may need to wait a few minutes for the configuration to propagate to all Grafana instances.
{{< /admonition >}}

Refer to [configuration options](#configuration-options) for more information.

## Configure generic OAuth authentication client using the Terraform provider

```terraform
resource "grafana_sso_settings" "generic_sso_settings" {
  provider_name = "generic_oauth"
  oauth2_settings {
    name              = "Auth0"
    auth_url          = "https://<domain>/authorize"
    token_url         = "https://<domain>/oauth/token"
    api_url           = "https://<domain>/userinfo"
    client_id         = "<client id>"
    client_secret     = "<client secret>"
    allow_sign_up     = true
    auto_login        = false
    scopes            = "openid profile email offline_access"
    use_pkce          = true
    use_refresh_token = true
  }
}
```

Refer to [Terraform Registry](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/sso_settings) for a complete reference on using the `grafana_sso_settings` resource.

## Configure generic OAuth authentication client using the Grafana configuration file

Ensure that you have access to the [Grafana configuration file](../../../configure-grafana/#configuration-file-location).

### Steps

To integrate your OAuth2 provider with Grafana using our Generic OAuth authentication, follow these steps:

1. Create an OAuth2 application in your chosen OAuth2 provider.
1. Set the callback URL for your OAuth2 app to `http://<my_grafana_server_name_or_ip>:<grafana_server_port>/login/generic_oauth`.

   Ensure that the callback URL is the complete HTTP address that you use to access Grafana via your browser, but with the appended path of `/login/generic_oauth`.

   For the callback URL to be correct, it might be necessary to set the `root_url` option in the `[server]`section of the Grafana configuration file. For example, if you are serving Grafana behind a proxy.

1. Refer to the following table to update field values located in the `[auth.generic_oauth]` section of the Grafana configuration file:

   | Field                        | Description                                                                                                                                                                                       |
   | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | `client_id`, `client_secret` | These values must match the client ID and client secret from your OAuth2 app.                                                                                                                     |
   | `auth_url`                   | The authorization endpoint of your OAuth2 provider.                                                                                                                                               |
   | `api_url`                    | The user information endpoint of your OAuth2 provider. Information returned by this endpoint must be compatible with [OpenID UserInfo](https://connect2id.com/products/server/docs/api/userinfo). |
   | `enabled`                    | Enables Generic OAuth authentication. Set this value to `true`.                                                                                                                                   |

   Review the list of other Generic OAuth [configuration options](#configuration-options) and complete them, as necessary.

1. Optional: [Configure a refresh token](#configure-a-refresh-token):

   a. Extend the `scopes` field of `[auth.generic_oauth]` section in Grafana configuration file with refresh token scope used by your OAuth2 provider.

   b. Set `use_refresh_token` to `true` in `[auth.generic_oauth]` section in Grafana configuration file.

   c. Enable the refresh token on the provider if required.

1. [Configure role mapping](#configure-role-mapping).
1. Optional: [Configure team synchronization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/).
1. Restart Grafana.

   You should now see a Generic OAuth login button on the login page and be able to log in or sign up with your OAuth2 provider.

### Configure login

Grafana can resolve a user's login from the OAuth2 ID token, user information retrieved from the OAuth2 UserInfo endpoint, or the OAuth2 access token.
Grafana looks at these sources in the order listed until it finds a login.
If no login is found, then the user's login is set to user's email address.

{{< admonition type="important" >}}
Email is required for successful sign-up and login with Generic OAuth. Even if you map `login` from another claim (for example `sub`), Grafana still requires the user to have an email. Ensure your provider returns an email claim or configure `email_attribute_path` so Grafana can resolve it. Including the `email` scope is strongly recommended (for OIDC providers use `openid profile email`).
{{< /admonition >}}

Refer to the following table for information on what to configure based on how your Oauth2 provider returns a user's login:

| Source of login                                                                 | Required configuration                           |
| ------------------------------------------------------------------------------- | ------------------------------------------------ |
| `login` or `username` field of the OAuth2 ID token.                             | N/A                                              |
| Another field of the OAuth2 ID token.                                           | Set `login_attribute_path` configuration option. |
| `login` or `username` field of the user information from the UserInfo endpoint. | N/A                                              |
| Another field of the user information from the UserInfo endpoint.               | Set `login_attribute_path` configuration option. |
| `login` or `username` field of the OAuth2 access token.                         | N/A                                              |
| Another field of the OAuth2 access token.                                       | Set `login_attribute_path` configuration option. |

#### Use the `sub` claim for login

Most of the OAuth2 providers expose a stable subject identifier in the `sub` claim. You can use it to populate the Grafana login by setting `login_attribute_path` to `sub`. Because email is still required, also make sure Grafana can resolve the user's email (for example by including the `email` scope or mapping a custom field via `email_attribute_path`).

Example configuration:

```ini
[auth.generic_oauth]
enabled = true
scopes = openid profile email
login_attribute_path = sub
# If your provider does not return `email` at the top level, map it explicitly
# email_attribute_path = user.email
```

### Configure display name

Grafana can resolve a user's display name from the OAuth2 ID token, user information retrieved from the OAuth2 UserInfo endpoint, or the OAuth2 access token.
Grafana looks at these sources in the order listed until it finds a display name.
If no display name is found, then user's login is displayed instead.

Refer to the following table for information on what you need to configure depending on how your Oauth2 provider returns a user's name:

| Source of display name                                                             | Required configuration                          |
| ---------------------------------------------------------------------------------- | ----------------------------------------------- |
| `name` or `display_name` field of the OAuth2 ID token.                             | N/A                                             |
| Another field of the OAuth2 ID token.                                              | Set `name_attribute_path` configuration option. |
| `name` or `display_name` field of the user information from the UserInfo endpoint. | N/A                                             |
| Another field of the user information from the UserInfo endpoint.                  | Set `name_attribute_path` configuration option. |
| `name` or `display_name` field of the OAuth2 access token.                         | N/A                                             |
| Another field of the OAuth2 access token.                                          | Set `name_attribute_path` configuration option. |

### Configure email address

Grafana can resolve the user's email address from the OAuth2 ID token, the user information retrieved from the OAuth2 UserInfo endpoint, the OAuth2 access token, or the OAuth2 `/emails` endpoint.
Grafana looks at these sources in the order listed until an email address is found.
If no email is found, then the email address of the user is set to an empty string.

Refer to the following table for information on what to configure based on how the Oauth2 provider returns a user's email address:

| Source of email address                                                                                                                                                 | Required configuration                                                                                             |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `email` field of the OAuth2 ID token.                                                                                                                                   | N/A                                                                                                                |
| `attributes` map of the OAuth2 ID token.                                                                                                                                | Set `email_attribute_name` configuration option. By default, Grafana searches for email under `email:primary` key. |
| `upn` field of the OAuth2 ID token.                                                                                                                                     | N/A                                                                                                                |
| `email` field of the user information from the UserInfo endpoint.                                                                                                       | N/A                                                                                                                |
| Another field of the user information from the UserInfo endpoint.                                                                                                       | Set `email_attribute_path` configuration option.                                                                   |
| `email` field of the OAuth2 access token.                                                                                                                               | N/A                                                                                                                |
| `attributes` map of the OAuth2 access token.                                                                                                                            | Set `email_attribute_name` configuration option. By default, Grafana searches for email under `email:primary` key. |
| `upn` field of the OAuth2 access token.                                                                                                                                 | N/A                                                                                                                |
| Another field of the OAuth2 access token.                                                                                                                               | Set `email_attribute_path` configuration option.                                                                   |
| Email address marked as primary from the `/emails` endpoint of <br /> the OAuth2 provider (obtained by appending `/emails` to the URL <br /> configured with `api_url`) | N/A                                                                                                                |

### Configure a refresh token

When a user logs in using an OAuth2 provider, Grafana verifies that the access token has not expired. When an access token expires, Grafana uses the provided refresh token (if any exists) to obtain a new access token.

Grafana uses a refresh token to obtain a new access token without requiring the user to log in again. If a refresh token doesn't exist, Grafana logs the user out of the system after the access token has expired.

To configure Generic OAuth to use a refresh token, set `use_refresh_token` configuration option to `true` and perform one or both of the following steps, if required:

1. Extend the `scopes` field of `[auth.generic_oauth]` section in Grafana configuration file with additional scopes.
1. Enable the refresh token on the provider.

{{< admonition type="note" >}}
The `accessTokenExpirationCheck` feature toggle has been removed in Grafana v10.3.0 and the `use_refresh_token` configuration value will be used instead for configuring refresh token fetching and access token expiration check.
{{< /admonition >}}

### Configure JWT ID token validation

By default, Grafana extracts user information from ID tokens without validating their cryptographic signatures. To enhance security, you can enable JWT signature validation to ensure that ID tokens are authentic and have not been tampered with.

To enable JWT ID token validation:

1. Set `validate_id_token` to `true` in the `[auth.generic_oauth]` section of the Grafana configuration file.
2. Configure `jwk_set_url` with the URL of your OAuth2 provider's JSON Web Key Set (JWKS) endpoint. This endpoint provides the public keys used to verify JWT signatures.

   Common JWKS endpoint locations:
   - OIDC providers: `https://<provider-domain>/.well-known/jwks.json`
   - Auth0: `https://<tenant>.auth0.com/.well-known/jwks.json`
   - Keycloak: `https://<keycloak-domain>/realms/<realm>/.well-known/openid-configuration` (contains `jwks_uri`)

Example configuration:

```ini
[auth.generic_oauth]
enabled = true
validate_id_token = true
jwk_set_url = https://your-provider.com/.well-known/jwks.json
client_id = <client id>
client_secret = <client secret>
auth_url = https://your-provider.com/authorize
token_url = https://your-provider.com/token
api_url = https://your-provider.com/userinfo
```

{{< admonition type="note" >}}
When JWT validation is enabled, Grafana caches the JWKS keys to improve performance. The cache respects the `Cache-Control` header from the JWKS endpoint response. If no cache expiration is specified, keys are cached for 5 minutes by default.
{{< /admonition >}}

{{< admonition type="caution" >}}
If `validate_id_token` is set to `true`, you must configure `jwk_set_url`. Authentication will fail if the JWK Set URL is not provided or if the ID token signature cannot be verified.
{{< /admonition >}}

### Configure role mapping

Unless `skip_org_role_sync` option is enabled, the user's role will be set to the role retrieved from the auth provider upon user login.

The user's role is retrieved using a [JMESPath](http://jmespath.org/examples.html) expression from the `role_attribute_path` configuration option.
Grafana will first evaluate the expression using the OAuth2 ID token. If no role is found, the expression will be evaluated using the user information obtained from the UserInfo endpoint. If still no role is found, the expression will be evaluated using the OAuth2 access token.
To map the server administrator role, use the `allow_assign_grafana_admin` configuration option.
Refer to [configuration options](#configuration-options) for more information.

If no valid role is found, the user is assigned the role specified by [the `auto_assign_org_role` option](../../../configure-grafana/#auto_assign_org_role).
You can disable this default role assignment by setting `role_attribute_strict = true`. This setting denies user access if no role or an invalid role is returned after evaluating the `role_attribute_path` and the `org_mapping` expressions.

You can use the `org_attribute_path` and `org_mapping` configuration options to assign the user to organizations and specify their role. For more information, refer to [Org roles mapping example](#org-roles-mapping-example). If both org role mapping (`org_mapping`) and the regular role mapping (`role_attribute_path`) are specified, then the user will get the highest of the two mapped roles.

To ease configuration of a proper JMESPath expression, go to [JMESPath](http://jmespath.org/) to test and evaluate expressions with custom payloads.

{{< admonition type="note" >}}
When using `org_attribute_path`, the value returned by the JMESPath expression must be an array, not a string.
{{< /admonition >}}

#### Role mapping examples

This section includes examples of JMESPath expressions used for role mapping.

##### Map user organization role

In this example, the user has been granted the role of an `Editor`. The role assigned is based on the value of the property `role`, which must be a valid Grafana role such as `Admin`, `Editor`, `Viewer` or `None`.

Payload:

```json
{
    ...
    "role": "Editor",
    ...
}
```

Config:

```bash
role_attribute_path = role
```

In the following more complex example, the user has been granted the `Admin` role. This is because they are a member of the `admin` group of their OAuth2 provider.
If the user was a member of the `editor` group, they would be granted the `Editor` role, otherwise `Viewer`.

Payload:

```json
{
    ...
    "groups": [
        "engineer",
        "admin",
    ],
    ...
}
```

Config:

```bash
role_attribute_path = contains(groups[*], 'admin') && 'Admin' || contains(groups[*], 'editor') && 'Editor' || 'Viewer'
```

##### Map server administrator role

In the following example, the user is granted the Grafana server administrator role.

Payload:

```json
{
    ...
    "roles": [
        "admin",
    ],
    ...
}
```

Config:

```ini
role_attribute_path = contains(roles[*], 'admin') && 'GrafanaAdmin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'
allow_assign_grafana_admin = true
```

##### Map one role to all users

In this example, all users will be assigned `Viewer` role regardless of the user information received from the identity provider.

Config:

```ini
role_attribute_path = "'Viewer'"
skip_org_role_sync = false
```

#### Org roles mapping

The field `org_mapping` accepts a space or comma separated list of `"<ExternalName>:<OrgIdOrName>:<Role>"` mappings.

It is possible to replace `ExternalName` or `OrgIdOrName` with a `*` as a match any.

Partial matches treat the `*` as part of the name (e.g. `*-org` does not match `my-org`)

##### Org roles mapping example

In this example, the user has been granted the role of a `Viewer` in the `org_foo` org, and the role of an `Editor` in the `org_bar` and `org_baz` orgs.

If the user was a member of the `admin` group, they would be granted the Grafana server administrator role.

Payload:

```json
{
  "roles": ["org_foo", "org_bar", "another_org"]
}
```

Config:

```ini
role_attribute_path = contains(roles[*], 'admin') && 'GrafanaAdmin' || 'None'
allow_assign_grafana_admin = true
org_attribute_path = roles
org_mapping = org_foo:org_foo:Viewer org_bar:org_bar:Editor *:org_baz:Editor
```

## Configure team synchronization

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and to customers on select Grafana Cloud plans. For pricing information, visit [pricing](https://grafana.com/pricing/) or contact our sales team.
{{< /admonition >}}

By using Team Sync, you can link your OAuth2 groups to teams within Grafana. This will automatically assign users to the appropriate teams.
Teams for each user are synchronized when the user logs in.

Generic OAuth groups can be referenced by group ID, such as `8bab1c86-8fba-33e5-2089-1d1c80ec267d` or `myteam`.
Group information can be extracted from the OAuth2 ID token, user information from the UserInfo endpoint, or the OAuth2 access token.
For information on configuring OAuth2 groups with Grafana using the `groups_attribute_path` configuration option, refer to [configuration options](#configuration-options).

To learn more about Team Sync, refer to [Configure team sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/).

### Team synchronization example

Configuration:

```bash
groups_attribute_path = groups
```

Payload:

```json
{
    ...
    "groups": [
        "engineers",
        "analysts",
    ],
    ...
}
```

## Configuration options

The following table outlines the various Generic OAuth configuration options. You can apply these options as environment variables, similar to any other configuration within Grafana. For more information, refer to [Override configuration with environment variables](../../../configure-grafana/#override-configuration-with-environment-variables).

{{< admonition type="note" >}}
If the configuration option requires a JMESPath expression that includes a colon, enclose the entire expression in quotes to prevent parsing errors. For example `role_attribute_path: "role:view"`
{{< /admonition >}}

| Setting                      | Required | Supported on Cloud | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | Default         |
| ---------------------------- | -------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- |
| `enabled`                    | No       | Yes                | Enables Generic OAuth authentication.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `false`         |
| `name`                       | No       | Yes                | Name that refers to the Generic OAuth authentication from the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `OAuth`         |
| `icon`                       | No       | Yes                | Icon used for the Generic OAuth authentication in the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `signin`        |
| `client_id`                  | Yes      | Yes                | Client ID provided by your OAuth2 app.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |                 |
| `client_secret`              | Yes      | Yes                | Client secret provided by your OAuth2 app.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |                 |
| `auth_url`                   | Yes      | Yes                | Authorization endpoint of your OAuth2 provider.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |                 |
| `token_url`                  | Yes      | Yes                | Endpoint used to obtain the OAuth2 access token.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                 |
| `token_exchange_timeout`     | No       | Yes                | Timeout in seconds for the OAuth token exchange request. If set to `0` or left unset, defaults to `15` seconds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `15`            |
| `api_url`                    | Yes      | Yes                | Endpoint used to obtain user information compatible with [OpenID UserInfo](https://connect2id.com/products/server/docs/api/userinfo).                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |                 |
| `auth_style`                 | No       | Yes                | Name of the [OAuth2 AuthStyle](https://pkg.go.dev/golang.org/x/oauth2#AuthStyle) to be used when ID token is requested from OAuth2 provider. It determines how `client_id` and `client_secret` are sent to Oauth2 provider. Available values are `AutoDetect`, `InParams` and `InHeader`.                                                                                                                                                                                                                                                                                                                   | `AutoDetect`    |
| `scopes`                     | No       | Yes                | List of comma- or space-separated OAuth2 scopes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | `user:email`    |
| `empty_scopes`               | No       | Yes                | Set to `true` to use an empty scope during authentication.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | `false`         |
| `allow_sign_up`              | No       | Yes                | Controls Grafana user creation through the Generic OAuth login. Only existing Grafana users can log in with Generic OAuth if set to `false`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `true`          |
| `auto_login`                 | No       | Yes                | Set to `true` to enable users to bypass the login screen and automatically log in. This setting is ignored if you configure multiple auth providers to use auto-login.                                                                                                                                                                                                                                                                                                                                                                                                                                      | `false`         |
| `login_prompt`               | No       | Yes                | Indicates the type of user interaction when the user logs in with the IdP. Available values are `login`, `consent` and `select_account`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |                 |
| `id_token_attribute_name`    | No       | Yes                | The name of the key used to extract the ID token from the returned OAuth2 token.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | `id_token`      |
| `login_attribute_path`       | No       | Yes                | [JMESPath](http://jmespath.org/examples.html) expression to use for user login lookup from the user ID token. For more information on how user login is retrieved, refer to [Configure login](#configure-login).                                                                                                                                                                                                                                                                                                                                                                                            |                 |
| `name_attribute_path`        | No       | Yes                | [JMESPath](http://jmespath.org/examples.html) expression to use for user name lookup from the user ID token. This name will be used as the user's display name. For more information on how user display name is retrieved, refer to [Configure display name](#configure-display-name).                                                                                                                                                                                                                                                                                                                     |                 |
| `email_attribute_path`       | No       | Yes                | [JMESPath](http://jmespath.org/examples.html) expression to use for user email lookup from the user information. For more information on how user email is retrieved, refer to [Configure email address](#configure-email-address).                                                                                                                                                                                                                                                                                                                                                                         |                 |
| `email_attribute_name`       | No       | Yes                | Name of the key to use for user email lookup within the `attributes` map of OAuth2 ID token. For more information on how user email is retrieved, refer to [Configure email address](#configure-email-address).                                                                                                                                                                                                                                                                                                                                                                                             | `email:primary` |
| `role_attribute_path`        | No       | Yes                | [JMESPath](http://jmespath.org/examples.html) expression to use for Grafana role lookup. Grafana will first evaluate the expression using the OAuth2 ID token. If no role is found, the expression will be evaluated using the user information obtained from the UserInfo endpoint. If still no role is found, the expression will be evaluated using the OAuth2 access token. The result of the evaluation should be a valid Grafana role (`None`, `Viewer`, `Editor`, `Admin` or `GrafanaAdmin`). For more information on user role mapping, refer to [Configure role mapping](#configure-role-mapping). |                 |
| `role_attribute_strict`      | No       | Yes                | Set to `true` to deny user login if the Grafana org role cannot be extracted using `role_attribute_path` or `org_mapping`. For more information on user role mapping, refer to [Configure role mapping](#configure-role-mapping).                                                                                                                                                                                                                                                                                                                                                                           | `false`         |
| `skip_org_role_sync`         | No       | Yes                | Set to `true` to stop automatically syncing user roles. This will allow you to set organization roles for your users from within Grafana manually.                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `false`         |
| `org_attribute_path`         | No       | No                 | [JMESPath](http://jmespath.org/examples.html) expression to use for Grafana org to role lookup. Grafana will first evaluate the expression using the OAuth2 ID token. If no value is returned, the expression will be evaluated using the user information obtained from the UserInfo endpoint. If still no value is returned, the expression will be evaluated using the OAuth2 access token. The result of the evaluation will be mapped to org roles based on `org_mapping`. For more information on org to role mapping, refer to [Org roles mapping example](#org-roles-mapping-example).              |                 |
| `org_mapping`                | No       | No                 | List of comma- or space-separated `<ExternalOrgName>:<OrgIdOrName>:<Role>` mappings. Value can be `*` meaning "All users". Role is optional and can have the following values: `None`, `Viewer`, `Editor` or `Admin`. For more information on external organization to role mapping, refer to [Org roles mapping example](#org-roles-mapping-example).                                                                                                                                                                                                                                                      |                 |
| `allow_assign_grafana_admin` | No       | No                 | Set to `true` to enable automatic sync of the Grafana server administrator role. If this option is set to `true` and the result of evaluating `role_attribute_path` for a user is `GrafanaAdmin`, Grafana grants the user the server administrator privileges and organization administrator role. If this option is set to `false` and the result of evaluating `role_attribute_path` for a user is `GrafanaAdmin`, Grafana grants the user only organization administrator role. For more information on user role mapping, refer to [Configure role mapping](#configure-role-mapping).                   | `false`         |
| `groups_attribute_path`      | No       | Yes                | [JMESPath](http://jmespath.org/examples.html) expression to use for user group lookup. Grafana will first evaluate the expression using the OAuth2 ID token. If no groups are found, the expression will be evaluated using the user information obtained from the UserInfo endpoint. If still no groups are found, the expression will be evaluated using the OAuth2 access token. The result of the evaluation should be a string array of groups.                                                                                                                                                        |                 |
| `allowed_groups`             | No       | Yes                | List of comma- or space-separated groups. The user should be a member of at least one group to log in. If you configure `allowed_groups`, you must also configure `groups_attribute_path`.                                                                                                                                                                                                                                                                                                                                                                                                                  |                 |
| `allowed_organizations`      | No       | Yes                | List of comma- or space-separated organizations. The user should be a member of at least one organization to log in.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                 |
| `allowed_domains`            | No       | Yes                | List of comma- or space-separated domains. The user should belong to at least one domain to log in.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                 |
| `team_ids`                   | No       | Yes                | String list of team IDs. If set, the user must be a member of one of the given teams to log in. If you configure `team_ids`, you must also configure `teams_url` and `team_ids_attribute_path`.                                                                                                                                                                                                                                                                                                                                                                                                             |                 |
| `team_ids_attribute_path`    | No       | Yes                | The [JMESPath](http://jmespath.org/examples.html) expression to use for Grafana team ID lookup within the results returned by the `teams_url` endpoint.                                                                                                                                                                                                                                                                                                                                                                                                                                                     |                 |
| `teams_url`                  | No       | Yes                | The URL used to query for team IDs. If not set, the default value is `/teams`. If you configure `teams_url`, you must also configure `team_ids_attribute_path`.                                                                                                                                                                                                                                                                                                                                                                                                                                             |                 |
| `tls_skip_verify_insecure`   | No       | No                 | If set to `true`, the client accepts any certificate presented by the server and any host name in that certificate. _You should only use this for testing_, because this mode leaves SSL/TLS susceptible to man-in-the-middle attacks.                                                                                                                                                                                                                                                                                                                                                                      | `false`         |
| `tls_client_cert`            | No       | No                 | The path to the certificate.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                 |
| `tls_client_key`             | No       | No                 | The path to the key.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                 |
| `tls_client_ca`              | No       | No                 | The path to the trusted certificate authority list.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                 |
| `use_pkce`                   | No       | Yes                | Set to `true` to use [Proof Key for Code Exchange (PKCE)](https://datatracker.ietf.org/doc/html/rfc7636). Grafana uses the SHA256 based `S256` challenge method and a 128 bytes (base64url encoded) code verifier.                                                                                                                                                                                                                                                                                                                                                                                          | `false`         |
| `use_refresh_token`          | No       | Yes                | Set to `true` to use refresh token and check access token expiration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `false`         |
| `validate_id_token`          | No       | Yes                | Set to `true` to enable JWT signature validation for ID tokens. When enabled, `jwk_set_url` must be configured.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `false`         |
| `jwk_set_url`                | No       | Yes                | URL of the JSON Web Key Set (JWKS) endpoint used to verify JWT ID token signatures. Required when `validate_id_token` is set to `true`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |                 |
| `signout_redirect_url`       | No       | Yes                | URL to redirect to after the user logs out.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |                 |

## Examples of setting up Generic OAuth

This section includes examples of setting up Generic OAuth integration.

### Set up OAuth2 with Descope

To set up Generic OAuth authentication with Descope, follow these steps:

1. Create a Descope Project [here](https://app.descope.com/gettingStarted), and go through the Getting Started Wizard to configure your authentication. You can skip step if you already have Descope project set up.

1. If you wish to use a flow besides `Sign Up or In`, go to the **IdP Applications** menu in the console, and select your IdP application. Then alter the **Flow Hosting URL** query parameter `?flow=sign-up-or-in` to change which flow id you wish to use.

1. Click **Save**.

1. Update the `[auth.generic_oauth]` section of the Grafana configuration file using the values from the **Settings** tab:

   {{< admonition type="note" >}}
   You can get your Client ID (Descope Project ID) under [Project Settings](https://app.descope.com/settings/project). Your Client Secret (Descope Access Key) can be generated under [Access Keys](https://app.descope.com/accesskeys).
   {{< /admonition >}}

   ```bash
   [auth.generic_oauth]
   enabled = true
   allow_sign_up = true
   auto_login = false
   team_ids =
   allowed_organizations =
   name = Descope
   client_id = <Descope Project ID>
   client_secret = <Descope Access Key>
   scopes = openid profile email descope.claims descope.custom_claims
   auth_url = https://api.descope.com/oauth2/v1/authorize
   token_url = https://api.descope.com/oauth2/v1/token
   api_url = https://api.descope.com/oauth2/v1/userinfo
   use_pkce = true
   use_refresh_token = true
   ```

### Set up OAuth2 with Auth0

{{< admonition type="note" >}}
Support for the Auth0 "audience" feature is not currently available in Grafana. For roles and permissions, the available options are described [here](../../../../administration/roles-and-permissions/).
{{< /admonition >}}

To set up Generic OAuth authentication with Auth0, follow these steps:

1. Create an Auth0 application using the following parameters:
   - Name: Grafana
   - Type: Regular Web Application

1. Go to the **Settings** tab of the application and set **Allowed Callback URLs** to `https://<grafana domain>/login/generic_oauth`.

1. Click **Save Changes**.

1. Update the `[auth.generic_oauth]` section of the Grafana configuration file using the values from the **Settings** tab:

   ```bash
   [auth.generic_oauth]
   enabled = true
   allow_sign_up = true
   auto_login = false
   team_ids =
   allowed_organizations =
   name = Auth0
   client_id = <client id>
   client_secret = <client secret>
   scopes = openid profile email offline_access
   auth_url = https://<domain>/authorize
   token_url = https://<domain>/oauth/token
   api_url = https://<domain>/userinfo
   use_pkce = true
   use_refresh_token = true
   ```

### Set up OAuth2 with Bitbucket

To set up Generic OAuth authentication with Bitbucket, follow these steps:

1. Navigate to **Settings > Workspace setting > OAuth consumers** in BitBucket.

1. Create an application by selecting **Add consumer** and using the following parameters:
   - Allowed Callback URLs: `https://<grafana domain>/login/generic_oauth`

1. Click **Save**.

1. Update the `[auth.generic_oauth]` section of the Grafana configuration file using the values from the `Key` and `Secret` from the consumer description:

   ```bash
   [auth.generic_oauth]
   name = BitBucket
   enabled = true
   allow_sign_up = true
   auto_login = false
   client_id = <client key>
   client_secret = <client secret>
   scopes = account email
   auth_url = https://bitbucket.org/site/oauth2/authorize
   token_url = https://bitbucket.org/site/oauth2/access_token
   api_url = https://api.bitbucket.org/2.0/user
   teams_url = https://api.bitbucket.org/2.0/user/permissions/workspaces
   team_ids_attribute_path = values[*].workspace.slug
   team_ids =
   allowed_organizations =
   use_refresh_token = true
   ```

By default, a refresh token is included in the response for the **Authorization Code Grant**.

### Set up OAuth2 with OneLogin

To set up Generic OAuth authentication with OneLogin, follow these steps:

1. Create a new Custom Connector in OneLogin with the following settings:
   - Name: Grafana
   - Sign On Method: OpenID Connect
   - Redirect URI: `https://<grafana domain>/login/generic_oauth`
   - Signing Algorithm: RS256
   - Login URL: `https://<grafana domain>/login/generic_oauth`

1. Add an app to the Grafana Connector:
   - Display Name: Grafana

1. Update the `[auth.generic_oauth]` section of the Grafana configuration file using the client ID and client secret from the **SSO** tab of the app details page:

   Your OneLogin Domain will match the URL you use to access OneLogin.

   ```bash
   [auth.generic_oauth]
   name = OneLogin
   enabled = true
   allow_sign_up = true
   auto_login = false
   client_id = <client id>
   client_secret = <client secret>
   scopes = openid email name
   auth_url = https://<onelogin domain>.onelogin.com/oidc/2/auth
   token_url = https://<onelogin domain>.onelogin.com/oidc/2/token
   api_url = https://<onelogin domain>.onelogin.com/oidc/2/me
   team_ids =
   allowed_organizations =
   ```

### Set up OAuth2 with Dex

To set up Generic OAuth authentication with [Dex IdP](https://dexidp.io/), follow these
steps:

1. Add Grafana as a client in the Dex config YAML file:

   ```yaml
   staticClients:
     - id: <client id>
       name: Grafana
       secret: <client secret>
       redirectURIs:
         - 'https://<grafana domain>/login/generic_oauth'
   ```

   {{< admonition type="note" >}}
   Unlike many other OAuth2 providers, Dex doesn't provide `<client secret>`.
   Instead, a secret can be generated with for example `openssl rand -hex 20`.
   {{< /admonition >}}

2. Update the `[auth.generic_oauth]` section of the Grafana configuration:

   ```bash
   [auth.generic_oauth]
   name = Dex
   enabled = true
   client_id = <client id>
   client_secret = <client secret>
   scopes = openid email profile groups offline_access
   auth_url = https://<dex base uri>/auth
   token_url = https://<dex base uri>/token
   api_url = https://<dex base uri>/userinfo
   ```

   `<dex base uri>` corresponds to the `issuer: ` configuration in Dex (e.g. the Dex
   domain possibly including a path such as e.g. `/dex`). The `offline_access` scope is
   needed when using [refresh tokens](#configure-a-refresh-token).

# Configure GitHub OAuth authentication

{{< docs/shared lookup="auth/intro.md" source="grafana" version="<GRAFANA VERSION>" >}}

This topic describes how to configure GitHub OAuth authentication.

{{< admonition type="note" >}}
If Users use the same email address in GitHub that they use with other authentication providers (such as Grafana.com), you need to do additional configuration to ensure that the users are matched correctly. Please refer to the [Using the same email address to login with different identity providers](../#using-the-same-email-address-to-login-with-different-identity-providers) documentation for more information.
{{< /admonition >}}

## Before you begin

Ensure you know how to create a GitHub OAuth app. Consult GitHub's documentation on [creating an OAuth app](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app) for more information.

### Create a GitHub OAuth App

1. Log in to your GitHub account.
   In **Profile > Settings > Developer settings**, select **OAuth Apps**.
1. Click **New OAuth App**.
1. Fill out the fields, using your Grafana homepage URL when appropriate.
   In the **Authorization callback URL** field, enter the following: `https://<YOUR-GRAFANA-URL>/login/github` .
1. Note your client ID.
1. Generate, then note, your client secret.

## Configure GitHub authentication client using the Grafana UI

As a Grafana Admin, you can configure GitHub OAuth client from within Grafana using the GitHub UI. To do this, navigate to **Administration > Authentication > GitHub** page and fill in the form. If you have a current configuration in the Grafana configuration file, the form will be pre-populated with those values. Otherwise the form will contain default values.

After you have filled in the form, click **Save**. If the save was successful, Grafana will apply the new configurations.

If you need to reset changes you made in the UI back to the default values, click **Reset**. After you have reset the changes, Grafana will apply the configuration from the Grafana configuration file (if there is any configuration) or the default values.

{{< admonition type="note" >}}
If you run Grafana in high availability mode, configuration changes may not get applied to all Grafana instances immediately. You may need to wait a few minutes for the configuration to propagate to all Grafana instances.
{{< /admonition >}}

Refer to [configuration options](#configuration-options) for more information.

## Configure GitHub authentication client using the Terraform provider

```terraform
resource "grafana_sso_settings" "github_sso_settings" {
  provider_name = "github"
  oauth2_settings {
    name                  = "Github"
    client_id             = "YOUR_GITHUB_APP_CLIENT_ID"
    client_secret         = "YOUR_GITHUB_APP_CLIENT_SECRET"
    allow_sign_up         = true
    auto_login            = false
    scopes                = "user:email,read:org"
    team_ids              = "150,300"
    allowed_organizations = "[\"My Organization\", \"Octocats\"]"
    allowed_domains       = "mycompany.com mycompany.org"
    role_attribute_path   = "[login=='octocat'][0] && 'GrafanaAdmin' || 'Viewer'"
  }
}
```

Go to [Terraform Registry](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/sso_settings) for a complete reference on using the `grafana_sso_settings` resource.

## Configure GitHub authentication client using the Grafana configuration file

Ensure that you have access to the [Grafana configuration file](../../../configure-grafana/#configuration-file-location).

### Configure GitHub authentication

To configure GitHub authentication with Grafana, follow these steps:

1. Create an OAuth application in GitHub.
1. Set the callback URL for your GitHub OAuth app to `http://<my_grafana_server_name_or_ip>:<grafana_server_port>/login/github`.

   Ensure that the callback URL is the complete HTTP address that you use to access Grafana via your browser, but with the appended path of `/login/github`.

   For the callback URL to be correct, it might be necessary to set the `root_url` option in the `[server]`section of the Grafana configuration file. For example, if you are serving Grafana behind a proxy.

1. Refer to the following table to update field values located in the `[auth.github]` section of the Grafana configuration file:

   | Field                        | Description                                                                         |
   | ---------------------------- | ----------------------------------------------------------------------------------- |
   | `client_id`, `client_secret` | These values must match the client ID and client secret from your GitHub OAuth app. |
   | `enabled`                    | Enables GitHub authentication. Set this value to `true`.                            |

   Review the list of other GitHub [configuration options](#configuration-options) and complete them, as necessary.

1. [Configure role mapping](#configure-role-mapping).
1. Optional: [Configure team synchronization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/).
1. Restart Grafana.

   You should now see a GitHub login button on the login page and be able to log in or sign up with your GitHub accounts.

### Configure role mapping

Unless the `skip_org_role_sync` option is enabled, the user's role will be set to the role retrieved from GitHub upon user login.

The user's role is retrieved using a [JMESPath](http://jmespath.org/examples.html) expression from the `role_attribute_path` configuration option.
To map the server administrator role, use the `allow_assign_grafana_admin` configuration option.
Refer to [configuration options](#configuration-options) for more information.

If no valid role is found, the user is assigned the role specified by [the `auto_assign_org_role` option](../../../configure-grafana/#auto_assign_org_role).
You can disable this default role assignment by setting `role_attribute_strict = true`. This setting denies user access if no role or an invalid role is returned after evaluating the `role_attribute_path` and the `org_mapping` expressions.

You can use the `org_mapping` configuration options to assign the user to organizations and specify their role based on their GitHub team membership. For more information, refer to [Org roles mapping example](#org-roles-mapping-example). If both org role mapping (`org_mapping`) and the regular role mapping (`role_attribute_path`) are specified, then the user will get the highest of the two mapped roles.

To ease configuration of a proper JMESPath expression, go to [JMESPath](http://jmespath.org/) to test and evaluate expressions with custom payloads.

#### Role mapping examples

This section includes examples of JMESPath expressions used for role mapping.

##### Org roles mapping example

The GitHub integration uses the external users' teams in the `org_mapping` configuration to map organizations and roles based on their GitHub team membership.

In this example, the user has been granted the role of a `Viewer` in the `org_foo` organization, and the role of an `Editor` in the `org_bar` and `org_baz` orgs.

The external user is part of the following GitHub teams: `@my-github-organization/my-github-team-1` and `@my-github-organization/my-github-team-2`.

Config:

```ini
org_mapping = @my-github-organization/my-github-team-1:org_foo:Viewer @my-github-organization/my-github-team-2:org_bar:Editor *:org_baz:Editor
```

##### Map roles using GitHub user information

In this example, the user with login `octocat` has been granted the `Admin` role.
All other users are granted the `Viewer` role.

```bash
role_attribute_path = [login=='octocat'][0] && 'Admin' || 'Viewer'
```

##### Map roles using GitHub teams

In this example, the user from GitHub team `my-github-team` has been granted the `Editor` role.
All other users are granted the `Viewer` role.

```bash
role_attribute_path = contains(groups[*], '@my-github-organization/my-github-team') && 'Editor' || 'Viewer'
```

##### Map roles using multiple GitHub teams

In this example, the users from GitHub teams `admins` and `devops` have been granted the `Admin` role,
the users from GitHub teams `engineers` and `managers` have been granted the `Editor` role,
the users from GitHub team `qa` have been granted the `Viewer` role and
all other users are granted the `None` role.

```bash
role_attribute_path = contains(groups[*], '@my-github-organization/admins') && 'Admin' || contains(groups[*], '@my-github-organization/devops') && 'Admin' || contains(groups[*], '@my-github-organization/engineers') && 'Editor' || contains(groups[*], '@my-github-organization/managers') && 'Editor' || contains(groups[*], '@my-github-organization/qa') && 'Viewer' || 'None'
```

##### Map server administrator role

In this example, the user with login `octocat` has been granted the `Admin` organization role as well as the Grafana server admin role.
All other users are granted the `Viewer` role.

```bash
role_attribute_path = [login=='octocat'][0] && 'GrafanaAdmin' || 'Viewer'
```

##### Map one role to all users

In this example, all users will be assigned `Viewer` role regardless of the user information received from the identity provider.

```ini
role_attribute_path = "'Viewer'"
skip_org_role_sync = false
```

### Example of GitHub configuration in Grafana

This section includes an example of GitHub configuration in the Grafana configuration file.

```bash
[auth.github]
enabled = true
client_id = YOUR_GITHUB_APP_CLIENT_ID
client_secret = YOUR_GITHUB_APP_CLIENT_SECRET
scopes = user:email,read:org
auth_url = https://github.com/login/oauth/authorize
token_url = https://github.com/login/oauth/access_token
api_url = https://api.github.com/user
allow_sign_up = true
auto_login = false
team_ids = 150,300
allowed_organizations = ["My Organization", "Octocats"]
allowed_domains = mycompany.com mycompany.org
role_attribute_path = [login=='octocat'][0] && 'GrafanaAdmin' || 'Viewer'
```

## Configure team synchronization

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/products/cloud/)
{{< /admonition >}}

By using Team Sync, you can map teams from your GitHub organization to teams within Grafana. This will automatically assign users to the appropriate teams.
Teams for each user are synchronized when the user logs in.

GitHub teams can be referenced in two ways:

- `https://github.com/orgs/<org>/teams/<slug>`
- `@<org>/<slug>`

Examples: `https://github.com/orgs/grafana/teams/developers` or `@grafana/developers`.

To learn more about Team Sync, refer to [Configure team sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/).

## Configuration options

The table below describes all GitHub OAuth configuration options. You can apply these options as environment variables, similar to any other configuration within Grafana. For more information, refer to [Override configuration with environment variables](../../../configure-grafana/#override-configuration-with-environment-variables).

{{< admonition type="note" >}}
If the configuration option requires a JMESPath expression that includes a colon, enclose the entire expression in quotes to prevent parsing errors. For example `role_attribute_path: "role:view"`
{{< /admonition >}}

| Setting                      | Required | Supported on Cloud | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Default                                       |
| ---------------------------- | -------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- |
| `enabled`                    | No       | Yes                | Whether GitHub OAuth authentication is allowed.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `false`                                       |
| `name`                       | No       | Yes                | Name used to refer to the GitHub authentication in the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | `GitHub`                                      |
| `icon`                       | No       | Yes                | Icon used for GitHub authentication in the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `github`                                      |
| `client_id`                  | Yes      | Yes                | Client ID provided by your GitHub OAuth app.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |                                               |
| `client_secret`              | Yes      | Yes                | Client secret provided by your GitHub OAuth app.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |                                               |
| `auth_url`                   | Yes      | Yes                | Authorization endpoint of your GitHub OAuth provider.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `https://github.com/login/oauth/authorize`    |
| `token_url`                  | Yes      | Yes                | Endpoint used to obtain GitHub OAuth access token.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `https://github.com/login/oauth/access_token` |
| `token_exchange_timeout`     | No       | Yes                | Timeout in seconds for the OAuth token exchange request. If set to `0` or left unset, defaults to `15` seconds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `15`                                          |
| `api_url`                    | Yes      | Yes                | Endpoint used to obtain GitHub user information compatible with [OpenID UserInfo](https://connect2id.com/products/server/docs/api/userinfo).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `https://api.github.com/user`                 |
| `scopes`                     | No       | Yes                | List of comma- or space-separated GitHub OAuth scopes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `user:email,read:org`                         |
| `allow_sign_up`              | No       | Yes                | Whether to allow new Grafana user creation through GitHub login. If set to `false`, then only existing Grafana users can log in with GitHub OAuth.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `true`                                        |
| `auto_login`                 | No       | Yes                | Set to `true` to enable users to bypass the login screen and automatically log in. This setting is ignored if you configure multiple auth providers to use auto-login.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `false`                                       |
| `login_prompt`               | No       | Yes                | Indicates the type of user interaction when the user logs in with GitHub. Available values are `login`, `consent` and `select_account`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                               |
| `role_attribute_path`        | No       | Yes                | [JMESPath](http://jmespath.org/examples.html) expression to use for Grafana role lookup. Grafana will first evaluate the expression using the user information obtained from the UserInfo endpoint. If no role is found, Grafana creates a JSON data with `groups` key that maps to GitHub teams obtained from GitHub's [`/api/user/teams`](https://docs.github.com/en/rest/teams/teams#list-teams-for-the-authenticated-user) endpoint, and evaluates the expression using this data. The result of the evaluation should be a valid Grafana role (`None`, `Viewer`, `Editor`, `Admin` or `GrafanaAdmin`). For more information on user role mapping, refer to [Configure role mapping](#org-roles-mapping-example). |                                               |
| `role_attribute_strict`      | No       | Yes                | Set to `true` to deny user login if the Grafana org role cannot be extracted using `role_attribute_path` or `org_mapping`. For more information on user role mapping, refer to [Configure role mapping](#org-roles-mapping-example).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | `false`                                       |
| `org_mapping`                | No       | No                 | List of comma- or space-separated `<ExternalGitHubTeamName>:<OrgIdOrName>:<Role>` mappings. Value can be `*` meaning "All users". Role is optional and can have the following values: `None`, `Viewer`, `Editor` or `Admin`. For more information on external organization to role mapping, refer to [Org roles mapping example](#org-roles-mapping-example).                                                                                                                                                                                                                                                                                                                                                         |                                               |
| `skip_org_role_sync`         | No       | Yes                | Set to `true` to stop automatically syncing user roles.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `false`                                       |
| `allow_assign_grafana_admin` | No       | No                 | Set to `true` to enable automatic sync of the Grafana server administrator role. If this option is set to `true` and the result of evaluating `role_attribute_path` for a user is `GrafanaAdmin`, Grafana grants the user the server administrator privileges and organization administrator role. If this option is set to `false` and the result of evaluating `role_attribute_path` for a user is `GrafanaAdmin`, Grafana grants the user only organization administrator role. For more information on user role mapping, refer to [Configure role mapping](#configure-role-mapping).                                                                                                                             | `false`                                       |
| `allowed_organizations`      | No       | Yes                | List of comma- or space-separated organizations. User must be a member of at least one organization to log in.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                                               |
| `allowed_domains`            | No       | Yes                | List of comma- or space-separated domains. User must belong to at least one domain to log in.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                                               |
| `team_ids`                   | No       | Yes                | Integer list of team IDs. If set, user has to be a member of one of the given teams to log in.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                                               |
| `tls_skip_verify_insecure`   | No       | No                 | If set to `true`, the client accepts any certificate presented by the server and any host name in that certificate. _You should only use this for testing_, because this mode leaves SSL/TLS susceptible to man-in-the-middle attacks.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `false`                                       |
| `tls_client_cert`            | No       | No                 | The path to the certificate.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |                                               |
| `tls_client_key`             | No       | No                 | The path to the key.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |                                               |
| `tls_client_ca`              | No       | No                 | The path to the trusted certificate authority list.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |                                               |
| `signout_redirect_url`       | No       | Yes                | URL to redirect to after the user logs out.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |                                               |

# Configure GitLab OAuth authentication

{{< docs/shared lookup="auth/intro.md" source="grafana" version="<GRAFANA VERSION>" >}}

This topic describes how to configure GitLab OAuth authentication.

{{< admonition type="note" >}}
If Users use the same email address in GitLab that they use with other authentication providers (such as Grafana.com), you need to do additional configuration to ensure that the users are matched correctly. Please refer to the [Using the same email address to login with different identity providers](../#using-the-same-email-address-to-login-with-different-identity-providers) documentation for more information.
{{< /admonition >}}

## Before you begin

Ensure you know how to create a GitLab OAuth application. Consult GitLab's documentation on [creating a GitLab OAuth application](https://docs.gitlab.com/ee/integration/oauth_provider.html) for more information.

### Create a GitLab OAuth Application

1. Log in to your GitLab account and go to **Profile > Preferences > Applications**.
1. Click **Add new application**.
1. Fill out the fields.
   - In the **Redirect URI** field, enter the following: `https://<YOUR-GRAFANA-URL>/login/gitlab` and check `openid`, `email`, `profile` in the **Scopes** list.
   - Leave the **Confidential** checkbox checked.
1. Click **Save application**.
1. Note your **Application ID** (this is the `Client Id`) and **Secret** (this is the `Client Secret`).

## Configure GitLab authentication client using the Grafana UI

As a Grafana Admin, you can configure GitLab OAuth client from within Grafana using the GitLab UI. To do this, navigate to **Administration > Authentication > GitLab** page and fill in the form. If you have a current configuration in the Grafana configuration file then the form will be pre-populated with those values otherwise the form will contain default values.

After you have filled in the form, click **Save** to save the configuration. If the save was successful, Grafana will apply the new configurations.

If you need to reset changes you made in the UI back to the default values, click **Reset**. After you have reset the changes, Grafana will apply the configuration from the Grafana configuration file (if there is any configuration) or the default values.

{{< admonition type="note" >}}
If you run Grafana in high availability mode, configuration changes may not get applied to all Grafana instances immediately. You may need to wait a few minutes for the configuration to propagate to all Grafana instances.
{{< /admonition >}}

Refer to [configuration options](#configuration-options) for more information.

## Configure GitLab authentication client using the Terraform provider

```terraform
resource "grafana_sso_settings" "gitlab_sso_settings" {
  provider_name = "gitlab"
  oauth2_settings {
    name                  = "Gitlab"
    client_id             = "YOUR_GITLAB_APPLICATION_ID"
    client_secret         = "YOUR_GITLAB_APPLICATION_SECRET"
    allow_sign_up         = true
    auto_login            = false
    scopes                = "openid email profile"
    allowed_domains       = "mycompany.com mycompany.org"
    role_attribute_path   = "contains(groups[*], 'example-group') && 'Editor' || 'Viewer'"
    role_attribute_strict = false
    allowed_groups        = "[\"admins\", \"software engineers\", \"developers/frontend\"]"
    use_pkce              = true
    use_refresh_token     = true
  }
}
```

Go to [Terraform Registry](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/sso_settings) for a complete reference on using the `grafana_sso_settings` resource.

## Configure GitLab authentication client using the Grafana configuration file

Ensure that you have access to the [Grafana configuration file](../../../configure-grafana/#configuration-file-location).

### Steps

To configure GitLab authentication with Grafana, follow these steps:

1. Create an OAuth application in GitLab.
   1. Set the redirect URI to `http://<my_grafana_server_name_or_ip>:<grafana_server_port>/login/gitlab`.

      Ensure that the Redirect URI is the complete HTTP address that you use to access Grafana via your browser, but with the appended path of `/login/gitlab`.

      For the Redirect URI to be correct, it might be necessary to set the `root_url` option in the `[server]`section of the Grafana configuration file. For example, if you are serving Grafana behind a proxy.

   1. Set the OAuth2 scopes to `openid`, `email` and `profile`.

1. Refer to the following table to update field values located in the `[auth.gitlab]` section of the Grafana configuration file:

   | Field                        | Description                                                                                   |
   | ---------------------------- | --------------------------------------------------------------------------------------------- |
   | `client_id`, `client_secret` | These values must match the `Application ID` and `Secret` from your GitLab OAuth application. |
   | `enabled`                    | Enables GitLab authentication. Set this value to `true`.                                      |

   Review the list of other GitLab [configuration options](#configuration-options) and complete them, as necessary.

1. Optional: [Configure a refresh token](#configure-a-refresh-token):

   a. Set `use_refresh_token` to `true` in `[auth.gitlab]` section in Grafana configuration file.

1. [Configure role mapping](#configure-role-mapping).
1. Optional: [Configure team synchronization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/).
1. Restart Grafana.

   You should now see a GitLab login button on the login page and be able to log in or sign up with your GitLab accounts.

### Configure a refresh token

When a user logs in using an OAuth provider, Grafana verifies that the access token has not expired. When an access token expires, Grafana uses the provided refresh token (if any exists) to obtain a new access token.

Grafana uses a refresh token to obtain a new access token without requiring the user to log in again. If a refresh token doesn't exist, Grafana logs the user out of the system after the access token has expired.

By default, GitLab provides a refresh token.

Refresh token fetching and access token expiration check is enabled by default for the GitLab provider since Grafana v10.1.0. If you would like to disable access token expiration check then set the `use_refresh_token` configuration value to `false`.

{{< admonition type="note" >}}
The `accessTokenExpirationCheck` feature toggle has been removed in Grafana v10.3.0. Use the `use_refresh_token` configuration value instead for configuring refresh token fetching and access token expiration check.
{{< /admonition >}}

### Configure JWT ID token validation

By default, Grafana extracts user information from ID tokens without validating their cryptographic signatures. To enhance security, you can enable JWT signature validation to ensure that ID tokens are authentic and have not been tampered with.

To enable JWT ID token validation:

1. Set `validate_id_token` to `true` in the `[auth.gitlab]` section of the Grafana configuration file.
2. Configure `jwk_set_url` with the URL of your GitLab instance's JSON Web Key Set (JWKS) endpoint. This endpoint provides the public keys used to verify JWT signatures.

   For GitLab.com, the JWKS endpoint is: `https://gitlab.com/oauth/discovery/keys`

   For self-hosted GitLab instances, the JWKS endpoint is typically: `https://<your-gitlab-domain>/oauth/discovery/keys`

Example configuration:

```ini
[auth.gitlab]
enabled = true
validate_id_token = true
jwk_set_url = https://gitlab.com/oauth/discovery/keys
client_id = <client id>
client_secret = <client secret>
scopes = openid email profile
```

{{< admonition type="note" >}}
When JWT validation is enabled, Grafana caches the JWKS keys to improve performance. The cache respects the `Cache-Control` header from the JWKS endpoint response. If no cache expiration is specified, keys are cached for 5 minutes by default.
{{< /admonition >}}

{{< admonition type="caution" >}}
If `validate_id_token` is set to `true`, you must configure `jwk_set_url`. Authentication will fail if the JWK Set URL is not provided or if the ID token signature cannot be verified.
{{< /admonition >}}

### Configure allowed groups

To limit access to authenticated users that are members of one or more [GitLab
groups](https://docs.gitlab.com/ce/user/group/index.html), set `allowed_groups`
to a comma or space-separated list of groups.

GitLab's groups are referenced by the group name. For example, `developers`. To reference a subgroup `frontend`, use `developers/frontend`.
Note that in GitLab, the group or subgroup name does not always match its display name, especially if the display name contains spaces or special characters.
Make sure you always use the group or subgroup name as it appears in the URL of the group or subgroup.

### Configure role mapping

Unless `skip_org_role_sync` option is enabled, the user's role will be set to the role retrieved from GitLab upon user login.

The user's role is retrieved using a [JMESPath](http://jmespath.org/examples.html) expression from the `role_attribute_path` configuration option.
To map the server administrator role, use the `allow_assign_grafana_admin` configuration option.
Refer to [configuration options](#configuration-options) for more information.

You can use the `org_mapping` configuration option to assign the user to multiple organizations and specify their role based on their GitLab group membership. For more information, refer to [Org roles mapping example](#org-roles-mapping-example). If the org role mapping (`org_mapping`) is specified and Entra ID returns a valid role, then the user will get the highest of the two roles.

If no valid role is found, the user is assigned the role specified by [the `auto_assign_org_role` option](../../../configure-grafana/#auto_assign_org_role).
You can disable this default role assignment by setting `role_attribute_strict = true`. This setting denies user access if no role or an invalid role is returned after evaluating the `role_attribute_path` and the `org_mapping` expressions.

To ease configuration of a proper JMESPath expression, go to [JMESPath](http://jmespath.org/) to test and evaluate expressions with custom payloads.

### Role mapping examples

This section includes examples of JMESPath expressions used for role mapping.

##### Org roles mapping example

The GitLab integration uses the external users' groups in the `org_mapping` configuration to map organizations and roles based on their GitLab group membership.

In this example, the user has been granted the role of a `Viewer` in the `org_foo` organization, and the role of an `Editor` in the `org_bar` and `org_baz` orgs.

The external user is part of the following GitLab groups: `groupd-1` and `group-2`.

Config:

```ini
org_mapping = group-1:org_foo:Viewer groupd-1:org_bar:Editor *:org_baz:Editor
```

#### Map roles using user information from OAuth token

In this example, the user with email `admin@company.com` has been granted the `Admin` role.
All other users are granted the `Viewer` role.

```ini
role_attribute_path = email=='admin@company.com' && 'Admin' || 'Viewer'
```

#### Map roles using groups

In this example, the user from GitLab group 'example-group' have been granted the `Editor` role.
All other users are granted the `Viewer` role.

```ini
role_attribute_path = contains(groups[*], 'example-group') && 'Editor' || 'Viewer'
```

#### Map server administrator role

In this example, the user with email `admin@company.com` has been granted the `Admin` organization role as well as the Grafana server admin role.
All other users are granted the `Viewer` role.

```bash
role_attribute_path = email=='admin@company.com' && 'GrafanaAdmin' || 'Viewer'
```

#### Map one role to all users

In this example, all users will be assigned `Viewer` role regardless of the user information received from the identity provider.

```ini
role_attribute_path = "'Viewer'"
skip_org_role_sync = false
```

### Example of GitLab configuration in Grafana

This section includes an example of GitLab configuration in the Grafana configuration file.

```bash
[auth.gitlab]
enabled = true
allow_sign_up = true
auto_login = false
client_id = YOUR_GITLAB_APPLICATION_ID
client_secret = YOUR_GITLAB_APPLICATION_SECRET
scopes = openid email profile
auth_url = https://gitlab.com/oauth/authorize
token_url = https://gitlab.com/oauth/token
api_url = https://gitlab.com/api/v4
role_attribute_path = contains(groups[*], 'example-group') && 'Editor' || 'Viewer'
role_attribute_strict = false
allow_assign_grafana_admin = false
allowed_groups = ["admins", "software engineers", "developers/frontend"]
allowed_domains = mycompany.com mycompany.org
tls_skip_verify_insecure = false
use_pkce = true
use_refresh_token = true
```

## Configure team synchronization

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/products/cloud/)
{{< /admonition >}}

By using Team Sync, you can map GitLab groups to teams within Grafana. This will automatically assign users to the appropriate teams.
Teams for each user are synchronized when the user logs in.

GitLab groups are referenced by the group name. For example, `developers`. To reference a subgroup `frontend`, use `developers/frontend`.
Note that in GitLab, the group or subgroup name does not always match its display name, especially if the display name contains spaces or special characters.
Make sure you always use the group or subgroup name as it appears in the URL of the group or subgroup.

To learn more about Team Sync, refer to [Configure team sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/).

## Configuration options

The following table describes all GitLab OAuth configuration options. You can apply these options as environment variables, similar to any other configuration within Grafana. For more information, refer to [Override configuration with environment variables](../../../configure-grafana/#override-configuration-with-environment-variables).

{{< admonition type="note" >}}
If the configuration option requires a JMESPath expression that includes a colon, enclose the entire expression in quotes to prevent parsing errors. For example `role_attribute_path: "role:view"`
{{< /admonition >}}

| Setting                      | Required | Supported on Cloud | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | Default                              |
| ---------------------------- | -------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| `enabled`                    | Yes      | Yes                | Whether GitLab OAuth authentication is allowed.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `false`                              |
| `client_id`                  | Yes      | Yes                | Client ID provided by your GitLab OAuth app.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |                                      |
| `client_secret`              | Yes      | Yes                | Client secret provided by your GitLab OAuth app.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                                      |
| `auth_url`                   | Yes      | Yes                | Authorization endpoint of your GitLab OAuth provider. If you use your own instance of GitLab instead of gitlab.com, adjust `auth_url` by replacing the `gitlab.com` hostname with your own.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `https://gitlab.com/oauth/authorize` |
| `token_url`                  | Yes      | Yes                | Endpoint used to obtain GitLab OAuth access token. If you use your own instance of GitLab instead of gitlab.com, adjust `token_url` by replacing the `gitlab.com` hostname with your own.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `https://gitlab.com/oauth/token`     |
| `token_exchange_timeout`     | No       | Yes                | Timeout in seconds for the OAuth token exchange request. If set to `0` or left unset, defaults to `15` seconds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `15`                                 |
| `api_url`                    | No       | Yes                | Grafana uses `<api_url>/user` endpoint to obtain GitLab user information compatible with [OpenID UserInfo](https://connect2id.com/products/server/docs/api/userinfo).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `https://gitlab.com/api/v4`          |
| `name`                       | No       | Yes                | Name used to refer to the GitLab authentication in the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | `GitLab`                             |
| `icon`                       | No       | Yes                | Icon used for GitLab authentication in the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `gitlab`                             |
| `scopes`                     | No       | Yes                | List of comma or space-separated GitLab OAuth scopes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `openid email profile`               |
| `allow_sign_up`              | No       | Yes                | Whether to allow new Grafana user creation through GitLab login. If set to `false`, then only existing Grafana users can log in with GitLab OAuth.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `true`                               |
| `auto_login`                 | No       | Yes                | Set to `true` to enable users to bypass the login screen and automatically log in. This setting is ignored if you configure multiple auth providers to use auto-login.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `false`                              |
| `login_prompt`               | No       | Yes                | Indicates the type of user interaction when the user logs in with GitLab. Available values are `login`, `consent` and `select_account`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |                                      |
| `role_attribute_path`        | No       | Yes                | [JMESPath](http://jmespath.org/examples.html) expression to use for Grafana role lookup. Grafana will first evaluate the expression using the GitLab OAuth token. If no role is found, Grafana creates a JSON data with `groups` key that maps to groups obtained from GitLab's `/oauth/userinfo` endpoint, and evaluates the expression using this data. Finally, if a valid role is still not found, the expression is evaluated against the user information retrieved from `api_url/users` endpoint and groups retrieved from `api_url/groups` endpoint. The result of the evaluation should be a valid Grafana role (`None`, `Viewer`, `Editor`, `Admin` or `GrafanaAdmin`). For more information on user role mapping, refer to [Configure role mapping](#configure-role-mapping). |                                      |
| `role_attribute_strict`      | No       | Yes                | Set to `true` to deny user login if the Grafana role cannot be extracted using `role_attribute_path`. For more information on user role mapping, refer to [Configure role mapping](#configure-role-mapping).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `false`                              |
| `org_mapping`                | No       | No                 | List of comma- or space-separated `<ExternalGitlabGroupName>:<OrgIdOrName>:<Role>` mappings. Value can be `*` meaning "All users". Role is optional and can have the following values: `None`, `Viewer`, `Editor` or `Admin`. For more information on external organization to role mapping, refer to [Org roles mapping example](#org-roles-mapping-example).                                                                                                                                                                                                                                                                                                                                                                                                                           |                                      |
| `skip_org_role_sync`         | No       | Yes                | Set to `true` to stop automatically syncing user roles.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | `false`                              |
| `allow_assign_grafana_admin` | No       | No                 | Set to `true` to enable automatic sync of the Grafana server administrator role. If this option is set to `true` and the result of evaluating `role_attribute_path` for a user is `GrafanaAdmin`, Grafana grants the user the server administrator privileges and organization administrator role. If this option is set to `false` and the result of evaluating `role_attribute_path` for a user is `GrafanaAdmin`, Grafana grants the user only organization administrator role. For more information on user role mapping, refer to [Configure role mapping](#configure-role-mapping).                                                                                                                                                                                                | `false`                              |
| `allowed_domains`            | No       | Yes                | List of comma or space-separated domains. User must belong to at least one domain to log in.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |                                      |
| `allowed_groups`             | No       | Yes                | List of comma or space-separated groups. The user should be a member of at least one group to log in.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |                                      |
| `tls_skip_verify_insecure`   | No       | No                 | If set to `true`, the client accepts any certificate presented by the server and any host name in that certificate. _You should only use this for testing_, because this mode leaves SSL/TLS susceptible to man-in-the-middle attacks.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `false`                              |
| `tls_client_cert`            | No       | No                 | The path to the certificate.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |                                      |
| `tls_client_key`             | No       | No                 | The path to the key.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |                                      |
| `tls_client_ca`              | No       | No                 | The path to the trusted certificate authority list.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |                                      |
| `use_pkce`                   | No       | Yes                | Set to `true` to use [Proof Key for Code Exchange (PKCE)](https://datatracker.ietf.org/doc/html/rfc7636). Grafana uses the SHA256 based `S256` challenge method and a 128 bytes (base64url encoded) code verifier.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `true`                               |
| `use_refresh_token`          | No       | Yes                | Set to `true` to use refresh token and check access token expiration. The `accessTokenExpirationCheck` feature toggle should also be enabled to use refresh token.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `true`                               |
| `validate_id_token`          | No       | Yes                | If enabled, Grafana will validate the JWT signature of ID tokens using the JWKS endpoint. This enhances security by ensuring tokens are authentic and have not been tampered with.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `false`                              |
| `jwk_set_url`                | No       | Yes                | URL of the JSON Web Key Set (JWKS) endpoint used to verify JWT ID token signatures. Required when ID token validation is enabled. For GitLab.com, use `https://gitlab.com/oauth/discovery/keys`. For self-hosted GitLab instances, use `https://<your-gitlab-domain>/oauth/discovery/keys`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |                                      |
| `signout_redirect_url`       | No       | Yes                | URL to redirect to after the user logs out.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |                                      |

# Configure Google OAuth authentication

To enable Google OAuth you must register your application with Google. Google will generate a client ID and secret key for you to use.

{{< admonition type="note" >}}
If Users use the same email address in Google that they use with other authentication providers (such as Grafana.com), you need to do additional configuration to ensure that the users are matched correctly. Please refer to the [Using the same email address to login with different identity providers](../#using-the-same-email-address-to-login-with-different-identity-providers) documentation for more information.
{{< /admonition >}}

## Create Google OAuth keys

First, you need to create a Google OAuth Client:

1. Go to https://console.developers.google.com/apis/credentials.
1. Create a new project if you don't have one already.
   1. Enter a project name. The **Organization** and **Location** fields should both be set to your organization's information.
   1. In **OAuth consent screen** select the **External** User Type. Click **CREATE**.
   1. Fill out the requested information using the URL of your Grafana Cloud instance.
   1. Accept the defaults, or customize the consent screen options.
1. Click **Create Credentials**, then click **OAuth Client ID** in the drop-down menu
1. Enter the following:
   - **Application Type**: Web application
   - **Name**: Grafana
   - **Authorized JavaScript origins**: `https://<YOUR_GRAFANA_URL>`
   - **Authorized redirect URIs**: `https://<YOUR_GRAFANA_URL>/login/google`
   - Replace `<YOUR_GRAFANA_URL>` with the URL of your Grafana instance.
     {{< admonition type="note" >}}
     The URL you enter is the one for your Grafana instance home page, not your Grafana Cloud portal URL.
     {{< /admonition >}}
1. Click Create
1. Copy the Client ID and Client Secret from the 'OAuth Client' modal

## Configure Google authentication client using the Grafana UI

As a Grafana Admin, you can configure Google OAuth client from within Grafana using the Google UI. To do this, navigate to **Administration > Authentication > Google** page and fill in the form. If you have a current configuration in the Grafana configuration file then the form will be pre-populated with those values otherwise the form will contain default values.

After you have filled in the form, click **Save**. If the save was successful, Grafana will apply the new configurations.

If you need to reset changes made in the UI back to the default values, click **Reset**. After you have reset the changes, Grafana will apply the configuration from the Grafana configuration file (if there is any configuration) or the default values.

{{< admonition type="note" >}}
If you run Grafana in high availability mode, configuration changes may not get applied to all Grafana instances immediately. You may need to wait a few minutes for the configuration to propagate to all Grafana instances.
{{< /admonition >}}

## Configure Google authentication client using the Terraform provider

```terraform
resource "grafana_sso_settings" "google_sso_settings" {
  provider_name = "google"
  oauth2_settings {
    name            = "Google"
    client_id       = "CLIENT_ID"
    client_secret   = "CLIENT_SECRET"
    allow_sign_up   = true
    auto_login      = false
    scopes          = "openid email profile"
    allowed_domains = "mycompany.com mycompany.org"
    hosted_domain   = "mycompany.com"
    use_pkce        = true
  }
}
```

Go to [Terraform Registry](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/sso_settings) for a complete reference on using the `grafana_sso_settings` resource.

## Configure Google authentication client using the Grafana configuration file

Ensure that you have access to the [Grafana configuration file](../../../configure-grafana/#configuration-file-location).

### Enable Google OAuth in Grafana

Specify the Client ID and Secret in the [Grafana configuration file](../../../configure-grafana/#configuration-file-location). For example:

```bash
[auth.google]
enabled = true
allow_sign_up = true
auto_login = false
client_id = CLIENT_ID
client_secret = CLIENT_SECRET
scopes = openid email profile
auth_url = https://accounts.google.com/o/oauth2/v2/auth
token_url = https://oauth2.googleapis.com/token
api_url = https://openidconnect.googleapis.com/v1/userinfo
allowed_domains = mycompany.com mycompany.org
hosted_domain = mycompany.com
use_pkce = true
```

You may have to set the `root_url` option of `[server]` for the callback URL to be
correct. For example, in case you are serving Grafana behind a proxy.

Restart the Grafana backend. You should now see a Google login button
on the login page. You can now login or sign up with your Google
accounts. The `allowed_domains` option is optional, and domains were separated by space.

You may allow users to sign-up via Google authentication by setting the
`allow_sign_up` option to `true`. When this option is set to `true`, any
user successfully authenticating via Google authentication will be
automatically signed up.

You may specify a domain to be passed as `hd` query parameter accepted by Google's
OAuth 2.0 authentication API. Refer to Google's OAuth [documentation](https://developers.google.com/identity/openid-connect/openid-connect#hd-param).

{{< admonition type="note" >}}
Since Grafana 10.3.0, the `hd` parameter retrieved from Google ID token is also used to determine the user's hosted domain. The Google Oauth `allowed_domains` configuration option is used to restrict access to users from a specific domain. If the `allowed_domains` configuration option is set, the `hd` parameter from the Google ID token must match the `allowed_domains` configuration option. If the `hd` parameter from the Google ID token does not match the `allowed_domains` configuration option, the user is denied access.

When an account does not belong to a google workspace, the `hd` claim will not be available.

This validation is enabled by default. To disable this validation, set the `validate_hd` configuration option to `false`. The `allowed_domains` configuration option will use the email claim to validate the domain.
{{< /admonition >}}

#### PKCE

IETF's [RFC 7636](https://datatracker.ietf.org/doc/html/rfc7636)
introduces "proof key for code exchange" (PKCE) which provides
additional protection against some forms of authorization code
interception attacks. PKCE will be required in [OAuth 2.1](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1-03).

{{< admonition type="note" >}}
You can disable PKCE in Grafana by setting `use_pkce` to `false` in the`[auth.google]` section.
{{< /admonition >}}

#### Configure refresh token

When a user logs in using an OAuth provider, Grafana verifies that the access token has not expired. When an access token expires, Grafana uses the provided refresh token (if any exists) to obtain a new access token.

Grafana uses a refresh token to obtain a new access token without requiring the user to log in again. If a refresh token doesn't exist, Grafana logs the user out of the system after the access token has expired.

By default, Grafana includes the `access_type=offline` parameter in the authorization request to request a refresh token.

Refresh token fetching and access token expiration check is enabled by default for the Google provider since Grafana v10.1.0. If you would like to disable access token expiration check then set the `use_refresh_token` configuration value to `false`.

{{< admonition type="note" >}}
The `accessTokenExpirationCheck` feature toggle has been removed in Grafana v10.3.0 and the `use_refresh_token` configuration value will be used instead for configuring refresh token fetching and access token expiration check.
{{< /admonition >}}

#### Configure JWT ID token validation

By default, Grafana extracts user information from ID tokens without validating their cryptographic signatures. To enhance security, you can enable JWT signature validation to ensure that ID tokens are authentic and have not been tampered with.

To enable JWT ID token validation:

1. Set `validate_id_token` to `true` in the `[auth.google]` section of the Grafana configuration file.
2. Configure `jwk_set_url` with the URL of Google's JSON Web Key Set (JWKS) endpoint. This endpoint provides the public keys used to verify JWT signatures.

   For Google, the JWKS endpoint is: `https://www.googleapis.com/oauth2/v3/certs`

Example configuration:

```ini
[auth.google]
enabled = true
validate_id_token = true
jwk_set_url = https://www.googleapis.com/oauth2/v3/certs
client_id = <client id>
client_secret = <client secret>
scopes = openid email profile
```

{{< admonition type="note" >}}
When JWT validation is enabled, Grafana caches the JWKS keys to improve performance. The cache respects the `Cache-Control` header from the JWKS endpoint response. If no cache expiration is specified, keys are cached for 5 minutes by default.
{{< /admonition >}}

{{< admonition type="caution" >}}
If `validate_id_token` is set to `true`, you must configure `jwk_set_url`. Authentication will fail if the JWK Set URL is not provided or if the ID token signature cannot be verified.
{{< /admonition >}}

#### Configure automatic login

Set the `auto_login` option to true to attempt log in automatically, skipping the login screen.
This setting is ignored if multiple auth providers are configured to use auto login.

```
auto_login = true
```

### Configure team synchronization

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/products/cloud/).
{{< /admonition >}}

With team sync, you can easily add users to teams by utilizing their Google groups. To set up team sync for Google OAuth, refer to the following example.

To set up team sync for Google OAuth:

1. Enable the Google Cloud Identity API on your [organization's dashboard](https://console.cloud.google.com/apis/api/cloudidentity.googleapis.com/).

1. Add the `https://www.googleapis.com/auth/cloud-identity.groups.readonly` scope to your Grafana `[auth.google]` configuration:

   Example:

   ```ini
   [auth.google]
   # ..
   scopes = openid email profile https://www.googleapis.com/auth/cloud-identity.groups.readonly
   ```

1. Configure team sync in your Grafana team's `External group sync` tab.
   The external group ID for a Google group is the group's email address, such as `dev@grafana.com`.

To learn more about Team Sync, refer to [Configure Team Sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/).

#### Configure allowed groups

To limit access to authenticated users that are members of one or more groups, set `allowed_groups`
to a comma or space separated list of groups.

Google groups are referenced by the group email key. For example, `developers@google.com`.

{{< admonition type="note" >}}
Add the `https://www.googleapis.com/auth/cloud-identity.groups.readonly` scope to your Grafana `[auth.google]` scopes configuration to retrieve groups.
{{< /admonition >}}

#### Configure role mapping

Unless the `skip_org_role_sync` option is enabled, the user's role will be set to the role mapped from Google upon user login. If no mapping is set the default instance role is used.

The user's role is retrieved using a [JMESPath](http://jmespath.org/examples.html) expression from the `role_attribute_path` configuration option.
To map the server administrator role, use the `allow_assign_grafana_admin` configuration option.

If no valid role is found, the user is assigned the role specified by [the `auto_assign_org_role` option](../../../configure-grafana/#auto_assign_org_role).
You can disable this default role assignment by setting `role_attribute_strict = true`. This setting denies user access if no role or an invalid role is returned after evaluating the `role_attribute_path` and the `org_mapping` expressions.

To ease configuration of a proper JMESPath expression, go to [JMESPath](http://jmespath.org/) to test and evaluate expressions with custom payloads.
{{< admonition type="note" >}}
By default the `skip_org_role_sync` option is enabled. The `skip_org_role_sync` option defaults to false in Grafana v10.3.0 and later versions.
{{< /admonition >}}

##### Role mapping examples

This section includes examples of JMESPath expressions used for role mapping.

##### Org roles mapping example

The Google integration uses the external users' groups in the `org_mapping` configuration to map organizations and roles based on their Google group membership.

In this example, the user has been granted the role of a `Viewer` in the `org_foo` organization, and the role of an `Editor` in the `org_bar` and `org_baz` orgs.

The external user is part of the following Google groups: `group-1` and `group-2`.

Config:

```ini
org_mapping = group-1:org_foo:Viewer group-2:org_bar:Editor *:org_baz:Editor
```

###### Map roles using user information from OAuth token

In this example, the user with email `admin@company.com` has been granted the `Admin` role.
All other users are granted the `Viewer` role.

```ini
role_attribute_path = email=='admin@company.com' && 'Admin' || 'Viewer'
skip_org_role_sync = false
```

###### Map roles using groups

In this example, the user from Google group 'example-group@google.com' have been granted the `Editor` role.
All other users are granted the `Viewer` role.

```ini
role_attribute_path = contains(groups[*], 'example-group@google.com') && 'Editor' || 'Viewer'
skip_org_role_sync = false
```

{{< admonition type="note" >}}
Add the `https://www.googleapis.com/auth/cloud-identity.groups.readonly` scope to your Grafana `[auth.google]` scopes configuration to retrieve groups.
{{< /admonition >}}

###### Map server administrator role

In this example, the user with email `admin@company.com` is granted the `Admin` organization role as well as the Grafana server admin role.
All other users are granted the `Viewer` role.

```ini
allow_assign_grafana_admin = true
skip_org_role_sync = false
role_attribute_path = email=='admin@company.com' && 'GrafanaAdmin' || 'Viewer'
```

###### Map one role to all users

In this example, all users are assigned the `Viewer` role regardless of the user information received from the identity provider.

```ini
role_attribute_path = "'Viewer'"
skip_org_role_sync = false
```

## Configuration options

The following table outlines the various Google OAuth configuration options. You can apply these options as environment variables, similar to any other configuration within Grafana. For more information, refer to [Override configuration with environment variables](../../../configure-grafana/#override-configuration-with-environment-variables).

| Setting                      | Required | Supported on Cloud | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Default                                            |
| ---------------------------- | -------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| `enabled`                    | No       | Yes                | Enables Google authentication.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | `false`                                            |
| `name`                       | No       | Yes                | Name that refers to the Google authentication from the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                  | `Google`                                           |
| `icon`                       | No       | Yes                | Icon used for the Google authentication in the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                          | `google`                                           |
| `client_id`                  | Yes      | Yes                | Client ID of the App.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |                                                    |
| `client_secret`              | Yes      | Yes                | Client secret of the App.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |                                                    |
| `auth_url`                   | Yes      | Yes                | Authorization endpoint of the Google OAuth provider.                                                                                                                                                                                                                                                                                                                                                                                                                                            | `https://accounts.google.com/o/oauth2/v2/auth`     |
| `token_url`                  | Yes      | Yes                | Endpoint used to obtain the OAuth2 access token.                                                                                                                                                                                                                                                                                                                                                                                                                                                | `https://oauth2.googleapis.com/token`              |
| `token_exchange_timeout`     | No       | Yes                | Timeout in seconds for the OAuth token exchange request. If set to `0` or left unset, defaults to `15` seconds.                                                                                                                                                                                                                                                                                                                                                                                 | `15`                                               |
| `api_url`                    | Yes      | Yes                | Endpoint used to obtain user information compatible with [OpenID UserInfo](https://connect2id.com/products/server/docs/api/userinfo).                                                                                                                                                                                                                                                                                                                                                           | `https://openidconnect.googleapis.com/v1/userinfo` |
| `auth_style`                 | No       | Yes                | Name of the [OAuth2 AuthStyle](https://pkg.go.dev/golang.org/x/oauth2#AuthStyle) to be used when ID token is requested from OAuth2 provider. It determines how `client_id` and `client_secret` are sent to Oauth2 provider. Available values are `AutoDetect`, `InParams` and `InHeader`.                                                                                                                                                                                                       | `AutoDetect`                                       |
| `scopes`                     | No       | Yes                | List of comma- or space-separated OAuth2 scopes.                                                                                                                                                                                                                                                                                                                                                                                                                                                | `openid email profile`                             |
| `allow_sign_up`              | No       | Yes                | Controls Grafana user creation through the Google login. Only existing Grafana users can log in with Google if set to `false`.                                                                                                                                                                                                                                                                                                                                                                  | `true`                                             |
| `auto_login`                 | No       | Yes                | Set to `true` to enable users to bypass the login screen and automatically log in. This setting is ignored if you configure multiple auth providers to use auto-login.                                                                                                                                                                                                                                                                                                                          | `false`                                            |
| `login_prompt`               | No       | Yes                | Indicates the type of user interaction when the user logs in with Google. Available values are `login`, `consent` and `select_account`.                                                                                                                                                                                                                                                                                                                                                         |                                                    |
| `hosted_domain`              | No       | Yes                | Specifies the domain to restrict access to users from that domain. This value is appended to the authorization request using the `hd` parameter.                                                                                                                                                                                                                                                                                                                                                |                                                    |
| `validate_hd`                | No       | Yes                | Set to `false` to disable the validation of the `hd` parameter from the Google ID token. For more informatiion, refer to [Enable Google OAuth in Grafana](#enable-google-oauth-in-grafana).                                                                                                                                                                                                                                                                                                     | `true`                                             |
| `role_attribute_strict`      | No       | Yes                | Set to `true` to deny user login if the Grafana org role cannot be extracted using `role_attribute_path` or `org_mapping`. For more information on user role mapping, refer to [Configure role mapping](#configure-role-mapping).                                                                                                                                                                                                                                                               | `false`                                            |
| `org_attribute_path`         | No       | No                 | [JMESPath](http://jmespath.org/examples.html) expression to use for Grafana org to role lookup. Grafana will first evaluate the expression using the OAuth2 ID token. If no value is returned, the expression will be evaluated using the user information obtained from the UserInfo endpoint. The result of the evaluation will be mapped to org roles based on `org_mapping`. For more information on org to role mapping, refer to [Org roles mapping example](#org-roles-mapping-example). |                                                    |
| `org_mapping`                | No       | No                 | List of comma- or space-separated `<ExternalOrgName>:<OrgIdOrName>:<Role>` mappings. Value can be `*` meaning "All users". Role is optional and can have the following values: `None`, `Viewer`, `Editor` or `Admin`. For more information on external organization to role mapping, refer to [Org roles mapping example](#org-roles-mapping-example).                                                                                                                                          |                                                    |
| `allow_assign_grafana_admin` | No       | No                 | Set to `true` to automatically sync the Grafana server administrator role. When enabled, if the Google user's App role is `GrafanaAdmin`, Grafana grants the user server administrator privileges and the organization administrator role. If disabled, the user will only receive the organization administrator role. For more details on user role mapping, refer to [Map roles](#map-roles).                                                                                                | `false`                                            |
| `skip_org_role_sync`         | No       | Yes                | Set to `true` to stop automatically syncing user roles. This will allow you to set organization roles for your users from within Grafana manually.                                                                                                                                                                                                                                                                                                                                              | `false`                                            |
| `allowed_groups`             | No       | Yes                | List of comma- or space-separated groups. The user should be a member of at least one group to log in. If you configure `allowed_groups`, you must also configure Google to include the `groups` claim following [Configure allowed groups](#configure-allowed-groups).                                                                                                                                                                                                                         |                                                    |
| `allowed_organizations`      | No       | Yes                | List of comma- or space-separated Azure tenant identifiers. The user should be a member of at least one tenant to log in.                                                                                                                                                                                                                                                                                                                                                                       |                                                    |
| `allowed_domains`            | No       | Yes                | List of comma- or space-separated domains. The user should belong to at least one domain to log in.                                                                                                                                                                                                                                                                                                                                                                                             |                                                    |
| `tls_skip_verify_insecure`   | No       | No                 | If set to `true`, the client accepts any certificate presented by the server and any host name in that certificate. _You should only use this for testing_, because this mode leaves SSL/TLS susceptible to man-in-the-middle attacks.                                                                                                                                                                                                                                                          | `false`                                            |
| `tls_client_cert`            | No       | No                 | The path to the certificate.                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |                                                    |
| `tls_client_key`             | No       | No                 | The path to the key.                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                    |
| `tls_client_ca`              | No       | No                 | The path to the trusted certificate authority list.                                                                                                                                                                                                                                                                                                                                                                                                                                             |                                                    |
| `use_pkce`                   | No       | Yes                | Set to `true` to use [Proof Key for Code Exchange (PKCE)](https://datatracker.ietf.org/doc/html/rfc7636). Grafana uses the SHA256 based `S256` challenge method and a 128 bytes (base64url encoded) code verifier.                                                                                                                                                                                                                                                                              | `true`                                             |
| `use_refresh_token`          | No       | Yes                | Enables the use of refresh tokens and checks for access token expiration. When enabled, Grafana automatically adds the `promp=consent` and `access_type=offline` parameters to the authorization request.                                                                                                                                                                                                                                                                                       | `true`                                             |
| `validate_id_token`          | No       | Yes                | If enabled, Grafana will validate the JWT signature of ID tokens using the JWKS endpoint. This enhances security by ensuring tokens are authentic and have not been tampered with.                                                                                                                                                                                                                                                                                                              | `false`                                            |
| `jwk_set_url`                | No       | Yes                | URL of the JSON Web Key Set (JWKS) endpoint used to verify JWT ID token signatures. Required when ID token validation is enabled. For Google, use `https://www.googleapis.com/oauth2/v3/certs`.                                                                                                                                                                                                                                                                                                 |                                                    |
| `signout_redirect_url`       | No       | Yes                | URL to redirect to after the user logs out.                                                                                                                                                                                                                                                                                                                                                                                                                                                     |                                                    |

# Configure basic authentication

Grafana provides a basic authentication system with password authentication enabled by default. This document details configuration options to manage and enhance basic authentication.

## Disable basic authentication

To disable basic authentication, use the following configuration:

```bash
[auth.basic]
enabled = false
```

## Password policy

By default, Grafana’s password policy requires a minimum of four characters for basic auth users. For a stronger password policy, enable the `password_policy` configuration option.

With the `password_policy` option enabled, new and updated passwords must meet the following criteria:

- At least 12 characters
- At least one uppercase letter
- At least one lowercase letter
- At least one number
- At least one special character

```bash
[auth.basic]
password_policy = true
```

{{< admonition type="note" >}}
Existing passwords that do not comply with the new password policy will not be affected until the user updates their password.
{{< /admonition >}}

## Disable login form

To hide the Grafana login form, use the following configuration setting:

```bash
[auth]
disable_login_form = true
```

This can be helpful in setups where authentication is handled entirely through external mechanisms or single sign-on (SSO).

# Configure Grafana Cloud authentication

To enable Grafana Cloud as the Identity Provider for a Grafana instance, generate a client ID and client secret and apply the configuration to Grafana.

## Create Grafana Cloud OAuth Client Credentials

To use Grafana Cloud authentication:

1. Log in to [Grafana Cloud](/).
1. To create an OAuth client, locate your organization and click **OAuth Clients**.
1. Click **Add OAuth Client Application**.
1. Add the name and URL of your running Grafana instance.
1. Click **Add OAuth Client**.
1. Copy the client ID and client secret or the configuration that has been generated.

The following snippet shows an example configuration:

```ini
[auth.grafana_com]
enabled = true
allow_sign_up = true
auto_login = false
client_id = 450bc21c10dc2194879d
client_secret = eyJ0Ijoib2F1dGgyYyIhlmlkIjoiNzUwYmMzM2MxMGRjMjE6NDh3OWQiLCJ2IjoiZmI1YzVlYmIwYzFmN2ZhYzZmNjIwOGI1NmVkYTRlNWYxMzgwM2NkMiJ9
scopes = user:email
allowed_organizations = sampleorganization
enabled = true
```

### Configure automatic login

Set `auto_login` option to true to attempt login automatically, skipping the login screen.
This setting is ignored if multiple auth providers are configured to use auto login.

```
auto_login = true
```

## Skip organization role sync

If a user signs in with their Grafana Cloud credentials, their assigned org role overrides the role defined in the Grafana instance. To prevent Grafana Cloud roles from synchronizing, set `skip_org_role_sync` to `true`. This is useful if you want to manage the organization roles for your users from within Grafana.

```ini
[auth.grafana_com]
# ..
# prevents the sync of org roles from Grafana.com
skip_org_role_sync = true
```

# Configure JWT authentication

You can configure Grafana to accept a JWT token provided in the HTTP header. The token is verified using any of the following:

- PEM-encoded key file
- JSON Web Key Set (JWKS) in a local file
- JWKS provided by the configured JWKS endpoint

This method of authentication is useful for integrating with other systems that
use JWKS but can't directly integrate with Grafana or if you want to use pass-through
authentication in an app embedding Grafana.

{{< admonition type="note" >}}
Grafana does not currently support refresh tokens.
{{< /admonition >}}

## Enable JWT

To use JWT authentication:

1. Enable JWT in the [main config file](../../../configure-grafana/).
1. Specify the header name that contains a token.

```ini
[auth.jwt]
# By default, auth.jwt is disabled.
enabled = true

# HTTP header to look into to get a JWT token.
header_name = X-JWT-Assertion
```

## Configure login claim

To identify the user, some of the claims needs to be selected as a login info. The subject claim called `"sub"` is mandatory and needs to identify the principal that is the subject of the JWT.

Typically, the subject claim called `"sub"` would be used as a login but it might also be set to some application specific claim.

```ini
# [auth.jwt]
# ...

# Specify a claim to use as a username to sign in.
username_claim = sub

# Specify a claim to use as an email to sign in.
email_claim = sub

# auto-create users if they are not already matched
# auto_sign_up = true
```

If `auto_sign_up` is enabled, then the `sub` claim is used as the "external Auth ID". The `name` claim is used as the user's full name if it is present.

Additionally, if the login username or the email claims are nested inside the JWT structure, you can specify the path to the attributes using the `username_attribute_path` and `email_attribute_path` configuration options using the JMESPath syntax.

JWT structure example.

```json
{
  "user": {
    "UID": "1234567890",
    "name": "John Doe",
    "username": "johndoe",
    "emails": ["personal@example.com", "professional@example.com"]
  }
}
```

```ini
# [auth.jwt]
# ...

# Specify a nested attribute to use as a username to sign in.
username_attribute_path = user.username # user's login is johndoe

# Specify a nested attribute to use as an email to sign in.
email_attribute_path = user.emails[1] # user's email is professional@example.com
```

## Iframe Embedding

If you want to embed Grafana in an iframe while maintaining user identity and role checks,
you can use JWT authentication to authenticate the iframe.

{{< admonition type="note" >}}
For Grafana Cloud, or scenarios where verifying viewer identity is not required,
embed [shared dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/share-dashboards-panels/shared-dashboards/).
{{< /admonition >}}

In this scenario, you will need to configure Grafana to accept a JWT
provided in the HTTP header and a reverse proxy should rewrite requests to the
Grafana instance to include the JWT in the request's headers.

{{< admonition type="note" >}}
For embedding to work, you must enable `allow_embedding` in the [security section](../../../configure-grafana/#allow_embedding). This setting is not available in Grafana Cloud.
{{< /admonition >}}

In a scenario where it is not possible to rewrite the request headers you
can use URL login instead.

### URL login

`url_login` allows grafana to search for a JWT in the URL query parameter
`auth_token` and use it as the authentication token.

{{< admonition type="note" >}}
You need to enable JWT before setting this setting. Refer to [Enabled JWT](#enable-jwt).
{{< /admonition >}}

{{< admonition type="warning" >}}
This can lead to JWTs being exposed in logs and possible session hijacking if the server is not
using HTTP over TLS.
{{< /admonition >}}

```ini
# [auth.jwt]
# ...
url_login = true # enable JWT authentication in the URL
```

An example of an URL for accessing grafana with JWT URL authentication is:

```
http://env.grafana.local/d/RciOKLR4z/board-identifier?orgId=1&kiosk&auth_token=eyJhbxxxxxxxxxxxxx
```

A sample repository using this authentication method is available
at [grafana-iframe-oauth-sample](https://github.com/grafana/grafana-iframe-oauth-sample).

## Signature verification

JSON web token integrity needs to be verified so cryptographic signature is used for this purpose. So we expect that every token must be signed with some known cryptographic key.

You have a variety of options on how to specify where the keys are located.

### Verify token using a JSON Web Key Set loaded from https endpoint

For more information on JWKS endpoints, refer to [Auth0 docs](https://auth0.com/docs/tokens/json-web-tokens/json-web-key-sets).

```ini
# [auth.jwt]
# ...

jwk_set_url = https://your-auth-provider.example.com/.well-known/jwks.json

# When the JWKS url requires an 'Authorization: Bearer <TOKEN>' header
# jwk_set_bearer_token_file = /path/to/bearer_token

# Cache duration for https endpoint response.
cache_ttl = 60m

# Path to file containing one or more custom PEM-encoded CA certificates.
# Used with jwk_set_url when the JWKS endpoint uses a certificate that is not
# trusted by the default CA bundle (e.g. self-signed certificates).
# tls_client_ca = /path/to/ca.crt

# Skip CA Verification entirely
# tls_skip_verify_insecure = false
```

{{< admonition type="note" >}}
If the JWKS endpoint includes cache control headers and the value is less than the configured `cache_ttl`, then the cache control header value is used instead. If the `cache_ttl` is not set, the default of `60m` is used. `no-store` and `no-cache` cache control headers are ignored. To disable JWKS caching, set `cache_ttl = 0s`
{{< /admonition >}}

### Verify token using a JSON Web Key Set loaded from JSON file

Key set in the same format as in JWKS endpoint but located on disk.

```ini
jwk_set_file = /path/to/jwks.json
```

### Verify token using a single key loaded from PEM-encoded file

PEM-encoded public key file in PKIX or PKCS #1 format. Private keys are not accepted.

```ini
key_file = /path/to/key.pem
```

If the JWT token's header specifies a `kid` (Key ID), then the Key ID must be set using the `key_id` configuration option.

```ini
key_id = my-key-id
```

### Verify token using an inline key or key set

When you can't place a file on disk, put the key material directly in the configuration. This is useful when you provision configuration to a managed instance. Both options use the same content as their file-based counterparts, base64-encoded so it fits on one line.

Inline a single PEM-encoded public key instead of `key_file`:

```ini
key_value = <base64-encoded PEM key>
```

Inline a JWKS document instead of `jwk_set_file`:

```ini
jwk_set_value = <base64-encoded JWKS JSON>
```

To generate a value from an existing file, run `base64 < key.pem | tr -d '\n'`. The `key_id` option applies to `key_value` just as it does to `key_file`.

{{< admonition type="note" >}}
Configure the key set with exactly one of `key_file`, `key_value`, `jwk_set_file`, `jwk_set_value`, or `jwk_set_url`. Setting more than one is an error.
{{< /admonition >}}

## Validate claims

By default, only `"exp"`, `"nbf"` and `"iat"` claims are validated.

Consider validating that other claims match your expectations by using the `expect_claims` configuration option.
Token claims must match exactly the values set here.

```ini
# This can be seen as a required "subset" of a JWT Claims Set.
expect_claims = {"iss": "https://your-token-issuer", "your-custom-claim": "foo"}
```

## Roles

Grafana checks for the presence of a role using the [JMESPath](http://jmespath.org/examples.html) specified via the `role_attribute_path` configuration option. The JMESPath is applied to JWT token claims. The result after evaluation of the `role_attribute_path` JMESPath expression should be a valid Grafana role, for example, `None`, `Viewer`, `Editor` or `Admin`.

To assign the role to a specific organization include the `X-Grafana-Org-Id` header along with your JWT when making API requests to Grafana.
To learn more about the header, please refer to the [documentation](../../../../developers/http_api/#x-grafana-org-id-header).

### Configure role mapping

Unless `skip_org_role_sync` option is enabled, the user's role will be set to the role retrieved from the JWT.

The user's role is retrieved using a [JMESPath](http://jmespath.org/examples.html) expression from the `role_attribute_path` configuration option.
To map the server administrator role, use the `allow_assign_grafana_admin` configuration option.

If no valid role is found, the user is assigned the role specified by [the `auto_assign_org_role` option](../../../configure-grafana/#auto_assign_org_role).
You can disable this default role assignment by setting `role_attribute_strict = true`. This setting denies user access if no role or an invalid role is returned after evaluating the `role_attribute_path` and the `org_mapping` expressions.

You can use the `org_attribute_path` and `org_mapping` configuration options to assign the user to organizations and specify their role. For more information, refer to [Org roles mapping example](#org-roles-mapping-example). If both org role mapping (`org_mapping`) and the regular role mapping (`role_attribute_path`) are specified, then the user will get the highest of the two mapped roles.

To ease configuration of a proper JMESPath expression, go to [JMESPath](http://jmespath.org/) to test and evaluate expressions with custom payloads.

**Basic example:**

In the following example user will get `Editor` as role when authenticating. The value of the property `role` will be the resulting role if the role is a proper Grafana role, i.e. `None`, `Viewer`, `Editor` or `Admin`.

Payload:

```json
{
    ...
    "role": "Editor",
    ...
}
```

Configuration:

```ini
role_attribute_path = role
```

**Advanced example:**

In the following example user will get `Admin` as role when authenticating since it has a role `admin`. If a user has a role `editor` it will get `Editor` as role, otherwise `Viewer`.

Payload:

```json
{
    ...
    "info": {
        ...
        "roles": [
            "engineer",
            "admin",
        ],
        ...
    },
    ...
}
```

Configuration:

```ini
role_attribute_path = contains(info.roles[*], 'admin') && 'Admin' || contains(info.roles[*], 'editor') && 'Editor' || 'Viewer'
```

**Org roles mapping example**

In the following example, the , the user has been granted the role of a `Viewer` in the `org_foo` organization, and the role of an `Editor` in the `org_bar` and `org_baz` organizations.

Payload:

```json
{
    ...
    "info": {
        ...
        "orgs": [
            "engineer",
            "admin",
        ],
        ...
    },
    ...
}
```

Configuration:

```ini
org_attribute_path = info.orgs
org_mapping = engineer:org_foo:Viewer admin:org_bar:Editor *:org_baz:Editor
```

### Grafana Admin Role

If the `role_attribute_path` property returns a `GrafanaAdmin` role, Grafana Admin is not assigned by default, instead the `Admin` role is assigned. To allow `Grafana Admin` role to be assigned set `allow_assign_grafana_admin = true`.

### Skip organization role mapping

To skip the assignment of roles and permissions upon login via JWT and handle them via other mechanisms like the user interface, we can skip the organization role synchronization with the following configuration.

```ini
[auth.jwt]
# ...

skip_org_role_sync = true
```

# Configure Keycloak OAuth2 authentication

Keycloak OAuth2 authentication allows users to log in to Grafana using their Keycloak credentials. This guide explains how to set up Keycloak as an authentication provider in Grafana.

Refer to [Generic OAuth authentication](../generic-oauth/) for extra configuration options available for this provider.

{{< admonition type="note" >}}
If you use the same email address in Keycloak as in other authentication providers (such as Grafana.com), you need to do additional configuration to ensure that the users are matched correctly. Please refer to the [Using the same email address to login with different identity providers](../#using-the-same-email-address-to-login-with-different-identity-providers) documentation for more information.
{{< /admonition >}}

You may have to set the `root_url` option of `[server]` for the callback URL to be
correct. For example in case you are serving Grafana behind a proxy.

Example config:

```ini
[auth.generic_oauth]
enabled = true
name = Keycloak-OAuth
allow_sign_up = true
client_id = YOUR_APP_CLIENT_ID
client_secret = YOUR_APP_CLIENT_SECRET
scopes = openid email profile offline_access roles
email_attribute_path = email
login_attribute_path = username
name_attribute_path = full_name
auth_url = https://<PROVIDER_DOMAIN>/realms/<REALM_NAME>/protocol/openid-connect/auth
token_url = https://<PROVIDER_DOMAIN>/realms/<REALM_NAME>/protocol/openid-connect/token
api_url = https://<PROVIDER_DOMAIN>/realms/<REALM_NAME>/protocol/openid-connect/userinfo
role_attribute_path = contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'
```

As an example, `<PROVIDER_DOMAIN>` can be `keycloak-demo.grafana.org`
and `<REALM_NAME>` can be `grafana`.

To configure the `kc_idp_hint` parameter for Keycloak, you need to change the `auth_url` configuration to include the `kc_idp_hint` parameter. For example if you want to hint the Google identity provider:

```ini
auth_url = https://<PROVIDER_DOMAIN>/realms/<REALM_NAME>/protocol/openid-connect/auth?kc_idp_hint=google
```

{{< admonition type="note" >}}
api_url is not required if the id_token contains all the necessary user information and can add latency to the login process.
It is useful as a fallback or if the user has more than 150 group memberships.
{{< /admonition >}}

## Keycloak configuration

1. Create a client in Keycloak with the following settings:

- Client ID: `grafana-oauth`
- Enabled: `ON`
- Client Protocol: `openid-connect`
- Access Type: `confidential`
- Standard Flow Enabled: `ON`
- Implicit Flow Enabled: `OFF`
- Direct Access Grants Enabled: `ON`
- Root URL: `<grafana_root_url>`
- Valid Redirect URIs: `<grafana_root_url>/login/generic_oauth`
- Web Origins: `<grafana_root_url>`
- Admin URL: `<grafana_root_url>`
- Base URL: `<grafana_root_url>`

As an example, `<grafana_root_url>` can be `https://play.grafana.org`.
Non-listed configuration options can be left at their default values.

2. In the client scopes configuration, _Assigned Default Client Scopes_ should match:

```
email
offline_access
profile
roles
```

{{< admonition type="warning" >}}
These scopes do not add group claims to the `id_token`. Without group claims, teamsync will not work. Teamsync is covered further down in this document.
{{< /admonition >}}

3. For role mapping to work with the example configuration above,
   you need to create the following roles and assign them to users:

```
admin
editor
viewer
```

## Team sync

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/products/cloud/).
{{< /admonition >}}

[Team Sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/) is a feature that allows you to map groups from your identity provider to Grafana teams. This is useful if you want to give your users access to specific dashboards or folders based on their group membership.

To enable teamsync, you need to add a `groups` mapper to the client configuration in Keycloak.
This will add the `groups` claim to the id_token. You can then use the `groups` claim to map groups to teams in Grafana.

1. In the client configuration, head to `Mappers` and create a mapper with the following settings:

- Name: `Group Mapper`
- Mapper Type: `Group Membership`
- Token Claim Name: `groups`
- Full group path: `OFF`
- Add to ID token: `ON`
- Add to access token: `OFF`
- Add to userinfo: `ON`

2. In Grafana's configuration add the following option:

```ini
[auth.generic_oauth]
groups_attribute_path = groups
```

If you use nested groups containing special characters such as quotes or colons, the JMESPath parser can perform a harmless reverse function so Grafana can properly evaluate nested groups. The following example shows a parent group named `Global` with nested group `department` that contains a list of groups:

```ini
[auth.generic_oauth]
groups_attribute_path = reverse("Global:department")
```

## Enable Single Logout

To enable Single Logout, you need to add the following option to the configuration of Grafana:

```ini
[auth.generic_oauth]
signout_redirect_url = https://<PROVIDER_DOMAIN>/realms/<REALM_NAME>/protocol/openid-connect/logout?post_logout_redirect_uri=https%3A%2F%2F<GRAFANA_DOMAIN>%2Flogin
```

As an example, `<PROVIDER_DOMAIN>` can be `keycloak-demo.grafana.org`,
`<REALM_NAME>` can be `grafana` and `<GRAFANA_DOMAIN>` can be `play.grafana.org`.

{{< admonition type="note" >}}
Grafana supports ID token hints for single logout. Grafana automatically adds the `id_token_hint` parameter to the logout request if it detects OAuth as the authentication method.
{{< /admonition >}}

## Allow assigning Grafana Admin

If the application role received by Grafana is `GrafanaAdmin` , Grafana grants the user server administrator privileges.

This is useful if you want to grant server administrator privileges to a subset of users.
Grafana also assigns the user the `Admin` role of the default organization.

```ini
role_attribute_path = contains(roles[*], 'grafanaadmin') && 'GrafanaAdmin' || contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'
allow_assign_grafana_admin = true
```

### Configure refresh token

When a user logs in using an OAuth provider, Grafana verifies that the access token has not expired. When an access token expires, Grafana uses the provided refresh token (if any exists) to obtain a new access token.

Grafana uses a refresh token to obtain a new access token without requiring the user to log in again. If a refresh token doesn't exist, Grafana logs the user out of the system after the access token has expired.

To enable a refresh token for Keycloak, do the following:

1. Extend the `scopes` in `[auth.generic_oauth]` with `offline_access`.

1. Add `use_refresh_token = true` to `[auth.generic_oauth]` configuration.

# Multiple providers with Keycloak in Grafana

While Grafana offers a variety of authentication providers, you can only configure one provider of one type at a time. However, you can configure multiple providers of the same type with the help of Keycloak.

This guide explains how to set up multiple providers of the same type with Keycloak as an authentication provider in Grafana.

The idea is to set up multiple OIDC providers in Keycloak with different tenants and configure Grafana to use the same Keycloak instance as the authentication provider.

## Entra ID configuration

For Entra ID, repeat the following steps for each tenant you want to set up in Keycloak.

### Overview

1. Register your application in Entra ID.
1. Give access to the application to the users in the tenant.
1. Create credentials for the application.
1. Configure the application in Keycloak.
1. Configure Grafana to use Keycloak.

#### Register your application in Entra ID

Registering an application in Entra ID is a one-time process. You can follow the steps in the [Entra ID documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) to register your application.

1. Go to the Azure portal and ensure you are using the correct tenant also known as directory.
1. Search for **App Registrations** and click on **New registration**.
1. Fill in the details for the application and click **Register**. You'll be redirected to the application's overview page.

#### Give access to the application to the users in the tenant

Assigning the correct access to users ensures only intended users or groups have access to the application.

1. Search for **Enterprise Applications** and look for the application you just created in the previous step.
1. Under the **Manage** section, click on **Users and groups**.
1. Click on **Add user/group** and add the users or groups that should have access to the application.

#### Create credentials for the application

To authenticate with Entra ID, the Keycloak application needs a client ID and client secret.

1. Search for **App Registrations** and look for the application ypu just created.
1. Click on **Certificates & Secrets**.
1. Click on **New client secret** and fill in the details. Make sure to copy the secret value as it will not be shown again.

#### Configure the application in Keycloak

1. Go to the Keycloak admin console.
1. Go to the Realm where you want to configure the Entra ID tenant.
1. Go to the Identity Providers section and click on **Add provider**.
1. Select **OpenID Connect v1.0**.
   1. Select a unique **Alias** and **Display name**.
   1. Copy the **Redirect URI**.
   1. Back in Azure Portal, go to the application's **Authentication** section.
   1. Add a **new platform** and select **Web**.
   1. Paste the **Redirect URI** from Keycloak.
   1. Save the changes.
   1. Navigate to the Azure Application overview and look for the **Endpoints** tab.
   1. Copy the **OpenID Connect metadata document** URL.
   1. Head back to Keycloak and paste the URL in the **Discovery endpoint** field.
   1. Navigate to the Azure application overview and look for the **Application (client) ID**.
   1. Copy the **Application ID** and paste it in the **Client ID** field in Keycloak.
   1. Paste the client secret you created in the previous step in the **Client secret** field.
   1. Click Add.

{{< admonition type="note" >}}
Up to this point, you have created an App Registration in Entra ID, assigned users to the application, created credentials for the application, and configured the application in Keycloak. In the Keycloak Client's section, the client with ID `account` Home URL can be used to test the configuration. This will open a new tab where you can login into the correct Keycloak realm with the Entra ID tenant you just configured.
{{< /admonition >}}

Repeat this steps, for every Entra ID tenant you want to configure in Keycloak.

#### Configure Grafana to use Keycloak

Now that the Entra ID tenants are configured in Keycloak, you can configure Grafana to use Keycloak as the authentication provider.

Refer to the [Keycloak documentation](https://grafana.com/docs/grafana/latest/auth/keycloak/) to configure Grafana to use Keycloak as the authentication provider.

# Configure LDAP authentication

The LDAP integration in Grafana allows your Grafana users to login with their LDAP credentials. You can also specify mappings between LDAP
group memberships and Grafana Organization user roles.

{{< admonition type="note" >}}
[Enhanced LDAP authentication](../enhanced-ldap/) is available in [Grafana Cloud](/docs/grafana-cloud/) and in [Grafana Enterprise](../../../../introduction/grafana-enterprise/).
{{< /admonition >}}

Refer to [Role-based access control](../../../../administration/roles-and-permissions/access-control/) to understand how you can control access with role-based permissions.

## Supported LDAP Servers

Grafana uses a [third-party LDAP library](https://github.com/go-ldap/ldap) under the hood that supports basic LDAP v3 functionality.
This means that you should be able to configure LDAP integration using any compliant LDAPv3 server, for example [OpenLDAP](#openldap) or
[Active Directory](#active-directory) among [others](https://en.wikipedia.org/wiki/Directory_service#LDAP_implementations).

## Enable LDAP

In order to use LDAP integration you'll first need to enable LDAP in the [main config file](../../../configure-grafana/) as well as specify the path to the LDAP
specific configuration file (default: `/etc/grafana/ldap.toml`).

After enabling LDAP, the default behavior is for Grafana users to be created automatically upon successful LDAP authentication. If you prefer for only existing Grafana users to be able to sign in, you can change `allow_sign_up` to `false` in the `[auth.ldap]` section.

```ini
[auth.ldap]
# Set to `true` to enable LDAP integration (default: `false`)
enabled = true

# Path to the LDAP specific configuration file (default: `/etc/grafana/ldap.toml`)
config_file = /etc/grafana/ldap.toml

# Allow sign-up should be `true` (default) to allow Grafana to create users on successful LDAP authentication.
# If set to `false` only already existing Grafana users will be able to login.
allow_sign_up = true
```

## Disable org role synchronization

If you use LDAP to authenticate users but don't use role mapping, and prefer to manually assign organizations
and roles, you can use the `skip_org_role_sync` configuration option.

```ini
[auth.ldap]
# Set to `true` to enable LDAP integration (default: `false`)
enabled = true

# Path to the LDAP specific configuration file (default: `/etc/grafana/ldap.toml`)
config_file = /etc/grafana/ldap.toml

# Allow sign-up should be `true` (default) to allow Grafana to create users on successful LDAP authentication.
# If set to `false` only already existing Grafana users will be able to login.
allow_sign_up = true

# Prevent synchronizing ldap users organization roles
skip_org_role_sync = true
```

## Grafana LDAP Configuration

Depending on which LDAP server you're using and how that's configured, your Grafana LDAP configuration may vary.
See [configuration examples](#configuration-examples) for more information.

**LDAP specific configuration file (ldap.toml) example:**

```bash
[[servers]]
# Ldap server host (specify multiple hosts space separated)
host = "ldap.my_secure_remote_server.org"
# Default port is 389 or 636 if use_ssl = true
port = 636
# Set to true if LDAP server should use an encrypted TLS connection (either with STARTTLS or LDAPS)
use_ssl = true
# If set to true, use LDAP with STARTTLS instead of LDAPS
start_tls = false
# The value of an accepted TLS cipher. By default, this value is empty. Example value: ["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"])
# For a complete list of supported ciphers and TLS versions, refer to: https://go.dev/src/crypto/tls/cipher_suites.go
# Starting with Grafana v11.0 only ciphers with ECDHE support are accepted for TLS 1.2 connections.
tls_ciphers = []
# This is the minimum TLS version allowed. By default, this value is empty. Accepted values are: TLS1.1 (only for Grafana v10.4 or earlier), TLS1.2, TLS1.3.
min_tls_version = ""
# set to true if you want to skip SSL cert validation
ssl_skip_verify = false
# set to the path to your root CA certificate or leave unset to use system defaults
# root_ca_cert = "/path/to/certificate.crt"
# Authentication against LDAP servers requiring client certificates
# client_cert = "/path/to/client.crt"
# client_key = "/path/to/client.key"

# Search user bind dn
bind_dn = "cn=admin,dc=grafana,dc=org"
# Search user bind password
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
bind_password = "grafana"
# We recommend using variable expansion for the bind_password, for more info https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#variable-expansion
# bind_password = '$__env{LDAP_BIND_PASSWORD}'

# Timeout in seconds. Applies to each host specified in the 'host' entry (space separated).
timeout = 10

# User search filter, for example "(cn=%s)" or "(sAMAccountName=%s)" or "(uid=%s)"
# Allow login from email or username, example "(|(sAMAccountName=%s)(userPrincipalName=%s))"
search_filter = "(cn=%s)"

# An array of base dns to search through
search_base_dns = ["dc=grafana,dc=org"]

# group_search_filter = "(&(objectClass=posixGroup)(memberUid=%s))"
# group_search_filter_user_attribute = "distinguishedName"
# group_search_base_dns = ["ou=groups,dc=grafana,dc=org"]

# Specify names of the LDAP attributes your LDAP uses
[servers.attributes]
member_of = "memberOf"
email =  "email"
```

{{< admonition type="note" >}}
Whenever you modify the ldap.toml file, you must restart Grafana in order for the change(s) to take effect.
{{< /admonition >}}

### Using the Grafana user interface

You can configure LDAP using the Grafana user interface by navigating to **Administration > Authentication > LDAP**. Please refer to the [LDAP user interface](../ldap-ui/) documentation for more information.

### Using environment variables

You can interpolate variables in the TOML configuration from environment variables. For instance, you could externalize your `bind_password` that way:

```bash
bind_password = "${LDAP_ADMIN_PASSWORD}"
```

### Bind and bind password

By default the configuration expects you to specify a bind DN and bind password. This should be a read only user that can perform LDAP searches.
When the user DN is found a second bind is performed with the user provided username and password (in the normal Grafana login form).

```bash
bind_dn = "cn=admin,dc=grafana,dc=org"
bind_password = "grafana"
```

#### Single bind example

If you can provide a single bind expression that matches all possible users, you can skip the second bind and bind against the user DN directly.
This allows you to not specify a bind_password in the configuration file.

```bash
bind_dn = "cn=%s,o=users,dc=grafana,dc=org"
```

In this case you skip providing a `bind_password` and instead provide a `bind_dn` value with a `%s` somewhere. This will be replaced with the username entered in on the Grafana login page.
The search filter and search bases settings are still needed to perform the LDAP search to retrieve the other LDAP information (like LDAP groups and email).

### POSIX schema

If your LDAP server does not support the `memberOf` attribute, add the following options:

```bash
## Group search filter, to retrieve the groups of which the user is a member (only set if memberOf attribute is not available)
group_search_filter = "(&(objectClass=posixGroup)(memberUid=%s))"
## An array of the base DNs to search through for groups. Typically uses ou=groups
group_search_base_dns = ["ou=groups,dc=grafana,dc=org"]
## the %s in the search filter will be replaced with the attribute defined below
group_search_filter_user_attribute = "uid"
```

### Group mappings

In `[[servers.group_mappings]]` you can map an LDAP group to a Grafana organization and role. These will be synced every time the user logs in, with LDAP being the authoritative source.

The first group mapping that an LDAP user is matched to will be used for the sync. If you have LDAP users that fit multiple mappings, the topmost mapping in the TOML configuration will be used.

**LDAP specific configuration file (ldap.toml) example:**

```bash
[[servers]]
# other settings omitted for clarity

[[servers.group_mappings]]
group_dn = "cn=superadmins,dc=grafana,dc=org"
org_role = "Admin"
grafana_admin = true

[[servers.group_mappings]]
group_dn = "cn=admins,dc=grafana,dc=org"
org_role = "Admin"

[[servers.group_mappings]]
group_dn = "cn=users,dc=grafana,dc=org"
org_role = "Editor"

[[servers.group_mappings]]
group_dn = "*"
org_role = "Viewer"
```

| Setting         | Required | Description                                                                                                                                           | Default              |
| --------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
| `group_dn`      | Yes      | LDAP distinguished name (DN) of LDAP group. If you want to match all (or no LDAP groups) then you can use wildcard (`"*"`)                            |
| `org_role`      | Yes      | Assign users of `group_dn` the organization role `Admin`, `Editor`, or `Viewer`. The organization role name is case sensitive.                        |
| `org_id`        | No       | The Grafana organization database id. Setting this allows for multiple group_dn's to be assigned to the same `org_role` provided the `org_id` differs | `1` (default org id) |
| `grafana_admin` | No       | When `true` makes user of `group_dn` Grafana server admin. A Grafana server admin has admin access over all organizations and users.                  | `false`              |

{{< admonition type="note" >}}
Commenting out a group mapping requires also commenting out the header of
said group or it will fail validation as an empty mapping.
{{< /admonition >}}

Example:

```bash
[[servers]]
# other settings omitted for clarity

[[servers.group_mappings]]
group_dn = "cn=superadmins,dc=grafana,dc=org"
org_role = "Admin"
grafana_admin = true

# [[servers.group_mappings]]
# group_dn = "cn=admins,dc=grafana,dc=org"
# org_role = "Admin"

[[servers.group_mappings]]
group_dn = "cn=users,dc=grafana,dc=org"
org_role = "Editor"
```

### Nested/recursive group membership

Users with nested/recursive group membership must have an LDAP server that supports `LDAP_MATCHING_RULE_IN_CHAIN`
and configure `group_search_filter` in a way that it returns the groups the submitted username is a member of.

To configure `group_search_filter`:

- You can set `group_search_base_dns` to specify where the matching groups are defined.
- If you do not use `group_search_base_dns`, then the previously defined `search_base_dns` is used.

**Active Directory example:**

Active Directory groups store the Distinguished Names (DNs) of members, so your filter needs to know the DN for the user based only on the submitted username.
Multiple DN templates are searched by combining filters with the LDAP OR-operator. Two examples:

```bash
group_search_filter = "(member:1.2.840.113556.1.4.1941:=%s)"
group_search_base_dns = ["DC=mycorp,DC=mytld"]
group_search_filter_user_attribute = "dn"
```

```bash
group_search_filter = "(member:1.2.840.113556.1.4.1941:=CN=%s,[user container/OU])"
group_search_filter = "(|(member:1.2.840.113556.1.4.1941:=CN=%s,[user container/OU])(member:1.2.840.113556.1.4.1941:=CN=%s,[another user container/OU]))"
group_search_filter_user_attribute = "cn"
```

For more information on AD searches refer to [Microsoft's Search Filter Syntax](https://docs.microsoft.com/en-us/windows/desktop/adsi/search-filter-syntax) documentation.

For troubleshooting, changing `member_of` in `[servers.attributes]` to "dn" will show you more accurate group memberships when [debug is enabled](#troubleshooting).

## Configuration examples

The following examples describe different LDAP configuration options.

### OpenLDAP

[OpenLDAP](http://www.openldap.org/) is an open source directory service.

**LDAP specific configuration file (ldap.toml):**

```bash
[[servers]]
host = "127.0.0.1"
port = 389
use_ssl = false
start_tls = false
ssl_skip_verify = false
bind_dn = "cn=admin,dc=grafana,dc=org"
bind_password = "grafana"
search_filter = "(cn=%s)"
search_base_dns = ["dc=grafana,dc=org"]

[servers.attributes]
member_of = "memberOf"
email =  "email"

# [[servers.group_mappings]] omitted for clarity
```

### Multiple LDAP servers

Grafana does support receiving information from multiple LDAP servers.

**LDAP specific configuration file (ldap.toml):**

```bash
# --- First LDAP Server ---

[[servers]]
host = "10.0.0.1"
port = 389
use_ssl = false
start_tls = false
ssl_skip_verify = false
bind_dn = "cn=admin,dc=grafana,dc=org"
bind_password = "grafana"
search_filter = "(cn=%s)"
search_base_dns = ["ou=users,dc=grafana,dc=org"]

[servers.attributes]
member_of = "memberOf"
email =  "email"

[[servers.group_mappings]]
group_dn = "cn=admins,ou=groups,dc=grafana,dc=org"
org_role = "Admin"
grafana_admin = true

# --- Second LDAP Server ---

[[servers]]
host = "10.0.0.2"
port = 389
use_ssl = false
start_tls = false
ssl_skip_verify = false

bind_dn = "cn=admin,dc=grafana,dc=org"
bind_password = "grafana"
search_filter = "(cn=%s)"
search_base_dns = ["ou=users,dc=grafana,dc=org"]

[servers.attributes]
member_of = "memberOf"
email =  "email"

[[servers.group_mappings]]
group_dn = "cn=editors,ou=groups,dc=grafana,dc=org"
org_role = "Editor"

[[servers.group_mappings]]
group_dn = "*"
org_role = "Viewer"
```

### Active Directory

[Active Directory](<https://technet.microsoft.com/en-us/library/hh831484(v=ws.11).aspx>) is a directory service which is commonly used in Windows environments.

Assuming the following Active Directory server setup:

- IP address: `10.0.0.1`
- Domain: `CORP`
- DNS name: `corp.local`

**LDAP specific configuration file (ldap.toml):**

```bash
[[servers]]
host = "10.0.0.1"
port = 3269
use_ssl = true
start_tls = false
ssl_skip_verify = true
bind_dn = "CORP\\%s"
search_filter = "(sAMAccountName=%s)"
search_base_dns = ["dc=corp,dc=local"]

[servers.attributes]
member_of = "memberOf"
email =  "mail"

# [[servers.group_mappings]] omitted for clarity
```

#### Port requirements

In the previous example, SSL is enabled and an encrypted port has been configured. If your Active Directory doesn't support SSL, use `enable_ssl = false` and `port = 389` instead.

Inspect your Active Directory configuration and documentation to find the correct settings. For more information about Active Directory and port requirements, refer to the [Microsoft documentation](https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/service-overview-and-network-port-requirements#active-directory-local-security-authority).

## Troubleshooting

To troubleshoot and get more log information, enable LDAP debug logging in the [`grafana.ini` or `custom.ini`](../../../configure-grafana/) file:

```bash
[log]
filters = ldap:debug
```

# Configure Okta OIDC authentication

{{< docs/shared lookup="auth/intro.md" source="grafana" version="<GRAFANA VERSION>" >}}

{{< admonition type="note" >}}
If Users use the same email address in Okta that they use with other authentication providers (such as Grafana.com), you need to do additional configuration to ensure that the users are matched correctly. Please refer to the [Using the same email address to login with different identity providers](../#using-the-same-email-address-to-login-with-different-identity-providers) documentation for more information.
{{< /admonition >}}

## Before you begin

To follow this guide, ensure you have permissions in your Okta workspace to create an OIDC app.

## Create an Okta app

1. From the Okta Admin Console, select **Create App Integration** from the **Applications** menu.
1. For **Sign-in method**, select **OIDC - OpenID Connect**.
1. For **Application type**, select **Web Application** and click **Next**.
1. Configure **New Web App Integration Operations**:
   - **App integration name**: Choose a name for the app.
   - **Logo (optional)**: Add a logo.
   - **Grant type**: Select **Authorization Code** and **Refresh Token**.
   - **Sign-in redirect URIs**: Replace the default setting with the Grafana Cloud Okta path, replacing <YOUR_ORG> with the name of your Grafana organization: https://<YOUR_ORG>.grafana.net/login/okta. For on-premises installation, use the Grafana server URL: http://<my_grafana_server_name_or_ip>:<grafana_server_port>/login/okta.
   - **Sign-out redirect URIs (optional)**: Replace the default setting with the Grafana Cloud Okta path, replacing <YOUR_ORG> with the name of your Grafana organization: https://<YOUR_ORG>.grafana.net/logout. For on-premises installation, use the Grafana server URL: http://<my_grafana_server_name_or_ip>:<grafana_server_port>/logout.
   - **Base URIs (optional)**: Add any base URIs
   - **Controlled access**: Select whether to assign the app integration to everyone in your organization, or only selected groups. You can assign this option after you create the app.

1. Make a note of the following:
   - **ClientID**
   - **Client Secret**
   - **Auth URL**
     For example: https://<TENANT_ID>.okta.com/oauth2/v1/authorize
   - **Token URL**
     For example: https://<TENANT_ID>.okta.com/oauth2/v1/token
   - **API URL**
     For example: https://<TENANT_ID>.okta.com/oauth2/v1/userinfo

### Configure Okta to Grafana role mapping

1. In the **Okta Admin Console**, select **Directory > Profile Editor**.
1. Select the Okta Application Profile you created previously (the default name for this is `<App name> User`).
1. Select **Add Attribute** and fill in the following fields:
   - **Data Type**: string
   - **Display Name**: Meaningful name. For example, `Grafana Role`.
   - **Variable Name**: Meaningful name. For example, `grafana_role`.
   - **Description (optional)**: A description of the role.
   - **Enum**: Select **Define enumerated list of values** and add the following:
     - Display Name: Admin Value: Admin
     - Display Name: Editor Value: Editor
     - Display Name: Viewer Value: Viewer

   The remaining attributes are optional and can be set as needed.

1. Click **Save**.
1. (Optional) You can add the role attribute to the default User profile. To do this, please follow the steps in the [Optional: Add the role attribute to the User (default) Okta profile](#optional-add-the-role-attribute-to-the-user-default-okta-profile) section.

### Configure Groups claim

1. In the **Okta Admin Console**, select **Application > Applications**.
1. Select the OpenID Connect application you created.
1. Go to the **Sign On** tab and click **Edit** in the **OpenID Connect ID Token** section.
1. In the **Group claim type** section, select **Filter**.
1. In the **Group claim filter** section, leave the default name `groups` (or add it if the box is empty), then select **Matches regex** and add the following regex: `.*`.
1. Click **Save**.
1. Click the **Back to applications** link at the top of the page.
1. From the **More** button dropdown menu, click **Refresh Application Data**.
1. Include the `groups` scope in the **Scopes** field in Grafana of the Okta integration.
   For Terraform or in the Grafana configuration file, include the `groups` scope in `scopes` field.

{{< admonition type="note" >}}
If you configure the `groups` claim differently, ensure that the `groups` claim is a string array.
{{< /admonition >}}

#### Optional: Add the role attribute to the User (default) Okta profile

If you want to configure the role for all users in the Okta directory, you can add the role attribute to the User (default) Okta profile.

1. Return to the **Directory** section and select **Profile Editor**.
1. Select the User (default) Okta profile, and click **Add Attribute**.
1. Set all of the attributes in the same way you did in **Step 3**.
1. Select **Add Mapping** to add your new attributes.
   For example, **user.grafana_role -> grafana_role**.
1. To add a role to a user, select the user from the **Directory**, and click **Profile -> Edit**.
1. Select an option from your new attribute and click **Save**.
1. Update the Okta integration by setting the `Role attribute path` (`role_attribute_path` in Terraform and config file) to `<YOUR_ROLE_VARIABLE>`. For example: `role_attribute_path = grafana_role` (using the configuration).

## Configure Okta authentication client using the Grafana UI

As a Grafana Admin, you can configure Okta OAuth2 client from within Grafana using the Okta UI. To do this, navigate to **Administration > Authentication > Okta** page and fill in the form. If you have a current configuration in the Grafana configuration file then the form will be pre-populated with those values otherwise the form will contain default values.

After you have filled in the form, click **Save**. If the save was successful, Grafana will apply the new configurations.

If you need to reset changes you made in the UI back to the default values, click **Reset**. After you have reset the changes, Grafana will apply the configuration from the Grafana configuration file (if there is any configuration) or the default values.

{{< admonition type="note" >}}
If you run Grafana in high availability mode, configuration changes may not get applied to all Grafana instances immediately. You may need to wait a few minutes for the configuration to propagate to all Grafana instances.
{{< /admonition >}}

Refer to [configuration options](#configuration-options) for more information.

## Configure Okta authentication client using the Terraform provider

```terraform
resource "grafana_sso_settings" "okta_sso_settings" {
  provider_name = "okta"
  oauth2_settings {
    name                  = "Okta"
    auth_url              = "https://<okta tenant id>.okta.com/oauth2/v1/authorize"
    token_url             = "https://<okta tenant id>.okta.com/oauth2/v1/token"
    api_url               = "https://<okta tenant id>.okta.com/oauth2/v1/userinfo"
    client_id             = "CLIENT_ID"
    client_secret         = "CLIENT_SECRET"
    allow_sign_up         = true
    auto_login            = false
    scopes                = "openid profile email offline_access"
    role_attribute_path   = "contains(groups[*], 'Example::DevOps') && 'Admin' || 'None'"
    role_attribute_strict = true
    allowed_groups        = "Example::DevOps,Example::Dev,Example::QA"
  }
}
```

Go to [Terraform Registry](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/sso_settings) for a complete reference on using the `grafana_sso_settings` resource.

## Configure Okta authentication client using the Grafana configuration file

Ensure that you have access to the [Grafana configuration file](../../../configure-grafana/#configuration-file-location).

### Steps

To integrate your Okta OIDC provider with Grafana using our Okta OIDC integration, follow these steps:

1. Follow the [Create an Okta app](#create-an-okta-app) steps to create an OIDC app in Okta.

1. Refer to the following table to update field values located in the `[auth.okta]` section of the Grafana configuration file:

   | Field       | Description                                                                                                 |
   | ----------- | ----------------------------------------------------------------------------------------------------------- |
   | `client_id` | These values must match the client ID from your Okta OIDC app.                                              |
   | `auth_url`  | The authorization endpoint of your OIDC provider. `https://<okta-tenant-id>.okta.com/oauth2/v1/authorize`   |
   | `token_url` | The token endpoint of your Okta OIDC provider. `https://<okta-tenant-id>.okta.com/oauth2/v1/token`          |
   | `api_url`   | The user information endpoint of your Okta OIDC provider. `https://<tenant-id>.okta.com/oauth2/v1/userinfo` |
   | `enabled`   | Enables Okta OIDC authentication. Set this value to `true`.                                                 |

1. Review the list of other Okta OIDC [configuration options](#configuration-options) and complete them as necessary.

1. Optional: [Configure a refresh token](#configure-a-refresh-token).
1. [Configure role mapping](#configure-role-mapping).
1. Optional: [Configure team synchronization](#configure-team-synchronization-enterprise-only).
1. Restart Grafana.

   You should now see a Okta OIDC login button on the login page and be able to log in or sign up with your OIDC provider.

The following is an example of a minimally functioning integration when
configured with the instructions above:

```ini
[auth.okta]
name = Okta
icon = okta
enabled = true
allow_sign_up = true
client_id = <client id>
scopes = openid profile email offline_access
auth_url = https://<okta tenant id>.okta.com/oauth2/v1/authorize
token_url = https://<okta tenant id>.okta.com/oauth2/v1/token
api_url = https://<okta tenant id>.okta.com/oauth2/v1/userinfo
role_attribute_path = grafana_role
role_attribute_strict = true
allowed_groups = "Example::DevOps" "Example::Dev" "Example::QA"
```

### Configure a refresh token

When a user logs in using an OAuth provider, Grafana verifies that the access token has not expired. When an access token expires, Grafana uses the provided refresh token (if any exists) to obtain a new access token without requiring the user to log in again.

If a refresh token doesn't exist, Grafana logs the user out of the system after the access token has expired.

To enable the `Refresh Token` head over the Okta application settings and:

1. Under `General` tab, find the `General Settings` section.
1. Within the `Grant Type` options, enable the `Refresh Token` checkbox.

At the configuration file, extend the `scopes` in `[auth.okta]` section with `offline_access` and set `use_refresh_token` to `true`.

### Configure JWT ID token validation

By default, Grafana extracts user information from ID tokens without validating their cryptographic signatures. To enhance security, you can enable JWT signature validation to ensure that ID tokens are authentic and have not been tampered with.

To enable JWT ID token validation:

1. Set `validate_id_token` to `true` in the `[auth.okta]` section of the Grafana configuration file.
2. Configure `jwk_set_url` with the URL of your Okta tenant's JSON Web Key Set (JWKS) endpoint. This endpoint provides the public keys used to verify JWT signatures.

   For Okta, the JWKS endpoint is: `https://<tenant-id>.okta.com/oauth2/v1/keys`

Example configuration:

```ini
[auth.okta]
enabled = true
validate_id_token = true
jwk_set_url = https://<tenant-id>.okta.com/oauth2/v1/keys
client_id = <client id>
client_secret = <client secret>
scopes = openid profile email groups
```

{{< admonition type="note" >}}
When JWT validation is enabled, Grafana caches the JWKS keys to improve performance. The cache respects the `Cache-Control` header from the JWKS endpoint response. If no cache expiration is specified, keys are cached for 5 minutes by default.
{{< /admonition >}}

{{< admonition type="caution" >}}
If `validate_id_token` is set to `true`, you must configure `jwk_set_url`. Authentication will fail if the JWK Set URL is not provided or if the ID token signature cannot be verified.
{{< /admonition >}}

### Configure role mapping

{{< admonition type="note" >}}
Unless `skip_org_role_sync` option is enabled, the user's role will be set to the role retrieved from the auth provider upon user login.
{{< /admonition >}}

The user's role is retrieved using a [JMESPath](http://jmespath.org/examples.html) expression from the `role_attribute_path` configuration option against the `api_url` (`/userinfo` OIDC endpoint) endpoint payload.

If no valid role is found, the user is assigned the role specified by [the `auto_assign_org_role` option](../../../configure-grafana/#auto_assign_org_role).
You can disable this default role assignment by setting `role_attribute_strict = true`. This setting denies user access if no role or an invalid role is returned after evaluating the `role_attribute_path` and the `org_mapping` expressions.

You can use the `org_attribute_path` and `org_mapping` configuration options to assign the user to organizations and specify their role. For more information, refer to [Org roles mapping example](#org-roles-mapping-example). If both org role mapping (`org_mapping`) and the regular role mapping (`role_attribute_path`) are specified, then the user will get the highest of the two mapped roles.

To allow mapping Grafana server administrator role, use the `allow_assign_grafana_admin` configuration option.
Refer to [configuration options](../generic-oauth/#configuration-options) for more information.

In [Create an Okta app](#create-an-okta-app), you created a custom attribute in Okta to store the role. You can use this attribute to map the role to a Grafana role by setting the `role_attribute_path` configuration option to the custom attribute name: `role_attribute_path = grafana_role`.

If you want to map the role based on the user's group, you can use the `groups` attribute from the user info endpoint. An example of this is `role_attribute_path = contains(groups[*], 'Example::DevOps') && 'Admin' || 'None'`. You can find more examples of JMESPath expressions on the Generic OAuth page for [JMESPath examples](../generic-oauth/#role-mapping-examples).

To learn about adding custom claims to the user info in Okta, refer to [add custom claims](https://developer.okta.com/docs/guides/customize-tokens-returned-from-okta/main/#add-a-custom-claim-to-a-token).

#### Org roles mapping example

{{< admonition type="note" >}}
Available in self-managed Grafana installations.
{{< /admonition >}}

In this example, the `org_mapping` uses the `groups` attribute as the source (`org_attribute_path`) to map the current user to different organizations and roles. The user has been granted the role of a `Viewer` in the `org_foo` org if they are a member of the `Group 1` group, the role of an `Editor` in the `org_bar` org if they are a member of the `Group 2` group, and the role of an `Editor` in the `org_baz`(OrgID=3) org.

Config:

```ini
org_attribute_path = groups
org_mapping = ["Group 1:org_foo:Viewer", "Group 2:org_bar:Editor", "*:3:Editor"]
```

### Configure team synchronization

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/products/cloud/).
{{< /admonition >}}

By using Team Sync, you can link your Okta groups to teams within Grafana. This will automatically assign users to the appropriate teams.

Map your Okta groups to teams in Grafana so that your users will automatically be added to
the correct teams.

Okta groups can be referenced by group names, like `Admins` or `Editors`.

To learn more about Team Sync, refer to [Configure Team Sync](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-team-sync/).

## Configuration options

The following table outlines the various Okta OIDC configuration options. You can apply these options as environment variables, similar to any other configuration within Grafana. For more information, refer to [Override configuration with environment variables](../../../configure-grafana/#override-configuration-with-environment-variables).

{{< admonition type="note" >}}
If the configuration option requires a JMESPath expression that includes a colon, enclose the entire expression in quotes to prevent parsing errors. For example `role_attribute_path: "role:view"`
{{< /admonition >}}

| Setting                  | Required | Supported on Cloud | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | Default                       |
| ------------------------ | -------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
| `enabled`                | No       | Yes                | Enables Okta OIDC authentication.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `false`                       |
| `name`                   | No       | Yes                | Name that refers to the Okta OIDC authentication from the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                                   | `Okta`                        |
| `icon`                   | No       | Yes                | Icon used for the Okta OIDC authentication in the Grafana user interface.                                                                                                                                                                                                                                                                                                                                                                                                                                           | `okta`                        |
| `client_id`              | Yes      | Yes                | Client ID provided by your Okta OIDC app.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |                               |
| `client_secret`          | Yes      | Yes                | Client secret provided by your Okta OIDC app.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |                               |
| `auth_url`               | Yes      | Yes                | Authorization endpoint of your Okta OIDC provider.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |                               |
| `token_url`              | Yes      | Yes                | Endpoint used to obtain the Okta OIDC access token.                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |                               |
| `token_exchange_timeout` | No       | Yes                | Timeout in seconds for the OAuth token exchange request. If set to `0` or left unset, defaults to `15` seconds.                                                                                                                                                                                                                                                                                                                                                                                                     | `15`                          |
| `api_url`                | Yes      | Yes                | Endpoint used to obtain user information.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |                               |
| `scopes`                 | No       | Yes                | List of comma- or space-separated Okta OIDC scopes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `openid profile email groups` |
| `allow_sign_up`          | No       | Yes                | Controls Grafana user creation through the Okta OIDC login. Only existing Grafana users can log in with Okta OIDC if set to `false`.                                                                                                                                                                                                                                                                                                                                                                                | `true`                        |
| `auto_login`             | No       | Yes                | Set to `true` to enable users to bypass the login screen and automatically log in. This setting is ignored if you configure multiple auth providers to use auto-login.                                                                                                                                                                                                                                                                                                                                              | `false`                       |
| `role_attribute_path`    | No       | Yes                | [JMESPath](http://jmespath.org/examples.html) expression to use for Grafana role lookup. Grafana will first evaluate the expression using the Okta OIDC ID token. If no role is found, the expression will be evaluated using the user information obtained from the UserInfo endpoint. The result of the evaluation should be a valid Grafana role (`None`, `Viewer`, `Editor`, `Admin` or `GrafanaAdmin`). For more information on user role mapping, refer to [Configure role mapping](#configure-role-mapping). |                               |
| `role_attribute_strict`  | No       | Yes                | Set to `true` to deny user login if the Grafana org role cannot be extracted using `role_attribute_path` or `org_mapping`. For more information on user role mapping, refer to [Configure role mapping](#configure-role-mapping).                                                                                                                                                                                                                                                                                   | `false`                       |
| `org_attribute_path`     | No       | No                 | [JMESPath](http://jmespath.org/examples.html) expression to use for Grafana org to role lookup. The result of the evaluation will be mapped to org roles based on `org_mapping`. For more information on org to role mapping, refer to [Org roles mapping example](#org-roles-mapping-example).                                                                                                                                                                                                                     |                               |
| `org_mapping`            | No       | No                 | List of comma- or space-separated `<ExternalOrgName>:<OrgIdOrName>:<Role>` mappings. Value can be `*` meaning "All users". Role is optional and can have the following values: `None`, `Viewer`, `Editor` or `Admin`. For more information on external organization to role mapping, refer to [Org roles mapping example](#org-roles-mapping-example).                                                                                                                                                              |                               |
| `skip_org_role_sync`     | No       | Yes                | Set to `true` to stop automatically syncing user roles. This will allow you to set organization roles for your users from within Grafana manually.                                                                                                                                                                                                                                                                                                                                                                  | `false`                       |
| `allowed_groups`         | No       | Yes                | List of comma- or space-separated groups. The user should be a member of at least one group to log in.                                                                                                                                                                                                                                                                                                                                                                                                              |                               |
| `allowed_domains`        | No       | Yes                | List of comma- or space-separated domains. The user should belong to at least one domain to log in.                                                                                                                                                                                                                                                                                                                                                                                                                 |                               |
| `use_pkce`               | No       | Yes                | Set to `true` to use [Proof Key for Code Exchange (PKCE)](https://datatracker.ietf.org/doc/html/rfc7636). Grafana uses the SHA256 based `S256` challenge method and a 128 bytes (base64url encoded) code verifier.                                                                                                                                                                                                                                                                                                  | `true`                        |
| `use_refresh_token`      | No       | Yes                | Set to `true` to use refresh token and check access token expiration.                                                                                                                                                                                                                                                                                                                                                                                                                                               | `false`                       |
| `validate_id_token`      | No       | Yes                | If enabled, Grafana will validate the JWT signature of ID tokens using the JWKS endpoint. This enhances security by ensuring tokens are authentic and have not been tampered with.                                                                                                                                                                                                                                                                                                                                  | `false`                       |
| `jwk_set_url`            | No       | Yes                | URL of the JSON Web Key Set (JWKS) endpoint used to verify JWT ID token signatures. Required when ID token validation is enabled. For Okta, use `https://<tenant-id>.okta.com/oauth2/v1/keys`.                                                                                                                                                                                                                                                                                                                      |                               |
| `signout_redirect_url`   | No       | Yes                | URL to redirect to after the user logs out.                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                               |

# Configure SAML with Okta catalog application

Grafana offers multiple ways to configure the SAML authentication flow. This guide focuses on configuring the authentication flow using the Okta Integration Network (OIN) application.

The Grafana Labs application can be found in the [Okta Integration Network catalog](https://www.okta.com/integrations/).

## Prerequisites

- Grafana Enterprise or a paid Grafana Cloud account.
- Admin privileges in both Grafana and Okta.

## Supported features

- SAML Single Sign-On (SSO)
- SAML Attribute Mapping
- SAML Group Mapping
- SAML External ID Mapping for SCIM provisioning

## Configure SAML using the OIN application

### At the Okta Integration Network catalog

1. Visit the [Okta Integration Network catalog](https://www.okta.com/integrations/) and search for **Grafana Labs**.
1. Within the **Grafana Labs** application page, click on **+Add Integration**.
1. Select the tenant to add the integration to.

### At the Grafana Labs application page

1. If needed, update the **Application label**.
1. Set the domain name. For example, `your-grafana-domain.grafana.net`.
1. Click on **Done**.

| Field                 | Description                              |
| --------------------- | ---------------------------------------- |
| **Application label** | The name of the application.             |
| **Domain name**       | The domain name of the Grafana instance. |

### At the Grafana Labs Integration page

1. At the **Assignments** tab, add the groups or users that should have access to the application.
1. At the **Sign On** tab, copy the _Metadata URL_.

## Update SAML configuration at Grafana

### At the Grafana Labs SAML settings page

1. Navigate to the **SAML settings** page within the **Authentication** section from the left-hand menu.
1. The only required step is pasting the _Metadata URL_ in the **IdP Metadata URL** field, located at the **3. Connect Grafana with Identity Provider** tab.
1. Save and apply the changes.

With this configuration, the users will be able to access Grafana using their Okta credentials.

# Configure custom branding

Custom branding enables you to replace the Grafana Labs brand and logo with your corporate brand and logo.

{{< admonition type="note" >}}
Available in [Grafana Enterprise](../../../introduction/grafana-enterprise/) and Grafana Cloud paid accounts. Grafana Cloud Free plans support login title and footer link customization only. For Cloud users, click **Open a Support Ticket** from the Cloud Portal to create a ticket to configure custom branding.
{{< /admonition >}}

The `grafana.ini` file includes Grafana Enterprise custom branding. As with all configuration options, you can use environment variables to set custom branding.

With custom branding, you can modify the following elements. Hover your cursor over a number to learn more about each element.

{{< image-map key="branding-login-page" >}}

{{< image-map key="branding-home-page" >}}

You can also customize the **loading logo** (`loading_logo`), which appears during the initial page load splash screen.

> You will have to host your logo and other images used by the custom branding feature separately. Make sure Grafana can access the URL where the assets are stored.

The configuration file in Grafana Enterprise contains the following options. For more information about configuring Grafana, refer to [Configure Grafana](../).

```ini
# Enterprise only
[white_labeling]
# Set to your company name to override application title
;app_title =

# Set to main title on the login page (Will not appear if a login logo is set)
;login_title =

# Set to login subtitle (Will not appear if a login logo is set)
;login_subtitle =

# Set to complete URL to override login logo
;login_logo =

# Set to complete CSS background expression to override login background
# example: login_background = url(http://www.bhmpics.com/wallpapers/starfield-1920x1080.jpg)
;login_background =

# Set to complete CSS background expression to override login box background
;login_box_background =

# Set to complete URL to override menu logo
;menu_logo =

# Set to complete URL to override fav icon (icon shown in browser tab)
;fav_icon =

# Set to complete URL to override apple/ios icon
;apple_touch_icon =

# Set to complete URL to override loading logo
;loading_logo =

# Set to `true` to remove the Grafana edition from appearing in the footer
;hide_edition =
```

{{< admonition type="note" >}}
For the `login_logo` option, Grafana recommends using SVG files that are 48 pixels by 48 pixels or smaller. You also don't need to use the `url()` function for `login_logo`.

Additionally, you can copy images to the local Grafana image directory, `/usr/share/grafana/public/img/`, and set `login_logo` to the stored image. For example:

```ini
login_logo = /public/img/<YOUR_LOGO.svg>
```

{{< /admonition >}}

You have the option of adding custom links in place of the default footer links (Documentation, Support, Community). Below is an example of how to replace the default footer and help links with custom links.

```ini
footer_links = support guides extracustom
footer_links_support_text = Support
footer_links_support_url = http://your.support.site
footer_links_guides_text = Guides
footer_links_guides_url = http://your.guides.site
footer_links_extracustom_text = Custom text
footer_links_extracustom_url = http://your.custom.site
```

The following example shows configuring custom branding using environment variables instead of the `custom.ini` or `grafana.ini` files.

```
GF_WHITE_LABELING_FOOTER_LINKS=support guides extracustom
GF_WHITE_LABELING_FOOTER_LINKS_SUPPORT_TEXT=Support
GF_WHITE_LABELING_FOOTER_LINKS_SUPPORT_URL=http://your.support.site
GF_WHITE_LABELING_FOOTER_LINKS_GUIDES_TEXT=Guides
GF_WHITE_LABELING_FOOTER_LINKS_GUIDES_URL=http://your.guides.site
GF_WHITE_LABELING_FOOTER_LINKS_EXTRACUSTOM_TEXT=Custom Text
GF_WHITE_LABELING_FOOTER_LINKS_EXTRACUSTOM_URL=http://your.custom.site
```

{{< admonition type="note" >}}
The following two links are always present in the footer:
{{< /admonition >}}

- Grafana edition
- Grafana version with build number

If you specify `footer_links` or `GF_WHITE_LABELING_FOOTER_LINKS`, then all other default links are removed from the footer, and only what is specified is included.

## Custom branding for shared dashboards

In addition to the customizations described below, you can customize the footer of your shared dashboards.
To customize the footer of a shared dashboard, add the following section to the `grafana.ini` file.

```ini
[white_labeling.public_dashboards]

# Hides the footer for the shared dashboards if set to `true`.
# example: footer_hide = "true"
;footer_hide =

# Set to text shown in the footer
;footer_text =

# Set to complete url to override shared dashboard footer logo. Default is `grafana-logo` and will display the Grafana logo.
# An empty value will hide the footer logo.
;footer_logo =

# Set to link for the footer
;footer_link =

# Set to `true` to hide the Grafana logo next to the title
;header_logo_hide =
```

If you specify `footer_hide` to `true`, all the other values are ignored because the footer will not be shown.

# Configure profiling and tracing to troubleshoot Grafana

You can set up the `grafana-server` process to enable certain diagnostics when it starts. This can be useful
when investigating certain performance problems. It's _not_ recommended to have these enabled by default.

## Turn on profiling and collect profiles

The `grafana-server` can be started with the command-line option `-profile` to enable profiling, `-profile-addr` to override the default HTTP address (`localhost`), and
`-profile-port` to override the default HTTP port (`6060`) where the `pprof` debugging endpoints are available. Further, [`-profile-block-rate`](https://pkg.go.dev/runtime#SetBlockProfileRate) controls the fraction of goroutine blocking events that are reported in the blocking profile, default `1` (i.e. track every event) for backward compatibility reasons, and [`-profile-mutex-rate`](https://pkg.go.dev/runtime#SetMutexProfileFraction) controls the fraction of mutex contention events that are reported in the mutex profile, default `0` (i.e. track no events). The higher the fraction (that is, the smaller this value) the more overhead it adds to normal operations.

Running Grafana with profiling enabled and without block and mutex profiling enabled should only add a fraction of overhead and is suitable for [continuous profiling](https://grafana.com/oss/pyroscope/). Adding a small fraction of block and mutex profiling, such as 10-5 (10%-20%) should in general be fine.

Enable profiling:

```bash
./grafana server -profile -profile-addr=0.0.0.0 -profile-port=8080
```

Enable profiling with block and mutex profiling enabled with a fraction of 20%:

```bash
./grafana server -profile -profile-addr=0.0.0.0 -profile-port=8080 -profile-block-rate=5 -profile-mutex-rate=5
```

Note that `pprof` debugging endpoints are served on a different port than the Grafana HTTP server. Check what debugging endpoints are available by browsing `http://<profile-addr><profile-port>/debug/pprof`.

There are some additional [godeltaprof](https://github.com/grafana/pyroscope-go/tree/main/godeltaprof) endpoints available which are more suitable in a continuous profiling scenario. These endpoints are `/debug/pprof/delta_heap`, `/debug/pprof/delta_block`, `/debug/pprof/delta_mutex`.

You can configure or override profiling settings using environment variables:

```bash
export GF_DIAGNOSTICS_PROFILING_ENABLED=true
export GF_DIAGNOSTICS_PROFILING_ADDR=0.0.0.0
export GF_DIAGNOSTICS_PROFILING_PORT=8080
export GF_DIAGNOSTICS_PROFILING_BLOCK_RATE=5
export GF_DIAGNOSTICS_PROFILING_MUTEX_RATE=5
```

In general, you use the [Go command pprof](https://golang.org/cmd/pprof/) to both collect and analyze profiling data. You can also use [curl](https://curl.se/) or similar to collect profiles which could be convenient in environments where you don't have the Go/pprof command available. Next, some usage examples of using curl and pprof to collect and analyze memory and CPU profiles.

**Analyzing high memory usage/memory leaks:**

When experiencing high memory usage or potential memory leaks it's useful to collect several heap profiles and later when analyzing, compare them. It's a good idea to wait some time, e.g. 30 seconds, between collecting each profile to allow memory consumption to increase.

```bash
curl http://<profile-addr>:<profile-port>/debug/pprof/heap > heap1.pprof
sleep 30
curl http://<profile-addr>:<profile-port>/debug/pprof/heap > heap2.pprof
```

You can then use pprof tool to compare two heap profiles:

```bash
go tool pprof -http=localhost:8081 --base heap1.pprof heap2.pprof
```

**Analyzing high CPU usage:**

When experiencing high CPU usage it's suggested to collect CPU profiles over a period of time, e.g. 30 seconds.

```bash
curl 'http://<profile-addr>:<profile-port>/debug/pprof/profile?seconds=30' > profile.pprof
```

You can then use pprof tool to analyze the collected CPU profile:

```bash
go tool pprof -http=localhost:8081 profile.pprof
```

## Use tracing

The `grafana-server` can be started with the arguments `-tracing` to enable tracing and `-tracing-file` to override the default trace file (`trace.out`) where trace result is written to. For example:

```bash
./grafana server -tracing -tracing-file=/tmp/trace.out
```

You can configure or override profiling settings using environment variables:

```bash
export GF_DIAGNOSTICS_TRACING_ENABLED=true
export GF_DIAGNOSTICS_TRACING_FILE=/tmp/trace.out
```

View the trace in a web browser (Go required to be installed):

```bash
go tool trace <trace file>
2019/11/24 22:20:42 Parsing trace...
2019/11/24 22:20:42 Splitting trace...
2019/11/24 22:20:42 Opening browser. Trace viewer is listening on http://127.0.0.1:39735
```

For more information about how to analyze trace files, refer to [Go command trace](https://golang.org/cmd/trace/).

## Export traces to a file

Grafana can write its own distributed traces to a local file in OpenTelemetry Protocol (OTLP) JSON format, without a collector or a tracing backend. This helps you get traces for debugging without setting up the backend: turn on the exporter, reproduce the issue, then collect and share the file.

{{< admonition type="note" >}}
Don't use this exporter as a permanent configuration. Use it to troubleshoot issues in Grafana, then turn it off.
{{< /admonition >}}

This is different from the `-tracing` option in the previous section. That option writes a low-level Go runtime execution trace for `go tool trace`, whereas the file exporter emits the same OpenTelemetry spans that Grafana sends to Jaeger or OTLP endpoints, which describe how requests flow through Grafana.

To turn it on, set a `path` in the `[tracing.opentelemetry.file]` section. Grafana uses this exporter only when you haven't configured a `tracing.opentelemetry.jaeger` or `tracing.opentelemetry.otlp` endpoint. Grafana limits each capture by file size and duration to protect disk space.

```ini
[tracing.opentelemetry.file]
path = /var/lib/grafana/traces/capture.json
```

For the available options, refer to [`[tracing.opentelemetry.file]`](../#tracingopentelemetryfile) in the configuration reference.

To view the captured file, open Grafana, go to **Explore**, select a Tempo data source, and select **Import trace**.

# Configure Grafana Enterprise

This page describes Grafana Enterprise-specific configuration options that you can specify in a `.ini` configuration file or using environment variables. Refer to [Configuration](../) for more information about available configuration options.

## [enterprise]

### license_path

Local filesystem path to Grafana Enterprise's license file.
Defaults to `<paths.data>/license.jwt`.

### license_text

When set to the text representation (i.e. content of the license file)
of the license, Grafana will evaluate and apply the given license to
the instance.

### auto_refresh_license

When enabled, Grafana will send the license and usage statistics to
the license issuer. If the license has been updated on the issuer's
side to be valid for a different number of users or a new duration,
your Grafana instance will be updated with the new terms
automatically. Defaults to `true`.

{{< admonition type="note" >}}
The license only automatically updates once per day. To immediately update the terms for a license, use the Grafana UI to renew your license token.
{{< /admonition >}}

### license_validation_type

When set to `aws`, Grafana will validate its license status with Amazon Web Services (AWS) instead of with Grafana Labs. Only use this setting if you purchased an Enterprise license from AWS Marketplace. Defaults to empty, which means that by default Grafana Enterprise will validate using a license issued by Grafana Labs. For details about licenses issued by AWS, refer to [Activate a Grafana Enterprise license purchased through AWS Marketplace](../../../administration/enterprise-licensing/activate-aws-marketplace-license/).

## [white_labeling]

### app_title

Set to your company name to override application title.

### login_logo

Set to complete URL to override login logo.

### login_background

Set to complete CSS background expression to override login background. Example:

```bash
[white_labeling]
login_background = url(http://www.bhmpics.com/wallpapers/starfield-1920x1080.jpg)
```

### menu_logo

Set to complete URL to override menu logo.

### fav_icon

Set to complete URL to override fav icon (icon shown in browser tab).

### apple_touch_icon

Set to complete URL to override Apple/iOS icon.

### hide_edition

Set to `true` to remove the Grafana edition from appearing in the footer.

### footer_links

List the link IDs to use here. Grafana will look for matching link configurations, the link IDs should be space-separated and contain no whitespace.

## [usage_insights.export]

By [exporting usage logs](../../configure-security/export-logs/), you can directly query them and create dashboards of the information that matters to you most, such as dashboard errors, most active organizations, or your top-10 most-used queries.

### enabled

Enable the usage insights export feature.

### storage

Specify a storage type. Defaults to `loki`.

## [usage_insights.export.storage.loki]

### type

Set the communication protocol to use with Loki, which is either `grpc` or `http`. Defaults to `grpc`.

### url

Set the address for writing logs to Loki (format must be host:port).

### tls

Decide whether or not to enable the TLS (Transport Layer Security) protocol when establishing the connection to Loki. Defaults to true.

### tenant_id

Set the tenant ID for Loki communication, which is disabled by default. The tenant ID is required to interact with Loki running in [multi-tenant mode](/docs/loki/latest/operations/multi-tenancy/).

### batch_wait_duration

How long to wait before sending a request to Loki with the batch of events. Uses duration format: e.g. 5s, 1m. Defaults to 5s.

Whatever happens first between `batch_wait_duration` and `batch_size_bytes` will trigger the batch to be sent to Loki.

If the wait duration is very long and the `batch_size_bytes` is very high, events may take a long time to be sent.

### batch_size_bytes

How many events (in bytes) to accumulate in a single batch before sending it to Loki. Defaults to 100 KiB.

Whatever happens first between `batch_wait_duration` and `batch_size_bytes` will trigger the batch to be sent to Loki.

If you wish to always wait for the `batch_wait_duration`, set this to a very high number.

## [analytics.summaries]

### buffer_write_interval

Interval for writing dashboard usage stats buffer to database.

### buffer_write_timeout

Timeout for writing dashboard usage stats buffer to database.

### rollup_interval

Interval for trying to roll up per dashboard usage summary. Only rolled up at most once per day.

### rollup_timeout

Timeout for trying to rollup per dashboard usage summary.

## [analytics.views]

### recent_users_age

Age for recent active users.

## [reporting]

### enabled

Enable or disable the reporting feature. When disabled, no reports are generated, and the UI is hidden. By default, reporting is enabled (`true`).

### rendering_timeout

Timeout for the following reporting rendering requests: generating PDFs, generating embedded dashboard images for report emails, and generating attached CSV files. Default is 30 seconds (`30s`).

### concurrent_render_limit

Maximum number of concurrent calls to the rendering service. Default is `4`.

### image_scale_factor

Scale factor for rendering images. Value `2` is enough for monitor resolutions, `4` would be better for printed material. Setting a higher value affects performance and memory. Default is `2`.

### max_attachment_size_mb

Set the maximum file size in megabytes for the report email attachments. Default is `10`.

### fonts_path

Path to the directory containing font files.

### font_regular

Name of the TrueType font file with regular style. Default is `DejaVuSansCondensed.ttf`.

### font_bold

Name of the TrueType font file with bold style.

### font_italic

Name of the TrueType font file with italic style. Default is `DejaVuSansCondensed-Oblique.ttf`.

### font_min_text_size

The minimum pixel size that Grafana uses when rendering fonts. Default is `4`.

### max_request_retries

Maximum number of times the following reporting rendering requests are retried before returning an error: generating PDFs, generating embedded dashboard images for report emails, and generating attached CSV files. Default is `0`, which means it is disabled.

### allowed_domains

Allowed domains to receive reports. Use an asterisk (`*`) to allow all domains. Use a comma-separated list to allow multiple domains. Example: `allowed_domains = grafana.com`, example.org. Default is `*`.

## [auditing]

[Auditing](../../configure-security/audit-grafana/) allows you to track important changes to your Grafana instance. By default, audit logs are logged to file but the auditing feature also supports sending logs directly to Loki.

### enabled

Enable the auditing feature. Defaults to false.

### loggers

List of enabled loggers.

### log_dashboard_content

Keep dashboard content in the logs (request or response fields). This can significantly increase the size of your logs.

### log_datasource_query_request_body

Whether to record data source queries' request body. This can significantly increase the size of your logs. Disabled by default.

### log_datasource_query_response_body

Whether to record data source queries' response body. This can significantly increase the size of your logs. Disabled by default.

### verbose

Log all requests and keep requests and responses body. This can significantly increase the size of your logs.

### log_all_status_codes

Set to false to only log requests with 2xx, 3xx, 401, 403, 500 responses.

### max_response_size_bytes

Maximum response body (in bytes) to be recorded. May help reducing the memory footprint caused by auditing.

### `snapshot_timeout`

Grafana uses `snapshot_timeout` as a cooperative deadline when resolving resource names before serving a request that may delete resources. Grafana currently uses this setting for `DELETE /api/admin/users/{id}`. Because the backing store must honor context cancellation, the value isn't a strict upper bound on request latency. Use a duration such as `500ms` or `2s`. If you configure zero or a negative duration, Grafana logs a warning and uses the default of `500ms`.

## [auditing.logs.file]

### path

Path to logs folder.

### max_files

Maximum log files to keep.

### max_file_size_mb

Max size in megabytes per log file.

## [auditing.logs.loki]

### url

Set the URL for writing logs to Loki.

### tls

If true, it establishes a secure connection to Loki. Defaults to true.

### tenant_id

Set the tenant ID for Loki communication, which is disabled by default. The tenant ID is required to interact with Loki running in [multi-tenant mode](/docs/loki/latest/operations/multi-tenancy/).

### retries

The amount of times the HTTP or gRPC client will retry a failed request to Loki. The default is `10`.

### timeout

The timeout duration of an HTTP request or gRPC call to Loki. The default is `3s`.

### batch_wait_duration

Only available for the HTTP client. Disabled by default.
How long to wait before sending a request to Loki with the batch of events. Uses duration format: e.g. 5s, 1m
Whichever threshold is hit first between `batch_wait_duration` and `batch_size_bytes` triggers the batch to be sent to Loki.
If the wait duration is very long and the `batch_size_bytes` is very high, events might take a long time to be sent.
To enable batching, you must also set the `batch_size_bytes` configuration option.

### batch_size_bytes

Only available for the HTTP client. Disabled by default.
How many events, in bytes, to accumulate in a single batch before sending it to Loki.
Whichever threshold is hit first between `batch_wait_duration` and `batch_size_bytes` triggers the batch to be sent to Loki.
If you want to always wait for the `batch_wait_duration`, set `batch_size_bytes` to a very high number.
To enable batching, you must also set the `batch_wait_duration` configuration option.

## [auth.saml]

### enabled

If true, the feature is enabled. Defaults to false.

### allow_sign_up

If true, allow new Grafana users to be created through SAML logins. Defaults to true.

### certificate

Base64-encoded public X.509 certificate. Used to sign requests to the IdP.

### certificate_path

Path to the public X.509 certificate. Used to sign requests to the IdP.

### private_key

Base64-encoded private key. Used to decrypt assertions from the IdP.

### private_key_path

Path to the private key. Used to decrypt assertions from the IdP.

### idp_metadata

Base64-encoded IdP SAML metadata XML. Used to verify and obtain binding locations from the IdP.

### idp_metadata_path

Path to the SAML metadata XML. Used to verify and obtain binding locations from the IdP.

### idp_metadata_url

URL to fetch SAML IdP metadata. Used to verify and obtain binding locations from the IdP.

### max_issue_delay

Time since the IdP issued a response and the SP is allowed to process it. Defaults to 90 seconds.

### metadata_valid_duration

How long the SPs metadata is valid. Defaults to 48 hours.

### assertion_attribute_name

Friendly name or name of the attribute within the SAML assertion to use as the user name. Alternatively, this can be a template with variables that match the names of attributes within the SAML assertion.

### assertion_attribute_login

Friendly name or name of the attribute within the SAML assertion to use as the user login handle. Defaults to `login`.

### assertion_attribute_email

Friendly name or name of the attribute within the SAML assertion to use as the user email. Defaults to `email`.

### assertion_attribute_groups

Friendly name or name of the attribute within the SAML assertion to use as the user groups.

### assertion_attribute_role

Friendly name or name of the attribute within the SAML assertion to use as the user roles.

### assertion_attribute_org

Friendly name or name of the attribute within the SAML assertion to use as the user organization.

### assertion_attribute_external_uid

Friendly name or name of the attribute within the SAML assertion to use as the user external UID. Defaults to `userUID`.

### allowed_organizations

List of comma- or space-separated organizations. Each user must be a member of at least one organization to log in.

### org_mapping

List of comma- or space-separated Organization:OrgId:Role mappings. Organization can be `*` meaning "All users". Role is optional and can have the following values: `Admin`, `Editor` ,`Viewer` or `None`.

### role_values_none

List of comma- or space-separated roles that will be mapped to the None role.

### role_values_viewer

List of comma- or space-separated roles that will be mapped to the Viewer role.

### role_values_editor

List of comma- or space-separated roles that will be mapped to the Editor role.

### role_values_admin

List of comma- or space-separated roles that will be mapped to the Admin role.

### role_values_grafana_admin

List of comma- or space-separated roles that will be mapped to the Grafana Admin (Super Admin) role.

## [keystore.vault]

### url

Location of the Vault server.

### namespace

Vault namespace if using Vault with multi-tenancy.

### auth_method

Method for authenticating towards Vault. Vault is inactive if this option is not set. Current possible values: `token`.

### token

Secret token to connect to Vault when auth_method is `token`.

### lease_renewal_interval

Time between checking if there are any secrets which needs to be renewed.

### lease_renewal_expires_within

Time until expiration for tokens which are renewed. Should have a value higher than lease_renewal_interval.

### lease_renewal_increment

New duration for renewed tokens. Vault may be configured to ignore this value and impose a stricter limit.

## [security.egress]

Security egress makes it possible to control outgoing traffic from the Grafana server.

### host_deny_list

A list of hostnames or IP addresses separated by spaces for which requests are blocked.

### host_allow_list

A list of hostnames or IP addresses separated by spaces for which requests are allowed. All other requests are blocked.

### header_drop_list

A list of headers that are stripped from the outgoing data source and alerting requests.

### cookie_drop_list

A list of cookies that are stripped from the outgoing data source and alerting requests.

## [security.encryption]

### algorithm

Encryption algorithm used to encrypt secrets stored in the database and cookies. Possible values are `aes-cfb` (default) and `aes-gcm`. AES-CFB stands for _Advanced Encryption Standard_ in _cipher feedback_ mode, and AES-GCM stands for _Advanced Encryption Standard_ in _Galois/Counter Mode_.

## [caching]

When query caching is enabled, Grafana can temporarily store the results of data source queries and serve cached responses to similar requests.

### backend

The caching backend to use when storing cached queries. Options: `memory`, `redis`, and `memcached`.

The default is `memory`.

### enabled

Setting 'enabled' to `true` allows users to configure query caching for data sources.

This value is `true` by default.

{{< admonition type="note" >}}
This setting enables the caching feature, but it does not turn on query caching for any data source. To turn on query caching for a data source, update the setting on the data source configuration page. For more information, refer to the [query caching docs](../../../administration/data-source-management/#enable-and-configure-query-caching).
{{< /admonition >}}

### ttl

_Time to live_ (TTL) is the time that a query result is stored in the caching system before it is deleted or refreshed. This setting defines the time to live for query caching, when TTL is not configured in data source settings. The default value is `1m` (1 minute).

### max_ttl

The max duration that a query result is stored in the caching system before it is deleted or refreshed. This value will override `ttl` config option or data source setting if the `ttl` value is greater than `max_ttl`. To disable this constraint, set this value to `0s`.

The default is `0s` (disabled).

{{< admonition type="note" >}}
Disabling this constraint is not recommended in production environments.
{{< /admonition >}}

### max_value_mb

This value limits the size of a single cache value. If a cache value (or query result) exceeds this size, then it is not cached. To disable this limit, set this value to `0`.

The default is `1`.

### connection_timeout

This setting defines the duration to wait for a connection to the caching backend.

The default is `5s`.

### read_timeout

This setting defines the duration to wait for the caching backend to return a cached result. To disable this timeout, set this value to `0s`.

The default is `0s` (disabled).

{{< admonition type="note" >}}
Disabling this timeout is not recommended in production environments.
{{< /admonition >}}

### write_timeout

This setting defines the number of seconds to wait for the caching backend to store a result. To disable this timeout, set this value to `0s`.

The default is `0s` (disabled).

{{< admonition type="note" >}}
Disabling this timeout is not recommended in production environments.
{{< /admonition >}}

## [caching.encryption]

### enabled

When 'enabled' is `true`, query values in the cache are encrypted.

The default is `false`.

### encryption_key

A string used to generate a key for encrypting the cache. For the encrypted cache data to persist between Grafana restarts, you must specify this key. If it is empty when encryption is enabled, then the key is automatically generated on startup, and the cache clears upon restarts.

The default is `""`.

## [caching.memory]

### gc_interval

When storing cache data in-memory, this setting defines how often a background process cleans up stale data from the in-memory cache. More frequent "garbage collection" can keep memory usage from climbing but will increase CPU usage.

The default is `1m`.

### max_size_mb

The maximum size of the in-memory cache in megabytes. Once this size is reached, new cache items are rejected. For more flexible control over cache eviction policies and size, use the Redis or Memcached backend.

To disable the maximum, set this value to `0`.

The default is `25`.

{{< admonition type="note" >}}
Disabling the maximum is not recommended in production environments.
{{< /admonition >}}

## [caching.redis]

Use these settings to configure Redis as the query caching backend.

{{< admonition type="note" >}}
Redis Sentinel isn't supported for query caching. Use a standalone Redis instance with `url`, or a Redis Cluster with `cluster`.
{{< /admonition >}}

### url

The full Redis URL of your Redis server. For example: `redis://username:password@localhost:6379`. To enable TLS, use the `rediss` scheme.

The default is `"redis://localhost:6379"`.

### cluster

A comma-separated list of Redis cluster members, either in `host:port` format or using the full Redis URLs (`redis://username:password@localhost:6379`). For example, `localhost:7000, localhost: 7001, localhost:7002`.
If you use the full Redis URLs, then you can specify the scheme, username, and password only once. For example, `redis://username:password@localhost:0000,localhost:1111,localhost:2222`. You cannot specify a different username and password for each URL.

{{< admonition type="note" >}}
If you have specify `cluster`, the value for `url` is ignored.
{{< /admonition >}}

{{< admonition type="note" >}}
You can enable TLS for cluster mode using the `rediss` scheme in Grafana Enterprise v8.5 and later versions.
{{< /admonition >}}

### prefix

A string that prefixes all Redis keys. This value must be set if using a shared database in Redis. If `prefix` is empty, then one will not be used.

The default is `"grafana"`.

## [caching.memcached]

### servers

A space-separated list of memcached servers. Example: `memcached-server-1:11211 memcached-server-2:11212 memcached-server-3:11211`. Or if there's only one server: `memcached-server:11211`.

The default is `"localhost:11211"`.

### tls_enabled

Enables TLS authentication for memcached. Defaults to `false`.

### tls_cert_path

Path to the client certificate, which will be used for authenticating with the server. Also requires the key path to be configured.

### tls_key_path

Path to the key for the client certificate. Also requires the client certificate to be configured.

### tls_ca_path

Path to the CA certificates to validate the server certificate against. If not set, the host's root CA certificates are used.

### tls_server_name

Override the expected name on the server certificate.

### connection_timeout

Timeout for the memcached client to connect to memcached. Defaults to `0`, which uses the memcached client default timeout per connection scheme.

## [recorded_queries]

### enabled

Whether the recorded queries feature is enabled

### min_interval

Sets the minimum interval to enforce between query evaluations. The default value is `10s`. Query evaluation will be
adjusted if they are less than this value. Higher values can help with resource management.

The interval string is a possibly signed sequence of decimal numbers, followed by a unit suffix (ms, s, m, h, d), e.g.
30s or 1m.

### max_queries

The maximum number of recorded queries that can exist.

### default_remote_write_datasource_uid

The UID of the datasource where the query data will be written.

If all `default_remote_write_*` properties are set, this information will be populated at startup. If a remote write target has
already been configured, nothing will happen.

### default_remote_write_path

The api path where metrics will be written

If all `default_remote_write_*` properties are set, this information will be populated at startup. If a remote write target has
already been configured, nothing will happen.

### default_remote_write_datasource_org_id

The org id of the datasource where the query data will be written.

If all `default_remote_write_*` properties are set, this information will be populated at startup. If a remote write target has
already been configured, nothing will happen.

# Configure feature toggles

<!-- DO NOT EDIT THIS PAGE, it is machine generated by running the test in -->
<!-- https://github.com/grafana/grafana/blob/main/pkg/services/featuremgmt/toggles_gen_test.go#L27 -->

# Configure feature toggles

You use feature toggles, also known as feature flags, to enable or disable features in Grafana. You can turn on feature toggles to try out new functionality in development or test environments.

This page contains a list of available feature toggles. To learn how to turn on feature toggles, refer to our [Configure Grafana documentation](../#feature_toggles). Feature toggles are also available to Grafana Cloud Advanced customers. If you use Grafana Cloud Advanced, you can open a support ticket and specify the feature toggles and stack for which you want them enabled.

Experimental feature toggles are listed in the respective feature's documentation only.

For more information about feature release stages, refer to [Release life cycle for Grafana Labs](https://grafana.com/docs/release-life-cycle/) and [Manage feature toggles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/feature-toggles/#manage-feature-toggles).

## General availability feature toggles

Most [generally available](https://grafana.com/docs/release-life-cycle/#general-availability) features are enabled by default. You can disable these feature by setting the feature flag to "false" in the configuration.

| Feature toggle name                          | Description                                                                                                                         | Enabled by default |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `cloudWatchCrossAccountQuerying`             | Enables cross-account querying in CloudWatch datasources                                                                            | Yes                |
| `lokiQuerySplitting`                         | Split large interval queries into subqueries with smaller time intervals                                                            | Yes                |
| `renderAuthJWT`                              | Uses JWT-based auth for rendering instead of relying on remote cache                                                                | Yes                |
| `awsDatasourcesTempCredentials`              | Support temporary security credentials in AWS plugins for Grafana Cloud customers                                                   | Yes                |
| `provisioningFolderMetadata`                 | Allow setting folder metadata for provisioned folders                                                                               | Yes                |
| `awsAsyncQueryCaching`                       | Enable caching for async queries for Redshift and Athena. Requires that the data source has caching and async query support enabled | Yes                |
| `reportingHeaderSettings`                    | Enables configuration of PDF report settings                                                                                        |                    |
| `reportingFooterSettings`                    | Enables the configurable footer settings for PDF reports                                                                            |                    |
| `grafana.newPanelQueryErrorsUI`              | Enables a new UI for query errors and notices                                                                                       | Yes                |
| `useKubernetesShortURLsAPI`                  | Routes short URL requests from /api to the /apis endpoint in the frontend. Depends on kubernetesShortURLs                           | Yes                |
| `cloudWatchBatchQueries`                     | Runs CloudWatch metrics queries as separate batches                                                                                 |                    |
| `annotationPermissionUpdate`                 | Change the way annotation permissions work by scoping them to folders and dashboards.                                               | Yes                |
| `dashboardNewLayouts`                        | Enables new dashboard layouts                                                                                                       | Yes                |
| `timeComparison`                             | Enables time comparison option in supported panels                                                                                  | Yes                |
| `alertingQueryOptimization`                  | Optimizes eligible queries in order to reduce load on datasources                                                                   |                    |
| `sqlExpressions`                             | Enables SQL Expressions, which can execute SQL queries against data source results.                                                 | Yes                |
| `grafana.filterablePanels`                   | Enables interactive grouped-label filtering through the tooltip in state timeline, status history and histogram panels              | Yes                |
| `cloudWatchNewLabelParsing`                  | Updates CloudWatch label parsing to be more accurate                                                                                | Yes                |
| `playlistsRBAC`                              | Enables RBAC for playlists                                                                                                          |                    |
| `savedQueriesRBAC`                           | Enables Saved queries (query library) RBAC permissions                                                                              | Yes                |
| `newSavedQueriesExperience`                  | Enables the new Saved queries (query library) modal experience                                                                      | Yes                |
| `alertingListViewV2`                         | Enables the new alert list view design                                                                                              | Yes                |
| `alertingNavigationV2`                       | Enables the new Alerting navigation structure with improved menu grouping                                                           | Yes                |
| `pluginProxyPreserveTrailingSlash`           | Preserve plugin proxy trailing slash.                                                                                               |                    |
| `azureMonitorPrometheusExemplars`            | Allows configuration of Azure Monitor as a data source that can provide Prometheus exemplars                                        | Yes                |
| `cloudWatchRoundUpEndTime`                   | Round up end time for metric queries to the next minute to avoid missing data                                                       | Yes                |
| `useSessionStorageForRedirection`            | Use session storage for handling the redirection after login                                                                        | Yes                |
| `azureMonitorDisableLogLimit`                | Disables the log limit restriction for Azure Monitor when true. The limit is enabled by default.                                    |                    |
| `enableSCIM`                                 | Enables SCIM support for user and group management                                                                                  | Yes                |
| `azureMonitorEnableUserAuth`                 | Enables user auth for Azure Monitor datasource only                                                                                 | Yes                |
| `lokiLabelNamesQueryApi`                     | Defaults to using the Loki `/labels` API instead of `/series`                                                                       | Yes                |
| `unifiedNavbars`                             | Enables unified navbars                                                                                                             |                    |
| `grafanaAssistantInProfilesDrilldown`        | Enables integration with Grafana Assistant in Profiles Drilldown                                                                    | Yes                |
| `otelLogsFormatting`                         | Applies OTel formatting templates to displayed logs                                                                                 | Yes                |
| `alertingNotificationHistory`                | Enables the notification history feature                                                                                            | Yes                |
| `newClickhouseConfigPageDesign`              | Enables new design for the Clickhouse data source configuration page                                                                | Yes                |
| `azureResourcePickerUpdates`                 | Enables the updated Azure Monitor resource picker                                                                                   | Yes                |
| `panelTimeSettings`                          | Enables a new panel time settings drawer                                                                                            | Yes                |
| `inlineLogDetailsNoScrolls`                  | Enables an inline version of Log Details that creates no new scrolls                                                                | Yes                |
| `lokiAlignedQuerySplitting`                  | Aligns query splitting chunks with UTC midnight                                                                                     | Yes                |
| `profilesHeatmap`                            | Enables heatmap visualization support for Pyroscope profiles                                                                        | Yes                |
| `datasources.useNewStackInfoToSettingsCache` | Use the new cache for datasource.StackInfoToSettings, backend flag                                                                  |                    |
| `grafana.queryVarEditorRedesign`             | Enables a redesigned query variable editor with split-pane preview and a spreadsheet for managing static options                    | Yes                |
| `grafana.dashboardSettingsRedesign`          | Redesigns dashboard settings page into Advanced Settings in a modal window                                                          | Yes                |
| `grafana.thresholdsInterpolation`            | Enables using dashboard variables in panel threshold values                                                                         | Yes                |
| `grafana.dashboardAutoGridDefault`           | Uses auto grid as the default layout for new dashboards                                                                             | Yes                |

## Public preview feature toggles

[Public preview](https://grafana.com/docs/release-life-cycle/#public-preview) features are supported by our Support teams, but might be limited to enablement, configuration, and some troubleshooting.

| Feature toggle name                | Description                                                                                                                    |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `panelTitleSearch`                 | Search for dashboards using panel title                                                                                        |
| `faroDatasourceSelector`           | Enable the data source selector within the Frontend Apps section of the Frontend Observability                                 |
| `externalServiceAccounts`          | Automatic service account and token setup for plugins                                                                          |
| `grafana.mobileEditDeprecation`    | Disables dashboard edit mode (manual and assistant-driven) on mobile/small-screen viewports                                    |
| `dashboard.notebooks`              | Enable notebooks, a resource in the dashboard API group for mixing text cells, code cells, and visualization panels            |
| `feedbackButton`                   | Enables the feedback button in the dashboard edit sidebar                                                                      |
| `pdfTables`                        | Enables generating table data as PDF in reporting                                                                              |
| `canvasPanelPanZoom`               | Allow pan and zoom in canvas panel                                                                                             |
| `secretsManagementAppPlatformUI`   | Enable the secrets management app platform UI                                                                                  |
| `secretsKeeperUI`                  | Enable the Secrets Keeper management UI for configuring external secret storage                                                |
| `grafana.secretsReferenceValueUI`  | Enable referencing an existing secret in an active keeper when creating a secure value                                         |
| `dashboardTemplates`               | Enables a flow to get started with a new dashboard from a template                                                             |
| `alertRuleRestore`                 | Enables the alert rule restore feature                                                                                         |
| `alertingMigrationWizardUI`        | Enables the migration wizard UI to migrate alert rules and notification resources from external sources to Grafana Alerting    |
| `azureMonitorLogsBuilderEditor`    | Enables the logs builder mode for the Azure Monitor data source                                                                |
| `datasources.azureMonitorBatchAPI` | Enables the Metrics Batch API for the Azure Monitor data source, allowing up to 50 resources to be queried in a single request |
| `alertingListViewV2PreviewToggle`  | Enables the alerting list view v2 preview toggle                                                                               |
| `alertingImportAlertmanagerAPI`    | Enables the API to import Alertmanager configuration                                                                           |
| `grafana.newTextPanel`             | Enables the new text panel                                                                                                     |
| `text.newFeatures`                 | Enables the new features in text panel                                                                                         |
| `interactiveLearning`              | Enables the interactive learning app                                                                                           |
| `transformationsEmptyPlaceholder`  | Show transformation quick-start cards in empty transformations state                                                           |
| `grafana.dashboardGlobalVariables` | Enables global and folder-scoped dashboard variables via dashboard.grafana.app                                                 |
| `pyroscopeUTF8LabelNames`          | Enables support for UTF-8 label names in Pyroscope label selectors                                                             |
| `queryEditorNext`                  | Enables next generation query editor experience                                                                                |
| `grafana.viewPanelPane`            | Enables the sidebar pane with new toggles and options in panel view mode                                                       |
| `splashScreen`                     | Enables the splash screen modal for introducing new Grafana features on first session                                          |
| `dataviz.tabularNums`              | Enables tabular numerals for visualization legend values                                                                       |
| `grafana.dynamicTraceToLogs`       | Check for the existence of logs when linking from the Trace View                                                               |

## Development feature toggles

The following toggles require explicitly setting Grafana's [app mode](../#app_mode) to 'development' before you can enable this feature toggle. These features tend to be experimental.

| Feature toggle name                    | Description                                                                   |
| -------------------------------------- | ----------------------------------------------------------------------------- |
| `grafanaAPIServerWithExperimentalAPIs` | Register experimental APIs with the k8s API server, including all datasources |

# Configure a data source SOCKS5 connection proxy

Grafana provides support for proxying data source connections through a secure SOCKS5 proxy. This enables you to securely connect to data sources hosted in a different network than Grafana.

To make use of this functionality, you need to deploy a SOCKS5 proxy server that supports TLS on a machine accessible from where your Grafana instance is running, and within the same network as your data source. From there, Grafana establishes a mutually trusted connection from Grafana to the proxy. Then the SOCKS5 proxy can proxy the Grafana data source connection to your private data source instance without exposing your data source to the public internet.

## Known limitations

- You can configure only one SOCKS5 proxy per Grafana instance. This is because the SOCKS5 proxy address and TLS configuration is set at the Grafana instance level.
- All built-in core data sources are compatible, but not all external data sources are. For a list of supported data sources, refer to [private data source connect](/docs/grafana-cloud/data-configuration/configure-private-datasource-connect/#known-limitations).

## Before you begin

To complete this task, you must first have a SOCKS5 proxy server running, using host certificates signed by a Certificate Authority (CA) that you can get the public certificate for. You must also have a TLS client key pair for Grafana, with the client certificate signed by the same CA used to sign the proxy server certificate.

## Steps

1. For Grafana to send data source connections to the socks5 server, use the following table to configure the `secure_socks_datasource_proxy` section of the `config.ini`:

   | Key              | Description                                                                 | Example                         |
   | ---------------- | --------------------------------------------------------------------------- | ------------------------------- |
   | `enabled`        | Enable this feature in Grafana                                              | true                            |
   | `root_ca_cert`   | The file path of the root ca cert used to sign the proxy server certificate | /etc/ca.crt                     |
   | `client_key`     | The file path of the client private key                                     | /etc/client.key                 |
   | `client_cert`    | The file path of the client public key                                      | /etc/client.crt                 |
   | `server_name`    | The domain name of the proxy, used for SNI                                  | proxy.grafana.svc.cluster.local |
   | `proxy_address`  | The address of the proxy                                                    | localhost:9090                  |
   | `allow_insecure` | Disable TLS in the socks proxy                                              | false                           |

1. Set up a data source and configure it to send data source connections through the proxy.

   To configure your data sources to send connections through the proxy, `enableSecureSocksProxy=true` must be specified in the data source json. You can do this in the [API](../../../developers/http_api/data_source/) or use [file based provisioning](../../../administration/provisioning/#data-sources).

   Additionally, if using SOCKS5 authentication, you can set the SOCKS5 username and password by adding `secureSocksProxyUsername` in the data source's `jsonData` field and `secureSocksProxyPassword` in the data source's `secureJsonData` field.

# Settings updates at runtime

{{< admonition type="note" >}}
This functionality is deprecated and will be removed in a future release. For configuring SAML authentication, please use the new [SSO settings API](../../../developers/http_api/sso-settings/).
{{< /admonition >}}

By updating settings at runtime, you can update Grafana settings without needing to restart the Grafana server.

Updates that happen at runtime are stored in the database and override
[settings from other sources](../)
(arguments, environment variables, settings file, etc). Therefore, every time a specific setting key is removed at runtime,
the value used for that key is the inherited one from the other sources in the reverse order of precedence
(`arguments > environment variables > settings file`). When no value is provided through any of these options, then the value used will be the application default

Currently, **it only supports updates on the `auth.saml` section.**

{{< admonition type="warning" >}}
Stored settings override the configuration file, environment variables, and command line arguments, and nothing in the UI or the file says so. If you rotate a credential by editing the file while a stored value exists, Grafana keeps using the stored value and your change has no effect.

Refer to [Check for stored settings](#check-for-stored-settings) when a configuration change appears to do nothing.
{{< /admonition >}}

## Update settings via the API

You can update settings through the [Admin API](../../../developers/http_api/admin/#update-settings).

When you submit a settings update via API, Grafana verifies if the given settings updates are allowed and valid. If they are, then Grafana stores the settings in the database and reloads
Grafana services with no need to restart the instance.

So, the payload of a `PUT` request to the update settings endpoint (`/api/admin/settings`)
should contain (either one or both):

- An `updates` map with a key, and a value per section you want to set.
- A `removals` list with keys per section you want to unset.

For example, if you provide the following `updates`:

```json
{
  "updates": {
    "auth.saml": {
      "enabled": "true",
      "single_logout": "false"
    }
  }
}
```

it would enable SAML and disable single logouts. And, if you provide the following `removals`:

```json
{
  "removals": {
    "auth.saml": ["allow_idp_initiated"]
  }
}
```

it would remove the key/value setting identified by `allow_idp_initiated` within the `auth.saml`.
So, the SAML service would be reloaded and that value would be inherited for either (settings `.ini` file,
environment variable, command line arguments or any other accepted mechanism to provide configuration).

Therefore, the complete HTTP payload would looks like:

```json
{
  "updates": {
    "auth.saml": {
      "enabled": "true",
      "single_logout": "false"
    }
  },
  "removals": {
    "auth.saml": ["allow_idp_initiated"]
  }
}
```

In case any of these settings cannot be overridden nor valid, it would return an error and these settings
won't be persisted into the database.

## Check for stored settings

Two stores can override your configuration file. Grafana resolves each key in this order: an SSO settings record, then the settings table this API writes, then command line arguments, environment variables, the configuration file, and defaults.

An SSO settings record comes from the [SSO Settings API](../../../developers/http_api/sso-settings/), the SAML and OAuth user interfaces, or Terraform. To check for one:

```sh
curl -s -u "<USERNAME>:<PASSWORD>" https://<GRAFANA_URL>/api/v1/sso-settings/saml
```

Replace _`<USERNAME>`_ and _`<PASSWORD>`_ with the credentials of a user that has the `settings:read` permission, and _`<GRAFANA_URL>`_ with your Grafana address.

A `"source": "database"` response means a record exists and wins over everything below it. A `"system"` response rules out a record, but not a value in the settings table. To inspect that table key by key, request `GET /api/admin/settings-verbose`, which marks each key `db` or `system`. That endpoint requires Grafana Enterprise.

While an SSO settings record exists, Grafana also ignores lower-precedence values for any key containing `certificate`, `private_key`, or `idp_metadata`, even when the record doesn't set them.

## Remove stored settings

For SAML, one request clears both stores:

```sh
curl -X DELETE -u "<USERNAME>:<PASSWORD>" https://<GRAFANA_URL>/api/v1/sso-settings/saml
```

This clears the SSO settings record and every `auth.saml` key in the settings table, so your configuration file takes over again. It returns `404` and clears nothing when no record exists.

To remove single keys from the settings table instead, send a `removals` list:

```sh
curl -X PUT -u "<USERNAME>:<PASSWORD>" \
  -H "Content-Type: application/json" \
  -d '{"removals": {"auth.saml": ["private_key"]}}' \
  https://<GRAFANA_URL>/api/admin/settings
```

Don't try to remove a key by leaving it out of a `PUT /api/v1/sso-settings/{provider}` payload. That request replaces the whole record, so anything you omit is deleted.

## Background job (high availability set-ups)

Grafana Enterprise has a built-in scheduled background job that looks into the database every minute for
settings updates. If there are updates, it reloads the Grafana services affected by the detected changes.

The background job synchronizes settings between instances in a highly available set-up. So, after you perform some changes through the
HTTP API, then the other instances are synchronized through the database and the background job.

## Control access with role-based access control

If you have [role-based access control](../../../administration/roles-and-permissions/access-control/) enabled, you can control who can read or update settings.
Refer to the [Admin API](../../../developers/http_api/admin/#update-settings) for more information.

# Audit a Grafana instance

Auditing allows you to track important changes to your Grafana instance. Depending on your deployment, you can send audit logs to a file, a Loki-compatible endpoint, or the Grafana default logger.

{{< admonition type="note" >}}
Available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/docs/grafana-cloud/).
{{< /admonition >}}

Only API requests or UI actions that trigger an API request generate an audit log.

- **Grafana Enterprise** supports all three exporters: file, Loki, and console.
- **Grafana Cloud** uses the Loki exporter. To enable audit logs for your Grafana Cloud instance, [file a support ticket](/profile/org/tickets/new). Standard ingest and retention rates apply.

## Understand audit logs

Audit logs are JSON objects representing user actions like:

- Modifications to resources such as dashboards and data sources.
- A user failing to log in.

### Review the audit log format

Audit logs contain the following fields. The fields followed by **\*** are always available, the others depend on the type of action logged.

| Field name              | Type    | Description                                                                                                                                                                                                              |
| ----------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `timestamp`\*           | string  | The date and time the request was made, in coordinated universal time (UTC) using the [RFC3339](https://tools.ietf.org/html/rfc3339#section-5.6) format.                                                                 |
| `user`\*                | object  | Information about the user that made the request. Either one of the `UserID` or `ApiKeyID` fields contains content if `isAnonymous=false`.                                                                               |
| `user.userId`           | number  | ID of the Grafana user that made the request.                                                                                                                                                                            |
| `user.orgId`\*          | number  | Current organization of the user that made the request.                                                                                                                                                                  |
| `user.orgRole`          | string  | Current role of the user that made the request.                                                                                                                                                                          |
| `user.name`             | string  | Name of the Grafana user that made the request.                                                                                                                                                                          |
| `user.authTokenId`      | number  | ID of the user authentication token.                                                                                                                                                                                     |
| `user.apiKeyId`         | number  | ID of the Grafana API key used to make the request.                                                                                                                                                                      |
| `user.isAnonymous`\*    | boolean | If an anonymous user made the request, `true`. Otherwise, `false`.                                                                                                                                                       |
| `action`\*              | string  | The request action. For example, `create`, `update`, or `manage-permissions`.                                                                                                                                            |
| `request`\*             | object  | Information about the HTTP request.                                                                                                                                                                                      |
| `request.params`        | object  | Request’s path parameters.                                                                                                                                                                                               |
| `request.query`         | object  | Request’s query parameters.                                                                                                                                                                                              |
| `request.body`          | string  | Request’s body. Filled with `<non-marshalable format>` when it isn't a valid JSON.                                                                                                                                       |
| `result`\*              | object  | Information about the HTTP response.                                                                                                                                                                                     |
| `result.statusType`     | string  | If the request action was successful, `success`. Otherwise, `failure`.                                                                                                                                                   |
| `result.statusCode`     | number  | HTTP status of the request.                                                                                                                                                                                              |
| `result.failureMessage` | string  | HTTP error message.                                                                                                                                                                                                      |
| `result.body`           | string  | Response body. Filled with `<non-marshalable format>` when it isn't a valid JSON.                                                                                                                                        |
| `resources`             | array   | Information about the resources that the request action affected. This field can be null for non-resource actions such as `login` or `logout`.                                                                           |
| `resources[x].id`       | number  | ID of the resource, when available.                                                                                                                                                                                      |
| `resources[x].type`\*   | string  | The type of the resource that was logged: `alert`, `alert-notification`, `annotation`, `api-key`, `auth-token`, `dashboard`, `datasource`, `folder`, `org`, `panel`, `playlist`, `report`, `team`, `user`, or `version`. |
| `resources[x].uid`      | string  | UID of the resource, when available.                                                                                                                                                                                     |
| `resources[x].name`     | string  | Name of the resource, when available. Grafana currently populates this field for `user` resources.                                                                                                                       |
| `requestUri`\*          | string  | Request URI.                                                                                                                                                                                                             |
| `ipAddress`\*           | string  | IP address that the request was made from.                                                                                                                                                                               |
| `userAgent`\*           | string  | Agent through which the request was made.                                                                                                                                                                                |
| `grafanaVersion`\*      | string  | Current version of Grafana when this log is created.                                                                                                                                                                     |
| `additionalData`        | object  | Additional information that can be provided about the request.                                                                                                                                                           |

The `additionalData` field can contain the following information:

| Field name          | Action                | Description                                                                      |
| ------------------- | --------------------- | -------------------------------------------------------------------------------- |
| `loginUsername`     | `login`               | Login used in the Grafana authentication form.                                   |
| `extUserInfo`       | `login`               | User information provided by the external system that was used to log in.        |
| `authTokenCount`    | `login`               | Number of active authentication tokens for the user that logged in.              |
| `terminationReason` | `logout`              | The reason why the user logged out, such as a manual logout or a token expiring. |
| `billing_role`      | `billing-information` | The billing role associated with the billing information being sent.             |

### Identify recorded actions

The audit logs include records about the following categories of actions. Each action is
distinguished by the `action` and `resources[...].type` fields in the JSON record.

For example, creating an API key produces an audit log like this:

```json {hl_lines=4}
{
  "action": "create",
  "resources": [
    {
      "id": 1,
      "type": "api-key"
    }
  ],
  "timestamp": "2021-11-12T22:12:36.144795692Z",
  "user": {
    "userId": 1,
    "orgId": 1,
    "orgRole": "Admin",
    "username": "admin",
    "isAnonymous": false,
    "authTokenId": 1
  },
  "request": {
    "body": "{\"name\":\"example\",\"role\":\"Viewer\",\"secondsToLive\":null}"
  },
  "result": {
    "statusType": "success",
    "statusCode": 200,
    "responseBody": "{\"id\":1,\"name\":\"example\"}"
  },
  "resources": [
    {
      "id": 1,
      "type": "api-key"
    }
  ],
  "requestUri": "/api/auth/keys",
  "ipAddress": "127.0.0.1:54652",
  "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0",
  "grafanaVersion": "8.3.0-pre"
}
```

Some actions can only be distinguished by their `requestUri` fields. For those actions, the relevant
pattern of the `requestUri` field is given.

Almost all recorded actions correspond to API requests or UI actions that trigger an API request. For example, the action `{"action": "email", "resources": [{"type": "report"}]}` corresponds to a user-initiated email preview of a report, not a scheduled send.

#### Sessions

| Action                           | Distinguishing fields                                                                      |
| -------------------------------- | ------------------------------------------------------------------------------------------ |
| Log in                           | `{"action": "login-AUTH-MODULE"}` \*                                                       |
| Log out \*\*                     | `{"action": "logout"}`                                                                     |
| Force logout for user            | `{"action": "logout-user"}`                                                                |
| Remove user authentication token | `{"action": "revoke-auth-token", "resources": [{"type": "auth-token"}, {"type": "user"}]}` |
| Create API key                   | `{"action": "create", "resources": [{"type": "api-key"}]}`                                 |
| Delete API key                   | `{"action": "delete", "resources": [{"type": "api-key"}]}`                                 |

\* Where `AUTH-MODULE` is the name of the authentication module: `grafana`, `saml`,
`ldap`, etc. \
\*\* Includes manual log out, token expired/revoked, and [SAML Single Logout](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-security/configure-authentication/saml/#single-logout).

#### Service accounts

| Action                       | Distinguishing fields                                                                                 |
| ---------------------------- | ----------------------------------------------------------------------------------------------------- |
| Create service account       | `{"action": "create", "resources": [{"type": "service-account"}]}`                                    |
| Update service account       | `{"action": "update", "resources": [{"type": "service-account"}]}`                                    |
| Delete service account       | `{"action": "delete", "resources": [{"type": "service-account"}]}`                                    |
| Create service account token | `{"action": "create", "resources": [{"type": "service-account"}, {"type": "service-account-token"}]}` |
| Delete service account token | `{"action": "delete", "resources": [{"type": "service-account"}, {"type": "service-account-token"}]}` |
| Hide API keys                | `{"action": "hide-api-keys"}`                                                                         |
| Migrate API keys             | `{"action": "migrate-api-keys"}`                                                                      |
| Migrate API key              | `{"action": "migrate-api-keys"}, "resources": [{"type": "api-key"}]}`                                 |

#### Access control

| Action                                   | Distinguishing fields                                                                                                       |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Create role                              | `{"action": "create", "resources": [{"type": "role"}]}`                                                                     |
| Update role                              | `{"action": "update", "resources": [{"type": "role"}]}`                                                                     |
| Delete role                              | `{"action": "delete", "resources": [{"type": "role"}]}`                                                                     |
| Assign built-in role                     | `{"action": "assign-builtin-role", "resources": [{"type": "role"}, {"type": "builtin-role"}]}`                              |
| Remove built-in role                     | `{"action": "remove-builtin-role", "resources": [{"type": "role"}, {"type": "builtin-role"}]}`                              |
| Grant team role                          | `{"action": "grant-team-role", "resources": [{"type": "team"}]}`                                                            |
| Set team roles                           | `{"action": "set-team-roles", "resources": [{"type": "team"}]}`                                                             |
| Revoke team role                         | `{"action": "revoke-team-role", "resources": [{"type": "role"}, {"type": "team"}]}`                                         |
| Grant user role                          | `{"action": "grant-user-role", "resources": [{"type": "role"}, {"type": "user"}]}`                                          |
| Set user roles                           | `{"action": "set-user-roles", "resources": [{"type": "user"}]}`                                                             |
| Revoke user role                         | `{"action": "revoke-user-role", "resources": [{"type": "role"}, {"type": "user"}]}`                                         |
| Set user permissions on folder           | `{"action": "set-user-permissions-on-folder", "resources": [{"type": "folder"}, {"type": "user"}]}`                         |
| Set team permissions on folder           | `{"action": "set-team-permissions-on-folder", "resources": [{"type": "folder"}, {"type": "team"}]}`                         |
| Set basic role permissions on folder     | `{"action": "set-basic-role-permissions-on-folder", "resources": [{"type": "folder"}, {"type": "builtin-role"}]}`           |
| Set user permissions on dashboard        | `{"action": "set-user-permissions-on-dashboards", "resources": [{"type": "dashboard"}, {"type": "user"}]}`                  |
| Set team permissions on dashboard        | `{"action": "set-team-permissions-on-dashboards", "resources": [{"type": "dashboard"}, {"type": "team"}]}`                  |
| Set basic role permissions on dashboard  | `{"action": "set-basic-role-permissions-on-dashboards", "resources": [{"type": "dashboard"}, {"type": "builtin-role"}]}`    |
| Set user permissions on team             | `{"action": "set-user-permissions-on-teams", "resources": [{"type": "teams"}, {"type": "user"}]}`                           |
| Set user permissions on service account  | `{"action": "set-user-permissions-on-service-accounts", "resources": [{"type": "service-account"}, {"type": "user"}]}`      |
| Set user permissions on datasource       | `{"action": "set-user-permissions-on-data-sources", "resources": [{"type": "datasource"}, {"type": "user"}]}`               |
| Set team permissions on datasource       | `{"action": "set-team-permissions-on-data-sources", "resources": [{"type": "datasource"}, {"type": "team"}]}`               |
| Set basic role permissions on datasource | `{"action": "set-basic-role-permissions-on-data-sources", "resources": [{"type": "datasource"}, {"type": "builtin-role"}]}` |

#### User management

| Action                    | Distinguishing fields                                               |
| ------------------------- | ------------------------------------------------------------------- |
| Create user               | `{"action": "create", "resources": [{"type": "user"}]}`             |
| Update user               | `{"action": "update", "resources": [{"type": "user"}]}`             |
| Delete user               | `{"action": "delete", "resources": [{"type": "user"}]}`             |
| Disable user              | `{"action": "disable", "resources": [{"type": "user"}]}`            |
| Enable user               | `{"action": "enable", "resources": [{"type": "user"}]}`             |
| Update password           | `{"action": "update-password", "resources": [{"type": "user"}]}`    |
| Send password reset email | `{"action": "send-reset-email"}`                                    |
| Reset password            | `{"action": "reset-password"}`                                      |
| Update permissions        | `{"action": "update-permissions", "resources": [{"type": "user"}]}` |
| Send signup email         | `{"action": "signup-email"}`                                        |
| Click signup link         | `{"action": "signup"}`                                              |
| Reload LDAP configuration | `{"action": "ldap-reload"}`                                         |
| Get user in LDAP          | `{"action": "ldap-search"}`                                         |
| Sync user with LDAP       | `{"action": "ldap-sync", "resources": [{"type": "user"}]`           |

#### Team and organization management

| Action                               | Distinguishing fields                                                        |
| ------------------------------------ | ---------------------------------------------------------------------------- |
| Add team                             | `{"action": "create", "requestUri": "/api/teams"}`                           |
| Update team                          | `{"action": "update", "requestUri": "/api/teams/TEAM-ID"}`\*                 |
| Delete team                          | `{"action": "delete", "requestUri": "/api/teams/TEAM-ID"}`\*                 |
| Add external group for team          | `{"action": "create", "requestUri": "/api/teams/TEAM-ID/groups"}`\*          |
| Remove external group for team       | `{"action": "delete", "requestUri": "/api/teams/TEAM-ID/groups/GROUP-ID"}`\* |
| Add user to team                     | `{"action": "create", "resources": [{"type": "user"}, {"type": "team"}]}`    |
| Update team member permissions       | `{"action": "update", "resources": [{"type": "user"}, {"type": "team"}]}`    |
| Remove user from team                | `{"action": "delete", "resources": [{"type": "user"}, {"type": "team"}]}`    |
| Create organization                  | `{"action": "create", "resources": [{"type": "org"}]}`                       |
| Update organization                  | `{"action": "update", "resources": [{"type": "org"}]}`                       |
| Delete organization                  | `{"action": "delete", "resources": [{"type": "org"}]}`                       |
| Add user to organization             | `{"action": "create", "resources": [{"type": "org"}, {"type": "user"}]}`     |
| Change user role in organization     | `{"action": "update", "resources": [{"type": "user"}, {"type": "org"}]}`     |
| Remove user from organization        | `{"action": "delete", "resources": [{"type": "user"}, {"type": "org"}]}`     |
| Invite external user to organization | `{"action": "org-invite", "resources": [{"type": "org"}, {"type": "user"}]}` |
| Revoke invitation                    | `{"action": "revoke-org-invite", "resources": [{"type": "org"}]}`            |

\* Where `TEAM-ID` is the ID of the affected team, and `GROUP-ID` (if present) is the ID of the
external group.

#### Folder and dashboard management

| Action                        | Distinguishing fields                                                    |
| ----------------------------- | ------------------------------------------------------------------------ |
| Create folder                 | `{"action": "create", "resources": [{"type": "folder"}]}`                |
| Update folder                 | `{"action": "update", "resources": [{"type": "folder"}]}`                |
| Update folder permissions     | `{"action": "manage-permissions", "resources": [{"type": "folder"}]}`    |
| Delete folder                 | `{"action": "delete", "resources": [{"type": "folder"}]}`                |
| Create/update dashboard       | `{"action": "create-update", "resources": [{"type": "dashboard"}]}`      |
| Import dashboard              | `{"action": "create", "resources": [{"type": "dashboard"}]}`             |
| Update dashboard permissions  | `{"action": "manage-permissions", "resources": [{"type": "dashboard"}]}` |
| Restore old dashboard version | `{"action": "restore", "resources": [{"type": "dashboard"}]}`            |
| Delete dashboard              | `{"action": "delete", "resources": [{"type": "dashboard"}]}`             |

#### Library elements management

| Action                 | Distinguishing fields                                              |
| ---------------------- | ------------------------------------------------------------------ |
| Create library element | `{"action": "create", "resources": [{"type": "library-element"}]}` |
| Update library element | `{"action": "update", "resources": [{"type": "library-element"}]}` |
| Delete library element | `{"action": "delete", "resources": [{"type": "library-element"}]}` |

#### Data sources management

| Action                                             | Distinguishing fields                                                                     |
| -------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| Create datasource                                  | `{"action": "create", "resources": [{"type": "datasource"}]}`                             |
| Update datasource                                  | `{"action": "update", "resources": [{"type": "datasource"}]}`                             |
| Delete datasource                                  | `{"action": "delete", "resources": [{"type": "datasource"}]}`                             |
| Enable permissions for datasource                  | `{"action": "enable-permissions", "resources": [{"type": "datasource"}]}`                 |
| Disable permissions for datasource                 | `{"action": "disable-permissions", "resources": [{"type": "datasource"}]}`                |
| Grant datasource permission to role, team, or user | `{"action": "create", "resources": [{"type": "datasource"}, {"type": "dspermission"}]}`\* |
| Remove datasource permission                       | `{"action": "delete", "resources": [{"type": "datasource"}, {"type": "dspermission"}]}`   |
| Enable caching for datasource                      | `{"action": "enable-cache", "resources": [{"type": "datasource"}]}`                       |
| Disable caching for datasource                     | `{"action": "disable-cache", "resources": [{"type": "datasource"}]}`                      |
| Update datasource caching configuration            | `{"action": "update", "resources": [{"type": "datasource"}]}`                             |

\* `resources` may also contain a third item with `"type":` set to `"user"` or `"team"`.

#### Data source query

| Action           | Distinguishing fields                                        |
| ---------------- | ------------------------------------------------------------ |
| Query datasource | `{"action": "query", "resources": [{"type": "datasource"}]}` |

#### Reporting

| Action                    | Distinguishing fields                                                            |
| ------------------------- | -------------------------------------------------------------------------------- |
| Create report             | `{"action": "create", "resources": [{"type": "report"}, {"type": "dashboard"}]}` |
| Update report             | `{"action": "update", "resources": [{"type": "report"}, {"type": "dashboard"}]}` |
| Delete report             | `{"action": "delete", "resources": [{"type": "report"}]}`                        |
| Send report by email      | `{"action": "email", "resources": [{"type": "report"}]}`                         |
| Update reporting settings | `{"action": "change-settings"}`                                                  |

#### Annotations, playlists and snapshots management

| Action                            | Distinguishing fields                                                                |
| --------------------------------- | ------------------------------------------------------------------------------------ |
| Create annotation                 | `{"action": "create", "resources": [{"type": "annotation"}]}`                        |
| Create Graphite annotation        | `{"action": "create-graphite", "resources": [{"type": "annotation"}]}`               |
| Update annotation                 | `{"action": "update", "resources": [{"type": "annotation"}]}`                        |
| Patch annotation                  | `{"action": "patch", "resources": [{"type": "annotation"}]}`                         |
| Delete annotation                 | `{"action": "delete", "resources": [{"type": "annotation"}]}`                        |
| Delete all annotations from panel | `{"action": "mass-delete", "resources": [{"type": "dashboard"}, {"type": "panel"}]}` |
| Create playlist                   | `{"action": "create", "resources": [{"type": "playlist"}]}`                          |
| Update playlist                   | `{"action": "update", "resources": [{"type": "playlist"}]}`                          |
| Delete playlist                   | `{"action": "delete", "resources": [{"type": "playlist"}]}`                          |
| Create a snapshot                 | `{"action": "create", "resources": [{"type": "dashboard"}, {"type": "snapshot"}]}`   |
| Delete a snapshot                 | `{"action": "delete", "resources": [{"type": "snapshot"}]}`                          |
| Delete a snapshot by delete key   | `{"action": "delete", "resources": [{"type": "snapshot"}]}`                          |

#### Provisioning

| Action                            | Distinguishing fields                      |
| --------------------------------- | ------------------------------------------ |
| Reload provisioned dashboards     | `{"action": "provisioning-dashboards"}`    |
| Reload provisioned datasources    | `{"action": "provisioning-datasources"}`   |
| Reload provisioned plugins        | `{"action": "provisioning-plugins"}`       |
| Reload provisioned alerts         | `{"action": "provisioning-alerts"}`        |
| Reload provisioned access control | `{"action": "provisioning-accesscontrol"}` |

#### Plugins management

| Action           | Distinguishing fields     |
| ---------------- | ------------------------- |
| Install plugin   | `{"action": "install"}`   |
| Uninstall plugin | `{"action": "uninstall"}` |

#### Miscellaneous

| Action                   | Distinguishing fields                                        |
| ------------------------ | ------------------------------------------------------------ |
| Set licensing token      | `{"action": "create", "requestUri": "/api/licensing/token"}` |
| Save billing information | `{"action": "billing-information"}`                          |

#### Cloud migration management

| Action                           | Distinguishing fields                                       |
| -------------------------------- | ----------------------------------------------------------- |
| Connect to a cloud instance      | `{"action": "connect-instance"}`                            |
| Disconnect from a cloud instance | `{"action": "disconnect-instance"}`                         |
| Build a snapshot                 | `{"action": "build", "resources": [{"type": "snapshot"}]}`  |
| Upload a snapshot                | `{"action": "upload", "resources": [{"type": "snapshot"}]}` |

#### Generic actions

In addition to the actions listed above, any HTTP request (`POST`, `PATCH`, `PUT`, and `DELETE`)
against the API is recorded with one of the following generic actions.

You can also record `GET` requests. Refer to [Configure audit logging](#configure-audit-logging) to enable this.

| Action         | Distinguishing fields          |
| -------------- | ------------------------------ |
| POST request   | `{"action": "post-action"}`    |
| PATCH request  | `{"action": "partial-update"}` |
| PUT request    | `{"action": "update"}`         |
| DELETE request | `{"action": "delete"}`         |
| GET request    | `{"action": "retrieve"}`       |

## Configure audit logging

{{< admonition type="note" >}}
The auditing feature is disabled by default.
{{< /admonition >}}

Audit logs can be saved into files, sent to a Loki-compatible endpoint, or sent to the Grafana default logger. You can choose which exporter to use in the [configuration file](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/).

Options are `file`, `loki`, and `logger`. Use spaces to separate multiple modes, such as `file loki`.

Not all exporters are available in every deployment:

| Exporter | Grafana Enterprise | Grafana Cloud |
| -------- | ------------------ | ------------- |
| File     | Yes (default)      | No            |
| Loki     | Yes                | Yes (default) |
| Console  | Yes                | No            |

By default, dashboard content doesn't appear in audit logs because it can significantly increase log size. To include dashboard content, enable the `log_dashboard_content` option.

Data source query request and response bodies are also excluded by default. Enabling `log_datasource_query_request_body` or `log_datasource_query_response_body` significantly increases log volume and may expose sensitive data such as query parameters, credentials, or personally identifiable information.

Before Grafana serves `DELETE /api/admin/users/{id}`, it tries to capture the user's display name, or login when the display name is empty, so the audit log can retain it after a successful deletion. The `snapshot_timeout` option sets a cooperative context deadline for this lookup. The user store must honor context cancellation, so the setting isn't a strict upper bound on request latency. If the lookup fails or observes an expired deadline, Grafana continues to process the request. If the deletion succeeds, the audit log omits `resources[x].name`.

```ini
[auditing]
# Enable the auditing feature
enabled = false
# List of enabled loggers
loggers = file
# Keep dashboard content in the logs (request or response fields); this can significantly increase the size of your logs.
log_dashboard_content = false
# Whether to record data source queries' request body. This can significantly increase the size of your logs. Disabled by default.
log_datasource_query_request_body = false
# Whether to record data source queries' response body. This can significantly increase the size of your logs. Disabled by default.
log_datasource_query_response_body = false
# Keep requests and responses body; this can significantly increase the size of your logs.
verbose = false
# Write an audit log for every status code.
# By default it only logs the following ones: 2XX, 3XX, 401, 403 and 500.
log_all_status_codes = false
# Maximum response body (in bytes) to be audited; 500KiB by default.
# May help reducing the memory footprint caused by auditing.
max_response_size_bytes = 512000
# Timeout for resolving resource names before Grafana serves a request that
# may delete resources. Uses duration format, for example, 500ms or 2s.
# The default is 500ms.
snapshot_timeout = 500ms
```

Each exporter has its own configuration fields.

### File exporter

{{< admonition type="note" >}}
The file exporter is only available in Grafana Enterprise.
{{< /admonition >}}

Audit logs are saved into files. You can configure the folder to use to save these files. Logs are rotated when the file size is exceeded and at the start of a new day.

```ini
[auditing.logs.file]
# Path to logs folder
path = data/log
# Maximum log files to keep
max_files = 5
# Max size in megabytes per log file
max_file_size_mb = 256
```

### Loki exporter

Audit logs are sent to a [Loki](https://grafana.com/oss/loki/)-compatible endpoint through HTTP or gRPC. You can use this exporter to send logs to Grafana Cloud Logs, a self-hosted Loki instance, or any other service that accepts the Loki push API.

{{< admonition type="note" >}}
The HTTP option for the Loki exporter is available only in Grafana Enterprise version 7.4 and later.
{{< /admonition >}}

```ini
[auditing.logs.loki]
# Set the communication protocol to use with Loki (can be grpc or http)
type = grpc
# Set the address for writing logs to Loki
url = localhost:9095
# Defaults to true. If true, it establishes a secure connection to Loki
tls = true
# Set the tenant ID for Loki communication, which is disabled by default.
# The tenant ID is required to interact with Loki running in multi-tenant mode.
tenant_id =
# Only available for the HTTP client. Disabled by default.
# How long to wait before sending a request to Loki with the batch of events. Uses duration format, for example, 5s, 1m
# Whichever threshold is hit first between `batch_wait_duration` and `batch_size_bytes` triggers the batch to be sent to Loki.
# If the wait duration is very long and the `batch_size_bytes` is very high, events might take a long time to be sent.
# To enable batching, you must also set the `batch_size_bytes` configuration option.
batch_wait_duration =
# Only available for the HTTP client. Disabled by default.
# How many events, in bytes, to accumulate in a single batch before sending it to Loki.
# Whichever threshold is hit first between `batch_wait_duration` and `batch_size_bytes` triggers the batch to be sent to Loki.
# If you want to always wait for the `batch_wait_duration`, set `batch_size_bytes` to a very high number.
# To enable batching, you must also set the `batch_wait_duration` configuration option.
batch_size_bytes =
```

If you have multiple Grafana instances sending logs to the same Loki service or if you use Loki for non-audit logs, audit logs come with additional labels to help identify them:

| Label            | Value                                                |
| ---------------- | ---------------------------------------------------- |
| host             | OS hostname on which the Grafana instance is running |
| grafana_instance | Application URL                                      |
| kind             | `auditing`                                           |

If your Loki instance requires basic authentication, specify credentials in the URL field. For example:

```ini
# Set the communication protocol to use with Loki (can be grpc or http)
type = http
# Set the address for writing logs to Loki
url = user:password@localhost:3000
```

### Console exporter

{{< admonition type="note" >}}
The console exporter is only available in Grafana Enterprise.
{{< /admonition >}}

Audit logs are sent to the Grafana default logger. The audit logs use the `auditing.console` logger and are logged on `debug`-level. To learn how to enable debug logging, refer to the [log configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#log) section. Accessing the audit logs in this way isn't recommended for production use.

# Configure request security

Request security allows you to limit requests from the Grafana server by targeting requests generated by users, such as data source metric queries and alert notifications.

This can be used to limit access to internal systems that the server Grafana runs on can access but that users of Grafana should not be able to access. This feature does not affect traffic from the Grafana users browser.

{{< admonition type="note" >}}
Available in [Grafana Enterprise](../../../introduction/grafana-enterprise/) and to users on select Grafana Cloud account plans. For pricing information, visit our [pricing page](https://grafana.com/pricing/) or contact our sales team.
{{< /admonition >}}

{{< admonition type="note" >}}
Although request security works with backend plugins, you can create a backend plugin that bypasses this security.
{{< /admonition >}}

## IP and hostname blocking

You can limit requests based on a hostname, an IP address, or both.

### Deny list

Grafana blocks any request to a hostname or IP address on the deny list.

### Allow list

If there is at least one entry on the list, then any request to a hostname or IP address not on the list is denied.

For example:

```toml
[security.egress]
# A list of hostnames or IP addresses separated by spaces for which requests are blocked.
host_deny_list = supersecret.internal 192.168.1.10
# a list of hostnames or IP addresses separated by spaces for which requests will be allowed, all other requests will be blocked
host_allow_list = prometheus.internal

```

## Drop headers and cookies

You can set a list of cookies or headers that are to be dropped from outgoing requests.

Example:

```toml
[security.egress]
# a list of headers that will be stripped from outgoing datasource and alerting requests
header_drop_list = user
# a list of cookies that will be stripped from outgoing datasource requests (case sensitive)
cookie_drop_list = session_id
```

# Export logs of usage insights

{{< admonition type="note" >}}
Available in [Grafana Enterprise](../../../introduction/grafana-enterprise/) and to customers on select Grafana Cloud plans. For pricing information, visit [pricing](https://grafana.com/pricing/) or contact our sales team.
{{< /admonition >}}

By exporting usage logs to Loki, you can directly query them and create dashboards of the information that matters to you most, such as dashboard errors, most active organizations, or your top-10 most-used queries. This configuration is done for you in Grafana Cloud, with provisioned dashboards. Read about them in the [Grafana Cloud documentation](/docs/grafana-cloud/usage-insights/).

## Usage insights logs

Usage insights logs are JSON objects that represent certain user activities, such as:

- A user opens a dashboard.
- A query is sent to a data source.

### Scope

A log is created every time:

- A user opens a dashboard.
- A query is sent to a data source in the dashboard view.
- A query is performed via Explore.

### Format

Logs of usage insights contain the following fields, where the fields followed by \* are always available, and the others depend on the logged event:
| Field name | Type | Description |
| ---------- | ---- | ----------- |
| `eventName`\* | string | Type of the event, which can be either `data-request` or `dashboard-view`. |
| `folderName`\* | string | Name of the dashboard folder. |
| `dashboardName`\* | string | Name of the dashboard where the event happened. |
| `dashboardId`\* | number | ID of the dashboard where the event happened. |
| `datasourceName`| string | Name of the data source that was queried. |
| `datasourceType` | string | Type of the data source that was queried. For example, `prometheus`, `elasticsearch`, or `loki`. |
| `datasourceId` | number | ID of the data source that was queried. |
| `panelId` | number | ID of the panel of the query. |
| `panelName` | string | Name of the panel of the query. |
| `error` | string | Error returned by the query. |
| `duration` | number | Duration of the query. |
| `source` | string | Source of the query. For example, `dashboard` or `explore`. |
| `orgId`\* | number | ID of the user’s organization. |
| `orgName`\* | string | Name of the user’s organization. |
| `timestamp`\* | string | The date and time that the request was made, in Coordinated Universal Time (UTC) in [RFC3339](https://tools.ietf.org/html/rfc3339#section-5.6) format. |
| `tokenId`\* | number | ID of the user’s authentication token. |
| `username`\* | string | Name of the Grafana user that made the request. |
| `userId`\* | number | ID of the Grafana user that made the request. |
| `totalQueries`\* | number | Number of queries executed for the data request. |
| `cachedQueries`\* | number | Number of fetched queries that came from the cache. |

## Configuration

{{< admonition type="caution" >}}
Grafana Cloud provides provisioned dashboards for usage insights. Refer to the [Grafana Cloud documentation](/docs/grafana-cloud/usage-insights/) for more information. If you export usage insights to a separate Loki instance, these dashboards stop receiving usage insights data. This is a current limitation of the feature in Grafana Cloud.
{{< /admonition >}}

To export your logs, enable the usage insights feature and [configure](../../configure-grafana/) an export location in the configuration file:

```ini
[usage_insights.export]
# Enable the usage insights export feature
enabled = true
# Storage type
storage = loki
```

The options for storage type are `loki` and `logger` (added in Grafana Enterprise 8.2).

If the storage type is set to `loki` you'll need to also configure Grafana
to export to a Loki ingestion server. To do this, you'll need Loki installed.
Refer to [Install Loki](/docs/loki/latest/installation/) for instructions
on how to install Loki.

```ini
[usage_insights.export.storage.loki]
# Set the communication protocol to use with Loki (can be grpc or http)
type = grpc
# Set the address for writing logs to Loki (format must be host:port)
url = localhost:9095
# Defaults to true. If true, it establishes a secure connection to Loki
tls = true
# Set the tenant ID for Loki communication, which is disabled by default.
# The tenant ID is required to interact with Loki running in multi-tenant mode.
tenant_id =
# How long to wait before sending a request to Loki with the batch of events. Uses duration format: e.g. 5s, 1m
# Whatever happens first between `batch_wait_duration` and `batch_size_bytes` will trigger the batch to be sent to Loki.
# If the wait duration is very long and the `batch_size_bytes` is very high, events may take a long time to be sent.
batch_wait_duration = 5s
# How many events (in bytes) to accumulate in a single batch before sending it to Loki.
# Whatever happens first between `batch_wait_duration` and `batch_size_bytes` will trigger the batch to be sent to Loki.
# If you wish to always wait for the `batch_wait_duration`, set this to a very high number.
batch_size_bytes = 102400 # 100KiB
```

Using `logger` will print usage insights to your [Grafana server log](../../configure-grafana/#log).
There is no option for configuring the `logger` storage type.

## Visualize Loki usage insights in Grafana

If you export logs into Loki, you can build Grafana dashboards to understand your Grafana instance usage.

1. Add Loki as a data source. Refer to [Grafana fundamentals tutorial](/tutorials/grafana-fundamentals/#6).
1. Import one of the following dashboards:
   - [Usage insights](/grafana/dashboards/13785)
   - [Usage insights datasource details](/grafana/dashboards/13786)
1. Play with usage insights to understand them:
   - In Explore, you can use the query `{datasource="gdev-loki",kind="usage_insights"}` to retrieve all logs related to your `gdev-loki` data source.
   - In a dashboard, you can build a table panel with the query `topk(10, sum by (error) (count_over_time({kind="usage_insights", datasource="gdev-prometheus"} | json | error != "" [$__interval])))` to display the 10 most common errors your users see using the `gdev-prometheus` data source.
   - In a dashboard, you can build a graph panel with the queries `sum by(host) (count_over_time({kind="usage_insights"} | json | eventName="data-request" | error != "" [$__interval]))` and `sum by(host) (count_over_time({kind="usage_insights"} | json | eventName="data-request" | error = "" [$__interval]))` to show the evolution of the data request count over time. Using `by (host)` allows you to have more information for each Grafana server you have if you have set up Grafana for [high availability](../../set-up-for-high-availability/).

# Configure Grafana secret scanning and notifications

With Grafana, you can use the GitHub Secret Scanning service to determine if your [service account tokens](../../../administration/service-accounts/) have been leaked on GitHub.

When GitHub Secret Scanning detects a Grafana secret, its hash is stored in Grafana Labs' secret scanning service.

Grafana instances, whether on-premises or on the cloud, can use this service to verify if a token generated by the instance has been made public. This verification is done by comparing the token's hash with the exposed token's hash.

If the service detects a leaked token, it immediately revokes it, making it useless, and logs the event.

{{< admonition type="note" >}}
If the `revoke` option is disabled, the service only sends a notification to the configured webhook URL and logs the event. The token is not automatically revoked.
{{< /admonition >}}

You can also configure the service to send an outgoing webhook notification to a webhook URL.

The notification includes a JSON payload that contains the following data:

```json
{
  "alert_uid": "c9ce50a1-d66b-45e4-9b5d-175766cfc026",
  "link_to_upstream_details": <URL to token leak>,
  "message": "Token of type grafana_service_account_token with name
sa-the-toucans has been publicly exposed in <URL to token leak>.
Grafana has revoked this token",
  "state": "alerting",
  "title": "SecretScan Alert: Grafana Token leaked"
}
```

{{< admonition type="note" >}}
Secret scanning is disabled by default. Outgoing connections are made once you enable it.
{{< /admonition >}}

## Before you begin

- Ensure all your API keys have been migrated to service accounts.
  For more information about service account migration, refer to [Migrate API keys to Grafana service accounts](/docs/grafana/<GRAFANA_VERSION>/administration/service-accounts/migrate-api-keys/).

## Configure secret scanning

1. Open the Grafana configuration file.

1. In the `[secretscan]` section, update the following parameters:

```ini
[secretscan]
# Enable secretscan feature
enabled = true

# Whether to revoke the token if a leak is detected or just send a notification
revoke = true
```

Save the configuration file and restart Grafana.

## Configure outgoing webhook notifications

1. Create an oncall integration of the type **Webhook** and set up alerts.
   To learn how to create a Grafana OnCall integration, refer to [Inbound Webhook integrations for Grafana OnCall](/docs/oncall/latest/integrations/webhook/).

1. Copy the webhook URL of the new integration.

1. Open the Grafana configuration file.

1. In the `[secretscan]` section, update the following parameters,
   replacing the URL with the webhook URL you copied in step 2.

```ini
[secretscan]
# URL to send a webhook payload in oncall format
oncall_url = https://example.url/integrations/v1/webhook/3a359nib9eweAd9lAAAETVdOx/
```

Save the configuration file and restart Grafana.

# Encrypt database secrets using AWS KMS

You can use an encryption key from AWS Key Management Service to encrypt secrets in the Grafana database.

**Prerequisites:**

- An AWS account with permission to view and create KMS keys and programmatic credentials to access those keys
- Access to the Grafana [configuration](../../../configure-grafana/#configuration-file-location) file

1. Create a symmetric API key either from the AWS Management Console or by using the AWS KMS API.
   <br><br>For detailed instructions, refer to [Creating keys](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html).

2. Retrieve the Key ID.
   <br><br>In AWS terms, this can be a key ID, a key ARN (Amazon Resource Name), an alias name, or an alias ARN. For more information about how to retrieve a key ID from AWS, refer to [Finding the key ID and key ARN](https://docs.aws.amazon.com/kms/latest/developerguide/find-cmk-id-arn.html).

3. Create a [programmatic credential](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys) (access key ID and secret access key), which has permission to view the key that you created.
   <br><br>In AWS, you can control access to your KMS keys by using [key policies](https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html), [IAM policies](https://docs.aws.amazon.com/kms/latest/developerguide/iam-policies.html), and [grants](https://docs.aws.amazon.com/kms/latest/developerguide/grants.html). You can also create [temporary credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html), which must provide a session token along with an access key ID and a secret access key.

4. From within Grafana, turn on envelope encryption.
5. Add your AWS KMS details to the Grafana configuration file; depending on your operating system, it is usually named `grafana.ini`:
   <br><br>a. Add a new section to the configuration file, with a name in the format of `[security.encryption.awskms.<KEY-NAME>]`, where `<KEY-NAME>` is any name that uniquely identifies this key among other provider keys.
   <br><br>b. Fill in the section with the following values:
   <br>
   - `key_id`: a reference to a key stored in the KMS. This can be a key ID, a key Amazon Resource Name (ARN), an alias name, or an alias ARN. If you are using an alias, use the prefix `alias/`. To specify a KMS key in a different AWS account, use its ARN or alias. For more information about how to retrieve a key ID from AWS, refer to [Finding the key ID and key ARN](https://docs.aws.amazon.com/kms/latest/developerguide/find-cmk-id-arn.html).<br>
     | `key_id` option | Example value |
     | --- | --- |
     | Key ID | `1234abcd-12ab-34cd-56ef-1234567890ab` |
     | Key ARN | `arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab` |
     | Alias name | `alias/ExampleAlias` |
     | Alias ARN | `arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias` |

   - `access_key_id`: The AWS Access Key ID that you previously generated.
   - `secret_access_key`: The AWS Secret Access Key you previously generated.
   - `region`: The AWS region where you created the KMS key. The region is contained in the key’s ARN. For example: `arn:aws:kms:*us-east-2*:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab`

   An example of an AWS KMS provider section in the `grafana.ini` file is as follows:

   ```
   # AWS key management service provider setup
   ;[security.encryption.awskms.example-encryption-key]
   # Reference to a KMS key - either key ID, key ARN, alias name, or ARN
   ;key_id = 1234abcd-12ab-34cd-56ef-1234567890ab
   # AWS access key ID
   ;access_key_id = AKIAIOSFODNN7EXAMPLE
   # AWS secret access key
   ;secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
   # AWS region, for example eu-north-1
   ;region = eu-north-1
   ```

6. Update the `[security]` section of the `grafana.ini` configuration file with the new Encryption Provider key that you created:

   ```
   [security]
   # previous encryption key, used for legacy alerts, decrypting existing secrets or used as default provider when external providers are not configured
   secret_key = AaaaAaaa
   # encryption provider key in the format <PROVIDER>.<KEY_NAME>
   encryption_provider = awskms.example-encryption-key
   # list of configured key providers, space separated
   available_encryption_providers = awskms.example-encryption-key
   ```

7. [Restart Grafana](/docs/grafana/latest/installation/restart-grafana/).

8. (Optional) From the command line and the root directory of Grafana, re-encrypt all of the secrets within the Grafana database with the new key using the following command:

   `grafana cli admin secrets-migration re-encrypt`

   If you do not re-encrypt existing secrets, then they will remain encrypted by the previous encryption key. Users will still be able to access them.

   **> Note:** This process could take a few minutes to complete, depending on the number of secrets (such as data sources) in your database. Users might experience errors while this process is running, and alert notifications might not be sent.

   **> Note:** If you are updating this encryption key during the initial setup of Grafana before any data sources or dashboards have been created, then this step is not necessary because there are no secrets in Grafana to migrate.

# Encrypt database secrets using Azure Key Vault

You can use an encryption key from Azure Key Vault to encrypt secrets in the Grafana database.

**Prerequisites:**

- An Azure account with permission to view and create Key Vault keys and programmatic credentials to access those keys
- Access to the Grafana [configuration](../../../configure-grafana/#configuration-file-location) file

1. [Create a vault](https://docs.microsoft.com/en-us/azure/key-vault/general/quick-create-portal#create-a-vault).

2. Create a key in the **Key Vault** with the name that you want by using **RSA** as the type and `2048` as the size with encrypt and decrypt permissions.

3. [Register an application](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#register-an-application) and generate a client secret for it.

4. [Assign a Key Vault access policy](https://docs.microsoft.com/en-us/azure/key-vault/general/assign-access-policy?tabs=azure-portal) for the key vault that you created:

5. In the Key Permissions section, set encrypt and decrypt permissions, and click **Save**.

6. From within Grafana, turn on envelope encryption.

7. Add your Azure Key Vault details to the Grafana configuration file; depending on your operating system, is usually named `grafana.ini`:
   <br><br>a. Add a new section to the configuration file, with a name in the format of `[security.encryption.azurekv.<KEY-NAME>]`, where `<KEY-NAME>` is any name that uniquely identifies this key among other provider keys.
   <br><br>b. Fill in the section with the following values:
   <br>
   - `tenant_id`: the **Directory ID** (tenant) from the application that you registered.
   - `client_id`: the **Application ID** (client) from the application that you registered.
   - `client_secret`: the VALUE of the secret that you generated in your app. (Don't use the Secret ID).
   - `key_id`: the key name that you created in the key vault.
   - `vault_uri`: the URL of your key vault.

   An example of an Azure Key Vault provider section in the `grafana.ini` file is as follows:

   ```
   # Azure Key Vault provider setup
   ;[security.encryption.azurekv.example-encryption-key]
   # Azure Application directory ID (tenant)
   tenant_id = 1234abcd-12ab-34cd-56ef-1234567890ab
   # Azure Application application ID (client).
   client_id = 1356dfgh-12ab-34cd-56ef-3322114455cc
   # Azure Application client secret.
   client_secret = FbE4X~4Jq45ERKxx823Aheb9plBjQqHHe81Sc
   # Azure Key Vault key name.
   key_id = mysecretkey
   # Azure Key Vault uri.
   vault_uri = https://my-vault-name.vault.azure.net
   ```

8. Update the `[security]` section of the `grafana.ini` configuration file with the new Encryption Provider key that you created:

   ```
   [security]
   # previous encryption key, used for legacy alerts, decrypting existing secrets or used as default provider when external providers are not configured
   secret_key = AaaaAaaa
   # encryption provider key in the format <PROVIDER>.<KEY-NAME>
   encryption_provider = azurekv.example-encryption-key
   # list of configured key providers, space separated
   available_encryption_providers =  azurekv.example-encryption-key
   ```

9. [Restart Grafana](/docs/grafana/latest/installation/restart-grafana/).

10. (Optional) From the command line and the root directory of Grafana Enterprise, re-encrypt all of the secrets within the Grafana database with the new key using the following command:

    `grafana cli admin secrets-migration re-encrypt`

    If you do not re-encrypt existing secrets, then they will remain encrypted by the previous encryption key. Users will still be able to access them.

    **> Note:** This process could take a few minutes to complete, depending on the number of secrets (such as data sources) in your database. Users might experience errors while this process is running, and alert notifications might not be sent.

    **> Note:** If you are updating this encryption key during the initial setup of Grafana before any data sources or dashboards have been created, then this step is not necessary because there are no secrets in Grafana to migrate.

# Encrypt database secrets using Google Cloud KMS

You can use an encryption key from Google Cloud Key Management Service to encrypt secrets in the Grafana database.

**Prerequisites:**

- A Google Cloud account with permission to list and create KMS keys and service accounts to access those keys
- Access to the Grafana [configuration](../../../configure-grafana/#configuration-file-location) file

1. [Create a key ring](https://cloud.google.com/kms/docs/creating-keys#kms-create-key-ring-console) in Google Cloud KMS.

2. [Create a symmetric encryption key](https://cloud.google.com/kms/docs/creating-keys#create_a_key) in the key ring.

3. [Create a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating) and assign it a role: it can be a predefined role or custom role with permissions to encrypt and decrypt secrets with Key Management Service.

4. [Create a service account key and save its JSON file](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating) to you computer, for example, as `~/.config/gcloud/sample-project-credentials.json`.

5. From within Grafana, turn on envelope encryption.

6. Add your Google Cloud KMS details to the Grafana configuration file; depending on your operating system, is usually named `grafana.ini`:
   <br><br>a. Add a new section to the configuration file, with a name in the format of `[security.encryption.azurekv.<KEY-NAME>]`, where `<KEY-NAME>` is any name that uniquely identifies this key among other provider keys.
   <br><br>b. Fill in the section with the following values:
   <br>
   - `key_id`: encryption key ID, refer to [Getting the ID for a Key](https://cloud.google.com/kms/docs/getting-resource-ids#getting_the_id_for_a_key_and_version).
   - `credentials_file`: full path to service account key JSON file on your computer.

   An example of a Google Cloud KMS provider section in the `grafana.ini` file is as follows:

   ```
   # Example of Google Cloud KMS provider setup
   ;[security.encryption.googlekms.example-encryption-key]
   # Google Cloud KMS key ID
   key_id = 1234abcd-12ab-34cd-56ef-1234567890ab
   # Full path to a JSON file with a service account key
   credentials_file = ~/.config/gcloud/sample-project-credentials.json
   ```

7. Update the `[security]` section of the `grafana.ini` configuration file with the new Encryption Provider key that you created:

   ```
   [security]
   # previous encryption key, used for legacy alerts, decrypting existing secrets or used as default provider when external providers are not configured
   secret_key = AaaaAaaa
   # encryption provider key in the format <PROVIDER>.<KEY-NAME>
   encryption_provider = googlekms.example-encryption-key
   # list of configured key providers, space separated
   available_encryption_providers = googlekms.example-encryption-key
   ```

8. [Restart Grafana](/docs/grafana/latest/installation/restart-grafana/).

9. (Optional) From the command line and the root directory of Grafana Enterprise, re-encrypt all of the secrets within the Grafana database with the new key using the following command:

   `grafana cli admin secrets-migration re-encrypt`

   If you do not re-encrypt existing secrets, then they will remain encrypted by the previous encryption key. Users will still be able to access them.

   **> Note:** This process could take a few minutes to complete, depending on the number of secrets (such as data sources) in your database. Users might experience errors while this process is running, and alert notifications might not be sent.

   **> Note:** If you are updating this encryption key during the initial setup of Grafana before any data sources or dashboards have been created, then this step is not necessary because there are no secrets in Grafana to migrate.

# Encrypt database secrets using Hashicorp Vault

You can use an encryption key from Hashicorp Vault to encrypt secrets in the Grafana database.

**Prerequisites:**

- Permissions to manage Hashicorp Vault to enable secrets engines and issue tokens.
- Access to the Grafana [configuration](../../../configure-grafana/#configuration-file-location) file

1. [Enable the transit secrets engine](https://www.vaultproject.io/docs/secrets/transit#setup) in Hashicorp Vault.

2. [Create a named encryption key](https://www.vaultproject.io/docs/secrets/transit#setup).

3. [Create a periodic service token](https://learn.hashicorp.com/tutorials/vault/tokens#periodic-service-tokens).

4. From within Grafana, turn on envelope encryption.

5. Add your Hashicorp Vault details to the Grafana configuration file; depending on your operating system, is usually named `grafana.ini`:
   <br><br>a. Add a new section to the configuration file, with a name in the format of `[security.encryption.hashicorpvault.<KEY-NAME>]`, where `<KEY-NAME>` is any name that uniquely identifies this key among other provider keys.
   <br><br>b. Fill in the section with the following values:
   <br>
   - `token`: a periodic service token used to authenticate within Hashicorp Vault.
   - `url`: URL of the Hashicorp Vault server.
   - `transit_engine_path`: mount point of the transit engine.
   - `key_ring`: name of the encryption key.
   - `token_renewal_interval`: specifies how often to renew token; should be less than the `period` value of a periodic service token.

   An example of a Hashicorp Vault provider section in the `grafana.ini` file is as follows:

   ```
   # Example of Hashicorp Vault provider setup
   ;[security.encryption.hashicorpvault.example-encryption-key]
   # Token used to authenticate within Vault. We suggest to use periodic tokens: more on token types https://www.vaultproject.io/docs/concepts/tokens#service-tokens
   ;token =
   # Location of the Hashicorp Vault server
   ;url = http://localhost:8200
   # Mount point of the transit secret engine
   ;transit_engine_path = transit
   # Key ring name
   ;key_ring = grafana-encryption-key
   # Specifies how often to check if a token needs to be renewed, should be less than a token's period value
   token_renewal_interval = 5m
   ```

6. Update the `[security]` section of the `grafana.ini` configuration file with the new Encryption Provider key that you created:

   ```
   [security]
   # previous encryption key, used for legacy alerts, decrypting existing secrets or used as default provider when external providers are not configured
   secret_key = AaaaAaaa
   # encryption provider key in the format <PROVIDER>.<KEY-NAME>
   encryption_provider = hashicorpvault.example-encryption-key
   # list of configured key providers, space separated
   available_encryption_providers = hashicorpvault.example-encryption-key
   ```

7. [Restart Grafana](/docs/grafana/latest/installation/restart-grafana/).

8. (Optional) From the command line and the root directory of Grafana Enterprise, re-encrypt all of the secrets within the Grafana database with the new key using the following command:

   `grafana cli admin secrets-migration re-encrypt`

   If you do not re-encrypt existing secrets, then they will remain encrypted by the previous encryption key. Users will still be able to access them.

   **> Note:** This process could take a few minutes to complete, depending on the number of secrets (such as data sources) in your database. Users might experience errors while this process is running, and alert notifications might not be sent.

   **> Note:** If you are updating this encryption key during the initial setup of Grafana before any data sources or dashboards have been created, then this step is not necessary because there are no secrets in Grafana to migrate.

# Integrate Grafana with Hashicorp Vault

If you manage your secrets with [Hashicorp Vault](https://www.hashicorp.com/products/vault), you can use them for [Configuration](../../../configure-grafana/) and [Provisioning](../../../../administration/provisioning/).

{{< admonition type="note" >}}
Available in [Grafana Enterprise](../../../../introduction/grafana-enterprise/).
{{< /admonition >}}

{{< admonition type="note" >}}
If you have Grafana [set up for high availability](../../../set-up-for-high-availability/), then we advise not to use dynamic secrets for provisioning files.
Each Grafana instance is responsible for renewing its own leases. Your data source leases might expire when one of your Grafana servers shuts down.
{{< /admonition >}}

## Configuration

Before using Vault, you need to activate it by providing a URL, authentication method (currently only token),
and a token for your Vault service. Grafana automatically renews the service token if it is renewable and
set up with a limited lifetime.

If you're using short-lived leases, then you can also configure how often Grafana should renew the lease and for how long. We recommend keeping the defaults unless you run into problems.

```ini
[keystore.vault]
# Location of the Vault server
;url =
# Vault namespace if using Vault with multi-tenancy
;namespace =
# Method for authenticating towards Vault. Vault is inactive if this option is not set
# Possible values: token
;auth_method =
# Secret token to connect to Vault when auth_method is token
;token =
# Time between checking if there are any secrets which needs to be renewed.
;lease_renewal_interval = 5m
# Time until expiration for tokens which are renewed. Should have a value higher than lease_renewal_interval
;lease_renewal_expires_within = 15m
# New duration for renewed tokens. Vault may be configured to ignore this value and impose a stricter limit.
;lease_renewal_increment = 1h
```

Example for `vault server -dev`:

```ini
[keystore.vault]
url = http://127.0.0.1:8200 # HTTP should only be used for local testing
auth_method = token
token = s.sAZLyI0r7sFLMPq6MWtoOhAN # replace with your key
```

## Using the Vault expander

After you configure Vault, you must set the configuration or provisioning files you wish to
use Vault. Vault configuration is an extension of configuration's [variable expansion](../../../configure-grafana/#variable-expansion) and follows the
`$__vault{<argument>}` syntax.

The argument to Vault consists of three parts separated by a colon:

- The first part specifies which secrets engine should be used.
- The second part specifies which secret should be accessed.
- The third part specifies which field of that secret should be used.

For example, if you place a Key/Value secret for the Grafana admin user in _secret/grafana/admin_defaults_
the syntax for accessing its _password_ field would be `$__vault{kv:secret/grafana/admin_defaults:password}`.

### Secrets engines

Vault supports many secrets engines which represents different methods for storing or generating secrets when requested by an
authorized user. Grafana supports a subset of these which are most likely to be relevant for a Grafana installation.

#### Key/Value

Grafana supports Vault's [K/V version 2](https://www.vaultproject.io/docs/secrets/kv/kv-v2) storage engine which
is used to store and retrieve arbitrary secrets as `kv`.

```ini
$__vault{kv:secret/grafana/smtp:username}
```

#### Databases

The Vault [databases secrets engines](https://www.vaultproject.io/docs/secrets/databases) is a family of
secret engines which shares a similar syntax and grants the user dynamic access to a database.
You can use this both for setting up Grafana's own database access and for provisioning data sources.

```ini
$__vault{database:database/creds/grafana:username}
```

### Examples

The following examples show you how to set your [configuration](../../../configure-grafana/) or [provisioning](../../../../administration/provisioning/) files to use Vault to retrieve configuration values.

#### Configuration

The following is a partial example for using Vault to set up a Grafana configuration file's email and database credentials.
Refer to [Configuration](../../../configure-grafana/) for more information.

```ini
[smtp]
enabled = true
host = $__vault{kv:secret/grafana/smtp:hostname}:587
user = $__vault{kv:secret/grafana/smtp:username}
password = $__vault{kv:secret/grafana/smtp:password}

[database]
type = mysql
host = mysqlhost:3306
name = grafana
user = $__vault{database:database/creds/grafana:username}
password = $__vault{database:database/creds/grafana:password}
```

#### Provisioning

The following is a full examples of a provisioning YAML file setting up a MySQL data source using Vault's
database secrets engine.
Refer to [Provisioning](../../../../administration/provisioning/) for more information.

**provisioning/custom.yaml**

```ini
apiVersion: 1

datasources:
  - name: statistics
    type: mysql
    url: localhost:3306
    database: stats
    user: $__vault{database:database/creds/ro/stats:username}
    secureJsonData:
      password: $__vault{database:database/creds/ro/stats:password}
```

# Configure security hardening

Security hardening enables you to apply additional security, which can help stop certain vulnerabilities from being exploited by a malicious attacker.

{{< admonition type="note" >}}
These settings are available in the [grafana.ini configuration file](../../configure-grafana/#configuration-file-location). To apply changes to the configuration file, restart the Grafana server.
{{< /admonition >}}

## Additional security for cookies

If Grafana uses HTTPS, you can further secure the cookie that the system uses to authenticate access to the web UI. By applying additional security to the cookie, you might mitigate certain attacks that result from an attacker obtaining the cookie value.

{{< admonition type="note" >}}
Grafana must use HTTPS for the following configurations to work properly.
{{< /admonition >}}

### Add a secure attribute to cookies

To provide mitigation against some MITM attacks, add the `Secure` attribute to the cookie that is used to authenticate users. This attribute forces users only to send the cookie over a valid HTTPS secure connection.

Example:

```toml
[security]
# Set to true if you host Grafana behind HTTPS. The default value is false.
cookie_secure = true
```

### Add a SameSite attribute to cookies

To mitigate almost all CSRF-attacks, set the _cookie_samesite_ option to `strict`. This setting prevents clients from sending the cookie in requests that are made cross-site, but only from the site that creates the cookie.

Example:

```toml
[security]
# set cookie SameSite attribute. defaults to `lax`. can be set to "lax", "strict", "none" and "disabled"
cookie_samesite = strict
```

{{< admonition type="note" >}}
By setting the SameSite attribute to "strict," only the user clicks within a Grafana instance work. The default option, "lax," does not produce this behavior.

If you want to use OAuth/SAML for login, it is necessary to configure this attribute as `lax`.
{{< /admonition >}}

### Add a prefix to cookie names

You can further secure the cookie authentication by adding a [Cookie Prefix](https://googlechrome.github.io/samples/cookie-prefixes/). Cookies without a special prefix can be overwritten in a man-in-the-middle attack, even if the site uses HTTPS. A cookie prefix forces clients only to accept the cookie if certain criteria are met.
Add a prefix to the current cookie name with either `__Secure-` or `__Host-` where the latter provides additional protection by only allowing the cookie to be created from the host that sent the Set-Cookie header.

Example:

```toml
[auth]
# Login cookie name
login_cookie_name = __Host-grafana_session
```

## Security headers

Grafana includes a few additional headers that you can configure to help mitigate against certain attacks, such as XSS.

### Add a Content Security Policy

A content security policy (CSP) is an HTTP response header that controls how the web browser handles content, such as allowing inline scripts to execute or loading images from certain domains. The default CSP template is already configured to provide sufficient protection against some attacks. This makes it more difficult for attackers to execute arbitrary JavaScript if such a vulnerability is present.

Example:

```toml
[security]
# Enable adding the Content-Security-Policy header to your requests.
# CSP enables you to control the resources the user agent can load and helps prevent XSS attacks.
content_security_policy = true

# Set the Content Security Policy template that is used when the Content-Security-Policy header is added to your requests.
# $NONCE in the template includes a random nonce.
# $ROOT_PATH is server.root_url without the protocol.
content_security_policy_template = """script-src 'self' 'unsafe-eval' 'unsafe-inline' 'strict-dynamic' $NONCE;object-src 'none';font-src 'self';style-src 'self' 'unsafe-inline' blob:;img-src * data: blob:;base-uri 'self';connect-src 'self' grafana.com ws://$ROOT_PATH wss://$ROOT_PATH;manifest-src 'self';media-src 'none';form-action 'self';"""
```

### Enable trusted types

**Currently in development. [Trusted types](https://github.com/w3c/trusted-types/blob/main/explainer.md) is an experimental Javascript API with [limited browser support](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/trusted-types#browser_compatibility).**

Trusted types reduce the risk of DOM XSS by enforcing developers to sanitize strings that are used in injection sinks, such as setting `innerHTML` on an element. Furthermore, when enabling trusted types, these injection sinks need to go through a policy that will sanitize, or leave the string intact and return it as "safe". This provides some protection from client side injection vulnerabilities in third party libraries, such as jQuery, Angular and even third party plugins.

To enable trusted types in enforce mode, where injection sinks are automatically sanitized:

- Enable `content_security_policy` in the configuration.
- Add `require-trusted-types-for 'script'` to the `content_security_policy_template` in the configuration.

To enable trusted types in report mode, where inputs that have not been sanitized with trusted types will be logged to the console:

- Enable `content_security_policy_report_only` in the configuration.
- Add `require-trusted-types-for 'script'` to the `content_security_policy_report_only_template` in the configuration.

As this is a feature currently in development, things may break. If they do, or if you have any other feedback, feel free to [open an issue](https://github.com/grafana/grafana/issues/new/choose).

## Additional security hardening

The Grafana server has several built-in security features that you can opt-in to enhance security. This section describes additional techniques you can use to harden security.

### Hide the version number

If set to `true`, the Grafana server hides the running version number for unauthenticated users. Version numbers might reveal if you are running an outdated and vulnerable version of Grafana.

Example:

```toml
[auth.anonymous]
# mask the Grafana version number for unauthenticated users
hide_version = true
```

### Enable auth for metrics

By default, metrics from Grafana itself can be accessed without authentication. This can lead to information leakage.

Example:

```toml
[metrics]
# If both are set, basic auth will be required for the metrics endpoints
basic_auth_username =
basic_auth_password =
```

### Enforce domain verification

If set to `true`, the Grafana server redirects requests that have a Host-header value that is mismatched to the actual domain. This might help to mitigate some DNS rebinding attacks.

Example:

```toml
[server]
# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
enforce_domain = true
```

# Install Grafana on Debian or Ubuntu

This topic explains how to install Grafana dependencies, install Grafana on Linux Debian or Ubuntu, and start the Grafana server on your Debian or Ubuntu system.

There are multiple ways to install Grafana: using the Grafana Labs APT repository, by downloading a `.deb` package, or by downloading a binary `.tar.gz` file. Choose only one of the methods below that best suits your needs.

{{< admonition type="note" >}}
If you install via the `.deb` package or `.tar.gz` file, then you must manually update Grafana for each new version.
{{< /admonition >}}

The following video demonstrates how to install Grafana on Debian and Ubuntu as outlined in this document:

{{< youtube id="_Zk_XQSjF_Q" >}}

## Install from APT repository

If you install from the APT repository, Grafana automatically updates when you run `apt-get update`.

| Grafana Version           | Package            | Repository                            |
| ------------------------- | ------------------ | ------------------------------------- |
| Grafana Enterprise        | grafana-enterprise | `https://apt.grafana.com stable main` |
| Grafana Enterprise (Beta) | grafana-enterprise | `https://apt.grafana.com beta main`   |
| Grafana OSS               | grafana            | `https://apt.grafana.com stable main` |
| Grafana OSS (Beta)        | grafana            | `https://apt.grafana.com beta main`   |

{{< admonition type="note" >}}
Grafana Enterprise is the recommended and default edition. It is available for free and includes all the features of the OSS edition. You can also upgrade to the [full Enterprise feature set](/products/enterprise/?utm_source=grafana-install-page), which has support for [Enterprise plugins](/grafana/plugins/?enterprise=1&utcm_source=grafana-install-page).
{{< /admonition >}}

Complete the following steps to install Grafana from the APT repository:

1. Install the prerequisite packages:

   ```bash
   sudo apt-get install -y apt-transport-https wget gnupg
   ```

1. Import the GPG key:

   ```bash
   sudo mkdir -p /etc/apt/keyrings
   sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
   sudo chmod 644 /etc/apt/keyrings/grafana.asc
   ```

1. To add a repository for stable releases, run the following command:

   ```bash
   echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
   ```

1. To add a repository for beta releases, run the following command:

   ```bash
   echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
   ```

1. Run the following command to update the list of available packages:

   ```bash
   # Updates the list of available packages
   sudo apt-get update
   ```

1. To install Grafana OSS, run the following command:

   ```bash
   # Installs the latest OSS release:
   sudo apt-get install grafana
   ```

1. To install Grafana Enterprise, run the following command:

   ```bash
   # Installs the latest Enterprise release:
   sudo apt-get install grafana-enterprise
   ```

## Install Grafana using a deb package

If you install Grafana manually using the deb package, then you must manually update Grafana for each new version.

Complete the following steps to install Grafana using a deb package:

1. Navigate to the [Grafana download page](/grafana/download).
1. Select the Grafana version you want to install.
   - The most recent Grafana version is selected by default.
   - The **Version** field displays only tagged releases. If you want to install a nightly build, click **Nightly Builds** and then select a version.
1. Select an **Edition**.
   - **Enterprise:** This is the recommended version. It is functionally identical to the open source version, but includes features you can unlock with a license, if you so choose.
   - **Open Source:** This version is functionally identical to the Enterprise version, but you will need to download the Enterprise version if you want Enterprise features.
1. Depending on which system you are running, click the **Linux** or **ARM** tab on the [download page](/grafana/download).
1. Copy and paste the code from the [download page](/grafana/download) into your command line and run.

## Install Grafana as a standalone binary

Complete the following steps to install Grafana using the standalone binaries:

1. Navigate to the [Grafana download page](/grafana/download).
1. Select the Grafana version you want to install.
   - The most recent Grafana version is selected by default.
   - The **Version** field displays only tagged releases. If you want to install a nightly build, click **Nightly Builds** and then select a version.
1. Select an **Edition**.
   - **Enterprise:** This is the recommended version. It is functionally identical to the open source version but includes features you can unlock with a license if you so choose.
   - **Open Source:** This version is functionally identical to the Enterprise version, but you will need to download the Enterprise version if you want Enterprise features.
1. Depending on which system you are running, click the **Linux** or **ARM** tab on the [download page](/grafana/download).
1. Copy and paste the code from the [download page](/grafana/download) page into your command line and run.
1. Create a user account for Grafana on your system:

   ```shell
   sudo useradd -r -s /bin/false grafana
   ```

1. Move the unpacked binary to `/usr/local/grafana`:

   ```shell
   sudo mv <DOWNLOAD PATH> /usr/local/grafana
   ```

1. Change the owner of `/usr/local/grafana` to Grafana users:

   ```shell
   sudo chown -R grafana:users /usr/local/grafana
   ```

1. Create a Grafana server systemd unit file:

   ```shell
   sudo touch /etc/systemd/system/grafana-server.service
   ```

1. Add the following to the unit file in a text editor of your choice:

   ```ini
   [Unit]
   Description=Grafana Server
   After=network.target

   [Service]
   Type=simple
   User=grafana
   Group=users
   ExecStart=/usr/local/grafana/bin/grafana server --config=/usr/local/grafana/conf/grafana.ini --homepath=/usr/local/grafana
   Restart=on-failure

   [Install]
   WantedBy=multi-user.target
   ```

1. Use the binary to manually start the Grafana server:

   ```shell
   /usr/local/grafana/bin/grafana server --homepath /usr/local/grafana
   ```

   {{< admonition type="note" >}}
   Manually invoking the binary in this step automatically creates the `/usr/local/grafana/data` directory, which needs to be created and configured before the installation can be considered complete.
   {{< /admonition >}}

1. Press `CTRL+C` to stop the Grafana server.
1. Change the owner of `/usr/local/grafana` to Grafana users again to apply the ownership to the newly created `/usr/local/grafana/data` directory:

   ```shell
   sudo chown -R grafana:users /usr/local/grafana
   ```

1. [Configure the Grafana server to start at boot time using systemd](https://grafana.com/docs/grafana/latest/setup-grafana/start-restart-grafana/#configure-the-grafana-server-to-start-at-boot-using-systemd).

## Uninstall on Debian or Ubuntu

Complete any of the following steps to uninstall Grafana.

To uninstall Grafana, run the following commands in a terminal window:

1. If you configured Grafana to run with systemd, stop the systemd service for Grafana server:

   ```shell
   sudo systemctl stop grafana-server
   ```

1. If you configured Grafana to run with init.d, stop the init.d service for Grafana server:

   ```shell
   sudo service grafana-server stop
   ```

1. To uninstall Grafana OSS:

   ```shell
   sudo apt-get remove grafana
   ```

1. To uninstall Grafana Enterprise:

   ```shell
   sudo apt-get remove grafana-enterprise
   ```

1. Optional: To remove the Grafana repository:

   ```bash
   sudo rm -i /etc/apt/sources.list.d/grafana.list
   ```

## Next steps

- [Start the Grafana server](../../start-restart-grafana/)

# Run Grafana Docker image

{{< admonition type="caution" >}}
Starting with Grafana release `12.4.0` , the `grafana/grafana-oss` Docker Hub repository will no longer be updated.
Instead, we encourage you to use the `grafana/grafana` Docker Hub repository. These two repositories have the same Grafana OSS docker images.
{{< /admonition >}}

# Run Grafana Docker image

This topic guides you through installing Grafana via the official Docker images. Specifically, it covers running Grafana via the Docker command line interface (CLI) and docker-compose.

{{< youtube id="FlDfcMbSLXs" start="703">}}

Grafana Docker images come in two editions:

- **Grafana Enterprise**: `grafana/grafana-enterprise`
- **Grafana Open Source**: `grafana/grafana`

> **Note:** The recommended and default edition of Grafana is Grafana Enterprise. It is free and includes all the features of the OSS edition. Additionally, you have the option to upgrade to the [full Enterprise feature set](/products/enterprise/?utm_source=grafana-install-page), which includes support for [Enterprise plugins](/grafana/plugins/?enterprise=1&utcm_source=grafana-install-page).

The default images for Grafana are created using the Alpine Linux project and can be found in the Alpine official image. For instructions on configuring a Docker image for Grafana, refer to [Configure a Grafana Docker image](../../configure-docker/).

## Run Grafana via Docker CLI

This section shows you how to run Grafana using the Docker CLI.

> **Note:** If you are on a Linux system (for example, Debian or Ubuntu), you might need to add `sudo` before the command or add your user to the `docker` group. For more information, refer to [Linux post-installation steps for Docker Engine](https://docs.docker.com/engine/install/linux-postinstall/).

To run the latest stable version of Grafana, run the following command:

```bash
docker run -d -p 3000:3000 --name=grafana grafana/grafana-enterprise
```

Where:

- [`docker run`](https://docs.docker.com/engine/reference/commandline/run/) is a Docker CLI command that runs a new container from an image
- `-d` (`--detach`) runs the container in the background
- `-p <host-port>:<container-port>` (`--publish`) publish a container's port(s) to the host, allowing you to reach the container's port via a host port. In this case, we can reach the container's port `3000` via the host's port `3000`
- `--name` assign a logical name to the container (e.g. `grafana`). This allows you to refer to the container by name instead of by ID.
- `grafana/grafana-enterprise` is the image to run

### Stop the Grafana container

To stop the Grafana container, run the following command:

```bash
# The `docker ps` command shows the processes running in Docker
docker ps

# This will display a list of containers that looks like the following:
CONTAINER ID   IMAGE  COMMAND   CREATED  STATUS   PORTS    NAMES
cd48d3994968   grafana/grafana-enterprise   "/run.sh"   8 seconds ago   Up 7 seconds   0.0.0.0:3000->3000/tcp   grafana

# To stop the grafana container run the command
# docker stop CONTAINER-ID or use
# docker stop NAME, which is `grafana` as previously defined
docker stop grafana
```

### Save your Grafana data

By default, Grafana uses an embedded SQLite version 3 database to store configuration, users, dashboards, and other data. When you run Docker images as containers, changes to these Grafana data are written to the filesystem within the container, which will only persist for as long as the container exists. If you stop and remove the container, any filesystem changes (i.e. the Grafana data) will be discarded. To avoid losing your data, you can set up persistent storage using [Docker volumes](https://docs.docker.com/storage/volumes/) or [bind mounts](https://docs.docker.com/storage/bind-mounts/) for your container.

> **Note:** Though both methods are similar, there is a slight difference. If you want your storage to be fully managed by Docker and accessed only through Docker containers and the Docker CLI, you should choose to use persistent storage. However, if you need full control of the storage and want to allow other processes besides Docker to access or modify the storage layer, then bind mounts is the right choice for your environment.

#### Use Docker volumes (recommended)

Use Docker volumes when you want the Docker Engine to manage the storage volume.

To use Docker volumes for persistent storage, complete the following steps:

1. Create a Docker volume to be used by the Grafana container, giving it a descriptive name (e.g. `grafana-storage`). Run the following command:

   ```bash
   # create a persistent volume for your data
   docker volume create grafana-storage

   # verify that the volume was created correctly
   # you should see some JSON output
   docker volume inspect grafana-storage
   ```

1. Start the Grafana container by running the following command:
   ```bash
   # start grafana
   docker run -d -p 3000:3000 --name=grafana \
     --volume grafana-storage:/var/lib/grafana \
     grafana/grafana-enterprise
   ```

#### Use bind mounts

If you plan to use directories on your host for the database or configuration when running Grafana in Docker, you must start the container with a user with permission to access and write to the directory you map.

To use bind mounts, run the following command:

```bash
# create a directory for your data
mkdir data

# start grafana with your user id and using the data directory
docker run -d -p 3000:3000 --name=grafana \
  --user "$(id -u)" \
  --volume "$PWD/data:/var/lib/grafana" \
  grafana/grafana-enterprise
```

### Use environment variables to configure Grafana

Grafana supports specifying custom configuration settings using [environment variables](../../configure-grafana/#override-configuration-with-environment-variables).

```bash
# enable debug logs

docker run -d -p 3000:3000 --name=grafana \
  -e "GF_LOG_LEVEL=debug" \
  grafana/grafana-enterprise
```

## Install plugins in the Docker container

You can install plugins in Grafana from the official and community [plugins page](/grafana/plugins) or by using a custom URL to install a private plugin. These plugins allow you to add new visualization types, data sources, and applications to help you better visualize your data.

Grafana currently supports three types of plugins: panel, data source, and app. For more information on managing plugins, refer to [Plugin Management](../../../administration/plugin-management/).

To install plugins in the Docker container, complete the following steps:

1. Pass the plugins you want to be installed to Docker with the `GF_PLUGINS_PREINSTALL` environment variable as a comma-separated list.

   This starts a background process that installs the list of plugins while Grafana server starts.

   For example:

   ```bash
   docker run -d -p 3000:3000 --name=grafana \
     -e "GF_PLUGINS_PREINSTALL=grafana-clock-panel, yesoreyeram-infinity-datasource" \
     grafana/grafana-enterprise
   ```

1. To specify the version of a plugin, add the version number to the `GF_PLUGINS_PREINSTALL` environment variable.

   For example:

   ```bash
   docker run -d -p 3000:3000 --name=grafana \
     -e "GF_PLUGINS_PREINSTALL=grafana-clock-panel@1.0.1" \
     grafana/grafana-enterprise
   ```

   > **Note:** If you do not specify a version number, the latest version is used.

1. To install a plugin from a custom URL, use the following convention to specify the URL: `<plugin ID>@[<plugin version>]@<url to plugin zip>`.

   For example:

   ```bash
   docker run -d -p 3000:3000 --name=grafana \
     -e "GF_PLUGINS_PREINSTALL=custom-plugin@@https://github.com/VolkovLabs/custom-plugin.zip" \
     grafana/grafana-enterprise
   ```

## Example

The following example runs the latest stable version of Grafana, listening on port 3000, with the container named `grafana`, persistent storage in the `grafana-storage` docker volume, the server root URL set, and the official [clock panel](/grafana/plugins/grafana-clock-panel) plugin installed.

```bash
# create a persistent volume for your data
docker volume create grafana-storage

# start grafana by using the above persistent storage
# and defining environment variables

docker run -d -p 3000:3000 --name=grafana \
  --volume grafana-storage:/var/lib/grafana \
  -e "GF_SERVER_ROOT_URL=http://my.grafana.server/" \
  -e "GF_PLUGINS_PREINSTALL=grafana-clock-panel" \
  grafana/grafana-enterprise
```

## Run Grafana via Docker Compose

Docker Compose is a software tool that makes it easy to define and share applications that consist of multiple containers. It works by using a YAML file, usually called `docker-compose.yaml`, which lists all the services that make up the application. You can start the containers in the correct order with a single command, and with another command, you can shut them down. For more information about the benefits of using Docker Compose and how to use it refer to [Use Docker Compose](https://docs.docker.com/get-started/08_using_compose/).

### Before you begin

To run Grafana via Docker Compose, install the compose tool on your machine. To determine if the compose tool is available, run the following command:

```bash
docker compose version
```

If the compose tool is unavailable, refer to [Install Docker Compose](https://docs.docker.com/compose/install/).

### Run the latest stable version of Grafana

This section shows you how to run Grafana using Docker Compose. The examples in this section use Compose version 3. For more information about compatibility, refer to [Compose and Docker compatibility matrix](https://docs.docker.com/compose/compose-file/compose-file-v3/).

> **Note:** If you are on a Linux system (for example, Debian or Ubuntu), you might need to add `sudo` before the command or add your user to the `docker` group. For more information, refer to [Linux post-installation steps for Docker Engine](https://docs.docker.com/engine/install/linux-postinstall/).

To run the latest stable version of Grafana using Docker Compose, complete the following steps:

1. Create a `docker-compose.yaml` file.

   ```bash
   # first go into the directory where you have created this docker-compose.yaml file
   cd /path/to/docker-compose-directory

   # now create the docker-compose.yaml file
   touch docker-compose.yaml
   ```

1. Now, add the following code into the `docker-compose.yaml` file.

   For example:

   ```yaml
   services:
     grafana:
       image: grafana/grafana-enterprise
       container_name: grafana
       restart: unless-stopped
       ports:
         - '3000:3000'
   ```

1. To run `docker-compose.yaml`, run the following command:

   ```bash
   # start the grafana container
   docker compose up -d
   ```

   Where:

   d = detached mode

   up = to bring the container up and running

To determine that Grafana is running, open a browser window and type `IP_ADDRESS:3000`. The sign in screen should appear.

### Stop the Grafana container

To stop the Grafana container, run the following command:

```bash
docker compose down
```

> **Note:** For more information about using Docker Compose commands, refer to [docker compose](https://docs.docker.com/engine/reference/commandline/compose/).

### Save your Grafana data

By default, Grafana uses an embedded SQLite version 3 database to store configuration, users, dashboards, and other data. When you run Docker images as containers, changes to these Grafana data are written to the filesystem within the container, which will only persist for as long as the container exists. If you stop and remove the container, any filesystem changes (i.e. the Grafana data) will be discarded. To avoid losing your data, you can set up persistent storage using [Docker volumes](https://docs.docker.com/storage/volumes/) or [bind mounts](https://docs.docker.com/storage/bind-mounts/) for your container.

#### Use Docker volumes (recommended)

Use Docker volumes when you want the Docker Engine to manage the storage volume.

To use Docker volumes for persistent storage, complete the following steps:

1. Create a `docker-compose.yaml` file

   ```bash
   # first go into the directory where you have created this docker-compose.yaml file
   cd /path/to/docker-compose-directory

   # now create the docker-compose.yaml file
   touch docker-compose.yaml
   ```

1. Add the following code into the `docker-compose.yaml` file.

   ```yaml
   services:
     grafana:
       image: grafana/grafana-enterprise
       container_name: grafana
       restart: unless-stopped
       ports:
         - '3000:3000'
       volumes:
         - grafana-storage:/var/lib/grafana
   volumes:
     grafana-storage: {}
   ```

1. Save the file and run the following command:

   ```bash
   docker compose up -d
   ```

#### Use bind mounts

If you plan to use directories on your host for the database or configuration when running Grafana in Docker, you must start the container with a user that has the permission to access and write to the directory you map.

To use bind mounts, complete the following steps:

1. Create a `docker-compose.yaml` file

   ```bash
   # first go into the directory where you have created this docker-compose.yaml file
   cd /path/to/docker-compose-directory

   # now create the docker-compose.yaml file
   touch docker-compose.yaml
   ```

1. Create the directory where you will be mounting your data, in this case is `/data` e.g. in your current working directory:

   ```bash
   mkdir $PWD/data
   ```

1. Now, add the following code into the `docker-compose.yaml` file.

   ```yaml
   services:
     grafana:
       image: grafana/grafana-enterprise
       container_name: grafana
       restart: unless-stopped
       # if you are running as root then set it to 0
       # else find the right id with the id -u command
       user: '0'
       ports:
         - '3000:3000'
       # adding the mount volume point which we create earlier
       volumes:
         - '$PWD/data:/var/lib/grafana'
   ```

1. Save the file and run the following command:

   ```bash
   docker compose up -d
   ```

### Example

The following example runs the latest stable version of Grafana, listening on port 3000, with the container named `grafana`, persistent storage in the `grafana-storage` docker volume, the server root URL set, and the official [clock panel](/grafana/plugins/grafana-clock-panel/) plugin installed.

```yaml
services:
  grafana:
    image: grafana/grafana-enterprise
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SERVER_ROOT_URL=http://my.grafana.server/
      - GF_PLUGINS_PREINSTALL=grafana-clock-panel
    ports:
      - '3000:3000'
    volumes:
      - 'grafana_storage:/var/lib/grafana'
volumes:
  grafana_storage: {}
```

{{< admonition type="note" >}}
If you want to specify the version of a plugin, add the version number to the `GF_PLUGINS_PREINSTALL` environment variable. For example: `-e "GF_PLUGINS_PREINSTALL=grafana-clock-panel@1.0.1,yesoreyeram-infinity-datasource@3.8.0"`. If you do not specify a version number, the latest version is used.
{{< /admonition >}}

## Next steps

Refer to the [Getting Started](../../../getting-started/build-first-dashboard/) guide for information about logging in, setting up data sources, and so on.

## Configure Docker image

Refer to [Configure a Grafana Docker image](../../configure-docker/) page for details on options for customizing your environment, logging, database, and so on.

## Configure Grafana

Refer to the [Configuration](../../configure-grafana/) page for details on options for customizing your environment, logging, database, and so on.

# Deploy Grafana using Helm Charts

This topic includes instructions for installing and running Grafana on Kubernetes using Helm Charts.

[Helm](https://helm.sh/) is an open source command line tool used for managing Kubernetes applications. It is a graduate project in the [CNCF Landscape](https://www.cncf.io/projects/helm/).

{{< admonition type="note" >}}
The Grafana open source community offers Helm Charts for running it on Kubernetes. Please be aware that the code is provided without any warranties. If you encounter any problems, you can report them to the [Community-maintained GitHub repository](https://github.com/grafana-community/helm-charts).
{{< /admonition >}}

Watch this video to learn more about installing Grafana using Helm Charts: {{< youtube id="sgYrEleW24E">}}

## Before you begin

To install Grafana using Helm, ensure you have completed the following:

- Install a Kubernetes server on your machine. For information about installing Kubernetes, refer to [Install Kubernetes](https://kubernetes.io/docs/setup/).
- Install the latest stable version of Helm. For information on installing Helm, refer to [Install Helm](https://helm.sh/docs/intro/install/).

## Install Grafana using Helm

When you install Grafana using Helm, you complete the following tasks:

1. Set up the Grafana Helm repository, which provides a space in which you will install Grafana.

1. Deploy Grafana using Helm, which installs Grafana into a namespace.

1. Accessing Grafana, which provides steps to sign into Grafana.

### Set up the Grafana Helm repository

To set up the Grafana Helm repository so that you download the correct Grafana Helm charts on your machine, complete the following steps:

1. To add the Grafana repository, use the following command syntax:

   `helm repo add <DESIRED-NAME> <HELM-REPO-URL>`

   The following example adds the `grafana` Helm repository.

   ```bash
   helm repo add grafana-community https://grafana-community.github.io/helm-charts
   ```

1. Run the following command to verify the repository was added:

   ```bash
   helm repo list
   ```

   After you add the repository, you should see an output similar to the following:

   ```bash
   NAME              URL
   grafana-community https://grafana-community.github.io/helm-charts
   ```

1. Run the following command to update the repository to download the latest Grafana Helm charts:

   ```bash
   helm repo update
   ```

### Deploy the Grafana Helm charts

After you have set up the Grafana Helm repository, you can start to deploy it on your Kubernetes cluster.

When you deploy Grafana Helm charts, use a separate namespace instead of relying on the default namespace. The default namespace might already have other applications running, which can lead to conflicts and other potential issues.

When you create a new namespace in Kubernetes, you can better organize, allocate, and manage cluster resources. For more information, refer to [Namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).

1. To create a namespace, run the following command:

   ```bash
   kubectl create namespace monitoring
   ```

   You will see an output similar to this, which means that the namespace has been successfully created:

   ```bash
   namespace/monitoring created
   ```

1. Search for the official `grafana-community/grafana` repository using the command:

   `helm search repo <repo-name/package-name>`

   For example, the following command provides a list of the Grafana Helm Charts from which you will install the latest version of the Grafana chart.

   ```bash
   helm search repo grafana-community/grafana
   ```

1. Run the following command to deploy the Grafana Helm Chart inside your namespace.

   ```bash
   helm install my-grafana grafana-community/grafana --namespace monitoring
   ```

   Where:
   - `helm install`: Installs the chart by deploying it on the Kubernetes cluster
   - `my-grafana`: The logical chart name that you provided
   - `grafana-community/grafana`: The repository and package name to install
   - `--namespace`: The Kubernetes namespace (i.e. `monitoring`) where you want to deploy the chart

1. To verify the deployment status, run the following command and verify that `deployed` appears in the **STATUS** column:

   ```bash
   helm list -n monitoring
   ```

   You should see an output similar to the following:

   ```bash
   NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART          APP VERSION
   my-grafana      monitoring      1               2024-01-13 23:06:42.737989554 +0000 UTC deployed        grafana-6.59.0 10.1.0
   ```

1. To check the overall status of all the objects in the namespace, run the following command:

   ```bash
   kubectl get all -n monitoring
   ```

   If you encounter errors or warnings in the **STATUS** column, check the logs and refer to the Troubleshooting section of this documentation.

### Access Grafana

This section describes the steps you must complete to access Grafana via web browser.

1. Run the following `helm get notes` command:

   ```bash
   helm get notes my-grafana -n monitoring
   ```

   This command will print out the chart notes. You will the output `NOTES` that provide the complete instructions about:
   - How to decode the login password for the Grafana admin account
   - Access Grafana service to the web browser

1. To get the Grafana admin password, run the command as follows:

   ```bash
   kubectl get secret --namespace monitoring my-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
   ```

   It will give you a decoded `base64` string output which is the password for the admin account.

1. Save the decoded password to a file on your machine.

1. To access Grafana service on the web browser, run the following command:

   ```bash
   export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=my-grafana" -o jsonpath="{.items[0].metadata.name}")
   ```

   The above command will export a shell variable named `POD_NAME` that will save the complete name of the pod which got deployed.

1. Run the following port forwarding command to direct the Grafana pod to listen to port `3000`:

   ```bash
   kubectl --namespace monitoring port-forward $POD_NAME 3000
   ```

   For more information about port-forwarding, refer to [Use Port Forwarding to Access Applications in a Cluster](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/).

1. Navigate to `127.0.0.1:3000` in your browser.

1. The Grafana sign-in page appears.

1. To sign in, enter `admin` for the username.

1. For the password paste it which you have saved to a file after decoding it earlier.

## Customize Grafana default configuration

Helm is a popular package manager for Kubernetes. It bundles Kubernetes resource manifests to be re-used across different environments. These manifests are written in a templating language, allowing you to provide configuration values via `values.yaml` file, or in-line using Helm, to replace the placeholders in the manifest where these configurations should reside.

The `values.yaml` file allows you to customize the chart's configuration by specifying values for various parameters such as image versions, resource limits, service configurations, etc.

By modifying the values in the `values.yaml` file, you can tailor the deployment of a Helm chart to your specific requirements by using the helm install or upgrade commands. For more information about configuring Helm, refer to [Values Files](https://helm.sh/docs/chart_template_guide/values_files/).

### Download the values.yaml file

In order to make any configuration changes, download the `values.yaml` file from the Grafana Helm Charts repository:

https://github.com/grafana-community/helm-charts/blob/main/charts/grafana/values.yaml

{{< admonition type="note" >}}
Depending on your use case requirements, you can use a single YAML file that contains your configuration changes or you can create multiple YAML files.
{{< /admonition >}}

### Enable persistent storage **(recommended)**

By default, persistent storage is disabled, which means that Grafana uses ephemeral storage, and all data will be stored within the container's file system. This data will be lost if the container is stopped, restarted, or if the container crashes.

It is highly recommended that you enable persistent storage in Grafana Helm charts if you want to ensure that your data persists and is not lost in case of container restarts or failures.

Enabling persistent storage in Grafana Helm charts ensures a reliable solution for running Grafana in production environments.

To enable the persistent storage in the Grafana Helm charts, complete the following steps:

1. Open the `values.yaml` file in your favorite editor.

1. Edit the values and under the section of `persistence`, change the `enable` flag from `false` to `true`

   ```yaml
   .......
   ............
   ......
   persistence:
     type: pvc
     enabled: true
     # storageClassName: default
   .......
   ............
   ......
   ```

1. Run the following `helm upgrade` command by specifying the `values.yaml` file to make the changes take effect:

   ```bash
   helm upgrade my-grafana grafana-community/grafana -f values.yaml -n monitoring
   ```

The PVC will now store all your data such as dashboards, data sources, and so on.

### Install plugins (e.g. Zabbix app, Clock panel, etc.)

You can install plugins in Grafana from the official and community [plugins page](https://grafana.com/grafana/plugins). These plugins allow you to add new visualization types, data sources, and applications to help you better visualize your data.

Grafana currently supports three types of plugins: panel, data source, and app. For more information on managing plugins, refer to [Plugin Management](https://grafana.com/docs/grafana/latest/administration/plugin-management/).

To install plugins in the Grafana Helm Charts, complete the following steps:

1. Open the `values.yaml` file in your favorite editor.

1. Find the line that says `plugins:` and under that section, define the plugins that you want to install.

   ```yaml
   .......
   ............
   ......
   plugins:
   # here we are installing two plugins, make sure to keep the indentation correct as written here.

   - alexanderzobnin-zabbix-app
   - grafana-clock-panel
   .......
   ............
   ......
   ```

1. Save the changes and use the `helm upgrade` command to get these plugins installed:

   ```bash
   helm upgrade my-grafana grafana-community/grafana -f values.yaml -n monitoring
   ```

1. Navigate to `127.0.0.1:3000` in your browser.

1. Login with admin credentials when the Grafana sign-in page appears.

1. Navigate to UI -> Administration -> Plugins

1. Search for the above plugins and they should be marked as installed.

### Configure a Private CA (Certificate Authority)

In many enterprise networks, TLS certificates are issued by a private certificate authority and are not trusted by default (using the provided OS trust chain).

If your Grafana instance needs to interact with services exposing certificates issued by these private CAs, then you need to ensure Grafana trusts the root certificate.

You might need to configure this if you:

- have plugins that require connectivity to other self hosted systems. For example, if you've installed the Grafana Enterprise Metrics, Logs, or Traces (GEM, GEL, GET) plugins, and your GEM (or GEL/GET) cluster is using a private certificate.
- want to connect to data sources which are listening on HTTPS with a private certificate.
- are using a backend database for persistence, or caching service that uses private certificates for encryption in transit.

In some cases you can specify a self-signed certificate within Grafana (such as in some data sources), or choose to skip TLS certificate validation (this is not recommended unless absolutely necessary).

A simple solution which should work across your entire instance (plugins, data sources, and backend connections) is to add your self-signed CA certificate to your Kubernetes deployment.

1. Create a ConfigMap containing the certificate, and deploy it to your Kubernetes cluster

   ```yaml
   # grafana-ca-configmap.yaml
   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: grafana-ca-cert
   data:
     ca.pem: |
       -----BEGIN CERTIFICATE-----
       (rest of the CA cert)
       -----END CERTIFICATE-----
   ```

   ```bash
   kubectl apply --filename grafana-ca-configmap.yaml --namespace monitoring
   ```

1. Open the Helm `values.yaml` file in your favorite editor.

1. Find the line that says `extraConfigmapMounts:` and under that section, specify the additional ConfigMap that you want to mount.

   ```yaml
   .......
   ............
   ......
   extraConfigmapMounts:
      - name: ca-certs-configmap
        mountPath: /etc/ssl/certs/ca.pem
        subPath: ca.pem
        configMap: grafana-ca-cert
        readOnly: true
   .......
   ............
   ......
   ```

1. Save the changes and use the `helm upgrade` command to update your Grafana deployment and mount the new ConfigMap:

   ```bash
   helm upgrade my-grafana grafana-community/grafana --values values.yaml --namespace monitoring
   ```

## Troubleshooting

This section includes troubleshooting tips you might find helpful when deploying Grafana on Kubernetes via Helm.

### Collect logs

It is important to view the Grafana server logs while troubleshooting any issues.

To check the Grafana logs, run the following command:

```bash
# dump Pod logs for a Deployment (single-container case)

kubectl logs --namespace=monitoring deploy/my-grafana
```

If you have multiple containers running in the deployment, run the following command to obtain the logs only for the Grafana deployment:

```bash
# dump Pod logs for a Deployment (multi-container case)

kubectl logs --namespace=monitoring deploy/grafana -c my-grafana
```

For more information about accessing Kubernetes application logs, refer to [Pods](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#interacting-with-running-pods) and [Deployments](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#interacting-with-deployments-and-services).

### Increase log levels

By default, the Grafana log level is set to `info`, but you can increase it to `debug` mode to fetch information needed to diagnose and troubleshoot a problem. For more information about Grafana log levels, refer to [Configuring logs](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana#log).

To increase log level to `debug` mode, use the following steps:

1. Open the `values.yaml` file in your favorite editor and search for the string `grafana.ini` and there you will find a section about log mode.

1. Add level: `debug` just below the line `mode: console`

   ```yaml
   # This is the values.yaml file
      .....
   .......
   ....
   grafana.ini:
   paths:
      data: /var/lib/grafana/
      .....
   .......
   ....
      mode: console
      level: debug
   ```

   Make sure to keep the indentation level the same otherwise it will not work.

1. Now to apply this, run the `helm upgrade` command as follows:

   ```bash
   helm upgrade my-grafana grafana-community/grafana -f values.yaml -n monitoring
   ```

1. To verify it, access the Grafana UI in the browser using the provided `IP:Port`. The Grafana sign-in page appears.

1. To sign in to Grafana, enter `admin` for the username and paste the password which was decoded earlier. Navigate to Server Admin > Settings and then search for log. You should see the level to `debug` mode.

### Reset Grafana admin secrets (login credentials)

By default the login credentials for the super admin account are generated via `secrets`. However, this can be changed easily. To achieve this, use the following steps:

1. Edit the `values.yaml` file and search for the string `adminPassword`. There you can define a new password:

   ```yaml
   # Administrator credentials when not using an existing secret (see below)
   adminUser: admin
   adminPassword: admin
   ```

1. Then use the `helm upgrade` command as follows:

   ```bash
   helm upgrade my-grafana grafana-community/grafana -f values.yaml -n monitoring
   ```

   This command will now make your super admin login credentials as `admin` for both username and password.

1. To verify it, sign in to Grafana, enter `admin` for both username and password. You should be able to login as super admin.

## Uninstall the Grafana deployment

To uninstall the Grafana deployment, run the command:

`helm uninstall <RELEASE-NAME> <NAMESPACE-NAME>`

```bash
helm uninstall my-grafana -n monitoring
```

This deletes all of the objects from the given namespace monitoring.

If you want to delete the namespace `monitoring`, then run the command:

```bash
kubectl delete namespace monitoring
```

# Deploy Grafana on Kubernetes

On this page, you will find instructions for installing and running Grafana on Kubernetes using Kubernetes manifests for the setup. If Helm is your preferred option, refer to [Grafana Helm community charts](https://github.com/grafana/helm-charts).

Watch this video to learn more about installing Grafana on Kubernetes: {{< youtube id="DEv5wtZxNCk" start="1872">}}

## Before you begin

To follow this guide:

- You need the latest version of [Kubernetes](https://kubernetes.io/) running either locally or remotely on a public or private cloud.

- If you plan to use it in a local environment, you can use various Kubernetes options such as [minikube](https://minikube.sigs.k8s.io/docs/), [kind](https://kind.sigs.k8s.io/), [Docker Desktop](https://docs.docker.com/desktop/kubernetes/), and others.

- If you plan to use Kubernetes in a production setting, it's recommended to utilize managed cloud services like [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine), [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/), or [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/products/kubernetes-service/).

## System requirements

This section provides minimum hardware and software requirements.

### Minimum Hardware Requirements

- Disk space: 1 GB
- Memory: 750 MiB (approx 750 MB)
- CPU: 250m (approx 0.25 cores)

### Supported databases

For a list of supported databases, refer to [supported databases](/docs/grafana/latest/setup-grafana/installation#supported-databases).

### Supported web browsers

For a list of support web browsers, refer to [supported web browsers](/docs/grafana/latest/setup-grafana/installation#supported-web-browsers).

{{< admonition type="note" >}}
Enable port `3000` in your network environment, as this is the Grafana default port.
{{< /admonition >}}

## Deploy Grafana OSS on Kubernetes

This section explains how to install Grafana OSS using Kubernetes.
{{< admonition type="note" >}}
If you want to install Grafana Enterprise on Kubernetes, refer to [Deploy Grafana Enterprise on Kubernetes](#deploy-grafana-enterprise-on-kubernetes).
{{< /admonition >}}

If you deploy an application in Kubernetes, it will use the default namespace which may already have other applications running. This can result in conflicts and other issues.

It is recommended to create a new namespace in Kubernetes to better manage, organize, allocate, and manage cluster resources. For more information about Namespaces, refer to the official [Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).

1. To create a namespace, run the following command:

   ```bash
   kubectl create namespace my-grafana
   ```

   In this example, the namespace is `my-grafana`

1. To verify and view the newly created namespace, run the following command:

   ```bash
   kubectl get namespace my-grafana
   ```

   The output of the command provides more information about the newly created namespace.

1. Create a YAML manifest file named `grafana.yaml`. This file will contain the necessary code for deployment.

   ```bash
   touch grafana.yaml
   ```

   In the next step you define the following three objects in the YAML file.

   | Object                        | Description                                                                                                                   |
   | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
   | Persistent Volume Claim (PVC) | This object stores the data.                                                                                                  |
   | Service                       | This object provides network access to the Pod defined in the deployment.                                                     |
   | Deployment                    | This object is responsible for creating the pods, ensuring they stay up to date, and managing Replicaset and Rolling updates. |

1. Copy and paste the following contents and save it in the `grafana.yaml` file.

   ```yaml
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: grafana-pvc
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: grafana
     name: grafana
   spec:
     selector:
       matchLabels:
         app: grafana
     template:
       metadata:
         labels:
           app: grafana
       spec:
         securityContext:
           fsGroup: 472
           supplementalGroups:
             - 0
         containers:
           - name: grafana
             image: grafana/grafana:latest
             imagePullPolicy: IfNotPresent
             ports:
               - containerPort: 3000
                 name: http-grafana
                 protocol: TCP
             readinessProbe:
               failureThreshold: 3
               httpGet:
                 path: /robots.txt
                 port: 3000
                 scheme: HTTP
               initialDelaySeconds: 10
               periodSeconds: 30
               successThreshold: 1
               timeoutSeconds: 2
             livenessProbe:
               failureThreshold: 3
               initialDelaySeconds: 30
               periodSeconds: 10
               successThreshold: 1
               tcpSocket:
                 port: 3000
               timeoutSeconds: 1
             resources:
               requests:
                 cpu: 250m
                 memory: 750Mi
             volumeMounts:
               - mountPath: /var/lib/grafana
                 name: grafana-pv
         volumes:
           - name: grafana-pv
             persistentVolumeClaim:
               claimName: grafana-pvc
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: grafana
   spec:
     ports:
       - port: 3000
         protocol: TCP
         targetPort: http-grafana
     selector:
       app: grafana
     sessionAffinity: None
     type: LoadBalancer
   ```

1. Run the following command to send the manifest to the Kubernetes API server:

   ```bash
   kubectl apply -f grafana.yaml --namespace=my-grafana
   ```

   This command creates the PVC, Deployment, and Service objects.

1. Complete the following steps to verify the deployment status of each object.

   a. For PVC, run the following command:

   ```bash
   kubectl get pvc --namespace=my-grafana -o wide
   ```

   b. For Deployment, run the following command:

   ```bash
   kubectl get deployments --namespace=my-grafana -o wide
   ```

   c. For Service, run the following command:

   ```bash
   kubectl get svc --namespace=my-grafana -o wide
   ```

## Access Grafana on Managed K8s Providers

In this task, you access Grafana deployed on a Managed Kubernetes provider using a web browser. Accessing Grafana via a web browser is straightforward if it is deployed on a Managed Kubernetes Provider as it uses the cloud provider’s **LoadBalancer** to which the external load balancer routes are automatically created.

1. Run the following command to obtain the deployment information:

   ```bash
   kubectl get all --namespace=my-grafana
   ```

   The output returned should look similar to the following:

   ```bash
   NAME                           READY   STATUS    RESTARTS   AGE
   pod/grafana-69946c9bd6-kwjb6   1/1     Running   0          7m27s

   NAME              TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)          AGE
   service/grafana   LoadBalancer   10.5.243.226   1.120.130.330   3000:31171/TCP   7m27s

   NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/grafana   1/1     1            1           7m29s

   NAME                                 DESIRED   CURRENT   READY   AGE
   replicaset.apps/grafana-69946c9bd6   1         1         1       7m30s
   ```

1. Identify the **EXTERNAL-IP** value in the output and type it into your browser.

   The Grafana sign-in page appears.

1. To sign in, enter `admin` for both the username and password.

1. If you do not see the EXTERNAL-IP, complete the following steps:

   a. Run the following command to do a port-forwarding of the Grafana service on port `3000`.

   ```bash
   kubectl port-forward service/grafana 3000:3000 --namespace=my-grafana
   ```

   For more information about port-forwarding, refer to [Use Port Forwarding to Access Applications in a Cluster](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/).

   b. Navigate to `localhost:3000` in your browser.

   The Grafana sign-in page appears.

   c. To sign in, enter `admin` for both the username and password.

## Access Grafana using minikube

There are multiple ways to access the Grafana UI on a web browser when using minikube. For more information about minikube, refer to [How to access applications running within minikube](https://minikube.sigs.k8s.io/docs/handbook/accessing/).

This section lists the two most common options for accessing an application running in minikube.

### Option 1: Expose the service

This option uses the `type: LoadBalancer` in the `grafana.yaml` service manifest, which makes the service accessible through the `minikube service` command. For more information, refer to [minikube Service command usage](https://minikube.sigs.k8s.io/docs/commands/service/).

1. Run the following command to obtain the Grafana service IP:

   ```bash
   minikube service grafana --namespace=my-grafana
   ```

   The output returns the Kubernetes URL for service in your local cluster.

   ```bash
   |------------|---------|-------------|------------------------------|
   | NAMESPACE  |  NAME   | TARGET PORT |             URL              |
   |------------|---------|-------------|------------------------------|
   | my-grafana | grafana |        3000 | http://192.168.122.144:32182 |
   |------------|---------|-------------|------------------------------|
   Opening service my-grafana/grafana in default browser...
   http://192.168.122.144:32182
   ```

1. Run a `curl` command to verify whether a given connection should work in a browser under ideal circumstances.

   ```bash
   curl 192.168.122.144:32182
   ```

   The following example output shows that an endpoint has been located:

   `<a href="/login">Found</a>.`

1. Access the Grafana UI in the browser using the provided IP:Port from the command above. For example `192.168.122.144:32182`

   The Grafana sign-in page appears.

1. To sign in to Grafana, enter `admin` for both the username and password.

### Option 2: Use port forwarding

If Option 1 does not work in your minikube environment (this mostly depends on the network), then as an alternative you can use the port forwarding option for the Grafana service on port `3000`.

For more information about port forwarding, refer to [Use Port Forwarding to Access Applications in a Cluster](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/).

1. To find the minikube IP address, run the following command:

   ```bash
   minikube ip
   ```

   The output contains the IP address that you use to access the Grafana Pod during port forwarding.

   A Pod is the smallest deployment unit in Kubernetes and is the core building block for running applications in a Kubernetes cluster. For more information about Pods, refer to [Pods](https://kubernetes.io/docs/concepts/workloads/pods/).

1. To obtain the Grafana Pod information, run the following command:

   ```bash
   kubectl get pods --namespace=my-grafana
   ```

   The output should look similar to the following:

   ```bash
   NAME                       READY   STATUS    RESTARTS   AGE
   grafana-58445b6986-dxrrw   1/1     Running   0          9m54s
   ```

   The output shows the Grafana POD name in the `NAME` column, that you use for port forwarding.

1. Run the following command for enabling the port forwarding on the POD:

   ```bash
   kubectl port-forward pod/grafana-58445b6986-dxrrw --namespace=my-grafana --address 0.0.0.0 3000:3000
   ```

1. To access the Grafana UI on the web browser, type the minikube IP along with the forwarded port. For example `192.168.122.144:3000`

   The Grafana sign-in page appears.

1. To sign in to Grafana, enter `admin` for both the username and password.

## Update an existing deployment using a rolling update strategy

Rolling updates enable deployment updates to take place with no downtime by incrementally updating Pods instances with new ones. The new Pods will be scheduled on nodes with available resources. For more information about rolling updates, refer to [Performing a Rolling Update](https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/).

The following steps use the `kubectl annotate` command to add the metadata and keep track of the deployment. For more information about `kubectl annotate`, refer to [kubectl annotate documentation](https://jamesdefabia.github.io/docs/user-guide/kubectl/kubectl_annotate/).

{{< admonition type="note" >}}
Instead of using the `annotate` flag, you can still use the `--record` flag. However, it has been deprecated and will be removed in the future version of Kubernetes. See: https://github.com/kubernetes/kubernetes/issues/40422
{{< /admonition >}}

1. To view the current status of the rollout, run the following command:

   ```bash
   kubectl rollout history deployment/grafana --namespace=my-grafana
   ```

   The output will look similar to this:

   ```bash
   deployment.apps/grafana
   REVISION  CHANGE-CAUSE
   1         NONE
   ```

   The output shows that nothing has been updated or changed after applying the `grafana.yaml` file.

1. To add metadata to keep record of the initial deployment, run the following command:

   ```bash
   kubectl annotate deployment/grafana kubernetes.io/change-cause='deployed the default base yaml file' --namespace=my-grafana
   ```

1. To review the rollout history and verify the changes, run the following command:

   ```bash
   kubectl rollout history deployment/grafana --namespace=my-grafana
   ```

   You should see the updated information that you added in the `CHANGE-CAUSE` earlier.

### Change Grafana image version

1. To change the deployed Grafana version, run the following `kubectl edit` command:

   ```bash
   kubectl edit deployment grafana --namespace=my-grafana
   ```

1. In the editor, change the container image under the `kind: Deployment` section.

   For example:
   - From
     - `yaml image: grafana/grafana:12.1.0`

   - To
     - `yaml image: grafana/grafana-dev:12.2.0-17161637292`

1. Save the changes.

   Once you save the file, you receive a message similar to the following:

   ```bash
   deployment.apps/grafana edited
   ```

   This means that the changes have been applied.

1. To verify that the rollout on the cluster is successful, run the following command:

   ```bash
   kubectl rollout status deployment grafana --namespace=my-grafana
   ```

   A successful deployment rollout means that the Grafana Dev cluster is now available.

1. To check the statuses of all deployed objects, run the following command and include the `-o wide` flag to get more detailed output:

   ```bash
   kubectl get all --namespace=my-grafana -o wide
   ```

   You should see the newly deployed `grafana-dev` image.

1. To verify it, access the Grafana UI in the browser using the provided IP:Port from the command above.

   The Grafana sign-in page appears.

1. To sign in to Grafana, enter `admin` for both the username and password.
1. In the top-right corner, click the help icon.

   The version information appears.

1. Add the `change cause` metadata to keep track of things using the commands:

   ```bash
   kubectl annotate deployment grafana --namespace=my-grafana kubernetes.io/change-cause='using grafana-dev:12.2.0-17161637292 for testing'
   ```

1. To verify, run the `kubectl rollout history` command:

   ```bash
   kubectl rollout history deployment grafana --namespace=my-grafana
   ```

   You will see an output similar to this:

   ```bash
   deployment.apps/grafana
   REVISION  CHANGE-CAUSE
   1         deploying the default yaml
   2         using grafana-dev:12.2.0-17161637292 for testing
   ```

This means that `REVISION#2` is the current version.

{{< admonition type="note" >}}
The last line of the `kubectl rollout history deployment` command output is the one which is currently active and running on your Kubernetes environment.
{{< /admonition >}}

### Roll back a deployment

When the Grafana deployment becomes unstable due to crash looping, bugs, and so on, you can roll back a deployment to an earlier version (a `REVISION`).

By default, Kubernetes deployment rollout history remains in the system so that you can roll back at any time. For more information, refer to [Rolling Back to a Previous Revision](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-back-to-a-previous-revision).

1. To list all possible `REVISION` values, run the following command:

   ```bash
   kubectl rollout history deployment grafana --namespace=my-grafana
   ```

1. To roll back to a previous version, run the `kubectl rollout undo` command and provide a revision number.

   Example: To roll back to a previous version, specify the `REVISION` number, which appears after you run the `kubectl rollout history deployment` command, in the `--to-revision` parameter.

   ```bash
   kubectl rollout undo deployment grafana --to-revision=1 --namespace=my-grafana
   ```

1. To verify that the rollback on the cluster is successful, run the following command:

   ```bash
   kubectl rollout status deployment grafana --namespace=my-grafana
   ```

1. Access the Grafana UI in the browser using the provided IP:Port from the command above.

   The Grafana sign-in page appears.

1. To sign in to Grafana, enter `admin` for both the username and password.
1. In the top-right corner, click the help icon to display the version number.

1. To see the new rollout history, run the following command:

   ```bash
   kubectl rollout history deployment grafana --namespace=my-grafana
   ```

If you need to go back to any other `REVISION`, just repeat the steps above and use the correct revision number in the `--to-revision` parameter.

## Provision Grafana resources using configuration files

Provisioning can add, update, or delete resources specified in your configuration files when Grafana starts. For detailed information, refer to [Grafana Provisioning](/docs/grafana/<GRAFANA_VERSION>/administration/provisioning).

This section outlines general instructions for provisioning Grafana resources within Kubernetes, using a persistent volume to supply the configuration files to the Grafana pod.

1. Add a new `PersistentVolumeClaim` to the `grafana.yaml` file.

   ```yaml
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: grafana-provisioning-pvc
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Mi
   ```

1. In the `grafana.yaml` file, mount the persistent volume into `/etc/grafana/provisioning` as follows.

   ```yaml
   ...
       volumeMounts:
         - mountPath: /etc/grafana/provisioning
           name: grafana-provisioning-pv
         ...
   volumes:
     - name: grafana-provisioning-pv
       persistentVolumeClaim:
         claimName: grafana-provisioning-pvc
   ...
   ```

1. Find or create the provision resources you want to add. For instance, create a `alerting.yaml` file adding a mute timing (alerting resource).

   ```yaml
   apiVersion: 1
   muteTimes:
     - orgId: 1
       name: MuteWeekends
       time_intervals:
         - weekdays: [saturday, sunday]
   ```

1. By default, configuration files for alerting resources need to be placed in the `provisioning/alerting` directory.

   Save the `alerting.yaml` file in a directory named `alerting`, as we will next supply this `alerting` directory to the `/etc/grafana/provisioning` folder of the Grafana pod.

1. Verify first the content of the provisioning directory in the running Grafana pod.

   ```bash
   kubectl exec -n my-grafana <pod_name> -- ls /etc/grafana/provisioning/
   ```

   ```bash
   kubectl exec -n my-grafana <pod_name> -- ls /etc/grafana/provisioning/alerting
   ```

   Because the `alerting` folder is not available yet, the last command should output a `No such file or directory` error.

1. Copy the local `alerting` directory to `/etc/grafana/provisioning/` in the Grafana pod.

   ```bash
   kubectl cp alerting my-grafana/<pod_name>:/etc/grafana/provisioning/
   ```

   You can follow the same process to provision additional Grafana resources by supplying the following folders:
   - `provisioning/dashboards`
   - `provisioning/datasources`
   - `provisioning/plugins`

1. Verify the `alerting` directory in the running Grafana pod includes the `alerting.yaml` file.

   ```bash
   kubectl exec -n my-grafana <pod_name> -- ls /etc/grafana/provisioning/alerting
   ```

1. Restart the Grafana pod to provision the resources.

   ```bash
   kubectl rollout restart -n my-grafana deployment --selector=app=grafana
   ```

   Note that `rollout restart` kills the previous pod and scales a new pod. When the old pod terminates, you may have to enable port-forwarding in the new pod. For instructions, refer to the previous sections about port forwarding in this guide.

1. Verify the Grafana resources are properly provisioned within the Grafana instance.

## Troubleshooting

This section includes troubleshooting tips you might find helpful when deploying Grafana on Kubernetes.

### Collecting logs

It is important to view the Grafana server logs while troubleshooting any issues.

1. To check the Grafana logs, run the following command:

   ```bash
   # dump Pod logs for a Deployment (single-container case)
   kubectl logs --namespace=my-grafana deploy/grafana
   ```

1. If you have multiple containers running in the deployment, run the following command to obtain the logs only for the Grafana deployment:

   ```bash
   # dump Pod logs for a Deployment (multi-container case)
   kubectl logs --namespace=my-grafana deploy/grafana -c grafana
   ```

For more information about accessing Kubernetes application logs, refer to [Pods](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#interacting-with-running-pods) and [Deployments](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#interacting-with-deployments-and-services).

### Increasing log levels to debug mode

By default, the Grafana log level is set to `info`, but you can increase it to `debug` mode to fetch information needed to diagnose and troubleshoot a problem. For more information about Grafana log levels, refer to [Configuring logs](/docs/grafana/latest/setup-grafana/configure-grafana#log).

The following example uses the Kubernetes ConfigMap which is an API object that stores non-confidential data in key-value pairs. For more information, refer to [Kubernetes ConfigMap Concept](https://kubernetes.io/docs/concepts/configuration/configmap/).

1. Create a empty file and name it `grafana.ini` and add the following:

   ```bash
   [log]
   ; # Either "debug", "info", "warn", "error", "critical", default is "info"
   ; # we change from info to debug level
   level = debug
   ```

   This example adds the portion of the log section from the configuration file. You can refer to the [Configure Grafana](/docs/grafana/latest/setup-grafana/configure-grafana/) documentation to view all the default configuration settings.

1. To add the configuration file into the Kubernetes cluster via the ConfigMap object, run the following command:

   ```bash
   kubectl create configmap ge-config --from-file=/path/to/file/grafana.ini --namespace=my-grafana
   ```

1. To verify the ConfigMap object creation, run the following command:

   ```bash
   kubectl get configmap --namespace=my-grafana
   ```

1. Open the `grafana.yaml` file and In the Deployment section, provide the mount path to the custom configuration (`/etc/grafana`) and reference the newly created ConfigMap for it.

   ```bash
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: grafana
     name: grafana
   # the rest of the code remains the same.
   ...
   ....
   ...
               requests:
               cpu: 250m
               memory: 750Mi
           volumeMounts:
             - mountPath: /var/lib/grafana
               name: grafana-pv
              # This is to mount the volume for the custom configuration
             - mountPath: /etc/grafana
               name: ge-config
       volumes:
         - name: grafana-pv
           persistentVolumeClaim:
             claimName: grafana-pvc
          # This is to provide the reference to the ConfigMap for the volume
         - name: ge-config
           configMap:
             name: ge-config
   ```

1. Deploy the manifest using the following kubectl apply command:

   ```bash
   kubectl apply -f grafana.yaml --namespace=my-grafana
   ```

1. To verify the status, run the following commands:

   ```bash
   # first check the rollout status
   kubectl rollout status deployment grafana --namespace=my-grafana

   # then check the deployment and configMap information
   kubectl get all --namespace=my-grafana
   ```

1. To verify it, access the Grafana UI in the browser using the provided IP:Port

   The Grafana sign-in page appears.

1. To sign in to Grafana, enter `admin` for both the username and password.

1. Navigate to **Server Admin > Settings** and then search for log.

   You should see the level to debug mode.

### Using the --dry-run command

You can use the Kubernetes `--dry-run` command to send requests to modifying endpoints and determine if the request would have succeeded.

Performing a dry run can be useful for catching errors or unintended consequences before they occur. For more information, refer to [Kubernetes Dry-run](https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/576-dry-run/README.md).

Example:

The following example shows how to perform a dry run when you make changes to the `grafana.yaml` such as using a new image version, or adding new labels and you want to determine if there are syntax errors or conflicts.

To perform a dry run, run the following command:

```bash
kubectl apply -f grafana.yaml --dry-run=server --namespace=grafana
```

If there are no errors, then the output will look similar to this:

```bash
persistentvolumeclaim/grafana-pvc unchanged (server dry run)
deployment.apps/grafana unchanged (server dry run)
service/grafana unchanged (server dry run)
```

If there are errors or warnings, you will see them in the terminal.

## Remove Grafana

If you want to remove any of the Grafana deployment objects, use the `kubectl delete command`.

1. If you want to remove the complete Grafana deployment, run the following command:

   ```bash
   kubectl delete -f grafana.yaml --namespace=my-grafana
   ```

   This command deletes the deployment, persistentvolumeclaim, and service objects.

1. To delete the ConfigMap, run the following command:

   ```bash
   kubectl delete configmap ge-config --namespace=my-grafana
   ```

## Deploy Grafana Enterprise on Kubernetes

The process for deploying Grafana Enterprise is almost identical to the preceding process, except for additional steps that are required for adding your license file.

### Obtain Grafana Enterprise license

To run Grafana Enterprise, you need a valid license.
To obtain a license, [contact a Grafana Labs representative](/contact?about=grafana-enterprise).
This topic assumes that you have a valid license in a `license.jwt` file.
Associate your license with a URL that you can use later in the topic.

### Create license secret

Create a Kubernetes secret from your license file using the following command:

```bash
kubectl create secret generic ge-license --from-file=/path/to/your/license.jwt
```

### Create Grafana Enterprise configuration

1. Create a Grafana configuration file with the name `grafana.ini`

1. Paste the following YAML contents into the file you created:

   ```yaml
   [enterprise]
   license_path = /etc/grafana/license/license.jwt
   [server]
   root_url =/your/license/root/url
   ```

1. Update the `root_url` field to the url associated with the license provided to you.

### Create Configmap for Grafana Enterprise configuration

Create a Kubernetes Configmap from your `grafana.ini` file with the following command:

```bash
kubectl create configmap ge-config --from-file=/path/to/your/grafana.ini
```

### Create Grafana Enterprise Kubernetes manifest

1. Create a `grafana.yaml` file, and copy-and-paste the following content into it.

   The following YAML is identical to the one for a Grafana installation, except for the additional references to the Configmap that contains your Grafana configuration file and the secret that has your license.

   ```yaml
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: grafana-pvc
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: grafana
     name: grafana
   spec:
     selector:
       matchLabels:
         app: grafana
     template:
       metadata:
         labels:
           app: grafana
       spec:
         securityContext:
           fsGroup: 472
           supplementalGroups:
             - 0
         containers:
           - image: grafana/grafana-enterprise:latest
             imagePullPolicy: IfNotPresent
             name: grafana
             ports:
               - containerPort: 3000
                 name: http-grafana
                 protocol: TCP
             readinessProbe:
               failureThreshold: 3
               httpGet:
                 path: /robots.txt
                 port: 3000
                 scheme: HTTP
               initialDelaySeconds: 10
               periodSeconds: 30
               successThreshold: 1
               timeoutSeconds: 2
             resources:
               limits:
                 memory: 4Gi
               requests:
                 cpu: 100m
                 memory: 2Gi
             volumeMounts:
               - mountPath: /var/lib/grafana
                 name: grafana-pv
               - mountPath: /etc/grafana
                 name: ge-config
               - mountPath: /etc/grafana/license
                 name: ge-license
         volumes:
           - name: grafana-pv
             persistentVolumeClaim:
               claimName: grafana-pvc
           - name: ge-config
             configMap:
               name: ge-config
           - name: ge-license
             secret:
               secretName: ge-license
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: grafana
   spec:
     ports:
       - port: 3000
         protocol: TCP
         targetPort: http-grafana
     selector:
       app: grafana
     sessionAffinity: None
     type: LoadBalancer
   ```

   {{< admonition type="caution" >}}
   If you use `LoadBalancer` in the Service and depending on your cloud platform and network configuration, doing so might expose your Grafana instance to the Internet. To eliminate this risk, use `ClusterIP` to restrict access from within the cluster Grafana is deployed to.
   {{< /admonition >}}

1. To send the manifest to Kubernetes API Server, run the following command:
   `kubectl apply -f grafana.yaml`

1. To verify the manifest was sent, run the following command:
   `kubectl port-forward service/grafana 3000:3000`

1. Navigate to `localhost:3000` in your browser.

   You should see the Grafana login page.

1. Use `admin` for both the username and password to login.
1. To verify you are working with an enterprise license, scroll to the bottom of the page where you should see `Enterprise (Licensed)`.

# Install Grafana on macOS

This page explains how to install Grafana on macOS.

The following video demonstrates how to install Grafana on macOS as outlined in this document:

{{< youtube id="1zdm8SxOLYQ" >}}

## Install Grafana on macOS using Homebrew

To install Grafana on macOS using Homebrew, complete the following steps:

1. On the [Homebrew](http://brew.sh/) homepage, search for Grafana.

   The last stable and released version is listed.

1. Open a terminal and run the following commands:

   ```
   brew update
   brew install grafana
   ```

   The brew page downloads and untars the files into:
   - `/usr/local/Cellar/grafana/[version]` (Intel Silicon)
   - `/opt/homebrew/Cellar/grafana/[version]` (Apple Silicon)

1. To start Grafana, run the following command:

   ```bash
   brew services start grafana
   ```

### Using the Grafana CLI with Homebrew

To use the Grafana CLI with Homebrew, you need to append the home path, the config file path and - based on the command - some other configurations to the `cli` command:

For `admin` commands, you need to append the `--configOverrides cfg:default.paths.data=/opt/homebrew/var/lib/grafana` configuration. Example:

```bash
/opt/homebrew/opt/grafana/bin/grafana cli --config /opt/homebrew/etc/grafana/grafana.ini --homepath /opt/homebrew/opt/grafana/share/grafana --configOverrides cfg:default.paths.data=/opt/homebrew/var/lib/grafana admin reset-admin-password <new password>
```

For `plugins` commands, you need to append the `--pluginsDir /opt/homebrew/var/lib/grafana/plugins` configuration. Example:

```bash
/opt/homebrew/opt/grafana/bin/grafana cli --config /opt/homebrew/etc/grafana/grafana.ini --homepath /opt/homebrew/opt/grafana/share/grafana --pluginsDir "/opt/homebrew/var/lib/grafana/plugins" plugins install <plugin-id>
```

## Install standalone macOS binaries

To install Grafana on macOS using the standalone binaries, complete the following steps:

1. Navigate to the [Grafana download page](/grafana/download).
1. Select the Grafana version you want to install.
   - The most recent Grafana version is selected by default.
   - The **Version** field displays only tagged releases. If you want to install a nightly build, click **Nightly Builds** and then select a version.
1. Select an **Edition**.
   - **Enterprise:** This is the recommended version. It is functionally identical to the open source version, but includes features you can unlock with a license, if you so choose.
   - **Open Source:** This version is functionally identical to the Enterprise version, but you will need to download the Enterprise version if you want Enterprise features.
1. Click **Mac**.
1. Copy and paste the code from the [download page](/grafana/download) into your command line and run.
1. Untar the `gz` file and copy the files to the location of your preference.
1. To start Grafana service, go to the directory and run the command:

   ```bash
   ./bin/grafana server
   ```

Alternatively, watch the Grafana for Beginners video below:

{{< youtube id="T51Qa7eE3W8" >}}

## Next steps

- [Start the Grafana server](../../start-restart-grafana/)

# Install Grafana on RHEL or Fedora

This topic explains how to install Grafana dependencies, install Grafana on RHEL or Fedora, and start the Grafana server on your system.

You can install Grafana from the RPM repository, from standalone RPM, or with the binary `.tar.gz` file.

If you install via RPM or the `.tar.gz` file, then you must manually update Grafana for each new version.

The following video demonstrates how to install Grafana on RHEL or Fedora as outlined in this document:

{{< youtube id="4khbLlyoqzE" >}}

## Install Grafana from the RPM repository

If you install from the RPM repository, then Grafana is automatically updated every time you update your applications.

| Grafana Version           | Package            | Repository                     |
| ------------------------- | ------------------ | ------------------------------ |
| Grafana Enterprise        | grafana-enterprise | `https://rpm.grafana.com`      |
| Grafana Enterprise (Beta) | grafana-enterprise | `https://rpm-beta.grafana.com` |
| Grafana OSS               | grafana            | `https://rpm.grafana.com`      |
| Grafana OSS (Beta)        | grafana            | `https://rpm-beta.grafana.com` |

{{< admonition type="note" >}}
Grafana Enterprise is the recommended and default edition. It is available for free and includes all the features of the OSS edition. You can also upgrade to the [full Enterprise feature set](/products/enterprise/?utm_source=grafana-install-page), which has support for [Enterprise plugins](/grafana/plugins/?enterprise=1&utcm_source=grafana-install-page).
{{< /admonition >}}

To install Grafana from the RPM repository, complete the following steps:

{{< admonition type="note" >}}
If you wish to install beta versions of Grafana, substitute the repository URL for the beta URL listed above.
{{< /admonition >}}

1. Import the GPG key:

   ```bash
   wget -q -O gpg.key https://rpm.grafana.com/gpg.key
   sudo rpm --import gpg.key
   ```

1. Create `/etc/yum.repos.d/grafana.repo` with the following content:

   ```bash
   [grafana]
   name=grafana
   baseurl=https://rpm.grafana.com
   repo_gpgcheck=1
   enabled=1
   gpgcheck=1
   gpgkey=https://rpm.grafana.com/gpg.key
   sslverify=1
   ```

1. To install Grafana OSS, run the following command:

   ```bash
   sudo dnf install grafana
   ```

1. To install Grafana Enterprise, run the following command:

   ```bash
   sudo dnf install grafana-enterprise
   ```

## Install the Grafana RPM package manually

If you install Grafana manually using YUM or RPM, then you must manually update Grafana for each new version. This method varies according to which Linux OS you are running.

**Note:** The RPM files are signed. You can verify the signature with this [public GPG key](https://rpm.grafana.com/gpg.key).

1. On the [Grafana download page](/grafana/download), select the Grafana version you want to install.
   - The most recent Grafana version is selected by default.
   - The **Version** field displays only finished releases. If you want to install a beta version, click **Nightly Builds** and then select a version.
1. Select an **Edition**.
   - **Enterprise** - Recommended download. Functionally identical to the open source version, but includes features you can unlock with a license if you so choose.
   - **Open Source** - Functionally identical to the Enterprise version, but you will need to download the Enterprise version if you want Enterprise features.
1. Depending on which system you are running, click **Linux** or **ARM**.
1. Copy and paste the RPM package URL and the local RPM package information from the [download page](/grafana/download) into the pattern shown below and run the command.

   ```bash
   sudo yum install -y <rpm package url>
   ```

## Install Grafana as a standalone binary

If you install Grafana manually using the standalone binaries, then you must manually update Grafana for each new version.

Complete the following steps to install Grafana using the standalone binaries:

1. Navigate to the [Grafana download page](/grafana/download).
1. Select the Grafana version you want to install.
   - The most recent Grafana version is selected by default.
   - The **Version** field displays only tagged releases. If you want to install a nightly build, click **Nightly Builds** and then select a version.
1. Select an **Edition**.
   - **Enterprise:** This is the recommended version. It is functionally identical to the open source version but includes features you can unlock with a license if you so choose.
   - **Open Source:** This version is functionally identical to the Enterprise version, but you will need to download the Enterprise version if you want Enterprise features.
1. Depending on which system you are running, click the **Linux** or **ARM** tab on the [download page](/grafana/download).
1. Copy and paste the code from the [download page](/grafana/download) page into your command line and run.
1. Create a user account for Grafana on your system:

   ```shell
   sudo useradd -r -s /bin/false grafana
   ```

1. Move the unpacked binary to `/usr/local/grafana`:

   ```shell
   sudo mv <DOWNLOAD PATH> /usr/local/grafana
   ```

1. Change the owner of `/usr/local/grafana` to Grafana users:

   ```shell
   sudo chown -R grafana:users /usr/local/grafana
   ```

1. Create a Grafana server systemd unit file:

   ```shell
   sudo touch /etc/systemd/system/grafana-server.service
   ```

1. Add the following to the unit file in a text editor of your choice:

   ```ini
   [Unit]
   Description=Grafana Server
   After=network.target

   [Service]
   Type=simple
   User=grafana
   Group=users
   ExecStart=/usr/local/grafana/bin/grafana server --config=/usr/local/grafana/conf/grafana.ini --homepath=/usr/local/grafana
   Restart=on-failure

   [Install]
   WantedBy=multi-user.target
   ```

1. Use the binary to manually start the Grafana server:

   ```shell
   /usr/local/grafana/bin/grafana-server --homepath /usr/local/grafana
   ```

   {{< admonition type="note" >}}
   Manually invoking the binary in this step automatically creates the `/usr/local/grafana/data` directory, which needs to be created and configured before the installation can be considered complete.
   {{< /admonition >}}

1. Press `CTRL+C` to stop the Grafana server.
1. Change the owner of `/usr/local/grafana` to Grafana users again to apply the ownership to the newly created `/usr/local/grafana/data` directory:

   ```shell
   sudo chown -R grafana:users /usr/local/grafana
   ```

1. [Configure the Grafana server to start at boot time using systemd](../../start-restart-grafana/#configure-the-grafana-server-to-start-at-boot-using-systemd).

## Uninstall on RHEL or Fedora

To uninstall Grafana, run the following commands in a terminal window:

1. If you configured Grafana to run with systemd, stop the systemd service for Grafana server:

   ```shell
   sudo systemctl stop grafana-server
   ```

1. If you configured Grafana to run with init.d, stop the init.d service for Grafana server:

   ```shell
   sudo service grafana-server stop
   ```

1. To uninstall Grafana OSS:

   ```shell
   sudo dnf remove grafana
   ```

1. To uninstall Grafana Enterprise:

   ```shell
   sudo dnf remove grafana-enterprise
   ```

1. Optional: To remove the Grafana repository:

   ```shell
   sudo rm -i /etc/yum.repos.d/grafana.repo
   ```

## Next steps

Refer to [Start the Grafana server](../../start-restart-grafana/).

# Install Grafana on SUSE or openSUSE

This topic explains how to install Grafana dependencies, install Grafana on SUSE or openSUSE and start the Grafana server on your system.

You can install Grafana using the RPM repository, or by downloading a binary `.tar.gz` file.

If you install via RPM or the `.tar.gz` file, then you must manually update Grafana for each new version.

The following video demonstrates how to install Grafana on SUSE or openSUSE as outlined in this document:

{{< youtube id="2MWsu0xy5Xc" >}}

## Install Grafana from the RPM repository

If you install from the RPM repository, then Grafana is automatically updated every time you run `sudo zypper update`.

| Grafana Version    | Package            | Repository                |
| ------------------ | ------------------ | ------------------------- |
| Grafana Enterprise | grafana-enterprise | `https://rpm.grafana.com` |
| Grafana OSS        | grafana            | `https://rpm.grafana.com` |

{{< admonition type="note" >}}
Grafana Enterprise is the recommended and default edition. It is available for free and includes all the features of the OSS edition. You can also upgrade to the [full Enterprise feature set](/products/enterprise/?utm_source=grafana-install-page), which has support for [Enterprise plugins](/grafana/plugins/?enterprise=1&utcm_source=grafana-install-page).
{{< /admonition >}}

To install Grafana using the RPM repository, complete the following steps:

1. Import the GPG key:

   ```bash
   wget -q -O gpg.key https://rpm.grafana.com/gpg.key
   sudo rpm --import gpg.key
   ```

1. Use zypper to add the grafana repo.

   ```bash
   sudo zypper addrepo https://rpm.grafana.com grafana
   ```

1. To install Grafana OSS, run the following command:

   ```bash
   sudo zypper install grafana
   ```

1. To install Grafana Enterprise, run the following command:

   ```bash
   sudo zypper install grafana-enterprise
   ```

## Install the Grafana RPM package manually

If you install Grafana manually using RPM, then you must manually update Grafana for each new version. This method varies according to which Linux OS you are running.

**Note:** The RPM files are signed. You can verify the signature with this [public GPG key](https://rpm.grafana.com/gpg.key).

1. On the [Grafana download page](/grafana/download), select the Grafana version you want to install.
   - The most recent Grafana version is selected by default.
   - The **Version** field displays only finished releases. If you want to install a beta version, click **Nightly Builds** and then select a version.
1. Select an **Edition**.
   - **Enterprise** - Recommended download. Functionally identical to the open source version, but includes features you can unlock with a license if you so choose.
   - **Open Source** - Functionally identical to the Enterprise version, but you will need to download the Enterprise version if you want Enterprise features.
1. Depending on which system you are running, click **Linux** or **ARM**.
1. Copy and paste the RPM package URL and the local RPM package information from the installation page into the pattern shown below, then run the commands.

   ```bash
   sudo zypper install initscripts urw-fonts wget
   wget <rpm package url>
   sudo rpm -Uvh <local rpm package>
   ```

## Install Grafana as a standalone binary

If you install Grafana manually using the standalone binaries, then you must manually update Grafana for each new version.

Complete the following steps to install Grafana using the standalone binaries:

1. Navigate to the [Grafana download page](/grafana/download).
1. Select the Grafana version you want to install.
   - The most recent Grafana version is selected by default.
   - The **Version** field displays only tagged releases. If you want to install a nightly build, click **Nightly Builds** and then select a version.
1. Select an **Edition**.
   - **Enterprise:** This is the recommended version. It is functionally identical to the open source version but includes features you can unlock with a license if you so choose.
   - **Open Source:** This version is functionally identical to the Enterprise version, but you will need to download the Enterprise version if you want Enterprise features.
1. Depending on which system you are running, click the **Linux** or **ARM** tab on the [download page](/grafana/download).
1. Copy and paste the code from the [download page](/grafana/download) into your command line and run.
1. Create a user account for Grafana on your system:

   ```shell
   sudo useradd -r -s /bin/false grafana
   ```

1. Move the unpacked binary to `/usr/local/grafana`:

   ```shell
   sudo mv <DOWNLOAD PATH> /usr/local/grafana
   ```

1. Change the owner of `/usr/local/grafana` to Grafana users:

   ```shell
   sudo chown -R grafana:users /usr/local/grafana
   ```

1. Create a Grafana server systemd unit file:

   ```shell
   sudo touch /etc/systemd/system/grafana-server.service
   ```

1. Add the following to the unit file in a text editor of your choice:

   ```ini
   [Unit]
   Description=Grafana Server
   After=network.target

   [Service]
   Type=simple
   User=grafana
   Group=users
   ExecStart=/usr/local/grafana/bin/grafana server --config=/usr/local/grafana/conf/grafana.ini --homepath=/usr/local/grafana
   Restart=on-failure

   [Install]
   WantedBy=multi-user.target
   ```

1. Use the binary to manually start the Grafana server:

   ```shell
   /usr/local/grafana/bin/grafana-server --homepath /usr/local/grafana
   ```

   {{< admonition type="note" >}}
   Manually invoking the binary in this step automatically creates the `/usr/local/grafana/data` directory, which needs to be created and configured before the installation can be considered complete.
   {{< /admonition >}}

1. Press `CTRL+C` to stop the Grafana server.
1. Change the owner of `/usr/local/grafana` to Grafana users again to apply the ownership to the newly created `/usr/local/grafana/data` directory:

   ```shell
   sudo chown -R grafana:users /usr/local/grafana
   ```

1. [Configure the Grafana server to start at boot time using systemd](https://grafana.com/docs/grafana/latest/setup-grafana/start-restart-grafana/#configure-the-grafana-server-to-start-at-boot-using-systemd).

## Uninstall on SUSE or openSUSE

To uninstall Grafana, run the following commands in a terminal window:

1. If you configured Grafana to run with systemd, stop the systemd service for Grafana server:

   ```shell
   sudo systemctl stop grafana-server
   ```

1. If you configured Grafana to run with init.d, stop the init.d service for Grafana server:

   ```shell
   sudo service grafana-server stop
   ```

1. To uninstall Grafana OSS:

   ```shell
   sudo zypper remove grafana
   ```

1. To uninstall Grafana Enterprise:

   ```shell
   sudo zypper remove grafana-enterprise
   ```

1. Optional: To remove the Grafana repository:

   ```shell
   sudo zypper removerepo grafana
   ```

## Next steps

Refer to [Start the Grafana server](../../start-restart-grafana/).

# Install Grafana on Windows

The following video demonstrates how to install Grafana using the Windows standalone installer as outlined in this document:

{{< youtube id="js2bZijbhJM" >}}

You install Grafana using the Windows installer or using the standalone Windows binary file.

1. Navigate to the [Grafana download page](/grafana/download).
1. Select the Grafana version you want to install.
   - The most recent Grafana version is selected by default.
   - The **Version** field displays only tagged releases. If you want to install a nightly build, click **Nightly Builds** and then select a version.
1. Select an **Edition**.
   - **Enterprise:** This is the recommended version. It is functionally identical to the open source version, but includes features you can unlock with a license, if you so choose.
   - **Open Source:** This version is functionally identical to the Enterprise version, but you will need to download the Enterprise version if you want Enterprise features.
1. Click **Windows**.
1. To use the Windows installer, complete the following steps:

   a. Click **Download the installer**.

   b. Open and run the installer.

1. To install the standalone Windows binary, complete the following steps:

   a. Click **Download the zip file**.

   b. Right-click the downloaded file, select **Properties**, select the `unblock` checkbox, and click `OK`.

   c. Extract the ZIP file to any folder.

Start Grafana by executing `grafana-server.exe`, located in the `bin` directory, preferably from the command line. If you want to run Grafana as a Windows service, then download
[NSSM](https://nssm.cc/). It is very easy to add Grafana as a Windows service using that tool.

1. To run Grafana, open your browser and go to the Grafana port (http://localhost:3000/ is default) and then follow the instructions in [Getting Started](../../../getting-started/build-first-dashboard/).

   > **Note:** The default Grafana port is `3000`. This port might require extra permissions on Windows. If it does not appear in the default port, you can change the port number.

1. To change the port, perform the following steps:

   a. Open the `conf` directory and copy `sample.ini` to `custom.ini`.

   > **Note:** You should edit `custom.ini`, never `defaults.ini`.

   b. Edit `custom.ini` and uncomment the `http_port` configuration option.

   `;` is the comment character in ini files.

   c. Change the port to `8080` or something similar.

   Port `8080` should not require extra Windows privileges.

## Next steps

- [Start the Grafana server](../../start-restart-grafana/)

# Shared

# communicate

---
headless: true
comments: |
  Communicate with Grafana in About, Intro, Contribute.
---

Use the right place to ask questions, report problems, and propose changes.

- **GitHub issues and pull requests**: Use for reproducible bugs in core Grafana and maintained plugins, small and actionable feature requests, and code or docs changes via pull requests. Avoid general “how do I” questions. For security issues, follow the [security policy](https://github.com/grafana/grafana/security/policy).
- **Grafana community forums**: Use for questions, troubleshooting, best practices, plugin development Q&A, and early idea discussion. Forums create a searchable public knowledge base that helps others with the same problems and questions. Start here if you are unsure: [Grafana community forums](https://community.grafana.com/).
- **Grafana Community Slack**: Use for quick, time-sensitive chats and networking. Do not rely on Slack for long troubleshooting or decisions. Share outcomes back to a forum topic or GitHub issue/PR to keep a public record: [Grafana Community Slack](https://slack.grafana.com).

{{< admonition type="note" >}}
**Not sure where to start?** Open a new topic in the forum! Maintainers and community members will redirect you if a GitHub issue or pull request is more appropriate.
{{< /admonition >}}

# Shared Content

Use test.md to experiment and test formats and syntax.

## Standalone text

When you have a chunk of text or steps that stand alone, not part of an ordered or unordered list. This includes headings, paragraphs, full lists, and combinations of those content types.

The syntax to invoke this file would be the following, minus the backslash:

```
\{{< docs/shared lookup="example.md" source="grafana" version="<GRAFANA VERSION>" >}}
```

## Part of a list

When you have steps that you want to use in more than one numbered list. This format does not work as well for unordered lists, but it does work.

### Ordered list

Below is an example from the docs, with backslashes added. The initial spaces are not necessary for the numbered list to work and make no difference in the output, but they aid in code readability.

```
\{{< docs/list >}}
  \{{< docs/shared lookup="manage-users/view-server-user-list.md" source="grafana" version="<GRAFANA VERSION>" >}}
  1. Click the user account that you want to edit. If necessary, use the search field to find the account.
\{{< /docs/list >}}
```

You cannot use short codes in an ordered list with sublists. The shortcode breaks the sublist indentation.

### Unordered list

All unordered list steps included as part of a list will appear as second-level lists (with the hollow circle bullet) rather than first-level lists (solid circle bullet), even if the list is not indented in the shared file or the document file.

{{< docs/list >}}
{{< docs/shared lookup="test.md" source="grafana" version="<GRAFANA VERSION>" >}}

- Bullet text
  {{< /docs/list >}}

# index

---
headless: true
labels:
  products:
    - enterprise
    - oss
---

# Shared Content

Intro text

- test bullet 1
- test bullet 2

# View user list

1. Hover your cursor over the **Server Admin** (shield) icon until a menu appears.
1. Click **Orgs**.

# View user list

1. Hover your cursor over the **Server Admin** (shield) icon until a menu appears.
1. Click **Users**.
1. Click the user account that you want to edit. If necessary, use the search field to find the account.

# Add an alertmanager data source

1. Click **Connections** in the left-side menu.
1. Under Your connections, click **Data sources**.
1. Enter `Alertmanager` in the search bar.
1. Click **Alertmanager**.

   The **Settings** tab of the data source is displayed.

1. Set the data source's basic configuration options:

   | Name                            | Description                                                                                                                                                                                 |
   | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Name**                        | Sets the name you use to refer to the data source                                                                                                                                           |
   | **Default**                     | Sets whether the data source is pre-selected for new panels and queries                                                                                                                     |
   | **Alertmanager Implementation** | Alertmanager implementation. **Mimir**, **Cortex,** and **Prometheus** are supported                                                                                                        |
   | **Receive Grafana Alerts**      | When enabled, the Alertmanager can receive Grafana-managed alerts. **Important:** This works only if receiving alerts is enabled for the Alertmanager in the Grafana Alerting Settings page |
   | **HTTP URL**                    | Sets the HTTP protocol, IP, and port of your Alertmanager instance, such as `https://alertmanager.example.org:9093`                                                                         |
   | **Access**                      | Only **Server** access mode is functional                                                                                                                                                   |

# Alerting Produces

#### Produces

- `application/json`
- `application/yaml`
- `application/terraform+hcl`
- `text/yaml`
- `text/hcl`

These outputs are for [file provisioning](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/file-provisioning) or [Terraform provisioning](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/file-provisioning), and they-including the JSON output—cannot be used to update resources via the HTTP API.

# Alerting Provisioning HTTP API

The Alerting Provisioning HTTP API can be used to create, modify, and delete resources relevant to Grafana-managed alerts. This API is the one used by our [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).

> If you are running Grafana Enterprise, you need to add specific permissions for some endpoints. For more information, refer to [Role-based access control permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/custom-role-actions-scopes/).

{{< admonition type="warning" >}}
The contact points, notification policies, notification template groups, and mute timings endpoints in this API are deprecated and will be removed in a future release. These endpoints don't return resources from a staged Alertmanager configuration. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead.
{{< /admonition >}}

## Grafana-managed endpoints

{{< admonition type="note" >}}
In the Alerting provisioning HTTP API, the endpoints use a JSON format that differs from the format returned by the `export` endpoints.

The `export` endpoints allow you to export alerting resources in a JSON format suitable for [provisioning via files](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/file-provisioning/). However, this format cannot be used to update resources via the HTTP API.

{{< /admonition >}}

### Alert rules

The following endpoints can be used to manage both alert rules and recording rules. To create a recording rule, include a `record` block in your request instead of a `condition` field.

| Method | URI                                                              | Name                                                                    | Summary                                                 |
| ------ | ---------------------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------- |
| DELETE | /api/v1/provisioning/alert-rules/:uid                            | [route delete alert rule](#route-delete-alert-rule)                     | Delete a specific alert rule by UID.                    |
| GET    | /api/v1/provisioning/alert-rules/:uid                            | [route get alert rule](#route-get-alert-rule)                           | Get a specific alert rule by UID.                       |
| POST   | /api/v1/provisioning/alert-rules                                 | [route post alert rule](#route-post-alert-rule)                         | Create a new alert rule.                                |
| PUT    | /api/v1/provisioning/alert-rules/:uid                            | [route put alert rule](#route-put-alert-rule)                           | Update an existing alert rule.                          |
| GET    | /api/v1/provisioning/alert-rules/:uid/export                     | [route get alert rule export](#route-get-alert-rule-export)             | Export an alert rule in provisioning file format.       |
| DELETE | /api/v1/provisioning/folder/:folderUid/rule-groups/:group        | [route delete alert rule group](#route-delete-alert-rule-group)         | Delete a rule group.                                    |
| GET    | /api/v1/provisioning/folder/:folderUid/rule-groups/:group        | [route get alert rule group](#route-get-alert-rule-group)               | Get a rule group.                                       |
| PUT    | /api/v1/provisioning/folder/:folderUid/rule-groups/:group        | [route put alert rule group](#route-put-alert-rule-group)               | Create or update a rule group.                          |
| GET    | /api/v1/provisioning/folder/:folderUid/rule-groups/:group/export | [route get alert rule group export](#route-get-alert-rule-group-export) | Export an alert rule group in provisioning file format. |
| GET    | /api/v1/provisioning/alert-rules                                 | [route get alert rules](#route-get-alert-rules)                         | Get all the alert rules.                                |
| GET    | /api/v1/provisioning/alert-rules/export                          | [route get alert rules export](#route-get-alert-rules-export)           | Export all alert rules in provisioning file format.     |

**Example request for new alert rule:**

```http
POST /api/v1/provisioning/alert-rules
Accept: application/json
Content-Type: application/json
Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk

{
  "title": "TEST-API_1",
  "ruleGroup": "API",
  "folderUID": "SET_FOLDER_UID",
  "noDataState": "OK",
  "execErrState": "OK",
  "for": "5m",
  "keepFiringFor": "2m",
  "orgId": 1,
  "uid": "",
  "condition": "B",
  "annotations": {
    "summary": "test_api_1"
  },
  "labels": {
    "API": "test1"
  },
  "data": [
    {
      "refId": "A",
      "queryType": "",
      "relativeTimeRange": {
        "from": 600,
        "to": 0
      },
      "datasourceUid": "XXXXXXXXX-XXXXXXXXX-XXXXXXXXXX",
      "model": {
        "expr": "up",
        "hide": false,
        "intervalMs": 1000,
        "maxDataPoints": 43200,
        "refId": "A"
      }
    },
    {
      "refId": "B",
      "queryType": "",
      "relativeTimeRange": {
        "from": 0,
        "to": 0
      },
      "datasourceUid": "-100",
      "model": {
        "conditions": [
          {
            "evaluator": {
              "params": [6],
              "type": "gt"
            },
            "operator": {
              "type": "and"
            },
            "query": {
              "params": ["A"]
            },
            "reducer": {
              "params": [],
              "type": "last"
            },
            "type": "query"
          }
        ],
        "datasource": {
          "type": "__expr__",
          "uid": "-100"
        },
        "hide": false,
        "intervalMs": 1000,
        "maxDataPoints": 43200,
        "refId": "B",
        "type": "classic_conditions"
      }
    }
  ]
}

```

#### Example Response:

```http
HTTP/1.1 201 Created
Content-Type: application/json

{
  "id": 1,
  "uid": "XXXXXXXXX",
  "orgID": 1,
  "folderUID": "SET_FOLDER_UID",
  "ruleGroup": "API3",
  "title": "TEST-API_1",
  "condition": "B",
  "data": [
    {
      "refId": "A",
      "queryType": "",
      "relativeTimeRange": {
        "from": 600,
        "to": 0
      },
      "datasourceUid": "XXXXXXXXX-XXXXXXXXX-XXXXXXXXXX",
      "model": {
        "expr": "up",
        "hide": false,
        "intervalMs": 1000,
        "maxDataPoints": 43200,
        "refId": "A"
      }
    },
    {
      "refId": "B",
      "queryType": "",
      "relativeTimeRange": {
        "from": 0,
        "to": 0
      },
      "datasourceUid": "-100",
      "model": {
        "conditions": [
          {
            "evaluator": {
              "params": [
                6
              ],
              "type": "gt"
            },
            "operator": {
              "type": "and"
            },
            "query": {
              "params": [
                "A"
              ]
            },
            "reducer": {
              "params": [],
              "type": "last"
            },
            "type": "query"
          }
        ],
        "datasource": {
          "type": "__expr__",
          "uid": "-100"
        },
        "hide": false,
        "intervalMs": 1000,
        "maxDataPoints": 43200,
        "refId": "B",
        "type": "classic_conditions"
      }
    }
  ],
  "updated": "2024-08-02T13:19:32.609640048Z",
  "noDataState": "OK",
  "execErrState": "OK",
  "for": "5m",
  "keepFiringFor": "2m",
  "annotations": {
    "summary": "test_api_1"
  },
  "labels": {
    "API": "test1"
  },
  "provenance": "api",
  "isPaused": false,
  "notification_settings": null,
  "record": null
}
```

#### Recording rules

Recording rules allow you to pre-compute frequently used or computationally expensive queries and save the results as a new time series metric. The same alert rule provisioning endpoints support creating recording rules by including the `record` field instead of alert-specific fields like `condition`, `noDataState`, or `execErrState`.

**Example request for new recording rule:**

```http
POST /api/v1/provisioning/alert-rules
Accept: application/json
Content-Type: application/json
Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk

{
  "title": "my_recording_rule",
  "ruleGroup": "recording_rules_group",
  "folderUID": "SET_FOLDER_UID",
  "for": "0s",
  "orgId": 1,
  "uid": "",
  "labels": {
    "team": "backend"
  },
  "data": [
    {
      "refId": "A",
      "queryType": "",
      "relativeTimeRange": {
        "from": 600,
        "to": 0
      },
      "datasourceUid": "PROMETHEUS_DATASOURCE_UID",
      "model": {
        "expr": "sum(rate(http_requests_total[5m]))",
        "intervalMs": 1000,
        "maxDataPoints": 43200,
        "refId": "A"
      }
    }
  ],
  "record": {
    "metric": "http_requests:rate5m:sum",
    "from": "A",
    "target_datasource_uid": "TARGET_PROMETHEUS_UID"
  }
}
```

**Example response:**

```http
HTTP/1.1 201 Created
Content-Type: application/json

{
  "id": 2,
  "uid": "YYYYYYYYY",
  "orgID": 1,
  "folderUID": "SET_FOLDER_UID",
  "ruleGroup": "recording_rules_group",
  "title": "my_recording_rule",
  "for": "0s",
  "data": [
    {
      "refId": "A",
      "queryType": "",
      "relativeTimeRange": {
        "from": 600,
        "to": 0
      },
      "datasourceUid": "PROMETHEUS_DATASOURCE_UID",
      "model": {
        "expr": "sum(rate(http_requests_total[5m]))",
        "intervalMs": 1000,
        "maxDataPoints": 43200,
        "refId": "A"
      }
    }
  ],
  "updated": "2024-08-02T14:30:15.123456789Z",
  "labels": {
    "team": "backend"
  },
  "provenance": "api",
  "isPaused": false,
  "record": {
    "metric": "http_requests:rate5m:sum",
    "from": "A",
    "target_datasource_uid": "TARGET_PROMETHEUS_UID"
  }
}
```

**Important notes for recording rules:**

- The `metric` field must be a valid Prometheus metric name and contain no whitespace.
- The `from` field specifies which query reference (refId) to use as the source for the recorded metric.
- The `target_datasource_uid` specifies which Prometheus-compatible data source to write the results to. If not specified, the default data source configured in `[recording_rules].default_datasource_uid` is used.
- Recording rules do not support `condition`, `noDataState`, `execErrState`, or `notification_settings` fields.
- Set `for` to `0s` for recording rules as they do not have a pending state.

### Contact points

The contact point endpoints support only v1 integrations. `GET` requests omit integrations that use another version. `PUT` and `DELETE` requests for those integrations return `404 Not Found`.

| Method | URI                                        | Name                                                              | Summary                                                |
| ------ | ------------------------------------------ | ----------------------------------------------------------------- | ------------------------------------------------------ |
| DELETE | /api/v1/provisioning/contact-points/:uid   | [route delete contactpoints](#route-delete-contactpoints)         | Delete a contact point.                                |
| GET    | /api/v1/provisioning/contact-points        | [route get contactpoints](#route-get-contactpoints)               | Get all the contact points.                            |
| POST   | /api/v1/provisioning/contact-points        | [route post contactpoints](#route-post-contactpoints)             | Create a contact point.                                |
| PUT    | /api/v1/provisioning/contact-points/:uid   | [route put contactpoint](#route-put-contactpoint)                 | Update an existing contact point.                      |
| GET    | /api/v1/provisioning/contact-points/export | [route get contactpoints export](#route-get-contactpoints-export) | Export all contact points in provisioning file format. |

**Example Request for all the contact points:**

```http
GET /api/v1/provisioning/contact-points
Accept: application/json
Content-Type: application/json
Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
```

**Example Response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "uid": "",
    "name": "email receiver",
    "type": "email",
    "settings": {
      "addresses": "<example@example.com>"
    },
    "disableResolveMessage": false
  }
]
```

### Receiver permissions

The receiver permissions endpoints manage access control for contact point receivers. These endpoints allow you to assign permissions to users, teams, or built-in roles for specific receivers.

| Method | URI                                                          | Name                                                                            | Summary                                                     |
| ------ | ------------------------------------------------------------ | ------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| POST   | /api/access-control/receivers/:uid/users/:userID             | [route set user receiver permission](#route-set-user-receiver-permission)       | Set permissions for a user on a specific receiver.          |
| POST   | /api/access-control/receivers/:uid/teams/:teamID             | [route set team receiver permission](#route-set-team-receiver-permission)       | Set permissions for a team on a specific receiver.          |
| POST   | /api/access-control/receivers/:uid/builtInRoles/:builtInRole | [route set builtin receiver permission](#route-set-builtin-receiver-permission) | Set permissions for a built-in role on a specific receiver. |

**Example Request to assign permissions to a user:**

```http
POST /api/access-control/receivers/abc123/users/5
Accept: application/json
Content-Type: application/json
Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk

{
  "permission": "Edit"
}
```

**Example Response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json

{
  "message": "Permission updated"
}
```

**Example Request to remove permissions from a team:**

```http
POST /api/access-control/receivers/abc123/teams/3
Accept: application/json
Content-Type: application/json
Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk

{
  "permission": ""
}
```

**Example Response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json

{
  "message": "Permission removed"
}
```

**Example Request to assign permissions to a built-in role:**

```http
POST /api/access-control/receivers/abc123/builtInRoles/Viewer
Accept: application/json
Content-Type: application/json
Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk

{
  "permission": "View"
}
```

**Example Response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json

{
  "message": "Permission updated"
}
```

**Available Permissions sets:**

- `View` - Read-only access to the receiver.
  - Grants `alert.notifications.receivers:read`.
- `Edit` - Ability to update and test the receiver.
  - Grants `View` actions plus:
    - `alert.notifications.receivers:write`
    - `alert.notifications.receivers:delete`
    - `alert.notifications.receivers.test:create`
- `Admin` - Full access including managing permissions and reading secrets.
  - Grants `Edit` actions plus:
    - `alert.notifications.receivers.secrets:read`
    - `receivers.permissions:read`
    - `receivers.permissions:write`
    - `alert.notifications.receivers.protected:write`
- Empty string (`""`) - Removes the permission.

### Notification policies

| Method | URI                                  | Name                                                          | Summary                                                          |
| ------ | ------------------------------------ | ------------------------------------------------------------- | ---------------------------------------------------------------- |
| DELETE | /api/v1/provisioning/policies        | [route reset policy tree](#route-reset-policy-tree)           | Clears the notification policy tree.                             |
| GET    | /api/v1/provisioning/policies        | [route get policy tree](#route-get-policy-tree)               | Get the notification policy tree.                                |
| PUT    | /api/v1/provisioning/policies        | [route put policy tree](#route-put-policy-tree)               | Sets the notification policy tree.                               |
| GET    | /api/v1/provisioning/policies/export | [route get policy tree export](#route-get-policy-tree-export) | Export the notification policy tree in provisioning file format. |

**Example Request for exporting the notification policy tree in YAML format:**

```http
GET /api/v1/provisioning/policies/export?format=yaml
Accept: application/json
Content-Type: application/json
Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
```

**Example Response:**

```http
HTTP/1.1 200 OK
Content-Type: text/yaml

apiVersion: 1
policies:
    - orgId: 1
      receiver: My Contact Email Point
      group_by:
        - grafana_folder
        - alertname
      routes:
        - receiver: My Contact Email Point
          object_matchers:
            - - monitor
              - =
              - testdata
          mute_time_intervals:
            - weekends
```

### Notification template groups

Template groups enable you to define multiple notification templates (`{{ define "" }}`) within a single group. They can be managed from the Grafana Alerting UI.

| Method | URI                                  | Name                                            | Summary                                         |
| ------ | ------------------------------------ | ----------------------------------------------- | ----------------------------------------------- |
| DELETE | /api/v1/provisioning/templates/:name | [route delete template](#route-delete-template) | Delete a notification template group.           |
| GET    | /api/v1/provisioning/templates/:name | [route get template](#route-get-template)       | Get a notification template group.              |
| GET    | /api/v1/provisioning/templates       | [route get template](#route-get-templates)      | Get all notification template groups.           |
| PUT    | /api/v1/provisioning/templates/:name | [route put template](#route-put-template)       | Create or update a notification template group. |

**Example Request for all notification template groups:**

```http
GET /api/v1/provisioning/templates
Accept: application/json
Content-Type: application/json
Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
```

**Example Response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "name": "custom_email.message",
    "template": "{{ define \"custom_email.message\" }}\n  Custom alert!\n{{ end }}",
    "provenance": "file"
  },
  {
    "name": "custom_email.subject",
    "template": "{{ define \"custom_email.subject\" }}\n{{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s)\n{{ end }}",
    "provenance": "file"
  }
]
```

### Mute timings

| Method | URI                                            | Name                                                            | Summary                                              |
| ------ | ---------------------------------------------- | --------------------------------------------------------------- | ---------------------------------------------------- |
| DELETE | /api/v1/provisioning/mute-timings/:name        | [route delete mute timing](#route-delete-mute-timing)           | Delete a mute timing.                                |
| GET    | /api/v1/provisioning/mute-timings/:name        | [route get mute timing](#route-get-mute-timing)                 | Get a mute timing.                                   |
| GET    | /api/v1/provisioning/mute-timings              | [route get mute timings](#route-get-mute-timings)               | Get all the mute timings.                            |
| POST   | /api/v1/provisioning/mute-timings              | [route post mute timing](#route-post-mute-timing)               | Create a new mute timing.                            |
| PUT    | /api/v1/provisioning/mute-timings/:name        | [route put mute timing](#route-put-mute-timing)                 | Replace an existing mute timing.                     |
| GET    | /api/v1/provisioning/mute-timings/export       | [route get mute timings export](#route-get-mute-timings-export) | Export all mute timings in provisioning file format. |
| GET    | /api/v1/provisioning/mute-timings/:name/export | [route get mute timing export](#route-get-mute-timing-export)   | Export a mute timing in provisioning file format.    |

**Example Request for all mute timings:**

```http
GET /api/v1/provisioning/mute-timings
Accept: application/json
Content-Type: application/json
Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
```

**Example Response:**

```http
HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "name": "weekends",
    "time_intervals": [
      {
        "weekdays": [
          "saturday",
          "sunday"
        ]
      }
    ],
    "version": "",
    "provenance": "file"
  }
]
```

### Edit resources in the Grafana UI

By default, you cannot edit API-provisioned alerting resources in Grafana.

To enable editing these resources in the Grafana UI, add the **`X-Disable-Provenance: true`** header to the following API requests:

- [`PUT /api/v1/provisioning/folder/{FolderUID}/rule-groups/{Group}`](#route-put-alert-rule-group): This action also sets the provenance for the rule group and all its alert rules.
- [`POST /api/v1/provisioning/alert-rules`](#route-post-alert-rule): The provenance of the new alert rule must match the provenance value configured for its rule group.
- [`POST /api/v1/provisioning/contact-points`](##route-post-contactpoints)
- [`POST /api/v1/provisioning/mute-timings`](#route-post-mute-timing)
- [`PUT /api/v1/provisioning/templates/{name}`](#route-put-template)
- [`PUT /api/v1/provisioning/policies`](#route-put-policy-tree)

To reset the notification policy tree to the default and unlock it for editing in the Grafana UI, use:

- [`DELETE /api/v1/provisioning/policies`](#route-reset-policy-tree)

## Data source-managed resources

{{< admonition type="caution" >}}
Pre-provisioned Loki and Prometheus data source-managed alerts have been deprecated in Grafana Cloud and can not be created in new stacks.
New Grafana Cloud stacks will use Grafana-managed alerting (GMA) by default. Datasource-managed alerting (DMA) is not provisioned in new stacks. Existing stacks are not affected.

This applies to the default Loki and Prometheus Grafana Cloud data sources managed by Grafana Labs and Cloud Alertmanager, which will not be available nor will Grafana provision the data source for it. If you add your own Mimir, Loki, or Alertmanager data sources, you can continue to use data source-managed alerts.

Cloud users can import DMA rules into GMA rules with the import tool. See the [import data source-managed alerts to Grafana-managed alerts documentation](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/alerting-migration/) for information on how to do this.
{{< /admonition >}}

## Paths

### <span id="route-delete-alert-rule"></span> Delete a specific alert rule by UID. (_RouteDeleteAlertRule_)

```
DELETE /api/v1/provisioning/alert-rules/:uid
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/rules.alerting.grafana.app-v1beta1.json) instead: `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/alertrules/{name}` for alert rules and `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/recordingrules/{name}` for recording rules.
{{< /admonition >}}

#### Parameters

{{% responsive-table %}}

| Name                         | Source | Type   | Go type | Required | Default | Description                                               |
| ---------------------------- | ------ | ------ | ------- | :------: | ------- | --------------------------------------------------------- |
| `UID`                        | path   | string | string  |    ✓     |         | Alert rule UID                                            |
| `X-Disable-Provenance: true` | header | string | string  |          |         | Allows editing of provisioned resources in the Grafana UI |

{{% /responsive-table %}}

#### All responses

| Code                                | Status     | Description                              | Has headers | Schema                                        |
| ----------------------------------- | ---------- | ---------------------------------------- | :---------: | --------------------------------------------- |
| [204](#route-delete-alert-rule-204) | No Content | The alert rule was deleted successfully. |             | [schema](#route-delete-alert-rule-204-schema) |

#### Responses

##### <span id="route-delete-alert-rule-204"></span> 204 - The alert rule was deleted successfully.

Status: No Content

###### <span id="route-delete-alert-rule-204-schema"></span> Schema

### <span id="route-delete-alert-rule-group"></span> Delete a rule group. (_RouteDeleteAlertRuleGroup_)

```
DELETE /api/v1/provisioning/folder/:folderUid/rule-groups/:group
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/rules.alerting.grafana.app-v1beta1.json) instead: `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/alertrules` for alert rules and `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/recordingrules` for recording rules.
{{< /admonition >}}

#### Parameters

{{% responsive-table %}}

| Name                         | Source | Type   | Go type | Required | Default | Description                                               |
| ---------------------------- | ------ | ------ | ------- | :------: | ------- | --------------------------------------------------------- |
| `FolderUID`                  | path   | string | string  |    ✓     |         |                                                           |
| `Group`                      | path   | string | string  |    ✓     |         |                                                           |
| `X-Disable-Provenance: true` | header | string | string  |          |         | Allows editing of provisioned resources in the Grafana UI |

{{% /responsive-table %}}

#### All responses

| Code                                      | Status     | Description                                    | Has headers | Schema                                              |
| ----------------------------------------- | ---------- | ---------------------------------------------- | :---------: | --------------------------------------------------- |
| [204](#route-delete-alert-rule-group-204) | No Content | The alert rule group was deleted successfully. |             | [schema](#route-delete-alert-rule-group-204-schema) |
| [403](#route-delete-alert-rule-group-403) | Forbidden  | ForbiddenError                                 |             | [schema](#route-delete-alert-rule-group-403-schema) |
| [404](#route-delete-alert-rule-group-404) | Not Found  | NotFound                                       |             | [schema](#route-delete-alert-rule-group-404-schema) |

#### Responses

##### <span id="route-delete-alert-rule-group-204"></span> 204 - The alert rule group was deleted successfully.

Status: No Content

###### <span id="route-delete-alert-rule-group-204-schema"></span> Schema

##### <span id="route-delete-alert-rule-group-403"></span> 403 - ForbiddenError

Status: Forbidden

###### <span id="route-delete-alert-rule-group-403-schema"></span> Schema

[ForbiddenError](#forbidden-error)

##### <span id="route-delete-alert-rule-group-404"></span> 404 - NotFound

Status: Not Found

###### <span id="route-delete-alert-rule-group-404-schema"></span> Schema

[NotFound](#not-found)

### <span id="route-delete-contactpoints"></span> Delete a contact point. (_RouteDeleteContactpoints_)

```
DELETE /api/v1/provisioning/contact-points/:uid
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/receivers/{name}` for receivers.
{{< /admonition >}}

#### Parameters

| Name  | Source | Type   | Go type | Required | Default | Description                                |
| ----- | ------ | ------ | ------- | :------: | ------- | ------------------------------------------ |
| `UID` | path   | string | string  |    ✓     |         | UID is the contact point unique identifier |

#### All responses

| Code                                   | Status     | Description                                 | Has headers | Schema                                           |
| -------------------------------------- | ---------- | ------------------------------------------- | :---------: | ------------------------------------------------ |
| [204](#route-delete-contactpoints-204) | No Content | The contact point was deleted successfully. |             | [schema](#route-delete-contactpoints-204-schema) |

#### Responses

##### <span id="route-delete-contactpoints-204"></span> 204 - The contact point was deleted successfully.

Status: No Content

###### <span id="route-delete-contactpoints-204-schema"></span> Schema

### <span id="route-delete-mute-timing"></span> Delete a mute timing. (_RouteDeleteMuteTiming_)

```
DELETE /api/v1/provisioning/mute-timings/:name
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/timeintervals/{name}` for mute timings.
{{< /admonition >}}

#### Parameters

| Name      | Source | Type   | Go type | Required | Default | Description                                                                                                   |
| --------- | ------ | ------ | ------- | :------: | ------- | ------------------------------------------------------------------------------------------------------------- |
| `name`    | path   | string | string  |    ✓     |         | Mute timing name                                                                                              |
| `version` | query  | string | string  |          |         | Current version of the resource. Used for optimistic concurrency validation. Keep empty to bypass validation. |

#### All responses

| Code                                 | Status     | Description                               | Has headers | Schema                                         |
| ------------------------------------ | ---------- | ----------------------------------------- | :---------: | ---------------------------------------------- |
| [204](#route-delete-mute-timing-204) | No Content | The mute timing was deleted successfully. |             | [schema](#route-delete-mute-timing-204-schema) |
| [409](#route-delete-mute-timing-409) | Conflict   | GenericPublicError                        |             | [schema](#route-delete-mute-timing-409-schema) |

#### Responses

##### <span id="route-delete-mute-timing-204"></span> 204 - The mute timing was deleted successfully.

Status: No Content

###### <span id="route-delete-mute-timing-204-schema"></span> Schema

##### <span id="route-delete-mute-timing-409"></span> 409 - Conflict

Status: Conflict

###### <span id="route-delete-mute-timing-409-schema"></span> Schema

[GenericPublicError](#generic-public-error)

### <span id="route-delete-template"></span> Delete a notification template group. (_RouteDeleteTemplate_)

```
DELETE /api/v1/provisioning/templates/:name
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/templategroups/{name}` for notification template groups.
{{< /admonition >}}

#### Parameters

| Name      | Source | Type   | Go type | Required | Default | Description                                                                                                   |
| --------- | ------ | ------ | ------- | :------: | ------- | ------------------------------------------------------------------------------------------------------------- |
| `name`    | path   | string | string  |    ✓     |         | Name of the template group                                                                                    |
| `version` | query  | string | string  |          |         | Current version of the resource. Used for optimistic concurrency validation. Keep empty to bypass validation. |

#### All responses

| Code                              | Status     | Description                            | Has headers | Schema                                      |
| --------------------------------- | ---------- | -------------------------------------- | :---------: | ------------------------------------------- |
| [204](#route-delete-template-204) | No Content | The template was deleted successfully. |             | [schema](#route-delete-template-204-schema) |
| [409](#route-delete-template-409) | Conflict   | GenericPublicError                     |             | [schema](#route-delete-template-409-schema) |

#### Responses

##### <span id="route-delete-template-204"></span> 204 - The template was deleted successfully.

Status: No Content

###### <span id="route-delete-template-204-schema"></span> Schema

##### <span id="route-delete-template-409"></span> 409 - Conflict

Status: Conflict

###### <span id="route-delete-template-409-schema"></span> Schema

[GenericPublicError](#generic-public-error)

### <span id="route-get-alert-rule"></span> Get a specific alert rule by UID. (_RouteGetAlertRule_)

```
GET /api/v1/provisioning/alert-rules/:uid
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/rules.alerting.grafana.app-v1beta1.json) instead: `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/alertrules/{name}` for alert rules and `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/recordingrules/{name}` for recording rules.
{{< /admonition >}}

#### Parameters

| Name  | Source | Type   | Go type | Required | Default | Description    |
| ----- | ------ | ------ | ------- | :------: | ------- | -------------- |
| `UID` | path   | string | string  |    ✓     |         | Alert rule UID |

#### All responses

| Code                             | Status    | Description          | Has headers | Schema                                     |
| -------------------------------- | --------- | -------------------- | :---------: | ------------------------------------------ |
| [200](#route-get-alert-rule-200) | OK        | ProvisionedAlertRule |             | [schema](#route-get-alert-rule-200-schema) |
| [404](#route-get-alert-rule-404) | Not Found | Not found.           |             | [schema](#route-get-alert-rule-404-schema) |

#### Responses

##### <span id="route-get-alert-rule-200"></span> 200 - ProvisionedAlertRule

Status: OK

###### <span id="route-get-alert-rule-200-schema"></span> Schema

[ProvisionedAlertRule](#provisioned-alert-rule)

##### <span id="route-get-alert-rule-404"></span> 404 - Not found.

Status: Not Found

###### <span id="route-get-alert-rule-404-schema"></span> Schema

### <span id="route-get-alert-rule-export"></span> Export an alert rule in provisioning file format. (_RouteGetAlertRuleExport_)

```
GET /api/v1/provisioning/alert-rules/:uid/export
```

{{< docs/shared lookup="alerts/alerting-provisioning-export-produces.md" source="grafana" version="<GRAFANA_VERSION>" >}}

#### Parameters

| Name       | Source | Type    | Go type | Required | Default | Description                                                                                                                                  |
| ---------- | ------ | ------- | ------- | :------: | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `UID`      | path   | string  | string  |    ✓     |         | Alert rule UID                                                                                                                               |
| `download` | query  | boolean | `bool`  |          |         | Whether to initiate a download of the file or not.                                                                                           |
| `format`   | query  | string  | string  |          | `yaml`  | Format of the downloaded file, either `yaml`, `json` or `hcl`. Accept header can also be used, but the query parameter will take precedence. |

#### All responses

| Code                                    | Status    | Description        | Has headers | Schema                                            |
| --------------------------------------- | --------- | ------------------ | :---------: | ------------------------------------------------- |
| [200](#route-get-alert-rule-export-200) | OK        | AlertingFileExport |             | [schema](#route-get-alert-rule-export-200-schema) |
| [404](#route-get-alert-rule-export-404) | Not Found | Not found.         |             | [schema](#route-get-alert-rule-export-404-schema) |

#### Responses

##### <span id="route-get-alert-rule-export-200"></span> 200 - AlertingFileExport

Status: OK

###### <span id="route-get-alert-rule-export-200-schema"></span> Schema

[AlertingFileExport](#alerting-file-export)

##### <span id="route-get-alert-rule-export-404"></span> 404 - Not found.

Status: Not Found

###### <span id="route-get-alert-rule-export-404-schema"></span> Schema

### <span id="route-get-alert-rule-group"></span> Get a rule group. (_RouteGetAlertRuleGroup_)

```
GET /api/v1/provisioning/folder/:folderUid/rule-groups/:group
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/rules.alerting.grafana.app-v1beta1.json) instead: `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/alertrules` for alert rules and `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/recordingrules` for recording rules.
{{< /admonition >}}

#### Parameters

| Name        | Source | Type   | Go type | Required | Default | Description |
| ----------- | ------ | ------ | ------- | :------: | ------- | ----------- |
| `FolderUID` | path   | string | string  |    ✓     |         |             |
| `Group`     | path   | string | string  |    ✓     |         |             |

#### All responses

| Code                                   | Status    | Description    | Has headers | Schema                                           |
| -------------------------------------- | --------- | -------------- | :---------: | ------------------------------------------------ |
| [200](#route-get-alert-rule-group-200) | OK        | AlertRuleGroup |             | [schema](#route-get-alert-rule-group-200-schema) |
| [404](#route-get-alert-rule-group-404) | Not Found | Not found.     |             | [schema](#route-get-alert-rule-group-404-schema) |

#### Responses

##### <span id="route-get-alert-rule-group-200"></span> 200 - AlertRuleGroup

Status: OK

###### <span id="route-get-alert-rule-group-200-schema"></span> Schema

[AlertRuleGroup](#alert-rule-group)

##### <span id="route-get-alert-rule-group-404"></span> 404 - Not found.

Status: Not Found

###### <span id="route-get-alert-rule-group-404-schema"></span> Schema

### <span id="route-get-alert-rule-group-export"></span> Export an alert rule group in provisioning file format. (_RouteGetAlertRuleGroupExport_)

```
GET /api/v1/provisioning/folder/:folderUid/rule-groups/:group/export
```

{{< docs/shared lookup="alerts/alerting-provisioning-export-produces.md" source="grafana" version="<GRAFANA_VERSION>" >}}

#### Parameters

| Name        | Source | Type    | Go type | Required | Default | Description                                                                                                                                  |
| ----------- | ------ | ------- | ------- | :------: | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `FolderUID` | path   | string  | string  |    ✓     |         |                                                                                                                                              |
| `Group`     | path   | string  | string  |    ✓     |         |                                                                                                                                              |
| `download`  | query  | boolean | `bool`  |          |         | Whether to initiate a download of the file or not.                                                                                           |
| `format`    | query  | string  | string  |          | `yaml`  | Format of the downloaded file, either `yaml`, `json` or `hcl`. Accept header can also be used, but the query parameter will take precedence. |

#### All responses

| Code                                          | Status    | Description        | Has headers | Schema                                                  |
| --------------------------------------------- | --------- | ------------------ | :---------: | ------------------------------------------------------- |
| [200](#route-get-alert-rule-group-export-200) | OK        | AlertingFileExport |             | [schema](#route-get-alert-rule-group-export-200-schema) |
| [404](#route-get-alert-rule-group-export-404) | Not Found | Not found.         |             | [schema](#route-get-alert-rule-group-export-404-schema) |

#### Responses

##### <span id="route-get-alert-rule-group-export-200"></span> 200 - AlertingFileExport

Status: OK

###### <span id="route-get-alert-rule-group-export-200-schema"></span> Schema

[AlertingFileExport](#alerting-file-export)

##### <span id="route-get-alert-rule-group-export-404"></span> 404 - Not found.

Status: Not Found

###### <span id="route-get-alert-rule-group-export-404-schema"></span> Schema

### <span id="route-get-alert-rules"></span> Get all the alert rules. (_RouteGetAlertRules_)

```
GET /api/v1/provisioning/alert-rules
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/rules.alerting.grafana.app-v1beta1.json) instead: `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/alertrules` for alert rules and `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/recordingrules` for recording rules.
{{< /admonition >}}

#### All responses

| Code                              | Status | Description           | Has headers | Schema                                      |
| --------------------------------- | ------ | --------------------- | :---------: | ------------------------------------------- |
| [200](#route-get-alert-rules-200) | OK     | ProvisionedAlertRules |             | [schema](#route-get-alert-rules-200-schema) |

#### Responses

##### <span id="route-get-alert-rules-200"></span> 200 - ProvisionedAlertRules

Status: OK

###### <span id="route-get-alert-rules-200-schema"></span> Schema

[ProvisionedAlertRules](#provisioned-alert-rules)

### <span id="route-get-alert-rules-export"></span> Export all alert rules in provisioning file format. (_RouteGetAlertRulesExport_)

```
GET /api/v1/provisioning/alert-rules/export
```

{{< docs/shared lookup="alerts/alerting-provisioning-export-produces.md" source="grafana" version="<GRAFANA_VERSION>" >}}

#### Parameters

| Name       | Source | Type    | Go type | Required | Default | Description                                                                                                                                  |
| ---------- | ------ | ------- | ------- | :------: | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `download` | query  | boolean | `bool`  |          |         | Whether to initiate a download of the file or not.                                                                                           |
| `format`   | query  | string  | string  |          | `yaml`  | Format of the downloaded file, either `yaml`, `json` or `hcl`. Accept header can also be used, but the query parameter will take precedence. |

#### All responses

| Code                                     | Status    | Description        | Has headers | Schema                                             |
| ---------------------------------------- | --------- | ------------------ | :---------: | -------------------------------------------------- |
| [200](#route-get-alert-rules-export-200) | OK        | AlertingFileExport |             | [schema](#route-get-alert-rules-export-200-schema) |
| [404](#route-get-alert-rules-export-404) | Not Found | Not found.         |             | [schema](#route-get-alert-rules-export-404-schema) |

#### Responses

##### <span id="route-get-alert-rules-export-200"></span> 200 - AlertingFileExport

Status: OK

###### <span id="route-get-alert-rules-export-200-schema"></span> Schema

[AlertingFileExport](#alerting-file-export)

##### <span id="route-get-alert-rules-export-404"></span> 404 - Not found.

Status: Not Found

###### <span id="route-get-alert-rules-export-404-schema"></span> Schema

### <span id="route-get-contactpoints"></span> Get all the contact points. (_RouteGetContactpoints_)

```
GET /api/v1/provisioning/contact-points
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/receivers` for receivers.
{{< /admonition >}}

#### Parameters

| Name   | Source | Type   | Go type | Required | Default | Description    |
| ------ | ------ | ------ | ------- | :------: | ------- | -------------- |
| `name` | query  | string | string  |          |         | Filter by name |

#### All responses

| Code                                | Status | Description   | Has headers | Schema                                        |
| ----------------------------------- | ------ | ------------- | :---------: | --------------------------------------------- |
| [200](#route-get-contactpoints-200) | OK     | ContactPoints |             | [schema](#route-get-contactpoints-200-schema) |

#### Responses

##### <span id="route-get-contactpoints-200"></span> 200 - ContactPoints

Status: OK

###### <span id="route-get-contactpoints-200-schema"></span> Schema

[ContactPoints](#contact-points)

### <span id="route-get-contactpoints-export"></span> Export all contact points in provisioning file format. (_RouteGetContactpointsExport_)

```
GET /api/v1/provisioning/contact-points/export
```

{{< docs/shared lookup="alerts/alerting-provisioning-export-produces.md" source="grafana" version="<GRAFANA_VERSION>" >}}

#### Parameters

| Name       | Source | Type    | Go type | Required | Default | Description                                                                                                                                                                                     |
| ---------- | ------ | ------- | ------- | :------: | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `decrypt`  | query  | boolean | `bool`  |          |         | Whether any contained secure settings should be decrypted or left redacted. Redacted settings will contain RedactedValue instead. Currently, only org admin can view decrypted secure settings. |
| `download` | query  | boolean | `bool`  |          |         | Whether to initiate a download of the file or not.                                                                                                                                              |
| `format`   | query  | string  | string  |          | `yaml`  | Format of the downloaded file, either `yaml`, `json` or `hcl`. Accept header can also be used, but the query parameter will take precedence.                                                    |
| `name`     | query  | string  | string  |          |         | Filter by name                                                                                                                                                                                  |

#### All responses

| Code                                       | Status    | Description        | Has headers | Schema                                               |
| ------------------------------------------ | --------- | ------------------ | :---------: | ---------------------------------------------------- |
| [200](#route-get-contactpoints-export-200) | OK        | AlertingFileExport |             | [schema](#route-get-contactpoints-export-200-schema) |
| [403](#route-get-contactpoints-export-403) | Forbidden | PermissionDenied   |             | [schema](#route-get-contactpoints-export-403-schema) |

#### Responses

##### <span id="route-get-contactpoints-export-200"></span> 200 - AlertingFileExport

Status: OK

###### <span id="route-get-contactpoints-export-200-schema"></span> Schema

[AlertingFileExport](#alerting-file-export)

##### <span id="route-get-contactpoints-export-403"></span> 403 - PermissionDenied

Status: Forbidden

###### <span id="route-get-contactpoints-export-403-schema"></span> Schema

[PermissionDenied](#permission-denied)

### <span id="route-set-user-receiver-permission"></span> Set permissions for a user on a specific receiver. (_RouteSetUserReceiverPermission_)

```
POST /api/access-control/receivers/:uid/users/:userID
```

This endpoint sets or removes permissions for a specific user on a receiver. To remove a permission, send an empty string as the permission value.

{{< admonition type="note" >}}
This endpoint requires Grafana Enterprise and the user must have the `receivers.permissions:write` permission for the specified receiver.
{{< /admonition >}}

#### Parameters

| Name     | Source | Type   | Go type | Required | Default | Description                                                                           |
| -------- | ------ | ------ | ------- | :------: | ------- | ------------------------------------------------------------------------------------- |
| `uid`    | path   | string | string  |    ✓     |         | UID of the receiver                                                                   |
| `userID` | path   | int64  | int64   |    ✓     |         | ID of the user to assign permissions to                                               |
| `body`   | body   | object | object  |    ✓     |         | JSON body with `permission` field: `View`, `Edit`, `Admin`, or `""` (empty to remove) |

#### All responses

| Code                                           | Status      | Description        | Has headers | Schema                                                   |
| ---------------------------------------------- | ----------- | ------------------ | :---------: | -------------------------------------------------------- |
| [200](#route-set-user-receiver-permission-200) | OK          | Permission updated |             | [schema](#route-set-user-receiver-permission-200-schema) |
| [400](#route-set-user-receiver-permission-400) | Bad Request | Invalid request    |             | [schema](#route-set-user-receiver-permission-400-schema) |
| [403](#route-set-user-receiver-permission-403) | Forbidden   | Permission denied  |             | [schema](#route-set-user-receiver-permission-403-schema) |

#### Responses

##### <span id="route-set-user-receiver-permission-200"></span> 200 - Permission updated

Status: OK

###### <span id="route-set-user-receiver-permission-200-schema"></span> Schema

```json
{
  "message": "Permission updated"
}
```

##### <span id="route-set-user-receiver-permission-400"></span> 400 - Invalid request

Status: Bad Request

###### <span id="route-set-user-receiver-permission-400-schema"></span> Schema

##### <span id="route-set-user-receiver-permission-403"></span> 403 - Permission denied

Status: Forbidden

###### <span id="route-set-user-receiver-permission-403-schema"></span> Schema

[PermissionDenied](#permission-denied)

### <span id="route-set-team-receiver-permission"></span> Set permissions for a team on a specific receiver. (_RouteSetTeamReceiverPermission_)

```
POST /api/access-control/receivers/:uid/teams/:teamID
```

This endpoint sets or removes permissions for a specific team on a receiver. To remove a permission, send an empty string as the permission value.

{{< admonition type="note" >}}
This endpoint requires Grafana Enterprise and the user must have the `receivers.permissions:write` permission for the specified receiver.
{{< /admonition >}}

#### Parameters

| Name     | Source | Type   | Go type | Required | Default | Description                                                                           |
| -------- | ------ | ------ | ------- | :------: | ------- | ------------------------------------------------------------------------------------- |
| `uid`    | path   | string | string  |    ✓     |         | UID of the receiver                                                                   |
| `teamID` | path   | int64  | int64   |    ✓     |         | ID of the team to assign permissions to                                               |
| `body`   | body   | object | object  |    ✓     |         | JSON body with `permission` field: `View`, `Edit`, `Admin`, or `""` (empty to remove) |

#### All responses

| Code                                           | Status      | Description        | Has headers | Schema                                                   |
| ---------------------------------------------- | ----------- | ------------------ | :---------: | -------------------------------------------------------- |
| [200](#route-set-team-receiver-permission-200) | OK          | Permission updated |             | [schema](#route-set-team-receiver-permission-200-schema) |
| [400](#route-set-team-receiver-permission-400) | Bad Request | Invalid request    |             | [schema](#route-set-team-receiver-permission-400-schema) |
| [403](#route-set-team-receiver-permission-403) | Forbidden   | Permission denied  |             | [schema](#route-set-team-receiver-permission-403-schema) |

#### Responses

##### <span id="route-set-team-receiver-permission-200"></span> 200 - Permission updated

Status: OK

###### <span id="route-set-team-receiver-permission-200-schema"></span> Schema

```json
{
  "message": "Permission updated"
}
```

##### <span id="route-set-team-receiver-permission-400"></span> 400 - Invalid request

Status: Bad Request

###### <span id="route-set-team-receiver-permission-400-schema"></span> Schema

##### <span id="route-set-team-receiver-permission-403"></span> 403 - Permission denied

Status: Forbidden

###### <span id="route-set-team-receiver-permission-403-schema"></span> Schema

[PermissionDenied](#permission-denied)

### <span id="route-set-builtin-receiver-permission"></span> Set permissions for a built-in role on a specific receiver. (_RouteSetBuiltinReceiverPermission_)

```
POST /api/access-control/receivers/:uid/builtInRoles/:builtInRole
```

This endpoint sets or removes permissions for a built-in role on a receiver. To remove a permission, send an empty string as the permission value.

{{< admonition type="note" >}}
This endpoint requires Grafana Enterprise and the user must have the `receivers.permissions:write` permission for the specified receiver.
{{< /admonition >}}

#### Parameters

| Name          | Source | Type   | Go type | Required | Default | Description                                                                           |
| ------------- | ------ | ------ | ------- | :------: | ------- | ------------------------------------------------------------------------------------- |
| `uid`         | path   | string | string  |    ✓     |         | UID of the receiver                                                                   |
| `builtInRole` | path   | string | string  |    ✓     |         | Built-in role name: `Viewer`, `Editor`, or `Admin`                                    |
| `body`        | body   | object | object  |    ✓     |         | JSON body with `permission` field: `View`, `Edit`, `Admin`, or `""` (empty to remove) |

#### All responses

| Code                                              | Status      | Description        | Has headers | Schema                                                      |
| ------------------------------------------------- | ----------- | ------------------ | :---------: | ----------------------------------------------------------- |
| [200](#route-set-builtin-receiver-permission-200) | OK          | Permission updated |             | [schema](#route-set-builtin-receiver-permission-200-schema) |
| [400](#route-set-builtin-receiver-permission-400) | Bad Request | Invalid request    |             | [schema](#route-set-builtin-receiver-permission-400-schema) |
| [403](#route-set-builtin-receiver-permission-403) | Forbidden   | Permission denied  |             | [schema](#route-set-builtin-receiver-permission-403-schema) |

#### Responses

##### <span id="route-set-builtin-receiver-permission-200"></span> 200 - Permission updated

Status: OK

###### <span id="route-set-builtin-receiver-permission-200-schema"></span> Schema

```json
{
  "message": "Permission updated"
}
```

##### <span id="route-set-builtin-receiver-permission-400"></span> 400 - Invalid request

Status: Bad Request

###### <span id="route-set-builtin-receiver-permission-400-schema"></span> Schema

##### <span id="route-set-builtin-receiver-permission-403"></span> 403 - Permission denied

Status: Forbidden

###### <span id="route-set-builtin-receiver-permission-403-schema"></span> Schema

[PermissionDenied](#permission-denied)

### <span id="route-get-mute-timing"></span> Get a mute timing. (_RouteGetMuteTiming_)

```
GET /api/v1/provisioning/mute-timings/:name
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/timeintervals/{name}` for mute timings.
{{< /admonition >}}

#### Parameters

| Name   | Source | Type   | Go type | Required | Default | Description      |
| ------ | ------ | ------ | ------- | :------: | ------- | ---------------- |
| `name` | path   | string | string  |    ✓     |         | Mute timing name |

#### All responses

| Code                              | Status    | Description      | Has headers | Schema                                      |
| --------------------------------- | --------- | ---------------- | :---------: | ------------------------------------------- |
| [200](#route-get-mute-timing-200) | OK        | MuteTimeInterval |             | [schema](#route-get-mute-timing-200-schema) |
| [404](#route-get-mute-timing-404) | Not Found | Not found.       |             | [schema](#route-get-mute-timing-404-schema) |

#### Responses

##### <span id="route-get-mute-timing-200"></span> 200 - MuteTimeInterval

Status: OK

###### <span id="route-get-mute-timing-200-schema"></span> Schema

[MuteTimeInterval](#mute-time-interval)

##### <span id="route-get-mute-timing-404"></span> 404 - Not found.

Status: Not Found

###### <span id="route-get-mute-timing-404-schema"></span> Schema

### <span id="route-get-mute-timings"></span> Get all the mute timings. (_RouteGetMuteTimings_)

```
GET /api/v1/provisioning/mute-timings
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/timeintervals` for mute timings.
{{< /admonition >}}

#### All responses

| Code                               | Status | Description | Has headers | Schema                                       |
| ---------------------------------- | ------ | ----------- | :---------: | -------------------------------------------- |
| [200](#route-get-mute-timings-200) | OK     | MuteTimings |             | [schema](#route-get-mute-timings-200-schema) |

#### Responses

##### <span id="route-get-mute-timings-200"></span> 200 - MuteTimings

Status: OK

###### <span id="route-get-mute-timings-200-schema"></span> Schema

[MuteTimings](#mute-timings)

### <span id="route-get-mute-timings-export"></span> Export all mute timings in provisioning file format. (_RouteGetMuteTimingsExport_)

```
GET /api/v1/provisioning/mute-timings/export
```

{{< docs/shared lookup="alerts/alerting-provisioning-export-produces.md" source="grafana" version="<GRAFANA_VERSION>" >}}

#### Parameters

| Name       | Source | Type    | Go type | Required | Default | Description                                                                                                                                  |
| ---------- | ------ | ------- | ------- | :------: | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `download` | query  | boolean | `bool`  |          |         | Whether to initiate a download of the file or not.                                                                                           |
| `format`   | query  | string  | string  |          | `yaml`  | Format of the downloaded file, either `yaml`, `json` or `hcl`. Accept header can also be used, but the query parameter will take precedence. |

#### All responses

| Code                                      | Status    | Description       | Has headers | Schema                                              |
| ----------------------------------------- | --------- | ----------------- | :---------: | --------------------------------------------------- |
| [200](#route-get-mute-timings-export-200) | OK        | MuteTimingsExport |             | [schema](#route-get-mute-timings-export-200-schema) |
| [403](#route-get-mute-timings-export-403) | Forbidden | PermissionDenied  |             | [schema](#route-get-mute-timings-export-403-schema) |

#### Responses

##### <span id="route-get-mute-timings-export-200"></span> 200 - MuteTimingsExport

Status: OK

###### <span id="route-get-mute-timings-export-200-schema"></span> Schema

[AlertingFileExport](#alerting-file-export)

##### <span id="route-get-mute-timings-export-403"></span> 403 - PermissionDenied

Status: Forbidden

###### <span id="route-get-mute-timings-export-403-schema"></span> Schema

[PermissionDenied](#permission-denied)

### <span id="route-get-mute-timing-export"></span> Export a mute timing in provisioning file format. (_RouteGetMuteTimingExport_)

```
GET /api/v1/provisioning/mute-timings/:name/export
```

{{< docs/shared lookup="alerts/alerting-provisioning-export-produces.md" source="grafana" version="<GRAFANA_VERSION>" >}}

#### Parameters

| Name       | Source | Type    | Go type | Required | Default | Description                                                                                                                                  |
| ---------- | ------ | ------- | ------- | :------: | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`     | path   | string  | string  |    ✓     |         | Mute timing name.                                                                                                                            |
| `download` | query  | boolean | `bool`  |          |         | Whether to initiate a download of the file or not.                                                                                           |
| `format`   | query  | string  | string  |          | `yaml`  | Format of the downloaded file, either `yaml`, `json` or `hcl`. Accept header can also be used, but the query parameter will take precedence. |

#### All responses

| Code                                     | Status    | Description      | Has headers | Schema                                              |
| ---------------------------------------- | --------- | ---------------- | :---------: | --------------------------------------------------- |
| [200](#route-get-mute-timing-export-200) | OK        | MuteTimingExport |             | [schema](#route-get-mute-timings-export-200-schema) |
| [403](#route-get-mute-timing-export-403) | Forbidden | PermissionDenied |             | [schema](#route-get-mute-timings-export-403-schema) |

#### Responses

##### <span id="route-get-mute-timing-export-200"></span> 200 - MuteTimingExport

Status: OK

###### <span id="route-get-mute-timing-export-200-schema"></span> Schema

[AlertingFileExport](#alerting-file-export)

##### <span id="route-get-mute-timing-export-403"></span> 403 - PermissionDenied

Status: Forbidden

###### <span id="route-get-mute-timing-export-403-schema"></span> Schema

[PermissionDenied](#permission-denied)

### <span id="route-get-policy-tree"></span> Get the notification policy tree. (_RouteGetPolicyTree_)

```
GET /api/v1/provisioning/policies
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/routingtrees` for notification policies.
{{< /admonition >}}

#### All responses

| Code                              | Status | Description | Has headers | Schema                                      |
| --------------------------------- | ------ | ----------- | :---------: | ------------------------------------------- |
| [200](#route-get-policy-tree-200) | OK     | Route       |             | [schema](#route-get-policy-tree-200-schema) |

#### Responses

##### <span id="route-get-policy-tree-200"></span> 200 - Route

Status: OK

###### <span id="route-get-policy-tree-200-schema"></span> Schema

[Route](#route)

### <span id="route-get-policy-tree-export"></span> Export the notification policy tree in provisioning file format. (_RouteGetPolicyTreeExport_)

```
GET /api/v1/provisioning/policies/export
```

{{< docs/shared lookup="alerts/alerting-provisioning-export-produces.md" source="grafana" version="<GRAFANA_VERSION>" >}}

#### Parameters

| Name        | Source | Type    | Go type | Required | Default | Description                                                                                                                                  |
| ----------- | ------ | ------- | ------- | :------: | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `download`  | query  | boolean | `bool`  |          |         | Whether to initiate a download of the file or not.                                                                                           |
| `format`    | query  | string  | string  |          | `yaml`  | Format of the downloaded file, either `yaml`, `json` or `hcl`. Accept header can also be used, but the query parameter will take precedence. |
| `routeName` | query  | string  | string  |          |         | Name of a specific routing tree to export. If omitted, the default policy tree is exported.                                                  |

#### All responses

| Code                                     | Status    | Description        | Has headers | Schema                                             |
| ---------------------------------------- | --------- | ------------------ | :---------: | -------------------------------------------------- |
| [200](#route-get-policy-tree-export-200) | OK        | AlertingFileExport |             | [schema](#route-get-policy-tree-export-200-schema) |
| [404](#route-get-policy-tree-export-404) | Not Found | NotFound           |             | [schema](#route-get-policy-tree-export-404-schema) |

#### Responses

##### <span id="route-get-policy-tree-export-200"></span> 200 - AlertingFileExport

Status: OK

###### <span id="route-get-policy-tree-export-200-schema"></span> Schema

[AlertingFileExport](#alerting-file-export)

##### <span id="route-get-policy-tree-export-404"></span> 404 - NotFound

Status: Not Found

###### <span id="route-get-policy-tree-export-404-schema"></span> Schema

[NotFound](#not-found)

### <span id="route-get-template"></span> Get a notification template group. (_RouteGetTemplate_)

```
GET /api/v1/provisioning/templates/:name
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/templategroups/{name}` for notification template groups.
{{< /admonition >}}

#### Parameters

| Name   | Source | Type   | Go type | Required | Default | Description                |
| ------ | ------ | ------ | ------- | :------: | ------- | -------------------------- |
| `name` | path   | string | string  |    ✓     |         | Name of the template group |

#### All responses

| Code                           | Status    | Description          | Has headers | Schema                                   |
| ------------------------------ | --------- | -------------------- | :---------: | ---------------------------------------- |
| [200](#route-get-template-200) | OK        | NotificationTemplate |             | [schema](#route-get-template-200-schema) |
| [404](#route-get-template-404) | Not Found | GenericPublicError   |             | [schema](#route-get-template-404-schema) |

#### Responses

##### <span id="route-get-template-200"></span> 200 - NotificationTemplate

Status: OK

###### <span id="route-get-template-200-schema"></span> Schema

[NotificationTemplate](#notification-template)

##### <span id="route-get-template-404"></span> 404 - Not found.

[GenericPublicError](#generic-public-error)

###### <span id="route-get-template-404-schema"></span> Schema

### <span id="route-get-templates"></span> Get all notification template groups. (_RouteGetTemplates_)

```
GET /api/v1/provisioning/templates
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/templategroups` for notification template groups.
{{< /admonition >}}

#### All responses

| Code                            | Status | Description           | Has headers | Schema                                    |
| ------------------------------- | ------ | --------------------- | :---------: | ----------------------------------------- |
| [200](#route-get-templates-200) | OK     | NotificationTemplates |             | [schema](#route-get-templates-200-schema) |

#### Responses

##### <span id="route-get-templates-200"></span> 200 - NotificationTemplates

Status: OK

###### <span id="route-get-templates-200-schema"></span> Schema

[NotificationTemplates](#notification-templates)

### <span id="route-post-alert-rule"></span> Create a new alert rule. (_RoutePostAlertRule_)

```
POST /api/v1/provisioning/alert-rules
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/rules.alerting.grafana.app-v1beta1.json) instead: `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/alertrules` for alert rules and `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/recordingrules` for recording rules.
{{< /admonition >}}

This action creates a new alert rule.

The provenance (`X-Disable-Provenance`) of the new rule must match the provenance configured for its rule group. Mixing provisioned and unprovisioned alert rules within the same rule group is not allowed.

#### Parameters

{{% responsive-table %}}

| Name                         | Source | Type                                            | Go type                       | Required | Default | Description                                               |
| ---------------------------- | ------ | ----------------------------------------------- | ----------------------------- | :------: | ------- | --------------------------------------------------------- |
| `X-Disable-Provenance: true` | header | string                                          | string                        |          |         | Allows editing of provisioned resources in the Grafana UI |
| `Body`                       | body   | [ProvisionedAlertRule](#provisioned-alert-rule) | `models.ProvisionedAlertRule` |          |         |                                                           |

{{% /responsive-table %}}

#### All responses

| Code                              | Status      | Description          | Has headers | Schema                                      |
| --------------------------------- | ----------- | -------------------- | :---------: | ------------------------------------------- |
| [201](#route-post-alert-rule-201) | Created     | ProvisionedAlertRule |             | [schema](#route-post-alert-rule-201-schema) |
| [400](#route-post-alert-rule-400) | Bad Request | ValidationError      |             | [schema](#route-post-alert-rule-400-schema) |

#### Responses

##### <span id="route-post-alert-rule-201"></span> 201 - ProvisionedAlertRule

Status: Created

###### <span id="route-post-alert-rule-201-schema"></span> Schema

[ProvisionedAlertRule](#provisioned-alert-rule)

##### <span id="route-post-alert-rule-400"></span> 400 - ValidationError

Status: Bad Request

###### <span id="route-post-alert-rule-400-schema"></span> Schema

[ValidationError](#validation-error)

### <span id="route-post-contactpoints"></span> Create a contact point. (_RoutePostContactpoints_)

```
POST /api/v1/provisioning/contact-points
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/receivers` for receivers.
{{< /admonition >}}

When creating a contact point, the `EmbeddedContactPoint.name` property determines if the new contact point is added to an existing one. In the UI, contact points with the same name are grouped together under a single contact point.

#### Parameters

{{% responsive-table %}}

| Name                         | Source | Type                                            | Go type                       | Required | Default | Description                                               |
| ---------------------------- | ------ | ----------------------------------------------- | ----------------------------- | :------: | ------- | --------------------------------------------------------- |
| `X-Disable-Provenance: true` | header | string                                          | string                        |          |         | Allows editing of provisioned resources in the Grafana UI |
| `Body`                       | body   | [EmbeddedContactPoint](#embedded-contact-point) | `models.EmbeddedContactPoint` |          |         |                                                           |

{{% /responsive-table %}}

#### All responses

| Code                                 | Status      | Description          | Has headers | Schema                                         |
| ------------------------------------ | ----------- | -------------------- | :---------: | ---------------------------------------------- |
| [202](#route-post-contactpoints-202) | Accepted    | EmbeddedContactPoint |             | [schema](#route-post-contactpoints-202-schema) |
| [400](#route-post-contactpoints-400) | Bad Request | ValidationError      |             | [schema](#route-post-contactpoints-400-schema) |

#### Responses

##### <span id="route-post-contactpoints-202"></span> 202 - EmbeddedContactPoint

Status: Accepted

###### <span id="route-post-contactpoints-202-schema"></span> Schema

[EmbeddedContactPoint](#embedded-contact-point)

##### <span id="route-post-contactpoints-400"></span> 400 - ValidationError

Status: Bad Request

###### <span id="route-post-contactpoints-400-schema"></span> Schema

[ValidationError](#validation-error)

### <span id="route-post-mute-timing"></span> Create a new mute timing. (_RoutePostMuteTiming_)

```
POST /api/v1/provisioning/mute-timings
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/timeintervals` for mute timings.
{{< /admonition >}}

#### Parameters

{{% responsive-table %}}

| Name                         | Source | Type                                    | Go type                   | Required | Default | Description                                               |
| ---------------------------- | ------ | --------------------------------------- | ------------------------- | :------: | ------- | --------------------------------------------------------- |
| `X-Disable-Provenance: true` | header | string                                  | string                    |          |         | Allows editing of provisioned resources in the Grafana UI |
| `Body`                       | body   | [MuteTimeInterval](#mute-time-interval) | `models.MuteTimeInterval` |          |         |                                                           |

{{% /responsive-table %}}

#### All responses

| Code                               | Status      | Description      | Has headers | Schema                                       |
| ---------------------------------- | ----------- | ---------------- | :---------: | -------------------------------------------- |
| [201](#route-post-mute-timing-201) | Created     | MuteTimeInterval |             | [schema](#route-post-mute-timing-201-schema) |
| [400](#route-post-mute-timing-400) | Bad Request | ValidationError  |             | [schema](#route-post-mute-timing-400-schema) |

#### Responses

##### <span id="route-post-mute-timing-201"></span> 201 - MuteTimeInterval

Status: Created

###### <span id="route-post-mute-timing-201-schema"></span> Schema

[MuteTimeInterval](#mute-time-interval)

##### <span id="route-post-mute-timing-400"></span> 400 - ValidationError

Status: Bad Request

###### <span id="route-post-mute-timing-400-schema"></span> Schema

[ValidationError](#validation-error)

### <span id="route-put-alert-rule"></span> Update an existing alert rule. (_RoutePutAlertRule_)

```
PUT /api/v1/provisioning/alert-rules/:uid
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/rules.alerting.grafana.app-v1beta1.json) instead: `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/alertrules/{name}` for alert rules and `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/recordingrules/{name}` for recording rules.
{{< /admonition >}}

#### Parameters

{{% responsive-table %}}

| Name                         | Source | Type                                            | Go type                       | Required | Default | Description                                               |
| ---------------------------- | ------ | ----------------------------------------------- | ----------------------------- | :------: | ------- | --------------------------------------------------------- | --- |
| `UID`                        | path   | string                                          | string                        |    ✓     |         | Alert rule UID                                            |
| `X-Disable-Provenance: true` | header | string                                          | string                        |          |         | Allows editing of provisioned resources in the Grafana UI |
| `Body`                       | body   | [ProvisionedAlertRule](#provisioned-alert-rule) | `models.ProvisionedAlertRule` |          |         |                                                           |     |

{{% /responsive-table %}}

#### All responses

| Code                             | Status      | Description          | Has headers | Schema                                     |
| -------------------------------- | ----------- | -------------------- | :---------: | ------------------------------------------ |
| [200](#route-put-alert-rule-200) | OK          | ProvisionedAlertRule |             | [schema](#route-put-alert-rule-200-schema) |
| [400](#route-put-alert-rule-400) | Bad Request | ValidationError      |             | [schema](#route-put-alert-rule-400-schema) |

#### Responses

##### <span id="route-put-alert-rule-200"></span> 200 - ProvisionedAlertRule

Status: OK

###### <span id="route-put-alert-rule-200-schema"></span> Schema

[ProvisionedAlertRule](#provisioned-alert-rule)

##### <span id="route-put-alert-rule-400"></span> 400 - ValidationError

Status: Bad Request

###### <span id="route-put-alert-rule-400-schema"></span> Schema

[ValidationError](#validation-error)

### <span id="route-put-alert-rule-group"></span> Create or update a rule group. (_RoutePutAlertRuleGroup_)

```
PUT /api/v1/provisioning/folder/:folderUid/rule-groups/:group
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/rules.alerting.grafana.app-v1beta1.json) instead: `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/alertrules` for alert rules and `/apis/rules.alerting.grafana.app/v1beta1/namespaces/{namespace}/recordingrules` for recording rules.
{{< /admonition >}}

This action also changes the provenance setting (`X-Disable-Provenance`) for all alert rules in the alert group.

#### Parameters

{{% responsive-table %}}

| Name                         | Source | Type                                | Go type                 | Required | Default | Description                                                                                                             |
| ---------------------------- | ------ | ----------------------------------- | ----------------------- | :------: | ------- | ----------------------------------------------------------------------------------------------------------------------- |
| `FolderUID`                  | path   | string                              | string                  |    ✓     |         |                                                                                                                         |
| `Group`                      | path   | string                              | string                  |    ✓     |         |                                                                                                                         |
| `X-Disable-Provenance: true` | header | string                              | string                  |          |         | Allows editing of provisioned resources in the Grafana UI. This also applies to all alert rules within the alert group. |
| `Body`                       | body   | [AlertRuleGroup](#alert-rule-group) | `models.AlertRuleGroup` |          |         | This action is idempotent and rules included in this body will overwrite configured rules for the group                 |

{{% /responsive-table %}}

#### All responses

| Code                                   | Status      | Description     | Has headers | Schema                                           |
| -------------------------------------- | ----------- | --------------- | :---------: | ------------------------------------------------ |
| [200](#route-put-alert-rule-group-200) | OK          | AlertRuleGroup  |             | [schema](#route-put-alert-rule-group-200-schema) |
| [400](#route-put-alert-rule-group-400) | Bad Request | ValidationError |             | [schema](#route-put-alert-rule-group-400-schema) |

#### Responses

##### <span id="route-put-alert-rule-group-200"></span> 200 - AlertRuleGroup

Status: OK

###### <span id="route-put-alert-rule-group-200-schema"></span> Schema

[AlertRuleGroup](#alert-rule-group)

##### <span id="route-put-alert-rule-group-400"></span> 400 - ValidationError

Status: Bad Request

###### <span id="route-put-alert-rule-group-400-schema"></span> Schema

[ValidationError](#validation-error)

### <span id="route-put-contactpoint"></span> Update an existing contact point. (_RoutePutContactpoint_)

```
PUT /api/v1/provisioning/contact-points/:uid
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/receivers/{name}` for receivers.
{{< /admonition >}}

#### Parameters

{{% responsive-table %}}

| Name                         | Source | Type                                            | Go type                       | Required | Default | Description                                               |
| ---------------------------- | ------ | ----------------------------------------------- | ----------------------------- | :------: | ------- | --------------------------------------------------------- |
| `UID`                        | path   | string                                          | string                        |    ✓     |         | UID is the contact point unique identifier                |
| `X-Disable-Provenance: true` | header | string                                          | string                        |          |         | Allows editing of provisioned resources in the Grafana UI |
| `Body`                       | body   | [EmbeddedContactPoint](#embedded-contact-point) | `models.EmbeddedContactPoint` |          |         |                                                           |

{{% /responsive-table %}}

#### All responses

| Code                               | Status      | Description     | Has headers | Schema                                       |
| ---------------------------------- | ----------- | --------------- | :---------: | -------------------------------------------- |
| [202](#route-put-contactpoint-202) | Accepted    | Ack             |             | [schema](#route-put-contactpoint-202-schema) |
| [400](#route-put-contactpoint-400) | Bad Request | ValidationError |             | [schema](#route-put-contactpoint-400-schema) |

#### Responses

##### <span id="route-put-contactpoint-202"></span> 202 - Ack

Status: Accepted

###### <span id="route-put-contactpoint-202-schema"></span> Schema

[Ack](#ack)

##### <span id="route-put-contactpoint-400"></span> 400 - ValidationError

Status: Bad Request

###### <span id="route-put-contactpoint-400-schema"></span> Schema

[ValidationError](#validation-error)

### <span id="route-put-mute-timing"></span> Replace an existing mute timing. (_RoutePutMuteTiming_)

```
PUT /api/v1/provisioning/mute-timings/:name
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/timeintervals/{name}` for mute timings.
{{< /admonition >}}

#### Parameters

{{% responsive-table %}}

| Name                         | Source | Type                                    | Go type                   | Required | Default | Description                                               |
| ---------------------------- | ------ | --------------------------------------- | ------------------------- | -------- | :-----: | --------------------------------------------------------- |
| `name`                       | path   | string                                  | string                    | ✓        |         | Mute timing name                                          |
| `X-Disable-Provenance: true` | header | string                                  | string                    |          |         | Allows editing of provisioned resources in the Grafana UI |
| `Body`                       | body   | [MuteTimeInterval](#mute-time-interval) | `models.MuteTimeInterval` |          |         |                                                           |

{{% /responsive-table %}}

#### All responses

| Code                              | Status      | Description        | Has headers | Schema                                      |
| --------------------------------- | ----------- | ------------------ | :---------: | ------------------------------------------- |
| [200](#route-put-mute-timing-200) | OK          | MuteTimeInterval   |             | [schema](#route-put-mute-timing-200-schema) |
| [400](#route-put-mute-timing-400) | Bad Request | ValidationError    |             | [schema](#route-put-mute-timing-400-schema) |
| [409](#route-put-mute-timing-409) | Conflict    | GenericPublicError |             | [schema](#route-put-mute-timing-409-schema) |

#### Responses

##### <span id="route-put-mute-timing-200"></span> 200 - MuteTimeInterval

Status: OK

###### <span id="route-put-mute-timing-200-schema"></span> Schema

[MuteTimeInterval](#mute-time-interval)

##### <span id="route-put-mute-timing-400"></span> 400 - ValidationError

Status: Bad Request

###### <span id="route-put-mute-timing-400-schema"></span> Schema

[ValidationError](#validation-error)

##### <span id="route-put-mute-timing-409"></span> 409 - Conflict

Status: Conflict

###### <span id="route-put-mute-timing-409-schema"></span> Schema

[GenericPublicError](#generic-public-error)

### <span id="route-put-policy-tree"></span> Sets the notification policy tree. (_RoutePutPolicyTree_)

{{< docs/shared lookup="alerts/warning-provisioning-tree.md" source="grafana" version="<GRAFANA_VERSION>" >}}

```
PUT /api/v1/provisioning/policies
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/routingtrees` for notification policies.
{{< /admonition >}}

#### Parameters

{{% responsive-table %}}

| Name                         | Source | Type            | Go type        | Required | Default | Description                                               |
| ---------------------------- | ------ | --------------- | -------------- | :------: | ------- | --------------------------------------------------------- |
| `X-Disable-Provenance: true` | header | string          | string         |          |         | Allows editing of provisioned resources in the Grafana UI |
| `Body`                       | body   | [Route](#route) | `models.Route` |          |         | The new notification routing tree to use                  |

{{% /responsive-table %}}

#### All responses

| Code                              | Status      | Description     | Has headers | Schema                                      |
| --------------------------------- | ----------- | --------------- | :---------: | ------------------------------------------- |
| [202](#route-put-policy-tree-202) | Accepted    | Ack             |             | [schema](#route-put-policy-tree-202-schema) |
| [400](#route-put-policy-tree-400) | Bad Request | ValidationError |             | [schema](#route-put-policy-tree-400-schema) |

#### Responses

##### <span id="route-put-policy-tree-202"></span> 202 - Ack

Status: Accepted

###### <span id="route-put-policy-tree-202-schema"></span> Schema

[Ack](#ack)

##### <span id="route-put-policy-tree-400"></span> 400 - ValidationError

Status: Bad Request

###### <span id="route-put-policy-tree-400-schema"></span> Schema

[ValidationError](#validation-error)

### <span id="route-put-template"></span> Create or update a notification template group. (_RoutePutTemplate_)

```
PUT /api/v1/provisioning/templates/:name
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/templategroups/{name}` for notification template groups.
{{< /admonition >}}

{{% responsive-table %}}

#### Parameters

| Name                         | Source | Type                                                          | Go type                              | Required | Default | Description                                               |
| ---------------------------- | ------ | ------------------------------------------------------------- | ------------------------------------ | -------- | :-----: | --------------------------------------------------------- | --- |
| `name`                       | path   | string                                                        | string                               | ✓        |         | Name of the template group                                |
| `X-Disable-Provenance: true` | header | string                                                        | string                               |          |         | Allows editing of provisioned resources in the Grafana UI |
| `Body`                       | body   | [NotificationTemplateContent](#notification-template-content) | `models.NotificationTemplateContent` |          |         |                                                           |     |

{{% /responsive-table %}}

#### All responses

| Code                           | Status      | Description          | Has headers | Schema                                   |
| ------------------------------ | ----------- | -------------------- | :---------: | ---------------------------------------- |
| [202](#route-put-template-202) | Accepted    | NotificationTemplate |             | [schema](#route-put-template-202-schema) |
| [400](#route-put-template-400) | Bad Request | GenericPublicError   |             | [schema](#route-put-template-400-schema) |
| [409](#route-put-template-409) | Conflict    | GenericPublicError   |             | [schema](#route-put-template-409-schema) |

#### Responses

##### <span id="route-put-template-202"></span> 202 - NotificationTemplate

Status: Accepted

###### <span id="route-put-template-202-schema"></span> Schema

[NotificationTemplate](#notification-template)

##### <span id="route-put-template-400"></span> 400 - ValidationError

Status: Bad Request

###### <span id="route-put-template-400-schema"></span> Schema

[GenericPublicError](#generic-public-error)

##### <span id="route-put-template-409"></span> 409 - Conflict

Status: Conflict

###### <span id="route-put-template-409-schema"></span> Schema

[GenericPublicError](#generic-public-error)

### <span id="route-reset-policy-tree"></span> Clears the notification policy tree. (_RouteResetPolicyTree_)

```
DELETE /api/v1/provisioning/policies
```

{{< admonition type="warning" >}}
This API is deprecated and will be removed in a future release. Use the [Grafana App Platform alerting APIs](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/packages/grafana-openapi/src/apis/notifications.alerting.grafana.app-v1beta1.json) instead: `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/routingtrees` for notification policies.
{{< /admonition >}}

#### All responses

| Code                                | Status   | Description | Has headers | Schema                                        |
| ----------------------------------- | -------- | ----------- | :---------: | --------------------------------------------- |
| [202](#route-reset-policy-tree-202) | Accepted | Ack         |             | [schema](#route-reset-policy-tree-202-schema) |

#### Responses

##### <span id="route-reset-policy-tree-202"></span> 202 - Ack

Status: Accepted

###### <span id="route-reset-policy-tree-202-schema"></span> Schema

[Ack](#ack)

## Models

### <span id="ack"></span> Ack

[interface{}](#interface)

### <span id="alert-query"></span> AlertQuery

**Properties**

{{% responsive-table %}}

| Name                | Type                                      | Go type             | Required | Default | Description                                                                                                          | Example |
| ------------------- | ----------------------------------------- | ------------------- | :------: | ------- | -------------------------------------------------------------------------------------------------------------------- | ------- |
| `datasourceUid`     | string                                    | string              |          |         | Grafana data source unique identifier; it should be '**expr**' for a Server Side Expression operation.               |         |
| `model`             | [interface{}](#interface)                 | `interface{}`       |          |         | JSON is the raw JSON query and includes the above properties as well as custom properties.                           |         |
| `queryType`         | string                                    | string              |          |         | QueryType is an optional identifier for the type of query. It can be used to distinguish different types of queries. |         |
| `refId`             | string                                    | string              |          |         | RefID is the unique identifier of the query, set by the frontend call.                                               |         |
| `relativeTimeRange` | [RelativeTimeRange](#relative-time-range) | `RelativeTimeRange` |          |         |                                                                                                                      |         |

{{% /responsive-table %}}

### <span id="alert-query-export"></span> AlertQueryExport

**Properties**

{{% responsive-table %}}

| Name                | Type                                      | Go type             | Required | Default | Description | Example |
| ------------------- | ----------------------------------------- | ------------------- | :------: | ------- | ----------- | ------- |
| `datasourceUid`     | string                                    | string              |          |         |             |         |
| `model`             | [interface{}](#interface)                 | `interface{}`       |          |         |             |         |
| `queryType`         | string                                    | string              |          |         |             |         |
| `refId`             | string                                    | string              |          |         |             |         |
| `relativeTimeRange` | [RelativeTimeRange](#relative-time-range) | `RelativeTimeRange` |          |         |             |         |

{{% /responsive-table %}}

### <span id="alert-rule-export"></span> AlertRuleExport

**Properties**

{{% responsive-table %}}

| Name            | Type                                      | Go type               | Required | Default | Description                                                                                         | Example |
| --------------- | ----------------------------------------- | --------------------- | :------: | ------- | --------------------------------------------------------------------------------------------------- | ------- |
| `annotations`   | map of string                             | `map[string]string`   |          |         |                                                                                                     |         |
| `condition`     | string                                    | string                |          |         |                                                                                                     |         |
| `dashboardUid`  | string                                    | string                |          |         |                                                                                                     |         |
| `data`          | [][AlertQueryExport](#alert-query-export) | `[]*AlertQueryExport` |          |         |                                                                                                     |         |
| `execErrState`  | string                                    | string                |          |         |                                                                                                     |         |
| `for`           | [Duration](#duration)                     | Duration              |          |         |                                                                                                     |         |
| `keepFiringFor` | [Duration](#duration)                     | Duration              |          |         | How long the alert continues to fire after the condition is no longer met. Prevents alert flapping. | `2m`    |
| `isPaused`      | boolean                                   | `bool`                |          |         |                                                                                                     |         |
| `labels`        | map of string                             | `map[string]string`   |          |         |                                                                                                     |         |
| `noDataState`   | string                                    | string                |          |         |                                                                                                     |         |
| `panelId`       | int64 (formatted integer)                 | int64                 |          |         |                                                                                                     |         |
| `title`         | string                                    | string                |          |         |                                                                                                     |         |
| `uid`           | string                                    | string                |          |         |                                                                                                     |         |

{{% /responsive-table %}}

### <span id="alert-rule-group"></span> AlertRuleGroup

**Properties**

{{% responsive-table %}}

| Name        | Type                                              | Go type                   | Required | Default | Description | Example |
| ----------- | ------------------------------------------------- | ------------------------- | :------: | ------- | ----------- | ------- |
| `folderUid` | string                                            | string                    |          |         |             |         |
| `interval`  | int64 (formatted integer)                         | int64                     |          |         |             |         |
| `rules`     | [][ProvisionedAlertRule](#provisioned-alert-rule) | `[]*ProvisionedAlertRule` |          |         |             |         |
| `title`     | string                                            | string                    |          |         |             |         |

{{% /responsive-table %}}

### <span id="alert-rule-group-export"></span> AlertRuleGroupExport

**Properties**

{{% responsive-table %}}

| Name       | Type                                    | Go type              | Required | Default | Description | Example |
| ---------- | --------------------------------------- | -------------------- | :------: | ------- | ----------- | ------- |
| `folder`   | string                                  | string               |          |         |             |         |
| `interval` | [Duration](#duration)                   | Duration             |          |         |             |         |
| `name`     | string                                  | string               |          |         |             |         |
| `orgId`    | int64 (formatted integer)               | int64                |          |         |             |         |
| `rules`    | [][AlertRuleExport](#alert-rule-export) | `[]*AlertRuleExport` |          |         |             |         |

{{% /responsive-table %}}

### <span id="alerting-file-export"></span> AlertingFileExport

**Properties**

{{% responsive-table %}}

| Name            | Type                                                      | Go type                       | Required | Default | Description | Example |
| --------------- | --------------------------------------------------------- | ----------------------------- | :------: | ------- | ----------- | ------- |
| `apiVersion`    | int64 (formatted integer)                                 | int64                         |          |         |             |         |
| `contactPoints` | [][ContactPointExport](#contact-point-export)             | `[]*ContactPointExport`       |          |         |             |         |
| `groups`        | [][AlertRuleGroupExport](#alert-rule-group-export)        | `[]*AlertRuleGroupExport`     |          |         |             |         |
| `policies`      | [][NotificationPolicyExport](#notification-policy-export) | `[]*NotificationPolicyExport` |          |         |             |         |

{{% /responsive-table %}}

### <span id="contact-point-export"></span> ContactPointExport

**Properties**

| Name        | Type                                 | Go type             | Required | Default | Description | Example |
| ----------- | ------------------------------------ | ------------------- | :------: | ------- | ----------- | ------- |
| `name`      | string                               | string              |          |         |             |         |
| `orgId`     | int64 (formatted integer)            | int64               |          |         |             |         |
| `receivers` | [][ReceiverExport](#receiver-export) | `[]*ReceiverExport` |          |         |             |         |

### <span id="contact-points"></span> ContactPoints

[][EmbeddedContactPoint](#embedded-contact-point)

### <span id="duration"></span> Duration

| Name       | Type   | Go type | Default | Description | Example |
| ---------- | ------ | ------- | ------- | ----------- | ------- |
| `Duration` | string | int64   |         |             |         |

### <span id="embedded-contact-point"></span> EmbeddedContactPoint

EmbeddedContactPoint is the contact point type used by Grafana-managed alerts.

When creating a contact point, the `EmbeddedContactPoint.name` property determines if the new contact point is added to an existing one. In the UI, contact points with the same name are grouped together under a single contact point.

**Properties**

{{% responsive-table %}}

| Name                    | Type          | Go type | Required | Default | Description                                                                        | Example                 |
| ----------------------- | ------------- | ------- | :------: | ------- | ---------------------------------------------------------------------------------- | ----------------------- |
| `disableResolveMessage` | boolean       | `bool`  |          |         |                                                                                    | false                   |
| `name`                  | string        | string  |          |         | `name` groups multiple contact points with the same name in the UI.                | `webhook_1`             |
| `provenance`            | string        | string  |          |         |                                                                                    |                         |
| `settings`              | [JSON](#json) | JSON    |    ✓     |         |                                                                                    |                         |
| `type`                  | string        | string  |    ✓     |         |                                                                                    | `webhook`               |
| `uid`                   | string        | string  |          |         | UID is the unique identifier of the contact point. The UID can be set by the user. | `my_external_reference` |

{{% /responsive-table %}}

### <span id="json"></span> Json

[interface{}](#interface)

### <span id="match-regexps"></span> MatchRegexps

[MatchRegexps](#match-regexps)

### <span id="match-type"></span> MatchType

| Name        | Type                      | Go type | Default | Description | Example |
| ----------- | ------------------------- | ------- | ------- | ----------- | ------- |
| `MatchType` | int64 (formatted integer) | int64   |         |             |         |

### <span id="matcher"></span> Matcher

**Properties**

{{% responsive-table %}}

| Name    | Type                     | Go type   | Required | Default | Description | Example |
| ------- | ------------------------ | --------- | :------: | ------- | ----------- | ------- |
| `Name`  | string                   | string    |          |         |             |         |
| `Type`  | [MatchType](#match-type) | MatchType |          |         |             |         |
| `Value` | string                   | string    |          |         |             |         |

{{% /responsive-table %}}

### <span id="matchers"></span> Matchers

> Matchers is a slice of Matchers that is sortable, implements Stringer, and
> provides a Matches method to match a LabelSet against all Matchers in the
> slice. Note that some users of Matchers might require it to be sorted.

[][Matcher](#matcher)

### <span id="mute-time-interval"></span> MuteTimeInterval

**Properties**

{{% responsive-table %}}

| Name             | Type                             | Go type           | Required | Default | Description         | Example |
| ---------------- | -------------------------------- | ----------------- | :------: | ------- | ------------------- | ------- |
| `name`           | string                           | string            |          |         |                     |         |
| `time_intervals` | [][TimeInterval](#time-interval) | `[]*TimeInterval` |          |         |                     |         |
| `version`        | string                           | string            |          |         | Version of resource |         |

{{% /responsive-table %}}

### <span id="mute-timing-export"></span> MuteTimingExport

**Properties**

### <span id="mute-timings-export"></span> MuteTimingsExport

**Properties**

### <span id="mute-timings"></span> MuteTimings

[][MuteTimeInterval](#mute-time-interval)

### <span id="not-found"></span> NotFound

[interface{}](#interface)

### <span id="notification-policy-export"></span> NotificationPolicyExport

**Properties**

| Name     | Type                         | Go type                      | Required | Default | Description | Example |
| -------- | ---------------------------- | ---------------------------- | :------: | ------- | ----------- | ------- |
| `Policy` | [RouteExport](#route-export) | [RouteExport](#route-export) |          |         | inline      |         |
| `orgId`  | int64 (formatted integer)    | int64                        |          |         |             |         |

### <span id="notification-template"></span> NotificationTemplate

**Properties**

{{% responsive-table %}}

| Name         | Type                      | Go type                   | Required | Default | Description         | Example |
| ------------ | ------------------------- | ------------------------- | :------: | ------- | ------------------- | ------- |
| `name`       | string                    | string                    |          |         |                     |         |
| `provenance` | [Provenance](#provenance) | [Provenance](#provenance) |          |         |                     |         |
| `template`   | string                    | string                    |          |         |                     |         |
| `version`    | string                    | string                    |          |         | Version of resource |         |

{{% /responsive-table %}}

### <span id="notification-template-content"></span> NotificationTemplateContent

**Properties**

{{% responsive-table %}}

| Name       | Type   | Go type | Required | Default | Description                                             | Example |
| ---------- | ------ | ------- | :------: | ------- | ------------------------------------------------------- | ------- |
| `template` | string | string  |          |         |                                                         |         |
| `version`  | string | string  |          |         | Version of resource. Should be empty for new templates. |         |

{{% /responsive-table %}}

### <span id="notification-templates"></span> NotificationTemplates

[][NotificationTemplate](#notification-template)

### <span id="object-matchers"></span> ObjectMatchers

[Matchers](#matchers)

#### Inlined models

### <span id="permission-denied"></span> PermissionDenied

[interface{}](#interface)

### <span id="provenance"></span> Provenance

| Name         | Type   | Go type | Default | Description | Example |
| ------------ | ------ | ------- | ------- | ----------- | ------- |
| `Provenance` | string | string  |         |             |         |

### <span id="provisioned-alert-rule"></span> ProvisionedAlertRule

**Properties**

{{% responsive-table %}}

| Name          | Type                         | Go type                   | Required | Default | Description                                                                                                               | Example                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ------------- | ---------------------------- | ------------------------- | :------: | ------- | ------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `annotations` | map of string                | `map[string]string`       |          |         | Optional key-value pairs. `__dashboardUid__` and `__panelId__` must be set together; one cannot be set without the other. | `{"runbook_url":"https://supercoolrunbook.com/page/13"}`                                                                                                                                                                                                                                                                                                                                                                         |
| `condition`   | string                       | string                    |    ✓     |         |                                                                                                                           | `A`                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `data`        | [][AlertQuery](#alert-query) | `[]*AlertQuery`           |    ✓     |         |                                                                                                                           | `[{"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0,0],"type":"gt"},"operator":{"type":"and"},"query":{"params":[]},"reducer":{"params":[],"type":"avg"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"1 == 1","hide":false,"intervalMs":1000,"maxDataPoints":43200,"refId":"A","type":"math"},"queryType":"","refId":"A","relativeTimeRange":{"from":0,"to":0}}]` |
| execErrState  | string                       | string                    |    ✓     |         |                                                                                                                           |                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `folderUID`   | string                       | string                    |    ✓     |         |                                                                                                                           | `project_x`                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `for`         | [Duration](#duration)        | [Duration](#duration)     |    ✓     |         |                                                                                                                           |                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `id`          | int64 (formatted integer)    | int64                     |          |         |                                                                                                                           |                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `isPaused`    | boolean                      | `bool`                    |          |         |                                                                                                                           | `false`                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `labels`      | map of string                | `map[string]string`       |          |         |                                                                                                                           | `{"team":"sre-team-1"}`                                                                                                                                                                                                                                                                                                                                                                                                          |
| `noDataState` | string                       | string                    |    ✓     |         |                                                                                                                           |                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `orgID`       | int64 (formatted integer)    | `int64                    |    ✓     |         |                                                                                                                           |                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `provenance`  | [Provenance](#provenance)    | [Provenance](#provenance) |          |         |                                                                                                                           |                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `record`      | [Record](#record)            | [Record](#record)         |          |         | Recording rule configuration. If present, this is a recording rule instead of an alert rule.                              | `{"metric":"http_requests:rate5m:sum","from":"A","target_datasource_uid":"my-prom"}`                                                                                                                                                                                                                                                                                                                                             |
| `ruleGroup`   | string                       | string                    |    ✓     |         |                                                                                                                           | `eval_group_1`                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `title`       | string                       | string                    |    ✓     |         |                                                                                                                           | `Always firing`                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `uid`         | string                       | string                    |          |         |                                                                                                                           |                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `updated`     | date-time (formatted string) | `strfmt.DateTime`         |          |         |                                                                                                                           |                                                                                                                                                                                                                                                                                                                                                                                                                                  |

{{% /responsive-table %}}

### <span id="provisioned-alert-rules"></span> ProvisionedAlertRules

[][ProvisionedAlertRule](#provisioned-alert-rule)

### <span id="raw-message"></span> RawMessage

[interface{}](#interface)

### <span id="record"></span> Record

> Record defines the configuration for a recording rule, which pre-computes query results and saves them as a new time series metric.

**Properties**

{{% responsive-table %}}

| Name                    | Type   | Go type | Required | Default | Description                                                                                                          | Example                        |
| ----------------------- | ------ | ------- | :------: | ------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| `metric`                | string | string  |    ✓     |         | The name of the new metric to create. Must be a valid Prometheus metric name with no whitespace.                     | `http_requests:rate5m:sum`     |
| `from`                  | string | string  |    ✓     |         | The query reference ID (refId) to use as the source for the recorded metric.                                         | `A`                            |
| `target_datasource_uid` | string | string  |          |         | UID of the Prometheus-compatible data source to write results to. Falls back to configured default if not specified. | `my-prometheus-datasource-uid` |

{{% /responsive-table %}}

### <span id="receiver-export"></span> ReceiverExport

**Properties**

| Name                    | Type                       | Go type    | Required | Default | Description | Example |
| ----------------------- | -------------------------- | ---------- | :------: | ------- | ----------- | ------- |
| `disableResolveMessage` | boolean                    | `bool`     |          |         |             |         |
| `settings`              | [RawMessage](#raw-message) | RawMessage |          |         |             |         |
| `type`                  | string                     | string     |          |         |             |         |
| `uid`                   | string                     | string     |          |         |             |         |

### <span id="regexp"></span> Regexp

> A Regexp is safe for concurrent use by multiple goroutines,
> except for configuration methods, such as Longest.

[interface{}](#interface)

### <span id="relative-time-range"></span> RelativeTimeRange

> RelativeTimeRange is the per query start and end time
> for requests.

**Properties**

{{% responsive-table %}}

| Name   | Type                  | Go type  | Required | Default | Description | Example |
| ------ | --------------------- | -------- | :------: | ------- | ----------- | ------- |
| `from` | [Duration](#duration) | Duration |          |         |             |         |
| `to`   | [Duration](#duration) | Duration |          |         |             |         |

{{% /responsive-table %}}

### <span id="route"></span> Route

> A Route is a node that contains definitions of how to handle alerts. This is modified
> from the upstream alertmanager in that it adds the ObjectMatchers property.

**Properties**

{{% responsive-table %}}

| Name                  | Type                               | Go type             | Required | Default | Description                             | Example |
| --------------------- | ---------------------------------- | ------------------- | :------: | ------- | --------------------------------------- | ------- |
| `continue`            | boolean                            | `bool`              |          |         |                                         |         |
| `group_by`            | []string                           | `[]string`          |          |         |                                         |         |
| `group_interval`      | string                             | string              |          |         |                                         |         |
| `group_wait`          | string                             | string              |          |         |                                         |         |
| `match`               | map of string                      | `map[string]string` |          |         | Deprecated. Remove before v1.0 release. |         |
| `match_re`            | [MatchRegexps](#match-regexps)     | `MatchRegexps`      |          |         |                                         |         |
| `matchers`            | [Matchers](#matchers)              | `Matchers`          |          |         |                                         |         |
| `mute_time_intervals` | []string                           | `[]string`          |          |         |                                         |         |
| `object_matchers`     | [ObjectMatchers](#object-matchers) | `ObjectMatchers`    |          |         |                                         |         |
| `provenance`          | [Provenance](#provenance)          | Provenance          |          |         |                                         |         |
| `receiver`            | string                             | string              |          |         |                                         |         |
| `repeat_interval`     | string                             | string              |          |         |                                         |         |
| `routes`              | [][Route](#route)                  | `[]*Route`          |          |         |                                         |         |

{{% /responsive-table %}}

### <span id="route-export"></span> RouteExport

> RouteExport is the provisioned file export of definitions.Route. This is needed to hide fields that aren't usable in
> provisioning file format. An alternative would be to define a custom MarshalJSON and MarshalYAML that excludes them.

**Properties**

| Name                  | Type                               | Go type             | Required | Default | Description                             | Example |
| --------------------- | ---------------------------------- | ------------------- | :------: | ------- | --------------------------------------- | ------- |
| `continue`            | boolean                            | `bool`              |          |         |                                         |         |
| `group_by`            | []string                           | `[]string`          |          |         |                                         |         |
| `group_interval`      | string                             | string              |          |         |                                         |         |
| `group_wait`          | string                             | string              |          |         |                                         |         |
| `match`               | map of string                      | `map[string]string` |          |         | Deprecated. Remove before v1.0 release. |         |
| `match_re`            | [MatchRegexps](#match-regexps)     | `MatchRegexps`      |          |         |                                         |         |
| `matchers`            | [Matchers](#matchers)              | `Matchers`          |          |         |                                         |         |
| `mute_time_intervals` | []string                           | `[]string`          |          |         |                                         |         |
| `object_matchers`     | [ObjectMatchers](#object-matchers) | `ObjectMatchers`    |          |         |                                         |         |
| `receiver`            | string                             | string              |          |         |                                         |         |
| `repeat_interval`     | string                             | string              |          |         |                                         |         |
| `routes`              | [][RouteExport](#route-export)     | `[]*RouteExport`    |          |         |                                         |         |

### <span id="time-interval"></span> TimeInterval

> TimeInterval describes intervals of time. ContainsTime will tell you if a golang time is contained
> within the interval.

**Properties**

{{% responsive-table %}}

| Name            | Type                       | Go type        | Required | Default | Description | Example |
| --------------- | -------------------------- | -------------- | :------: | ------- | ----------- | ------- |
| `days_of_month` | []string                   | []string       |          |         |             |         |
| `location`      | string                     | string         |          |         |             |         |
| `months`        | []string                   | []string       |          |         |             |         |
| `times`         | [][TimeRange](#time-range) | `[]*TimeRange` |          |         |             |         |
| `weekdays`      | []string                   | []string       |          |         |             |         |
| `years`         | []string                   | []string       |          |         |             |         |

{{% /responsive-table %}}

### <span id="time-range"></span> TimeRange

> For example, 4:00PM to End of the day would Begin at 1020 and End at 1440.

**Properties**

{{% responsive-table %}}

| Name         | Type   | Go type | Required | Default | Description | Example                 |
| ------------ | ------ | ------- | :------: | ------- | ----------- | ----------------------- |
| `end_time`   | string | string  |          |         |             | `"end_time": "24:00"`   |
| `start_time` | string | string  |          |         |             | `"start_time": "18:00"` |

{{% /responsive-table %}}

### <span id="validation-error"></span> ValidationError

**Properties**

{{% responsive-table %}}

| Name  | Type   | Go type | Required | Default | Description | Example         |
| ----- | ------ | ------- | :------: | ------- | ----------- | --------------- |
| `msg` | string | string  |          |         |             | `error message` |

{{% /responsive-table %}}

### <span id="generic-public-error"></span> GenericPublicError

**Properties**

{{% responsive-table %}}

| Name         | Type       | Go type          | Required | Default | Description                                                              | Example |
| ------------ | ---------- | ---------------- | :------: | ------- | ------------------------------------------------------------------------ | ------- |
| `statusCode` | string     | string           |    ✓     |         | HTTP Status Code                                                         |         |
| `messageId`  | string     | string           |    ✓     |         | Unique code of the error                                                 |         |
| `message`    | string     | string           |          |         | Error message                                                            |         |
| `extra`      | map of any | `map[string]any` |          |         | Extra information about the error. Format is specific to the error code. |         |

{{% /responsive-table %}}

# Set alert rule name

1. Click **Alerts & IRM** -> **Alert rules** -> **+ New alert rule**.
1. Enter a name to identify your alert rule.

   This name is displayed in the alert rule list. It is also the `alertname` label for every alert instance that is created from this rule.

# Configure notification message

Use [annotations](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#annotations) to add information to alert messages that can help respond to the alert.

Annotations are included by default in notification messages, and can use text or [templates](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/) to display dynamic data from queries.

Grafana provides several optional annotations.

1. Optional: Add a summary.

   Short summary of what happened and why.

1. Optional: Add a description.

   Description of what the alert rule does.

1. Optional: Add a Runbook URL.

   Webpage where you keep your runbook for the alert

1. Optional: Add a custom annotation.

   Add any additional information that could help address the alert.

1. Optional: **Link dashboard and panel**.

   [Link the alert rule to a panel](/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/link-alert-rules-to-panels/) to facilitate alert investigation.

1. Click **Save rule**.

# Provisioning

### Provisioning

Note that if you delete an alert resource created in the UI, you can no longer retrieve it.

To backup and manage alert rules, you can [provision alerting resources](/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/) using options such as configuration files, Terraform, or the Alerting API.

# Custom webhook example

```go
{- /* Generates a pretty-printed JSON payload with alert group info and individual alerts metadata. */ -}}
{{ define "webhook.custom.payload" -}}
  {{ coll.Dict
  "receiver" .Receiver
  "status" .Status
  "alerts" (tmpl.Exec "webhook.custom.simple_alerts" .Alerts | data.JSON)
  "groupLabels" .GroupLabels
  "commonLabels" .CommonLabels
  "commonAnnotations" .CommonAnnotations
  "externalURL" .ExternalURL
  "version" "1"
  "orgId"  (index .Alerts 0).OrgID
  "truncatedAlerts"  .TruncatedAlerts
  "groupKey" .GroupKey
  "state"  (tmpl.Inline "{{ if eq .Status \"resolved\" }}ok{{ else }}alerting{{ end }}" . )
  "allVariables"  .Vars
  "title" (tmpl.Exec "default.title" . )
  "message" (tmpl.Exec "default.message" . )
  | data.ToJSONPretty " "}}
{{- end }}

{{- /* Embed json templates in other json templates. */ -}}
{{ define "webhook.custom.simple_alerts" -}}
  {{- $alerts := coll.Slice -}}
  {{- range . -}}
    {{ $alerts = coll.Append (coll.Dict
    "status" .Status
    "labels" .Labels
    "startsAt" .StartsAt
    "endsAt" .EndsAt
    ) $alerts}}
  {{- end -}}
  {{- $alerts | data.ToJSON -}}
{{- end }}
```

# TBD

Use the instructions in [Getting started with Grafana](/docs/grafana/<GRAFANA_VERSION>/getting-started/getting-started/) to:

- Install Grafana.
- Log in to Grafana.
- Create your first dashboard.

# Grafana managed alerts

The current alerting system doesn't support high availability. Alert notifications are not deduplicated and load balancing is not supported between instances; for example, silences from one instance will not appear in the other.

## Alert evaluation

Grafana managed alerts are evaluated by the Grafana backend. Rule evaluations are scheduled, according to the alert rule configuration, and queries are evaluated by an engine that is part of core Grafana.

Alerting rules can only query backend data sources with alerting enabled:

- builtin or developed and maintained by grafana: `Graphite`, `Prometheus`, `Loki`, `InfluxDB`, `Elasticsearch`,
  `Google Cloud Monitoring`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL`, `OpenTSDB`, `Oracle`, and `Azure Data Explorer`
- any community backend data sources with alerting enabled (`backend` and `alerting` properties are set in the [plugin.json](https://grafana.com/developers/plugin-tools/reference/plugin-json)

## Metrics from the alerting engine

The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes [internal metrics](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/set-up-grafana-monitoring/). See also, [View alert rules and their current state](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/state-and-health/).

| Metric Name                                         | Type      | Description                                                                              |
| --------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------- |
| `alerting.alerts`                                   | gauge     | How many alerts by state                                                                 |
| `alerting.request_duration_seconds`                 | histogram | Histogram of requests to the Alerting API                                                |
| `alerting.active_configurations`                    | gauge     | The number of active, non default alertmanager configurations for grafana managed alerts |
| `alerting.rule_evaluations_total`                   | counter   | The total number of rule evaluations                                                     |
| `alerting.rule_evaluation_failures_total`           | counter   | The total number of rule evaluation failures                                             |
| `alerting.rule_evaluation_duration_seconds`         | histogram | The time to evaluate a rule                                                              |
| `alerting.rule_process_evaluation_duration_seconds` | histogram | The time to process the evaluation results for a rule                                    |
| `alerting.rule_send_alerts_duration_seconds`        | histogram | The time to send the alerts to Alertmanager                                              |
| `alerting.rule_group_rules`                         | gauge     | The number of rules                                                                      |
| `alerting.state_calculation_duration_seconds`       | histogram | The duration of calculation of a single state                                            |

# How label matching works

{{< collapse title="How label matching works" >}}

Use [labels](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/) and label matchers to link alert rules to [notification policies](/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/) and [silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/). This allows for a flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.

A label matchers consists of 3 distinct parts, the **label**, the **value** and the **operator**.

- The **Label** field is the name of the label to match. It must exactly match the label name.

- The **Value** field matches against the corresponding value for the specified **Label** name. How it matches depends on the **Operator** value.

- The **Operator** field is the operator to match against the label value. The available operators are:

  | Operator | Description                                        |
  | -------- | -------------------------------------------------- |
  | `=`      | Select labels that are exactly equal to the value. |
  | `!=`     | Select labels that are not equal to the value.     |
  | `=~`     | Select labels that regex-match the value.          |
  | `!~`     | Select labels that do not regex-match the value.   |

{{< admonition type="note" >}}
If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match in order to link a rule to a policy.
{{< /admonition >}}

**Label matching example**

If you define the following set of labels for your alert:

`{ foo=bar, baz=qux, id=12 }`

then:

- A label matcher defined as `foo=bar` matches this alert rule.
- A label matcher defined as `foo!=bar` does _not_ match this alert rule.
- A label matcher defined as `id=~[0-9]+` matches this alert rule.
- A label matcher defined as `baz!~[0-9]+` matches this alert rule.
- Two label matchers defined as `foo=bar` and `id=~[0-9]+` match this alert rule.

**Exclude labels**

You can also write label matchers to exclude labels.

Here is an example that shows how to exclude the label `Team`. You can choose between any of the values below to exclude labels.

| Label  | Operator | Value |
| ------ | -------- | ----- |
| `team` | `=`      | `""`  |
| `team` | `!~`     | `.+`  |
| `team` | `=~`     | `^$`  |

{{< /collapse >}}

# Math example

- `$A` returns series `{host="web01"} 30` and `{host="web02"} 20`.
- `$B` returns series `{host="web01"} 10` and `{host="web02"} 0`.
- `$A + $B` returns `{host="web01"} 40` and `{host="web02"} 20`.

# Modify error state

- **Set Error state** (Default)

  If the pending period is `0`, the alert instance transitions immediately to **Error**. Otherwise, it transitions to the **Pending** state, and then to **Error** after the pending period has elapsed and the last evaluation returns an error.

  When entering the **Error** state, the alert rule creates a new `DatasourceError` alert instance with labels for the alert rule name, UID, and data source UID.

- **Set Alerting state**

  If the pending period is `0`, the alert instance transitions immediately to **Alerting**. Otherwise, it transitions to the **Pending** state, and then to **Alerting** after the pending period has elapsed and the last evaluation returns an error or a threshold breach.

- **Set Normal state**

  The alert instance transitions immediately to the **Normal** state.

- **Keep last state**

  The alert instance keeps its current state.

# Modify no data state

- **Set No Data state** (Default)

  If the pending period is `0`, the alert instance transitions immediately to **No Data**. Otherwise, it transitions to the **Pending** state, and then to **No Data** after the pending period has elapsed and the last evaluation returns no data.

  When entering the **No Data** state, the alert rule creates a new `DatasourceNoData` alert instance with labels for the alert rule name, UID, and data source UID.

- **Set Alerting state**

  If the pending period is `0`, the alert instance transitions immediately to **Alerting**. Otherwise, it transitions to the **Pending** state, and then to **Alerting** after the pending period has elapsed and the last evaluation returns no data or a threshold breach.

- **Set Normal state**

  The alert instance transitions immediately to the **Normal** state.

- **Keep last state**

  The alert instance keeps its current state.

# Mute and active timings vs silences

The function of [Mute timing and active timing](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/) differs from [silences](/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/), as they are two are distinct methods to suppress notifications. They do not prevent alert rules from being evaluated or stop alert instances from appearing in the user interface; they only prevent notifications from being created.

The following table highlights the key differences of mute timing and active timing compared with silences.

|            | Mute timing and active timing                               | Silence                                                          |
| ---------- | ----------------------------------------------------------- | ---------------------------------------------------------------- |
| **Setup**  | Created and then added to notification policies             | Matches alerts using labels to determine whether to silence them |
| **Period** | Uses time interval definitions that can repeat periodically | Has a fixed start and end time                                   |

**When to use mute timings**

Use mute timing for predictable, recurring time periods when you don't want to receive notifications:

- Regular maintenance windows (for example, every Sunday from 2:00 AM to 4:00 AM)
- Non-business hours (for example, nights and weekends)
- Scheduled deployments or known change windows
- Regular testing periods

**When to use silences**

Use silences for one-time or as-needed suppression of notifications:

- Active incident response (suppress notifications while investigating)
- Immediate suppression of a specific alert or group of alerts
- One-time maintenance or deployment events
- Temporarily suppress alerts for specific services or components

# Note Dynamic labels

Pending

# JSON alert object

| Key            | Type   | Description                                                                         |
| -------------- | ------ | ----------------------------------------------------------------------------------- |
| `status`       | string | Current status of the alert, `firing` or `resolved`.                                |
| `labels`       | object | Labels that are part of this alert, map of string keys to string values.            |
| `annotations`  | object | Annotations that are part of this alert, map of string keys to string values.       |
| `startsAt`     | string | Start time of the alert.                                                            |
| `endsAt`       | string | End time of the alert, default value when not resolved is `0001-01-01T00:00:00Z`.   |
| `values`       | object | Values that triggered the current status.                                           |
| `generatorURL` | string | URL of the alert rule in the Grafana UI.                                            |
| `fingerprint`  | string | The labels fingerprint, alarms with the same labels will have the same fingerprint. |
| `silenceURL`   | string | URL to silence the alert rule in the Grafana UI.                                    |
| `dashboardURL` | string | A link to the Grafana Dashboard if the alert has a Dashboard UID annotation.        |
| `panelURL`     | string | A link to the panel if the alert has a Panel ID annotation.                         |
| `imageURL`     | string | URL of a screenshot of a panel assigned to the rule that created this notification. |

# Template language

You can use `if` statements in templates. For example, you can print `Variable empty` when a variable is empty:

```go
{{ if $element }}
Element value: {{$element}}
{{ else }}
Element is empty
{{ end }}
```

## With

`with` is similar to `if` statements, but unlike `if`, it updates dot(`.`) to refer to the value of the expression in `with`:

```go
{{ with $array }}
There are {{ len . }} item(s)
{{ else }}
There are no alerts
{{ end }}
```

## Range

`range` iterates over an array or map, and dot (`.`) is set to the current element of the array:

```go
{{ range $array }}
{{ .itemPropertyName }}
{{ end }}
```

Optionally, you can handle empty objects using `else`:

```go
{{ range $array }}
  {{ .itemPropertyName }}
{{ else }}
  Empty array
{{ end }}
```

You can also get the index of each item within a range by defining index and value variables at the start of the range:

```go
{{ $num_items := len $array }}
{{ range $index, $item := $array }}
This is item {{ $index }} out of {{ $num_items }}
{{ end }}
```

Additionally, you can use `{{break}}` to stop the remaining iterations or `{{continue}}` to stop the current iteration and continue with the next one.

## Functions

The global functions available in `text/template` are:

| Function   | Description                                                                                                                                                                    |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `and`      | Returns the boolean AND of its arguments by returning the first empty argument or the last argument.                                                                           |
| `call`     | Returns the result of calling the first argument, which must be a function, with the remaining arguments as parameters.                                                        |
| `html`     | Returns the escaped HTML equivalent of the textual representation of its arguments.                                                                                            |
| `index`    | Returns the result of indexing its first argument by the following arguments, e.g., `{{ index $labels "instance" }}` returns the `instance` key in the `$labels` map variable. |
| `slice`    | Returns the result of slicing its first argument by the remaining arguments.                                                                                                   |
| `js`       | Returns the escaped JavaScript equivalent of the textual representation of its arguments.                                                                                      |
| `len`      | Returns the integer length of its argument, e.g., `{{ len $array }}`                                                                                                           |
| `not`      | Returns the boolean negation of its single argument.                                                                                                                           |
| `or`       | Returns the boolean OR of its arguments by returning the first non-empty argument or the last argument.                                                                        |
| `print`    | An alias for fmt.Sprint                                                                                                                                                        |
| `printf`   | An alias for fmt.Sprintf                                                                                                                                                       |
| `println`  | An alias for fmt.Sprintln                                                                                                                                                      |
| `urlquery` | Returns the escaped value of the textual representation of its arguments in a form suitable for embedding in a URL query                                                       |

For more details, refer to the official documentation on [functions in `text/template`](https://pkg.go.dev/text/template#hdr-Functions).

## Comparison operators

Boolean comparison operators are also available in `text/template`:

| Function | Description                               |
| -------- | ----------------------------------------- |
| `eq`     | Returns the boolean truth of arg1 == arg2 |
| `ne`     | Returns the boolean truth of arg1 != arg2 |
| `lt`     | Returns the boolean truth of arg1 < arg2  |
| `le`     | Returns the boolean truth of arg1 <= arg2 |
| `gt`     | Returns the boolean truth of arg1 > arg2  |
| `ge`     | Returns the boolean truth of arg1 >= arg2 |

## Variables

Variables in `text/template` must be created within the template. For example, you can create a variable with the current value of dot (`.`) and assign a string or another object to the variable like this:

```go
{{ $variable := . }}
{{ $variable := "This is a test" }}
{{ $variable }}
```

This template outputs:

```
This is a test
```

## Templates

You can create reusable templates that can be executed from other templates or within the same template.

Define templates using `define` and the name of the template in double quotes:

```go
{{ define "print_labels" }}
{{ end }}
```

You should not define templates with the same name as other templates, including default templates such as `__subject`, `__text_values_list`, `__text_alert_list`, `default.title` and `default.message`. Where a template has been created with the same name as a default template, or a template in another notification template, Grafana might use either template. Grafana does not prevent, or show an error message, when there are two or more templates with the same name.

### Execute templates

You can execute defined templates using `template`, the name of the template in double quotes, and the cursor that should be passed to the template:

```go
{{ template "print_labels" . }}
```

Within a template dot refers to the value that is passed to the template.

For example, if a template is passed a list of firing alerts then dot refers to that list of firing alerts:

```go
{{ template "print_alerts" .Alerts }}
```

If the template is passed the sorted labels for an alert then dot refers to the list of sorted labels:

```go
{{ template "print_labels" .SortedLabels }}
```

This is useful when writing reusable templates. For example, to print all alerts you might write the following:

```go
{{ template "print_alerts" .Alerts }}
```

Then to print just the firing alerts you could write this:

```go
{{ template "print_alerts" .Alerts.Firing }}
```

This works because both `.Alerts` and `.Alerts.Firing` is a list of alerts.

```go
{{ define "print_alerts" }}
{{ range . }}
{{ template "print_labels" .SortedLabels }}
{{ end }}
{{ end }}
```

{{< admonition type="note" >}}
You cannot create independent, reusable templates for labels and annotations as you can with notification templates. In alert rule templates, you need to write each template inline within the label or annotation field.
{{< /admonition >}}

## Comments

You can add comments with `{{/*` and `*/}}`:

```go
{{/* This is a comment */}}
```

To avoid adding line breaks, use:

```go
{{- /* This is a comment with no leading or trailing line breaks */ -}}
```

## Indentation

You can use indentation, both tabs and spaces, and line breaks, to make templates more readable:

```go
{{ range .Alerts }}
  {{ range .Labels.SortedPairs }}
    {{ .Name }} = {{ .Value }}
  {{ end }}
{{ end }}
```

However, indentation in the template is also present in the text.

### Remove spaces and line breaks

In text/template use `{{-` and `-}}` to remove leading and trailing spaces and line breaks.

For example, when using indentation and line breaks to make a template more readable:

```go
{{ range .Alerts }}
  {{ range .Labels.SortedPairs }}
    {{ .Name }} = {{ .Value }}
  {{ end }}
{{ end }}
```

The indentation and line breaks are also present in the text:

```
    alertname = "Test"

    grafana_folder = "Test alerts"
```

You can remove the indentation and line breaks from the text changing `}}` to `-}}` at the start of each range:

```go
{{ range .Alerts -}}
  {{ range .Labels.SortedPairs -}}
    {{ .Name }} = {{ .Value }}
  {{ end }}
{{ end }}
```

The indentation and line breaks in the template are now absent from the text:

```
alertname = "Test"
grafana_folder = "Test alerts"
```

# Warning Provisioning Tree

{{< admonition type="warning" >}}

Since the policy tree is a single resource, provisioning it will overwrite all policies in the notification policy tree. However, it does not affect internal policies created when alert rules directly select a contact point.

{{< /admonition >}}

# Intro paragraph

There are numerous authentication methods available in Grafana to verify user identity. The authentication configuration dictates which users can access Grafana and the methods they can use for logging in.
You can also configure Grafana to automatically update users' roles and team memberships in Grafana based on the information returned by the auth provider integration.

When deciding on an authentication method, it's important to take into account your current identity and access management system as well as the specific authentication and authorization features you require.
For a complete list of the available authentication options and the features they support, refer to [Configure authentication](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication).

# Back up Grafana

This topic explains how to back up a local Grafana deployment, including configuration, plugin data, and the Grafana database.

## Back up the Grafana configuration file

Copy Grafana configuration files that you might have modified in your Grafana deployment to a backup directory.

The Grafana configuration files are located in the following directories:

- Default configuration: `$WORKING_DIR/defaults.ini` (Don't change this file)
- Custom configuration: `$WORKING_DIR/custom.ini`

For more information on where to find configuration files, refer to [Configuration file location](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#configuration-file-location).

{{< admonition type="note" >}}
If you installed Grafana using the `deb` or `rpm` packages, then your configuration file is located at
`/etc/grafana/grafana.ini`. This path is specified in the Grafana `init.d` script using `--config` file parameter.
{{< /admonition >}}

## Back up plugin data

Installing plugins in Grafana creates a folder for each plugin with its associated files and data. Copy all files and folders recursively from this location to your backup repository.

The Grafana plugin files are located in the following directories:

- Default location for plugins in a binary or source installation: `$WORKING_DIR/data/plugins`
- Default location for plugins in a `deb` or `rpm` package: `/var/lib/grafana/plugins`. This path is specified in the Grafana init.d script using `--config` file parameter.

## Back up the Grafana database

We recommend that you back up your Grafana database so that you can roll back to a previous version, if required.

### SQLite

The default Grafana database is SQLite, which stores its data in a single file on disk. To back up this file, copy it to your backup repository.

{{< admonition type="note" >}}
To ensure data integrity, shut down your Grafana service before backing up the SQLite database.
{{< /admonition >}}

The SQLite database file is located in one of the following directories:

- Default location for SQLite data in a binary or source installation: `$WORKING_DIR/data/grafana.db`
- Default location for SQLite data in a `deb` or `rpm` package: `/var/lib/grafana/grafana.db`. This path is specified in the Grafana
  init.d script using `--config` file parameter.

### MySQL

To back up or restore a MySQL Grafana database, run the following commands:

```bash
backup:
> mysqldump -u root -p[root_password] [grafana] > grafana_backup.sql

restore:
> mysql -u root -p grafana < grafana_backup.sql
```

### Postgres

To back up or restore a Postgres Grafana database, run the following commands:

```bash
backup:
> pg_dump grafana > grafana_backup

restore:
> psql grafana < grafana_backup
```

# Grafana Cloud

Grafana Cloud is a highly available, fast, fully managed OpenSaaS logging and metrics platform. It is everything you love about Grafana, but Grafana Labs hosts it for you and handles all the headaches.

# Grafana Enterprise

Grafana Enterprise is the commercial edition of Grafana that includes additional features not found in the open source version. Building on everything you already know and love about Grafana, Grafana Enterprise adds enterprise data sources, advanced authentication options, more permission controls, 24x7x365 support, and training from the core Grafana team.

[Learn more about Grafana Enterprise](https://grafana.com/enterprise) and contact the Grafana Labs [Sales Team](https://grafana.com/contact?about=grafana-enterprise-stack) to purchase an Enterprise license. You can also obtain a trial license before the purchase.

# Grafana Open Source

Grafana open source is open source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics, logs, and traces no matter where they are stored. It provides you with tools to turn your time-series database (TSDB) data into insightful graphs and visualizations.

### Grafana Loki

Grafana Loki is an open-source set of components that can be composed into a fully featured logging stack. For more information, refer to [Loki documentation](https://grafana.com/docs/loki/<LOKI_VERSION>/).

### Grafana Tempo

Grafana Tempo is an open source, easy-to-use and high-volume distributed tracing backend. For more information, refer to [Tempo documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/).

### Grafana Mimir

Grafana Mimir is an open source software project that provides a scalable long-term storage for Prometheus. For more information about Grafana Mimir, refer to [Grafana Mimir documentation](https://grafana.com/docs/mimir/<MIMIR_VERSION>/).

### Grafana Pyroscope

Grafana Pyroscope is an open source software project for aggregating continuous profiling data. Continuous profiling is an observability signal that helps you understand your workload’s resources usage. For more information, refer to [Grafana Pyroscope documentation](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/).

# Time range URLs

You can control the time range of a dashboard by providing the following query parameters in the dashboard URL:

- `from` - Defines the lower limit of the time range, specified in ms, epoch, or relative time.
- `to` - Defines the upper limit of the time range, specified in ms, epoch, or relative time.
- `time` and `time.window` - Defines a time range from `time-time.window/2` to `time+time.window/2`. Both parameters should be specified in `ms`. For example `?time=1500000000000&time.window=10000` results in a 10-second time range from 1499999995000 to 1500000005000`.
- `timezone` - Defines the time zone. For example `timezone=Europe/Madrid`.

Since these aren't variables, they don't require the `var-` prefix.

The following example shows a dashboard with the time range of the last five minutes:

```
https://${your-domain}/path/to/your/dashboard?from=now-5m&to=now
```

# datasouce-authentication

---
headless: true
labels:
  products:
    - enterprise
    - oss
---

[//]: # 'This file documents the Authentication section for data sources.'
[//]: # 'This shared file is included in these locations:'
[//]: # '/grafana/docs/sources/datasources/pyroscope/configure-pyroscope-data-source.md'
[//]: # '/grafana/docs/sources/datasources/tempo/configure-tempo-data-source.md'
[//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.'
[//]: # 'Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/.'

<!-- Authentication procedure from shared file -->

To set up authentication:

1. Select an authentication method from the drop-down list:
   - **Basic authentication**: Authenticates your data source using a username and password
   - **Forward OAuth identity**: Forwards the OAuth access token and the OIDC ID token, if available, of the user querying to the data source
   - **No authentication**: No authentication is required to access the data source

1. For **Basic authentication** only: Enter the **User** and **Password**.
1. Optional: Complete the **TLS settings** for additional security methods.

   **TLS Client Authentication**
   : Toggle on to use client authentication. When enabled, it adds the **Server name**, **Client cert**, and **Client key** fields. The client provides a certificate that is validated by the server to establish the client's trusted identity. The client key encrypts the data between client and server. These details are encrypted and stored in the Grafana database.

   **Add self-signed certificate**
   : Activate this option to use a self-signed TLS certificate. You can add your own Certificate Authority (CA) certificate on top of one generated by the certificate authorities for additional security measure.

   **Skip TLS certification validation**
   : When activated, it bypasses TLS certificate verification. Not recommended, unless absolutely necessary for testing.
   ![Authentication section showing the TLS client certificate options](/media/docs/grafana/data-sources/tempo/tempo-data-source-authentication.png)

1. Optional: Add **HTTP Headers**. You can pass along additional context and metadata data about the request and response. Select **Add header** to add **Header** and **Value** fields.

1. Select **Save & test** to preserve your changes.

# datasouce-private-ds-connect

---
headless: true
labels:
  products:
    - enterprise
    - oss
---

[//]: # 'This file documents the Private data source section for data sources.'
[//]: # 'This shared file is included in these locations:'
[//]: # '/grafana/docs/sources/datasources/pyroscope/configure-pyroscope-data-source.md'
[//]: # '/grafana/docs/sources/datasources/tempo/configure-tempo-data-source.md'
[//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.'
[//]: # 'Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/.'

<!-- Procedure for using private data source connect section in the data sources -->

{{< admonition type="note" >}}
This feature is only available in Grafana Cloud.
{{< /admonition >}}

Use private data source connect (PDC) to connect to and query data within a secure network without opening that network to inbound traffic from Grafana Cloud.

Refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) for more information on how PDC works and [Configure Grafana private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc) for steps on setting up a PDC connection.

Use the drop-down list to select a configured private data source. If you make changes, select **Test & save** to preserve your changes.

Use **Manage private data source connect** to configure and manage any private data sources you have configured.

# pyroscope-profile-tracing-intro

---
headless: true
labels:
  products:
    - enterprise
    - oss
---

[//]: # 'This file documents the introductory material for traces to profiling for the Pyroscope data source.'
[//]: # 'This shared file is included in these locations:'
[//]: # '/website/docs/grafana-cloud/data-configuration/traces/traces-query-editor.md'
[//]: # '/docs/sources/view-and-analyze-profile-data/profile-tracing/_index.md'
[//]: #
[//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.'
[//]: # 'Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/.'

<!-- Profiling and tracing integration -->

Profiles, continuous profiling, and distributed traces are all tools that can be used to improve the performance and reliability of applications.
However, each tool has its own strengths and weaknesses, and it is important to choose the right tool for the job as well as understand when to use both.

## Profiling

Profiling offers a deep-dive into an application's performance at the code level, highlighting resource usage and performance hotspots.

<table>
  <tr>
    <th scope="row">Usage</th>
    <td>During development, major releases, or upon noticing performance quirks.</td>
  </tr>
  <tr>
    <th scope="row">Benefits</th>
    <td>
      <ul>
        <li>Business: Boosts user experience through enhanced application performance.</li>
        <li>Technical: Gives clear insights into code performance and areas of refinement.</li>
      </ul>
    </td>
  </tr>
  <tr>
    <th scope="row">Example</th>
    <td>A developer uses profiling upon noting slow app performance, identifies a CPU-heavy function, and optimizes it.</td>
  </tr>
</table>

## Continuous profiling

Continuous profiling provides ongoing performance insights, capturing long-term trends and intermittent issues.

<table>
  <tr>
    <th scope="row">Usage</th>
    <td>Mainly in production, especially for high-priority applications.</td>
  </tr>
  <tr>
    <th scope="row">Benefits</th>
    <td>
      <ul>
        <li>Business: Preemptively addresses inefficiencies, potentially saving costs.</li>
        <li>Technical: Highlights performance trends and issues like potential memory leaks over time.</li>
      </ul>
    </td>
  </tr>
  <tr>
    <th scope="row">Example</th>
    <td>A month-long data from continuous profiling suggests increasing memory consumption, hinting at a memory leak.</td>
  </tr>
</table>

## Distributed tracing

Traces requests as they cross multiple services, revealing interactions and service dependencies.

<table>
  <tr>
    <th scope="row">Usage</th>
    <td>Essential for systems like microservices where requests touch multiple services.</td>
  </tr>
  <tr>
    <th scope="row">Benefits</th>
    <td>
      <ul>
        <li>Business: Faster issue resolution, reduced downtimes, and strengthened customer trust.</li>
        <li>Technical: A broad view of the system's structure, revealing bottlenecks and inter-service dependencies.</li>
      </ul>
    </td>
  </tr>
  <tr>
    <th scope="row">Example</th>
    <td>In e-commerce, a user's checkout request might involve various services. Tracing depicts this route, pinpointing where time is most spent.</td>
  </tr>
</table>

## Combined power of tracing and profiling

When used together, tracing and profiling provide a powerful tool for understanding system and application performance.

<table>
  <tr>
    <th scope="row">Usage</th>
    <td>For comprehensive system-to-code insights, especially when diagnosing complex issues spread across services and codebases.</td>
  </tr>
  <tr>
    <th scope="row">Benefits</th>
    <td>
      <ul>
        <li>Business: Reduces downtime, optimizes user experience, and safeguards revenues.</li>
        <li>Technical:
          <ul>
            <li>Holistic view: Tracing pinpoints bottle-necked services, while profiling delves into the responsible code segments.</li>
            <li>End-to-end insight: Visualizes a request's full journey and the performance of individual code parts.</li>
            <li>Efficient diagnosis: Tracing identifies service latency; profiling zeroes in on its cause, be it database queries, API calls, or specific code inefficiencies.</li>
          </ul>
        </li>
      </ul>
    </td>
  </tr>
  <tr>
    <th scope="row">Example</th>
    <td>Tracing reveals latency in a payment service. Combined with profiling, it's found that a particular function, making third-party validation calls, is the culprit. This insight guides optimization, refining system efficiency.</td>
  </tr>
</table>

# query-with-gcx

---
headless: true
labels:
  products:
    - cloud
    - enterprise
    - oss
---

[//]: # 'Shared note: querying a data source with the gcx Grafana CLI.'
[//]: # 'This shared file is included in these locations:'
[//]: # '/docs/sources/datasources/aws-cloudwatch/_index.md'
[//]: # '/docs/sources/datasources/azure-monitor/_index.md'
[//]: # '/docs/sources/datasources/elasticsearch/_index.md'
[//]: # '/docs/sources/datasources/google-cloud-monitoring/_index.md'
[//]: # '/docs/sources/datasources/influxdb/_index.md'
[//]: # '/docs/sources/datasources/loki/_index.md'
[//]: # '/docs/sources/datasources/mssql/_index.md'
[//]: # '/docs/sources/datasources/mysql/_index.md'
[//]: # '/docs/sources/datasources/postgres/_index.md'
[//]: # '/docs/sources/datasources/prometheus/_index.md'
[//]: # '/docs/sources/datasources/pyroscope/_index.md'
[//]: # '/docs/sources/datasources/tempo/_index.md'
[//]: #
[//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.'
[//]: # 'Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/.'

## Query with the Grafana CLI

You can query this data source from the command line and from AI coding agents using the Grafana CLI, `gcx`. `gcx` gives you and your agent structured, terminal-based access to your Grafana data sources, which is useful for automation and agent-driven investigations.

`gcx` supports Grafana Cloud and Grafana OSS or Enterprise v12 and later. Before you query, install and authenticate `gcx`. Refer to the [`gcx` CLI documentation](/docs/grafana/<GRAFANA_VERSION>/ai-tools/grafana-cli/gcx/) for installation, configuration, and the full command reference.

# sql-query-builder-macros

---
headless: true
labels:
  products:
    - enterprise
    - oss
---

#### Macros

You can enable macros support in the select clause to create time-series queries.

Use the **Data operations** drop-down to select a macro like `$__timeGroup` or `$__timeGroupAlias`.
Select a time column from the **Column** drop-down and a time interval from the **Interval** drop-down to create a time-series query.

{{< figure src="/media/docs/grafana/data-sources/screenshot-sql-builder-time-series-query.png" class="docs-image--no-shadow" caption="SQL query builder time-series query" >}}

You can also add custom value to the **Data operations**.
For example, a function that's not in the drop-down list.
This allows you to add any number of parameters.

# tempo-editor-traceql

---
headless: true
labels:
  products:
    - enterprise
    - oss
---

[//]: # 'This file documents the TraceQL query editor for the Tempo data source.'
[//]: # 'This shared file is included in these locations:'
[//]: # '/grafana/docs/sources/datasources/tempo/query-editor/traceql-editor.md'
[//]: # '/website/docs/grafana-cloud/data-configuration/traces/traces-query-editor.md'
[//]: # '/tempo/docs/sources/tempo/traceql/query_editor.md'
[//]: #
[//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.'
[//]: # 'Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/.'

<!-- # Write TraceQL queries using the query editor -->

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces.
TraceQL provides a method for formulating precise queries so you can zoom in to the data you need.
Query results are returned faster because the queries limit what is searched.

To learn more about how to query by TraceQL, refer to the [TraceQL documentation](https://grafana.com/docs/tempo/latest/traceql/).

The TraceQL query editor in Grafana **Explore** lets you search by trace ID and write TraceQL queries using autocomplete.

![The TraceQL query editor](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-traceql-v11.png)

## Before you begin

This feature is automatically available in Grafana 10 (and newer) and Grafana Cloud.

To use the TraceQL query editor in self-hosted Grafana 9.3.2 and older, you need to [enable the `traceqlEditor` feature toggle](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/feature-toggles/).

### Streaming and gRPC

If you're trying to query a self-managed Grafana Tempo or Grafana Enterprise Traces database with a gateway, such as nginx, in front of it from your hosted Grafana, that gateway (for example, nginx) must allow gRPC connections.
If it doesn't, streaming won't work and queries will fail to return results.

If you can't configure your gateway to allow gRPC, deactivate streaming in your hosted Grafana.
In Grafana 11.2 and newer, you can deactivate the **Streaming** option in your Tempo data source settings from **Connections** > **Data sources** in the Grafana main menu.
You can also open a support escalation to request streaming query results be disabled in your hosted Grafana.

## Write TraceQL queries using the query editor

The Tempo data source’s TraceQL query editor helps you query and display traces from Tempo in **Explore**.

To access the query editor, follow these steps:

1. Sign into Grafana or Grafana Cloud.
1. Select **Explore** from the main menu.
1. Select a Tempo data source.
1. Select the **TraceQL** tab.
1. Start your query on the text line by entering `{`. For help with TraceQL syntax, refer to the [Construct a TraceQL query documentation](https://grafana.com/docs/tempo/latest/traceql/#construct-a-traceql-query).

   Optional: Select **Copy query from Search** to transfer a builder query to the editor.

1. Optional: Use the **Time picker** drop-down list to change the time and range for the query (refer to the [documentation for instructions](https://grafana.com/docs/grafana/latest/dashboards/use-dashboards/#set-dashboard-time-range)).
1. Once you've finished your query, select **Run query**.

![Query editor showing span results](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-ed-example-v11-a.png)

This video provides an example of creating a TraceQL query using the custom tag grouping.

{{< youtube id="fraepWra00Y" >}}

## Query by TraceID

To query a particular trace by its trace ID:

1. From the menu, choose **Explore**, select the desired Tempo data source, and navigate to the **TraceQL** tab.
1. Enter the trace ID into the query field. For example: `41928b92edf1cdbe0ba6594baee5ae9`
1. Click **Run query** or use the keyboard shortcut Shift + Enter.

![Search for a trace ID using the TraceQL query editor](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-trace-id-v11.png)

## Use autocomplete to write queries

You can use the query editor’s autocomplete suggestions to write queries.
The editor detects spansets to provide relevant autocomplete options.
It uses regular expressions (regex) to detect where it's inside a spanset and provide attribute names, scopes, intrinsic names, logic operators, or attribute values from the Tempo API, depending on what's expected for the current situation.

![Query editor showing the auto-complete feature](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-editor-autocomplete.png)

### Anchored regular expressions

Regular expressions are anchored at both ends.
This anchoring makes the queries faster and matches the behavior of PromQL, where regular expressions are also fully anchored.

An unanchored query, such as:
`{ span.foo =~ "bar" }`
is now treated as:
`{ span.foo =~ "^bar$" }`.

If you use TraceQL with regular expressions in your Grafana dashboards and you want the unanchored behavior, update the queries to use the unanchored version, such as `{ span.foo =~ ".*bar.*"}`.

### Create a query with autocomplete

To create a query using autocomplete, follow these steps:

1. From the menu, choose **Explore**, select the desired Tempo data source, and navigate to the **TraceQL** tab.

1. Enter your query. As you type your query, autocomplete suggestions appear as a drop-down. Each letter you enter refines the autocomplete options to match.

1. Use your mouse or arrow keys to select any option you wish. When the desired option is highlighted, press Tab on your keyboard to add the selection to your query.

1. Once your query is complete, select **Run query**.

## View query results

Query results appear in a table, such as **Table - Traces**, under the query editor.
Each span (and the trace it belongs to) matching the query conditions is returned by the query.
If there are no filter conditions, all spans are matching and thus returned with their associated traces.

A query is performed against a defined time interval, relative (for example, the last 3 hours) or absolute (for example, from X date-time to Y date-time).
The query response is also limited by the number of traces (**Limit**) and spans per spanset (**Span Limit**).

![TraceQL in Grafana](/media/docs/tempo/traceql/TraceQL-in-Grafana-v11.png)

1. TraceQL query editor
1. Query options: **Limit**, **Span Limit** and **Table Format** (Traces or Spans).
1. Trace (by Trace ID). The **Name** and **Service** columns are displaying the trace root span name and associated service.
1. Spans associated with the Trace.

Selecting the trace ID from the returned results opens a trace diagram.
Selecting a span from the returned results opens a trace diagram and reveals the relevant span in the trace diagram.

For more information on span details, refer to [Traces in Explore](https://grafana.com/docs/grafana/latest/explore/trace-integration/#span-details).

![Selecting a trace ID or a span to view span details](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-span-details-v11.png)

Querying spansets with a large number of spans can negatively impact performance.
You can use the **Span Limit** field in **Options** section of the TraceQL query editor.
This field sets the maximum number of spans to return for each span set.
By default, the maximum value that you can set for the **Span Limit** value (or the spss query) is 100.
In Tempo configuration, this value is controlled by the `max_spans_per_span_set` parameter and can be modified by your Tempo administrator.
Entering a value higher than the default results in an error.

{{< admonition type="note" >}}
Changing the value of `max_spans_per_span_set` isn't supported in Grafana Cloud.
{{< /admonition >}}

### Focus on traces or spans

Under **Options**, you can choose to display the table as **Traces** or **Spans** focused.

When the **Table Type** option is set to **Spans**, the traces and spansets are flattened into a list of spans.
The trace service and trace name are added to the row of each span to add context.

Using the **Spans** option makes it easier access the spans to apply transformations and plot them in dashboards.

### Stream results

The Tempo data source supports streaming responses to TraceQL queries so you can see partial query results as they come in without waiting for the whole query to finish.

Streaming is available for both the **Search** and **TraceQL** query types, and you'll get immediate visibility of incoming traces on the results table.

To learn how to activate streaming, refer to [Streaming](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#streaming) in the Tempo data source documentation.

{{< video-embed src="/media/docs/grafana/data-sources/tempo-streaming-v2.mp4" >}}

# tempo-search-traceql

---
headless: true
labels:
  products:
    - enterprise
    - oss
---

[//]: # 'This file documents the Search query type for the Tempo data source.'
[//]: # 'This shared file is included in these locations:'
[//]: # '/grafana/docs/sources/datasources/tempo/query-editor/index.md'
[//]: # '/website/docs/grafana-cloud/data-configuration/traces/traces-query-editor.md'
[//]: # '/tempo/docs/sources/tempo/traceql/query_editor.md'
[//]: #
[//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.'
[//]: # 'Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/.'

# Write TraceQL queries using Search

The **Search** query builder, located on the **Explore** > **Query type** > **Search** in Grafana, provides drop-down lists and text fields to help you write a query.
The selections you make automatically generate a [TraceQL query](/docs/tempo/latest/traceql).

![The Search query builder](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-search-v11.png)

The builder lets you run the most common queries in as few clicks as possible. You don't need to know the underlying query language or database architecture to use it.

The builder supports a subset of TraceQL capabilities, including some structural operators (`>>`, `>`, `=~`, `!=`).

To access **Search**, select your Tempo data source, and then choose **Explore** and select **Query type** > **Search**.
You can use the query builder to search trace data by resource service name, span name, duration, one or more tags. The examples on this page use the default filters.

In addition, you can add query builder blocks, view the query history, and use the **Inspector** to see details.

{{< figure src="/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-builder-v11.png" class="docs-image--no-shadow" max-width="750px" caption="Screenshot of the Tempo Search query type" >}}

## Perform a search

To perform a search, you need to select filters and then run the query. The results appear underneath the query builder.
The screenshot identifies the areas used to perform a search.

![Parts of the Search query builder](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-build-numbered-v11.png)

| Number | Name                  | Action                                                                                                                                             | Comment                                                                                                                                                          |
| :----- | :-------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1      | Data source           | Use the data source drop-down list to select a Tempo data source.                                                                                  | Each data source has its own version of search. This **Search** is specific to the Tempo data source.                                                            |
| 2      | Query type            | Select **Search**.                                                                                                                                 |                                                                                                                                                                  |
| 3      | Choose filter         | Choose one or more of the filters.                                                                                                                 | Optional. You can execute an empty query in the Search tab. In TraceQL, `{}` is a valid query and is the default query to provide a list of all traces or spans. |
| 4      | Filters conditions    | Select options for one or more filters. For example, you can define a filter where **Service Name** (`resource.service.name`) equals (`=`) `user`. | Optional. At least one tag or filter must be defined.                                                                                                            |
| 5      | Tags and Aggregate by | Add tags for span, resource, or unscoped and define their conditions. Use **Aggregate by** to group results.                                       | Optional.                                                                                                                                                        |
| 6      | TraceQL query         | Displays the TraceQL query constructed by your selections.                                                                                         | This TraceQL query is executed when you select **Run query**.                                                                                                    |

Every query searches the data for the selected time frame.
By default, queries run against data from the last hour.
Select **Time range** to the left of **Run query** to choose the time range for the data your query runs against.
Read the [dashboard time range](/docs/grafana/latest/dashboards/use-dashboards/#set-dashboard-time-range) documentation to learn more.

To access the **Search** query builder, use the following steps:

1. Sign into Grafana.
1. Select your Tempo data source.
1. From the menu, choose **Explore** and select **Query type > Search**.

## Define filters

Using filters, you refine the data returned from the query by selecting **Service Name**, **Span Name**, **Status**, or **Duration**.
**Duration** represents span time, calculated by subtracting the end time from the start time of the span.

Grafana administrators can change the default filters using the Tempo data source configuration.
Filters can be limited by the operators.
The field type determines the available operators.
For example, **Span Name** and **Service Name** are string fields so the comparison operators are equals (`=`), not equal (`!=`), matches regular expressions (`=~`), or doesn't match regular expression (`!~`).
**Duration** is a duration field type and uses range selections (`>`, `>=`, `<`, `<=`).

You can select multiple values for the same filter.
The generated query depends on the operator you choose:

- Equals (`=`) or other non-regular expression operators: Each value becomes a separate condition joined with `||` inside parentheses. For the not-equals (`!=`) operator, conditions are joined with `&&` instead.
- Regular expressions match (`=~`) or not match (`!~`): Values are concatenated with `|` inside a single quoted string.

This capability only applies to fields with drop-down value selection.

For example, if you select **Span Name** with the `=` operator and choose both `get` and `log_results_cache` as values, the generated query uses separate conditions:

```traceql
{ name="get" || name="log_results_cache" }
```

If you select the regular expression operator `=~` instead, the values are combined into a single regular expression pattern:

```traceql
{ name=~"get|log_results_cache" }
```

To define filters, follow these steps:

1. Choose one of the filters.
1. Select a comparison operator from the drop-down.
1. **Service Name**, **Span Name**, and **Status** only: Select one or more values from the drop-down.
1. **Duration** only: Enter values and units for the range and choose comparison operators for the drop-downs. Units can be nanoseconds (`ns`), milliseconds (`ms`), seconds (`s`), minutes (`m`), and hours (`h`).

You can either select **Run query** to execute the query or define tags and then run the query.

### Define tags

You can add any tags to your query to further filter the results.
Tags can be selected by scoped (span or resource) or unscoped.
If you select unscoped, then all tags are searched for matches.

To add a tag, follow these steps:

1. Select span, resource, or unscoped.
1. Select a tag from the **Select tag** drop-down.
1. Select a comparison operator.
1. Select a value from the **Select value** drop-down. This field is populated based upon the tag.
1. Optional: Select **+** to add another tag.

### Optional: Use Aggregate by

{{< admonition type="warning" >}}
Metrics summary API and the **Aggregate by** feature are deprecated in Grafana Cloud and Grafana 11.3 and later.
It will be removed in a future release.
{{< /admonition >}}

Using **Aggregate by**, you can calculate RED metrics (total span count, percent erroring spans, and latency information) for spans of `kind=server` that match your filter criteria, grouped by one or more attributes.
This capability is based on the [metrics summary API](/docs/grafana-cloud/send-data/traces/metrics-summary-api/).
Metrics summary only calculates summaries based on spans received within the last hour.
For additional information, refer to [Traces to metrics: Ad-hoc RED metrics in Grafana Tempo with `Aggregate by`](https://grafana.com/blog/2023/12/07/traces-to-metrics-ad-hoc-red-metrics-in-grafana-tempo-with-aggregate-by/).

<!--Impromptu RED metrics with Aggregate by -->

{{< youtube id="xOolCpm2F8c" >}}

**Aggregate by** is disabled by default.
[Enable the `metricsSummary` feature toggle](/docs/grafana/latest/setup-grafana/configure-grafana/feature-toggles/) in Grafana to use this feature.

Your Grafana Tempo data source must also point to a Tempo database with the [Metrics Summary API](https://grafana.com/docs/grafana-cloud/observe-and-act/send-data/traces/configure/metrics-summary-api/) enabled.
Contact Grafana Support to enable this feature in Grafana Cloud.

#### Use Aggregate by

When you use **Aggregate by**, the selections you make determine how the information is reported in the Table.
Every combination that matches selections in your data is listed in the table.
Each aggregate value, for example `intrinsic`:`name`, has a corresponding column in the results table.

For example, **names** matching `GET /:endpoint` with a **span.http.user_agent** of `k6/0.46` appeared in 31,466 spans. Instead of being listed by traces and associated spans, the query results are grouped by the selections in **Aggregate by**.

The RED metrics are calculated for every name and user agent combination found in your data.

The screenshot shows all of the successful HTTP `status_code` API calls against the `mystical-server` service.
The results are shown in the same order used in **Aggregate by**.
For example, **Aggregate by** lists `intrinsic.name` followed by `span.http.user_agent`.
The first column in the results Table shows **name** and then **span.http.user_agent**.

![Use Aggregate by to calculate RED metrics for spans and group by attributes](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-build-aggregate-v11-a.png)

To use this capability:

1. In the **Aggregate by** row, select a scope from the first drop-down box. For example, `span`.
1. Select an attribute from the second drop-down.
1. Optional: Select **+** to add an **Aggregate by** row.
1. Optional: Select a **Time range** to expand or narrow the data set for an hour's range.
1. Select **Run query**.

<!-- Explanation of how to use feature -->

{{< youtube id="g97CjKOZqT4" >}}

### Optional: Add query and service graph blocks

Using **Add query**, you can have successive query or service node blocks that run in sequential order.
For example, query A runs and then query B.
You can reorder the queries by dragging and dropping them above or below other queries.
Select **+ Add query** to add another query block.

For more information, refer to [Use query types together](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/).

## Run queries and view results

Select **Run query** to run the TraceQL query (1 in the screenshot).

Queries can take a little while to return results. The results appear in a table underneath the query builder.
Selecting a Trace ID (2 in the screenshot) displays more detailed information (3 in the screenshot).
**Span Filters** (4 in the screenshot) provide an additional to refine the query results.

![Query results with numbered sections](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-results-numbered-v11.png)

### Stream results

The Tempo data source supports streaming responses to TraceQL queries so you can see partial query results as they come in without waiting for the whole query to finish.

When active, all configured Tempo data sources attempt to use streaming.
You can activate and control which Tempo data sources do and don't attempt to stream results at the per-data source level using the **Streaming** section of the Tempo data source configuration.
For more information, refer to the [Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#streaming) documentation.

Streaming is available for both the **Search** and **TraceQL** query types.
You'll get immediate visibility of incoming traces on the results table.

{{< video-embed src="/media/docs/grafana/data-sources/tempo-streaming-v2.mp4" >}}

### Use filters and tags on spans

Using **Span Filters**, you can use filters to refine results when viewing span details.
These filters are available when viewing details for a trace.

You can continue to apply filters until you have narrowed down your resulting spans to the select few you are most interested in.

**Service Name**, **Span Name**, **Duration**, and **Tags** have the same function and operation as the filters of the same name in the **Search** query builder.

In addition, you can search for a keyword, opt to **Show matches only**, opt to **Show critical path only**, and browse matches using **Prev** and **Next**.

Use **Clear** to reset the filters.

# tempo-traces-to-profiles

---
headless: true
labels:
  products:
    - enterprise
    - oss
---

[//]: # 'This file documents the Traces to profile configure and usage for the Tempo data source.'
[//]: # 'This shared file is included in these locations:'
[//]: # '/grafana/docs/sources/datasources/tempo/configure-tempo-data-source.md'
[//]: # '/website/docs/grafana-cloud/data-configuration/traces/traces-query-editor.md'
[//]: #
[//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.'
[//]: # 'Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/.'

<!-- # Trace to profiles  -->

Using Trace to profiles, you can use the ability to correlate different signals in Grafana by adding the functionality to link between traces and profiles.

**Trace to profiles** lets you link your Grafana Pyroscope data source to tracing data.
When configured, this connection lets you run queries from a trace span into the profile data using **Explore**.
Each span links to your queries. Clicking a link runs the query in a split panel.
If tags are configured, Grafana dynamically inserts the span attribute values into the query.
The query runs over the time range of the (span start time - 60) to (span end time + 60 seconds).

Embedded flame graphs are also inserted into each span details section that has a linked profile.
This lets you see resource consumption in a flame graph visualization for each span without having to navigate away from the current view.

{{< youtube id="AG8VzfFMLxo" >}}

There are two ways to configure the trace to profiles feature:

- Use a basic configuration with default query, or
- Configure a custom query where you can use a template language to interpolate variables from the trace or span.

![Traces to profiles section in the Tempo data source](/media/docs/grafana/data-sources/tempo/tempo-data-source-trace-to-profiles.png)

## Before you begin

To use Trace to profiles, you need:

- A [Grafana Pyroscope data source](/docs/grafana/<GRAFANA_VERSION>/datasources/grafana-pyroscope/) connected to your Pyroscope instance.
- A Tempo data source configured with the Trace to profiles settings described on this page.
- Your application instrumented with all three of the following:
  1. **Profiling**: A Pyroscope SDK or Grafana Alloy sending profiling data to Pyroscope. Refer to [Configure the client to send profiles](/docs/pyroscope/<PYROSCOPE_VERSION>/configure-client/) for setup instructions.
  1. **Tracing**: An OpenTelemetry SDK sending trace data to Tempo. Refer to the [OpenTelemetry getting started documentation](https://opentelemetry.io/docs/getting-started/) for setup instructions.
  1. **Span profiling bridge**: A language-specific OpenTelemetry integration package that links profiling data with trace spans. Without this package, traces and profiles are independent signals with no connection between them. Refer to [Link tracing and profiling with span profiles](/docs/pyroscope/<PYROSCOPE_VERSION>/configure-client/trace-span-profiles/) for per-language setup instructions.

## Use a basic configuration

To use a basic configuration, follow these steps:

1. In the left menu, select **Connections** > **Data sources**.
1. Select your configured Tempo data source from the **Data source** list.
1. Scroll down to the **Traces to profiles** section.
1. Select a Pyroscope data source in the **Data source** drop-down.
1. Optional: Add one or more tags to use in the query. If left blank, the default values of `service.name` and `service.namespace` are used.

   The tags you configure must be present in the spans attributes or resources for a trace-to-profiles span link to appear.

   You can optionally configure a new name for the tag. This is useful if the tag has dots in the name and the target data source doesn't allow dots in labels. In that case, you can remap `service.name` to `service_name`.

1. Select a profile type to use in the query. Select the drop-down and choose an option from the menu.

   The profile type or app must be selected for the query to be valid. Grafana doesn't show any data if the profile type or app isn't selected when a query runs.

1. Select **Save and Test**.

If you have configured a Pyroscope data source and no profile data is available or the **Profiles for this span**
button and the embedded flame graph isn't visible, verify that the `pyroscope.profile.id` key-value pair exists in your span tags.

## Configure a custom query {#configure-custom-query-traces-profiles}

To use a custom query with the configuration, follow these steps:

1.  In the left menu, select **Connections** > **Data sources**.
1.  Select a configured Tempo data source from the **Data source** list.
1.  Scroll down to the **Traces to profiles** section.
1.  Select a Pyroscope data source in the **Data source** drop-down.
1.  Optional: Choose any tags to use in the query. If left blank, the default values of `service.name` and `service.namespace` are used.

    These tags can be used in the custom query with `${__tags}` variable. This variable interpolates the mapped tags as list in an appropriate syntax for the data source. Only tags present in the span are included. Tags that aren't present are omitted.

    You can also configure a name for the tag. Tag names are useful where the tag has dots in the name and the target data source doesn't allow using dots in labels. For example, you can remap `service.name` to `service_name`. If you don't map any tags here, you can still use any tag in the query, for example: `method="${__span.tags.method}"`. Learn more about [custom query variables](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#custom-query-variables).

1.  Select a profile type to use in the query. Select the drop-down and choose an option from the menu.
1.  Switch on **Use custom query** to enter a custom query.
1.  Specify a custom query to be used to query profile data. You can use various variables to make that query relevant for current span. The link shows only if all the variables are interpolated with non-empty values to prevent creating an invalid query. You can interpolate the configured tags using the `$__tags` keyword.
1.  Select **Save and Test**.

## Configuration options

The following table describes options for configuring your **Trace to profiles** settings:

| Setting name     | Description                                                                                                                                                                                                                                                                                                    |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Data source      | Defines the target data source. You can select a Pyroscope \[profiling\] data source.                                                                                                                                                                                                                          |
| Tags             | Defines the tags to use in the profile query. Default: `cluster`, `hostname`, `namespace`, `pod`, `service.name`, `service.namespace`. You can change the tag name for example to remove dots from the name if they're not allowed in the target data source. For example, map `http.status` to `http_status`. |
| Profile type     | Defines the profile type to use in the query.                                                                                                                                                                                                                                                                  |
| Use custom query | Toggles use of custom query with interpolation.                                                                                                                                                                                                                                                                |
| Query            | Input to write custom query. Use variable interpolation to customize it with variables from span.                                                                                                                                                                                                              |

# authentication

---
headless: true
comments: |
  This file is used in the following files: developers/http_api/{_index.md,authentication.md}
---

You can authenticate HTTP API requests using basic authentication or a service account token.

### Basic auth

If basic auth is enabled (it is enabled by default), then you can authenticate your HTTP request via
standard basic auth. Basic auth will also authenticate LDAP users.

curl example:

```bash
curl http://admin:admin@localhost:3000/api/org
{"id":1,"name":"Main Org."}
```

### Service account token

To create a service account token, click on **Administration** in the left-side menu, click **Users and access**, then **Service Accounts**.
For more information on how to use service account tokens, refer to the [Service Accounts](/docs/grafana/<GRAFANA_VERSION>/administration/service-accounts/) documentation.

You use the token in all requests in the `Authorization` header, like this:

**Example**:

```http
GET http://your.grafana.com/api/dashboards/db/mydash HTTP/1.1
Accept: application/json
Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
```

The `Authorization` header value should be _`Bearer <YOUR_SERVICE_ACCOUNT_TOKEN>`_.

# deprecated-apis-nonmigrated

---
headless: true
comments: |
  This file is used in legacy APIs.
---

{{< admonition type="note" >}}

Starting in Grafana 13, `/api` endpoints are being deprecated in favor of the `/apis` route. **This change doesn't disrupt or break your current setup**. Legacy APIs are not being disabled and remain fully accessible and operative, but `/api` routes will no longer be updated.

**This API will not be migrated**. Refer to [API deprecation notes](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis-migration#deprecation-notes) for more details and alternatives.

{{< /admonition >}}

# deprecated-apis

---
headless: true
comments: |
  This file is used in legacy APIs.
---

{{< admonition type="note" >}}

Starting in Grafana 13, `/api` endpoints are being deprecated in favor of the `/apis` route. Note that while Grafana is working on migrating existing APIs, currently there may not be an exact match to the legacy API you're using.

**This change doesn't disrupt or break your current setup**. Legacy APIs are not being disabled and remain fully accessible and operative, but `/api` routes will no longer be updated.

To learn more refer to the [new API structure in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis/).

{{< /admonition >}}

# Intro paragraph

[InfluxDB](https://www.influxdata.com/products/) is an open-source time series database (TSDB) developed by [InfluxData](https://www.influxdata.com/). It is optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, IoT sensor data, and real-time analytics.

# View org list as server admin

{{< docs/list >}}
{{< docs/shared lookup="manage-users/view-server-org-list.md" source="grafana" version="<GRAFANA VERSION>" >}}

1. Click the name of the organization that you want to edit.

{{< /docs/list >}}

# View org list as server admin

1. Hover your cursor over the **Server Admin** (shield) icon until a menu appears.
1. Click **Orgs**.

# View user list and search - list format

{{< docs/list >}}
{{< docs/shared lookup="manage-users/view-server-user-list.md" source="grafana" version="<GRAFANA VERSION>" >}}

1. Click the user account that you want to edit. If necessary, use the search field to find the account.

{{< /docs/list >}}

# View user list as server admin

1. Hover your cursor over the **Server Admin** (shield) icon until a menu appears.
1. Click **Users**.

# View team list

1. Hover your cursor over the **Configuration** (gear) icon in the side menu.
1. Click **Teams**. Grafana displays the team list.

# Panel links intro

Each panel can have its own set of links that are shown in top left of a panel after the panel title. You can link to any available URL, including dashboards, panels, or external sites. You can even control the time range to ensure the user is zoomed in on the right data in Grafana.

# Repeat panel intro

Grafana lets you create dynamic dashboards using _template variables_. All variables in your queries expand to the current value of the variable before the query is sent to the database. Variables let you reuse a single dashboard for all your services.

# rename-note

---
headless: true
labels:
  products:
    - enterprise
    - oss
---

[//]: # 'This file contains a rename note for Explore to Drilldown apps.'
[//]: # 'This shared file is included in a lot of files. Check the app docs in'
[//]: # 'drilldown-traces, drilldown-logs, drilldown-profiles, grafana, and website/grafana-cloud.'
[//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.'
[//]: # 'Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/.'

{{< admonition type="note" >}}
The Grafana Explore apps have changed to Grafana Drilldown apps.
For example, Explore Logs is now Logs Drilldown.
To learn more, read [Grafana Drilldown apps: the improved queryless experience formerly known as the Explore apps](https://grafana.com/blog/2025/02/20/grafana-drilldown-apps-the-improved-queryless-experience-formerly-known-as-the-explore-apps/).
{{< /admonition >}}

# Navigate to user preferences list

1. On the left menu, hover your cursor over your avatar and then click **Preferences**.

# Click Org preferences

1. Hover your cursor over the **Configuration** (gear) icon.
1. Click **Preferences**.

# Select home dashboard list

1. In the **Home Dashboard** field, search for and select the dashboard that you want to use. Select **Grafana home** to show the instance default instead of a dashboard set at a higher level.
1. Click **Save**.

# Select home dashboard list

1. Click to select an option in the **Timezone** list. **Default** is either the browser local timezone or the timezone selected at a higher level.
1. Click **Save**.

# Select the UI theme

1. In the Preferences section, select the **UI theme**.
1. Click **Save**.

# Some tasks require permissions

Some tasks require certain permissions. For more information about roles, refer to [About users and permission](/docs/grafana/<GRAFANA_VERSION>/administration/manage-users-and-permissions/about-users-and-permissions/).

# Serving Grafana on a port < 1024

If you are using `systemd` and want to start Grafana on a port that is lower than 1024, you must add a `systemd` unit override.

1. Run the following command to create an override file in your configured editor.

   ```bash
   # Alternatively, create a file in /etc/systemd/system/grafana-server.service.d/override.conf
   sudo systemctl edit grafana-server.service
   ```

1. Add the following additional settings to grant the `CAP_NET_BIND_SERVICE` capability.

   To learn more about capabilities, refer to [capabilities(7) — Linux manual page](https://man7.org/linux/man-pages/man7/capabilities.7.html).

   ```ini
   [Service]
   # Give the CAP_NET_BIND_SERVICE capability
   CapabilityBoundingSet=CAP_NET_BIND_SERVICE
   AmbientCapabilities=CAP_NET_BIND_SERVICE

   # A private user cannot have process capabilities on the host's user
   # namespace and thus CAP_NET_BIND_SERVICE has no effect.
   PrivateUsers=false
   ```

# Create Plugin

Tooling for modern web development can be tricky to wrap your head around. While you certainly can write your own webpack configuration, for this guide, you'll be using grafana create-plugin tool

Grafana [create-plugin tool](https://www.npmjs.com/package/@grafana/create-plugin) is a CLI application that simplifies Grafana plugin development, so that you can focus on code. The tool scaffolds a starter plugin and all the required configuration for you.

1. In the plugin directory, create a plugin from template using create-plugin. When prompted for the kind of plugin, select `datasource`:

   ```
   npx @grafana/create-plugin@latest
   ```

1. Change directory to your newly created plugin:

   ```
   cd my-plugin
   ```

1. Install the dependencies:

   ```
   yarn install
   ```

1. Build the plugin:

   ```
   yarn dev
   ```

1. Restart the Grafana server for Grafana to discover your plugin.
1. Open Grafana and go to **Connections** -> **Connect Data**. Make sure that your data source is there.

By default, Grafana logs whenever it discovers a plugin:

```
INFO[01-01|12:00:00] Plugin registered       logger=plugin.loader pluginID=my-plugin
```

# Plugin Anatomy

Plugins come in different shapes and sizes. Before we dive deeper, let's look at some of the properties that are shared by all of them.

Every plugin you create will require at least two files: `plugin.json` and `src/module.ts`.

### plugin.json

When Grafana starts, it scans the plugin directory for any subdirectory that contains a `plugin.json` file. The `plugin.json` file contains information about your plugin, and tells Grafana about what capabilities and dependencies your plugin needs.

While certain plugin types can have specific configuration options, let's look at the mandatory ones:

- `type` tells Grafana what type of plugin to expect. Grafana supports three types of plugins: `panel`, `datasource`, and `app`.
- `name` is what users will see in the list of plugins. If you're creating a data source, this is typically the name of the database it connects to, such as Prometheus, PostgreSQL, or Stackdriver.
- `id` uniquely identifies your plugin, and should start with your Grafana username, to avoid clashing with other plugins. [Sign up for a Grafana account](https://grafana.com/signup/) to claim your username.

To see all the available configuration settings for the `plugin.json`, refer to the [plugin.json Schema](/docs/grafana/latest/plugins/developing/plugin.json/).

### module.ts

After discovering your plugin, Grafana loads the `module.ts` file, the entrypoint for your plugin. `module.ts` exposes the implementation of your plugin, which depends on the type of plugin you're building.

Specifically, `module.ts` needs to expose an object that extends [GrafanaPlugin](https://github.com/grafana/grafana/blob/08bf2a54523526a7f59f7c6a8dafaace79ab87db/packages/grafana-data/src/types/plugin.ts#L124), and can be any of the following:

- [PanelPlugin](https://github.com/grafana/grafana/blob/08bf2a54523526a7f59f7c6a8dafaace79ab87db/packages/grafana-data/src/types/panel.ts#L73)
- [DataSourcePlugin](https://github.com/grafana/grafana/blob/08bf2a54523526a7f59f7c6a8dafaace79ab87db/packages/grafana-data/src/types/datasource.ts#L33)
- [AppPlugin](https://github.com/grafana/grafana/blob/45b7de1910819ad0faa7a8aeac2481e675870ad9/packages/grafana-data/src/types/app.ts#L27)

# Package your plugin

Once you're happy with your plugin, it's time to package it, and submit to the plugin repository.

For users to be able to use the plugin without building it themselves, you need to make a production build of the plugin, and commit to a release branch in your repository.

To submit a plugin to the plugin repository, you need to create a release of your plugin. While we recommend following the branching strategy outlined below, you're free to use one that makes more sense to you.

#### Create a plugin release

Let's create version 0.1.0 of our plugin.

1. Create a branch called `release-0.1.x`.

   ```
   git checkout -b release-0.1.x
   ```

1. Do a production build.

   ```
   yarn build
   ```

1. Add the `dist` directory.

   ```
   git add -f dist
   ```

1. Create the release commit.

   ```
   git commit -m "Release v0.1.0"
   ```

1. Create a release tag.

   ```
   git tag -a v0.1.0 -m "Create release tag v0.1.0"
   ```

1. Push to GitHub. `follow-tags` tells Git to push the release tag along with our release branch.
   ```
   git push --set-upstream origin release-0.1.x --follow-tags
   ```

#### Submit the plugin

For a plugin to be published on [Grafana Plugins](/grafana/plugins/), it needs to be added to the [grafana-plugin-repository](https://github.com/grafana/grafana-plugin-repository).

1. Fork the [grafana-plugin-repository](https://github.com/grafana/grafana-plugin-repository)

1. Add your plugin to the `repo.json` file in the project root directory:

   ```json
   {
     "id": "<plugin id>",
     "type": "<plugin type>",
     "url": "https://github.com/<username>/my-plugin",
     "versions": [
       {
         "version": "<version>",
         "commit": "<git sha>",
         "url": "https://github.com/<username>/my-plugin"
       }
     ]
   }
   ```

1. [Create a pull request](https://github.com/grafana/grafana-plugin-repository/pull/new/master).

Once your plugin has been accepted, it'll be published on [Grafana Plugin](/grafana/plugins/), available for anyone to [install](/docs/grafana/latest/plugins/installation/)!

> We're auditing every plugin that's added to make sure it's ready to be published. This means that it might take some time before your plugin is accepted. We're working on adding more automated tests to improve this process.

# Set up Environment

Before you can get started building plugins, you need to set up your environment for plugin development.

To discover plugins, Grafana scans a _plugin directory_, the location of which depends on your operating system.

1. Create a directory called `grafana-plugins` in your preferred workspace.

1. Find the `plugins` property in the Grafana configuration file and set the `plugins` property to the path of your `grafana-plugins` directory. Refer to the [Grafana configuration documentation](/docs/grafana/latest/installation/configuration/#plugins) for more information.

   ```ini
   [paths]
   plugins = "/path/to/grafana-plugins"
   ```

1. Restart Grafana if it's already running, to load the new configuration.

### Alternative method: Docker

If you don't want to install Grafana on your local machine, you can use [Docker](https://www.docker.com).

To set up Grafana for plugin development using Docker, run the following command:

```
docker run -d -p 3000:3000 -v "$(pwd)"/grafana-plugins:/var/lib/grafana/plugins --name=grafana grafana/grafana:7.0.0
```

Since Grafana only loads plugins on start-up, you need to restart the container whenever you add or remove a plugin.

```
docker restart grafana
```

# Upgrade guide introduction

We recommend that you upgrade Grafana often to stay current with the latest fixes and enhancements.
Because Grafana upgrades are backward compatible, the upgrade process is straightforward, and dashboards and graphs will not change.

In addition to common tasks you should complete for all versions of Grafana, there might be additional upgrade tasks to complete for a version.

{{< admonition type="note" >}}
There might be breaking changes in some releases. We outline these changes in the [What's New ](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/whatsnew/) document for each release.
{{< /admonition >}}

For versions of Grafana prior to v9.2, we published additional information in the [Release Notes](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/release-notes/).

When available, we list all changes with links to pull requests or issues in the [Changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

{{< admonition type="note" >}}
When possible, we recommend that you test the Grafana upgrade process in a test or development environment.
{{< /admonition >}}

# Upgrade guide introduction

We recommend that you upgrade Grafana often to stay current with the latest fixes and enhancements.
Because Grafana upgrades are backward compatible, the upgrade process is straightforward, and dashboards and graphs will not change.

In addition to common tasks you should complete for all versions of Grafana, there might be additional upgrade tasks to complete for a version.

{{< admonition type="note" >}}
There might be breaking changes in some releases. We outline all these changes in the [What's New](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/whatsnew/) document.
{{< /admonition >}}

For versions of Grafana prior to v9.2, we published additional information in the [Release Notes](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/release-notes/).

When available, we list all changes with links to pull requests or issues in the [Changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

{{< admonition type="note" >}}
When possible, we recommend that you test the Grafana upgrade process in a test or development environment.
{{< /admonition >}}

# Upgrade guide common tasks

The following sections provide instructions for how to upgrade Grafana based on your installation method. For more information on where to find configuration files, refer to [Configuration file location](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#configuration-file-location).

### Debian

To upgrade Grafana installed from a Debian package (`.deb`), complete the following steps:

1. In your current installation of Grafana, save your custom configuration changes to a file named `<grafana_install_dir>/grafana.ini`.

   This enables you to upgrade Grafana without the risk of losing your configuration changes.

1. [Download](https://grafana.com/grafana/download?platform=linux) the latest version of Grafana.

1. Run the following `dpkg -i` command.

   ```bash
   wget <debian package url>
   sudo apt-get install -y adduser
   sudo dpkg -i grafana_<version>_amd64.deb
   ```

### APT repository

To upgrade Grafana installed from the Grafana Labs APT repository, complete the following steps:

1. In your current installation of Grafana, save your custom configuration changes to a file named `<grafana_install_dir>/grafana.ini`.

   This enables you to upgrade Grafana without the risk of losing your configuration changes.

1. Run the following commands:

   ```bash
   sudo apt-get update
   sudo apt-get upgrade
   ```

Grafana automatically updates when you run `apt-get upgrade`.

### Binary .tar file

To upgrade Grafana installed from the binary `.tar.gz` package, complete the following steps:

1. In your current installation of Grafana, save your custom configuration changes to the custom configuration file, `custom.ini` or `grafana.ini`.

   This enables you to upgrade Grafana without the risk of losing your configuration changes.

1. [Download](https://grafana.com/grafana/download) the binary `.tar.gz` package.

1. Extract the downloaded package and overwrite the existing files.

### RPM or YUM

To upgrade Grafana installed using RPM or YUM complete the following steps:

1. In your current installation of Grafana, save your custom configuration changes to a file named `<grafana_install_dir>/grafana.ini`.

   This enables you to upgrade Grafana without the risk of losing your configuration changes.

1. Perform one of the following steps based on your installation.
   - If you [downloaded an RPM package](https://grafana.com/grafana/download) to install Grafana, then complete the steps documented in [Install Grafana on Red Hat, RHEL, or Fedora](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/installation/redhat-rhel-fedora/) or [Install Grafana on SUSE or openSUSE](https://grafana.com/docs/grafana/<GRAFANA_VERSION>//setup-grafana/installation/suse-opensuse/) to upgrade Grafana.
   - If you used the Grafana YUM repository, run the following command:

     ```bash
     sudo yum update grafana
     ```

   - If you installed Grafana on openSUSE or SUSE, run the following command:

     ```bash
     sudo zypper update
     ```

### Docker

To upgrade Grafana running in a Docker container, complete the following steps:

1. Use Grafana [environment variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#override-configuration-with-environment-variables) to save your custom configurations; this is the recommended method. Alternatively, you can view your configuration files manually by accessing the deployed container.

   This enables you to upgrade Grafana without the risk of losing your configuration changes.

1. Run a commands similar to the following commands.

   {{< admonition type="note" >}}
   This is an example. The parameters you enter depend on how you configured your Grafana container.
   {{< /admonition >}}

   ```bash
   docker pull grafana/grafana
   docker stop my-grafana-container
   docker rm my-grafana-container
   docker run -d --name=my-grafana-container --restart=always -v /var/lib/grafana:/var/lib/grafana grafana/grafana
   ```

### Windows

To upgrade Grafana installed on Windows, complete the following steps:

1. In your current installation of Grafana, save your custom configuration changes to a file named `<grafana_install_dir>/conf/custom.ini`.

   This enables you to upgrade Grafana without the risk of losing your configuration changes.

1. [Download](https://grafana.com/grafana/download) the Windows binary package.

1. Extract the contents of the package to the location in which you installed Grafana.

   You can overwrite existing files and folders, when prompted.

### Mac

To upgrade Grafana installed on Mac, complete the following steps:

1. In your current installation of Grafana, save your custom configuration changes to the custom configuration file, `custom.ini`.

   This enables you to upgrade Grafana without the risk of losing your configuration changes.

1. [Download](https://grafana.com/grafana/download) the Mac binary package.

1. Extract the contents of the package to the location in which you installed Grafana.

   You can overwrite existing files and folders, when prompted.

## Update Grafana plugins

After you upgrade Grafana, we recommend that you update all plugins because a new version of Grafana
can make older plugins stop working properly.

Run the following command to update plugins:

```bash
grafana cli plugins update-all
```

# Axis options

Options under the **Axis** section control how the x- and y-axes are rendered. Some options don't take effect until you click outside of the field option box you're editing. You can also press `Enter`.

<!-- prettier-ignore-start -->

| Option                             | Description                                                                                                        |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Time zone                          | Add one or more time zones to display along the x-axis using the time zone picker. Use **+** to add entries and **x** to remove them. |
| [Placement](#placement)            | Select the placement of the y-axis.                                                                                |
| Label                              | Set a y-axis text label. If you have more than one y-axis, then you can assign different labels using an override. |
| Width                              | Set a fixed width for the axis. By default, Grafana dynamically calculates the width of an axis.                   |
| Show grid lines                    | Set the axis grid line visibility. Choose from: **Auto**, **On**, and **Off**.                                     |
| Color                              | Set the color of the axis. Choose from:<ul><li>**Text** - Uses panel text color.</li><li>**Series** - Uses the colors of the series.</li></ul> |
| Show border                        | Set the axis border visibility.                                                                                    |
| [Scale](#scale)                    | Set the y-axis values scale. Choose from: **Linear**, **Logarithmic**, and **Symlog**.                             |
| Centered zero                      | Set the y-axis so it's centered on zero.                                                                           |
| [Soft min](#soft-min-and-soft-max) | Set a soft min to better control the y-axis limits.                                                                |
| [Soft max](#soft-min-and-soft-max) | Set a soft max to better control the y-axis limits.                                                                |

<!-- prettier-ignore-end -->

#### Placement

Select the placement of the y-axis. Choose from the following:

- **Auto** - Automatically assigns the y-axis to the series. When there are two or more series with different units, Grafana assigns the left axis to the first unit and the right axis to the units that follow.
- **Left** - Display all y-axes on the left side.
- **Right** - Display all y-axes on the right side.
- **Hidden** - Hide all axes. To selectively hide axes, [Add a field override](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-overrides/#add-a-field-override) that targets specific fields.

#### Scale

Set the y-axis values scale. Choose from:

- **Linear** - Divides the scale into equal parts.
- **Logarithmic** - Use a logarithmic scale. When you select this option, a list appears for you to choose a binary (base 2) or common (base 10) logarithmic scale.
- **Symlog** - Use a symmetrical logarithmic scale. When you select this option, a list appears for you to choose a binary (base 2) or common (base 10) logarithmic scale. The linear threshold option allows you to set the threshold at which the scale changes from linear to logarithmic.

#### Soft min and soft max

Set a **Soft min** or **soft max** option for better control of y-axis limits. By default, Grafana sets the range for the y-axis automatically based on the dataset.

**Soft min** and **soft max** settings can prevent small variations in the data from being magnified when it's mostly flat. In contrast, hard min and max values help prevent obscuring useful detail in the data by clipping intermittent spikes past a specific point.

To define hard limits of the y-axis, set standard min/max options. For more information, refer to [Configure standard options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-standard-options/#max).

![Label example](/media/docs/grafana/panels-visualizations/screenshot-soft-min-max-v12.0.png)

# Axis options

Options under the **Axis** section control how the x- and y-axes are rendered. Some options don't take effect until you click outside of the field option box you're editing. You can also press `Enter`.

<!-- prettier-ignore-start -->

| Option                             | Description       |
| ---------------------------------- | --------------------------------------------------------------------------------- |
| [Placement](#placement)            | Select the placement of the y-axis.  Choose from: **Auto**, **Left**, **Right**, and **Hidden**.  |
| Label                              | Set a y-axis text label. If you have more than one y-axis, then you can assign different labels using an override. |
| Width                              | Set a fixed width for the axis. By default, Grafana dynamically calculates the width of an axis. |
| Show grid lines                    | Set the axis grid line visibility. Choose from: **Auto**, **On**, and **Off**. |
| Color                          | Set the color of the axis. Choose from:<ul><li>**Text** - Uses panel text color.</li><li>**Series** - Uses the colors of the series.</li></ul> |
| Show border                        | Set the axis border visibility. |
| [Scale](#scale)                    | Set the y-axis values scale. Choose from: **Linear**, **Logarithmic**, and **Symlog**.  |
| Centered zero                      | Set the y-axis so it's centered on zero. Applies to the **Linear** or **Symlog** scale options. |
| [Soft min](#soft-min-and-soft-max) | Set a soft min to better control the y-axis limits. |
| [Soft max](#soft-min-and-soft-max) | Set a soft max to better control the y-axis limits. |

<!-- prettier-ignore-end -->

#### Placement

Select the placement of the y-axis. Choose from the following:

- **Auto** - Automatically assigns the y-axis to the series. When there are two or more series with different units, Grafana assigns the left axis to the first unit and the right axis to the units that follow.
- **Left** - Display all y-axes on the left side.
- **Right** - Display all y-axes on the right side.
- **Hidden** - Hide all axes. To selectively hide axes, [Add a field override](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-overrides/#add-a-field-override) that targets specific fields.

#### Scale

Set the y-axis values scale. Choose from:

- **Linear** - Divides the scale into equal parts.
- **Logarithmic** - Use a logarithmic scale. If you select this option, in the **Log base** list, choose a binary (base 2) or common (base 10) logarithmic scale.
- **Symlog** - Use a symmetrical logarithmic scale. If you select this option, choose also:
  - **Log base** - Select a binary (base 2) or common (base 10) logarithmic scale.
  - **Linear threshold** - Set the threshold at which the scale changes from linear to logarithmic.

#### Soft min and soft max

Set a **Soft min** or **soft max** option for better control of y-axis limits. By default, Grafana sets the range for the y-axis automatically based on the dataset.

**Soft min** and **soft max** settings can prevent small variations in the data from being magnified when it's mostly flat. In contrast, hard min and max values help prevent obscuring useful detail in the data by clipping intermittent spikes past a specific point.

To define hard limits of the y-axis, set standard min/max options. For more information, refer to [Configure standard options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-standard-options/#max).
The following examples shows how this option works in a time series visualization:

![Label example](/media/docs/grafana/panels-visualizations/screenshot-soft-min-max-v12.0.png)

# Axis options

| Option    | Description                                                                                      |
| --------- | ------------------------------------------------------------------------------------------------ |
| Placement | Control the visibility of series names along the y-axis or time values along the x-axis.         |
| Width     | Set a fixed width for the axis. By default, Grafana dynamically calculates the width of an axis. |

# Cell options

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Cell value inspect | <p>Enables value inspection from table cells. When the switch is toggled on, clicking the inspect icon in a cell opens the **Inspect value** drawer which contains two tabs: **Plain text** and **Code editor**.</p><p>Grafana attempts to automatically detect the type of data in the cell and opens the drawer with the associated tab showing. However, you can switch back and forth between tabs.</p> |
| Tooltip from field | Toggle on the **Tooltip from field** switch to use the values from another field (or column) in a tooltip. For more information, refer to [Tooltip from field](#tooltip-from-field). |
| Styling from field | Toggle on the **Styling from field** switch to apply the styling from another field (or column). The referenced field must contain CSS properties formatted in JSON object syntax (for example, `{"name":"John"}`). For more information, refer to the [Styling from field](#styling-from-field). |
<!-- prettier-ignore-end -->

# Stack series link

For full instructions, refer to [Change axis display](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/time-series/change-axis-display/).

# Configuration options intro text

The following section describes the configuration options available in the panel editor pane for this visualization. These options are, as much as possible, ordered as they appear in Grafana.

# Connect null values

### Connect null values

Choose how null values, which are gaps in the data, appear on the graph. Null values can be connected to form a continuous line or set to a threshold above which gaps in the data are no longer connected.

- **Never** - Time series data points with gaps in the data are never connected.
- **Always** - Time series data points with gaps in the data are always connected.
- **Threshold** - Specify a threshold above which gaps in the data are no longer connected. This can be useful when the connected gaps in the data are of a known size and/or within a known range, and gaps outside this range should no longer be connected.

# Data links and actions options

Data links allow you to link to other panels, dashboards, and external resources while maintaining the context of the source panel.
You can create links that include the series name or even the value under the cursor.
To learn more, refer to [Configure data links and actions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-data-links/).

{{< admonition type="note" >}}
Actions are not supported for this visualization.
{{< /admonition >}}

For each data link, set the following options:

- **Title**
- **URL**
- **Open in new tab**

Data links for this visualization don't include the **One click** switch, however, if there's only one data link configured, that data link has single-click functionality.
If multiple data links are configured, then clicking the visualization opens a menu that displays all the data links.

# Data links and actions options

_Data links_ allow you to link to other panels, dashboards, and external resources and _actions_ let you trigger basic, unauthenticated, API calls.
In both cases, you can carry out these tasks while maintaining the context of the source panel.

For each data link, set the following options:

- **Title**
- **URL**
- **Open in new tab**
- **One click** - Opens the data link with a single click. Only one data link can have **One click** enabled at a time.

For each action, define the following API call settings:

<!-- prettier-ignore-start -->

| Option               | Description                                                       |
| -------------------- | ----------------------------------------------------------------- |
| Title                | A human-readable label for the action that's displayed in the UI. |
| Confirmation message | A descriptive prompt to confirm or cancel the action. |
| Connection           | Specify how the action's HTTP request is sent. Choose from: **Direct from browser** or routed through a configured data source. |
| Method               | Select from **POST**, **PUT**, or **GET**. |
| URL                  | The request URL.</p><p>To add a variable, click in the **URL** field and enter `$` or press Ctrl+Space or Cmd+Space to see a list of available variables. |
| Variables            | **Key** and **Name** pairs with a type selection. Click the **+** icon to add as many variables as you need. To add a variable to the request, prefix the key with `$`. You can set the values for the variables when performing an action. |
| Query parameters     | **Key** and **Value** pairs. Click the **+** icon to add as many key/value pairs as you need. |
| Headers              | Comprised of **Key** and **Value** pairs and a **Content-Type**.</p><p>Click the **+** icon to add as many key/value pairs as you need. |
| Content-Type         | Select from the following: **application/json**, **text/plain**, **application/XML**, and **application/x-www-form-urlencoded**. |
| Body                 | The body of the request. |

<!-- prettier-ignore-end -->

To learn more, refer to [Configure data links and actions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-data-links/).

# Data links and actions options

_Data links_ allow you to link to other panels, dashboards, and external resources and _actions_ let you trigger basic, unauthenticated, API calls.
In both cases, you can carry out these tasks while maintaining the context of the source panel.

For each data link, set the following options:

- **Title**
- **URL**
- **Open in new tab**

Data links for this visualization don't include the **One click** switch, however, if there's only one data link configured, that data link has single-click functionality.
If multiple data links are configured, then clicking the visualization opens a menu that displays all the data links.

For each action, define the following API call settings:

<!-- prettier-ignore-start -->

| Option               | Description                                                       |
| -------------------- | ----------------------------------------------------------------- |
| Title                | A human-readable label for the action that's displayed in the UI. |
| Confirmation message | A descriptive prompt to confirm or cancel the action. |
| Connection           | Specify how the action's HTTP request is sent. Choose from: **Direct from browser** or routed through a configured data source. |
| Method               | Select from **POST**, **PUT**, or **GET**. |
| URL                  | The request URL.</p><p>To add a variable, click in the **URL** field and enter `$` or press Ctrl+Space or Cmd+Space to see a list of available variables. |
| Variables            | **Key** and **Name** pairs with a type selection. Click the **+** icon to add as many variables as you need. To add a variable to the request, prefix the key with `$`. You can set the values for the variables when performing an action. |
| Query parameters     | **Key** and **Value** pairs. Click the **+** icon to add as many key/value pairs as you need. |
| Headers              | Comprised of **Key** and **Value** pairs and a **Content-Type**.</p><p>Click the **+** icon to add as many key/value pairs as you need. |
| Content-Type         | Select from the following: **application/json**, **text/plain**, **application/XML**, and **application/x-www-form-urlencoded**. |
| Body                 | The body of the request. |

<!-- prettier-ignore-end -->

To learn more, refer to [Configure data links and actions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-data-links/).

# Data links and actions options

Data links allow you to link to other panels, dashboards, and external resources while maintaining the context of the source panel.
You can create links that include the series name or even the value under the cursor.

{{< admonition type="note" >}}
Actions are not supported for this visualization.
{{< /admonition >}}

For each data link, set the following options:

- **Title**
- **URL**
- **Open in new tab**

To learn more, refer to [Configure data links and actions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-data-links/).

# Disconnect values

### Disconnect values

Choose whether to set a threshold above which values in the data should be disconnected.

- **Never** - Time series data points in the data are never disconnected.
- **Threshold** - Specify a threshold above which values in the data are disconnected. This can be useful when desired values in the data are of a known size and/or within a known range, and values outside this range should no longer be connected.

# Graph styles options

<!-- prettier-ignore-start -->

| Option                                      | Description                                                                                    |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| [Style](#style)                             | Choose whether to display your time-series data as **Lines**, **Bars**, or **Points**. |
| [Line interpolation](#line-interpolation)   | Choose how the graph interpolates the series line. |
| Line width                                  | Set the thickness of the series lines or the outline for bars using the **Line width** slider. |
| Fill opacity                                | Set the series area fill color using the **Fill opacity** slider. |
| [Gradient mode](#gradient-mode)             | Choose a gradient mode to control the gradient fill, which is based on the series color. |
| [Line style](#line-style)                   | Choose a solid, dashed, or dotted line style. |
| [Connect null values](#connect-null-values) | Choose how null values, which are gaps in the data, appear on the graph. |
| [Disconnect values](#disconnect-values)     | Choose whether to set a threshold above which values in the data should be disconnected. |
| [Show points](#show-points)                 | Set whether to show data points to lines or bars. |
| Show values                                 | Set whether to show data point values above the lines or bars. |
| Point size                                  | Set the size of the points, from 1 to 40 pixels in diameter. |
| [Stack series](#stack-series)               | Set whether Grafana displays series on top of each other. |
| [Bar alignment](#bar-alignment)             | Set the position of the bar relative to a data point. |
| Bar width factor                            | Set the width of the bar relative to minimum space between data points. A factor of 0.5 means that the bars take up half of the available space between data points. A factor of 1.0 means that the bars take up all available space. |

<!-- prettier-ignore-end -->

#### Style

Choose whether to display your time-series data as **Lines**, **Bars**, or **Points**. You can use overrides to combine multiple styles in the same graph. Choose from the following:

![Graph style examples](/media/docs/grafana/panels-visualizations/screenshot-time-style-v12.0.png)

#### Line interpolation

Choose how the graph interpolates the series line:

- **Linear** - Points are joined by straight lines.
- **Smooth** - Points are joined by curved lines that smooths transitions between points.
- **Step before** - The line is displayed as steps between points. Points are rendered at the end of the step.
- **Step after** - The line is displayed as steps between points. Points are rendered at the beginning of the step.

![Line interpolation examples](/media/docs/grafana/panels-visualizations/screenshot-time-interpolation-v12.0.png)

#### Gradient mode

Choose a gradient mode to control the gradient fill, which is based on the series color. To change the color, use the standard color scheme field option. For more information, refer to [Color scheme](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-standard-options/#color-scheme).

- **None** - No gradient fill. This is the default setting.
- **Opacity** - An opacity gradient where the opacity of the fill increases as y-axis values increase.
- **Hue** - A subtle gradient that's based on the hue of the series color.
- **Scheme** - A color gradient defined by your [Color scheme](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-standard-options/#color-scheme). This setting is used for the fill area and line. For more information about scheme, refer to [Scheme gradient mode](#scheme-gradient-mode).

Gradient appearance is influenced by the **Fill opacity** setting. The following image shows the **Fill opacity** set to 50.

![Gradient mode examples](/media/docs/grafana/panels-visualizations/screenshot-time-gradient-v12.0.png)

##### Scheme gradient mode

In **Scheme** gradient mode, the line or bar receives a gradient color defined from the selected **Color scheme** option in the visualization's **Standard** options.

The following image shows a line chart with the **Green-Yellow-Red (by value)** color scheme option selected:

{{< figure src="/static/img/docs/time-series-panel/gradient_mode_scheme_line.png" max-width="600px" alt="Color scheme: Green-Yellow-Red" >}}

If the **Color scheme** is set to **From thresholds (by value)** and **Gradient mode** is set to **Scheme**, then the line or bar color changes as it crosses the defined thresholds:

{{< figure src="/static/img/docs/time-series-panel/gradient_mode_scheme_thresholds_line.png" max-width="600px" alt="Colors scheme: From thresholds" >}}

#### Line style

Choose a solid, dashed, or dotted line style:

- **Solid** - Display a solid line. This is the default setting.
- **Dash** - Display a dashed line. When you choose this option, a list appears for you to select the length and gap (length, gap) for the line dashes. Dash spacing is 10, 10 by default.
- **Dots** - Display dotted lines. When you choose this option, a list appears for you to select the gap (length = 0, gap) for the dot spacing. Dot spacing is 0, 10 by default.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-time-line-style-v12.0.png" max-width="550px" alt="Line style examples" >}}

{{< docs/shared lookup="visualizations/connect-null-values.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="visualizations/disconnect-values.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

To change the color, use the standard [color scheme](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-standard-options/#color-scheme) field option.

#### Show points

Set whether to show data points as lines or bars. Choose from the following:

- **Auto** - Grafana determines a point's visibility based on the density of the data. If the density is low, then points appear.
- **Always** - Show the points regardless of how dense the dataset is.
- **Never** - Don't show points.

#### Stack series

Set whether Grafana stacks or displays series on top of each other. Be cautious when using stacking because it can create misleading graphs. To read more about why stacking might not be the best approach, refer to [The issue with stacking](https://www.data-to-viz.com/caveat/stacking.html). Choose from the following:

- **Off** - Turns off series stacking. When **Off**, all series share the same space in the visualization.
- **Normal** - Stacks series on top of each other.
- **100%** - Stack by percentage where all series add up to 100%.

##### Stack series in groups

The stacking group option is only available as an override. For more information about creating an override, refer to [Configure field overrides](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-overrides/).

1. Edit the panel and click **Overrides**.
1. Create a field override for the **Stack series** option.
1. In stacking mode, click **Normal**.
1. Name the stacking group in which you want the series to appear.

   The stacking group name option is only available when you create an override.

#### Bar alignment

Set the position of the bar relative to a data point. In the examples below, **Show points** is set to **Always** which makes it easier to see the difference this setting makes. The points don't change, but the bars change in relationship to the points. Choose from the following:

- **Before** ![Bar alignment before icon](/static/img/docs/time-series-panel/bar-alignment-before.png)
  The bar is drawn before the point. The point is placed on the trailing corner of the bar.
- **Center** ![Bar alignment center icon](/static/img/docs/time-series-panel/bar-alignment-center.png)
  The bar is drawn around the point. The point is placed in the center of the bar. This is the default.
- **After** ![Bar alignment after icon](/static/img/docs/time-series-panel/bar-alignment-after.png)
  The bar is drawn after the point. The point is placed on the leading corner of the bar.

# Legend options

Legend options control the series names and statistics that appear under or to the right of the graph. For more information about the legend, refer to [Configure a legend](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-legend/).

| Option     | Description                                                                                                                                                                                                                                                                                                                                        |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Visibility | Toggle the switch to turn the legend on or off.                                                                                                                                                                                                                                                                                                    |
| Mode       | Use these settings to define how the legend appears in your visualization. **List** displays the legend as a list. This is a default display mode of the legend. **Table** displays the legend as a table.                                                                                                                                         |
| Placement  | Choose where to display the legend. **Bottom** places the legend below the graph. **Right** places the legend to the right of the graph.                                                                                                                                                                                                           |
| Width      | Control how wide the legend is when placed on the right side of the visualization. The value may be a basic pixel width or a more complex [CSS width property](https://developer.mozilla.org/en-US/docs/Web/CSS/Reference/Properties/width) such as `max(200px, 20%)`. This option is only displayed if you set the legend placement to **Right**. |
| Limit      | Limit how many series items are shown by default. The rest become expandable using a **Show all** link.                                                                                                                                                                                                                                            |
| Values     | Choose which of the [standard calculations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/calculation-types/) to show in the legend. You can have more than one.                                                                                                                                   |
| Overflow   | Control how overflow for long series names is handled. **Ellipsis** truncates the series names with "..." at the end. **Wrap** word-wraps the series names to multiple lines. This option is only displayed if you set the legend placement to **Right** and mode is **Table**.                                                                    |

# Legend mode and legend placement

When the legend option is enabled it can show either the value mappings or the threshold brackets. To show the value mappings in the legend, it's important that the **Color scheme** as referenced in [Color scheme](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-standard-options/#color-scheme) is set to **Single color** or **Classic palette**. To see the threshold brackets in the legend set the **Color scheme** to **From thresholds**.

For more information about the legend, refer to [Configure a legend](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-legend/).

### Visibility

Toggle the switch to turn the legend on or off.

### Mode

Use these settings to define how the legend appears in your visualization.

- **List -** Displays the legend as a list. This is a default display mode of the legend.
- **Table -** Displays the legend as a table.

### Placement

Choose where to display the legend.

- **Bottom -** Below the graph.
- **Right -** To the right of the graph.

### Width

Control how wide the legend is when placed on the right side of the visualization. The value may be a simple pixel width or a more complex [CSS width property](https://developer.mozilla.org/en-US/docs/Web/CSS/Reference/Properties/width) such as `max(200px, 20%)`. This option is only displayed if you set the legend placement to **Right**.

### Overflow

Control how overflow for long series names is handled. This option is only displayed if you set the legend placement to **Right** and mode is **Table**.

- **Ellipsis -** Truncates the series names with "..." at the end.
- **Wrap -** Word-wraps the series names to multiple lines.

# Multiple y-axes

In some cases, you might want to display multiple y-axes. For example, if you have a dataset showing both temperature and humidity over time, you might want to show two y-axes with different units for the two series.

You can configure multiple y-axes and control where they're displayed in the visualization by adding field overrides. [This example of a dataset that includes temperature and humidity](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-overrides/#example-2-format-temperature-and-humidity) describes how you can configure that. Repeat the steps for every y-axis you wish to display.

# Field overrides options

Overrides allow you to customize visualization settings for specific fields or series. When you add an override rule, it targets a particular set of fields and lets you define multiple options for how that field is displayed.

Choose from the following override options:

| Option                         | Description                                                                                                   |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------- |
| Fields with name               | Select a field from the list of all available fields.                                                         |
| Field with name matching regex | Specify fields to override with a regular expression.                                                         |
| Fields with type               | Select fields by type, such as string, numeric, or time.                                                      |
| Fields returned by query       | Select all fields returned by a specific query, such as A, B, or C.                                           |
| Fields with values             | Select all fields returned by your defined reducer condition, such as **Min**, **Max**, **Count**, **Total**. |

To learn more, refer to [Configure field overrides](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-overrides/).

# Panel options

In the **Panel options** section of the panel editor pane, set options like the panel title, description, and show/hide rules.
To learn more, refer to [Configure panel options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-panel-options/).

<!-- prettier-ignore-start -->

| Option                 | Description                                                                           |
| ---------------------- | ------------------------------------------------------------------------------------- |
| Title                  | Text entered in this field appears at the top of your panel in the panel editor and in the dashboard. You can use [variables you have defined](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/) or [built-in variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/built-in-variables). |
| Description            | Add a description to a panel to share with users any important information about it, such as its purpose. Text entered in this field appears in a tooltip in the upper-left corner of the panel. Conversely, toggle on the **Use as subtitle** switch to display the description just below the panel title instead. You can use [variables you have defined](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/) or dashboard-scoped [built-in variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/built-in-variables) (such as `$__from`, `$__to`, `$__dashboard`, `$__org`, and `$__user`). Query-scoped global variables (such as `$__interval` and `$__interval_ms`) are not available in this field. |
| Transparent background | Toggle this switch on and off to control whether or not the panel has the same background color as the dashboard. | 
| Panel links | Add [links to the panel](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/manage-dashboard-links/#panel-links) to create shortcuts to other dashboards, panels, and external websites. Access panel links by clicking the icon next to the panel title. |
| Repeat options         | Set whether to repeat the panel for each value in the selected variable. |
| Show/hide rules        | Set whether to show or hide panels based on various rules. |

<!-- prettier-ignore-end -->

# Panel mouse time zoom

You can pan the panel time range left and right, and zoom it and in and out.
This, in turn, changes the dashboard time range.

**Zoom in** - Click and drag on the panel to zoom in on a particular time range.

**Zoom out** - Double-click anywhere on the panel to zoom out the time range.

When you zoom out, the range doubles with each double-click, adding equal time to each side of the range.
For example, if the original time range is from 9:00 to 9:59, the time range changes as follow with each double-click:

- Next range: 8:30 - 10:29
- Next range: 7:30 - 11:29

**Pan** - Click and drag the x-axis area of the panel to pan the time range.

The time range shifts by the distance you drag.
For example, if the original time range is from 9:00 to 9:59 and you drag 30 minutes to the right, the time range changes to 9:30 to 10:29.

For screen recordings showing these interactions, refer to the [Panel overview documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/panel-overview/#pan-and-zoom-panel-time-range).

# Stack series link

### Stack series

For full instructions, refer to [Graph stacked time series](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/time-series/graph-time-series-stacking/).

# Standard options

**Standard options** in the panel editor pane let you change how field data is displayed in your visualizations. When you set a standard option, the change is applied to all fields or series. For more granular control over the display of fields, refer to [Configure overrides](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-overrides/).

| Option        | Description                                                                                                                                       |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| Unit          | Choose which unit a field should use.                                                                                                             |
| Min/Max       | Set the minimum and maximum values used in percentage threshold calculations or leave these field empty for them to be calculated automatically.  |
| Field min/max | Enable **Field min/max** to have Grafana calculate the min or max of each field individually, based on the minimum or maximum value of the field. |
| Decimals      | Specify the number of decimals Grafana includes in the rendered value.                                                                            |
| Display name  | Set the display title of all fields. You can use variables in the field title.                                                                    |
| Color scheme  | Set single or multiple colors for your entire visualization.                                                                                      |
| No value      | Enter what Grafana should display if the field value is empty or null. The default value is a hyphen (-).                                         |

To learn more, refer to [Configure standard options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-standard-options/).

# Thresholds options

A threshold is a value or limit you set for a metric that’s reflected visually when it’s met or exceeded. Thresholds are one way you can conditionally style and color your visualizations based on query results.

For each threshold, set the following options:

| Option          | Description                                                                          |
| --------------- | ------------------------------------------------------------------------------------ |
| Value           | Set the value for each threshold.                                                    |
| Thresholds mode | Choose from **Absolute** and **Percentage**.                                         |
| Show thresholds | Choose from a variety of display options including not displaying thresholds at all. |

To learn more, refer to [Configure thresholds](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-thresholds/).

# Thresholds options

A threshold is a value or limit you set for a metric that’s reflected visually when it’s met or exceeded. Thresholds are one way you can conditionally style and color your visualizations based on query results.

For each threshold, set the following options:

| Option          | Description                                  |
| --------------- | -------------------------------------------- |
| Value           | Set the value for each threshold.            |
| Thresholds mode | Choose from **Absolute** and **Percentage**. |

To learn more, refer to [Configure thresholds](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-thresholds/).

# Tooltip options

Tooltip options control the information overlay that appears when you hover over data points in the visualization.

| Option                                  | Description                                                                                                                                                                            |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Tooltip mode](#tooltip-mode)           | When you hover your cursor over the visualization, Grafana can display tooltips. Choose how tooltips behave.                                                                           |
| [Values sort order](#values-sort-order) | This option controls the order in which values are listed in a tooltip.                                                                                                                |
| Hide zeros                              | When you set the **Tooltip mode** to **All**, the **Hide zeros** option is displayed. This option controls whether or not series with `0` values are shown in the list in the tooltip. |
| Max width                               | Set the maximum width of the tooltip box.                                                                                                                                              |
| Max height                              | Set the maximum height of the tooltip box. The default is 600 pixels.                                                                                                                  |

### Tooltip mode

When you hover your cursor over the visualization, Grafana can display tooltips. Choose how tooltips behave.

- **Single** - The hover tooltip shows only a single series, the one that you are hovering over on the visualization.
- **All** - The hover tooltip shows all series in the visualization. Grafana highlights the series that you are hovering over in bold in the series list in the tooltip.
- **Hidden** - Do not display the tooltip when you interact with the visualization.

Use an override to hide individual series from the tooltip.

### Values sort order

When you set the **Tooltip mode** to **All**, the **Values sort order** option is displayed. This option controls the order in which values are listed in a tooltip. Choose from the following:

- **None** - Grafana automatically sorts the values displayed in a tooltip.
- **Ascending** - Values in the tooltip are listed from smallest to largest.
- **Descending** - Values in the tooltip are listed from largest to smallest.

# Tooltip options

Tooltip options control the information overlay that appears when you hover over data points in the visualization.

| Option                                  | Description                                                                                                                                                                            |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Tooltip mode](#tooltip-mode)           | When you hover your cursor over the visualization, Grafana can display tooltips. Choose how tooltips behave.                                                                           |
| [Values sort order](#values-sort-order) | This option controls the order in which values are listed in a tooltip.                                                                                                                |
| Hide zeros                              | When you set the **Tooltip mode** to **All**, the **Hide zeros** option is displayed. This option controls whether or not series with `0` values are shown in the list in the tooltip. |
| [Hover proximity](#hover-proximity)     | Set the hover proximity (in pixels) to control how close the cursor must be to a data point to trigger the tooltip to display.                                                         |
| Max width                               | Set the maximum width of the tooltip box.                                                                                                                                              |
| Max height                              | Set the maximum height of the tooltip box. The default is 600 pixels.                                                                                                                  |

### Tooltip mode

When you hover your cursor over the visualization, Grafana can display tooltips. Choose how tooltips behave.

- **Single -** The hover tooltip shows only a single series, the one that you are hovering over on the visualization.
- **All -** The hover tooltip shows all series in the visualization. Grafana highlights the series that you are hovering over in bold in the series list in the tooltip.
- **Hidden -** Do not display the tooltip when you interact with the visualization.

Use an override to hide individual series from the tooltip.

### Values sort order

When you set the **Tooltip mode** to **All**, the **Values sort order** option is displayed. This option controls the order in which values are listed in a tooltip. Choose from the following:

- **None** - Grafana automatically sorts the values displayed in a tooltip.
- **Ascending** - Values in the tooltip are listed from smallest to largest.
- **Descending** - Values in the tooltip are listed from largest to smallest.

### Hover proximity

Set the hover proximity (in pixels) to control how close the cursor must be to a data point to trigger the tooltip to display.
The following screen recording shows this option in a time series visualization:

![Adding a hover proximity limit for tooltips](/media/docs/grafana/gif-grafana-10-4-hover-proximity.gif)

# Tooltip options

Tooltip options control the information overlay that appears when you hover over data points in the visualization.

| Option                                  | Description                                                                                                  |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| [Tooltip mode](#tooltip-mode)           | When you hover your cursor over the visualization, Grafana can display tooltips. Choose how tooltips behave. |
| [Values sort order](#values-sort-order) | This option controls the order in which values are listed in a tooltip.                                      |
| Max width                               | Set the maximum width of the tooltip box.                                                                    |
| Max height                              | Set the maximum height of the tooltip box. The default is 600 pixels.                                        |

### Tooltip mode

When you hover your cursor over the visualization, Grafana can display tooltips. Choose how tooltips behave.

- **Single** - The hover tooltip shows only a single series, the one that you are hovering over on the visualization.
- **All** - The hover tooltip shows all series in the visualization. Grafana highlights the series that you are hovering over in bold in the series list in the tooltip.
- **Hidden** - Do not display the tooltip when you interact with the visualization.

Use an override to hide individual series from the tooltip.

### Values sort order

When you set the **Tooltip mode** to **All**, the **Values sort order** option is displayed. This option controls the order in which values are listed in a tooltip. Choose from the following:

- **None** - Grafana automatically sorts the values displayed in a tooltip.
- **Ascending** - Values in the tooltip are listed from smallest to largest.
- **Descending** - Values in the tooltip are listed from largest to smallest.

# Value mappings options

Value mapping is a technique you can use to change how data appears in a visualization.

For each value mapping, set the following options:

- **Condition** - Choose what's mapped to the display text and (optionally) color:
  - **Value** - Specific values
  - **Range** - Numerical ranges
  - **Regex** - Regular expressions
  - **Special** - Special values like `Null`, `NaN` (not a number), or boolean values like `true` and `false`
- **Display text**
- **Color** (Optional)
- **Icon** (Canvas only)

To learn more, refer to [Configure value mappings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-value-mappings/).

# Annotation panel options

| Option                | Description                                                                                                                               |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Multi-row annotations | Breaks each annotation data frame into a separate row in the visualization. Only visible in panels that display more than one annotation. |
| Annotation clustering | Joins adjacent point annotations into a single annotation region.                                                                         |
| Hide lines and areas  | Controls display of annotation indicator lines, and shaded area for region annotations.                                                   |

# Troubleshooting

# Generate a data source diagnostic bundle

{{< admonition type="note" >}}
On-demand data source diagnostics is an [experimental](https://grafana.com/docs/release-life-cycle/#experimental) feature that helps you share troubleshooting evidence with Grafana Labs Technical Support. The feature is disabled by default. To switch it on, refer to [Enable the feature toggle](#enable-the-feature-toggle).
{{< /admonition >}}

When a panel returns an error, no data, or data that looks wrong, Grafana Labs Technical Support needs to know what your data source actually returned. In Grafana Cloud, you can grant support access to your stack so an engineer can investigate live. A self-managed instance isn't reachable by support, so you collect the evidence yourself in a diagnostic bundle and send it in.

When you generate a diagnostic bundle, Grafana re-runs the panel's queries with traffic capture active and packages the result as a single `.tar.gz` file containing the requests and responses exchanged with your data source, the data the data source plugin returned to Grafana, the panel and dashboard configuration, and metadata about the capture.

{{< admonition type="warning" >}}
A diagnostic bundle records upstream traffic and error messages verbatim, without redaction. Request headers, query parameters, request and response bodies, and error text are all stored as they occurred, so a bundle can contain authentication tokens, passwords, and the contents of the data your queries returned.

Treat a bundle as you would treat a credential. **Review it before you share it**, and **send it only to Grafana Labs Technical Support**, through your Grafana Labs support ticket.
{{< /admonition >}}

## Before you begin

To generate a diagnostic bundle, you need the following:

- A self-managed Grafana instance running version 13.2 or later. Diagnostic bundles aren't available in Grafana Cloud, and earlier self-managed versions don't include this feature.
- The Grafana server administrator role. The menu items described on this page don't appear for other users.
- The `grafana.onDemandDiagnostics` feature toggle enabled.
- A data source whose plugin supports diagnostic bundles. Refer to [Data source support for capturing upstream traffic](#data-source-support-for-capturing-upstream-traffic).

### Enable the feature toggle

The `grafana.onDemandDiagnostics` feature toggle is disabled by default. While it's disabled, Grafana hides the diagnostics UI elements and the underlying endpoints return `404`.

To enable it, add the following to your Grafana configuration file and restart Grafana:

```ini
[feature_toggles]
enable = grafana.onDemandDiagnostics
```

To set a feature toggle in other ways, such as environment variables, refer to the [`feature_toggles` section of Configure Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#feature_toggles) and [Manage feature toggles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/feature-toggles/#manage-feature-toggles).

## Generate a bundle for a single panel

1. Open the dashboard that contains the panel you want to diagnose.

1. Hover over any part of the panel to display the panel menu in the top right corner.

1. Click the menu and select **More** > **Download diagnostics**.

   Grafana opens a drawer describing what the bundle contains, including a **May contain sensitive data** warning.

1. Click **Download diagnostics**.

   Grafana re-runs the panel's queries with capture active and downloads the bundle. The button reads **Generating…** while this happens, which can take a moment.

1. Review the bundle, then attach it to your support ticket.

## Generate a bundle for an entire dashboard

Use a dashboard bundle when you don't know which panel is at fault, or when the problem involves more than one panel.

1. Open the dashboard you want to diagnose.

1. In the dashboard toolbar, click **Export** > **Download diagnostics**.

1. Click **Download diagnostics**.

   Grafana captures panels one at a time and reports its progress, for example **Capturing panel 3 of 12**. Large dashboards can take a while to capture.

1. Review the bundle, then attach it to your support ticket.

Grafana captures each panel independently, so one failing panel doesn't stop the rest of the bundle from being produced. Skipped and failed panels are recorded in `manifest.json`.

## What the bundle contains

Extract the bundle to review it:

```bash
tar -xzf <bundle>.tar.gz
```

A single-panel bundle contains the following files:

| File                  | Contents                                                                                                                                                                                                                                                                                                                                                                                  |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `traffic.har`         | The HTTP requests and responses exchanged with the data source, in [HAR 1.2](http://www.softwareishard.com/blog/har-12-spec/) format, including bodies and timings. Connection failures are recorded as an entry with a comment. Not available for every data source; refer to [Data source support for capturing upstream traffic](#data-source-support-for-capturing-upstream-traffic). |
| `querydata.json`      | The queries Grafana submitted and the data the data source plugin returned, including per-query status, errors, and the schema and values of each returned frame. Large responses are reduced to a summary that records what was truncated.                                                                                                                                               |
| `query-error.txt`     | The query error text, when the queries failed.                                                                                                                                                                                                                                                                                                                                            |
| `querydata-error.txt` | Present only when the query data couldn't be written. The rest of the bundle is still produced.                                                                                                                                                                                                                                                                                           |
| `panel.json`          | The panel configuration: queries, transformations, field options, and visualization settings.                                                                                                                                                                                                                                                                                             |
| `dashboard.json`      | The dashboard configuration, for context.                                                                                                                                                                                                                                                                                                                                                 |

A dashboard bundle contains a `panels/<id>-<title>/` directory holding those files for each panel, plus a `manifest.json` recording when the bundle was generated, how many panels ran, and the data sources, sizes, and any errors for each panel.

## Data source support for capturing upstream traffic

`traffic.har` is usually the most valuable file in the bundle, but it isn't available for every data source. When upstream traffic can't be captured, Grafana still generates the rest of the bundle, and `querydata.json` still shows what the data source plugin returned.

### Data sources that support capture today

- Amazon Managed Service for Prometheus
- Azure Data Explorer
- Azure Monitor
- Azure Prometheus
- CSV
- Dynatrace
- Elasticsearch
- Falcon LogScale
- Google Sheets
- Graphite
- InfluxDB
- Jaeger
- Loki
- OpenTSDB
- Prometheus
- Sentry
- Tempo, for its standard HTTP query path—TraceQL streaming is not supported.
- Zabbix
- Zipkin

## Limitations

As an experimental feature, on-demand data source diagnostics currently has the following limitations:

- **Grafana re-runs your queries.** A bundle reflects a fresh execution at the moment you generate it, not the original failure, and it bypasses the query cache. An intermittent failure might not reproduce, so a healthy-looking bundle doesn't prove the problem is resolved.

- **Queries run as you.** Because you generate the bundle as a server administrator, failures that depend on the affected user's identity might not reproduce. Examples include per-user OAuth forwarding and row-level database permissions. If a specific user is affected, note that in your ticket.

- A bundle captures panel and dashboard queries only. Failures in the variable picker, in a data source's **Save & test** button, in annotation queries, or during alert rule evaluation don't appear.

- Server-side expressions and panel transformations are recorded in `panel.json` as configuration, not as the data that flows between them. So if `querydata.json` looks correct but the panel doesn't, check that configuration.

- **Large responses might be trimmed**, and less commonly, generation fails outright. The following limits apply:
  - `querydata.json` is capped at 1 GiB for a single panel and 1.5 GiB across a whole-dashboard bundle. A response over that limit is replaced by a summary that records what was left out, and the rest of the bundle still generates. For a dashboard bundle, once the shared 1.5 GiB budget is used up, the remaining panels' query data is skipped, and the reason is recorded in `manifest.json`.
  - `traffic.har` is limited to 64 MiB per request or response body and 256 MiB of retained traffic per panel. A body over the per-body limit is kept as a truncated prefix, and its entry reports `bodySize: -1` with the actual size in `content.size`. When a panel's traffic exceeds the per-panel limit, further entries keep their headers, sizes, and timings but omit the body text.
- Generating a bundle can fail outright, instead of trimming, when the entire request to Grafana exceeds 100 MiB. That request includes the queries, the panel and dashboard JSON, and, for a single panel, the frames your browser is holding. This is uncommon, because it requires an unusually large payload from the browser itself—which a large data source response alone doesn't produce.

  If you hit one of these limits, or a bundle takes an unreasonably long time to generate, reproduce the problem on a minimal setup instead of the original dashboard:
  - Use a temporary panel or dashboard holding only the misbehaving panel.
  - Narrow the time range to the shortest window that still reproduces the problem. Fewer returned data points means a smaller bundle and a faster capture.

  If you generate the bundle from a reduced dashboard, note that in your ticket. A smaller, reproducible setup is also easier for Grafana Labs Technical Support to work with.

## Related resources

- [Send a panel to Grafana Labs support](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/troubleshooting/send-panel-to-grafana-support/)
- [Generate a HAR capture to send to Grafana Labs support](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/troubleshooting/har-captures/)
- [Troubleshoot data sources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/troubleshooting/)

# Troubleshoot with HAR captures

Support may request information about the network requests your browser generated in the form of a HAR file.

Important: HAR files can contain sensitive information, including cookies, session data, personal information, and sometimes credentials or tokens entered during troubleshooting. Before sharing a HAR file, make sure you review or sanitize it. You can then attach the redacted HAR file to your Grafana Support ticket.

To generate a HAR capture, follow the steps for your browser:

## Chrome

1. In your Chrome browser, go to the page where the issue is occurring.
1. Click the menu icon and go to **More Tools** > **Developer Tools**.
1. In the drawer that opens, click the **Network** tab.
1. In the upper left hand corner of the drawer, make sure the round Record button is red. If it's grey, click the button to begin recording.
1. Select the **Preserve log** option box if it isn't already selected.
1. Click the grey crossed circle button to clear any existing requests from the panel.
1. Reproduce the issue while recording the network requests.
1. Click the download button to download the HAR file.
   <img width="1293" height="239" alt="image" src="https://github.com/user-attachments/assets/2ddd5bd4-d207-4c59-8143-f10009e0cb5d" />
1. Upload the HAR file to your Grafana Support ticket for investigation.

## Firefox

1. In your Firefox browser, go to the page where the issue is occurring.
1. Click the menu icon and go to **More Tools** > **Web Developer Tools**.
1. In the drawer that opens, click the **Network** tab.
1. In the upper left corner of the drawer, click the trash can icon to clear any existing requests from the recording.
1. Reproduce the issue while recording the network requests.
1. To save the HAR file, click the gear icon in the upper right hand corner of the panel and select **Save All As HAR**.
   <img width="1837" height="196" alt="image" src="https://github.com/user-attachments/assets/785be5e4-10a2-4d16-838a-2b7bb97c8453" />
1. Upload the HAR file to your Grafana Support ticket for investigation.

## Safari

1. In your Safari browser, go to the page where the issue is occurring.
1. Open the **Develop** menu option and select **Show Web Inspector**.
   {{< admonition type="note" >}}
   The **Show features for web developers** option needs to be enabled for the **Develop** option to appear. For more information, refer to [Use the developer tools in the Develop menu in Safari on Mac](https://support.apple.com/en-ie/guide/safari/sfri20948/mac).
   {{< /admonition >}}
1. In the panel that appears, click the **Network** tab.
1. In the upper right corner of the panel, click the trash can icon to clear any existing requests from the recording.
1. Reproduce the issue while recording the network requests.
1. Click the **Export** button to save the HAR file:
   <img width="1915" height="219" alt="image" src="https://github.com/user-attachments/assets/2bda7bbd-43bf-4ecb-b8f9-f0be92c5a480" />
1. Upload the HAR file to your Grafana Support ticket for investigation.

## Edge

1. In your Edge browser, go to the page where the issue is occurring.
1. Click the ... button and go to **More Tools** > **Developer Tools**.
1. In the panel that appears, click on the **Network** tab.
1. In the upper left hand corner of the panel, make sure the round Record button is red. If it's grey, click the button to begin recording
1. Verify that the **Preserve log** option box is checked.
1. Click the grey crossed circle button to clear any existing requests from the panel.
1. Reproduce the issue while the network requests are recorded.
1. Click the download button to save the HAR file:
   <img width="916" height="305" alt="image" src="https://github.com/user-attachments/assets/54dfd40d-751f-47b4-9824-bc5be1f027fa" />
1. Upload the HAR file to your Grafana Support ticket for investigation.

# Send a panel to Grafana Labs support

When you encounter problems with any of your visualizations, you can send the panel JSON model to Grafana Labs Technical Support and request help with troubleshooting your issue.

The panel that you send includes all query response data and all visualizations settings. Upon receiving your panel, Grafana Labs Technical Support imports your data into a local version of Grafana and begins researching your problem.

1. Open the dashboard that contains the panel you want to send to Grafana Labs.

1. Hover over any part of the panel to display the actions menu on the top right corner.

1. Click the menu and select **More > Get help**.

   Grafana opens a standalone support dashboard that contains the data you are sending to Grafana Labs Technical Support.

   If you enable iframes, the support dashboard includes the visualization that looks similar to the following image.

   ![Support panel](/static/img/docs/troubleshooting/grafana-support-panel.png)

1. To send the panel data to Grafana Labs via Github:

   a. Click **Copy to clipboard**.

   b. In the [Grafana/Grafana](https://github.com/grafana/grafana) repository, create an issue, and paste the contents of the support dashboard.

1. To send the panel data to Grafana Labs via a support ticket:

   a. Click **Dashboard**.

   Grafana downloads the support dashboard to a TXT file.

   b. Attach the TXT file to a support ticket that you send to Grafana Labs Technical Support.

# Tutorials

# Get started with Grafana Alerting - Create and receive your first alert

<!-- INTERACTIVE page intro.md START -->

In this tutorial, we walk you through the process of setting up your first alert in just a few minutes. You'll witness your alert in action with real-time data, as well as sending alert notifications.

{{< youtube id="YVhKNbQNNss" >}}

In this tutorial you will:

- Create a contact point.
- Set up an alert rule.
- Receive firing and resolved alert notifications in a public webhook.

<!-- INTERACTIVE ignore START -->

{{< admonition type="tip" >}}

After you have completed Part 1, don’t forget to explore the advanced but essential alerting topics in [Multi-dimensional alerts and how to route them](http://www.grafana.com/tutorials/alerting-get-started-pt2/).

{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

> After you have completed Part 1, don’t forget to explore the advanced but essential alerting topics in [Multi-dimensional alerts and how to route them](http://www.grafana.com/tutorials/alerting-get-started-pt2/).

{{< /docs/ignore >}}

<!-- INTERACTIVE page intro.md END -->
<!-- INTERACTIVE page step1.md START -->
<!-- INTERACTIVE ignore START -->

## Before you begin

There are different ways you can follow along with this tutorial.

- **Grafana Cloud**
  - As a Grafana Cloud user, you don't have to install anything. [Create your free account](http://www.grafana.com/auth/sign-up/create-user).

  Continue to [Create a contact point](#create-a-contact-point).

- **Interactive learning environment**
  - Alternatively, you can [try out this example in our interactive learning environment](https://killercoda.com/grafana-labs/course/grafana/alerting-get-started/). It's a fully configured environment with all the dependencies already installed.

- **Grafana OSS**
  - If you opt to run a Grafana stack locally, ensure you have the following applications installed:

  - [Docker Compose](https://docs.docker.com/get-docker/) (included in Docker for Desktop for macOS and Windows)
  - [Git](https://git-scm.com/)

### Set up the Grafana stack (OSS users)

<!-- INTERACTIVE ignore END -->

To demonstrate the observation of data using the Grafana stack, download and run the following files.

1. Clone the [tutorial environment repository](https://www.github.com/grafana/tutorial-environment).

   <!-- INTERACTIVE exec START -->

   ```
   git clone https://github.com/grafana/tutorial-environment.git
   ```

   <!-- INTERACTIVE exec END -->

1. Change to the directory where you cloned the repository:

   <!-- INTERACTIVE exec START -->

   ```
   cd tutorial-environment
   ```

   <!-- INTERACTIVE exec END -->

1. Run the Grafana stack:

   <!-- INTERACTIVE ignore START -->

   ```
   docker compose up -d
   ```

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   <!-- INTERACTIVE exec START -->

   ```bash
   docker-compose up -d
   ```

   <!-- INTERACTIVE exec END -->

   {{< /docs/ignore >}}

   The first time you run `docker compose up -d`, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

   <!-- INTERACTIVE ignore START -->

   {{< admonition type="note" >}}
   If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.
   {{< /admonition >}}

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   NOTE:

   If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

   {{< /docs/ignore >}}

<!-- INTERACTIVE page step1.md END -->
<!-- INTERACTIVE page step2.md START -->

## Create a contact point

Besides being an open-source observability tool, Grafana has its own built-in alerting service. This means that you can receive notifications whenever there is an event of interest in your data, and even see these events graphed in your visualizations.

In this step, we set up a new contact point. This contact point uses the [webhook integration](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier/). In order to make this work, we also need an endpoint for our webhook integration to receive the alert. We can use [Webhook.site](https://webhook.site/) to quickly set up that test endpoint. This way we can make sure that our alert is actually sending a notification somewhere.

1. In your browser, **sign in** to your Grafana Cloud account.

   OSS users: To log in, navigate to [http://localhost:3000](http://localhost:3000), where Grafana is running.

1. In another tab, go to [Webhook.site](https://webhook.site/).
1. Copy Your unique URL.

Your webhook endpoint is now waiting for the first request.

Next, let's configure a contact point in Grafana's Alerting UI to send notifications to our webhook endpoint.

1. Return to Grafana. In Grafana's sidebar, hover over the **Alerting** (bell) icon and then click **Contact points**.
1. Click **+ New contact point**.
1. In **Name**, write **Webhook**.
1. In **Integration**, choose **Webhook**.
1. In **URL**, paste the endpoint to your webhook endpoint.
1. Click **Test**, and then click **Send test notification** to send a test alert to your webhook endpoint.
1. Navigate back to _Webhook.site_. On the left side, there's now a `POST /` entry. Click it to see what information Grafana sent.

   {{< figure src="/media/docs/alerting/alerting-webhook-detail.png" max-width="1200px" caption="A POST entry in Webhook.site" >}}

1. Return to Grafana and click **Save contact point**.

We have created a dummy Webhook endpoint and created a new Alerting contact point in Grafana. Now, we can create an alert rule and link it to this new integration.

<!-- INTERACTIVE page step2.md END -->

<!-- INTERACTIVE page step3.md START -->

## Create an alert

Next, we establish an [alert rule](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-grafana-managed-rule/) within Grafana Alerting to notify us whenever alert rules are triggered and resolved.

1. In Grafana, navigate to **Alerts & IRM > Alerting > Alert rules**. Click on **+ New alert rule**.

1. Enter alert rule name for your alert rule. Make it short and descriptive as this appears in your alert notification. For instance, **database-metrics**

### Define query and alert condition

In this section, we use the default options for Grafana-managed alert rule creation. The default options let us define the query, a expression (used to manipulate the data -- the `WHEN` field in the UI), and the condition that must be met for the alert to be triggered (in default mode is the threshold).

Grafana includes a [test data source](https://grafana.com/docs/grafana/latest/datasources/testdata/) that creates simulated time series data. This data source is included in the demo environment for this tutorial. If you're working in Grafana Cloud or your own local Grafana instance, you can add the data source through the **Connections** menu.

1. Select the **TestData** data source from the drop-down menu.

1. In the **Alert condition** section:
   - Keep **Random Walk** as the _Scenario_.
   - Keep `Last` as the value for the reducer function (`WHEN`), and `IS ABOVE 0` as the threshold value. This is the value above which the alert rule should trigger.

1. Click **Preview alert rule condition** to run the query.

   It should return random time series data. The alert rule state should be `Firing`.

   {{< figure src="/media/docs/alerting/random-walk-firing-alert-rule.png" max-width="1200px" caption="A preview of a firing alert" >}}

### Add folders and labels

1. In **Folder**, click **+ New folder** and enter a name. For example: `metric-alerts` . This folder contains our alert rules.

### Set evaluation behavior

The [alert rule evaluation](https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rules/rule-evaluation/) defines the conditions under which an alert rule triggers, based on the following settings:

- **Evaluation group**: every alert rule is assigned to an evaluation group. You can assign the alert rule to an existing evaluation group or create a new one.
- **Evaluation interval**: determines how frequently the alert rule is checked. For instance, the evaluation may occur every 10s, 30s, 1m, 10m, etc.
- **Pending period**: how long the condition must be met to trigger the alert rule.
- **Keep firing for**: defines how long an alert should remain in the Firing state after the alert condition stops being true. During this time, the alert enters a _Recovering_ state, suppressing additional notifications but keeping the alert active. It helps prevent alert flapping, where alerts rapidly switch between firing and resolved due to noisy or unstable metrics.

To set up the evaluation:

1. In the **Evaluation group and interval**, enter a name. For example: `1m-evaluation`.
1. Choose an **Evaluation interval** (how often the alert are evaluated).
   For example, every `1m` (1 minute).
1. Set the **pending period** to, `0s` (zero seconds), so the alert rule fires the moment the condition is met.
1. Set **Keep firing for** to, `0s`, so the alert stops firing immediately after the condition is no longer true. Use this when you want alerts to be resolved as soon as the system is healthy again.

### Configure notifications

Choose the contact point where you want to receive your alert notifications.

1. Under **Contact point**, select **Webhook** from the drop-down menu.
1. Click **Save rule and exit** at the top right corner.

<!-- INTERACTIVE page step3.md END -->

<!-- INTERACTIVE page step4.md START -->

## Trigger and resolve an alert

Now that the alert rule has been configured, you should receive alert [notifications](http://grafana.com/docs/grafana/next/alerting/fundamentals/alert-rule-evaluation/state-and-health/#notifications) in the contact point whenever alerts trigger and get resolved.

### Trigger an alert

Since the alert rule that you have created has been configured to always fire, once the evaluation interval has concluded, you should receive an alert notification in the Webhook endpoint.

{{< figure src="/media/docs/alerting/alerting-webhook-firing-alert.png" max-width="1200px" caption="Firing alert notification details" >}}

The alert notification details show that the alert rule state is Firing , and it includes the value that made the rule trigger by exceeding the threshold of the alert rule condition. The notification also includes links to see the alert rule details, and another link to add a [Silence](http://grafana.com/docs/grafana/next/alerting/configure-notifications/create-silence/) to it.

### Resolve an alert

To see how a resolved alert notification looks like, you can modify the current alert rule threshold.

To edit the Alert rule:

1. **Navigate to Alerting** > **Alert rules**.
1. Click on the metric-alerts folder to display the alert that you created earlier
1. Click the **edit** button on the right hand side of the screen
1. Increment the Threshold expression to 1.
1. Click **Save rule and exit**.

By incrementing the threshold, the condition is no longer met, and after the evaluation interval has concluded (1 minute approx.), you should receive an alert notification with status **“Resolved”**.

<!-- INTERACTIVE page step4.md END -->

<!-- INTERACTIVE page finish.md START -->

## Learn more in [Grafana Alerting: Multi-dimensional alerts and how to route them](http://www.grafana.com/tutorials/alerting-get-started-pt2/)

<!-- INTERACTIVE ignore START -->

{{< admonition type="tip" >}}

In [Grafana Alerting: Multi-dimensional alerts and how to route them](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you'll learn to use alert instances and route notifications by labels to contacts

{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

In [Grafana Alerting: Multi-dimensional alerts and how to route them](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you'll learn to use alert instances and route notifications by labels to contacts.

{{< /docs/ignore >}}

<!-- INTERACTIVE page finish.md END -->

# Get started with Grafana Alerting - Multi-dimensional alerts and how to route them

<!-- INTERACTIVE page intro.md START -->

This tutorial is a continuation of the [Grafana Alerting - Create and receive your first alert](http://www.grafana.com/tutorials/alerting-get-started/) tutorial.

In this guide, we dig into more complex yet equally fundamental elements of Grafana Alerting: **alert instances** and **notification policies**.

{{< youtube id="nI-_MEnFBQs" >}}

After introducing each component, you will learn how to:

- Configure an alert rule that returns more than one alert instance
- Create notification policies that route firing alert instances to different contact points
- Use labels to match alert instances and notification policies

Learning about alert instances and notification policies is useful if you have more than one contact point in your organization, or if your alert rule returns a number of metrics that you want to handle separately by routing each alert instance to a specific contact point. The tutorial will introduce each concept, followed by how to apply both concepts in a real-world scenario.

<!-- INTERACTIVE page intro.md END -->
<!-- INTERACTIVE page step1.md START -->

<!-- INTERACTIVE ignore START -->

{{< docs/ignore >}}

## Set up the Grafana stack

{{< /docs/ignore >}}

## Before you begin

There are different ways you can follow along with this tutorial.

- **Grafana Cloud**
  - As a Grafana Cloud user, you don't have to install anything. [Create your free account](http://www.grafana.com/auth/sign-up/create-user).

  Continue to [Alert instances](#alert-instances).

- **Interactive learning environment**
  - Alternatively, you can try out this example in our interactive learning environment: [Get started with Grafana Alerting - Alert routing](https://killercoda.com/grafana-labs/course/grafana/alerting-get-started-pt2/). It's a fully configured environment with all the dependencies already installed.

- **Grafana OSS**
  - If you opt to run a Grafana stack locally, ensure you have the following applications installed:

  - [Docker Compose](https://docs.docker.com/get-docker/) (included in Docker for Desktop for macOS and Windows)
  - [Git](https://git-scm.com/)

### Set up the Grafana stack (OSS users)

<!-- INTERACTIVE ignore END -->

To demonstrate the observation of data using the Grafana stack, download and run the following files.

1. Clone the [tutorial environment repository](https://www.github.com/grafana/tutorial-environment).

   <!-- INTERACTIVE exec START -->

   ```
   git clone https://github.com/grafana/tutorial-environment.git
   ```

   <!-- INTERACTIVE exec END -->

1. Change to the directory where you cloned the repository:

   <!-- INTERACTIVE exec START -->

   ```
   cd tutorial-environment
   ```

   <!-- INTERACTIVE exec END -->

1. Run the Grafana stack:

   <!-- INTERACTIVE ignore START -->

   ```
   docker compose up -d
   ```

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   <!-- INTERACTIVE exec START -->

   ```bash
   docker-compose up -d
   ```

   <!-- INTERACTIVE exec END -->

   {{< /docs/ignore >}}

   The first time you run `docker compose up -d`, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

   <!-- INTERACTIVE ignore START -->

   {{< admonition type="note" >}}
   If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.
   {{< /admonition >}}

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   NOTE:

   If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

   {{< /docs/ignore >}}

<!-- INTERACTIVE page step1.md END -->
<!-- INTERACTIVE page step2.md START -->

## Alert instances

An [alert instance](https://grafana.com/docs/grafana/latest/alerting/fundamentals/#alert-instances) is an event that matches a metric returned by an alert rule query.

Let's consider a scenario where you're monitoring website traffic using Grafana. You've set up an alert rule to trigger an alert instance if the number of page views exceeds a certain threshold (more than `1000` page views) within a specific time period, say, over the past `5` minutes.

If the query returns more than one time-series, each time-series represents a different metric or aspect being monitored. In this case, the alert rule is applied individually to each time-series.

{{< figure alt="Screenshot displaying alert instances in the context of an alert rule, highlighting the specific alerts triggered by the rule and their respective statuses" src="/media/docs/alerting/alert-instance-flow.jpg" max-width="1200px" caption="Alert Instances in the Context of an Alert Rule" >}}

In this scenario, each time-series is evaluated independently against the alert rule. It results in the creation of an alert instance for each time-series. The time-series corresponding to the desktop page views meets the threshold and, therefore, results in an alert instance in **Firing** state for which an alert notification is sent. The mobile alert instance state remains **Normal**.

<!-- INTERACTIVE page step2.md END -->
<!-- INTERACTIVE page step3.md START -->

## Notification policies

[Notification policies](https://grafana.com/docs/grafana/latest/alerting/fundamentals/notifications/notification-policies/) route alerts to different communication channels, reducing alert noise and providing control over when and how alerts are sent. For example, you might use notification policies to ensure that critical alerts about server downtime are sent immediately to the on-call engineer. Another use case could be routing performance alerts to the development team for review and action.

Key Characteristics:

- Route alert notifications by matching alerts and policies with labels
- Manage when to send notifications

{{< figure alt="Screenshot illustrating the routing of alerts with notification policies, including the configuration and flow of alerts through different notification channels" src="/media/docs/alerting/get-started-notification-policy-tree-combo.png" max-width="1200px" caption="Routing of alerts with notification policies" >}}

In the above diagram, alert instances and notification policies are matched by labels. For instance, the label `team=operations` matches the alert instance “**Pod stuck in CrashLoop**” and “**Disk Usage -80%**” to child policies that send alert notifications to a particular contact point (operations@grafana.com).

<!-- INTERACTIVE page step3.md END -->
<!-- INTERACTIVE page step4.md START -->

## Create notification policies

Create a notification policy if you want to handle metrics returned by alert rules separately by routing each alert instance to a specific contact point.

<!-- INTERACTIVE ignore START -->

1. In your browser, **sign in** to your Grafana Cloud account.

   OSS and interactive learning environment users: To log in, navigate to [http://localhost:3000](http://localhost:3000), where Grafana should be running.

1. Navigate to **Alerts & IRM > Alerting > Notification policies**.
1. In the Default policy, click **+ New child policy**.
1. In the field **Label** enter `device`, and in the field **Value** enter `desktop`.
1. From the **Contact point** drop-down, choose **Webhook**.
   {{< admonition type="note" >}}
   If you don’t have any contact points, add a [Contact point](https://grafana.com/tutorials/alerting-get-started/#create-a-contact-point).
   {{</ admonition >}}
1. Enable Continue matching:

   Turn on **Continue matching subsequent sibling nodes** so the evaluation continues even after one or more labels (i.e. `device` label) match.

1. Click **Save Policy**.

   This new child policy routes alerts that match the label `device=desktop` to the Webhook contact point.

1. **Repeat the steps above to create a second child policy** to match another alert instance. For labels use: `device=mobile`. Use the Webhook integration for the contact point. Alternatively, experiment by using a different Webhook endpoint or a [different integration](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/#supported-contact-point-integrations).

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

1. Visit [http://localhost:3000](http://localhost:3000), where Grafana should be running
1. Navigate to **Alerts & IRM > Alerting > Notification policies**.
1. In the Default policy, click **+ New child policy**.
1. In the field **Label** enter `device`, and in the field **Value** enter `desktop`.
1. From the **Contact point** drop-down, choose **Webhook**.

   If you don’t have any contact points, add a [Contact point](https://grafana.com/tutorials/alerting-get-started/#create-a-contact-point).

1. Click **Save Policy**.

   This new child policy routes alerts that match the label `device=desktop` to the Webhook contact point.

1. **Repeat the steps above to create a second child policy** to match another alert instance. For labels use: `device=mobile`. Use the Webhook integration for the contact point. Alternatively, experiment by using a different Webhook endpoint or a [different integration](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/#supported-contact-point-integrations).

{{< /docs/ignore >}}

<!-- INTERACTIVE page step4.md END -->
<!-- INTERACTIVE page step5.md START -->

## Create an alert rule that returns alert instances

The alert rule that you are about to create is meant to monitor web traffic page views. The objective is to explore what an alert instance is and how to leverage routing individual alert instances by using label matchers and notification policies.

   <!-- INTERACTIVE page step5.md END -->
   <!-- INTERACTIVE page step6.md START -->

### Create an alert rule

1. Navigate to **Alerts & IRM > Alerting > Alert rules**.
1. Click **+ New alert rule**.

### Enter an alert rule name

Make it short and descriptive as this will appear in your alert notification. For instance, `web-traffic`.

### Define query and alert condition

In this section, we use the default options for Grafana-managed alert rule creation. The default options let us define the query, a expression (used to manipulate the data -- the `WHEN` field in the UI), and the condition that must be met for the alert to be triggered (in default mode is the threshold).

Grafana includes a [test data source](https://grafana.com/docs/grafana/latest/datasources/testdata/) that creates simulated time series data. This data source is included in the demo environment for this tutorial. If you're working in Grafana Cloud or your own local Grafana instance, you can add the data source through the **Connections** menu.

1. Select **TestData** data source from the drop-down menu.
1. From **Scenario** select **CSV Content**.
1. Copy in the following CSV data:

   ```
   device,views
   desktop,1200
   mobile,900
   ```

   The above CSV data simulates a data source returning multiple time series, each leading to the creation of an alert instance for that specific time series. Note that the data returned matches the example in the [Alert instance](#alert-instances) section.

1. In the **Alert condition** section:
   - Keep `Last` as the value for the reducer function (`WHEN`), and `IS ABOVE 1000` as the threshold value. This is the value above which the alert rule should trigger.

1. Click **Preview alert rule condition** to run the queries.

It should return two series.`desktop` in Firing state, and `mobile` in Normal state. The values `1`, and `0` mean that the condition is either `true` or `false`.

{{< figure alt="Screenshot showing a preview of a query in Grafana that returns two alert instances, including the query results and relevant alert details" src="/media/docs/alerting/firing-instances.png" max-width="1200px" caption="Preview of a query returning two alert instances in Grafana." >}}

<!-- INTERACTIVE page step6.md END -->
<!-- INTERACTIVE page step7.md START -->

### Add folders and labels

1. In **Folder**, click **+ New folder** and enter a name. For example: `web-traffic-alerts` . This folder contains our alert rules.

### Set evaluation behavior

In the [life cycle](http://grafana.com/docs/grafana/next/alerting/fundamentals/alert-rule-evaluation/) of alert instances, when an alert condition (threshold) is not met, the alert instance state is **Normal**. Similarly, when the condition is breached (for longer than the pending period, which in this tutorial will be 0), the alert instance state switches back to **Alerting**, which means that the alert rule state is **Firing**, and a notification is sent.

To set up evaluation behavior:

1. In the **Evaluation group and interval**, enter a name. For example:  `1m-evaluation`.
1. Choose an **Evaluation interval** (how often the alert will be evaluated). Choose `1m`.
1. Set the **pending period** to `0s` (zero seconds), so the alert rule fires the moment the condition is met.
1. Set **Keep firing for** to, `0s`, so the alert stops firing immediately after the condition is no longer true.

### Configure notifications

In this section, you can select how you want to route your alert instances. Since we want to route by notification policy, we need to ensure that the labels match the alert instance.

1. Toggle the **Advanced options** button to display matching Notification policies.
1. Click **Preview routing**. Based on the existing labels, you should see a preview of what policies are matching with the alerts. There should be two alert instances matching the labels that were previously setup in each notification policy: `device=desktop`, `device=mobile`.

   These [types of labels](https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rules/annotation-label/#label-types) are generated by the data source query and they can be leveraged to match our notification policies without needing to manually add them to the alert rule.

   {{< figure alt="Screenshot showing a routing preview of matched notification policies, detailing how alerts are matched and routed to specific notification channels" src="/media/docs/alerting/get-started-alert-instace-routing-prev.png" max-width="1200px" caption="Routing preview of matched notification policies" >}}

   <!-- INTERACTIVE ignore START -->

   {{< admonition type="note" >}}
   Even if both labels match the policies, only the alert instance in Firing state produces an alert notification.
   {{</ admonition >}}
   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}
   Even if both labels match the policies, only the alert instance in Firing state produces an alert notification.
   {{< /docs/ignore >}}

1. Click **Save rule and exit**.

Now that we have set up the alert rule, it’s time to check the alert notification.

<!-- INTERACTIVE page step7.md END -->
<!-- INTERACTIVE page step8.md START -->

## Receive alert notifications

Now that the alert rule has been configured, you should receive alert [notifications](http://grafana.com/docs/grafana/next/alerting/fundamentals/alert-rule-evaluation/state-and-health/#notifications) in the contact point whenever the alert triggers and gets resolved. In our example, each alert instance should be routed separately as we configured labels to match notification policies. Once the evaluation interval has concluded (1m), you should receive an alert notification in the Webhook endpoint.

{{< figure alt="Screenshot showing the exploration of alert notification details in a webhook endpoint, displaying the content and structure of the alert payload received by the endpoint" src="/media/docs/alerting/get-started-webhook-alert-isntance.png" max-width="1200px" caption="Exploring alert notification details in webhook endpoint" >}}

The alert notification details show that the alert instance corresponding to the website views from desktop devices was correctly routed through the notification policy to the Webhook contact point. The notification also shows that the instance is in **Firing** state, as well as it includes the label `device=desktop`, which makes the routing of the alert instance possible.

Feel free to change the CSV data in the alert rule to trigger the routing of the alert instance that matches the label `device=mobile`.

<!-- INTERACTIVE page step8.md END -->
<!-- INTERACTIVE page finish.md START -->

## Summary

In this tutorial, you have learned how Grafana Alerting can route individual alert instances using the labels generated by the data-source query and match these labels with notification policies, which in turn routes alert notifications to specific contact points.

If you run into any problems, you are welcome to post questions in our [Grafana Community forum](https://community.grafana.com/).

## Learn more in [Grafana Alerting: Group alert notifications](http://www.grafana.com/tutorials/alerting-get-started-pt3/)

<!-- INTERACTIVE ignore START -->

{{< admonition type="tip" >}}

In [Get started with Grafana Alerting: Group alert notifications](http://www.grafana.com/tutorials/alerting-get-started-pt3/) you learn how to group alert notifications effectively.

{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

In [Get started with Grafana Alerting: Group alert notifications](http://www.grafana.com/tutorials/alerting-get-started-pt3/) you learn how to group alert notifications effectively.

{{< /docs/ignore >}}

<!-- INTERACTIVE page finish.md END -->

# Get started with Grafana Alerting - Group alert notifications

<!-- INTERACTIVE page intro.md START -->

This tutorial is a continuation of the [Get started with Grafana Alerting - Alert routing](http://www.grafana.com/tutorials/alerting-get-started-pt2/) tutorial.

{{< youtube id="WZ8gqKIQ5Oc" >}}

Grouping in Grafana Alerting reduces notification noise by combining related alert instances into a single, concise notification. This is useful for on-call engineers, ensuring they focus on resolving incidents instead of sorting through a flood of notifications.

Grouping is configured using labels in the notification policy. These labels reference those generated by alert instances or configured by the user.

Notification policies also allow you to define how often notifications are sent for each group of alert instances.

In this tutorial, you will:

- Learn how alert rule grouping works.
- Create a notification policy to handle grouping.
- Define alert rules for a real-world scenario.
- Receive and review grouped alert notifications.

<!-- INTERACTIVE page intro.md END -->
<!-- INTERACTIVE page step1.md START -->

<!-- INTERACTIVE ignore START -->

{{< docs/ignore >}}

## Set up the Grafana stack

{{< /docs/ignore >}}

## Before you begin

There are different ways you can follow along with this tutorial.

- **Grafana Cloud**
  - As a Grafana Cloud user, you don't have to install anything. [Create your free account](http://www.grafana.com/auth/sign-up/create-user).

  Continue to [How alert rule grouping works](#how-alert-rule-grouping-works).

- **Interactive learning environment**
  - Alternatively, you can try out this example in our interactive learning environment: [Get started with Grafana Alerting - Grouping](https://killercoda.com/grafana-labs/course/grafana/alerting-get-started-pt3/). It's a fully configured environment with all the dependencies already installed.

- **Grafana OSS**
  - If you opt to run a Grafana stack locally, ensure you have the following applications installed:

  - [Docker Compose](https://docs.docker.com/get-docker/) (included in Docker for Desktop for macOS and Windows)
  - [Git](https://git-scm.com/)

### Set up the Grafana stack (OSS users)

<!-- INTERACTIVE ignore END -->

To demonstrate the observation of data using the Grafana stack, download and run the following files.

1. Clone the [tutorial environment repository](https://www.github.com/grafana/tutorial-environment).

   <!-- INTERACTIVE exec START -->

   ```
   git clone https://github.com/grafana/tutorial-environment.git
   ```

   <!-- INTERACTIVE exec END -->

1. Change to the directory where you cloned the repository:

   <!-- INTERACTIVE exec START -->

   ```
   cd tutorial-environment
   ```

   <!-- INTERACTIVE exec END -->

1. Run the Grafana stack:

   <!-- INTERACTIVE ignore START -->

   ```
   docker compose up -d
   ```

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   <!-- INTERACTIVE exec START -->

   ```bash
   docker-compose up -d
   ```

   <!-- INTERACTIVE exec END -->

   {{< /docs/ignore >}}

   The first time you run `docker compose up -d`, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

   <!-- INTERACTIVE ignore START -->

   {{< admonition type="note" >}}
   If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.
   {{< /admonition >}}

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   NOTE:

   If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

   {{< /docs/ignore >}}

<!-- INTERACTIVE page step1.md END -->
<!-- INTERACTIVE page step2.md START -->

## How alert rule grouping works

Alert notification grouping is configured with **labels** and **timing options**:

- **Labels** map the alert rule with the notification policy and define the grouping.
- **Timing options** control when and how often notifications are sent.

{{< figure src="/media/docs/alerting/alerting-notification-policy-diagram-with-labels-v3.png" max-width="750px" alt="A diagram about the components of a notification policy, including labels and groups" >}}

### Types of Labels

**Reserved labels** (default):

- Automatically generated by Grafana, e.g., `alertname`, `grafana_folder`.
- Example: `alertname="High CPU usage"`.

**User-configured labels**:

- Added manually to the alert rule.
- Example: `severity`, `priority`.

**Query labels**:

- Returned by the data source query.
- Example: `region`, `service`, `environment`.

### Timing Options

**Group wait**: Time before sending the first notification.
**Group interval**: Time between notifications for a group.
**Repeat interval**: Time before resending notifications for an unchanged group.

Alerts sharing the **same label values** are grouped together, and timing options determine notification frequency.

For more details, see:

- [Grouping Alerts](https://grafana.com/docs/grafana/latest/alerting/fundamentals/notifications/group-alert-notifications/)
- [Alert Labels](https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rules/annotation-label/#label-types)

<!-- INTERACTIVE page step2.md END -->
<!-- INTERACTIVE page step3.md START -->

## A real-world example of alert grouping in action

### Scenario: monitoring a distributed application

You’re monitoring metrics like CPU usage, memory utilization, and network latency across multiple regions. Some of these alert rules include labels such as `region: us-west` and `region: us-east`. If multiple alert rules trigger across these regions, they can result in notification floods.

### How to manage grouping

To group alert rule notifications:

1. **Define labels**: Use `region`, `metric`, or `instance` labels to categorize alerts.
1. **Configure Notification policies**:
   - Group alerts by the **query label** "region".
   - Example:
     - Alert notifications for `region: us-west` go to the West Coast team.
     - Alert notifications for `region: us-east` go to the East Coast team.
1. Specify the **timing options** for sending notifications to control their frequency.
   - Example:
     - **Group interval**: setting determines how often updates for the same alert group are sent. By default, this interval is set to 5 minutes, but you can customize it to be shorter or longer based on your needs.

<!-- INTERACTIVE page step3.md END -->
<!-- INTERACTIVE page step4.md START -->

## Setting up alert rule grouping

### Notification Policy

Following the above example, [notification policies](ref:notification-policies) are created to route alert instances, which have a region label, to a specific contact point. The goal is to receive one consolidated notification per region. To demonstrate how grouping works, alert notifications for the East Coast team are not grouped. Regarding timing, a specific schedule is defined for that region. This setup overrides the parent's settings to fine-tune the behavior for specific labels (i.e., regions).

<!-- INTERACTIVE ignore START -->

1. Sign in to Grafana:
   - **Grafana Cloud** users: Log in via Grafana Cloud.
   - **OSS users**: Go to [http://localhost:3000](http://localhost:3000).

1. Navigate to **Notification Policies**:
   - Go to **Alerts & IRM > Alerting > Notification Policies**.

1. Add a child policy:
   - In the Default policy, click **+ New child policy**.
   - **Label**: `region`
   - **Operator**: `=`
   - **Value**: `us-west`

     This label matches alert rules where the region label is us-west.

1. Choose a **Contact point**:
   - Select **Webhook**.

   If you don’t have any contact points, add a [Contact point](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/#add-a-contact-point).

1. Enable Continue matching:
   - Turn on **Continue matching subsequent sibling nodes** so the evaluation continues even after one or more labels (i.e. region label) match.

1. Override grouping settings:
   - Toggle **Override grouping**.
   - **Group by**: Add `region` as label. Remove any existing labels.

     **Group by** consolidates alerts that share the same grouping label into a single notification. For example, all alerts with `region=us-west` will be combined into one notification, making it easier to manage and reducing alert fatigue.

1. Set custom timing:
   - Toggle **Override general timings**.
   - **Group interval**: `2m`. This ensures follow-up notifications for the same alert group will be sent at intervals of 2 minutes. While the default is 5 minutes, we chose 2 minutes here to provide faster feedback for demonstration purposes.

     **Timing options** control how often notifications are sent and can help balance timely alerting with minimizing noise.

1. Save and repeat:
   - Repeat the steps above for `region = us-east` but without overriding grouping and timing options. Use a different webhook endpoint as the contact point.

   {{< figure src="/media/docs/alerting/notificaiton-policies-region.png" max-width="750px" alt="Two nested notification policies to route and group alert notifications" >}}

   These nested policies should route alert instances where the region label is either us-west or us-east. Only the us-west region team should receive grouped alert notifications.

   {{< admonition type="note" >}}
   Label matchers are combined using the `AND` logical operator. This means that all matchers must be satisfied for a rule to be linked to a policy. If you attempt to use the same label key (e.g., region) with different values (e.g., us-west and us-east), the condition will not match, because it is logically impossible for a single key to have multiple values simultaneously.

   However, `region!=us-east && region=!us-west` can match. For example, it would match a label set where `region=eu-central`.
   Alternatively, for identical label keys use regular expression matchers (e.g., `region=~us-west|us-east`).
   {{</ admonition >}}

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

1. Visit [http://localhost:3000](http://localhost:3000), where Grafana should be running
1. Navigate to **Alerts & IRM > Alerting > Notification policies**.
1. In the Default policy, click **+ New child policy**.
   - In the Default policy, click **+ New child policy**.
   - **Label**: `region`
   - **Operator**: `=`
   - **Value**: `us-west`

   This label matches alert rules where the region label is us-west

1. Choose a **Contact point**:
   - Select **Webhook**.

   If you don’t have any contact points, add a Contact point.

1. Enable Continue matching:
   - Turn on **Continue matching subsequent sibling nodes** so the evaluation continues even after one or more labels (i.e. region label) match.

1. Override grouping settings:
   - Toggle **Override grouping**.
   - **Group by**: `region`.

     **Group by** consolidates alerts that share the same grouping label into a single notification. For example, all alerts with `region=us-west` will be combined into one notification, making it easier to manage and reducing alert fatigue.

1. Set custom timing:
   - Toggle **Override general timings**.
   - **Group interval**: `2m`. This ensures follow-up notifications for the same alert group will be sent at intervals of 2 minutes. While the default is 5 minutes, we chose 2 minutes here to provide faster feedback for demonstration purposes.

     **Timing options** control how often notifications are sent and can help balance timely alerting with minimizing noise.

1. Save and repeat:
   - Repeat for `region = us-east` with a different webhook or a different contact point.

     **Note**: Label matchers are combined using the `AND` logical operator. This means that all matchers must be satisfied for a rule to be linked to a policy. If you attempt to use the same label key (e.g., region) with different values (e.g., us-west and us-east), the condition will not match, because it is logically impossible for a single key to have multiple values simultaneously.

     However, `region!=us-east && region=!us-west` can match. For example, it would match a label set where `region=eu-central`.Alternatively, for identical label keys use regular expression matchers (e.g., `region=~us-west|us-east`).

{{< /docs/ignore >}}

<!-- INTERACTIVE page step4.md END -->
<!-- INTERACTIVE page step5.md START -->

## Create an alert rule

In this section we configure an alert rule based on our application monitoring example.

1. Navigate to **Alerts & IRM > Alerting > Alert rules**.
2. Click **+ New alert rule**.

### Enter an alert rule name

Make it short and descriptive as this appears in your alert notification. For instance, `High CPU usage - Multi-region`.

### Define query and alert condition

In this section, we use the default options for Grafana-managed alert rule creation. The default options let us define the query, a expression (used to manipulate the data -- the `WHEN` field in the UI), and the condition that must be met for the alert to be triggered (in default mode is the threshold).

Grafana includes a [test data source](https://grafana.com/docs/grafana/latest/datasources/testdata/) that creates simulated time series data. This data source is included in the demo environment for this tutorial. If you're working in Grafana Cloud or your own local Grafana instance, you can add the data source through the **Connections** menu.

1. From the drop-down menu, select **TestData** data source.
1. From **Scenario** select **CSV Content**.
1. Copy in the following CSV data:
   - Select **TestData** as the data source.
   - Set **Scenario** to **CSV Content**.
   - Use the following CSV data:

     ```csv
     region,cpu-usage,service,instance
     us-west,35,web-server-1,server-01
     us-west,81,web-server-1,server-02
     us-east,79,web-server-2,server-03
     us-east,52,web-server-2,server-04
     us-west,45,db-server-1,server-05
     us-east,77,db-server-2,server-06
     us-west,82,db-server-1,server-07
     us-east,93,db-server-2,server-08
     ```

   The returned data simulates a data source returning multiple time series, each leading to the creation of an alert instance for that specific time series.

1. In the **Alert condition** section:
   - Keep `Last` as the value for the reducer function (`WHEN`), and `IS ABOVE 75` as the threshold value. This is the value above which the alert rule should trigger.

1. Click **Preview alert rule condition** to run the queries.

   It should return 5 series in Firing state, two firing instances from the us-west region, and three from the us-east region.

   {{< figure src="/media/docs/alerting/regions-alert-instance-preview.png" max-width="750px" alt="Preview of a query returning alert instances." >}}

### Add folders and labels

1. In **Folder**, click **+ New folder** and enter a name. For example: `Multi-region alerts` . This folder contains our alert rules.

### Set evaluation behavior

Every alert rule is assigned to an evaluation group. You can assign the alert rule to an existing evaluation group or create a new one.

1. In the **Evaluation group and interval**, enter a name. For example: `Multi-region group`.
1. Choose an **Evaluation interval** (how often the alert are evaluated). Choose `1m`.
1. Set the **pending period** to `0s` (zero seconds), so the alert rule fires the moment the condition is met (this minimizes the waiting time for the demonstration).
1. Set **Keep firing for** to, `0s`, so the alert stops firing immediately after the condition is no longer true.

### Configure notifications

Select who should receive a notification when an alert rule fires.

1. Select **Use notification policy**.
1. Click **Preview routing** to ensure correct matching.

   {{< figure src="/media/docs/alerting/region-notification-policy-routing-preview.png" max-width="750px" alt="Preview of alert instance routing with the region label matcher" >}}

   The preview should show that the region label from our data source is successfully matching the notification policies that we created earlier thanks to the label matcher that we configured.

1. Click **Save rule and exit**.

### Create a second alert rule

Repeat the steps above to create a second alert rule that alerts on high memory usage.

1. Duplicate the alert rule by clicking on **More > Duplicate**.
1. Name it `High Memory usage - Multi-region`.
1. Use the below CSV data to simulate a data source returning memory usage.

   ```
   region,memory-usage,service,instance
   us-west,42,cache-server-1,server-09
   us-west,88,cache-server-1,server-10
   us-east,74,api-server-1,server-11
   us-east,90,api-server-1,server-12
   us-west,53,analytics-server-1,server-13
   us-east,81,analytics-server-2,server-14
   us-west,77,analytics-server-1,server-15
   us-east,94,analytics-server-2,server-16
   ```

1. Click Save rule and exit.

<!-- INTERACTIVE page step5.md END -->
<!-- INTERACTIVE page step6.md START -->

## Receiving grouped alert notifications

Now that the alert rules have been configured, you should receive alert notifications in the contact point(s) whenever alerts trigger.

When the configured alert rule detects CPU or memory usage higher than 75% across multiple regions, it will evaluate the metric every minute. If the condition persists, notifications will be grouped together, with a Group wait of 30 seconds before the first alert is sent. Follow-up notifications for the same alert group will be sent at intervals of 2 minutes (US-west alert instances only), increasing the frequency of the grouped alert notifications. US-east instances follow-up notifications should be sent at the default interval of 5 minutes. If the condition continues for an extended period, a Repeat interval of 4 hours ensures that the alert is only resent if the issue persists.

As a result, our notification policies should route three notifications: one grouped notification grouping both CPU and memory alert instances from the us-west region and two separate notifications with alert instances from the us-east region.

Grouped notifications example:

```json
{
  "receiver": "US-West-Alerts",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High CPU usage - Multi-region",
        "grafana_folder": "Multi-region alerts",
        "instance": "server-05",
...
  {
    "status": "firing",
      "labels": {
        "alertname": "High Memory usage - Multi-region",
        "grafana_folder": "Multi-region alerts",
        "instance": "server-10",
      },

...}
```

_Detail of CPU and memory alert instances grouped into a single notification for us-west contact point._

```json
{
  "receiver": "US-East-Alerts",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High CPU usage - Multi-region",
        "grafana_folder": "Multi-region alerts",
        "instance": "server-03",
        "region": "us-east",
        "service": "web-server-2"
...}}}
```

_Detail of CPU alert instances grouped into a separate notification for us-east contact point._

```json
{
  "receiver": "US-East-Alerts",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High memory usage - Multi-region",
        "grafana_folder": "Multi-region memory alerts",
        "instance": "server-12",
        "region": "us-east"
...}}}
```

_Detail of memory alert instances grouped into a separate notification for us-east contact point._

<!-- INTERACTIVE page step6.md END -->

<!-- INTERACTIVE page finish.md START -->

## Conclusion

By configuring **notification policies** and using **labels** (such as _region_), you can group alert notifications based on specific criteria and route them to the appropriate teams. Fine-tuning **timing options**—including group wait, group interval, and repeat interval—further can reduce noise and ensures notifications remain actionable without overwhelming on-call engineers.

## Learn more in [Grafana Alerting: Template your alert notifications](http://www.grafana.com/tutorials/alerting-get-started-pt4/)

<!-- INTERACTIVE ignore START -->

{{< admonition type="tip" >}}

In [Get started with Grafana Alerting: Template your alert notifications](http://www.grafana.com/tutorials/alerting-get-started-pt4/) you learn how to use templates to create customized and concise notifications.

{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

In [Get started with Grafana Alerting: Template your alert notifications](http://www.grafana.com/tutorials/alerting-get-started-pt4/) you learn how to use templates to create customized and concise notifications.

{{< /docs/ignore >}}

<!-- INTERACTIVE page finish.md END -->

<!-- INTERACTIVE page finish.md END -->

# Get started with Grafana Alerting - Template your alert notifications

<!-- INTERACTIVE page intro.md START -->

This tutorial is a continuation of the [Get started with Grafana Alerting - Grouping notifications](http://www.grafana.com/tutorials/alerting-get-started-pt3/) tutorial.

{{< youtube id="9CSrQGKeZwQ" >}}

In this tutorial, you will learn:

- The two types of templates in Grafana Alerting: labels and annotations and notification templates.
- How to configure alert rules with summary and description annotations.
- How to create a notification template that integrates with alert rule annotations.
- How to use a built-in notification template to group and format multiple alert instances.
- How to preview alert notifications by leveraging alert instances in the notification template payload.

<!-- INTERACTIVE page intro.md END -->
<!-- INTERACTIVE page step1.md START -->

<!-- INTERACTIVE ignore START -->

{{< docs/ignore >}}

## Set up the Grafana stack

{{< /docs/ignore >}}

## Before you begin

There are different ways you can follow along with this tutorial.

> Note: Some of the templating features in Grafana Alerting discussed in this tutorial are currently available in Grafana Cloud but have not yet been released to the Open Source (OSS) version.

- **Grafana Cloud**
  - As a Grafana Cloud user, you don't have to install anything. [Create your free account](http://www.grafana.com/auth/sign-up/create-user).

  Continue to [how templating works](#how-templating-works).

- **Interactive learning environment**
  - Alternatively, you can try out this example in our interactive learning environment: [Get started with Grafana Alerting - Templating](https://killercoda.com/grafana-labs/course/grafana/alerting-get-started-pt4/). It's a fully configured environment with all the dependencies already installed.

- **Grafana OSS**
  - If you opt to run a Grafana stack locally, ensure you have the following applications installed:

  - [Docker Compose](https://docs.docker.com/get-docker/) (included in Docker for Desktop for macOS and Windows)
  - [Git](https://git-scm.com/)

### Set up the Grafana stack (OSS users)

<!-- INTERACTIVE ignore END -->

To demonstrate the observation of data using the Grafana stack, download and run the following files.

1. Clone the [tutorial environment repository](https://www.github.com/grafana/tutorial-environment).

   <!-- INTERACTIVE exec START -->

   ```
   git clone https://github.com/grafana/tutorial-environment.git
   ```

   <!-- INTERACTIVE exec END -->

1. Change to the directory where you cloned the repository:

   <!-- INTERACTIVE exec START -->

   ```
   cd tutorial-environment
   ```

   <!-- INTERACTIVE exec END -->

1. Run the Grafana stack:

   <!-- INTERACTIVE ignore START -->

   ```
   docker compose up -d
   ```

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   <!-- INTERACTIVE exec START -->

   ```bash
   docker-compose up -d
   ```

   <!-- INTERACTIVE exec END -->

   {{< /docs/ignore >}}

   The first time you run `docker compose up -d`, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

   <!-- INTERACTIVE ignore START -->

   {{< admonition type="note" >}}
   If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.
   {{< /admonition >}}

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   NOTE:

   If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

   {{< /docs/ignore >}}

<!-- INTERACTIVE page step1.md END -->
<!-- INTERACTIVE page step2.md START -->

## How templating works

In Grafana, you can use [templates](https://grafana.com/docs/grafana/latest/alerting/fundamentals/templates/) to dynamically pull in specific data about the alert rule. This results in more flexible and informative alert notification messages. You can template either alert rule labels and annotations, or notification templates. Both use the Go template language.

{{< figure src="/media/docs/alerting/how-notification-templates-works.png" max-width="1200px" caption="How templating works" >}}

### Templating alert rule labels and annotations

[Labels and annotations](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/) are key fields where templates are applied. One of the main advantages of using templating in annotations is the ability to incorporate dynamic data from queries, allowing alerts to reflect real-time information relevant to the triggered condition. By using templating in annotations, you can customize the content of each alert instance, such as including instance names and metric values, so the notification becomes more informative.

### Notification templates

The real power of templating lies in how it helps you format notifications with dynamic alert data. [Notification templates](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/) let you pull in details from annotations to create clear and consistent messages. They also make it simple to reuse the same format across different contact points, saving time and effort.

Notification templates allow you to customize how information is presented in each notification. For example, you can use templates to organize and format details about firing or resolved alerts, making it easier for recipients to understand the status of each alert at a glance—all within a single notification.

This particular notification template pulls in summary and description annotations for each alert instance and organizes them into separate sections, such as "firing" and "resolved." This way, instead of getting a long list of individual alert notifications, users can receive one well-structured message with all the relevant details grouped together.

This approach is helpful when you want to reduce notification noise, especially in situations where multiple instances of an alert are firing at the same time (e.g., high CPU usage across several instances). You can leverage templates to create a unified, easy-to-read notification that includes all the pertinent details.

<!-- INTERACTIVE page step2.md END -->
<!-- INTERACTIVE page step3.md START -->

## Step 1: Template labels and annotations

Now that we've introduced how templating works, let’s move on to the next step. We guide you through creating an alert rule with a summary and description annotation. In doing so, we incorporate CPU usage and instance names, which we later use in our notification template.

### Create an alert rule

1. Sign in to Grafana:
   - **Grafana Cloud** users: Log in via Grafana Cloud.
   - **OSS users**: Go to [http://localhost:3000](http://localhost:3000).

1. Create an alert rule that includes a summary and description annotation:
   - Navigate to **Alerts & IRM > Alerting > Alert rules**.
   - Click **+ New alert rule**.
   - Enter an **alert rule name**. Name it `High CPU usage`
1. **Define query an alert condition** section:
   - Select TestData data source from the drop-down menu.

     [TestData](https://grafana.com/docs/grafana/latest/datasources/testdata/) is included in the demo environment. If you’re working in Grafana Cloud or your own local Grafana instance, you can add the data source through the Connections menu.

   - From **Scenario** select **CSV Content**.
   - Copy in the following CSV data:

     ```
     region,cpu-usage,service,instance
     us-west,88,web-server-1,server-01
     us-west,81,web-server-1,server-02
     us-east,79,web-server-2,server-03
     us-east,52,web-server-2,server-04
     ```

     This dataset simulates a data source returning multiple time series, with each time series generating a separate alert instance.

1. **Alert condition** section:
   - Keep Last as the value for the reducer function (`WHEN`), and `IS ABOVE 75` as the threshold value, representing CPU usage above 75% .This is the value above which the alert rule should trigger.
   - Click **Preview alert rule condition** to run the queries.

   It should return 3 series in Firing state, and 1 in Normal state.

   {{< figure src="/media/docs/alerting/part-4-firing-instances-preview.png" max-width="1200px" caption="Preview of a query returning alert instances" >}}

1. Add folders and labels section:
   - In **Folder**, click **+ New folder** and enter a name. For example: `System metrics` . This folder contains our alert rules.

     Note: while it's possible to template labels here, in this tutorial, we focus on templating the summary and annotations fields instead.

1. **Set evaluation behaviour** section:
   - In the **Evaluation group and interval**, enter a name. For example: `High usage`.
   - Choose an **Evaluation interval** (how often the alert will be evaluated). Choose `1m`.
   - Set the **pending period** to 0s (zero seconds), so the alert rule fires the moment the condition is met (this minimizes the waiting time for the demonstration.).
   - Set **Keep firing for** to, `0s`, so the alert stops firing immediately after the condition is no longer true.
1. **Configure notifications** section:

   Select who should receive a notification when an alert rule fires.
   - Select a **Contact point**. If you don’t have any contact points, click _View or create contact points_.

1. **Configure notification message** section:

   In this step, you’ll configure the **summary** and **description** annotations to make your alert notifications informative and easy to understand. These annotations use templates to dynamically include key information about the alert.
   - **Summary** annotation: Enter the following code as the value for the annotation.:

     ```go
     {{- "\n" -}}
     Instance: {{ index $labels "instance" }}
     {{- "\t" -}} Usage: {{ index $values "A"}}%{{- "\n" -}}
     ```

     This template automatically adds the instance name (from the [$labels](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/reference/#labels) data) and its current CPU usage (from [$values["A"]](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/reference/#values)) into the alert summary. `\t`: Adds a tab space between the instance name and the value. And, `\n`: Inserts a new line after the value.

     Output example:

     ```
     server-01	88
     ```

     This output helps you quickly see which instance is affected and its usage level.

1. Optional: Add a description to help the on-call engineer to better understand what the alert rule is about. Eg. This alert monitors CPU usage across instances and triggers if any instance exceeds a usage threshold of 75%.
1. Click **Save rule and exit**.

Now that we’ve configured an alert rule with dynamic templates for the **summary** annotation, the next step is to customize the alert notifications themselves. While the default notification message includes the summary annotation and works well, it can often be too verbose.

{{< figure src="/media/docs/alerting/templated-annotation-alert.png" max-width="1200px" caption="Default email alert notification with templated annotation" >}}

To make our alert notifications more concise and tailored to our needs, we’ll create a custom **notification template** that references the summary annotation we just set up. Notification templates are especially useful because they can be reused across multiple contact points, ensuring consistent alert messages.

<!-- INTERACTIVE page step3.md END -->
<!-- INTERACTIVE page step4.md START -->

## Step 2: Template notifications

In this step, we use a built-in notification template to format alert notifications in a clear and organized way. Notification templates allow us to customize the structure of alert messages, making them easier to read and more relevant.

Without a notification template, the alert messages would include the default Grafana formatting (`default.message`, see image above).

### Adding a notification template:

1. Navigate to **Alerts & IRM** > **Alerting** > **Contact point**s.
1. Select the **Notification Templates** tab.
1. Click **+ New notification template**.
1. Enter a name. E.g `instance-cpu-summary`.
1. From the **Add example** dropdown menu, choose `Print firing and resolved alerts`.

This template prints out alert instances into two sections: **firing alerts** and **resolved alerts**, and includes only the key details for each. In addition, it adds our summary and description annotations.

```
{{- /* Example displaying firing and resolved alerts separately in the notification. */ -}}
{{- /* Edit the template name and template content as needed. */ -}}
{{ define "custom.firing_and_resolved_alerts" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
  {{ template "alert.summary_and_description" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
  {{ template "alert.summary_and_description" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.summary_and_description" }}
  Summary: {{.Annotations.summary}}
  Status: {{ .Status }}
  Description: {{.Annotations.description}}
{{ end -}}
```

{{< docs/ignore >}}
Note: Your notification template name (`{{define "<NAME>"}}`) must be unique. You cannot have two templates with the same name in the same notification template group or in different notification template groups.
{{< /docs/ignore >}}

<!-- INTERACTIVE ignore START -->

{{< admonition type="note" >}}
Your notification template name (`{{define "<NAME>"}}`) must be unique. You cannot have two templates with the same name in the same notification template group or in different notification template groups.
{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

Here’s a breakdown of the template:

- `{{ define "custom.firing_and_resolved_alerts" -}}` section: Displays the number of resolved alerts and their summaries, using the `alert.summary_and_description` template to include the summary, status, and description for each alert.
- `.Alerts.Firing` section: Similarly lists the number of firing alert instances and their details.
- `alert.summary_and_description`: This sub-template pulls the summary annotation you configured earlier.

In the **Preview** area, you can see a sample of how the notification would look. Since we’ve already created our alert rule, you can take it a step further by previewing how an actual alert instance from your rule would appear in the notification.

1. Click **Edit payload**.
1. Click **Use existing alert instance**.

   You should see our alert rule listed on the left.

1. Click the alert rule.
1. Select an instance.
1. Click **Add alert data to payload**.

   The alert instance is added to the bottom of the preview.

   {{< figure src="/media/docs/alerting/alert-instance-preview-in-template.png" max-width="1200px" caption="Preview of an alert instance in a notification template" >}}

1. Click **Save**.

With the notification template ready, the next step is to apply it to your contact point to see it in action.

<!-- INTERACTIVE page step4.md END -->
<!-- INTERACTIVE page step5.md START -->

### Apply the template to your contact point

1. Apply the template to your contact point.
   - Navigate to **Alerts & IRM** > **Alerting** > **Contact points**.
   - Edit your contact point.
1. **Optional** [email] **settings** section:
   - Click **Edit Message**.
   - Under **Select notification template**, search `custom.firing_and_resolved_alerts`.
   - Click **Save**.
1. Save your contact point.

<!-- INTERACTIVE page step5.md END -->
<!-- INTERACTIVE page step6.md START -->

### Receiving notifications

Now that the template has been applied to the contact point, you should receive notifications in the specified contact point.

Note: you might need to pause the alert rule evaluation and resume it to trigger the notification.

{{< figure src="/media/docs/alerting/templated-notification-cpu.png" max-width="1200px" caption="Templated email notification for CPU and memory usage" >}}

In the screen capture, you can see how the notification template groups the alert instances into two sections: **firing alerts** and **resolved alerts**. Each section includes only the key details for each alert, ensuring the message remains concise and focused. Additionally, the summary and description annotations we created earlier are included, providing affected instance and CPU usage.

<!-- INTERACTIVE page step6.md END -->
<!-- INTERACTIVE page finish.md START -->

## Conclusion

In this tutorial, we learned how to use templating in Grafana Alerting to create dynamic and actionable alert notifications. We explored how to configure alert rules with annotations, design custom notification templates, and apply them to contact points to enhance the clarity and efficiency of alert messages. By organizing alert instances into concise notifications, you can reduce noise and ensure on-call engineers quickly understand and address critical issues.

To deepen your understanding of Grafana’s templating, explore the following resources:

- **Overview of the functions and operators used in templates**:
  - [Notification template language](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/language/)
  - [Alert rule template language](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/language/)

- [**Notification template reference**](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/reference/): Lists the data available for use in notification templates and explores specific functions.
- [**Alert rule template reference**](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/reference/): Covers the specifics of creating dynamic labels and annotations for alert rules using elements such as variables and functions.

## Learn more in [Grafana Alerting: Route alerts using dynamic labels](http://www.grafana.com/tutorials/alerting-get-started-pt5/)

<!-- INTERACTIVE ignore START -->

{{< admonition type="tip" >}}

In [Get started with Grafana Alerting: Route alerts using dynamic labels](http://www.grafana.com/tutorials/alerting-get-started-pt5/) you learn how to dynamically route alerts and link them to dashboards.

{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

In [Get started with Grafana Alerting: Route alerts using dynamic labels](http://www.grafana.com/tutorials/alerting-get-started-pt5/) you learn how to dynamically route alerts and link them to dashboards.

{{< /docs/ignore >}}

<!-- INTERACTIVE page finish.md END -->

# Get started with Grafana Alerting - Route alerts using dynamic labels

<!-- INTERACTIVE page intro.md START -->

The Get started with Grafana Alerting - Dynamic routing tutorial is a continuation of the [Get started with Grafana Alerting - Templating](http://www.grafana.com/tutorials/alerting-get-started-pt4/) tutorial.

{{< youtube id="hSejnv1cdYY" >}}

Imagine you are managing a web application or a fleet of servers, tracking critical metrics such as CPU, memory, and disk usage. While monitoring is essential, managing alerts allows your team to act on issues without necessarily feeling overwhelmed by the noise.

In this tutorial you will learn how to:

- Leverage notification policies for **dynamic routing based on query values**: Use notification policies to route alerts based on dynamically generated labels, in a way that critical alerts reach the on-call team and less urgent ones go to a general monitoring channel.

<!-- INTERACTIVE page intro.md END -->
<!-- INTERACTIVE page step1.md START -->

<!-- INTERACTIVE ignore START -->

## Before you begin

- **Interactive learning environment**
  - Alternatively, you can [try out this example in our interactive learning environment](https://killercoda.com/grafana-labs/course/grafana/alerting-get-started-pt5/). It’s a fully configured environment with all the dependencies already installed.

- **Grafana OSS**
  - If you opt to run a Grafana stack locally, ensure you have the following applications installed:
    - Docker Compose (included in Docker for Desktop for macOS and Windows)
    - Git

<!-- INTERACTIVE ignore END -->

### Set up the Grafana stack

To observe data using the Grafana stack, download and run the following files.

1. Clone the [tutorial environment repository](https://github.com/tonypowa/grafana-prometheus-alerting-demo.git).

   ```bash
   git clone https://github.com/tonypowa/grafana-prometheus-alerting-demo.git
   ```

1. Change to the directory where you cloned the repository:

   ```bash
   cd grafana-prometheus-alerting-demo
   ```

1. Build the Grafana stack:

   <!-- INTERACTIVE ignore START -->

   ```
   docker compose build
   ```

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   <!-- INTERACTIVE exec START -->

   ```bash
   docker-compose build
   ```

   <!-- INTERACTIVE exec END -->

   {{< /docs/ignore >}}

1. Bring up the containers:

    <!-- INTERACTIVE ignore START -->

   ```
   docker compose up -d
   ```

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   <!-- INTERACTIVE exec START -->

   ```bash
   docker-compose up -d
   ```

   <!-- INTERACTIVE exec END -->

   {{< /docs/ignore >}}

   The first time you run `docker compose up -d`, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

 <!-- INTERACTIVE ignore START -->

{{< admonition type="note" >}}
If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.
{{< /admonition >}}

   <!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

NOTE:

If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

{{< /docs/ignore >}}

<!-- INTERACTIVE page step1.md END -->
<!-- INTERACTIVE page step2.md START -->

## Use case: monitoring and alerting for system health with Prometheus and Grafana

In this use case, we focus on monitoring the system's CPU, memory, and disk usage as part of a monitoring setup. This example is based on the [Grafana Prometheus Alerting Demo](https://github.com/tonypowa/grafana-prometheus-alerting-demo), which collects and visualizes system metrics via Prometheus and Grafana.

Your team is responsible for ensuring the health of your servers, and you want to leverage advanced alerting features in Grafana to:

- Set who should receive an alert notification based on query value.
- Suppress alerts based on query value.

### Scenario

In the provided demo setup, you're monitoring:

- CPU Usage.
- Memory Consumption.

You have a mixture of critical alerts (e.g., CPU usage over `75%`) and warning alerts (e.g., memory usage over `60%`).

This Flask-based Python script simulates a service that:

- Generates random CPU and memory usage values (10% to 100%) every **10 seconds**
- Exposes them as Prometheus metrics
- Each metric includes a default instance label based on the scrape target:
  - `instance="flask-prod:5000"`
  - `instance="flask-staging:5000"`
- A custom deployment label added explicitly in the app logic (this serves as an additional example for dynamically routing production instances):
  - `deployment="prod-us-cs30"`
  - `deployment="staging-us-cs20"`

### Objective

Use templates to dynamically populate a custom label that matches a notification policy, and therefore routes alerts to the correct contact point.

We'll automatically determine the environment associated with each firing alert by inspecting system metrics (e.g., CPU, memory) and extracting keywords using regular expressions with the Go templating language.

<!-- INTERACTIVE page step2.md END -->

<!-- INTERACTIVE page step3.md START -->

## Step 1: Create Notification Policies

Notification policies route alert instances to contact points via label matchers. Since we know what labels our application returns (e.g., `job`, `instance`, `deployment`), we can use them to match alert rules and define appropriate notification routing.

Although our application doesn't explicitly include an `environment` label, we can rely on other labels like `instance` or `deployment`, which may contain keywords (like prod or staging) that indicate the environment.

1. Sign in to Grafana:
   - **Grafana Cloud** users: Log in via Grafana Cloud.
   - **OSS users**: Go to [http://localhost:3000](http://localhost:3000).

1. Navigate to **Alerts & IRM > Alerting > Notification Policies**.

1. Add a child policy:
   - In the **Default policy**, click **+ New child policy**.
   - **Label**: `environment`.
   - **Operator**: `=`.
   - **Value**: `production`.
   - This label matches alert rules where the environment label is `prod`.

1. Choose a **contact point**:
   - If you don’t have any contact points, add a [Contact point](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/#add-a-contact-point).

   For a quick test, you can use a public webhook from [webhook.site](https://webhook.site/) to capture and inspect alert notifications. If you choose this method, select **Webhook** from the drop-down menu in contact points.

1. Enable continue matching:
   - Turn on **Continue matching subsequent sibling nodes** so the evaluation continues even after one or more labels (i.e., _environment_ labels) match.

1. Save and repeat
   - Create another child policy by following the same steps.
   - Use `environment = staging` as the label/value pair.
   - Feel free to use a different contact point.

Now that the labels are defined, we can create alert rules for CPU and memory metrics. These alert rules will use the labels from the collected and stored metrics in Prometheus.

<!-- INTERACTIVE page step3.md END -->
<!-- INTERACTIVE page step4.md START -->

## Step 2: Create alert rules to monitor CPU and memory usage

Follow these steps to manually create alert rules and link them to a visualization.

## Create an alert rule for CPU usage

1. Navigate to **Alerts & IRM > Alerting > Alert rules** from the Grafana sidebar.
1. Click **+ New alert rule** rule to create a new alert.

### Enter alert rule name

Make it short and descriptive, as this will appear in your alert notification. For instance, `cpu-usage` .

### Define query and alert condition

1. Select **Prometheus** data source from the drop-down menu.
1. In the query section, enter the following query:

   ** switch to **Code** mode if not already selected **

   ```
   flask_app_cpu_usage{}
   ```

1. **Alert condition** section:
   - Enter `75` as the value for **WHEN QUERY IS ABOVE** to set the threshold for the alert.
   - Click **Preview alert rule condition** to run the queries.

   {{< figure src="/media/docs/alerting/flask-app-metrics.png" max-width="1200px" caption="Preview of a query returning alert instances in Grafana." >}}

   Among the labels returned for `flask_app_cpu_usage`, the labels `instance` and `deployment` contain values that include the term _prod_ and _staging_. We will create a template later to detect these keywords, so that any firing alert instances are routed to the relevant contact points (e.g., alerts-prod, alerts-staging).

### Add folders and labels

In this section we add a [templated label based on query value](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/examples/#based-on-query-value) to map to the notification policies.

<!-- INTERACTIVE page step4.md END -->
<!-- INTERACTIVE page step5.md START -->

1. In **Folder**, click _+ New folder_ and enter a name. For example: `app-metrics` . This folder contains our alerts.
1. Click **+ Add labels**.
1. **Key** field: `environment` .
1. In the **value** field copy in the following template:

   ```go
   {{- $env := reReplaceAll ".*([pP]rod|[sS]taging|[dD]ev).*" "${1}" $labels.instance -}}
   {{- if eq $env "prod" -}}
   production
   {{- else if eq $env "staging" -}}
   staging
   {{- else -}}
   development
   {{- end -}}
   ```

   This template uses a regular expression to extract `prod`, `staging`, or `dev` from the instance label (`$labels.instance`) and maps it to a more readable label (like "production" for "prod").

As result, when alerts exceed a threshold, the template checks the labels, such as `instance="flask-prod:5000"`, `instance="flask-staging:5000"`, or custom labels like `deployment="prod-us-cs30"`, and assigns a value of production, staging or development to the custom environment **environment** label.

This label is then used by the alert notification policy to route alerts to the appropriate team, so that notifications are delivered efficiently, and reducing unnecessary noise.

### Set evaluation behaviour

1. Click + **New evaluation group**. Name it `system-usage`.
1. Choose an **Evaluation interval** (how often the alert will be evaluated). Choose `1m`.
1. Set the **pending period** to `0s` (zero seconds), so the alert rule fires the moment the condition is met (this minimizes the waiting time for the demonstration.).
1. Set **Keep firing for** to, `0s`, so the alert stops firing immediately after the condition is no longer true.

### Configure notifications

Select who should receive a notification when an alert rule fires.

1. Toggle the **Advance options** button.
1. Click **Preview routing**.
   The preview should display which firing alerts are routed to contact points based on notification policies that match the `environment` label.

   {{< figure src="/media/docs/alerting/dynamic-routing-preview-prod-staging.png" max-width="1200px" caption="Notification policies matched by the environment label matcher." >}}

   The environment label matcher should map to the notification policies created earlier. This makes sure that firing alert instances are routed to the appropriate contact points associated with each policy.

## Step 3: Create a second alert rule for memory usage

1. Duplicate the existing alert rule (**More > Duplicate**), or create a new alert rule for memory usage, defining a threshold condition (e.g., memory usage exceeding `60%`).
1. Give it a name. For example: `memory-usage`
1. Query: `flask_app_memory_usage{}`
1. Link to the same visualization to obtain memory usage annotations whenever the alert rule triggers or resolves.

Now that the CPU and memory alert rules are set up, they are linked to the notification policies through the custom label matcher we added. The value of the label dynamically changes based on the environment template, using `$labels.instance`. This ensures that the label value will be set to production, staging, or development, depending on the environment.

<!-- INTERACTIVE page step5.md END -->
<!-- INTERACTIVE page step6.md START -->

## Done! Your alerts are now dynamically routed

Based on your query's `instance` label values (which contain keywords like _prod_ or _staging_ ), Grafana dynamically assigns the value `production`, `staging` or `development` to the custom **environment** label using the template. This dynamic label then matches the label matchers in your notification policies, which route alerts to the correct contact points.

To see this in action go to **Alerts & IRM > Alerting > Active notifications**

This page shows grouped alerts that are currently triggering notifications. If you click on any alert group to view its label set, contact point, and number of alert instances. Notice that the **environment** label has been dynamically populated with values like `production`.

{{< figure src="/media/docs/alerting/routing-active-notification-detail.png" max-width="1200px" caption="Expanded alert in Active notifications section" >}}

Finally, you should receive notifications at the contact point associated with either `prod` or `staging`.

Feel free to experiment by changing the template to match other labels that contain any of the watched keywords. For example, you could reference:

```go
$labels.deployment
```

The template should be flexible enough to capture the target keywords (e.g., prod, staging) by adjusting which label the[`$labels`](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/reference/#labels) is referencing.

<!-- INTERACTIVE page step6.md END -->

<!-- INTERACTIVE page finish.md START -->

## Conclusion

By using notification policies, you can route alerts based on query values, directing them to the appropriate teams.

## Learn more in [Grafana Alerting - Link alerts to visualizations](http://www.grafana.com/tutorials/alerting-get-started-pt6/)

<!-- INTERACTIVE ignore START -->

{{< admonition type="tip" >}}

In [Grafana Alerting - Link alerts to visualizations](http://www.grafana.com/tutorials/alerting-get-started-pt6/) you will create alerts using Prometheus data and link them to your graphs.

{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

In [Grafana Alerting - Link alerts to visualizations](http://www.grafana.com/tutorials/alerting-get-started-pt6/) you will create alerts using Prometheus data and link them to your graphs.

{{< /docs/ignore >}}

Explore related topics covered in this tutorial:

- Understand how alert routing works in [Get started with Grafana Alerting - Alert routing](http://www.grafana.com/tutorials/alerting-get-started-pt2/).
- Learn how templating works in [Get started with Grafana Alerting - Templating](http://www.grafana.com/tutorials/alerting-get-started-pt4/).
  - More [examples on templating labels](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/examples/).

<!-- INTERACTIVE page finish.md END -->

# Get started with Grafana Alerting - Link alerts to visualizations

<!-- INTERACTIVE page intro.md START -->

This tutorial is a continuation of the [Get started with Grafana Alerting - Route alerts using dynamic labels](http://www.grafana.com/tutorials/alerting-get-started-pt5/) tutorial.

{{< youtube id="mqj_hN24zLU" >}}

<!-- USE CASE -->

In this tutorial you will learn how to:

- Link alert rules to time series panels for better visualization
- View alert annotations directly on dashboards for better context
- Write Prometheus queries

<!-- INTERACTIVE page intro.md END -->
<!-- INTERACTIVE page step1.md START -->

<!-- INTERACTIVE ignore START -->

## Before you begin

- **Interactive learning environment**
  - Alternatively, you can [try out this example in our interactive learning environment](https://killercoda.com/grafana-labs/course/grafana/alerting-get-started-pt6/). It’s a fully configured environment with all the dependencies already installed.

- **Grafana OSS**
  - If you opt to run a Grafana stack locally, ensure you have the following applications installed:
    - Docker Compose (included in Docker for Desktop for macOS and Windows)
    - Git

<!-- INTERACTIVE ignore END -->

### Set up the Grafana stack

To observe data using the Grafana stack, download and run the following files.

1. Clone the [tutorial environment repository](https://github.com/tonypowa/grafana-prometheus-alerting-demo.git).

   ```bash
   git clone https://github.com/tonypowa/grafana-prometheus-alerting-demo.git
   ```

1. Change to the directory where you cloned the repository:

   ```bash
   cd grafana-prometheus-alerting-demo
   ```

1. Build the Grafana stack:

   <!-- INTERACTIVE ignore START -->

   ```
   docker compose build
   ```

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   <!-- INTERACTIVE exec START -->

   ```bash
   docker-compose build
   ```

   <!-- INTERACTIVE exec END -->

   {{< /docs/ignore >}}

1. Bring up the containers:

    <!-- INTERACTIVE ignore START -->

   ```
   docker compose up -d
   ```

   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   <!-- INTERACTIVE exec START -->

   ```bash
   docker-compose up -d
   ```

   <!-- INTERACTIVE exec END -->

   {{< /docs/ignore >}}

   The first time you run `docker compose up -d`, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

 <!-- INTERACTIVE ignore START -->

{{< admonition type="note" >}}
If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.
{{< /admonition >}}

   <!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

NOTE:

If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

{{< /docs/ignore >}}

<!-- INTERACTIVE page step1.md END -->
<!-- INTERACTIVE page step2.md START -->

## Use case: monitoring and alerting for system health with Prometheus and Grafana

In this use case, we focus on monitoring the system's CPU, memory, and disk usage as part of a monitoring setup. The [demo app](https://github.com/tonypowa/grafana-prometheus-alerting-demo), launches a stack that includes a Python script to simulate metrics, which Grafana collects and visualizes as a time-series visualization.

The script simulates random CPU and memory usage values (10% to 100%) every **10 seconds** and exposes them as Prometheus metrics.

### Objective

You'll build a time series visualization to monitor CPU and memory usage, define alert rules with threshold-based conditions, and link those alerts to your dashboards to display real-time annotations when thresholds are breached.

<!-- INTERACTIVE page step2.md END -->
<!-- INTERACTIVE page step3.md START -->

## Step 1: Create a visualization to monitor metrics

To keep track of these metrics you can set up a visualization for CPU usage and memory consumption. This will make it easier to see how the system is performing.

The time-series visualization supports alert rules to provide more context in the form of annotations and alert rule state. Follow these steps to create a visualization to monitor the application’s metrics.

1. Log in to Grafana:
   - Navigate to [http://localhost:3000](http://localhost:3000), where Grafana should be running.
   - Username and password: `admin`

1. Create a time series panel:
   - Navigate to **Dashboards**.
   - Click **+ Create dashboard**.
   - Click **+ Add visualization**.
   - Select **Prometheus** as the data source (provided with the demo).
   - Enter a title for your panel, e.g., **CPU and Memory Usage**.

1. Add queries for metrics:
   - In the query area, copy and paste the following PromQL query:

     ** switch to **Code** mode if not already selected **

     ```promql
     flask_app_cpu_usage{instance="flask-prod:5000"}
     ```

   - Click **Run queries**.

   This query should display the simulated CPU usage data for the **prod** environment.

1. Add memory usage query:
   - Click **+ Add query**.
   - In the query area, paste the following PromQL query:

     ```promql
     flask_app_memory_usage{instance="flask-prod:5000"}
     ```

   {{< figure src="/media/docs/alerting/cpu-mem-dash.png" max-width="1200px" caption="Time-series panel displaying CPU and memory usage metrics in production." >}}

1. Click **Save dashboard**. Name it: `cpu-and-memory-metrics`.

We have our time-series panel ready. Feel free to combine metrics with labels such as `flask_app_cpu_usage{instance=“flask-staging:5000”}`, or other labels like `deployment`.

<!-- INTERACTIVE page step3.md END -->
<!-- INTERACTIVE page step4.md START -->

## Step 2: Create alert rules

Follow these steps to manually create alert rules and link them to a visualization.

## Create an alert rule for CPU usage

1. Navigate to **Alerts & IRM > Alerting > Alert rules** from the Grafana sidebar.
1. Click **+ New alert rule** rule to create a new alert.

### Enter alert rule name

Make it short and descriptive, as this will appear in your alert notification. For instance, `cpu-usage` .

### Define query and alert condition

1. Select **Prometheus** data source from the drop-down menu.
1. In the query section, enter the following query:

   ** switch to **Code** mode if not already selected **

   ```
   flask_app_cpu_usage{instance="flask-prod:5000"}
   ```

1. **Alert condition**
   - Enter 75 as the value for **WHEN QUERY IS ABOVE** to set the threshold for the alert.
   - Click **Preview alert rule condition** to run the queries.

     {{< figure src="/media/docs/alerting/alert-condition-details-prod.png" max-width="1200px" caption="Preview of a query returning alert instances in Grafana." >}}

   The query returns the CPU usage of the Flask application in the production environment. In this case, the usage is `86.01%`, which exceeds the configured threshold of `75%`, causing the alert to fire.

<!-- INTERACTIVE page step4.md END -->
<!-- INTERACTIVE page step5.md START -->

### Add folders and labels

1. In **Folder**, click **+ New folder** and enter a name. For example: `system-metrics` . This folder contains our alert rules.

### Set evaluation behaviour

1. Click + **New evaluation group**. Name it `system-usage`.
1. Choose an **Evaluation interval** (how often the alert will be evaluated). Choose `1m`.
1. Set the **pending period** to `0s` (None), so the alert rule fires the moment the condition is met (this minimizes the waiting time for the demonstration.).
1. Set **Keep firing for** to, `0s`, so the alert stops firing immediately after the condition is no longer true.

### Configure notifications

- Select a **Contact point**. If you don’t have any contact points, add a [Contact point](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/#add-a-contact-point).

  For a quick test, you can use a public webhook from [webhook.site](https://webhook.site/) to capture and inspect alert notifications. If you choose this method, select **Webhook** from the drop-down menu in contact points.

### Configure notification message

To link this alert rule to our visualization click [**Link dashboard and panel**](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/link-alert-rules-to-panels/#link-alert-rules-to-panels)

- Select the folder that contains the dashboard. In this case: **system-metrics**
- Select the **cpu-and-memory-metrics** visualization
- Click **confirm**

You have successfully linked this alert rule to your visualization!

When the CPU usage exceeds the defined threshold, an annotation should appear on the graph to mark the event. Similarly, when the alert is resolved, another annotation is added to indicate the moment it returned to normal.

Try adding a second alert rule using the memory usage metric (`flask_app_memory_usage{instance="flask-prod:5000"`}) to see how combining multiple alerts can enhance your dashboard.

Check how your dashboard looks now that your alert has been linked to your dashboard panel.

<!-- INTERACTIVE page step5.md END -->
<!-- INTERACTIVE page step6.md START -->

## Step 3: Visualizing metrics and alert annotations

After the alert rules are linked to visualization, they should appear as **health indicators** (colored heart icons: a red heart when the alert is in **Alerting** state, and a green heart when in **Normal** state) on the linked panel. In addition, annotations provide helpful context, such as the time the alert was triggered.

{{< figure src="/media/docs/alerting/alert-in-panel.png" max-width="1200px" caption="Time series panel displaying health indicators and annotations." >}}

<!-- INTERACTIVE page step6.md END -->
<!-- INTERACTIVE page step7.md START -->

## Step 4: Receiving notifications

Finally, as part of the alerting process, you should receive notifications at the associated contact point. If you're receiving alerts via email, the default email template will include two buttons:

- **View dashboard**: links to the dashboard that contains the alerting panel

- **View panel**: links directly to the individual panel where the alert was triggered

{{< figure src="/media/docs/alerting/email-notification-w-url.png" max-width="1200px" caption="Alert notification with links to panel and dashboard." >}}

Clicking either button opens Grafana with a pre-applied time range relevant to the alert.

By default, this URL includes `from` and `to` query [parameters](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/reference/#alert) that reflect the time window around the alert event (one hour before and after the alert). This helps you land directly in the time window where the alert occurred, making it easier to analyze what happened.

If you want to define a more intentional time range, you can customize your notifications using a [notification template](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/examples/#print-a-link-to-a-dashboard-with-time-range). With a template, you can explicitly set `from` and `to` values for more precise control over what users see when they follow the dashboard link. The final URL is constructed using a custom annotation (e.g., `MyDashboardURL`) along with the `from` and `to` parameters, which are calculated in the notification template.

<!-- INTERACTIVE page step7.md END -->
<!-- INTERACTIVE page finish.md START -->

## Conclusion

You’ve now linked Prometheus-based alert rules to your Grafana visualizations, giving your dashboards real-time context with alert annotations and health indicators. By visualizing alerts alongside metrics, responders can quickly understand what’s happening and when. You also saw how alert notifications can include direct links to the affected dashboard or panel, helping teams jump straight into the right time window for faster troubleshooting.

Have feedback or ideas to improve this tutorial? [Let us know](https://github.com/grafana/tutorials/issues/new).

<!-- INTERACTIVE page finish.md END -->

# How to create Grafana alerts with InfluxDB and the Flux query language

[Grafana Alerting](/docs/grafana/latest/alerting/) represents a powerful new approach to systems observability and incident response management. While the alerting platform is perhaps best known for its strong integrations with Prometheus, the system works with numerous popular data sources including InfluxDB. In this tutorial we will learn how to create Grafana alerts using InfluxDB and the newer Flux query language. We will cover five common scenarios from the most basic to the most complex. Together, these five scenarios will provide an excellent guide for almost any type of alerting query that you wish to create using Grafana and Flux.

Before we dive into our alerting scenarios, it is worth considering the development of InfluxDB's two popular query languages: InfluxQL and Flux. Originally, InfluxDB used [InfluxQL](https://docs.influxdata.com/influxdb/v2.5/reference/syntax/influxql/spec/) as their query language, which uses a SQL-like syntax. But beginning with InfluxDB v1.8, the company introduced [Flux](https://docs.influxdata.com/flux/v0.x/), "an open source functional data scripting language designed for querying, analyzing, and acting on data." "Flux," its official documentation goes on to state, "unifies code for querying, processing, writing, and acting on data into a single syntax. The language is designed to be usable, readable, flexible, composable, testable, contributable, and shareable."

In the following five examples we will see just how powerful and flexible the new Flux query language can be. We will also see just how well Flux pairs with Grafana Alerting.

## Example 1: Create an alert when a value is above or below a set threshold

Our first example uses a common real-world scenario for InfluxDB and Grafana Alerting. Popular with IoT and edge applications, InfluxDB excels at on-site, real-time observability. In this example, and in fact for many of the following examples, we will consider the hypothetical scenario where we are monitoring a number of fluid tanks in a manufacturing plant. This scenario, [based on an actual application of InfluxDB and Alerting](/go/grafanaconline/2021/plant-efficiency-grafana-cloud/), will allow us to work through Grafana's various alerting setups, progressing from the simplest to the most complex.

For Example 1, let's consider the following scenario: we are monitoring one tank, `A5`, for which we are storing real-time temperature data. We need to make sure that the temperature in this tank is always greater than 30 °C and less than 60 °C.

We want to write a Grafana alert that will trigger whenever the temperature in tank `A5` crosses the lower threshold of 30 °C or the upper threshold of 60 °C.

To do this, we'll: create a Grafana alert rule, add a Flux query, and then add expressions to the alert rule.

### Create a Grafana Alert rule

1. Open the Grafana alerting menu and select **Alert rules**.
1. Click **New alert rule**.
1. Give your alert rule a name and then select **Grafana managed alert**.
   For InfluxDB, you will always create a [Grafana managed rule](/docs/grafana/latest/alerting/alerting-rules/create-grafana-managed-rule/#add-grafana-managed-rule).

### Add an initial Flux query to the alert rule

Still in the **Step 2** section of the Alert rule page, you will see three boxes: a query editor (`A`), and then two sections labelled `B` and `C`. You will use these three sections to construct your rule. Let's move through them one by one.

First, we want to query the data in our imaginary InfluxDB instance to obtain a time series graph of the temperature of tank A5. For this you would choose your InfluxDB data source from the dropdown and then write a query like this:

    ```
     from(bucket: "RetroEncabulator")
    |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
    |> filter(fn: (r) => r["_measurement"] == "TemperatureData")
    |> filter(fn: (r) => r["Tank"] == "A5")
    |> filter(fn: (r) => r["_field"] == "Temperature")
    |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
    |> yield(name: "mean")
    ```

This is a fairly typical Flux query. Let's go through it function by function. We begin using [the `from()` function](https://docs.influxdata.com/flux/v0.x/stdlib/influxdata/influxdb/from/) to choose the correct bucket where our tank data resides. Then we use [a `range()` function](https://docs.influxdata.com/flux/v0.x/stdlib/universe/range/) to filter our rows based on time constraints. Then we pass our data through three [`filter()` functions](https://docs.influxdata.com/flux/v0.x/stdlib/universe/filter/) to narrow our results. We choose a specific [`measurement` (a special keyword in InfluxDB)](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#measurement), then our tank in question (`A5`), and then a specific [`field` (another special keyword in InfluxDB)](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#field). After this we pass the data into [an `aggregateWindow()` function](https://docs.influxdata.com/flux/v0.x/stdlib/universe/aggregatewindow/), which downsamples our data into specific periods of time, and then finally [a `yield()` function](https://docs.influxdata.com/flux/v0.x/stdlib/universe/yield/), which specifies which final result we want: `mean`.

This Flux query will yield a time-series graph like this:

![grafana alerts from flux queries](/media/tutorials/screenshot-flux-timeseries-graph.png)

### Add expressions to your Grafana Alert rule

With data now appearing in our rule setup, our next step is to create an [expression](/docs/grafana/v9.0/panels/query-a-data-source/use-expressions-to-manipulate-data/about-expressions/#using-expressions). Move to section `B`. For this scenario, we want to create a Reduce expression that will reduce the above to a single value. In this image, you can see that we have chosen to reduce our time-series data the `Last` value from input `A`. In this case, it returns a value 53 degrees celsius for Tank A5:

![grafana alerts from flux queries](/media/tutorials/screenshot-flux-reduce-expression.png)

Finally, we need to create a math expression that Grafana will alert on. In our case we will write an expression with two conditions separated by the OR `||` operator. We want to trigger an alert any time our result in section `B` is less than 30 or more than 60. This looks like `$B < 30 || $B > 60`:

![grafana alerts from flux queries](/media/tutorials/screenshot-flux-math-expression.png)

Set the alert condition to `C - expression`. We can now preview our alert. Here is a preview of this alert when the state is `Normal`:

![grafana alerts from flux queries](/media/tutorials/screenshot-flux-alert-preview-state-normal.png)

And here is a preview of this alert when the state is `Alerting`:

![grafana alerts from flux queries](/media/tutorials/screenshot-flux-alert-alert-preview-state-alerting.png)

Note that the Reduce expression above is needed. Without it, when previewing the results, Grafana would display `invalid format of evaluation results for the alert definition B: looks like time series data, only reduced data can be alerted on`.

💡Tip: In case your locale is still stubbornly using Fahrenheit, we can modify the above Flux query by adding (before the aggregateWindow statement) a map() function to to convert (or map) the values from °C to °F. Note that we are not creating a new field. We are simply remapping the existing value.

```flux
|> map(fn: (r) => ({r with _value: r._value * 1.8 + 32.0}))
```

### Conclusion

Using these three steps you can create a Flux-based Grafana Alert that will trigger on either of two thresholds from a single data source. But what if you need to trigger an alert based on **multiple conditions and from multiple time-series**? In example two we will cover this very scenario.

## Example 2: how to create a Grafana alert from two queries and two conditions

Let's mix things up a bit for example two and leave our imaginary manufacturing plant. Imagine you're an assistant to the great Dr. Emmett Brown from Back to the Future, and Doc has tasked you with the following challenge: "I want an alert sent to me every time both conditions for time travel are met: when the velocity of a vehicle reaches 88 miles per hour and an object generates 1.21 jigowatts of electricity."

Let's assume we are tracking this data in InfluxDB and Grafana. Let's also assume that each of the above data sources comes from different buckets. How do we alert on this? How do we use Grafana and Flux to alert on two distinct conditions originating from two distinct data sources?

### Add two Flux queries to your Grafana Alert rule

Like we did in example 1, let's first mock up our queries. Our query for our vehicle data is very similar to our last query. We use a `from()`, `range()`, and a sequence of `filter()` functions. We then use `AggregateWindow()` and `yield()` to narrow our data even more. In this case, the result is a time series tracking the velocity of our 1983 DeLorean:

```flux
from(bucket: "vehicles")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "VehicleData")
|> filter(fn: (r) => r["VehicleType"] == "DeLorean")
|> filter(fn: (r) => r["VehicleYear"] == "1983")
|> filter(fn: (r) => r["_field"] == "velocity")
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: "mean")
```

Our second query will trigger an alert whenever our electricity resource (the lightning strike on the Hill Valley clocktower) reaches the needed 1.21 jigowatts. A query like this would look very similar to our vehicle velocity query:

```flux
from(bucket: "HillValley")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "ElectricityData")
|> filter(fn: (r) => r["Location"] == "clocktower")
|> filter(fn: (r) => r["Source"] == "lightning")
|> filter(fn: (r) => r["_field"] == "power")
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: "mean")
```

We are now ready to modify this data using expressions.

### Add expressions to your Grafana Alert rule

1. Let's now use the same steps to reduce each query to the last (most recent) value. Reducing Query `A` to a single value might look like this:

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-additional-queries-reduce-expression-A.png)

1. And here we are reducing query `B`:

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-additional-queries-reduce-expression-B.png)

1. Now, in section `C` we need to create a math expression to be alerted on. In this case we will use the AND `&&` operator to specify that two conditions must be met: the value of `C` (the reduced value from query `A`) must be greater than 88.0 while the value of `D` (the reduced value from query `B`) must be greater than 1.21. We write this as `$C > 88.0 && $D > 1.21`

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-additional-queries-math-expression.png)

And here is a preview of our alerts:

![grafana alerts from flux queries](https://raw.githubusercontent.com/grafana/tutorials/master/content/tutorials/assets/flux-additional-queries-alert-preview.png)

💡Tip: If your data in InfluxDB happens to have an unnecessarily large number of digits to the right of the decimal (such as 1.2104705741732575 shown above), and you want your Grafana alerts to be more legible, try using {{ printf "%.2f"  $values.D.Value }}. For example, in the annotation Summary, we could write the following:

```
{{  $values.D.Labels.Source }} at the {{  $values.D.Labels.Location }} has generated {{ printf "%.2f"  $values.D.Value }} jigowatts.`
```

This will display as follows:
![grafana alerts from flux queries](/media/tutorials/screenshot-flux-tip-significant-figures.png))

You can reference our documentation on [alert message templating](/docs/grafana/latest/alerting/contact-points/message-templating/) to learn more about this powerful feature.

### Conclusion

In this example we showed how to create a Flux-based alert that uses two distinct conditions from two distinct queries that use data from two distinct data sources. For example three we will switch gears and tackle another popular alerting scenario: how to create an alert based on an aggregated (per day) value.

## Example 3: how to create a Grafana Alert based on an aggregated (per-day) value

One of the most common requests in [Grafana's community forum](https://community.grafana.com) involves graphing daily electrical consumption and production. This sort of data is very often stored in InfluxDB. In this example we will see how to aggregate time series data into a per-day value and then alert on it.

Let’s assume our electricity meter sends a reading to InfluxDB once per hour and contains the total kWh used for that hour. We want to write a query that will aggregate these per-hour values into a per-day value, then create an alert that triggers when the power consumption (kWh) exceeds 5,000 kWh per day.

### Add an initial Flux query to your Grafana Alert rule

1. Let's begin by examining a typical query and the resulting time graph for our hourly data across a 7-day period. A query like this is shown below:

   ```flux
   from(bucket: "RetroEncabulator")
   |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
   |> filter(fn: (r) => r["_measurement"] == "ElectricityData")
   |> filter(fn: (r) => r["Location"] == "PlantD5")
   |> filter(fn: (r) => r["_field"] == "power_consumed")
   |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
   |> yield(name: "power")
   ```

   We can see the same pattern of Flux functions here that we say in examples 1 and 2. A query like this would produce a graph similar to the following:

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-aggregatewindow-timeseries-graph.png)

1. Now let's adjust our query to calculate daily usage. With many datasources, this can be a rather complex operation. But with Flux, by simply changing the aggregateWindow function parameters we can calculate the daily usage over the same 7-day period:

   ```flux
   from(bucket: "RetroEncabulator")
     |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
     |> filter(fn: (r) => r["_measurement"] == "ElectricityData")
     |> filter(fn: (r) => r["Location"] == "PlantD5")
     |> filter(fn: (r) => r["_field"] == "power_consumed")
     |> aggregateWindow(every: 1d, fn: sum)
     |> yield(name: "power")
   ```

   Note how we've adjusted our `aggregateWindow()` function to `aggregateWindow(every: 1d, fn: sum)`. This results in a graph like so:

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-aggregatewindow-aggregated.png)

1. Add expressions to your Grafana Alert rule.

   Now that we have our per-day query correct, we can continue using the same pattern as before, adding expressions to reduce and perform math on our results.

   As before, let's reduce our query to a single value:

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-aggregatewindow-reduce-expression.png)

   Now create a math expression to be alerted on and set the evaluation behavior. In this case we want to write `$B > 5000`:

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-aggregatewindow-math-expression.png)

   And now we are alerting on our daily electricity consumption whenever we exceed 5000 kWh. Here is preview of our alert:

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-aggregatewindow-alert-preview.png)

### Conclusion

Plotting and aggregating electrical consumption is a common use case for combining InfluxDB and Grafana. Using Flux, we saw just how easy it can be to group our data by day and then alert on that daily value. In our next two examples we will examine the more complex form of Grafana Alert: multidimensional alerts.

## Example 4: create a dynamic (multidimensional) Grafana Alert using Flux

Let’s return to our fluid tanks from example 1, but this time let’s assume we have 5 tanks (A5, B4, C3, D2, and E1). We are now tracking the temperature in five tanks: A5, B4, C3, D2, and E1.

We want to create one multidimensional alert that will notify us whenever the temperature in any tank is less than 30 °C or greater than 60 °C.

### Add an initial Flux query to your Grafana Alert rule

We begin, as always, by writing our initial query. This is very similar to our query in example 1, but note how our third `filter()` function captures the data from all five tanks and not just `A5`:

```flux
from(bucket: "HyperEncabulator")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "TemperatureData")
|> filter(fn: (r) => r["MeasType"] == "actual")
|> filter(fn: (r) => r["Tank"] == "A5" or r["Tank"] == "B4" or r["Tank"] == "C3" or r["Tank"] == "D2" or r["Tank"] == "E1")
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: "mean")
```

💡Tip: If the tanks were shut down every night from 23:00 to 07:00, they would possibly fall below the 30 °C threshold. If one did not want to receive alerts during those hours, one can use the Flux function hourSelection() which filters rows by time values in a specified hour range.

```flux
|> hourSelection(start: 7, stop: 23)`
```

A query like the one above will produce a time series graph like this:

![grafana alerts from flux queries](/media/tutorials/screenshot-flux-multidimensional-timeseries-graph.png)

### Add expressions to your Grafana Alert rule

1. We create a Reduce expression that will reduce the time series for each tank to a single value. This gives us five distinct temperatures:

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-multidimensional-reduce-expression.png))

1. Create a math expression to be alerted on. This is the exact same expression from example 1, `$B < 30 || $B > 60`:

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-multidimensional-math-expression.png)

As we can see three tanks are within the acceptable thresholds while two tanks have crossed the upper boundary. This would trigger an alert for tanks `D2` and `E1`.

### Conclusion

With multidimensional alerts we can avoid repeating ourselves. But what if the scenario were even more complex? In the next and final example, we will examine how to use multidimensional alerts to create the most dynamic alerts possible.

## Example 5: how to create a dynamic (multidimensional) Grafana Alert using multiple queries and multiple thresholds with Flux

For this final example let's continue with our five fluid tanks and their five datasets.Let’s assume again that each tank has a temperature controller with a setpoint value that is stored in InfluxDB. Let’s mix things up and assume that each tank has a _different_ setpoint, where we always need to be within 3 degrees of the setpoint.

We want to create one multidimensional alert that will cover each unique scenario for each tank, triggering an alert whenever any tank's temperature moves beyond its unique allowable range.

To better visualize this challenge, here is a table representing our five tanks, their temperature setpoints, and their allowable range:

| Tank | Setpoint | Allowable Range (±3) |
| ---- | -------- | -------------------- |
| A5   | 45       | 42 to 48             |
| B4   | 55       | 52 to 58             |
| C3   | 60       | 57 to 63             |
| D2   | 72       | 69 to 75             |
| E1   | 80       | 77 to 83             |

With Grafana Alerting, we can create a single multidimensional rule to cover all 5 tanks, and we can use Flux to compare the setpoint and actual value for each tank. In other words, one multidimensional alert can monitor 5 separate tanks, each with different setpoints and actual values, but all with one common "allowable threshold" (i.e. a temperature difference of ±3 degrees).

### Add an initial Flux query to your Grafana Alert rule

Let's begin with our data query. It is similar to our past queries, only now more complex. We must add extra functions to get our data into the proper format, including a `pivot()`, `map()`, `rename()`, `keep()`, and `drop()` function:

```flux
from(bucket: "HyperEncabulator")
 |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
 |> filter(fn: (r) => r["_measurement"] == "TemperatureData")
 |> filter(fn: (r) => r["MeasType"] == "actual" or r["MeasType"] == "setpoint")
 |> filter(fn: (r) => r["Tank"] == "A5" or r["Tank"] == "B4" or r["Tank"] == "C3" or r["Tank"] == "D2" or r["Tank"] == "E1")
 |> filter(fn: (r) => r["_field"] == "Temperature")
 |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
 |> pivot(rowKey:["_time"], columnKey: ["MeasType"], valueColumn: "_value")
 |> map(fn: (r) => ({ r with _value: (r.setpoint - r.actual)}))
 |> rename(columns: {_value: "difference"})
 |> keep(columns: ["_time", "difference", "Tank"])
 |> drop(columns: ["actual", "setpoint"])
 |> yield(name: "mean")
```

Note in the above that we are calculating the difference between the actual and the setpoint. The way Grafana parses the result from InfluxDB is that if a \_value column is found, it is assumed to be a time-series. The quick workaround is to add the following `rename()` function:

```flux
 |> rename(columns: {_value: "something"})
```

The above query results in this time series:

![grafana alerts from flux queries](/media/tutorials/screenshot-flux-complex-query-timeseries-graph.png)

### Add expressions to your Grafana Alert rule

1. Again, we create a Reduce expression for the above query to reduce each of the above to a single value. This value represents the temperature differential between each tank's setpoint and its actual real-time temperature:

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-complex-query-reduce-expression.png)

1. Now we create a math expression to be alerted on. This time we will create a condition that checks if the absolute value of our reduce calculation is greater than 3, `abs($(B))>3.0`:

   ![grafana alerts from flux queries](/media/tutorials/screenshot-flux-complex-query-math-expression.png)

We can now see that two tanks, `D2` and `E1`, are evaluating to true. When we preview the alert we can see that those two tanks will trigger a notification and change their state from `Normal` to `Alerting`:

![grafana alerts from flux queries](/media/tutorials/screenshot-flux-complex-query-alert-preview-state-normal.png)
![grafana alerts from flux queries](/media/tutorials/screenshot-flux-complex-query-alert-preview-state-alerting.png)

### Conclusion

Flux queries and Grafana Unified Alerting are a powerful combination to identify practically any alertable conditions in your dataset, or across your entire system. For more information on Grafana Alerting, [visit the documentation here](/docs/grafana/latest/alerting/). For more information on the Flux query language, [you can visit that documentation as well](https://docs.influxdata.com/flux/v0.x/).

# How to create alert rules with log data

<!-- INTERACTIVE page intro.md START -->

# How to create alert rules with log data

Loki stores your logs and only indexes labels for each log stream. Using Loki with Grafana Alerting is a powerful way to keep track of what’s happening in your environment. You can create metric alert rules based on content in your log lines to notify your team. What’s even better is that you can add label data from the log message directly into your alert notification.

In this tutorial, you'll:

- Generate sample logs and collect them with Grafana Alloy.
- Create an alert rule based on a Loki query (LogQL).
- Create a Webhook contact point to send alert notifications to.

<!-- INTERACTIVE ignore START -->

{{< admonition type="tip" >}}
In [Get started with Grafana Alerting - Part 2](https://grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.
{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

> In [Get started with Grafana Alerting - Part 2](https://grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.

{{< /docs/ignore >}}

<!-- INTERACTIVE page intro.md END -->

<!-- INTERACTIVE page step1.md START -->

## Before you begin

<!-- INTERACTIVE ignore START -->

There are different ways you can follow along with this tutorial.

- **Grafana OSS**

  To run a Grafana stack locally, ensure you have the following applications installed:
  - [Docker Compose](https://docs.docker.com/get-docker/) (included in Docker for Desktop for macOS and Windows)

- **Interactive learning environment**
  - Alternatively, you can [try out this example in our interactive learning environment](https://killercoda.com/grafana-labs/course/grafana/alerting-loki-logs). It's a fully configured environment with all the dependencies already installed.

## Set up the Grafana stack

<!-- INTERACTIVE ignore END -->

To demonstrate the observation of data using the Grafana stack, download and run the following files.

1. Create a folder for this tutorial, then download and save the Docker Compose file, the Alloy configuration, and the Grafana data source provisioning file for this tutorial. Also create a `logs` folder, which Alloy reads the sample logs from.

   ```bash
   mkdir alerts-with-logs && cd alerts-with-logs
   wget https://raw.githubusercontent.com/grafana/tutorial-environment/master/loki-logs/docker-compose.yaml
   wget https://raw.githubusercontent.com/grafana/tutorial-environment/master/loki-logs/config.alloy
   mkdir -p provisioning/datasources logs
   wget -P provisioning/datasources https://raw.githubusercontent.com/grafana/tutorial-environment/master/loki-logs/provisioning/datasources/datasources.yaml
   ```

2. Run the Grafana stack.

   ```bash
   docker compose up -d
   ```

The first time you run `docker compose up -d`, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

<!-- INTERACTIVE ignore START -->

{{< admonition type="note" >}}

If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

> If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

{{< /docs/ignore >}}

<!-- INTERACTIVE page step1.md END -->

<!-- INTERACTIVE page step2.md START -->

## Generate sample logs

To demonstrate how to create alert rules based on logs, you’ll use a script that generates realistic log entries to simulate typical monitoring data in Grafana. Running this script outputs logs continuously, each containing a timestamp, HTTP method (either GET or POST), status code (200 for success or 500 for failures), and request duration in milliseconds.

1. Download and save a Python file that generates logs.

   ```bash
   wget https://raw.githubusercontent.com/grafana/tutorial-environment/master/app/loki/web-server-logs-simulator.py
   ```

1. Execute the log-generating Python script.

   ```bash
   python3 ./web-server-logs-simulator.py | tee -a logs/web_requests.log
   ```

### Troubleshooting the script

If you don't see the sample logs in Explore:

- Does the output file exist, check `logs/web_requests.log` to see if it contains logs.
- If the file is empty, check that you followed the steps above to create the file.
- If the file exists, verify that the alloy container is running.
- In Grafana Explore, check that the time range is only for the last 5 minutes.

<!-- INTERACTIVE page step2.md END -->

<!-- INTERACTIVE page step3.md START -->

## Create a contact point

Besides being an open-source observability tool, Grafana has its own built-in alerting service. This means that you can receive notifications whenever there is an event of interest in your data, and even see these events graphed in your visualizations.

In this step, we set up a new contact point. This contact point uses the [webhook integration](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier/). This contact point uses the _webhooks_ integration. In order to make this work, we also need an endpoint for our webhook integration to receive the alert. We can use [Webhook.site](https://webhook.site/) to quickly set up that test endpoint. This way we can make sure that our alert is actually sending a notification somewhere.

<!-- INTERACTIVE ignore START -->

1. In your browser, **sign in** to your Grafana Cloud account.

   OSS users: To log in, navigate to [http://localhost:3000](http://localhost:3000), where Grafana should be running.

1. In another tab, go to [Webhook.site](https://webhook.site/).
1. Copy Your unique URL.
<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

1. Navigate to [http://localhost:3000](http://localhost:3000), where Grafana should be running.
1. In another tab, go to [Webhook.site](https://webhook.site/).
1. Copy Your unique URL.
   {{< /docs/ignore >}}

Your webhook endpoint is now waiting for the first request.

Next, let's configure a contact point in the Grafana Alerting UI to send notifications to our webhook endpoint.

1. Return to Grafana. In the Grafana sidebar, click **Alerting** > **Notification configuration**, and then select the **Contact points** tab.
1. Click **+ New contact point**.
1. In **Name**, write **Webhook**.
1. In **Integration**, choose **Webhook**.
1. In **URL**, paste the endpoint to your webhook endpoint.
1. Click **Test**, and then click **Send test notification** to send a test alert to your webhook endpoint.
1. Navigate back to _Webhook.site_. On the left side, there's now a `POST /` entry. Click it to see what information Grafana sent.

   {{< figure src="/media/docs/alerting/alerting-webhook-detail.png" max-width="1200px" caption="A POST entry in Webhook.site" >}}

1. Return to Grafana and click **Save contact point**.

We have created a dummy Webhook endpoint and created a new Alerting contact point in Grafana. Now, we can create an alert rule and link it to this new integration.

<!-- INTERACTIVE page step3.md END -->

<!-- INTERACTIVE page step4.md START -->

## Create an alert rule

Next, we'll establish an [alert rule](https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rule-evaluation/) within Grafana Alerting to notify us whenever alert rules are triggered and resolved.

1. In Grafana, **navigate to Alerting** > **Alert rules**.
1. Click on **New alert rule**.
1. Enter alert rule name for your alert rule. Make it short and descriptive as this appears in your alert notification. For instance, **web-requests-logs**

### Define query and alert condition

In this section, we use the default options for Grafana-managed alert rule creation. The default options let us define the query, an expression (used to manipulate the data -- the `WHEN` field in the UI), and the condition that must be met for the alert to be triggered (in default mode is the threshold).

1. Select the **Loki** data source from the drop-down.
1. The query editor opens in **Code** mode by default. If it's showing **Builder** mode instead, click **Code** on the right.
1. Paste the query below.

   ```
   sum by (message)(count_over_time({filename="/var/log/web_requests.log"} != "status=200" | pattern "<_> <message> duration<_>" [10m]))
   ```

   This query counts the number of log lines with a status code that is not 200 (OK), then sum the result set by message type using an **instant query** and the time interval indicated in brackets. It uses the LogQL pattern parser to add a new label called `message` that contains the level, method, url, and status from the log line.

   You can use the **explain query** toggle button for a full explanation of the query syntax. The optional log-generating script creates a sample log line similar to the one below:

   ```
   2023-04-22T02:49:32.562825+00:00 level=info method=GET url=test.com status=200 duration=171ms
   ```

   <!-- INTERACTIVE ignore START -->

   {{< admonition type="note" >}}
   If you're using your own logs, modify the LogQL query to match your own log message. Refer to the Loki docs to understand the [pattern parser](https://grafana.com/docs/loki/latest/logql/log_queries/#pattern).
   {{% / admonition %}}
   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}
   If you're using your own logs, modify the LogQL query to match your own log message. Refer to the Loki docs to understand the [pattern parser](https://grafana.com/docs/loki/latest/logql/log_queries/#pattern).
   {{< /docs/ignore >}}

1. In the **Alert condition** section, confirm the condition reads **WHEN QUERY IS ABOVE** `0`. This is the default, and `0` is the threshold above which the alert rule should trigger.

1. Click **Preview alert rule condition** to run the query.

   It should return alert instances from log lines with a status code that is not 200 (OK), and that has met the alert condition. The condition for the alert rule to fire is any occurrence that goes over the threshold of `0`. Since the Loki query has returned more than zero alert instances, the alert rule is `Firing`.

   {{< figure src="/media/docs/alerting/firing-loki-alert-rule.png" max-width="1200px" caption="Preview of a firing alert instances" >}}

### Set evaluation behavior

The [alert rule evaluation](https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rules/rule-evaluation/) defines the conditions under which an alert rule triggers, based on the following settings:

- **Evaluation group**: every alert rule is assigned to an evaluation group. You can assign the alert rule to an existing evaluation group or create a new one.
- **Evaluation interval**: determines how frequently the alert rule is checked. For instance, the evaluation may occur every 10s, 30s, 1m, 10m, etc.
- **Pending period**: how long the condition must be met to trigger the alert rule.
- **Keep firing for**: defines how long an alert should remain in the Firing state after the alert condition stops being true. During this time, the alert enters a _Recovering_ state, suppressing additional notifications but keeping the alert active. It helps prevent alert flapping, where alerts rapidly switch between firing and resolved due to noisy or unstable metrics.

To set up the evaluation:

1. In **Folder**, click **+ New folder** and enter a name. For example: _web-server-alerts_. This folder contains our alerts.
1. In **Evaluation group and interval**, click **+ New evaluation group**.
1. In **Evaluation group name**, enter _1m-evaluation_.
1. Keep **Evaluation interval** set to `1m` (1 minute), so the alert rule is evaluated every minute, and then click **Create**.
1. Set the pending period to, `0s` (zero seconds), so the alert rule fires the moment the condition is met.
1. Set **Keep firing for** to, `0s`, so the alert stops firing immediately after the condition is no longer true. Use this when you want alerts to be resolved as soon as the system is healthy again.

### Configure labels and notifications

Choose the contact point where you want to receive your alert notifications.

1. Under **Contact point**, select **Webhook** from the drop-down menu.
1. Click **Save** at the bottom of the page.

<!-- INTERACTIVE page step4.md END -->

<!-- INTERACTIVE page step5.md START -->

## Trigger the alert rule

Since the Python script continues to generate log data that matches the alert rule condition, once the evaluation interval has concluded, you should receive an alert notification in the Webhook endpoint.

{{< figure src="/media/docs/alerting/alerting-webhook-firing-alert.png" max-width="1200px" caption="Firing alert notification details" >}}

<!-- INTERACTIVE page step5.md END -->

<!-- INTERACTIVE page finish.md START -->

<!-- INTERACTIVE ignore START -->

{{< admonition type="tip" >}}
In [Get started with Grafana Alerting - Part 2](https://grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.
{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

> In [Get started with Grafana Alerting - Part 2](https://grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.

{{< /docs/ignore >}}

<!-- INTERACTIVE page finish.md END -->

# Create users and teams

This tutorial is for admins or anyone that wants to learn how to manage
users in Grafana. You'll add multiple local users, organize them into teams,
and make sure they're only able to access the resources they need.

### Scenario

_Graphona_, a fictional telemarketing company, has asked you to configure Grafana
for their teams.

In this scenario, you'll:

- Create users and organize them into teams.
- Manage resource access for each user and team through roles and folders.

{{% class "prerequisite-section" %}}

### Prerequisites

- Grafana 7.0 or newer, this tutorial was tested with Grafana 8.5.
- A user with the Admin or Server Admin role.
  {{% /class %}}

## Add users

In Grafana, all users are granted an _organization role_ that determines what
resources they can access.

There are three types of organization roles in Grafana. The **Grafana Admin** is
a global role, the default `admin` user has this role.

- **Grafana Admin -** Manage organizations, users, and view server-wide settings.
- **Organization Administrator -** Manage data sources, teams, and users within an organization.
- **Editor -** Create and edit dashboards.
- **Viewer -** View dashboards.

{{< admonition type="note" >}}
You can also configure Grafana to allow [anonymous access](/docs/grafana/latest/auth/overview/#anonymous-authentication), to make dashboards available even to those who don't have a Grafana user account. That's how Grafana Labs made https://play.grafana.org publicly available.
{{< /admonition >}}

### Exercise

Graphona has asked you to add a group of early adopters that work in the Marketing and Engineering teams. They'll need to be able to edit their own team's dashboards, but want to have view access to dashboards that belong to the other team.

| Name              | Email                         | Username          |
| ----------------- | ----------------------------- | ----------------- |
| Almaz Russom      | almaz.russom@example.com      | almaz.russom      |
| Brenda Tilman     | brenda.tilman@example.com     | brenda.tilman     |
| Mada Rawdha Tahan | mada.rawdha.tahan@example.com | mada.rawdha.tahan |
| Yuan Yang         | yuan.yang@example.com         | yuan.yang         |

#### Add users

Repeat the following steps for each of the employees in the table above to create the new user accounts:

1. Log in as a user that has the **Server Admin** role.
1. On the sidebar, click the **Server Admin** (shield) icon.
1. Choose **Users** from the menu drop-down, then click **New User**.
1. Enter the **Name**, **Email**, **Username**, and **Password** from the table above.
1. Click the **Create User** button to create the account.

When you create a user they are granted the Viewer role by default, which means that they won't be able to make any changes to any of the resources in Grafana. That's ok for now, you'll grant more user permissions by adding users to _teams_ in the next step.

## Assign users to teams

Teams let you grant permissions to a group of users, instead of granting permissions to individual users one at a time.

Teams are useful when onboarding new colleagues. When you add a user to a team, they get access to all resources assigned to that team.

### Exercise

In this step, you'll create two teams and assign users to them.

| Username          | Team        |
| ----------------- | ----------- |
| brenda.tilman     | Marketing   |
| mada.rawdha.tahan | Marketing   |
| almaz.russom      | Engineering |
| yuan.yang         | Engineering |

#### Create a team

Create the _Marketing_ and _Engineering_ teams.

1. In the sidebar, hover your mouse over the **Configuration** (gear) icon and
   then click **Teams**.
1. Click **New team**.
1. In **Name**, enter the name of the team: either _Marketing_ or _Engineering_.
   You do not need to enter an email.
1. Click **Create**.
1. Click on the **Teams** link at the top of the page to return to teams page and create the second team.

#### Add a user to a team

Repeat these steps for each user to assign them to their team. Refer to the table above for team assignments.

1. Click the team name _Marketing_ or _Engineering_ to add members to that team.
1. Click **Add member**.
1. In the **Add team member** box, click the drop-down arrow to choose the user you want to add to the team .
1. Click **Add to team**.

When you're done, you'll have two teams with two users assigned to each.

## Manage resource access with folders

It's a good practice to use folders to organize collections of related dashboards. You can assign permissions at the folder level to individual users or teams.

### Exercise

The Marketing team is going to use Grafana for analytics, while the Engineering team wants to monitor the application they're building.

You'll create two folders, _Analytics_ and _Application_, where each team can add their own dashboards. The teams still want to be able to view each other's dashboards.

| Folder      | Team        | Permissions |
| ----------- | ----------- | ----------- |
| Analytics   | Marketing   | Edit        |
|             | Engineering | View        |
| Application | Marketing   | View        |
|             | Engineering | Edit        |

Repeat the following steps for each folder. You'll move through all three steps for each folder before moving on to the next one.

#### Add a folder for each team

1. In the sidebar, hover your cursor over the **Dashboards** (four squares) icon and then click **Browse**.
1. To create a folder, click **New Folder**.
1. In **Name**, enter the folder name.
1. Click **Create**.
1. Stay in the folder view and move on to the next sections to edit permissions for this folder.

#### Remove the viewer role from folder permissions

By default, when you create a folder, all users with the Viewer role are granted permission to view the folder.

In this example, Graphona wants to explicitly grant teams access to folders. To support this, you need to remove the Viewer role from the list of permissions:

1. Go to the **Permissions** tab.
1. Remove the Viewer role from the list, by clicking the red button on the right.
1. Stay in the permissions tab and move on to the next section to grant folder permissions for each team.

#### Grant folder permissions to a team:

1. Click **Add Permission**.
1. In the **Add Permission For** dialog, make sure "Team" is selected in the first box.
1. In the second box, select the team to grant access to.
1. In the third box, select the access you want to grant.
1. Click **Save**.
1. Repeat for the other team.
1. Click the **Dashboards** link at the top of the page to return to the dashboard list.

When you're finished, you'll have two empty folders, the contents of which can only be viewed by members of the Marketing or Engineering teams. Only Marketing team members can edit the contents of the Analytics folder, only Engineering team members can edit the contents of the Application folder.

## Define granular permissions

By using folders and teams, you avoid having to manage permissions for individual users.

However, there are times when you need to configure permissions on a more granular level. For these cases, Grafana allows you to override permissions for specific dashboards.

### Exercise

Graphona has hired a consultant to assist the Marketing team. The consultant should only be able to access the SEO dashboard in the Analytics folder.

| Name       | Email                            | Username   |
| ---------- | -------------------------------- | ---------- |
| Luc Masson | luc.masson@exampleconsulting.com | luc.masson |

#### Add a new user

1. In the sidebar, click the **Server Admin** (shield) icon.
1. In the Users tab, click **New user**.
1. In **Name**, enter the name of the user.
1. In **E-mail**, enter the email of the user.
1. In **Username**, enter the username that the user will use to log in.
1. In **Password**, enter a password. The user can change their password once they log in.
1. Click **Create user** to create the user account.

#### Create a dashboard

1. In the sidebar, click the **Create** (plus) icon to create a new dashboard.
1. In the top right corner, click the cog icon to go to **Dashboard settings**.
1. In **Name**, enter **SEO**.
1. Click **Save Dashboard**.
1. In the **Save dashboard as...** pop-up, choose the **Analytics** folder from the drop-down and click **Save**.

#### Grant a user permission to view dashboard

1. In the top right corner of your dashboard, click the cog icon to go to **Dashboard settings**.
1. Go to the **Permissions** tab, and click **Add Permission**.
1. In the **Add Permission For** dialog, select **User** in the first box.
1. In the second box, select the user to grant access to: Luc Masson.
1. In the third box, select **View**.
1. Click **Save**.
1. Click **Save dashboard**.
1. Add a note about giving Luc Masson Viewer permission for the dashboard and then click **Save**.

You've created a new user and given them unique permissions to view a single dashboard within a folder.

#### Check your work

You can repeat these steps to log in as the other users you've created see the differences in the viewer and editor roles.

For this example, you can log in as the user `luc.masson` to see that they can only access the SEO dashboard.

1. Click the profile (avatar) button in the bottom left corner, choose **Sign out**.
1. Enter `luc.masson` as the username.
1. Enter the password you created for Luc.
1. Click **Log in**.
1. In the sidebar, hover your cursor over the **Dashboards** (four squares) icon and then click **Browse**.
1. You'll notice that you won't see the **Analytics** folder in the folder view because we did not give Luc folder permission.
1. Click on the list icon (3 lines) to see the dashboard list.
1. Click on the **SEO dashboard**, there shouldn't be any editing permissions since we assigned Luc the viewer role.

## Summary

In this tutorial, you've configured Grafana for an organization:

- You added users to your organization.
- You created teams to manage permissions for groups of users.
- You configured permissions for folders and dashboard.

### Learn more

- [Organization Roles](/docs/grafana/next/administration/manage-users-and-permissions/about-users-and-permissions/#organization-roles)
- [Permissions Overview](/docs/grafana/latest/administration/manage-users-and-permissions/about-users-and-permissions/#about-users-and-permissions)

# Grafana fundamentals

<!-- INTERACTIVE page intro.md START -->

## Introduction

In this tutorial, you'll learn how to use Grafana to set up a monitoring solution for your application, and:

- Explore metrics and logs
- Build dashboards
- Annotate dashboards
- Set up alert rules

<!-- INTERACTIVE ignore START -->

Alternatively, you can also watch our Grafana for Beginners series where we discuss fundamental concepts to help you get started with Grafana.

<div style="position:relative;padding-bottom:56.25%;height:0;overflow:hidden">
   <iframe src="https://www.youtube.com/embed/videoseries?si=ueLa_QEXz20IWnGt&amp;list=PLDGkOdUX1Ujo27m6qiTPPCpFHVfyKq9jT" style="position:absolute;top:0;left:0;width:100%;height:100%;border:0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen>
   </iframe>
</div>

{{% class "prerequisite-section" %}}

### Prerequisites

- [Docker](https://docs.docker.com/install/)
- [Docker Compose](https://docs.docker.com/compose/) (included in Docker for Desktop for macOS and Windows)
- [Git](https://git-scm.com/)

{{% /class %}}

{{< admonition type="tip" >}}
Alternatively, you can try out this example in our interactive learning environment: [Grafana Fundamentals](https://killercoda.com/grafana-labs/course/grafana/grafana-fundamentals).

It's a fully configured environment with all the dependencies already installed.

![Interactive](/media/docs/grafana/full-stack-ile.png)

Provide feedback, report bugs, and raise issues in the [Grafana Killercoda repository](https://github.com/grafana/killercoda).
{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

<!-- INTERACTIVE page intro.md END -->

<!-- INTERACTIVE page step1.md START -->

## Set up the sample application

This tutorial uses a sample application to demonstrate some of the features in Grafana. To complete the exercises in this tutorial, you need to download the files to your local machine.

In this step, you'll set up the sample application, as well as supporting services, such as [Loki](/oss/loki/).

> **Note:** [Prometheus](https://prometheus.io/), a popular time series database (TSDB), has already been configured as a data source as part of this tutorial.

1. Clone the [github.com/grafana/tutorial-environment](https://github.com/grafana/tutorial-environment) repository.

   ```bash
   git clone https://github.com/grafana/tutorial-environment.git
   ```

1. Change to the directory where you cloned this repository:

   ```bash
   cd tutorial-environment
   ```

1. Make sure Docker is running:

   ```bash
   docker ps
   ```

   No errors means it is running. If you get an error, then start Docker and then run the command again.

1. Start the sample application:

   ```bash
   docker-compose up -d
   ```

   The first time you run `docker-compose up -d`, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

   > **Note:** If you already have Grafana, Loki, or Prometheus running on your system, then you might see errors because the Docker image is trying to use ports that your local installations are already using. Stop the services, then run the command again.

1. Ensure all services are up-and-running:

   ```bash
   docker-compose ps
   ```

   In the **State** column, it should say `Up` for all services.

1. Browse to the sample application on [http://localhost:8081](http://localhost:8081).

### Grafana News

The sample application, Grafana News, lets you post links and vote for the ones you like.

To add a link:

1. In **Title**, enter **Example**.
1. In **URL**, enter **https://example.com**.
1. Click **Submit** to add the link.

   The link appears in the list under the Grafana News heading.

To vote for a link, click the triangle icon next to the name of the link.

<!-- INTERACTIVE page step1.md END -->

<!-- INTERACTIVE page step2.md START -->

## Open Grafana

Grafana is an open source platform for monitoring and observability that lets you visualize and explore the state of your systems.

<!-- INTERACTIVE ignore START -->

1. Open a new tab.
1. Browse to [http://localhost:3000](http://localhost:3000).
<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

1. Browse to [http://localhost:3000](http://localhost:3000).
   {{< /docs/ignore >}}

<!-- INTERACTIVE ignore START -->

{{< admonition type="note" >}}
This demo does not require a login page or credentials. However, if you choose to install Grafana locally, you need to log in and provide credentials. In that case, the default username and password is `admin`.
{{< /admonition >}}

<!-- INTERACTIVE ignore START -->

{{< docs/ignore >}}

> This demo does not require a login page or credentials. However, if you choose to install Grafana locally, you need to log in and provide credentials. In that case, the default username and password is `admin`.
> {{< /docs/ignore >}}

The first thing you see is the Home dashboard, which helps you get started.

In the top left corner, you can see the menu icon. Clicking it opens the _sidebar_, the main menu for navigating Grafana.

<!-- INTERACTIVE page step2.md END -->

<!-- INTERACTIVE page step3.md START -->

## Explore your metrics

Grafana Explore is a workflow for troubleshooting and data exploration. In this step, you'll be using Explore to create ad-hoc queries to understand the metrics exposed by the sample application. Specifically, you'll explore requests received by the sample application.

> Ad-hoc queries are queries that are made interactively, with the purpose of exploring data. An ad-hoc query is commonly followed by another, more specific query.

1. Click the menu icon and, in the sidebar, click **Explore**. A dropdown menu for the list of available data sources is on the upper-left side. The Prometheus data source should already be selected. If not, choose Prometheus.
1. Confirm that you're in code mode by checking the **Builder/Code** toggle at the top right corner of the query panel.
1. In the query editor, where it says _Enter a PromQL query…_, enter `tns_request_duration_seconds_count` and then press Shift + Enter.
   A graph appears.
1. In the top right corner, click the dropdown arrow on the **Run Query** button, and then select **5s**. Grafana runs your query and updates the graph every 5 seconds.

   You just made your first _PromQL_ query! [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) is a powerful query language that lets you select and aggregate time series data stored in Prometheus.

   `tns_request_duration_seconds_count` is a _counter_, a type of metric whose value only ever increases. Rather than visualizing the actual value, you can use counters to calculate the _rate of change_, i.e. how fast the value increases.

1. Add the [`rate`](https://prometheus.io/docs/prometheus/latest/querying/functions/#rate) function to your query to visualize the rate of requests per second. Enter the following in the query editor and then press Shift + Enter.

   ```
   rate(tns_request_duration_seconds_count[5m])
   ```

   Immediately below the graph there's an area where each time series is listed with a colored icon next to it. This area is called the _legend_.

   PromQL lets you group the time series by their labels, using the [`sum`](https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators) aggregation operator.

1. Add the `sum` aggregation operator to your query to group time series by route:

   ```
   sum(rate(tns_request_duration_seconds_count[5m])) by(route)
   ```

1. Go back to the [sample application](http://localhost:8081) and generate some traffic by adding new links, voting, or just refresh the browser.

1. Back in Grafana, in the upper-right corner, click the _time picker_, and select **Last 5 minutes**. By zooming in on the last few minutes, it's easier to see when you receive new data.

Depending on your use case, you might want to group on other labels. Try grouping by other labels, such as `status_code`, by changing the `by(route)` part of the query to `by(status_code)`.

<!-- INTERACTIVE page step3.md END -->

<!-- INTERACTIVE page step4.md START -->

## Add a logging data source

Grafana supports log data sources, like [Loki](/oss/loki/). Just like for metrics, you first need to add your data source to Grafana.

1. Click the menu icon and, in the sidebar, click **Connections** and then **Data sources**.
1. Click **+ Add new data source**.
1. In the list of data sources, click **Loki**.
1. In the URL box, enter `http://loki:3100`
1. Scroll to the bottom of the page and click **Save & Test** to save your changes.

You should see the message "Data source successfully connected." Loki is now available as a data source in Grafana.

<!-- INTERACTIVE page step4.md END -->

<!-- INTERACTIVE page step5.md START -->

## Explore your logs

Grafana Explore not only lets you make ad-hoc queries for metrics, but lets you explore your logs as well.

1. Click the menu icon and, in the sidebar, click **Explore**.
1. In the data source list at the top, select the **Loki** data source.
1. Confirm that you're in code mode by checking the **Builder/Code** toggle at the top right corner of the query panel.
1. Enter the following in the query editor, and then press Shift + Enter:

   ```
   {filename="/var/log/tns-app.log"}
   ```

1. Grafana displays all logs within the log file of the sample application. The height of each bar in the graph encodes the number of logs that were generated at that time.

1. Click and drag across the bars in the graph to filter logs based on time.

Not only does Loki let you filter logs based on labels, but on specific occurrences.

Let's generate an error, and analyze it with Explore.

1. In the [sample application](http://localhost:8081), post a new link without a URL to generate an error in your browser that says `empty url`.
1. Go back to Grafana and enter the following query to filter log lines based on a substring:

   ```
   {filename="/var/log/tns-app.log"} |= "error"
   ```

1. Click the log line that says `level=error msg="empty url"` to see more information about the error.

   > **Note:** If you're in Live mode, clicking logs does not show more information about the error. Instead, stop and exit the live stream, then click the log line there.

Logs are helpful for understanding what went wrong. Later in this tutorial, you'll see how you can correlate logs with metrics from Prometheus to better understand the context of the error.

<!-- INTERACTIVE page step5.md END -->

<!-- INTERACTIVE page step6.md START -->

## Build a dashboard

A _dashboard_ gives you an at-a-glance view of your data and lets you track metrics through different visualizations.

Dashboards consist of _panels_, each representing a part of the story you want your dashboard to tell.

Every panel consists of a _query_ and a _visualization_. The query defines _what_ data you want to display, whereas the visualization defines _how_ the data is displayed.

1. Click the menu icon and, in the sidebar, click **Dashboards**.
1. On the **Dashboards** page, click **New** in top right corner and select **New Dashboard** in the drop-down.
1. Click **+ Add visualization**.
1. In the modal that opens, select the Prometheus data source that you just added.
1. In the **Query** tab below the graph, enter the query from earlier and then press Shift + Enter:

   ```
   sum(rate(tns_request_duration_seconds_count[5m])) by(route)
   ```

1. In the panel editor on the right, under **Panel options**, change the panel title to "Traffic".
1. Click **Apply** in the top-right corner to save the panel and go back to the dashboard view.
1. Click the **Save dashboard** (disk) icon at the top of the dashboard to save your dashboard.
1. Enter a name in the **Dashboard name** field and then click **Save**.

   You should now have a panel added to your dashboard.

   {{< figure src="/media/tutorials/grafana-fundamentals-dashboard.png" alt="A panel in a Grafana dashboard" caption="A panel in a Grafana dashboard" >}}

<!-- INTERACTIVE page step6.md END -->

<!-- INTERACTIVE page step7.md START -->

## Annotate events

When things go bad, it often helps if you understand the context in which the failure occurred. Time of last deploy, system changes, or database migration can offer insight into what might have caused an outage. Annotations allow you to represent such events directly on your graphs.

In the next part of the tutorial, we simulate some common use cases that someone would add annotations for.

1. To manually add an annotation, click anywhere on a graph line to open the data tooltip, then click **Add annotation**.
   You can also press `Ctrl` or `Command` and click anywhere in the graph to open the **Add annotation** dialog box.
   Note: you might need to save the dashboard first.
1. In **Description**, enter **Migrated user database**.
1. Click **Save**.

   Grafana adds your annotation to the graph. Hover your mouse over the base of the annotation to read the text.

Grafana also lets you annotate a time interval, with _region annotations_.

Add a region annotation:

1. Press Ctrl (or Cmd on macOS) and hold, then click and drag across the graph to select an area.
1. In **Description**, enter **Performed load tests**.
1. In **Tags**, enter **testing**.
1. Click **Save**.

### Using annotations to correlate logs with metrics

Manually annotating your dashboard is fine for those single events. For regularly occurring events, such as deploying a new release, Grafana supports querying annotations from one of your data sources. Let's create an annotation using the Loki data source we added earlier.

1. At the top of the dashboard, click the **Dashboard settings** (gear) icon.
1. Go to **Annotations** and click **Add annotation query**.
1. In **Name**, enter **Errors**.
1. In **Data source**, select **Loki**.
1. In **Query**, enter the following query:

   ```
   {filename="/var/log/tns-app.log"} |= "error"
   ```

1. Click **Apply**. Grafana displays the Annotations list, with your new annotation.
1. Click your dashboard name to return to your dashboard.
1. At the top of your dashboard, there is now a toggle to display the results of the newly created annotation query. Press it if it's not already enabled.
1. Click the **Save dashboard** (disk) icon to save the changes.
1. To test the changes, go back to the [sample application](http://localhost:8081), post a new link without a URL to generate an error in your browser that says `empty url`.

The log lines returned by your query are now displayed as annotations in the graph.

{{< figure src="/media/tutorials/annotations-grafana-dashboard.png" alt="A panel in a Grafana dashboard with log queries from Loki displayed as annotations" caption="Displaying log queries from Loki as annotations" >}}

Being able to combine data from multiple data sources in one graph allows you to correlate information from both Prometheus and Loki.

Annotations also work very well alongside alert rules. In the next and final section, we set up an alert rules for our app `grafana.news` and then we trigger it. This provides a quick intro to our new Alerting platform.

<!-- INTERACTIVE page step7.md END -->

<!-- INTERACTIVE page step8.md START -->

## Create a Grafana-managed alert rule

Alert rules allow you to identify problems in your system moments after they occur. By quickly identifying unintended changes in your system, you can minimize disruptions to your services.

Grafana's new alerting platform debuted with Grafana 8. A year later, with Grafana 9, it became the default alerting method. In this step we create a Grafana-managed alert rule. Then we trigger our new alert rule and send a test message to a dummy endpoint.

The most basic alert rule consists of two parts:

1. A _Contact point_ - A Contact point defines how Grafana delivers an alert instance. When the conditions of an _alert rule_ are met, Grafana notifies the contact points, or channels, configured for that alert rule.

   <!-- INTERACTIVE ignore START -->

   {{< admonition type="note" >}}
   An [alert instance](https://grafana.com/docs/grafana/latest/alerting/fundamentals/#alert-instances) is a specific occurrence that matches a condition defined by an alert rule, such as when the rate of requests for a specific route suddenly increases.
   {{< /admonition >}}
   <!-- INTERACTIVE ignore END -->

   {{< docs/ignore >}}

   > An [alert instance](https://grafana.com/docs/grafana/latest/alerting/fundamentals/#alert-instances) is a specific occurrence that matches a condition defined by an alert rule, such as when the rate of requests for a specific route suddenly increases.
   > {{< /docs/ignore >}}

   Some popular channels include:
   - [Email](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/configure-email/)
   - [Webhooks](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier/)
   - [Telegram](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/configure-telegram/)
   - [Slack](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/configure-slack/)
   - [PagerDuty](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/pager-duty/)

1. An _Alert rule_ - An Alert rule defines one or more _conditions_ that Grafana regularly evaluates. When these evaluations meet the rule's criteria, the alert rule is triggered.

To begin, let's set up a webhook contact point. Once we have a usable endpoint, we write an alert rule and trigger a notification.

<!-- INTERACTIVE page step8.md END -->

<!-- INTERACTIVE page step9.md START -->

### Create a contact point for Grafana-managed alert rules

In this step, we set up a new contact point. This contact point uses the _webhooks_ channel. In order to make this work, we also need an endpoint for our webhook channel to receive the alert notification. We can use [Webhook.site](https://webhook.site/) to quickly set up that test endpoint. This way we can make sure that our alert manager is actually sending a notification somewhere.

1. Browse to [Webhook.site](https://webhook.site/).
1. Copy Your unique URL.

Your webhook endpoint is now waiting for the first request.

Next, let's configure a Contact Point in Grafana's Alerting UI to send notifications to our webhook endpoint.

1. Return to Grafana. In Grafana's sidebar, hover over the **Alerting** (bell) icon and then click **Manage Contact points**.
1. Click **+ Add contact point**.
1. In **Name**, write **Webhook**.
1. In **Integration**, choose **Webhook**.
1. In **URL**, paste the endpoint to your webhook endpoint.

1. Click **Test**, and then click **Send test notification** to send a test alert notification to your webhook endpoint.
1. Navigate back to the webhook endpoint you created earlier. On the left side, there's now a `POST /` entry. Click it to see what information Grafana sent.
1. Return to Grafana and click **Save contact point**.

We have now created a dummy webhook endpoint and created a new Alerting Contact Point in Grafana. Now we can create an alert rule and link it to this new channel.

<!-- INTERACTIVE page step9.md END -->

<!-- INTERACTIVE page step10.md START -->

### Add an alert rule to Grafana

Now that Grafana knows how to notify us, it's time to set up an alert rule:

1. In Grafana's sidebar, hover over the **Alerting** (bell) icon and then click **Alert rules**.

   In this tutorial, we use the advanced options for Grafana-managed alert rule creation. The advanced options let us define queries, expressions (used to manipulate the data), and the condition that must be met for the alert to be triggered (the default condition is the threshold).

1. Click **+ New alert rule**.
1. For **Section 1**, name the rule `fundamentals-test`.
1. For **Section 2**, toggle the **Advanced options** button.
1. Find the **query A** box, and choose your Prometheus data source.
1. Enter the same Prometheus query that we used in our earlier panel:

   ```
   sum(rate(tns_request_duration_seconds_count[5m])) by(route)
   ```

1. Keep expressions **B** and **C** as they are. These expressions (Reduce and Threshold, respectively) are included by default when creating a new rule. Enter `0.2` as threshold value. You can read more about queries and conditions [here](/docs/grafana/latest/alerting/fundamentals/alert-rules/queries-conditions/#expression-queries).
1. Scroll down to bottom of section #2 and click the **Preview** button. You should see some data returned.
1. In **Section 3**, in Folder, create a new folder, by clicking `New folder` and typing a name for the folder. This folder contains our alert rules. For example: `fundamentals`. Then, click `create`.
1. In the Evaluation group, repeat the above step to create a new one. Name it `fundamentals` too.
1. Choose an Evaluation interval (how often the alert rule are evaluated). For example, every `10s` (10 seconds).
1. Set the pending period. This is the time that a condition has to be met until the alert instance enters in Firing state and a notification is sent. Enter `0s`. For the purposes of this tutorial, the evaluation interval is intentionally short. This makes it easier to test. This setting makes Grafana wait until an alert instance has fired for a given time before Grafana sends the notification.
1. In **Section 4**, choose **Webhook** as the **Contact point**.
1. Click **Save rule and exit** at the top of the page.

### Trigger a Grafana-managed alert rule

We have now configured an alert rule and a contact point. Now let's see if we can trigger a Grafana-managed alert rule by generating some traffic on our sample application.

1. Browse to [localhost:8081](http://localhost:8081).
1. Add a new title and URL, repeatedly click the vote button, or refresh the page to generate a traffic spike.

Once the query `sum(rate(tns_request_duration_seconds_count[5m])) by(route)` returns a value greater than `0.2` Grafana triggers our alert rule. Browse to the webhook endpoint we created earlier and find the sent Grafana alert notification with details and metadata.

<!-- INTERACTIVE ignore START -->

{{< admonition type="note" >}}
The alert rule may be triggered by the `/metrics` endpoint which is frequently accessed by Grafana when pulling metrics from the application. If this happens, you can increase the **Threshold** value in **Section 2** for testing purposes.
{{< /admonition >}}

<!-- INTERACTIVE ignore START -->

{{< docs/ignore >}}

> The alert rule may be triggered by the `/metrics` endpoint which is frequently accessed by Grafana when pulling metrics from the application. If this happens, you can increase the **Threshold** value in **Section 2** for testing purposes.
> {{< /docs/ignore >}}

### Display Grafana-managed alert rules to your dashboard

In most cases, it's also valuable to display Grafana alert instances as annotations to your dashboard. Check out the video tutorial below to learn how to display alerting to your dashboard.

<!-- INTERACTIVE ignore START -->

{{< youtube id="ClLp-iSoaSY" >}}

<!-- INTERACTIVE ignore END -->

Let's see how we can configure this.

1. In Grafana's sidebar, hover over the **Alerting** (bell) icon and then click **Alert rules**.
1. Expand the `fundamentals > fundamentals` folder to view our created alert rule.
1. Click the **Edit** icon and scroll down to **Section 5**.
1. Click the **Link dashboard and panel** button and select the dashboard and panel to which you want the alert instance to be added as an annotation.
1. Click **Confirm** and **Save rule and exit** to save all the changes.
1. In Grafana's sidebar, navigate to the dashboard by clicking **Dashboards** and selecting the dashboard you created.
1. To test the changes, follow the steps listed to [trigger a Grafana-managed alert rule](#trigger-a-grafana-managed-alert).

   You should now see a red, broken heart icon beside the panel name, indicating that the alert rule has been triggered. An annotation for the alert instance, represented as a vertical red line, is also displayed.

   {{< figure src="/media/tutorials/grafana-alert-on-dashboard.png" alt="A panel in a Grafana dashboard with alerting and annotations configured" caption="Displaying Grafana-managed alert rules on a dashboard" >}}

<!-- INTERACTIVE ignore START -->

{{< admonition type="tip" >}}

In [Get started with Grafana Alerting - Part 2](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.

{{< /admonition >}}

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

> In [Get started with Grafana Alerting - Part 2](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.

{{< /docs/ignore >}}

<!-- INTERACTIVE page step10.md END -->

<!-- INTERACTIVE page finish.md START -->

## Summary

<!-- INTERACTIVE ignore START -->

In this tutorial you learned about fundamental features of Grafana. To do so, we ran several Docker containers on your local machine. When you are ready to clean up this local tutorial environment, run the following command:

```
docker-compose down -v
```

<!-- INTERACTIVE ignore END -->

{{< docs/ignore >}}

In this tutorial you learned about fundamental features of Grafana.

{{< /docs/ignore >}}

### Learn more

Check out the links below to continue your learning journey with Grafana's LGTM stack.

- [Prometheus](/docs/grafana/latest/features/datasources/prometheus/)
- [Loki](/docs/grafana/latest/features/datasources/loki/)
- [Explore](/docs/grafana/latest/explore/)
- [Alerting Overview](/docs/grafana/latest/alerting/)
- [Alert rules](/docs/grafana/latest/alerting/create-alerts/)
- [Contact points](/docs/grafana/latest/alerting/fundamentals/contact-points/)

<!-- INTERACTIVE page finish.md END -->

# Use IIS with URL Rewrite as a reverse proxy

If you want Grafana to be a subpath/subfolder under a website in IIS then the Application Request Routing (ARR) and URL Rewrite modules for ISS can be used to support this.

Example:

- Parent site: http://yourdomain.com:8080
- Grafana: http://localhost:3000

Grafana as a subpath: http://yourdomain.com:8080/grafana

Other Examples:

- If the application is only served on the local server, the parent site can also look like http://localhost:8080.
- If your domain is served using https on port 443, and thus the port is not normally entered in the address of your site, then the need to specify a port for the parent site in the configuration steps below can be eliminated.

## Setup

Install the URL Rewrite module for IIS.

- Download and install the URL Rewrite module for IIS: https://www.iis.net/downloads/microsoft/url-rewrite

You will also need the Application Request Routing (ARR) module for IIS for proxy forwarding

- Download and install ARR module for IIS: https://www.iis.net/downloads/microsoft/application-request-routing

## Grafana Config

The Grafana config can be set by creating a file named/editing the existing file named `custom.ini` in the `conf` subdirectory of your Grafana installation. See the [installation instructions](/docs/grafana/<GRAFANA_VERSION>/installation/windows/#configure) for more details.

Using the example from above, if the subpath is `grafana` (you can set this to whatever is required) and the parent site is `yourdomain.com:8080`, then you would add this to the `custom.ini` config file:

```bash
[server]
domain = yourdomain.com:8080
root_url = %(protocol)s://%(domain)s/grafana/
```

Restart the Grafana server after changing the config file.

Configured address to serve Grafana: http://yourdomain.com:8080/grafana

---

If you already have a subpath on your domain, configure it as follows:

- Your Parent Site Address: http://yourdomain.com/existingsubpath

```bash
[server]
domain = yourdomain.com/existingsubpath
root_url = %(protocol)s://%(domain)s/grafana/
```

Restart the Grafana server after changing the config file.

Configured address to serve Grafana: http://yourdomain.com/existingsubpath/grafana

## IIS Config

### Step 1: Forward Proxy

1. Open the IIS Manager and click on the server
2. In the admin console for the server, double click on the Application Request Routing option:
3. Click the `Server Proxy Settings` action on the right-hand pane
4. Select the `Enable proxy` checkbox so that it is enabled
5. Click `Apply` and proceed with the URL Rewriting configuration

**Note:** If you don't enable the Forward Proxy, you will most likely get 404 Not Found if you only apply the URL Rewrite rule

### Step 2: URL Rewriting

1. In the IIS Manager, click on the website that grafana will run under. For example, select the website that is bound to the http://yourdomain.com domain.
2. In the admin console for this website, double click on the URL Rewrite option:

{{< figure src="/static/img/docs/tutorials/IIS_admin_console.png"  max-width="800px" >}}

3. Click on the `Add Rule(s)...` action
4. Choose the Blank Rule template for an Inbound Rule

{{< figure src="/static/img/docs/tutorials/IIS_add_inbound_rule.png"  max-width="800px" >}}

5. Create an Inbound Rule for the website with the following settings:

- pattern: `grafana(/)?(.*)` (if you have customised the subpath that will be used, use that instead of `grafana`)
- check the `Ignore case` checkbox
- rewrite URL set to `http://localhost:3000/{R:2}`
- check the `Append query string` checkbox
- check the `Stop processing of subsequent rules` checkbox

{{< figure src="/static/img/docs/tutorials/IIS_url_rewrite.png"  max-width="800px" >}}

6. If your version of Grafana is greater than 8.3.5, you also need to configure the reverse proxy to preserve host headers.

- This can be achieved by configuring the IIS config file by running this in a cmd prompt
  `%windir%\system32\inetsrv\appcmd.exe set config -section:system.webServer/proxy -preserveHostHeader:true /commit:apphost`
- More information here https://github.com/grafana/grafana/issues/45261

Finally, navigate to `http://yourdomain.com:8080/grafana` and you should come to the Grafana login page.

## Troubleshooting

### 404 error

When navigating to the Grafana URL (`http://yourdomain.com:8080/grafana`) and a `HTTP Error 404.0 - Not Found` error is returned, then either:

- The pattern for the Inbound Rule is incorrect. Edit the rule, click on the `Test pattern...` button, test the part of the URL after `http://yourdomain.com:8080/` and make sure it matches. For `grafana/login` the test should return 3 capture groups: {R:0}: `grafana` {R:1}: `/` and {R:2}: `login`.
- The `root_url` setting in the Grafana config file does not match the parent URL with subpath.

### Grafana Website only shows text with no images or css

{{< figure src="/static/img/docs/tutorials/IIS_proxy_error.png"  max-width="800px" >}}

1. The `root_url` setting in the Grafana config file does not match the parent URL with subpath. This could happen if the root_url is commented out by mistake (`;` is used for commenting out a line in .ini files):

   `; root_url = %(protocol)s://%(domain)s/grafana/`

2. or if the subpath in the `root_url` setting does not match the subpath used in the pattern in the Inbound Rule in IIS:

   `root_url = %(protocol)s://%(domain)s/grafana/`

   pattern in Inbound Rule: `wrongsubpath(/)?(.*)`

3. or if the Rewrite URL in the Inbound Rule is incorrect.

   The Rewrite URL should not include the subpath.

   The Rewrite URL should contain the capture group from the pattern matching that returns the part of the URL after the subpath. The pattern used above returns three capture groups and the third one {R:2} returns the part of the URL after `http://yourdomain.com:8080/grafana/`.

### You see an 'Error updating options: origin not allowed' error

- Ensure you have undertaken step 6 above, to configure IIS to preserve host headers by edit IIS config by running this in cmd prompt:
  `%windir%\system32\inetsrv\appcmd.exe set config -section:system.webServer/proxy -preserveHostHeader:true /commit:apphost`

# Install Grafana on Raspberry Pi

The Raspberry Pi is a tiny, affordable, yet capable computer that can run a range of different applications. Even Grafana!

Many people are running Grafana on Raspberry Pi as a way to monitor their home, for things like indoor temperature, humidity, or energy usage.

In this tutorial, you'll:

- Set up a Raspberry Pi using a version of Raspberry Pi OS (previously called "Raspbian") that does not require you to connect a keyboard or monitor (this is often called "headless").
- Install Grafana on your Raspberry Pi.

{{% class "prerequisite-section" %}}

### Prerequisites

- Raspberry Pi
- SD card
  {{% /class %}}

## Set up your Raspberry Pi

Before we can install Grafana, you first need to set up your Raspberry Pi.

For this tutorial, you'll configure your Raspberry Pi to be _headless_. This means you don't need to connect a monitor, keyboard, or mouse to your Raspberry Pi. All configuration is done from your regular computer.

#### Download and install Raspberry Pi Imager

Before we get started, you need to download and install the [Raspberry Pi Imager](https://www.raspberrypi.org/software/).

We'll use the Raspberry Pi Imager to flash the operating system image to the SD card. You download the imager directly from the official Raspberry Pi website and it's available for Ubuntu Linux, macOS, and Windows.

Follow the directions on the website to download and install the imager.

#### Install Raspberry Pi OS

Now it's time to install Raspberry Pi OS.

1. Insert the SD card into your regular computer from which you plan to install Raspberry Pi OS.
1. Run the Raspberry Pi Imager that you downloaded and installed.
1. To select an operating system, click **Choose OS** in the imager. You will be shown a list of available options.
1. From the list, select **Raspberry Pi OS (other)** and then select **Raspberry Pi OS Lite**, which is a Debian-based operating system for the Raspberry Pi. Since you're going to run a headless Raspberry Pi, you won't need the desktop dependencies.
1. To select where you want to put the operating system image, click **Choose Storage** in the imager and then select the SD card you already inserted into your computer.
1. The final step in the imager to click **Write**. When you do, the imager will write the Raspberry Pi OS Lite image to the SD card and verify that it has been written correctly.
1. Eject the SD card from your computer, and insert it again.

While you _could_ fire up the Raspberry Pi now, we don't yet have any way of accessing it.

1. Create an empty file called `ssh` in the boot directory. This enables SSH so that you can log in remotely.

   The next step is only required if you want the Raspberry Pi to connect to your wireless network. Otherwise, connect the it to your network by using a network cable.

1. **(Optional)** Create a file called `wpa_supplicant.conf` in the boot directory:

   ```bash
   ctrl_interface=/var/run/wpa_supplicant
   update_config=1
   country=<Insert 2 letter ISO 3166-1 country code here>

   network={
    ssid="<Name of your WiFi>"
    psk="<Password for your WiFi>"
   }
   ```

All the necessary files are now on the SD card. Let's start up the Raspberry Pi.

1. Eject the SD card and insert it into the SD card slot on the Raspberry Pi.
1. Connect the power cable and make sure the LED lights are on.
1. Find the IP address of the Raspberry Pi. Usually you can find the address in the control panel for your WiFi router.

#### Connect remotely via SSH

1. Open up your terminal and enter the following command:

   ```bash
   ssh pi@<ip address>
   ```

1. SSH warns you that the authenticity of the host can't be established. Type "yes" to continue connecting.
1. When asked for a password, enter the default password: `raspberry`.
1. Once you're logged in, change the default password:

   ```bash
   passwd
   ```

Congratulations! You've now got a tiny Linux machine running that you can hide in a closet and access from your normal workstation.

## Install Grafana

Now that you've got the Raspberry Pi up and running, the next step is to install Grafana.

1. Add the APT key used to authenticate packages:

   ```bash
   sudo mkdir -p /etc/apt/keyrings/
   wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
   ```

1. Add the Grafana APT repository:

   ```bash
   echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
   ```

1. Install Grafana:

   ```bash
   sudo apt-get update
   sudo apt-get install -y grafana
   ```

Grafana is now installed, but not yet running. To make sure Grafana starts up even if the Raspberry Pi is restarted, we need to enable and start the Grafana Systemctl service.

1. Enable the Grafana server:

   ```bash
   sudo /bin/systemctl enable grafana-server
   ```

1. Start the Grafana server:

   ```bash
   sudo /bin/systemctl start grafana-server
   ```

   Grafana is now running on the machine and is accessible from any device on the local network.

1. Open a browser and go to `http://<ip address>:3000`, where the IP address is the address that you used to connect to the Raspberry Pi earlier. You're greeted with the Grafana login page.
1. Log in to Grafana with the default username `admin`, and the default password `admin`.
1. Change the password for the admin user when asked.

Congratulations, Grafana is now running on your Raspberry Pi. If the Raspberry Pi is ever restarted or turned off, Grafana will start up whenever the machine regains power.

## Summary

If you want to use Grafana without having to go through a full installation process, check out [Grafana Cloud](/products/cloud/), which is designed to get users up and running quickly and easily. Grafana Cloud offers a forever free plan that is genuinely useful for hobbyists, testing, and small teams.

### Learn more

- [Raspberry Pi Documentation](https://www.raspberrypi.org/documentation/)

# Integrate Hubot with Grafana

Grafana 2.0 shipped with a great feature that enables it to render any graph or panel to a PNG image.

No matter what data source you are using, the PNG image of the Graph will look the same as it does in your browser.

This guide shows you how to install and configure the [Hubot-Grafana](https://github.com/stephenyeargin/hubot-grafana) plugin. This plugin allows you to tell Hubot to render any dashboard or graph right from a channel in Slack, Basecamp, or any other supported Hubot adapter. The bot will respond with an image of the graph and a link that will take you to the graph.

{{< figure src="/static/img/docs/tutorials/hubot_grafana.png"  max-width="800px" >}}

## What is Hubot?

[Hubot](https://hubot.github.com/) is an universal and extensible chat bot that can be used with many chat services and has a huge library of third party plugins that allow you to automate anything from your chat rooms.

## Install Hubot

Hubot is very easy to install and host. If you do not already have a bot up and running please read the official [Getting Started With Hubot](https://hubot.github.com/docs/) guide.

## Install the Hubot-Grafana script

In your Hubot project repo install the Grafana plugin using `npm`:

```bash
npm install hubot-grafana --save
```

Edit the file external-scripts.json, and add hubot-grafana to the list of plugins.

```json
["hubot-pugme", "hubot-shipit", "hubot-grafana"]
```

## Configure

The Hubot-Grafana plugin requires two environment variables to be set in order to work properly.

```bash
export HUBOT_GRAFANA_HOST=https://play.grafana.org
export HUBOT_GRAFANA_API_KEY=abcd01234deadbeef01234
```

There are [additional environment variables](https://github.com/stephenyeargin/hubot-grafana?tab=readme-ov-file#general-settings) that you can set to control the appearance of the graphs.

### Grafana server side rendering

The hubot plugin will take advantage of the Grafana server side rendering feature that can render any panel on the server using phantomjs. Grafana ships with a phantomjs binary (Linux only).

To verify that this feature works try the `Direct link to rendered image` link in the panel share dialog. If you do not get an image when opening this link verify that the required font packages are installed for phantomjs to work.

### Grafana API Key

{{< figure src="/static/img/docs/v2/orgdropdown_api_keys.png" max-width="150px" class="docs-image--right">}}

You need to set the environment variable `HUBOT_GRAFANA_API_KEY` to a Grafana API Key. You can add these from the API Keys page which you find in the Organization dropdown.

### Image uploading

There are several approaches to uploading the rendered graphs. If you are using Slack, Rocket.Chat, or Telegram, the adapter's native uploader will take care of sending it through their respective API. If your Hubot is hosted on a platform that doesn't support uploads (such as IRC), you can use the [built-in S3 uploader](https://github.com/stephenyeargin/hubot-grafana/wiki/Amazon-S3-Image-Hosting). Note if you configure S3, it will not use the adapter's upload features.

## Hubot commands

- `hubot graf list`
  - Lists the available dashboards
- `hubot graf db graphite-carbon-metrics`
  - Graph all panels in the dashboard
- `hubot graf db graphite-carbon-metrics:3`
  - Graph only panel with id 3 of a particular dashboard
- `hubot graf db graphite-carbon-metrics:cpu`
  - Graph only the panels containing "cpu" (case insensitive) in the title
- `hubot graf db graphite-carbon-metrics now-12hr`
  - Get a dashboard with a window of 12 hours ago to now
- `hubot graf db graphite-carbon-metrics now-24hr now-12hr`
  - Get a dashboard with a window of 24 hours ago to 12 hours ago
- `hubot graf db graphite-carbon-metrics:3 now-8d now-1d`
  - Get only the third panel of a particular dashboard with a window of 8 days ago to yesterday
- `hubot graf db graphite-carbon-metrics host=carbon-a`
  - Get a templated dashboard with the `$host` parameter set to `carbon-a`

## Aliases

Some of the hubot commands above can lengthy and you might have to remember the dashboard slug (url id). If you have a few favorite graphs you want to be able check up on often (let's say from your mobile) you can create hubot command aliases with the hubot script `hubot-alias`.

Install it:

```bash
npm i --save hubot-alias
```

Now add `hubot-alias` to the list of plugins in `external-scripts.json` and restart hubot.

Now you can add an alias like this:

- `hubot alias graf-lb=graf db loadbalancers:2 now-20m`

{{< figure src="/static/img/docs/tutorials/hubot_grafana2.png"  max-width="800px" >}}

## Summary

Grafana is going to ship with integrated Slack and Hipchat features some day but you do not have to wait for that. Grafana 2 shipped with a very clever server side rendering feature that can render any panel to a png using phantomjs. The hubot plugin for Grafana is something you can install and use today!

# Provision dashboards and data sources

Learn how you can reuse dashboards and data sources across multiple teams by provisioning Grafana from version-controlled configuration files.

In this tutorial, you'll:

- Provision dashboards.
- Provision data sources.

{{% class "prerequisite-section" %}}

### Prerequisites

- Grafana 7.0
- Administrator privileges on the system you are doing the tutorial on
  {{% /class %}}

## Configuration as code

Configuration as code is the practice of storing the configuration of your system as a set of version controlled, human-readable configuration files, rather than in a database. These configuration files can be reused across environments to avoid duplicated resources.

As the number of dashboards and data sources grows within your organization, manually managing changes can become tedious and error-prone. Encouraging reuse becomes important to avoid multiple teams redesigning the same dashboards.

Grafana supports configuration as code through _provisioning_. The resources that currently supports provisioning are:

- [Dashboards](/docs/grafana/latest/administration/provisioning/#dashboards)
- [Data sources](/docs/grafana/latest/administration/provisioning/#datasources)

## Set the provisioning directory

Before you can start provisioning resources, Grafana needs to know where to find the _provisioning directory_. The provisioning directory contains configuration files that are applied whenever Grafana starts and continuously updated while running.

By default, Grafana looks for a provisioning directory in the configuration directory (grafana > conf) on the system where Grafana is installed. However, if you are a Grafana Administrator, then you might want to place the config files in a shared resource like a network folder, so you would need to change the path to the provisioning directory.

You can set a different path by setting the `paths.provisioning` property in the main config file:

```ini
[paths]
provisioning = <path to config files>
```

For more information about configuration files, refer to [Configuration](/docs/grafana/latest/installation/configuration/) in the [Grafana documentation](/docs/grafana/latest/).

The provisioning directory assumes the following structure:

```
provisioning/
  datasources/
    <yaml files>
  dashboards/
    <yaml files>
```

Next, we'll look at how to provision a data source.

## Provision a data source

Each data source provisioning config file contains a _manifest_ that specifies the desired state of a set of provisioned data sources.

At startup, Grafana loads the configuration files and provisions the data sources listed in the manifests.

Let's configure a [TestData](/docs/grafana/latest/features/datasources/testdata/) data source that you can use for your dashboards.

#### Create a data source manifest

1. In the `provisioning/datasources/` directory, create a file called `default.yaml` with the following content:

   ```yaml
   apiVersion: 1

   datasources:
     - name: TestData
       type: testdata
   ```

1. Restart Grafana to load the new changes.
1. In the sidebar, hover the cursor over the **Configuration** (gear) icon and click **Data Sources**. TestData appears in the list of data sources.

> The configuration options can vary between different types of data sources. For more information on how to configure a specific data source, refer to [Data sources](/docs/grafana/latest/administration/provisioning/#datasources).

## Provision a dashboard

Each dashboard config file contains a manifest that specifies the desired state of a set of _dashboard providers_.

A dashboard provider tells Grafana where to find the dashboard definitions and where to put them.

Grafana regularly checks for changes to the dashboard definitions (by default every 10 seconds).

Let's define a dashboard provider so that Grafana knows where to find the dashboards we want to provision.

#### Define a dashboard provider

In the `provisioning/dashboards/` directory, create a file called `default.yaml` with the following content:

```yaml
apiVersion: 1

providers:
  - name: Default # A uniquely identifiable name for the provider
    folder: Services # The folder where to place the dashboards
    type: file
    options:
      path:
        <path to dashboard definitions>
        # Default path for Windows: C:/Program Files/GrafanaLabs/grafana/public/dashboards
        # Default path for Linux is: /var/lib/grafana/dashboards
```

For more information on how to configure dashboard providers, refer to [Dashboards](/docs/grafana/latest/administration/provisioning/#dashboards).

#### Create a dashboard definition

1. In the dashboard definitions directory you specified in the dashboard provider, i.e. `options.path`, create a file called `cluster.json` with the following content:

   ```json
   {
     "__inputs": [],
     "__requires": [],
     "annotations": {
       "list": []
     },
     "editable": false,
     "gnetId": null,
     "graphTooltip": 0,
     "hideControls": false,
     "id": null,
     "links": [],
     "panels": [
       {
         "aliasColors": {},
         "bars": false,
         "dashLength": 10,
         "dashes": false,
         "datasource": "TestData",
         "fill": 1,
         "gridPos": {
           "h": 8,
           "w": 24,
           "x": 0,
           "y": 0
         },
         "id": 2,
         "legend": {
           "alignAsTable": false,
           "avg": false,
           "current": false,
           "max": false,
           "min": false,
           "rightSide": false,
           "show": true,
           "total": false,
           "values": false
         },
         "lines": true,
         "linewidth": 1,
         "links": [],
         "nullPointMode": "null",
         "percentage": false,
         "pointradius": 5,
         "points": false,
         "renderer": "flot",
         "repeat": null,
         "seriesOverrides": [],
         "spaceLength": 10,
         "stack": false,
         "steppedLine": false,
         "targets": [],
         "thresholds": [],
         "timeFrom": null,
         "timeShift": null,
         "title": "CPU Usage",
         "tooltip": {
           "shared": true,
           "sort": 0,
           "value_type": "individual"
         },
         "type": "graph",
         "xaxis": {
           "buckets": null,
           "mode": "time",
           "name": null,
           "show": true,
           "values": []
         },
         "yaxes": [
           {
             "format": "short",
             "label": null,
             "logBase": 1,
             "max": null,
             "min": null,
             "show": true
           },
           {
             "format": "short",
             "label": null,
             "logBase": 1,
             "max": null,
             "min": null,
             "show": true
           }
         ]
       }
     ],
     "refresh": "",
     "rows": [],
     "schemaVersion": 16,
     "tags": ["kubernetes"],
     "templating": {
       "list": []
     },
     "time": {
       "from": "now-6h",
       "to": "now"
     },
     "timepicker": {},
     "timezone": "browser",
     "title": "Cluster",
     "version": 0
   }
   ```

1. Restart Grafana to provision the new dashboard or wait 10 seconds for Grafana to automatically create the dashboard.
1. In the sidebar, hover the cursor over **Dashboards** (squares) icon, and then click **Manage**. The dashboard appears in a **Services** folder.

> If you don't specify an `id` in the dashboard definition, then Grafana assigns one during provisioning. You can set the `id` yourself if you want to reference the dashboard from other dashboards. Be careful to not use the same `id` for multiple dashboards, as this will cause a conflict.

## Summary

In this tutorial you learned how you to reuse dashboards and data sources across multiple teams by provisioning Grafana from version-controlled configuration files.

Dashboard definitions can get unwieldy as more panels and configurations are added to them. There are a number of open source tools available to make it easier to manage dashboard definitions:

- [grafana-dash-gen](https://github.com/uber/grafana-dash-gen) (Javascript)
- [grafanalib](https://github.com/weaveworks/grafanalib) (Python)
- [grafonnet-lib](https://github.com/grafana/grafonnet-lib) (Jsonnet)
- [grafyaml](https://docs.openstack.org/infra/grafyaml/) (YAML)

### Learn more

- [Provisioning Grafana](/docs/grafana/latest/administration/provisioning/)

# Run Grafana behind a reverse proxy

In this tutorial, you'll configure Grafana to run behind a reverse proxy.

When running Grafana behind a proxy, you need to configure the domain name to let Grafana know how to render links and redirects correctly.

- In the Grafana configuration file, change `server.domain` to the domain name you'll be using:

```bash
[server]
domain = example.com
```

- Restart Grafana for the changes to take effect.

## Configure reverse proxy

### Configure nginx

[nginx](https://www.nginx.com) is a high performance load balancer, web server, and reverse proxy.

- In your nginx configuration file inside the `http` section, add the following:

```nginx
# This is required to proxy Grafana Live WebSocket connections.
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

upstream grafana {
  server localhost:3000;
}

server {
  listen 80;
  root /usr/share/nginx/html;
  index index.html index.htm;

  location / {
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }

  # Proxy Grafana Live WebSocket connections.
  location /api/live/ {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }
}
```

- Reload the nginx configuration.
- Navigate to port 80 on the machine nginx is running on.
  You're greeted by the Grafana login page.

For Grafana Live which uses WebSocket connections you may have to raise the nginx value for [`worker_connections`](https://nginx.org/en/docs/ngx_core_module.html#worker_connections) option which is `512` by default. The default value limits the number of possible concurrent connections with Grafana Live.

Also, be aware that the preceding configuration only works when the `proxy_pass` value for `location /` is a literal string.
If you want to use a variable here, you must instead use [a rewrite rule](https://www.nginx.com/blog/creating-nginx-rewrite-rules/).
For more information, refer to [the GitHub issue #18299](https://github.com/grafana/grafana/issues/18299).

To configure nginx to serve Grafana under a _sub path_, update the `location` block:

```nginx
# This is required to proxy Grafana Live WebSocket connections.
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

upstream grafana {
  server localhost:3000;
}

server {
  listen 80;
  root /usr/share/nginx/www;
  index index.html index.htm;

  location /grafana/ {
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }

  # Proxy Grafana Live WebSocket connections.
  location /grafana/api/live/ {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }
}
```

Add a rewrite rule to each location block:

```
 rewrite  ^/grafana/(.*)  /$1 break;
```

{{< admonition type="note" >}}
If nginx is performing TLS termination, then you must set the `root_url` and `protocol` configuration accordingly.
If you're serving Grafana from `https://example.com/grafana/` then the `root_url` is `https://example.com/grafana/` or `https://%(domain)s/grafana/` with the corresponding `domain` configuration value set to `example.com` in the `server` section of the Grafana configuration file.
Set `protocol` to `http`.
{{< /admonition >}}

### Configure HAProxy

To configure HAProxy to serve Grafana under a _sub path_:

```bash
frontend http-in
  bind *:80
  use_backend grafana_backend if { path /grafana } or { path_beg /grafana/ }

backend grafana_backend
  server grafana localhost:3000
  # Requires haproxy >= 1.6
  http-request set-path %[path,regsub(^/grafana/?,/)]
  # Works for haproxy < 1.6
  # reqrep ^([^\ ]*\ /)grafana[/]?(.*) \1\2

  server grafana localhost:3000
```

### Configure IIS

{{< admonition type="note" >}}
To use IIS, you must have the URL Rewrite module installed.
{{< /admonition >}}

To configure IIS to serve Grafana under a _sub path_, create an `Inbound Rule` for the parent website in **IIS Manager** with the following settings:

- pattern: `grafana(/)?(.*)`
- check the `Ignore case` checkbox
- rewrite URL set to `http://localhost:3000/{R:2}`
- check the `Append query string` checkbox
- check the `Stop processing of subsequent rules` checkbox

This is the rewrite rule that's generated in the `web.config`:

```xml
  <rewrite>
      <rules>
          <rule name="Grafana" enabled="true" stopProcessing="true">
              <match url="grafana(/)?(.*)" />
              <action type="Rewrite" url="http://localhost:3000/{R:2}" logRewrittenUrl="false" />
          </rule>
      </rules>
  </rewrite>
```

For more detailed instruction, refer to the [tutorial on IIS URL Rewrites](/tutorials/iis/).

### Configure Apache

To use Apache as a proxy, ensure its proper installation and configuration.

1.  Ensure that the Apache proxy module [`mod_proxy`](https://httpd.apache.org/docs/current/mod/mod_proxy.html) is installed and enabled. To enable, run the following commands:

```bash
a2enmod proxy
a2enmod proxy_http
```

2. To configure the proxy, edit the site configuration file. To do so, inside the `<VirtualHost>` section, add the following code:

```bash
  ProxyPreserveHost on
  ProxyPass / http://your_grafana_server:3000
  ProxyPassReverse / http://your_grafana_server:3000
```

3. Finally, restart Apache for the settings to take effect.

After you've restarted, navigate to your Apache server on port 80 and you will be redirected to Grafana.

To configure Grafana hosted in a sub path, replace the sub path with the following code (assuming your Grafana instance is on the sub path `your_path`):

```bash
  ProxyPreserveHost on
  ProxyPass /your_path http://your_grafana_server:3000
  ProxyPassReverse /your_path http://your_grafana_server:3000
  ProxyPass / http://your_grafana_server:3000/your_path
  ProxyPassReverse / http://192.168.250.5:3000/your_path
```

Note that the lines containing `your_path` _must_ come before the lines referencing root path (`/`) in order for this to work correctly.

### Configure Traefik

[Traefik](https://traefik.io/traefik/) Cloud Native application proxy.

Using the Docker provider and the following labels configures the router and service for a domain or subdomain routing.

```yaml
labels:
  traefik.http.routers.grafana.rule: Host(`grafana.example.com`)
  traefik.http.services.grafana.loadbalancer.server.port: 3000
```

To deploy on a _sub path_:

```yaml
labels:
  traefik.http.routers.grafana.rule: Host(`example.com`) && PathPrefix(`/grafana`)
  traefik.http.services.grafana.loadbalancer.server.port: 3000
```

Examples using the file provider.

```yaml
http:
  routers:
    grafana:
      rule: Host(`grafana.example.com`)
      service: grafana
  services:
    grafana:
      loadBalancer:
        servers:
          - url: http://192.168.30.10:3000
```

```yaml
http:
  routers:
    grafana:
      rule: Host(`example.com`) && PathPrefix(`/grafana`)
      service: grafana
  services:
    grafana:
      loadBalancer:
        servers:
          - url: http://192.168.30.10:3000
```

## Alternative for serving Grafana under a sub path

{{< admonition type="note" >}}
You only need this if you don't handle the sub path serving via your reverse proxy configuration.
{{< /admonition >}}

If you don't want or can't use the reverse proxy to handle serving Grafana from a _sub path_, you can set the configuration variable `serve_from_sub_path` to `true`.

1. Include the sub path at the end of the `root_url`.
1. Set `serve_from_sub_path` to `true`:

```bash
[server]
domain = example.com
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/
serve_from_sub_path = true
```

# Stream metrics from Telegraf to Grafana

Grafana v8 introduced streaming capabilities – a way to push data to UI panels in near real-time. In this tutorial we show how Grafana real-time streaming capabilities can be used together with Telegraf to instantly display system measurements.

In this tutorial, you'll:

- Setup Telegraf and output measurements directly to Grafana time-series panel in near real-time

{{% class "prerequisite-section" %}}

#### Prerequisites

- Grafana 8.0+
- Telegraf
  {{% /class %}}

## Run Grafana and create admin token

1. Run Grafana following [installation instructions](/docs/grafana/latest/installation/) for your operating system.
1. Log in and go to Configuration -> API Keys.
1. Press "Add API key" button and create a new API token with **Admin** role.

## Configure and run Telegraf

Telegraf is a plugin-driven server agent for collecting and sending metrics and events from databases, systems, and IoT sensors.

You can install it following [official installation instructions](https://docs.influxdata.com/telegraf/latest/introduction/installation/).

In this tutorial we will be using Telegraf HTTP output plugin to send metrics in Influx format to Grafana. We can use a configuration like this:

```
[agent]
  interval = "1s"
  flush_interval = "1s"

[[inputs.cpu]]
  percpu = false
  totalcpu = true

[[outputs.http]]
  url = "http://localhost:3000/api/live/push/custom_stream_id"
  data_format = "influx"
  [outputs.http.headers]
    Authorization = "Bearer <Your API Key>"
```

Make sure to replace `<Your API Key>` placeholder with your actual API key created in the previous step. Save this config into `telegraf.conf` file and run Telegraf pointing to this config file. Telegraf will periodically (once in a second) report the state of total CPU usage on a host to Grafana (which is supposed to be running on `http://localhost:3000`). Of course you can replace `custom_stream_id` to something more meaningful for your use case.

Inside Grafana Influx data is converted to Grafana data frames and then frames are published to Grafana Live channels. In this case, the channel where CPU data will be published is `stream/custom_stream_id/cpu`. The `stream` scope is constant, the `custom_stream_id` namespace is the last part of API URL set in Telegraf configuration (`http://localhost:3000/api/live/push/telegraf`) and the path is `cpu` - the name of a measurement.

The only thing left here is to create a dashboard with streaming data.

## Create dashboard with streaming data

1. Click **Dashboards** in the left-side menu.
1. Click **New** and select **New Dashboard**.
1. On the empty dashboard, click **+ Add visualization**.
1. In the modal that opens, select the `-- Grafana --` data source.
1. Select `Live Measurements` query type.
1. Find and select `stream/custom_stream_id/cpu` measurement for Channel field.
1. Save dashboard changes.

After making these steps Grafana UI should subscribe to the channel `stream/custom_stream_id/cpu` and you should see CPU data updates coming from Telegraf in near real-time.

## Stream using WebSocket endpoint

If you aim for a high-frequency update sending then you may want to use the WebSocket output plugin of Telegraf (introduced in Telegraf v1.19.0) instead of the HTTP output plugin we used above. Configure WebSocket output plugin like this:

```
[agent]
  interval = "500ms"
  flush_interval = "500ms"

[[inputs.cpu]]
  percpu = false
  totalcpu = true

[[outputs.websocket]]
  url = "ws://localhost:3000/api/live/push/custom_stream_id"
  data_format = "influx"
  [outputs.websocket.headers]
    Authorization = "Bearer <Your API Key>"
```

WebSocket avoids running all Grafana HTTP middleware on each request from Telegraf thus reducing Grafana backend CPU usage significantly.

## Summary

In this tutorial you learned how to use Telegraf to stream live metrics to Grafana.

# Upgrade Guide

# Upgrade to Grafana v11.0

{{< docs/shared lookup="upgrade/intro.md" source="grafana" version="<GRAFANA VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA VERSION>" >}}

## Technical notes

# Upgrade to Grafana v11.1

{{< docs/shared lookup="upgrade/intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

# Upgrade to Grafana v11.2

{{< docs/shared lookup="upgrade/intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

### Grafana data source UID format enforcement

**Ensure that your data source UIDs follow the correct standard**

We've had a standard ways to define UIDs for Grafana objects for years (at least [since Grafana v5](https://github.com/grafana/grafana/issues/7883)). While all of our internal code complies to this format, we didn't yet have strict enforcement of this format in REST APIs and provisioning paths that allow the creation and update of data sources.

In Grafana v11.1, we [introduced](https://github.com/grafana/grafana/pull/86598) a warning that is sent to Grafana server logs every time a data source instance is being created or updated using an invalid UID format.

In Grafana v11.2, we [added](https://github.com/grafana/grafana/pull/89363/files) a new feature flag called `failWrongDSUID` that is turned off by default. When enabled, the REST APIs and provisioning start rejecting any requests to create or update data source instances that have an incorrect UID.

In Grafana v11.5, we're going to turn feature flag `failWrongDSUID` on by default, but there will still be an option to turn it off.

In Grafana v12, this will be the default behavior and will not be configurable.

#### Correct UID format

You can find the exact regex definition [in the grafana repository](https://github.com/grafana/grafana/blob/c92f5169d1c83508beb777f71a93336179fe426e/pkg/util/shortid_generator.go#L32-L45).

A data source UID can only contain:

- Latin characters (`a-Z`)
- Numbers (`0-9`)
- Dash symbols (`-`)

#### How do I know if I'm affected?

- You can fetch all your data sources using the `/api/datasources` API. Review the `uid` fields, comparing them to the correct format, as shown [in the docs](https://grafana.com/docs/grafana/latest/developers/http_api/data_source/#get-all-data-sources). Following is a script that can help, but note that it's missing authentication that you [have to add yourself](https://grafana.com/docs/grafana/latest/developers/http_api/#authenticating-api-requests):

```
curl http://localhost:3000/api/datasources | jq '.[] | select((.uid | test("^[a-zA-Z0-9\\-_]+$") | not) or (.uid | length > 40)) | {id, uid, name, type}'
```

- Alternatively, you can check the server logs for the `Invalid datasource uid` [error](https://github.com/grafana/grafana/blob/68751ed3107c4d15d33f34b15183ee276611785c/pkg/services/datasources/service/store.go#L429).

#### What do I do if I'm affected?

You'll need to create a new data source with the correct UID and update your dashboards and alert rules to use it.

#### How do I update my dashboards to use the new or updated data source?

1. Go to the dashboard using the data source and update it by selecting the new or updated data source from the picker below your panel.
1. Update the dashboard's JSON model directly using search and replace.

   Navigate to [dashboard json model](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/view-dashboard-json-model/) and carefully replace all the instances of old `uid` with the newly created `uid`.

   {{< figure src="/media/docs/grafana/screenshot-grafana-11-datasource-uid-enforcement.png" alt="Updating JSON Model of a Dashboard">}}

#### How do I update my alert rules to use the new or updated data source?

Open the alert rule you want to adjust and search for the data source that is being used for the query/alert condition. From there, select the new data source from the drop-down list and save the alert rule.

# Upgrade to Grafana v11.3

{{< docs/shared lookup="upgrade/intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

# Upgrade to Grafana v11.4

{{< docs/shared lookup="upgrade/intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

# Upgrade to Grafana v11.5

{{< docs/shared lookup="upgrade/intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

# Upgrade to Grafana v11.6

{{< docs/shared lookup="upgrade/intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

# Upgrade to Grafana v12.0

{{< docs/shared lookup="upgrade/intro_2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

### Grafana data source UID format enforcement

**Ensure that your data source UIDs follow the correct standard**

We've had standard ways to define UIDs for Grafana objects for years (at least [since Grafana v5](https://github.com/grafana/grafana/issues/7883)). While all of our internal code complies with this format, we haven't strictly enforced this format in REST APIs and provisioning paths that allow the creation and update of data sources.

In Grafana v11.1, we [introduced](https://github.com/grafana/grafana/pull/86598) a warning that is sent to Grafana server logs every time a data source instance is created or updated using an invalid UID format.

In Grafana v11.2, we [added](https://github.com/grafana/grafana/pull/89363/files) a new feature flag called `failWrongDSUID` that is turned off by default. When enabled, the REST APIs and provisioning reject any requests to create or update data source instances that have an incorrect UID.

In Grafana v12.0, we're turning the feature flag `failWrongDSUID` on by default.

#### Correct UID format

You can find the exact regex definition [in the `grafana/grafana` repository](https://github.com/grafana/grafana/blob/c92f5169d1c83508beb777f71a93336179fe426e/pkg/util/shortid_generator.go#L32-L45).

A data source UID can only contain:

- Latin characters (`a-Z`)
- Numbers (`0-9`)
- Dash symbols (`-`)

#### How do I know if I'm affected?

- You can fetch all your data sources using the `/api/datasources` API. Review the `uid` fields, comparing them to the correct format, as shown [in the docs](https://grafana.com/docs/grafana/latest/developers/http_api/data_source/#get-all-data-sources). The following script can help, but note that it's missing authentication that you [have to add yourself](https://grafana.com/docs/grafana/latest/developers/http_api/#authenticating-api-requests):

```
curl http://localhost:3000/api/datasources | jq '.[] | select((.uid | test("^[a-zA-Z0-9\\-_]+$") | not) or (.uid | length > 40)) | {id, uid, name, type}'
```

- Alternatively, you can check the server logs for the `Invalid datasource uid` [error](https://github.com/grafana/grafana/blob/68751ed3107c4d15d33f34b15183ee276611785c/pkg/services/datasources/service/store.go#L429).

#### What do I do if I'm affected?

You'll need to create a new data source with the correct UID and update your dashboards and alert rules to use it.

#### How do I update my dashboards to use the new or updated data source?

- Go to the dashboard using the data source and update it by selecting the new or updated data source from the picker below your panel.

OR

- Update the dashboard's JSON model directly using search and replace.

  Navigate to [dashboard json model](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/view-dashboard-json-model/) and carefully replace all the instances of the old `uid` with the newly created `uid`.

  {{< figure src="/media/docs/grafana/screenshot-grafana-11-datasource-uid-enforcement.png" alt="Updating JSON Model of a Dashboard">}}

#### How do I update my alert rules to use the new or updated data source?

Open the alert rule you want to adjust and search for the data source that is being used for the query/alert condition. From there, select the new data source from the drop-down list and save the alert rule.

### Enforcing stricter version compatibility checks in plugin CLI install commands

Since Grafana 10.2, the endpoint to check compatible versions when installing a plugin using `grafana cli plugins install` changed, which led to Grafana dependency version no longer being taken into account. This might have led to some behavior where the CLI would install plugins that are not fully compatible based on the plugins definition of compatibility via `grafanaDependency` property in the `plugin.json` file.

#### What if I want to ignore the compatibility check?

We _do not_ recommend installing plugins declared as incompatible. However, if you need to force install a plugin despite it being declared as incompatible, refer to the [Installing a plugin from a ZIP](https://grafana.com/docs/grafana/latest/administration/plugin-management/#install-a-plugin-from-a-zip-file) guidance.

### Annotation table migration

**Plan for increased disk usage when upgrading from Grafana v11.x**

Upgrading from Grafana v11.x to Grafana v12.x triggers a full-table rewrite of the `annotation` table. The migration populates the new `dashboard_uid` column, which causes the database to rewrite the entire table and rebuild its indexes.

Environments with large annotation datasets can experience significant temporary disk usage increase, which may lead to:

- Rapid disk consumption on the database data volume
- Database migration failures (for example, "could not extend file: No space left on device")
- Grafana startup failures
- Extended downtime during the upgrade process

#### How do I know if I'm affected?

You're affected if you're upgrading from Grafana v11.x to v12.x and you have a large `annotation` table in your database.

To check your annotation table size, connect to your database and check the table size.

For PostgreSQL, run the following query:

```sql
SELECT
    pg_size_pretty(pg_relation_size('annotation'))       AS table_size,
    pg_size_pretty(pg_indexes_size('annotation'))        AS indexes_size,
    pg_size_pretty(pg_total_relation_size('annotation')) AS total_size;
```

For MySQL, run the following query:

```sql
SELECT
    ROUND(data_length / 1024 / 1024, 2) AS table_size_mb,
    ROUND(index_length / 1024 / 1024, 2) AS indexes_size_mb,
    ROUND((data_length + index_length) / 1024 / 1024, 2) AS total_size_mb
FROM information_schema.tables
WHERE table_schema = DATABASE()
    AND table_name = 'annotation';
```

For SQLite, check the database file size directly, as SQLite stores all tables in a single file. You can run the following command from your terminal:

```bash
ls -lh <PATH/TO/GRAFANA.DB>
```

If your total size is several gigabytes or more, you should plan accordingly before upgrading.

#### What should I do before upgrading?

Before you upgrade, take the following steps:

1. **Verify available disk space**: Ensure you have at least 2-3 times the current `annotation` table size available as free disk space on your database data volume.

1. **Review your annotation data**: Consider whether you need to retain all historical annotations.

1. **Clean up old annotations (optional)**: If you have annotations you don't need, remove them before upgrading.

1. **Back up your database**: Always back up your Grafana database before performing an upgrade. For more information, refer to [Back up Grafana](#back-up-grafana).

#### What should I do after upgrading?

After successfully upgrading to Grafana v12.x, you can reclaim disk space by performing database maintenance operations during a maintenance window.

For PostgreSQL, run a `VACUUM FULL` operation on the `annotation` table:

```sql
VACUUM FULL annotation;
```

For MySQL, run an `OPTIMIZE TABLE` operation on the `annotation` table:

```sql
OPTIMIZE TABLE annotation;
```

For SQLite, run a `VACUUM` operation on the database:

```sql
VACUUM;
```

These operations require a lock on the table and may take significant time depending on the table size. Plan to run these during a low-traffic period.

# Upgrade to Grafana v12.1

{{< docs/shared lookup="upgrade/intro_2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

### Annotation table migration

**Plan for increased disk usage when upgrading from Grafana v11.x**

Upgrading from Grafana v11.x to Grafana v12.x triggers a full-table rewrite of the `annotation` table. The migration populates the new `dashboard_uid` column, which causes the database to rewrite the entire table and rebuild its indexes.

Environments with large annotation datasets can experience significant temporary disk usage increase, which may lead to:

- Rapid disk consumption on the database data volume
- Database migration failures (for example, "could not extend file: No space left on device")
- Grafana startup failures
- Extended downtime during the upgrade process

#### How do I know if I'm affected?

You're affected if you're upgrading from Grafana v11.x to v12.x and you have a large `annotation` table in your database.

To check your annotation table size, connect to your database and check the table size.

For PostgreSQL, run the following query:

```sql
SELECT
    pg_size_pretty(pg_relation_size('annotation'))       AS table_size,
    pg_size_pretty(pg_indexes_size('annotation'))        AS indexes_size,
    pg_size_pretty(pg_total_relation_size('annotation')) AS total_size;
```

For MySQL, run the following query:

```sql
SELECT
    ROUND(data_length / 1024 / 1024, 2) AS table_size_mb,
    ROUND(index_length / 1024 / 1024, 2) AS indexes_size_mb,
    ROUND((data_length + index_length) / 1024 / 1024, 2) AS total_size_mb
FROM information_schema.tables
WHERE table_schema = DATABASE()
    AND table_name = 'annotation';
```

For SQLite, check the database file size directly, as SQLite stores all tables in a single file. You can run the following command from your terminal:

```bash
ls -lh <PATH/TO/GRAFANA.DB>
```

If your total size is several gigabytes or more, you should plan accordingly before upgrading.

#### What should I do before upgrading?

Before you upgrade, take the following steps:

1. **Verify available disk space**: Ensure you have at least 2-3 times the current `annotation` table size available as free disk space on your database data volume.

1. **Review your annotation data**: Consider whether you need to retain all historical annotations.

1. **Clean up old annotations (optional)**: If you have annotations you don't need, remove them before upgrading.

1. **Back up your database**: Always back up your Grafana database before performing an upgrade. For more information, refer to [Back up Grafana](#back-up-grafana).

#### What should I do after upgrading?

After successfully upgrading to Grafana v12.x, you can reclaim disk space by performing database maintenance operations during a maintenance window.

For PostgreSQL, run a `VACUUM FULL` operation on the `annotation` table:

```sql
VACUUM FULL annotation;
```

For MySQL, run an `OPTIMIZE TABLE` operation on the `annotation` table:

```sql
OPTIMIZE TABLE annotation;
```

For SQLite, run a `VACUUM` operation on the database:

```sql
VACUUM;
```

These operations require a lock on the table and may take significant time depending on the table size. Plan to run these during a low-traffic period.

# Upgrade to Grafana v12.2

{{< docs/shared lookup="upgrade/intro_2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

### Annotation table migration

**Plan for increased disk usage when upgrading from Grafana v11.x**

Upgrading from Grafana v11.x to Grafana v12.x triggers a full-table rewrite of the `annotation` table. The migration populates the new `dashboard_uid` column, which causes the database to rewrite the entire table and rebuild its indexes.

Environments with large annotation datasets can experience significant temporary disk usage increase, which may lead to:

- Rapid disk consumption on the database data volume
- Database migration failures (for example, "could not extend file: No space left on device")
- Grafana startup failures
- Extended downtime during the upgrade process

#### How do I know if I'm affected?

You're affected if you're upgrading from Grafana v11.x to v12.x and you have a large `annotation` table in your database.

To check your annotation table size, connect to your database and check the table size.

For PostgreSQL, run the following query:

```sql
SELECT
    pg_size_pretty(pg_relation_size('annotation'))       AS table_size,
    pg_size_pretty(pg_indexes_size('annotation'))        AS indexes_size,
    pg_size_pretty(pg_total_relation_size('annotation')) AS total_size;
```

For MySQL, run the following query:

```sql
SELECT
    ROUND(data_length / 1024 / 1024, 2) AS table_size_mb,
    ROUND(index_length / 1024 / 1024, 2) AS indexes_size_mb,
    ROUND((data_length + index_length) / 1024 / 1024, 2) AS total_size_mb
FROM information_schema.tables
WHERE table_schema = DATABASE()
    AND table_name = 'annotation';
```

For SQLite, check the database file size directly, as SQLite stores all tables in a single file. You can run the following command from your terminal:

```bash
ls -lh <PATH/TO/GRAFANA.DB>
```

If your total size is several gigabytes or more, you should plan accordingly before upgrading.

#### What should I do before upgrading?

Before you upgrade, take the following steps:

1. **Verify available disk space**: Ensure you have at least 2-3 times the current `annotation` table size available as free disk space on your database data volume.

1. **Review your annotation data**: Consider whether you need to retain all historical annotations.

1. **Clean up old annotations (optional)**: If you have annotations you don't need, remove them before upgrading.

1. **Back up your database**: Always back up your Grafana database before performing an upgrade. For more information, refer to [Back up Grafana](#back-up-grafana).

#### What should I do after upgrading?

After successfully upgrading to Grafana v12.x, you can reclaim disk space by performing database maintenance operations during a maintenance window.

For PostgreSQL, run a `VACUUM FULL` operation on the `annotation` table:

```sql
VACUUM FULL annotation;
```

For MySQL, run an `OPTIMIZE TABLE` operation on the `annotation` table:

```sql
OPTIMIZE TABLE annotation;
```

For SQLite, run a `VACUUM` operation on the database:

```sql
VACUUM;
```

These operations require a lock on the table and may take significant time depending on the table size. Plan to run these during a low-traffic period.

# Upgrade to Grafana v12.3

{{< docs/shared lookup="upgrade/intro_2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

### Annotation table migration

**Plan for increased disk usage when upgrading from Grafana v11.x**

Upgrading from Grafana v11.x to Grafana v12.x triggers a full-table rewrite of the `annotation` table. The migration populates the new `dashboard_uid` column, which causes the database to rewrite the entire table and rebuild its indexes.

Environments with large annotation datasets can experience significant temporary disk usage increase, which may lead to:

- Rapid disk consumption on the database data volume
- Database migration failures (for example, "could not extend file: No space left on device")
- Grafana startup failures
- Extended downtime during the upgrade process

#### How do I know if I'm affected?

You're affected if you're upgrading from Grafana v11.x to v12.x and you have a large `annotation` table in your database.

To check your annotation table size, connect to your database and check the table size.

For PostgreSQL, run the following query:

```sql
SELECT
    pg_size_pretty(pg_relation_size('annotation'))       AS table_size,
    pg_size_pretty(pg_indexes_size('annotation'))        AS indexes_size,
    pg_size_pretty(pg_total_relation_size('annotation')) AS total_size;
```

For MySQL, run the following query:

```sql
SELECT
    ROUND(data_length / 1024 / 1024, 2) AS table_size_mb,
    ROUND(index_length / 1024 / 1024, 2) AS indexes_size_mb,
    ROUND((data_length + index_length) / 1024 / 1024, 2) AS total_size_mb
FROM information_schema.tables
WHERE table_schema = DATABASE()
    AND table_name = 'annotation';
```

For SQLite, check the database file size directly, as SQLite stores all tables in a single file. You can run the following command from your terminal:

```bash
ls -lh <PATH/TO/GRAFANA.DB>
```

If your total size is several gigabytes or more, you should plan accordingly before upgrading.

#### What should I do before upgrading?

Before you upgrade, take the following steps:

1. **Verify available disk space**: Ensure you have at least 2-3 times the current `annotation` table size available as free disk space on your database data volume.

1. **Review your annotation data**: Consider whether you need to retain all historical annotations.

1. **Clean up old annotations (optional)**: If you have annotations you don't need, remove them before upgrading.

1. **Back up your database**: Always back up your Grafana database before performing an upgrade. For more information, refer to [Back up Grafana](#back-up-grafana).

#### What should I do after upgrading?

After successfully upgrading to Grafana v12.x, you can reclaim disk space by performing database maintenance operations during a maintenance window.

For PostgreSQL, run a `VACUUM FULL` operation on the `annotation` table:

```sql
VACUUM FULL annotation;
```

For MySQL, run an `OPTIMIZE TABLE` operation on the `annotation` table:

```sql
OPTIMIZE TABLE annotation;
```

For SQLite, run a `VACUUM` operation on the database:

```sql
VACUUM;
```

These operations require a lock on the table and may take significant time depending on the table size. Plan to run these during a low-traffic period.

# Upgrade to Grafana v12.4

{{< docs/shared lookup="upgrade/intro_2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Upgrade to Grafana v13.0

{{< docs/shared lookup="upgrade/intro_2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

{{< admonition type="warning" >}}
**Git Sync early adopters:** A migration bug in Grafana v13.0.0 might cause dashboards and folders to be lost or reverted when upgrading from Grafana v12.x.x with Git Sync enabled. Upgrading from v13.0.0 to v13.0.1 does **not** recover lost data, and if you already upgraded, you must restore your database first, then upgrade to v13.0.1.

The bug only affects self-managed instances with Git Sync feature flags (`provisioning`, `kubernetesClientDashboardsFolders`, and `kubernetesDashboards`) enabled on Grafana v12.x.x.

Grafana v13.0.0 has been removed from distribution and the fix has shipped in Grafana v13.0.1.

If you're affected, use the following recovery paths:

- **If your instance used mixed local and Git Sync content:** restore from the database backup you took before upgrading, then upgrade to Grafana v13.0.1.
- **If your instance used full-instance Git Sync:** upgrade to Grafana v13.0.1 and resync from your Git repository.
- **If unsure:** restore from your database backup before upgrading to v13.0.1.
  {{< /admonition >}}

### React 19 related updates

As part of [the migration to React 19 in Grafana 13](https://grafana.com/blog/react-19-is-coming-to-grafana-what-plugin-developers-need-to-know/#next-steps-and-how-to-learn-more), make the following updates to ensure that your plugins are working properly and you don't experience any disruptions during the Grafana 13 upgrade.

Follow this sequence for best results:

#### Upgrade your Grafana to latest patch version for the version you are running

To ensure that the changes required for the upgrade to React 19 are in place in your Grafana version, update to the latest minor version available for your Grafana instance. You can check the available versions on the [downloads page](https://grafana.com/grafana/download).

#### Update all of your plugins

Update all of your installed plugins and check if they are still working properly. By using the latest version of a plugin you facilitate support to React 19.

#### Upgrade to Grafana 13

Finally you can continue your upgrade to Grafana 13.

### Legacy `/api` endpoints are now deprecated

Grafana is migrating existing APIs to the new `/apis` model, a Kubernetes-style API layer which follows a standardized API structure alongside consistent API versioning. Refer to the [New API structure in Grafana](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis) documentation for more details.

**Legacy APIs are not being disabled for the moment**. Removal of legacy APIs is planned for a future major release, and any breaking changes will be announced well in advance to avoid disruptions.

For more information and migration guidance, refer to [Migrate to the new APIs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/apis-migration).

### Deprecated data source APIs disabled

Data source APIs that reference data sources by numeric `id` have been deprecated since Grafana 9.
In Grafana 13, they're disabled by default.

#### You are affected if

You use data source API endpoints that reference data sources by numeric `id` rather than `uid`.

#### Migration

Update your API calls to reference data sources by `uid` instead of numeric `id`.
To temporarily re-enable the deprecated APIs, enable the `datasourceLegacyIdApi` feature flag.
Both the deprecated APIs and the feature toggle will be removed in a future release.

### Image Renderer plugin support removed

In Grafana 13, support for the Image Renderer plugin is removed.

#### You are affected if

You run the Image Renderer as a Grafana plugin.
After upgrading, the plugin no longer works for rendering screenshots or scheduled reports.

#### Migration

Deploy the image renderer as a separate service alongside Grafana.
For setup instructions, refer to [Set up image rendering](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/image-rendering/#set-up-image-rendering).

### Image Renderer default authentication changed to JWTs

The Image Renderer previously authenticated with Grafana using opaque tokens stored in the database when generating screenshots and PDFs. Grafana v13.0 enables the `renderAuthJWT` feature toggle by default, which switches authentication to stateless JSON Web Tokens (JWTs) that don't depend on the database.

#### Action required

After upgrading to Grafana v13.0, if you use the Image Renderer, you must set a `[rendering]renderer_token` in your Grafana configuration file to a value that isn't empty or the default value (`-`), and configure the Image Renderer with the same token value. Restart your Grafana instance for the changes to take effect.

For more information, refer to the [Image Renderer security configuration documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/image-rendering/#security).

#### Reverting to the previous behavior

After upgrading to Grafana v13.0, to revert to opaque token authentication, disable the feature toggle in your Grafana configuration file:

```ini
[feature_toggles]
renderAuthJWT = false
```

Restart your Grafana instance for the changes to take effect.

### Unified storage for folders and dashboards

Grafana v13.0 automatically migrates folders and dashboards from the legacy SQL database to unified storage on startup.
The migration runs once and is tracked in the `unifiedstorage_migration_log` table.

After the migration completes, the following legacy tables are deprecated:

- `dashboard`
- `dashboard_acl`
- `dashboard_provisioning`
- `dashboard_version`
- `dashboard_tag`
- `library_element_connection`
- `folder`

These tables will be removed in a future release.

If you downgrade to an earlier Grafana version after the migration, the older version reads the stale legacy tables and doesn't reflect changes made in unified storage.
To roll back, restore from the database backup you took before upgrading.

If you downgrade and then upgrade again without restoring a backup, any folders or dashboards created or modified during the downgrade aren't migrated automatically.
If a backup isn't available, contact Grafana support for assistance.

#### SQLite deployments

If you use SQLite, the migration might fail with `database is locked` or `database table is locked` errors due to lock contention.
Grafana automatically retries using the Parquet buffer, but if errors persist, increase `migration_cache_size_kb` or enable `migration_parquet_buffer` in the `[unified_storage]` section of your configuration file:

| Setting                    | Description                                                                                              | Default           |
| -------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------- |
| `migration_cache_size_kb`  | SQLite page cache size during migration.                                                                 | `1000000` (~1 GB) |
| `migration_parquet_buffer` | Stage data through a temporary Parquet file to separate read and write phases, avoiding lock contention. | `false`           |

### Removal of `grafana-cli` and `grafana-server` commands

The `grafana-cli` and `grafana-server` commands, deprecated since Grafana v10.0, are removed in Grafana v13.0.
Update any scripts, systemd units, Docker entrypoints, CI pipelines, or other automation to use `grafana cli` and `grafana server` instead.

### Legacy Alertmanager configuration API endpoints changed

Several legacy Alertmanager configuration API endpoints are removed or restricted in Grafana v13.0:

- `DELETE /api/alertmanager/grafana/config/api/v1/alerts` is removed.
- `POST /api/alertmanager/grafana/config/api/v1/receivers/test` is removed.
- `GET /api/alertmanager/grafana/config/api/v1/alerts` is restricted to admin users.
- `GET /api/alertmanager/grafana/config/history` is restricted to admin users.
- `POST /api/alertmanager/grafana/config/history/{id}/_activate` is restricted to admin users.

#### You are affected if

You call any of these endpoints in automation scripts, Terraform providers, or custom tooling.

#### Migration

Migrate to the Kubernetes-style resource APIs under `notifications.alerting.grafana.app/v1beta1`:

| Resource              | API path                                                                                       |
| --------------------- | ---------------------------------------------------------------------------------------------- |
| Receivers             | `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/receivers`            |
| Notification policies | `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/routingtrees`         |
| Templates             | `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/templategroups`       |
| Mute timings          | `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/timeintervals`        |
| Inhibition rules      | `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/inhibitionrules`      |
| Receiver testing      | `/apis/notifications.alerting.grafana.app/v1beta1/namespaces/{namespace}/receivers/{uid}/test` |

### Alertmanager status endpoint requires a new permission

#### You are affected if

You use the `GET /api/alertmanager/grafana/api/v2/status` endpoint and rely on the `alert.notifications:read` permission to access it.

#### Description

The `GET /api/alertmanager/grafana/api/v2/status` endpoint previously required the legacy `alert.notifications:read` permission. It now requires a dedicated `alert.notifications.system-status:read` permission. This new permission is included in the `fixed:alerting.notifications:writer` role, which is granted to Admin users by default.

#### Migration

If you have custom roles that need access to this endpoint, add the `alert.notifications.system-status:read` action to those roles. Admin users are unaffected as they receive this permission automatically through the built-in alerting notifications writer role.

### RBAC enforcement changes for custom roles

Grafana v13.0 tightens RBAC enforcement for custom roles, Terraform-managed roles, and role provisioning. Roles that contain deprecated permissions may fail during create, update, delete, or assignment operations. We recommend reviewing and updating affected roles proactively.

#### You are affected if

You manage custom RBAC roles through Terraform, the API, or provisioning, and those roles contain any of the following:

- Data source UID-scoped permissions (`datasources:uid:<uid>`) on a global role
- `fixed:annotations.dashboard:writer` or `fixed:annotations.dashboard:reader`
- `annotations:type:dashboard`
- `annotations:*`

#### Migration

**Data source permissions on global roles:** If a Terraform-managed `grafana_role` combines `global = true` with `datasources:uid:<uid>`, recreate the role as a non-global role with a new UID. Grafana can't change the scope of an existing role in place. Set `datasource_type` on data source permission resources when possible.

**Dashboard annotation fixed roles:** Remove `fixed:annotations.dashboard:writer` and `fixed:annotations.dashboard:reader` from your configurations. Use dashboard or folder permissions (View, Edit, Admin) to control dashboard annotation access instead. Use `annotations:type:organization` for organization-level annotations.

**`annotations:*` wildcard scope:** Replace with specific scopes: `annotations:type:organization` for organization annotations, and dashboard or folder permissions for dashboard annotations. Don't recreate `annotations:*` in Terraform after it has been removed.

If a role is stuck because Grafana refuses to update or delete it, the role may contain a deprecated permission that must be removed before Terraform or API updates can succeed. Contact Grafana support for assistance.

For detailed examples and troubleshooting, refer to [RBAC behavior changes with Grafana 13](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/whatsnew/whats-new-in-v13/whats-new-in-v13-0/#rbac-behavior-changes-with-grafana-13).

# Upgrade to Grafana v13.1

{{< docs/shared lookup="upgrade/intro_2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

# Upgrade to Grafana v13.2

{{< docs/shared lookup="upgrade/intro_2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Technical notes

# Upgrade to Grafana v13.3

{{< docs/shared lookup="upgrade/intro_2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

{{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Strategies for upgrading your self-managed Grafana instance

At Grafana Labs, we believe in shipping features early and often, and in recent years we’ve increased our commitment to that philosophy.

We no longer wait for the yearly major release to give you access to the next big improvement. Instead, we regularly make new features, bug fixes, and security patches available to our self-managing users ([Grafana OSS](https://grafana.com/oss/grafana/) and [Grafana Enterprise](https://grafana.com/products/enterprise/)) throughout the year.

Having a dependable release process provides users like you with the best Grafana experience possible, and it provides the flexibility to upgrade in a manner that works best for you and your organization.

{{< admonition type="note" >}}
Grafana Cloud follows a different release cadence than Grafana OSS and Enterprise. In Cloud, Grafana uses Rolling release channels. To learn more about release channels, refer to [Rolling release channels for Grafana Cloud](https://grafana.com/docs/rolling-release/).
{{< /admonition >}}

## What to expect from each release type

We split Grafana OSS and Grafana Enterprise releases into three main categories:

- **Minor release (every other month)**: These releases can include new features, deprecation notices, notices about upcoming breaking changes, previously announced breaking changes, bug fixes, and security vulnerability patches.
- **Major release (once a year, in April/May)**: These are like a minor release, but accompanied by [GrafanaCON](https://grafana.com/events/grafanacon/) and a comprehensive upgrade guide for users who like to upgrade only once a year.
- **Patching release (at least every month)**: These include bug fixes for currently supported versions, as well as any security vulnerability patches.

You can choose your cadence: For frequent self-managed updates, you should follow the minor release (for example, upgrade 11.1 to 11.2), which also gives you access to the latest features. If you need a longer period to review our new releases, you should follow the major releases. Both strategies get patching releases with security fixes (high severity security fixes also result in ad-hoc patch releases). We’ll get into additional guidance on upgrade cadences later in this guide.

## How to find the specifics for a release

We love sharing all our great features with you so you can leverage Grafana to its fullest. We also understand that complete release documentation allows you to upgrade with confidence.
Whether it’s knowing that a bug has been fixed, seeing that a security vulnerability is patched, or understanding how to mitigate the impact of breaking changes, proper documentation allows you to make informed decisions about when to upgrade your local Grafana instances.

We provide release documentation in multiple places to address different needs:

- [**What’s new**](https://grafana.com/docs/grafana/latest/whatsnew/?pg=blog&plcmt=body-txt) outlines new features debuting in each major and minor release, along with breaking changes that could impact you and recommendations to mitigate them.
- [**Upgrade guides**](https://grafana.com/docs/grafana/latest/upgrade-guide/?pg=blog&plcmt=body-txt) instruct you on how to upgrade to a newer minor or major version.
- And finally, a [**changelog**](https://github.com/grafana/grafana/blob/main/CHANGELOG.md) is generated for every release (major, minor, patching, security) and outlines all changes included in that release.

## When to expect releases

Grafana releases a new minor version every other month, with patching releases in between. Below are the planned releases for the rest of 2026. However, these dates may be subject to change:

| **Release date** | **Grafana versions**      | **Release type** |
| ---------------- | ------------------------- | ---------------- |
| Feb. 24, 2026    | 12.4 & Supported versions | Minor & patching |
| Apr. 14, 2026    | Grafana 13                | Major            |
| Jun. 23, 2026    | 13.1 & Supported versions | Minor & patching |
| Jul. 21, 2026    | Supported versions        | Patching         |
| Aug. 4, 2026     | Supported versions        | Patching         |
| Aug. 18, 2026    | 13.2 & Supported versions | Minor & patching |
| Sept. 1, 2026    | Supported versions        | Patching         |
| Oct. 20, 2026    | 13.3 & Supported versions | Minor & patching |

### A few important notes

- The schedule above outlines how we plan release dates. However, unforeseen events and circumstances may cause dates to change.
- High severity security and feature degradation incidents will result in ad-hoc releases that are not scheduled ahead of time.
- Patching releases are for all supported minor versions of Grafana. Each supported minor version receives patch releases with bug fixes and security patches until its end of life.
- Release freezes: Each year Grafana implements two release freezes to accommodate for the holiday season. During these times, no scheduled releases will be executed. However, this does not apply to changes that may be required during the course of an operational or security incident.

## Grafana security releases: improved version naming convention

We've enhanced our naming convention for security release versions to make it easier to clearly identify our security releases from our standard patching releases.

In the past, critical vulnerabilities triggered unscheduled releases that incremented the patch version (e.g., 10.3.0 to 10.3.1). However, we found that the naming convention for these releases didn't clearly communicate the nature of the update. For example, if there was a version change from 11.3.0 to 11.3.1, there was no indication whether it was a security fix, a bug fix, or a minor feature update. This lack of clarity led to confusion about the urgency and nature of the update. <br>

{{< admonition type="note" >}}
Docker does not allow the plus sign (`+`) in image tag names. A plus sign (`+`) will be a rendered as a dash (`-`) in the docker tag.
{{< /admonition >}}

Our new approach directly addresses this issue. Going forward, security releases will be appended with "+security" to indicate that the release is the indicated version PLUS the security fix.

**For example**: A release named "11.2.3+security-01" would consist of what was released in 11.2.3 PLUS the indicated security fix. Once released, the security fix will also then be automatically included in all future releases of the impacted version.

This naming convention should make it easier to identify security updates and the Grafana version they're based on, allowing for a better understanding of the importance and urgency of each release.

## What to know about version support

Self-managed Grafana users have control over when they upgrade to a new version of Grafana. To help you make an informed decision about whether it’s time to upgrade, it’s important that you understand the level of support provided for your current version.

For self-managed Grafana (both Enterprise and OSS), the support for versions follows these rules:

- Each minor release is supported for 9 months after its release date
- The last minor release of a major version receives extended support for 15 months after its release date
- Support levels change as new versions are released:
  - **Full Support**: The current major version receives new features through new minor releases
  - **Patch Support**: Individual minor versions receive patch releases (bug fixes and security patches) until end of life
  - **Not Supported**: Versions beyond their support period receive no updates

Here is an overview of version support through 2027:

| **Version**               | **Release date**   | **Support end date** | **Support level** |
| ------------------------- | ------------------ | -------------------- | ----------------- |
| 11.4.x                    | December 5, 2024   | September 5, 2025    | Not Supported     |
| 11.5.x                    | January 28, 2025   | October 28, 2025     | Not Supported     |
| 11.6.x (Last minor of 11) | March 25, 2025     | June 25, 2026        | Not Supported     |
| 12.0.x                    | May 5, 2025        | February 5, 2026     | Not Supported     |
| 12.1.x                    | July 22, 2025      | April 22, 2026       | Not Supported     |
| 12.2.x                    | September 23, 2025 | June 23, 2026        | Not Supported     |
| 12.3.x                    | November 19, 2025  | August 19, 2026      | Not Supported     |
| 12.4.x (Last minor of 12) | February 24, 2026  | May 24, 2027         | Patch Support     |
| 13.0.x                    | April 14, 2026     | January 9, 2027      | Patch Support     |
| 13.1.x                    | June 23, 2026      | March 20, 2027       | Patch Support     |
| 13.2.x                    | August 18, 2026    | May 18, 2027         | Patch Support     |
| 13.3.x                    | October 20, 2026   | July 20, 2027        | Patch Support     |

## How are these versions supported?

Self-managed Grafana follows semantic-like versioning (MAJOR.MINOR.PATCH). Here's how different types of releases work:

- **Major releases** (e.g., 12.0.0, 13.0.0):
  - Awesome new features as well as significant architectural improvements and modernizations
  - May include breaking changes that require migration steps
  - Released once per year

- **Minor releases** (e.g., 12.3.0, 12.4.0):
  - Contain new features and enhancements
  - Include bug fixes and security patches
  - Released every other month

- **Patch releases** (e.g., 12.3.1, 12.3.2):
  - **Only** contain bug fixes and security patches
  - **No new features** - these wait until the next minor release
  - Released monthly

**Support levels by version:**

- **Full Support** (current major version):
  - Gets new minor releases with new features approximately every other month
  - All minor versions within the major receive patch releases until end of life
  - Example: Major 12.x gets new features via 12.1.0, 12.2.0, 12.3.0, etc.

- **Patch Support** (individual minor versions):
  - Each minor version receives patch releases (bug fixes and security patches) until end of life
  - No new features - these only come with new minor releases
  - Example: 12.3.x gets 12.3.1, 12.3.2, etc. with fixes only

- **Not Supported**: Versions beyond their support period receive no updates and should be upgraded.

**Example**: When 12.3.0 is released, it includes new features. Subsequent releases like 12.3.1 and 12.3.2 only include bug fixes and security updates. All new features developed after 12.3.0 wait until 12.4.0 is released.

### What is a critical feature degradation?

A critical feature degradation usually meets one of the following criteria:

- Major functionality is universally unavailable (for example, cannot create dashboards, unable to authenticate).
- Major (critical) impact to a significant amount of customers.
- Major escalated incident for one or many customers.

## Self-managing upgrade strategies

Based on your needs, choose your ideal upgrade strategy. Here’s what that might look like in practice:

| **Strategy/cadence**                       | **Advantages/disadvantages**                                                                                                                                                                                                                                                                                           | **Example upgrade procedure**                                                                                                                                                                                                                                               |
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Minor / bi-monthly (11.1 to 11.2)**      | Our recommended strategy. It combines up-to-date, secure releases with access to latest features as soon as they're released. <ul><li>Small changelog to review</li><li>Highest compatibility with actively maintained plugins</li><li>Easy migration to [Grafana Cloud](https://grafana.com/products/cloud)</li></ul> | <ul><li>**January 2025**: You review the 11.5 changelog and deploy the release to testing</li><li>**February 2025**: You deploy 11.5 to production</li><li>**March 2025**: 11.6 is released</li></ul>                                                                       |
| **Major / yearly (10.0 to 11.0)**          | Yearly upgrade path that still gives access to up-to-date features presented at GrafanaCON.<ul><li>Big changelog to review</li><li>High compatibility with plugins</li><li>Relatively easy migration to [Grafana Cloud](https://grafana.com/products/cloud)</li></ul>                                                  | <ul><li>**May 2024**: 11.0 is released, you start a big changelog review</li><li>**June 2024**: You deploy 11.0 to testing</li><li>**July 2024**: You deploy 11.0 to production</li><li>**May 2025**: 12.0 is released</li></ul>                                            |
| **Previous major / yearly (10.4 to 11.6)** | Release with extended support timeline<ul><li>Limited compatibility with actively developed plugins</li><li>Big changelog to review</li><li>Migrations to Grafana Cloud might require professional support</li></ul>                                                                                                   | <ul><li>**May 2025**: 12.0 is released, marking the previous minor (11.6.x) with extended support, you start a big changelog review (10.4.x to 11.6.x)</li><li>**June 2025**: You deploy 11.6.x to testing</li><li>**July 2025**: You deploy 11.6.x to production</li></ul> |

Follow the “minor” strategy for the most flexibility, as you can also occasionally lengthen the cadence to a full quarter and still rely on your currently deployed minor release being supported with security fixes.

## How the Grafana team catches bugs and breaks during the release process

1. Each team writes automated tests for their code and we run [automated tests](https://github.com/grafana/grafana/blob/HEAD/contribute/developer-guide.md#test-grafana) which include unit tests, integration tests, end-to-end tests, and load tests. For plugins specifically, we test the frontend and backend parts of a data source (unit), integrating a data source with a matrix of Grafana versions (E2E), and the contract between the data source and the API it consumes (integration).
2. We perform manual acceptance and smoke testing internally for new features by deploying to our internal observability stack. After that, we progressively roll out in Grafana Cloud, and then cut an OSS and Enterprise release. Each stage catches bugs.
3. We ship new features in Experimental or Private Preview [release stages](https://grafana.com/docs/release-life-cycle/), behind feature toggles. This helps us improve the feature during development. If you are interested in gaining early access to features (including in your development or test environments), please let us know.
4. We scan Grafana, all plugins and their dependencies continuously for security vulnerabilities.

## Minimize the likelihood of bugs and problems during upgrade

Despite thorough testing, you can experience problems when upgrading:

### Bugs

Bugs are unexpected side effects of code changes in the release, which cause problems. Some bugs occur for all users, and we usually catch these in the early stages of testing. Others occur in a small number of Grafana instances with specific configuration or unusual use cases; for example a specific authentication setup or a combination of feature toggles. Grafana plugins also interact with external services via API to query data, and sometimes these APIs change without notice, causing issues for your dashboards that depend on these datasources. Grafana Labs has monitoring in place to regularly test these APIs, but at times they break in unexpected ways.

Reduce the risk of bugs by staying current and rolling out upgrades across dev or test environments before production. A Grafana Enterprise license entitles you to an additional dev and test instance for this purpose, available through your account team, and in Grafana Cloud you can create dev and test stacks that upgrade before production by using [rolling release channels](https://grafana.com/docs/rolling-release/).

- Back up your database on a regular basis, and especially before you upgrade.
- Roll back and report issues if you experience problems in dev or test.
- In Cloud, run a dev or test stack on the `fast` channel, and run your production stack on `steady` or `slow`. To change your release channel, open a support ticket.

### Known breaking changes

As a rule we always seek backward compatibility and migration, and reserve breaking changes for Grafana’s once-yearly major release. However occasionally small breaking changes (like updates to API payloads) will ship in minor releases. These are announced in [upgrade guides](https://grafana.com/docs/grafana/latest/upgrade-guide/), [What’s New](https://grafana.com/docs/grafana-cloud/whats-new/), and our [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

Always read the [upgrade guide](https://grafana.com/docs/grafana/latest/upgrade-guide/) and [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md) prior to upgrading to learn about and account for breaking changes.

### Plugin incompatibility

**Grafana core** ships as a single binary and consists of Dashboards, Alerts, Explore, Authentication and Authorization, Reporting, some core data sources, and other components. However, almost everyone who uses Grafana also uses **plugins**: panels, data sources, and applications that are released independently of Grafana. Every plugin version lists its Grafana version dependencies (you can see them at [https://grafana.com/grafana/plugins/](https://grafana.com/grafana/plugins/)) but different versions of different plugins can also interact with each other - for example you might visualize data from a data source in a panel in Grafana, all three of which are versioned independently of each other. That can create issues that are hard to catch in testing.

To minimize the likelihood of plugin incompatibility issues, run the latest available version of plugins and update them regularly. Always [update plugins](https://grafana.com/docs/grafana/latest/administration/plugin-management/#update-a-plugin) before updating Grafana. Plugins also follow Semver patterns, so review the plugin’s changelog for breaking changes before upgrading to a new major version of that plugin.

# Visualizations

# Assess dashboard usage

Usage insights enables you to have a better understanding of how your Grafana instance is used.

{{< admonition type="note" >}}
Available in [Grafana Enterprise](ref:grafana-enterprise) and [Grafana Cloud](https://grafana.com/docs/grafana-cloud/).
Grafana Cloud insights logs include additional fields with their own dashboards.
Read more in the [Grafana Cloud documentation](https://grafana.com/docs/grafana-cloud/account-management/usage-insights/).
{{< /admonition >}}

The usage insights feature collects a number of aggregated data and stores them in the database:

- Dashboard views (aggregated and per user)
- Data source errors
- Data source queries

The aggregated data provides you access to several features:

- [Dashboard and data source insights](#dashboard-and-data-source-insights)
- [Presence indicator](#presence-indicator)
- [Sort dashboards by using insights data](#sort-dashboards-by-using-insights-data)
- [Visualize usage insight data in a dashboard](#visualize-usage-insights-data)

This feature also generates detailed logs that can be exported to Loki. Refer to [Export logs of usage insights](ref:export-logs-of-usage-insights).

## Dashboard and data source insights

For every dashboard and data source, you can access usage information.

### Dashboard insights

To see dashboard usage information, click the dashboard insights icon in the toolbar.

{{< figure src="/media/docs/grafana/dashboards/screenshot-dashboard-insights-v12.4.png" max-width="500px" alt="Dashboard insights icon" >}}

Dashboard insights show the following information:

- **Stats:** The number of daily queries and errors for the past 30 days.
- **Users & activity:** The daily view count for the last 30 days; last activities on the dashboard and recent users (with a limit of 20).

{{< figure src="/static/img/docs/enterprise/dashboard_insights_stats.png" max-width="400px" class="docs-image--no-shadow" alt="Stats tab" >}}{{< figure src="/static/img/docs/enterprise/dashboard_insights_users.png" max-width="400px" class="docs-image--no-shadow" alt="Users and activity tab" >}}

If [dashboard sharing](ref:dashboard-sharing) is enabled, you'll also see a **Shared dashboards** tab in your analytics.

### Data source insights

Data source insights provides information about how a data source has been used in the past 30 days, such as:

- Queries per day
- Errors per day
- Query load time per day (averaged in ms)

To find data source insights:

1. Click **Connections** in the main navigation.
1. Under Your connections, click **Data sources**.
1. Click a data source.
1. Click the **Insights** tab.

{{< figure src="/media/docs/grafana/dashboards/screenshot-data-source-insights-9.5.png" max-width="650px" class="docs-image--no-shadow" alt="Insights tab for a data source" >}}

## Presence indicator

When you are signed in and looking at a dashboard, you can know who is looking at the same dashboard as you are via a presence indicator, which displays avatars of users who have recently interacted with the dashboard. The default time frame is 10 minutes. To see the user's name, hover over the user's avatar. The avatars come from [Gravatar](https://gravatar.com) based on the user's email.

When there are more active users on a dashboard than can fit within the presence indicator, click the **+X** icon. Doing so opens [dashboard insights](#dashboard-and-data-source-insights), which contains more details about recent user activity.

{{< figure src="/static/img/docs/enterprise/presence_indicators.png" max-width="400px" class="docs-image--no-shadow" alt="Presence indicator icons" >}}

To change _recent_ to something other than the past 10 minutes, edit the [configuration](ref:configuration) file:

```ini
[analytics.views]

# Set age for recent active users to 10 minutes
recent_users_age = 10m
```

To disable the presence indicator, edit the [configuration](ref:configuration) file as follows:

```ini
[analytics.views]


# Disables the presence indicator
recent_users_age = 0
```

The dashboard won't show any avatars and thus no recent user activity.

## Sort dashboards by using insights data

In the search view, you can use insights data to help you find most-used, broken, and unused dashboards.

You can sort the dashboards by:

- Errors total
- Errors 30 days (most and least)
- Views total
- Views 30 days (most and least)

Grafana counts the data behind these sort options once a day, up to the end of the previous day.
Because today's activity isn't included yet, these numbers can be lower than the view count in [dashboard insights](#dashboard-insights), which updates as activity happens.

**Views total** and **Errors total** are aggregates.
Grafana starts counting from the first time it aggregates data for a dashboard. That first pass looks back 30 days, so any activity older than that isn't included.

For activity closer to the present, use **Views 30 days** or **Errors 30 days**, or open [dashboard insights](#dashboard-insights) on a specific dashboard to see today's views.
To get same-day detail across all dashboards, [export logs of usage insights](ref:export-logs-of-usage-insights) and [visualize the exported logs](#visualize-usage-insights-data).

{{< figure src="/media/docs/grafana/dashboards/screenshot-dashboard-sort-9.5.png" max-width="650px" class="docs-image--no-shadow" alt="Open list of dashboard sort options" >}}

## Visualize usage insights data

If you set up your installation to [export logs of usage insights](ref:export-logs-of-usage-insights), there are two dashboards to help you take advantage of this data.

1. [Usage Insights overview](/grafana/dashboards/13785) provides a top-level perspective of user activity.
1. [Data source details](/grafana/dashboards/13786) dashboard provides a view of data source activity and health.

You can click the previous links to download the respective dashboard JSON, then import into your Grafana installation.

# Annotate visualizations

Annotations provide a way to mark points on a visualization with rich events. They are visualized as vertical lines and icons on all graph panels. When you hover over an annotation, you can get event description and event tags. The text field can include links to other systems with more detail.

{{< figure src="/static/img/docs/v46/annotations.png" max-width="800px" alt="Annotated visualization with annotation context menu open" >}}

To see annotations in action, explore this demo dashboard on Grafana Play:

{{< docs/play title="Annotations" url="https://play.grafana.org/d/000000010/" >}}

You can annotate visualizations in three ways:

- Directly in the panel, using the [built-in annotations query](#built-in-query)
- Using the HTTP API
- Configuring annotation queries

In the first two cases, you're creating new annotations, while in the last you're querying existing annotations from data sources. The built-in annotation query also supports this.

This page explains the first and third options; for information about using the HTTP API, refer to [Annotations API](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/annotations/).

Annotations are supported for the following visualization types:

- Time series
- State timeline
- Candlestick

## Create annotations in panels

Grafana comes with the ability to add annotation events directly from a panel using the [built-in annotation query](#built-in-query) that exists on all dashboards. Annotations that you create this way are stored in Grafana.

To add annotations directly in the panel:

- The dashboard must already be saved.
- The built-in query must be enabled. Learn more in [Built-in query](#built-in-query).

Watch the following video for a quick tutorial on creating annotations:

{{< youtube id="N5iOlyYyK6Q" >}}

### Add an annotation

To add an annotation, complete the following steps:

1. If you've just saved a dashboard, refresh the page.
1. Click a data point in a panel to open the tooltip.
1. In the tooltip, click **Add annotation**.
   ![Add annotation context menu](/static/img/docs/time-series-panel/time-series-annotations-context-menu.png)
1. Add an annotation description and tags (optional).
1. Click **Save**.

Alternatively, to add an annotation, press Ctrl/Cmd and click anywhere on the panel, and the **Add annotation** tooltip appears.

### Add a region annotation

1. If you've just saved a dashboard, refresh the page.
1. Press Ctrl/Cmd and click and drag on the panel to open the **Add annotation** dialog box.
   ![Add annotation dialog box](/static/img/docs/time-series-panel/time-series-annotations-add-region-annotation.gif)
1. Add an annotation description and tags (optional).
1. Click **Save**

### Edit an annotation

1. Hover over the annotation indicator at the bottom of the panel to open the tooltip.
1. Click the pencil icon to open the annotation dialog box.
1. Modify the description and tags.
1. Click **Save**.

### Delete an annotation

1. Hover over the annotation indicator at the bottom of the panel to open the tooltip.
1. Click the trash icon to open the annotation dialog box.

## Annotation queries

You can add new queries to fetch annotations using any data source, including the built-in data annotation data source. Annotation queries return events that can be visualized as event markers in graphs across the dashboard.

Check out the video below for a quick tutorial.

{{< youtube id="2istdJpPj2Y" >}}

### Add annotation queries

To add an annotation query to a dashboard, follow these steps:

{{< docs/list >}}

{{< shared id="add-annotation-query" >}}

1. Navigate to the dashboard you want to update.
1. Click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Annotation query**.
1. Enter a name for the annotation query.

   This name is given to the toggle that allows you to enable/disable showing annotation events from this query.

1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox.
1. Select a color for the annotation event markers.
1. Select one of the following options in the **Show annotation controls in** drop-down list to control where the annotation is displayed:
   - **Above dashboard** - The annotation toggle is displayed above the dashboard. This is the default.
   - **Controls menu** - The annotation toggle is displayed in the dashboard controls menu instead of above the dashboard. The dashboard controls menu appears as a button in the dashboard toolbar.
   - **Hidden** - The annotation toggle isn't displayed on the dashboard.

1. Select one of the following options in the **Show in** drop-down list to control the panels in which the annotation is displayed:
   - **All panels** - The annotations are displayed on all panels that support annotations.
   - **Selected panels** - The annotations are displayed on all the panels you select.
   - **All panels except** - The annotations are displayed on all panels except the ones you select.

   {{< figure src="/media/docs/grafana/dashboards/screenshot-annotation-filtering-10-v2.png" max-width="600px" caption="Annotation filtering" >}}

1. To add a query, do one of the following:
   - Click **Open query editor** to open the **Annotation Query** dialog box, select an option in the **Data source** drop-down list, and write or construct a query. The annotation query options are different for each data source. For information about annotations in a specific data source, refer to the specific [data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/) topic. Proceed to the next step.
   - Click **Use saved query** to open the **Saved queries** drawer. Choose a [saved query](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/#saved-queries) to reuse, click **Select query**, and proceed to step 13.

   {{< admonition type="note" >}}
   [Saved queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/#saved-queries) is only available on Grafana Enterprise and Grafana Cloud.
   {{< /admonition >}}

1. (Optional) Click **Test annotation query** to ensure that the query is working properly.
1. Click **Close** when you've completed the query setup.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

{{< /shared >}}

{{< /docs/list >}}

### Saved queries

{{< admonition type="note" >}}
Saved queries is only available on Grafana Enterprise and Grafana Cloud.
{{< /admonition >}}

You can reuse queries you and others in your organization have saved in annotations.
This helps users across your organization create annotations without having to create their own queries or know a query language.
It also helps you avoid having several users build the same queries for the same data sources multiple times.

Saved queries are supported in:

- [Dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/#create-a-dashboard)
- [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/explore/query-editor/)
- [Annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/annotate-visualizations/#add-new-annotation-queries)

Learn more about saved queries:

- [Saved queries dialog box](#saved-queries-dialog-box)
- [Roles, permissions, and RBAC](#roles-permissions-and-rbac)
- [Variables in saved queries](#variables-in-saved-queries)
- [Manage saved queries as code](#manage-saved-queries-as-code)
- [Known limitations](#known-limitations)

#### Saved queries dialog box

The **Saved queries** dialog box gives you access to all the saved queries in your organization:

{{< figure src="/media/docs/grafana/dashboards/screenshot-saved-queries-v13.0.png" max-width="750px" alt="List of saved queries" >}}

To access saved queries, click **Use saved query** in the annotations configuration.

{{< figure src="/media/docs/grafana/dashboards/screenshot-annotation-saved-query-v13.2.png" max-width="450px" alt="Access saved queries" >}}

{{< admonition type="note" >}}
To review your saved queries, press `Ctrl + K` or `Cmd + K` to open the command palette and search "Saved queries".
From this view, you can also select a query to open in Explore.
{{< /admonition >}}

From the **Saved queries** dialog box, you can:

- Search for queries by data source name, query content, title, or description.
- Sort queries alphabetically or by creation date.
- Filter by data source name, author name, and tags. The tag filter uses the `OR` operator, while the others use the `AND` operator. Use the **Remember filters** switch to persist your filter selections across sessions in your local storage.
- Star queries so that they appear in the **Starred queries** filter view.
- Duplicate or delete a saved query.
- Edit a query title, description, or tags.

You can apply all the same search, filter, and sort options in the **Starred queries** filter view.

{{< admonition type="tip">}}
When you select a query with a Loki, Mimir, Tempo, or Pyroscope data source, the **Saved queries** dialog box displays a **Drilldown** button.
Click the button to open the associated Drilldown app, while maintaining the context of the query.
Learn more about these apps in the [Drilldown documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/simplified-exploration/).
{{< /admonition >}}

#### Roles, permissions, and RBAC

Saved queries support role-based access controls.
By default, saved queries have two RBAC roles:

- **Writer**: Create, update, and delete all saved queries.
- **Reader**: Reuse saved queries.

If you used saved queries prior to the addition of RBAC support in Grafana v12.4, Grafana user roles are mapped as follows:

- Admin > Writer
- Editor > Writer
- Viewer > Reader

#### Variables in saved queries

If a saved query includes variables, you can substitute the variables in the query without modifying it.
This is useful in environments where variable names or available values differ between dashboards.

You can map the original variables to either:

- A variable in your dashboard
- A custom value that you enter

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-saved-query-variable-v13.0.png" max-width="450px" alt="A saved query with substituted variables" >}}

Grafana applies your selections to the query before inserting it into the dashboard.
However, the substitutions only apply to the query when it's reused, and the original saved query remains unchanged.

#### Manage saved queries as code

You can manage saved queries as code with the Grafana Terraform provider, which lets you version-control your query library and keep it consistent across instances.
For more information, refer to [Manage saved queries using Terraform](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/terraform/manage-saved-queries/).

#### Known limitations

- No validation is performed when you save a query, so it's possible to save an invalid query. You should confirm the query is working properly before you save it.
- You can save a maximum of 1000 queries.

## Built-in query

After you add an annotation, they are still visible. This is due to the built-in annotation query that exists on all dashboards. This annotation query fetches all annotation events that originate from the current dashboard, which are stored in Grafana, and show them on the panel where they were created. This includes alert state history annotations.

By default, the built-in annotation query uses the `-- Grafana --` special data source, and manual annotations are only supported using this data source. You can use another data source in the built-in annotation query, but you'll only be able to create automated annotations using the query editor for that data source.

To add annotations directly to the dashboard, this query must be enabled.

To confirm if the built-in query is enabled, take the following steps:

1. Click **Edit** in the top-right corner of the dashboard.
1. Click the **Dashboard options** icon in the toolbar.
1. In the sidebar, expand the **Annotations** section.
1. Expand the **Hidden** section of annotations.
1. Select the **Annotations & Alerts (Built-in)**.
1. Check if the **Enabled** checkbox is selected.

   If you don't want annotations to be fetched and drawn, clear the checkbox.

1. Click **Save**.
1. Enter an optional description of your changes and click **Save**.
1. Click **Exit edit**.

When you copy a dashboard using the **Save As** feature it gets a new dashboard id, so annotations created on the source dashboard is no longer be visible on the copy. You can still show them if you add a new **Annotation Query** and filter by tags. However, this only works if the annotations on the source dashboard had tags to filter by.

Following are some query options specific to the built-in annotation query.

### Filter queries by tag

You can create new queries to fetch annotations from the built-in annotation query using the `-- Grafana --` data source by setting _Filter by_ to `Tags`.

Grafana also supports typeahead of existing tags, provide at least one tag.

For example, create an annotation query name `outages` and specify a tag `outage`. This query shows all annotations (from any dashboard or via API) with the `outage` tag. If multiple tags are defined in an annotation query, then Grafana only shows annotations matching all the tags. To modify the behavior, enable `Match any`, and Grafana shows annotations that contain any one of the tags you provided.

{{< admonition type="warning" >}}
If you enable **Display annotations** on an externally shared dashboard that uses tag-based annotation queries, those queries return matching annotations from _all dashboards_ in the organization. This means annotations from dashboards that are not shared externally are visible to anyone with access to the shared dashboard. This is by design. Review which annotations might match your tags before enabling this option on a shared dashboard.
{{< /admonition >}}

{{< figure src="/media/docs/grafana/dashboards/screenshot-annotations-typeahead-support-10.0.png" max-width="600px" alt="Annotation query options" >}}

You can also use template variables in the tag query. This means if you have a dashboard showing stats for different services and a template variable that dictates which services to show, you can use the same template variable in your annotation query to only show annotations for those services.

{{< figure src="/media/docs/grafana/dashboards/screenshot-annotation-tag-filter-variable-10.0.png" max-width="600px" alt="Annotation query options with a template variable query tag" >}}

### Add time regions

When adding or editing an annotation, you can define a repeating time region by setting **Query type** to **Time regions**. Then, define the **From** and **To** sections with the preferred days of the week and time. You also have the option to change the timezone, which is set to the dashboard's timezone, by default.

{{< figure src="/media/docs/grafana/dashboards/screenshot-annotation-timeregions-10-v2.png" max-width="600px" alt="Time regions options set to business hours" >}}

The above configuration produces the following result in the Time series panel:

{{< figure src="/media/docs/grafana/screenshot-grafana-10-0-timeseries-time-regions.png" max-width="600px" alt="Time series visualization with time regions business hours" >}}

Toggle the **Advanced** switch and use [Cron syntax](https://en.wikipedia.org/wiki/Cron) to set more granular time region controls. The following example sets a time region of 9:00 AM, Monday to Friday:

{{< figure src="/media/docs/grafana/dashboards/screenshot-annotations-cron-option-v11.6.png" max-width="600px" alt="Time region query with cron syntax" >}}

# Grafana dashboard best practices

This section provides information about best practices for intermediate Grafana administrators and users about how to build and maintain Grafana dashboards.

For more information about the different kinds of dashboards you can create, refer to [Grafana dashboards: A complete guide to all the different types you can build](https://grafana.com/blog/2022/06/06/grafana-dashboards-a-complete-guide-to-all-the-different-types-you-can-build/?pg=webinar-getting-started-with-grafana-dashboard-design-amer&plcmt=related-content-1).

## Common observability strategies

When you have a lot to monitor, like a server farm, you need a strategy to decide what is important enough to monitor. This page describes several common methods for choosing what to monitor.

A logical strategy allows you to make uniform dashboards and scale your observability platform more easily.

### Guidelines for usage

- The USE method tells you how happy your machines are, the RED method tells you how happy your users are.
- USE reports on causes of issues.
- RED reports on user experience and is more likely to report symptoms of problems.
- The best practice of alerting is to alert on symptoms rather than causes, so alerting should be done on RED dashboards.

### USE method

USE stands for:

- **Utilization -** Percent time the resource is busy, such as node CPU usage
- **Saturation -** Amount of work a resource has to do, often queue length or node load
- **Errors -** Count of error events

This method is best for hardware resources in infrastructure, such as CPU, memory, and network devices. For more information, refer to [The USE Method](http://www.brendangregg.com/usemethod.html).

### RED method

RED stands for:

- **Rate -** Requests per second
- **Errors -** Number of requests that are failing
- **Duration -** Amount of time these requests take, distribution of latency measurements

This method is most applicable to services, especially a microservices environment. For each of your services, instrument the code to expose these metrics for each component. RED dashboards are good for alerting and SLAs. A well-designed RED dashboard is a proxy for user experience.

For more information, refer to Tom Wilkie's blog post [The RED method: How to instrument your services](https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services).

### The Four Golden Signals

According to the [Google SRE handbook](https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/#xref_monitoring_golden-signals), if you can only measure four metrics of your user-facing system, focus on these four.

This method is similar to the RED method, but it includes saturation.

- **Latency -** Time taken to serve a request
- **Traffic -** How much demand is placed on your system
- **Errors -** Rate of requests that are failing
- **Saturation -** How "full" your system is

{{< docs/play title="The Four Golden Signals" url="https://play.grafana.org/d/000000109/" >}}

## Dashboard management maturity model

_Dashboard management maturity_ refers to how well-designed and efficient your dashboard ecosystem is. It's recommended that you periodically review your dashboard setup to gauge where you are and how you can improve.

Broadly speaking, dashboard maturity can be defined as low, medium, or high.

Much of the content for this topic was taken from the KubeCon 2019 talk [Fool-Proof Kubernetes Dashboards for Sleep-Deprived Oncalls](https://www.youtube.com/watch?v=YE2aQFiMGfY).

### Low - default state

At this stage, you have no coherent dashboard management strategy. Almost everyone starts here.

How can you tell you are here?

- Everyone can modify your dashboards.
- Lots of copied dashboards, little to no dashboard reuse.
- One-off dashboards that hang around forever.
- No version control (dashboard JSON in version control).
- Lots of browsing for dashboards, searching for the right dashboard. This means lots of wasted time trying to find the dashboard you need.
- Not having any alerts to direct you to the right dashboard.

### Medium - methodical dashboards

At this stage, you are starting to manage your dashboard use with methodical dashboards. You might have laid out a strategy, but there are some things you could improve.

How can you tell you are here?

- Prevent sprawl by using template variables. For example, you don't need a separate dashboard for each node, you can use query variables. Even better, you can make the data source a template variable too, so you can reuse the same dashboard across different clusters and monitoring backends.

  Refer to the list of [Variable examples](ref:variable-examples) if you want some ideas.

- Methodical dashboards according to an [observability strategy](#common-observability-strategies).
- Hierarchical dashboards with drill-downs to the next level.

  {{< figure class="float-right"  max-width="100%" src="/static/img/docs/best-practices/drill-down-example.png" caption="Example of using drill-down" >}}

- Dashboard design reflects service hierarchies. The example shown below uses the RED method (request and error rate on the left, latency duration on the right) with one row per service. The row order reflects the data flow.

  {{< figure class="float-right"  max-width="100%" src="/static/img/docs/best-practices/service-hierarchy-example.png" caption="Example of a service hierarchy" >}}

- Compare like to like: split service dashboards when the magnitude differs. Make sure aggregated metrics don't drown out important information.
- Expressive charts with meaningful use of color and normalizing axes where you can.
  - Example of meaningful color: Blue means it's good, red means it's bad. [Thresholds](ref:thresholds) can help with that.
  - Example of normalizing axes: When comparing CPU usage, measure by percentage rather than raw number, because machines can have a different number of cores. Normalizing CPU usage by the number of cores reduces cognitive load because the viewer can trust that at 100% all cores are being used, without having to know the number of CPUs.
- Directed browsing cuts down on "guessing."
  - Template variables make it harder to "just browse" randomly or aimlessly.
  - Most dashboards should be linked to by alerts.
  - Browsing is directed with links. For more information, refer to [Manage dashboard links](ref:manage-dashboard-links).
- Version-controlled dashboard JSON.

### High - optimized use

At this stage, you have optimized your dashboard management use with a consistent and thoughtful strategy. It requires maintenance, but the results are worth it.

- Actively reducing sprawl.
  - Regularly review existing dashboards to make sure they are still relevant.
  - Only approved dashboards added to master dashboard list.
  - Tracking dashboard use. If you're an Enterprise user, you can take advantage of [Usage insights](ref:usage-insights).
- Consistency by design.
- Use scripting libraries to generate dashboards, ensure consistency in pattern and style.
  - grafonnet (Jsonnet)
  - grafanalib (Python)
- No editing in the browser. Dashboard viewers change views with variables.
- Browsing for dashboards is the exception, not the rule.
- Perform experimentation and testing in a separate Grafana instance dedicated to that purpose, not your production instance. When a dashboard in the test environment is proven useful, then add that dashboard to your main Grafana instance.

## Best practices for creating dashboards

This page outlines some best practices to follow when creating Grafana dashboards.

### Before you begin

Here are some principles to consider before you create a dashboard.

#### A dashboard should tell a story or answer a question

What story are you trying to tell with your dashboard? Try to create a logical progression of data, such as large to small or general to specific. What is the goal for this dashboard? (Hint: If the dashboard doesn't have a goal, then ask yourself if you really need the dashboard.)

Keep your graphs simple and focused on answering the question that you are asking. For example, if your question is "which servers are in trouble?", then maybe you don't need to show all the server data. Just show data for the ones in trouble.

#### Dashboards should reduce cognitive load, not add to it

<!-- vale Grafana.GoogleWill = NO -->

_Cognitive load_ is basically how hard you need to think about something in order to figure it out. Make your dashboard easy to interpret. Other users and future you (when you're trying to figure out what broke at 2 AM) will appreciate it.

Ask yourself:

- Can I tell what exactly each graph represents? Is it obvious, or do I have to think about it?
- If I show this to someone else, how long will it take them to figure it out? Will they get lost?

<!-- vale Grafana.GoogleWill = YES -->

#### Have a monitoring strategy

It's easy to make new dashboards. It's harder to optimize dashboard creation and adhere to a plan, but it's worth it. This strategy should govern both your overall dashboard scheme and enforce consistency in individual dashboard design.

Refer to [Common observability strategies](#common-observability-strategies) and [Dashboard management maturity levels](#dashboard-management-maturity-model) for more information.

#### Write it down

Once you have a strategy or design guidelines, write them down to help maintain consistency over time. Check out this [Wikimedia runbook example](https://wikitech.wikimedia.org/wiki/Performance/Runbook/Grafana_best_practices).

### Best practices to follow

- When creating a new dashboard, make sure it has a meaningful name.
  - If you are creating a dashboard to play or experiment, then put the word `TEST` or `TMP` in the name.
  - Consider including your name or initials in the dashboard name or as a tag so that people know who owns the dashboard.
  - Remove temporary experiment dashboards when you are done with them.
- If you create many related dashboards, think about how to cross-reference them for easy navigation. Refer to [Best practices for managing dashboards](#best-practices-for-managing-dashboards) for more information.
- Grafana retrieves data from a data source. A basic understanding of [data sources](ref:data-sources) in general and your specific is important.
- Avoid unnecessary dashboard refreshing to reduce the load on the network or backend. For example, if your data changes every hour, then you don't need to set the dashboard refresh rate to 30 seconds.
- Use the left and right Y-axes when displaying time series with different units or ranges.
- Add documentation to dashboards and panels.
  - To add documentation to a dashboard, add a [Text panel visualization](ref:text-panel-visualization) to the dashboard. Record things like the purpose of the dashboard, useful resource links, and any instructions users might need to interact with the dashboard. Check out this [Wikimedia example](https://grafana.wikimedia.org/d/000000066/resourceloader?orgId=1).
  - To add documentation to a panel, edit the panel settings and add a description. Any text you add appears if you hover your cursor over the small `i` in the top left corner of the panel.
- Reuse your dashboards and enforce consistency by using [templates and variables](ref:templates-and-variables).
- Be careful with stacking graph data. The visualizations can be misleading, and hide important data. It's recommended that you turn it off in most cases.

## Best practices for managing dashboards

This page outlines some best practices to follow when managing Grafana dashboards.

### Before you begin

Here are some principles to consider before you start managing dashboards.

#### Strategic observability

There are several [common observability strategies](#common-observability-strategies). You should research them and decide whether one of them works for you or if you want to come up with your own. Either way, have a plan, write it down, and stick to it.

Adapt your strategy to changing needs as necessary.

#### Maturity level

What is your dashboard maturity level? Analyze your current dashboard setup and compare it to the [Dashboard management maturity model](#dashboard-management-maturity-model). Understanding where you are can help you decide how to get to where you want to be.

### Best practices to follow

- Avoid dashboard sprawl, meaning the uncontrolled growth of dashboards. Dashboard sprawl negatively affects time to find the right dashboard. Duplicating dashboards and changing "one thing" (worse: keeping original tags) is the easiest kind of sprawl.
  - Periodically review the dashboards and remove unnecessary ones.
  - If you create a temporary dashboard, perhaps to test something, prefix the name with `TEST:`. Delete the dashboard when you are finished.
- Copying dashboards with no significant changes is not a good idea.
  - You miss out on updates to the original dashboard, such as documentation changes, bug fixes, or additions to metrics.
  - In many cases copies are being made to simply customize the view by setting template parameters. This should instead be done by maintaining a link to the master dashboard and customizing the view with [URL parameters](ref:url-parameters).
- When you must copy a dashboard, clearly rename it and _do not_ copy the dashboard tags. Tags are important metadata for dashboards that are used during search. Copying tags can result in false matches.
- Maintain a dashboard of dashboards or cross-reference dashboards. This can be done in several ways:
  - Create dashboard links, panel, or data links. Links can go to other dashboards or to external systems. For more information, refer to [Manage dashboard links](ref:manage-dashboard-links).
  - Add a [Dashboard list panel](ref:dashboard-list-panel). You can then customize what you see by doing tag or folder searches.
  - Add a [Text panel](ref:text-panel) and use markdown to customize the display.

# Dashboard controls

In the **Dashboard controls** section of the sidebar, you can add variables, annotation queries, dashboard links, and controls to filter and group data, without leaving the dashboard.

{{< figure src="/media/docs/grafana/dashboards/screenshot-dashboard-controls-v13.1.png" max-width="400px" alt="Dashboard controls in the Dashboard options sidebar view" >}}

## Add filters and group by controls

<!-- vale Grafana.WordList = NO -->
<!-- vale Grafana.Spelling = NO -->

{{< admonition type="note" >}}
The **Filter and Group by** feature renames the **Ad hoc filters** variable and extends it by adding grouping for Prometheus and Loki data sources.
However, in the dashboard schema, it's still referred to as `"kind": "AdhocVariable"` under the `variables` schema property.
{{< /admonition >}}

<!-- vale Grafana.WordList = YES -->
<!-- vale Grafana.Spelling = YES -->

{{< shared-snippet path="/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/filter-group-by/index.md" id="add-filter-group-by" >}}

For more detailed information on the **Filter and Group by** option, refer to the full [Filter and group by controls documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/filter-group-by/).

## Add variables

To add a variable, follow these steps:

{{< docs/list >}}

{{< shared-snippet path="/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/index.md" id="add-variable" >}}

1. Complete the variable configuration. For more detailed configuration information, click the one of the following links to complete the steps for adding your selected variable type:

{{< column-list >}}

- [Query](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-query-variable)
- [Custom](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-custom-variable)
- [Textbox](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-text-box-variable)
- [Constant](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-constant-variable)
- [Data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-data-source-variable)
- [Interval](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-an-interval-variable)
- [Switch](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-switch-variable)

{{< /column-list >}}

1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

{{< /docs/list >}}

For more detailed information on variables, refer to the full [Variables documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/).

## Add annotation queries

To add an annotation query, follow these steps:

{{< shared-snippet path="/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/annotate-visualizations/index.md" id="add-annotation-query" >}}

For more detailed information about annotations, refer to the full [Annotations documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/annotate-visualizations/).

## Add links

For more detailed information on dashboard links, refer to the full [Dashboard links documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/manage-dashboard-links/).

### Add links to dashboards

To add a link to another dashboard at the top of your current dashboard, follow these steps:

{{< docs/list >}}

1. Navigate to the dashboard you want to update.
1. Click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Link**.

{{< shared-snippet path="/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/manage-dashboard-links/index.md" id="add-dashboard-link" >}}

1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

{{< /docs/list >}}

### Add a URL link to a dashboard

Add a link to a URL at the top of your current dashboard.
You can link to any available URL, including dashboards, panels, or external sites.
You can even control the time range to ensure the user is zoomed in on the right data in Grafana.

To add a URL link to your dashboard, follow these steps:

{{< docs/list >}}

1. Navigate to the dashboard you want to update.
1. Click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Link**.

{{< shared-snippet path="/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/manage-dashboard-links/index.md" id="url-dashboard-link" >}}

1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

{{< /docs/list >}}

## Manage dashboard controls

After you add dashboard controls, you can manage them from the dashboard options.
In this view, the sidebar includes collapsible sections for filters, variables, annotations, and links, including hidden controls that aren't otherwise visible on the dashboard:

{{< figure src="/media/docs/grafana/dashboards/screenshot-dashboard-controls-mgmt-v13.1.png" max-width="450px" alt="Dashboard controls in the Dashboard options sidebar view" >}}

To manage dashboard controls, follow these steps:

1. Click the **Dashboard options** icon in the sidebar.
1. In the sidebar, expand the appropriate collapsible section.
1. Do one or more of the following:
   - **Edit**: Click the **Edit** icon on the control to open it in the sidebar so you can make updates.
   - **Reorder**: Drag and drop controls to reorder them.
   - **Change display**: Drag and drop controls between sub-sections **Above dashboard**, **Controls menu**, and **Hidden** to update the control display option. Note that links can't be hidden.

   {{< admonition type="tip" >}}
   To get back to controls management from configuration, click the **Dashboard options** icon.
   {{< /admonition >}}

1. Make any other changes as needed.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

# Dashboard panel groupings

To help create meaningful sections in your dashboard, you can group panels into rows or tabs.
Rows and tabs let you break up big dashboards or make one dashboard out of several smaller ones.

You can think of the dashboard as a series of nested containers: the dashboard is the largest container and it contains panels, rows, or tabs.
Rows and tabs are the next largest containers, and they contain panels or other rows and tabs.

You can nest:

- Rows in a row
- Rows in a tab
- Tabs in a row

You can't nest tabs directly inside a tab, but you can nest them inside a row that's within a tab (tab > row > tabs).

You can nest up to four levels deep, which means a dashboard can have a maximum of six configuration levels:

- Dashboard
- Grouping 1 - Row or tab
- Grouping 2 - Row or tab
- Grouping 3 - Row or tab
- Grouping 4 - Row or tab
- Panels

The following screenshot demonstrates how groupings work:

{{< image-map key="panel-groupings" >}}

In addition to the option to add groupings from the sidebar, when you hover your cursor over relevant parts of the dashboard, **+ Add panel**, **+ New row**, **+ New tab**, **Group panels**, and ungroup buttons are visible.

To see a repeating row in action, explore this demo dashboard on Grafana Play:

{{< docs/play title="Repeat rows" url="https://play.grafana.org/d/000000153/" >}}

The following sections describe:

- [Grouping configuration options](#grouping-configuration-options)
- [Grouping layouts](#grouping-layouts)
- [How to group panels](#group-panels)
- [How to ungroup panels](#ungroup-panels)

## Grouping configuration options

The following table describes the options you can set for a row or tab:

<!-- prettier-ignore-start -->

| Option          | Description                                                                 |
| ----------------| --------------------------------------------------------------------------- |
| Title           | Title of the row or tab.                                                    |
| Fill screen     | Toggle the switch on to make the row fill the screen. Rows only. |
| Hide row header | Toggle the switch on to hide row headers in view mode. In edit mode, the row header is visible, but crossed out with the hidden icon next to it. Rows only. |
| Filters       | Add filters that apply to only the panels in the grouping. For more information, refer to [Section-level variables and filters](#grouping-level-variables-and-filters). For information on configuring the **Filter and Group by** feature, refer to the [documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/dashboard-controls/#add-filters-and-group-by-controls). |
| Variables       | Add variables that apply to only the panels in the grouping. For more information, refer to [Section-level variables and filters](#grouping-level-variables-and-filters). For information on configuring variables, refer to [Add variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/dashboard-controls/#add-variables). |
| Layout          | Select the layout. If the grouping contains another grouping, choose from **Rows** or **Tabs**. If the grouping contains panels, choose from **Custom** or **Auto**. For more information, refer to [Panel layouts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/#panel-layouts) or [Grouping layouts](#grouping-layouts). |
| Repeat options > [Repeat by variable](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/#configure-repeat-options) | Configure the dashboard to dynamically add panels, rows, or tabs based on the value of a variable. |
| Show / hide rules > [Panel/Row/Tab visibility](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/#configure-showhide-rules) | Control whether or not panels, rows, or tabs are displayed based on variable values, a time range, or query results (panels only). |

<!-- prettier-ignore-end -->

## Grouping layouts

When you have panels grouped into rows or tabs, the **Layout** options available depend on which dashboard element is selected and the nesting level of that element.

You can nest up to four levels deep, which means a dashboard can have a maximum of six configuration levels, with the following layout options:

- **Dashboard**: Layout options allow you to choose between rows and tabs or custom and auto layouts.
- **Grouping 1 (outer)**: Layout options allow you to choose between rows and tabs.
- **Grouping 2 (middle)**: Layout options allow you to choose between rows and tabs.
- **Grouping 3 (middle)**: Layout options allow you to choose between rows and tabs.
- **Grouping 4 (inner)**: Layout options allow you to choose between custom and auto grid (refer to [Panel layouts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/#panel-layouts)).
- **Panels**: No layout options

You can switch between rows and tabs or update the panel layout by clicking the parent container and changing the layout selection.

## Group panels

You can group all the panels on a dashboard or select specific ones.

### Group all panels

To group all the panels on a dashboard, follow these steps:

1. Navigate to the dashboard you want to update.
1. Click **Edit**.
1. Do one of the following:
   - Click the **Add new element** icon (blue plus sign) and select **Group into rows** or **Group into tabs**.
   - Under a panel or group of panels, hover your mouse to reveal the **Group panels** button, click it and select **Group into row** or **Group into tab**.

   All the panels are moved into the grouping, and a dotted blue line surrounds the row or tab.
   The sidebar opens, displaying the relevant options.

   {{< admonition type="tip" >}}
   While grouping is typically used for multiple panels, you can start a grouping with just one panel.
   {{< /admonition >}}

1. Set the [grouping configuration options](#grouping-configuration-options) in the sidebar.
1. (Optional) Add one or both of the following:
   - Other [groupings at the same level](#add-more-groupings-at-the-same-level).
   - A [nested grouping](#add-nested-groupings)

1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

### Group select panels

To group select panels on a dashboard:

1. Navigate to the dashboard you want to update.
1. Click **Edit**.
1. Press the `Shift` key and select all the panels you want to group.
   A dotted blue line surrounds the selected panels.
   You can select panels that are already in other groupings.

1. In the sidebar, select **Group into row** or **Group into tab**.
   The panels you've selected are grouped and any remaining panels are added to another grouping of the same type.

1. Set the [grouping configuration options](#grouping-configuration-options) in the sidebar.
1. (Optional) Add one or both of the following:
   - Other [groupings at the same level](#add-more-groupings-at-the-same-level).
   - A [nested grouping](#add-nested-groupings)

1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

### Add more groupings at the same level

To add more groupings at the same level, follow these steps:

1. Click the dashboard or grouping where you want to more groupings, and click **+ New row** or **+ New tab** (only one option will be available).
1. Set the configuration options for the new grouping.
1. Click **+ Add panel** to begin adding panels to the new grouping.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

### Add nested groupings

To add a nested grouping, follow these steps:

1. Click the grouping where you want to add the nested grouping.
1. Under a panel or group of panels in the grouping, click **Group panels** and select **Group into row** or **Group into tab** (**Group into tab** is only available if the parent grouping is a row).

   {{< figure src="/media/docs/grafana/dashboards/screenshot-nest-group-v13.0.png" alt="Adding a nested grouping" max-width="500px" >}}

   The new grouping is added inside the first grouping, and the panels are moved into the nested grouping.
   The sidebar opens displaying the relevant configuration options.

1. Set the configuration options for the nested grouping.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

You can add more one more level of grouping if needed.

## Ungroup panels

You can ungroup some or all of the dashboard groupings without losing your panels.

What happens when you ungroup rows or tabs depends on what they contain:

- **Groupings that contain only panels**: The grouping level is removed and the panels are merged into the next higher-level grouping. If there are no groupings left, the panels are moved onto the dashboard.
- **Groupings that contain nested groupings**: Only the grouping level you're ungrouping is removed. The nested groupings move up a level, and any groupings at the same level that contain only panels are converted to the same grouping type, so the structure of your dashboard content is preserved.

When the groupings you're ungrouping contain nested groupings, the following rules apply:

- If the nested groupings are all rows or all tabs, Grafana automatically converts everything at that level to that grouping type.
- If the nested groupings are a mix of rows and tabs, or if converting to tabs isn't possible because tabs can't be directly nested inside a tab, you're prompted to choose between **Convert to rows** and **Convert to tabs**.
- If any of the ungrouped groupings have repeat options configured, those repeat options are lost. Grafana warns you before making the change.

Section-level variables and filters configured on the ungrouped groupings aren't lost; they move up to the next level.

{{< image-map key="ungrouping" >}}

The **Ungroup rows** and **Ungroup tabs** buttons are only visible when you hover your mouse over the relevant part of the dashboard

{{< admonition type="caution" >}}
If you delete a grouping, rather than ungrouping it, its panels are deleted as well.
{{< /admonition >}}

To remove groupings, follow these steps:

1. Navigate to the dashboard you want to update.
1. Click **Edit**.
1. (Optional) Click the **Content outline** icon to quickly navigate to the grouping you want to remove.
1. Hover your mouse over the relevant area to show the **Ungroup rows** or **Ungroup tabs** button, then click it.
1. If you've ungrouped panels that were previously in different panel layouts, you'll be prompted to select a common layout type for all the panels; click **Convert to Auto** or **Convert to Custom**.
1. If the ungrouped groupings contain a mix of nested rows and tabs, you'll be prompted to select how to convert them; click **Convert to rows** or **Convert to tabs**.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## Drag tabs between rows

In dashboards that include multiple rows with nested tabs, you can drag and drop nested tabs between rows.
To do this, the receiving row just needs to have one nested tab already.

## Place panels outside of groupings with headerless rows

If you have a row with panels, but you want the panels to appear ungrouped, you can use the **Hide row header** switch in the row configuration to achieve this.

In view mode, the row header is hidden, so that the panels in that row appear ungrouped.
In the following image, the panels are grouped into two rows, but the header of the second row is hidden:

{{< figure src="/media/docs/grafana/dashboards/screenshot-headerless-row-v13.1.png" max-width="750px" alt="Dashboard including a row with a hidden header" >}}

When you hide the header of a row, you can't collapse the row.

## Section-level variables and filters {#grouping-level-variables-and-filters}

You can add variables and filters to groupings that apply only to the panels in that grouping.

For example, if your dashboard includes both an API gateway and a database, you might want to apply two different `$instance` variables.
Section-level variables and filters for address this by letting each row or tab have its own independent scope.
In the API gateway and database scenario, the API gateway grouping can use one set of instances, while a database grouping uses another set.
However, both groupings still share the same time range, and the underlying dashboard remains unchanged.

The following image shows an example using two rows of panels:

{{< figure src="/media/docs/grafana/dashboards/screenshot-section-variables-v13.1.png" max-width="750px" alt="A dashboard with two rows, each with its own variable filter above the panels" >}}

Panels in the grouping resolve section-level variables and filters first, then fall back to dashboard-level variables.

The panel query editor is context-aware, so the autocomplete only shows the variables available to the panel you're editing.
Also, section-level variables and filters carry over when you convert between rows and tabs, change layouts, and work with repeating rows and tabs.
When you ungroup rows or tabs, their section-level variables and filters move up to the next level.

# Dashboard URL variables

Dashboard URL [variables](ref:variables) allow you to provide more context when you share a dashboard URL.

For example, you could share a basic URL to your dashboard that looks like this:

```
https://${your-domain}/path/to/your/dashboard
```

This allows someone to navigate to the dashboard, but doesn't provide any helpful context that might be available.

Instead, you can add dashboard variables, passed as query parameters in the dashboard URL, to provide a URL like this:

```
https://${your-domain}/path/to/your/dashboard?var-example=value
```

This allows you to provide added context to the dashboard when someone navigates to it.

## Variables as query parameters

Grafana interprets query string parameters prefixed with `var-` as variables in the given dashboard.

For example:

```
https://${your-domain}/path/to/your/dashboard?var-example=value
```

In this URL, the query parameter `var-example=value` represents the dashboard variable `example` with a value of `value`.

### Multiple values for a variable

To pass multiple values, repeat the variable parameter once for each value:

```
https://${your-domain}/path/to/your/dashboard?var-example=value1&var-example=value2
```

Grafana interprets `var-example=value1&var-example=value2` as the dashboard variable `example` with two values: `value1` and `value2`.

### Example

[This dashboard in Grafana Play](https://play.grafana.org/d/000000074/alerting?var-app=backend&var-server=backend_01&var-server=backend_03&var-interval=1h) passes the variable `server` with multiple values, and the variables `app` and `interval` with a single value each.

## Filters and group by dimensions

Filters automatically query your data source for available dimensions and lets users add or remove filters and group by dimensions on the dashboard dynamically.
This allows you to quickly apply filters dashboard-wide to all metric queries that use the specified data source.
The group by function allows you to then group data by keys, letting you split it up.

For more information, refer to [Filter and Group by documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/filter-group-by).

To pass a filter as a query parameter, use the variable syntax to pass the filter. Then provide the key, operator, and value as a pipe-separated list.

For example:

```
https://${your-domain}/path/to/your/dashboard?var-filter=example_key|=|example_value
```

In this URL, the query parameter `var-filter=key|=|value` applies the configured filter using the `example_key` key, the `=` operator, and the `example_value` value.

{{< admonition type="note" >}}
When sharing URLs with filters, remember to encode the URL. In the preceding example, replace the pipes (`|`) with `%7C` and the equality operator (`=`) with `%3D`.
{{< /admonition >}}

### Example

[This dashboard in Grafana Play](https://play.grafana.org/d/canvvlh/filter-variables?orgId=1&from=now-1h&to=now&var-filter=job%7C%3D%7Cecommerce-prod%2Fcartservice) passes the filter variable `filter` with the filter value `job = ecommerce-prod/cartservice`.

## Time range control using the URL

{{< docs/shared lookup="dashboards/time-range-URLs.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Variables in dashboard links

When you create dashboard links the dashboard settings, you can have current dashboard variables included in the link by selecting that option:

{{< figure src="/media/docs/grafana/dashboards/screenshot-dashboard-link-variables-11.1.png" max-width="500px" alt="Dashboard link page with variables option selected" >}}

For steps to add variables to dashboard links, refer to [Manage dashboard links](ref:manage-dashboard-links).

# Filter and group by controls

<!-- vale Grafana.WordList = NO -->
<!-- vale Grafana.Spelling = NO -->

{{< admonition type="note" >}}
The **Filter and Group by** feature renames the **Ad hoc filters** variable and extends it by adding grouping for Prometheus and Loki data sources.
However, in the dashboard schema, it's still referred to as `"kind": "AdhocVariable"` under the `variables` schema property.
{{< /admonition >}}

<!-- vale Grafana.WordList = YES -->
<!-- vale Grafana.Spelling = YES -->

The **Filter and Group by** option is one of the most complex and flexible dashboard controls available.
Instead of creating a variable for each dimension by which you want to filter, it automatically queries your data source for available dimensions and lets users add or remove filters and group by dimensions on the dashboard dynamically.
This allows you to quickly apply filters dashboard-wide.

The group by function allows you to then group data by keys, letting you split it up.
This function is typically used with aggregation queries, such as `sum(your_metric_here)`, to split aggregated results by the selected dimensions.
Then, you can use filters within panels to filter data in or out, drilling down further into the data.

The filter and group by feature lets you add label/value filter pairs that are automatically added to all queries that use the specified data source.
Unlike variables, you don't use these filters in queries.
Instead, you use them to write filters for existing queries.

To try filter and group by controls with a Prometheus data source, explore this demo dashboard on Grafana Play:

{{< docs/play title="Filter variables" url="https://play.grafana.org/d/canvvlh/" >}}

The following data sources support filters.
Data sources with an asterisk also support the group by function:

- Prometheus\*
- Loki\*
- InfluxDB
- Elasticsearch
- OpenSearch.
- Special Dashboard data source - Use this special data source to [apply filters to data from unsupported data sources](#filter-any-data-using-the-dashboard-data-source).

## Add filters and group by controls

{{< shared id="add-filter-group-by" >}}

<!-- prettier-ignore-start -->

To add filters and group by controls, follow these steps:

1. Navigate to the dashboard you want to update.
1. Click **Edit**.
1. Click the **Add new element** icon (blue plus sign).
1. Click **Filter and Group by**.
1. Enter a **Name** for the filter.
1. (Optional) In the **Label** field, enter the display name for the filter drop-down list.

   If you don't enter a display name, then the drop-down list label is the filter name.

1. (Optional) In the **Description** field, enter a description of the filter. The description appears as an info icon tooltip next to the filter name on the dashboard.

   Descriptions support links. You can use Markdown-style links (`[link text](https://example.com)`) or paste bare URLs (`https://example.com`). Only `http` and `https` URLs are rendered as clickable links—other protocols are displayed as plain text.

1. Choose a **Display** option:
   - **Above dashboard**: The filter drop-down list displays above the dashboard with the filter **Name** or **Label** value. This is the default.
   - **Above dashboard, label hidden**: The filter drop-down list displays above the dashboard, but without showing the name of the filter.
   - **Controls menu**: The filter is displayed in the dashboard controls menu instead of above the dashboard. The dashboard controls menu appears as a button in the dashboard toolbar.
   - **Hidden**: No filter drop-down list is displayed on the dashboard.

1. Under the **Filter options** section of the sidebar, set the following options:

   | Option                    | Description                 |
   | ------------------------- | --------------------------- |
   | Data source               | Select a target data source in the drop-down list. You can also click **Open advanced data source picker** to see more options, including adding a data source (Admins only). For more information about data sources, refer to [Add a data source](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/data-sources/).                 |
   | Default filters           | Set a default key/value pair. Optional. In the dashboard filter control, the default value is indicated with an information icon.|
   | Enable group by           | This option only appears if you selected a Prometheus or Loki data source. Toggle the switch on to enable data grouping. |
   | Default group by          | Set a default key for the dashboard. Optional. In the dashboard filter control, the default value is indicated with an information icon.                                         |
   | Use static key dimensions | To provide the filter dimensions as comma-separated values (CSV), toggle the switch on, and then enter the values in the space provided. Optional.                                       |
   | Allow custom values       | Toggle the switch on to allow dashboard users to add custom values to the filter and group by lists. Optional. |

1. Click **Save**.
1. Enter an optional description of your dashboard changes, and then click **Save**.
1. Click **Exit edit**.

<!-- prettier-ignore-end -->

{{< /shared >}}

Now you can filter and group data on the dashboard.

You can remove and reset default filters and group by dimensions, and see your recent ones:

{{< figure src="/media/docs/grafana/dashboards/screenshot-reset-default-v13.0.png" max-width="500px" alt="Dashboard with the filters and group by selections" caption="_Reset default filters and group by selections_" >}}

{{< figure src="/media/docs/grafana/screenshot-filters-group-recent-v13.0.png" max-width="500px" alt="Dashboard with the filters and group by selections" caption="_Recent filters and group by dimensions_" >}}

To see all active filters and group by dimensions across the dashboard all at once, click the **Filters overview** icon (filter) in the toolbar to open an overview.
The overview lets you search for specific keys, and adjust them, without scrolling through the dashboard controls:

{{< figure src="/media/docs/grafana/screenshot-filters-overview-v12.0.png" max-width="500px" alt="Dashboard with the filters and group by dimensions" >}}

Add an operator and value for a key to add it as a filter or select the **GroupBy** checkbox to set a group by dimension.
You can use a key for both a filter and a group by.

## Cross-filtering {#dashboard-drilldown-with-filters}

In some visualizations, you can apply filters directly from the visualization.
Click on a series in the panel and then click **Filter on this value** or **Filter out this value**.
This filters by the labels found on that series:

{{< figure src="/media/docs/grafana/dashboards/screenshot-panel-filters-v13.0.png" max-width="675px" alt="Panel with tooltip open showing options to filter on a value or filter it out" >}}

To enable this functionality, you need to add an override for every field by which you want to filter.
In the override:

- Select a field name.
- Add the **Standard options > Filter from values** override property.
- Enable the **Filter from values** switch to allow viewers to update dashboard filters by interacting with the panel.

You can also do this programmatically by returning the data frame with the appropriate `filterable` property on the desired fields.

For provisioned dashboards, **Filter from values** uses the JSON property ID `filterable`.
For example, add the following override to the panel's `fieldConfig.overrides` array to enable filtering from the `service` field:

```json
{
  "matcher": { "id": "byName", "options": "service" },
  "properties": [{ "id": "filterable", "value": true }]
}
```

Replace `service` with the name of the field you want to filter by.
This requires a supported visualization and data source support for dashboard filters.
The standard `filterable` property updates dashboard filters.

Cross-filtering is supported for the following visualizations:

| Visualization  | Requires Group by |
| -------------- | ----------------- |
| Bar chart      | no                |
| Histogram      | yes               |
| State timeline | yes               |
| Status history | yes               |
| Table          | no                |
| Time series    | yes               |

### Apply cross-filtering

To apply cross-filtering, follow these steps:

1. To display the filter icons, hover your cursor over the data point with the value for which you want to filter. In this example, the table cell value is `ConfigMap Updated`, which is in the `alertname` column:

   {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-adhoc-filter-icon-v12.2.png" max-width="550px" alt="Table and bar chart with a filter icon displayed on a table cell" >}}

   In bar chart visualizations, hover and click the bar to display the filter button:

   {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-adhoc-filter-icon-bar-v12.2.png" max-width="300px" alt="The filter button in a bar chart tooltip">}}

1. Click the add filter icon.

   The filter pair `alertname = ConfigMap Updated` is added, and all panels using the same data source that include that value are filtered by it:

   {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-adhoc-filter-applied-v12.2.png" max-width="550px" alt="Table and bar chart, filtered" >}}

If one of the panels in the dashboard using that data source doesn't include that filter value, the panel won't return any data. In this example, the filter pair `_name_ = ALERTS` has been added so the bar chart doesn't return any results:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-adhoc-filter-no-data-v12.2.png" max-width="650px" alt="Table, filtered and bar chart returning no results" >}}

For some visualizations, if your query doesn't include some type of data grouping, you might need to group the data first.
For more information, refer to [Group from the panel](#group-from-the-panel).

In cases where the data source you're using doesn't support filtering, consider using the special Dashboard data source.
For more information, refer to [Filter any data using the Dashboard data source](#filter-any-data-using-the-dashboard-data-source).

### Group from the panel

When the **Group by** switch is toggled on, you can also set a group by dimension from a panel rather than from the dashboard-level control.
Hover the cursor over any panel using the data source of the filter to show the **Group by** selector:

{{< figure src="/media/docs/grafana/dashboards/screenshot-panel-groupby-v13.0.png" max-width="550px" alt="Group by control on a panel" >}}

The panel-level **Group by** control only includes keys available in the panel's query. This is in contrast with the dashboard-level control that includes all available keys for the dashboard.
Your selection is applied to all the panels in the dashboard with the same data source.

## Filter any data using the Dashboard data source

In cases where a data source doesn't support the use of filters, you can use the Dashboard data source to reference that data, and then filter it in a new panel.
This allows you to bypass the limitations of the data source in the source panel.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-adhoc-filter-dashboard-ds-v12.2.png" max-width="750px" alt="The query section of a panel with the Dashboard data source configured" >}}

To use filters on data from an unsupported data source, follow these steps:

1. Navigate to the dashboard with the panel with the data you want to filter.
1. Click **Edit** in top-right corner of the dashboard.
1. Click the **Add new element** icon and click or drag a panel onto the dashboard.
1. Click **Configure visualization**.
1. In the **Queries** tab of the edit panel view, enter `Dashboard` in the **Data source** field and select **-- Dashboard --**.
1. In the query configuration section, make the following selections:
   - **Source panel** - Choose the panel with the source data.
   - **Data** - Select **All Data** to use the data of the panel, and not just the annotations. This is the default selection.
   - **Filters** - Toggle on the switch to make the data from the referenced panel filterable.

   {{< admonition type="note">}}
   If you're referencing multiple panels in a dashboard with the Dashboard data source, you can only use one of those source panels at a time for filtering.
   {{< /admonition >}}

1. Configure any other needed options for the panel.
1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Back** and then **Exit edit**.

Now you can filter the data from the source panel by way of the Dashboard data source.
Add as many panels as you need.

# Import dashboards

You can import preconfigured dashboards into your Grafana instance or Cloud stack using the UI or the [HTTP API](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developer-resources/api-reference/http-api/).

## Import a dashboard

To import a dashboard, follow these steps:

1. Click **Dashboards** in the primary menu.
1. Click **New** and select **Import dashboard** in the drop-down menu.
1. Perform one of the following steps:
   - Upload a dashboard JSON file.
   - Paste a [Grafana.com dashboard](#discover-dashboards-on-grafanacom) URL or ID into the field provided, and click **Load**.
   - Paste dashboard JSON text directly into the text area, and click **Load**.

1. (Optional) Change the dashboard name, folder, or UID, and specify metric prefixes, if the dashboard uses any.
1. Select a data source, if required.
1. Click **Import**.

## Discover dashboards on grafana.com

The [Dashboards page](https://grafana.com/grafana/dashboards/) on grafana.com provides you with dashboards for common server applications. Browse our library of official and community-built dashboards and import them to quickly get up and running.

{{< figure src="/media/docs/grafana/dashboards/screenshot-gcom-dashboards.png" alt="Preconfigured dashboards on grafana.com">}}

You can also add to this library by exporting one of your own dashboards. For more information, refer to [Share dashboards and panels](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/share-dashboards-panels/).

## More examples

Your Grafana Cloud stack comes with several default dashboards in the **Grafana Cloud** folder in **Dashboards**. If you're running your own installation of Grafana, you can find more example dashboards in the `public/dashboards/` directory.

## Frequently asked questions

{{< qa-list >}}
{{< qa question="Where can I find dashboards to import?" >}}
Grafana.com provides a library of official and community-created dashboards for common technologies and applications.
You can browse the library, copy a dashboard's URL or ID, and import it directly into your Grafana instance.
{{< /qa >}}
{{< qa question="Why do I need to select a data source when importing a dashboard?" >}}
Many imported dashboards contain queries that reference a data source.
During import, Grafana lets you map those references to data sources that exist in your environment so the dashboard can display data correctly.
{{< /qa >}}
{{< qa question="Can I import dashboards into both Grafana Cloud and self-managed Grafana?" >}}
Yes.
Dashboard import is supported in both Grafana Cloud and self-managed Grafana.
Regardless of where your Grafana instance is running, you can import dashboards from a JSON file, by pasting dashboard JSON, or by using a dashboard URL or ID from Grafana.com.
{{< /qa >}}
{{< /qa-list >}}

# Manage dashboard links

You can use links to navigate between commonly-used dashboards or to connect others to your visualizations. Links let you create shortcuts to other dashboards, panels, and even external websites.

Grafana supports dashboard links, panel links, and data links. Dashboard links are displayed at the top of the dashboard. Panel links are accessible by clicking the icon next to the panel title.

## Which link should you use?

Start by figuring out how you're currently navigating between dashboards. If you're often jumping between a set of dashboards and struggling to find the same context in each, links can help optimize your workflow.

The next step is to figure out which link type is right for your workflow. Even though all the link types in Grafana are used to create shortcuts to other dashboards or external websites, they work in different contexts.

- If the link relates to most if not all of the panels in the dashboard, use [dashboard links](#dashboard-links).
- If you want to drill down into specific panels, use [panel links](#panel-links).
- If you want to link to an external site, you can use either a dashboard link or a panel link.
- If you want to drill down into a specific series, or even a single measurement, use [data links](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-data-links/).

## Controlling time range using the URL

To control the time range of a panel or dashboard, you can provide query parameters in the dashboard URL:

- `from` - defines lower limit of the time range, specified in ms epoch
- `to` - defines upper limit of the time range, specified in ms epoch
- `time` and `time.window` - defines a time range from `time-time.window/2` to `time+time.window/2`. Both params should be specified in ms. For example `?time=1500000000000&time.window=10000` will result in 10s time range from 1499999995000 to 1500000005000

## Dashboard links

When you create a dashboard link, you can include the time range and current template variables to directly jump to the same context in another dashboard. This way, you don’t have to worry whether the person you send the link to is looking at the right data. For other types of links, refer to [Data link variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-data-links/#data-link-variables).

Dashboard links can also be used as shortcuts to external systems, such as submitting [a GitHub issue with the current dashboard name](https://github.com/grafana/grafana/issues/new?title=Dashboard%3A%20HTTP%20Requests).

To see an example of dashboard links in action, check out:

- [Dashboard links with variables](https://play.grafana.org/d/rUpVRdamz/dashboard-links-with-variables?orgId=1)
- [Prometheus repeat](https://play.grafana.org/d/dbg1erw7kech6oe/placeholder-dashboard)

After you've added a dashboard link, it appears in the upper right corner of your dashboard.

### Add links to dashboards

To add a link to another dashboard at the top of your current dashboard, follow these steps:

{{< docs/list >}}

1. In the dashboard you want to link, click **Edit**.
1. In the toolbar, click the **Dashboard options** icon.
1. In the sidebar, click **View all settings**.
1. Go to the **Links** tab and then click **Add dashboard link**. If the dashboard already has links, click **New link**.

{{< shared id="add-dashboard-link" >}}

The default link type is **Dashboards**.

1. In the **With tags** drop-down list, enter tags to limit the linked dashboards to only the ones with the tags you enter.

   If you don't add any tags, Grafana includes links to all other dashboards.

1. Set link options:
   - **Show as dropdown** – If you are linking to lots of dashboards, then you probably want to select this option and add an optional title to the dropdown. Otherwise, Grafana displays the dashboard links side by side across the top of your dashboard.
   - **Include current time range** – Select this option to include the dashboard time range in the link. When the user clicks the link, the linked dashboard opens with the indicated time range already set. **Example:** https://play.grafana.org/d/000000010/annotations?orgId=1&from=now-3h&to=now
   - **Include current template variable values** – Select this option to include template variables currently used as query parameters in the link. When the user clicks the link, any matching templates in the linked dashboard are set to the values from the link. For more information, see [Dashboard URL variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/create-dashboard-url-variables/).
   - **Open link in new tab** – Select this option if you want the dashboard link to open in a new tab or window.
   - **Show in controls menu** – Select this option to display the link in the dashboard controls menu instead of at the top of the dashboard. The dashboard controls menu appears as a button in the dashboard toolbar.

{{< /shared >}}

1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Back** and **Exit edit**.

{{< /docs/list >}}

{{< admonition type="tip" >}}
To add links without leaving the dashboard, click the **Add new element** icon in the dashboard toolbar, and select **Link**.
For more information, refer to the [Dashboard controls documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/dashboard-controls/).
{{< /admonition >}}

### Add a URL link to a dashboard

Add a link to a URL at the top of your current dashboard.
You can link to any available URL, including dashboards, panels, or external sites.
You can even control the time range to ensure the user is zoomed in on the right data in Grafana.

To add a URL link to your dashboard, follow these steps:

{{< docs/list >}}

1. In the dashboard you want to link, click **Edit**.
1. In the toolbar, click the **Dashboard options** icon.
1. In the sidebar, click **View all settings**.
1. Go to the **Links** tab and then click **Add dashboard link**. If the dashboard already has links, click **New link**.

{{< shared id="url-dashboard-link" >}}

1. In the **Type** drop-down list, select **Link**.
1. In the **URL** field, enter the URL to which you want to link.

   Depending on the target, you might want to include field values. **Example:** https://github.com/grafana/grafana/issues/new?title=Dashboard%3A%20HTTP%20Requests

1. In the **Tooltip** field, enter the tooltip you want the link to display when the user hovers their mouse over it.
1. In the **Icon** drop-down, choose the icon you want displayed with the link.
1. Set link options; by default, these options are enabled for URL links:
   - **Include current time range** – Select this option to include the dashboard time range in the link. When the user clicks the link, the linked dashboard opens with the indicated time range already set. **Example:** https://play.grafana.org/d/000000010/annotations?orgId=1&from=now-3h&to=now
   - **Include current template variable values** – Select this option to include template variables currently used as query parameters in the link. When the user clicks the link, any matching templates in the linked dashboard are set to the values from the link.
   - **Open link in new tab** – Select this option if you want the dashboard link to open in a new tab or window.
   - **Show in controls menu** – Select this option to display the link in the dashboard controls menu instead of at the top of the dashboard. The dashboard controls menu appears as a button in the dashboard header.

{{< /shared >}}

1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Back** and then **Exit edit**.

{{< /docs/list >}}

### Update a dashboard link

To edit, duplicate, or delete dashboard link, follow these steps:

1. In the dashboard you want to link, click **Edit**.
1. In the toolbar, click the **Dashboard options** icon.
1. In the sidebar, click **View all settings**.
1. Go to the **Links** tab.
1. Do one of the following:
   - **Edit** - Click the name of the link and update the link settings.
   - **Duplicate** - Click the copy link icon next to the link that you want to duplicate.
   - **Delete** - Click the red **X** next to the link that you want to delete, and then **Delete**.

1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Back** and then **Exit edit**.

## Panel links

Each panel can have its own set of links that are shown in the upper left of the panel after the panel title. You can link to any available URL, including dashboards, panels, or external sites. You can even control the time range to ensure the user is zoomed in on the right data in Grafana.

Click the icon next to the panel title to see available panel links.

{{< figure src="/media/docs/grafana/dashboards/screenshot-panel-links-v11.3.png" max-width="550px" alt="List of panel links displayed" >}}

### Add a panel link

1. Hover over any part of the panel to which you want to add the link to display the actions menu on the top right corner.
1. Click the menu and select **Edit**.

   To use a keyboard shortcut to open the panel, hover over the panel and press `e`.

1. Expand the **Panel options** section, scroll down to **Panel links**.
1. Click **Add link**.
1. Enter a **Title**. **Title** is a human-readable label for the link that will be displayed in the UI.
1. Enter the **URL** you want to link to.
   You can even add one of the template variables defined in the dashboard. Press Ctrl+Space or Cmd+Space and click in the **URL** field to see the available variables. By adding template variables to your panel link, the link sends the user to the right context, with the relevant variables already set. You can also use time variables:
   - `from` - Defines the lower limit of the time range, specified in ms epoch.
   - `to` - Defines the upper limit of the time range, specified in ms epoch.
   - `time` and `time.window` - Define a time range from `time-time.window/2` to `time+time.window/2`. Both params should be specified in ms. For example `?time=1500000000000&time.window=10000` will result in 10s time range from 1499999995000 to 1500000005000.
1. If you want the link to open in a new tab, then select **Open in new tab**.
1. Click **Save** to save changes and close the dialog box.
1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Back** and then **Exit edit**.

### Update a panel link

1. Hover over any part of the panel to display the actions menu on the top right corner.
1. Click the menu and select **Edit**.

   To use a keyboard shortcut to open the panel, hover over the panel and press `e`.

1. Expand the **Panel options** section, scroll down to Panel links.
1. Find the link that you want to make changes to.
1. Click the Edit (pencil) icon to open the Edit link window.
1. Make any necessary changes.
1. Click **Save** to save changes and close the dialog box.
1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Back** and then **Exit edit**.

### Delete a panel link

1. Hover over any part of the panel to display the actions menu on the top right corner.
1. Click the menu and select **Edit**.

   To use a keyboard shortcut to open the panel, hover over the panel and press `e`.

1. Expand the **Panel options** section, scroll down to Panel links.
1. Find the link that you want to delete.
1. Click the **X** icon next to the link you want to delete.
1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Back** and then **Exit edit**.

# Manage library panels

A library panel is a reusable panel that you can use in any dashboard. When you make a change to a library panel, that change propagates to all instances of where the panel is used. Library panels streamline reuse of panels across multiple dashboards.

You can save a library panel in a folder alongside saved dashboards.

## Role-based access control

You can control permissions for library panels using [role-based access control (RBAC)](ref:rbac). RBAC provides a standardized way of granting, changing, and revoking access when it comes to viewing and modifying Grafana resources, such as dashboards, reports, and administrative settings.

## Create a library panel

Library panels can be reused in different dashboards throughout Grafana. When you create a library panel, the panel on the source dashboard is converted to a library panel as well. You need to save the original dashboard once a panel is converted.

To create a library panel, follow these steps:

1. In the top-right corner of the dashboard, click **Edit**.
1. Hover over any part of the panel you want to share to display the actions menu on the top right corner.
1. Click **More > New library panel**.
1. In the **Library panel name** field, enter the name.
1. In the **Save in folder** drop-down list, select the folder in which to save the library panel. By default, the root level is selected.
1. Click **Create library panel** to save your changes.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

Once created, you can modify the library panel using any dashboard on which it appears. After you save the changes, all instances of the library panel reflect these modifications.

## Add a library panel to a dashboard

Add a Grafana library panel to a dashboard when you want to provide visualizations to other dashboard users.

To add a library panel, follow these steps:

1. Click **Dashboards** in the main menu.
1. Click **New** and select **New dashboard** in the drop-down list.
1. On the empty dashboard, click or drag the panel onto the dashboard.
1. Click **Use library panel**.

   The **Add panel from panel library** drawer opens.

1. Filter the list or search to find the panel you want to add.
1. Click a panel to add it to the dashboard.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.

## Unlink a library panel

Unlink a library panel when you want to make a change to the panel and not affect other instances of the library panel.

To unlink a library panel, follow these steps:

1. Click **Dashboards** in the main menu.
1. Click **Library panels**.
1. Select a library panel that is being used in dashboards.
1. Click the panel you want to unlink.
1. In the dialog box, select the dashboard from which you want to unlink the panel.
1. Click **View panel in \<dashboard name\>**.
1. Click **Edit** in the top-right corner of the dashboard.
1. Hover over any part of the panel you want to unlink to display the menu icon on the top-right corner.
1. Click the menu icon and select **More > Unlink library panel**.
1. Click **Yes, unlink**.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.

Alternatively, if you know where the library panel is being used, you can go directly to that dashboard and start at step 7.

## Replace a library panel

To replace a library panel with a different one, follow these steps:

1. Click **Dashboards** in the main menu.
1. Click **Library panels**.
1. Select a library panel that is being used in different dashboards.
1. Click the panel you want to unlink.
1. In the dialog box, select the dashboard from which you want to unlink the panel.
1. Click **View panel in \<dashboard name\>**.
1. Click **Edit** in the top-right corner of the dashboard.
1. Hover over any part of the panel you want to unlink to display the menu icon on the top-right corner.
1. Click the menu icon and select **More > Replace library panel**.
1. Select the replacement library panel.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save** and **Exit edit**.

Alternatively, if you know where the library panel that you want to replace is being used, you can go directly to that dashboard and start at step 7.

## View a list of library panels

You can view a list of available library panels and see where those panels are being used.

To view and manage library panels, follow these steps:

1. Click **Dashboards > Library panels** in the main menu.

   You can see a list of previously defined library panels.
   {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-library-panel-list-9-5.png" class="docs-image--no-shadow" max-width= "900px" alt="Library panels page with list of library panels" >}}

1. Search for a specific library panel if you know its name.

   You can also filter the panels by folder or type.

1. Click the panel to see if it's being used in any dashboards.
1. (Optional) If the library panel is in use, select one of the dashboards using it.
1. (Optional) Click **View panel in \<dashboard name\>** to see the panel in context.

## Delete a library panel

To delete a library panel that you no longer need, follow these steps:

1. Click **Dashboards > Library panels** in the main menu.
1. Click the delete icon next to the library panel name.
1. Click **Delete**.

# Manage dashboard version history

Whenever you save a version of your dashboard, a copy of that version is saved so that previous versions of your dashboard are never lost. You can see a list of dashboard versions in the **Versions** tab of the dashboard settings:

![Dashboards versions list](/media/docs/grafana/dashboards/screenshot-dashboard-version-list-11.2.png)

The dashboard version history feature lets you compare dashboards and restore previously saved dashboard versions. Grafana saves 20 versions by default. To change this limit, configure the [`[dashboards] versions_to_keep`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#dashboards) setting.

## Compare two dashboard versions

To compare two dashboard versions, follow these steps:

1. Click **Edit**.
1. In the toolbar, click the **Dashboard options** icon.
1. In the sidebar, click **View all settings**.
1. Go to the **Versions** tab.
1. Select the two dashboard versions that you want to compare.
1. Click **Compare versions** to view the diff between the two versions.
1. Review the text descriptions of the differences between the versions.
1. (Optional) Expand the **View JSON diff** section of the page to see the diff of the raw JSON that represents your dashboard.
1. When you've finished comparing versions, click **Back** and **Exit edit**.

When you're comparing versions, if one of the versions you've selected is the latest version, a button to restore the previous version is also displayed, so you can restore a version from the compare view:

![Dashboards versions diff](/media/docs/grafana/dashboards/screenshot-dashboard-compare-versions-restore-11.2.png)

## Restore a previously dashboard version

To restore to a previously saved dashboard version, follow these steps:

1. Click **Edit**.
1. Click the **Dashboard options** icon.
1. In the sidebar, click **View all settings**.
1. Go to the **Versions** tab.
1. Click the **Restore** button next to the version.

When you restore a version, the dashboard is immediately saved and you're no longer in edit mode.

After you restore a previous version, a new version of the dashboard is created containing the same exact data as the previous version, but with a different version number. This is indicated in the **Notes column** in the **Versions** tab of the dashboard settings. This is done simply to ensure your previous dashboard versions are not affected by the change.

# Modify dashboard settings

The dashboard settings page allows you to:

- Edit general dashboard properties, including time settings
- Add links
- View the dashboard JSON model

To access the dashboard setting page:

1. Click **Edit**.
1. In the toolbar, click the **Dashboard options** icon.
1. In the sidebar, click **View all settings**.

## Modify dashboard time settings

Adjust dashboard time settings when you want to change the dashboard timezone, the local browser time, and specify auto-refresh time intervals.

1. On the the **General** tab of the **Settings** page, scroll down to the **Time options** section.
1. Specify time settings as follows.
   - **Time zone:** Specify the local time zone of the service or system that you are monitoring. This can be helpful when monitoring a system or service that operates across several time zones.
     - **Default:** Grafana uses the default selected time zone for the user profile, team, or organization. If no time zone is specified for the user profile, a team the user is a member of, or the organization, then Grafana uses the local browser time.
     - **Browser time:** The time zone configured for the viewing user browser is used. This is usually the same time zone as set on the computer.
     - Standard [ISO 8601 time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), including UTC.
   - **Auto refresh:** Customize the options displayed for relative time and the auto-refresh options Entries are comma separated and accept any valid time unit.
   - **Now delay:** Override the `now` time by entering a time delay. Use this option to accommodate known delays in data aggregation to avoid null values.
   - **Week start:** Choose the day the week starts on.
   - **Hide time picker:** Select this option if you do not want Grafana to display the time picker.
   - **Refresh live dashboards:** Select this option to continuously update panels when the time range includes the current time.

1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## Modify graph tooltip behavior

Use this option to control tooltip and hover highlight behavior across graph panels (for example, time series).

1. On the the **General** tab of the **Settings** page, scroll down to the **Panel options** section.
1. Choose from the following options to control the tooltip and hover highlight behavior across graph panels:
   - **Default** - Tooltip and hover highlight behavior isn't shared across panels.
   - **Shared crosshair** - When you hover the cursor over one graph panel in the dashboard, the crosshair is also displayed on all other graph panels in the dashboard.
   - **Shared tooltip** - When you hover the cursor over one graph panel in the dashboard, the crosshair and tooltips are also displayed on all other graph panels in the dashboard.

1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## Add tags

You can add metadata to your dashboards using tags. Tags also give you the ability to filter the list of dashboards.

Tags can be up to 50 characters long, including spaces.

To add tags to a dashboard, follow these steps:

1. On the the **General** tab of the **Settings** page, scroll down to the **Tags** section.
1. In the field, enter a new or existing tag.

   If you're entering an existing tag, make sure that you spell it the same way or a new tag is created.

1. Click **Add** or press the Enter key.
1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

When you're on the **Dashboards** page, any tags you've entered show up under the **Tags** column.

## Add an annotation query

Annotations setup has been removed from dashboard settings.
You can now add and manage annotation queries in the dashboard sidebar.
For more information, refer to [Add annotation queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/dashboard-controls/#add-annotation-queries).

## Add a variable

Variables setup has been removed from dashboard settings.
You can now add and manage variables in the dashboard sidebar.
For more information, refer to [Add variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/dashboard-controls/#add-variables).

## Add a link

Dashboard links enable you to place links to other dashboards and web sites directly below the dashboard header. Links provide for quick navigation to other, related dashboards and content.

1. On the **Settings** page, click the **Links** tab.
1. Click **New link**.
1. Enter title for the link.
1. In the **Type** drop-down, select **Dashboards** or **Link**.
1. Set the rest of the link options.

   For more detailed directions on creating links, refer to [Dashboard links](ref:dashboard-links)

1. Click **Save**.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save**.
1. Click **Exit edit**.

## View dashboard JSON model

A dashboard in Grafana is represented by a JSON object, which stores metadata of its dashboard. Dashboard metadata includes dashboard properties, metadata from panels, template variables, panel queries, and so on.

To view a dashboard JSON model, on the **Settings** page, click the **JSON Model** tab.

For more information about the JSON fields, refer to [JSON fields](ref:json-fields).

# Dashboard JSON model

Grafana dashboards are represented as JSON objects that store metadata, panels, variables, and settings.

## Different dashboard schema models

There are currently three dashboard JSON schema models:

- [V2 Resource](#v2-resource-model): The current schema, supporting new features such as advanced layouts and conditional rendering. It models all dashboard elements as Kubernetes kinds, following Kubernetes conventions for declaring dashboard components.
- [V1 Resource](#v1-resource-model): The Classic dashboard schema formatted as a Kubernetes-style resource. Its `spec` property contains the Classic model of the schema. It was the default format for API communication between Grafana v12.2.0 and v13.0.0 and has been used for exporting, importing, and sharing dashboards.
- [Classic](#classic-model): A non-Kubernetes resource that was the default before Grafana v12.2.0. It's been widely used for exporting, importing, and sharing dashboards in the Grafana dashboards collection at [grafana.com/dashboards](https://grafana.com/grafana/dashboards/). Dashboards created using this model can be exported using either this model or V2.

{{< admonition type="note" >}}
[Observability as Code](https://grafana.com/docs/grafana/latest/as-code/) works with both versions of the JSON model, but it's fully compatible with version 2.
{{< /admonition >}}

## Access and update the JSON model {#view-json}

To access the JSON representation of a dashboard:

1. Navigate to the dashboard.
1. Click **Edit** in the top-right corner.
1. In the toolbar, click the **Dashboard options** icon.
1. In the sidebar, click **View all settings**.
1. Go to the **JSON Model** tab.
1. When you've finished updating the JSON, click **Save changes** at the bottom of the tab.
1. Click **Back** and **Exit edit**.

## V2 Resource model

For the detailed V2 Resource model schema, refer to the [Swagger documentation](https://play.grafana.org/swagger?api=dashboard.grafana.app-v2beta1).
You can also reach the Swagger schema documentation from your Grafana Cloud instance at: https://grafana.com/launch/swagger.

## V1 Resource model

The V1 Resource schema model formats the [Classic JSON model](#classic-model) schema as a Kubernetes-style resource.
The `spec` property of the schema contains the Classic-style model of the schema.

For the detailed V1 Resource model schema, refer to the [Swagger documentation](https://play.grafana.org/swagger?api=dashboard.grafana.app-v1).
You can also reach the Swagger schema documentation from your Grafana Cloud instance at: https://grafana.com/launch/swagger.

The following code snippet shows the fields included in the V1 Resource model:

```json
{
  "apiVersion": "dashboard.grafana.app/v1",
  "kind": "Dashboard",
  "metadata": {
    "name": "isnt5ss",
    "namespace": "stacks-521104",
    "uid": "92674c0e-0360-4bb4-99ab-fb150581376d",
    "resourceVersion": "1764705030717045",
    "generation": 1,
    "creationTimestamp": "2025-12-02T19:50:30Z",
    "labels": {
      "grafana.app/deprecatedInternalID": "1329"
    },
    "annotations": {
      "grafana.app/createdBy": "user:u000000002",
      "grafana.app/folder": "",
      "grafana.app/saved-from-ui": "Grafana Cloud (instant)"
    }
  },
  "spec": {
    "annotations": {
      "list": [
        {
          "builtIn": 1,
          "datasource": {
            "type": "grafana",
            "uid": "-- Grafana --"
          },
          "enable": true,
          "hide": true,
          "iconColor": "rgba(0, 211, 255, 1)",
          "name": "Annotations & Alerts",
          "type": "dashboard"
        }
      ]
    },
    "editable": true,
    "fiscalYearStartMonth": 0,
    "graphTooltip": 0,
    "id": 1329,
    "links": [],
    "panels": [],
    "preload": false,
    "schemaVersion": 42,
    "tags": [],
    "templating": {
      "list": []
    },
    "time": {
      "from": "now-6h",
      "to": "now"
    },
    "timepicker": {},
    "timezone": "Africa/Abidjan",
    "title": "Graphite suggestions",
    "uid": "isnt5ss",
    "version": 1,
    "weekStart": ""
  },
  "status": {}
}
```

## Classic model

When you create a new dashboard in self-managed Grafana, a new dashboard JSON object was initialized with the following fields:

{{< admonition type="note" >}}
In the following JSON, id is shown as null which is the default value assigned to it until a dashboard is saved.
After a dashboard is saved, an integer value is assigned to the `id` field.
{{< /admonition >}}

```json
{
  "id": null,
  "uid": "cLV5GDCkz",
  "title": "New dashboard",
  "tags": [],
  "timezone": "browser",
  "editable": true,
  "graphTooltip": 1,
  "panels": [],
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": []
  },
  "templating": {
    "list": []
  },
  "annotations": {
    "list": []
  },
  "refresh": "5s",
  "schemaVersion": 17,
  "version": 0,
  "links": []
}
```

Each field in the dashboard JSON is explained below with its usage:

<!--prettier-ignore-start -->

| Name              | Usage                                                                                      |
| ----------------- | ------------------------------------------------------------------------------------------ |
| **id**            | unique numeric identifier for the dashboard. (generated by the db) |
| **uid**           | unique dashboard identifier that can be generated by anyone. string (8-40) |
| **title**         | current title of dashboard |
| **tags**          | tags associated with dashboard, an array of strings |
| **style**         | theme of dashboard, i.e. `dark` or `light` |
| **timezone**      | timezone of dashboard, i.e. `utc` or `browser` |
| **editable**      | whether a dashboard is editable or not |
| **graphTooltip**  | 0 for no shared crosshair or tooltip (default), 1 for shared crosshair, 2 for shared crosshair AND shared tooltip |
| **time**          | time range for dashboard, i.e. last 6 hours, last 7 days, etc |
| **timepicker**    | timepicker metadata, see [timepicker section](#timepicker) for details |
| **templating**    | templating metadata, see [templating section](#templating) for details |
| **annotations**   | annotations metadata, see [annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/) for how to add them |
| **refresh**       | auto-refresh interval|
| **schemaVersion** | version of the JSON schema (integer), incremented each time a Grafana update brings changes to said schema |
| **version**       | version of the dashboard (integer), incremented each time the dashboard is updated |
| **panels**        | panels array, see below for detail. |

<!--prettier-ignore-end -->

### Panels

Panels are the building blocks of a dashboard. It consists of data source queries, type of graphs, aliases, etc. Panel JSON consists of an array of JSON objects, each representing a different panel. Most of the fields are common for all panels but some fields depend on the panel type. Following is an example of panel JSON of a text panel.

```json
"panels": [
  {
    "type": "text",
    "title": "Panel Title",
    "gridPos": {
      "x": 0,
      "y": 0,
      "w": 12,
      "h": 9
    },
    "id": 4,
    "mode": "markdown",
    "content": "# title"
  }
```

### Panel size and position

The gridPos property describes the panel size and position in grid coordinates.

- `w` 1-24 (the width of the dashboard is divided into 24 columns)
- `h` In grid height units, each represents 30 pixels.
- `x` The x position, in same unit as `w`.
- `y` The y position, in same unit as `h`.

The grid has a negative gravity that moves panels up if there is empty space above a panel.

### timepicker

```json
"timepicker": {
    "collapse": false,
    "enable": true,
    "notice": false,
    "now": true,
    "hidden": false,
    "nowDelay": "",
    "quick_ranges": [
      {
        "display": "Last 6 hours",
        "from": "now-6h",
        "to": "now"
      },
      {
        "display": "Last 7 days",
        "from": "now-7d",
        "to": "now"
      }
    ],
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "status": "Stable",
    "type": "timepicker"
  }
```

Usage of the fields is explained below:

<!--prettier-ignore-start -->

| Name                  | Usage                                                     |
| --------------------- | --------------------------------------------------------- |
| **collapse**          | whether timepicker is collapsed or not                    |
| **enable**            | whether timepicker is enabled or not                      |
| **notice**            |                                                           |
| **now**               |                                                           |
| **hidden**            | whether timepicker is hidden or not                       |
| **nowDelay**          | override the now time by entering a time delay. Use this option to accommodate known delays in data aggregation to avoid null values.                                                   |
| **quick_ranges**      | custom quick ranges                                       |
| **refresh_intervals** | interval options available in the refresh picker dropdown |
| **status**            |                                                           |
| **type**              |                                                           |

<!--prettier-ignore-end -->

### templating

The `templating` field contains an array of template variables with their saved values along with some other metadata, for example:

```json
 "templating": {
    "enable": true,
    "list": [
      {
        "allFormat": "wildcard",
        "current": {
          "tags": [],
          "text": "prod",
          "value": "prod"
        },
        "datasource": null,
        "includeAll": true,
        "name": "env",
        "options": [
          {
            "selected": false,
            "text": "All",
            "value": "*"
          },
          {
            "selected": false,
            "text": "stage",
            "value": "stage"
          },
          {
            "selected": false,
            "text": "test",
            "value": "test"
          }
        ],
        "query": "tag_values(cpu.utilization.average,env)",
        "refresh": false,
        "type": "query"
      },
      {
        "allFormat": "wildcard",
        "current": {
          "text": "apache",
          "value": "apache"
        },
        "datasource": null,
        "includeAll": false,
        "multi": false,
        "multiFormat": "glob",
        "name": "app",
        "options": [
          {
            "selected": true,
            "text": "tomcat",
            "value": "tomcat"
          },
          {
            "selected": false,
            "text": "cassandra",
            "value": "cassandra"
          }
        ],
        "query": "tag_values(cpu.utilization.average,app)",
        "refresh": false,
        "regex": "",
        "type": "query"
      }
    ]
  }
```

Usage of the above mentioned fields in the templating section is explained below:

| Name            | Usage                                                                                                   |
| --------------- | ------------------------------------------------------------------------------------------------------- |
| **enable**      | whether templating is enabled or not                                                                    |
| **list**        | an array of objects each representing one template variable                                             |
| **allFormat**   | format to use while fetching all values from data source, eg: `wildcard`, `glob`, `regex`, `pipe`, etc. |
| **current**     | shows current selected variable text/value on the dashboard                                             |
| **data source** | shows data source for the variables                                                                     |
| **includeAll**  | whether all value option is available or not                                                            |
| **multi**       | whether multiple values can be selected or not from variable value list                                 |
| **multiFormat** | format to use while fetching timeseries from data source                                                |
| **name**        | name of variable                                                                                        |
| **options**     | array of variable text/value pairs available for selection on dashboard                                 |
| **query**       | data source query used to fetch values for a variable                                                   |
| **refresh**     | configures when to refresh a variable                                                                   |
| **regex**       | extracts part of a series name or metric node segment                                                   |
| **type**        | type of variable, i.e. `custom`, `query` or `interval`                                                  |

# Manage playlists

A _playlist_ is a list of dashboards that are displayed in a sequence. You might use a playlist to build situational awareness or to present your metrics to your team or visitors.

Grafana automatically scales dashboards to any resolution, which makes them perfect for big screens.

You can access the **Playlist** feature from Grafana's side menu, in the Dashboards submenu.

{{< admonition type="note" >}}
You must have at least Editor role permissions to create and manage playlists.
{{< /admonition >}}

## Access, share, and control a playlist

Use the information in this section to access playlists. Start and control the display of a playlist using one of the six available modes.

### Access a playlist

1. Click **Dashboards** in the main menu.
1. Click **Playlists** to see a list of playlists.

### Start a playlist

You can start a playlist in four different view modes. View modes determine how the menus and navigation bar appear on the dashboards as well as how panels are sized.

1. Click **Dashboards** in the main menu.
1. Click **Playlists**.
1. Find the desired playlist and click **Start playlist**.
1. In the dialog box that opens, select one of the [four playlist modes](#playlist-modes) available.
1. Disable any dashboard controls that you don't want displayed while the list plays; these controls are enabled and visible by default. Select from:
   - **Time and refresh**
   - **Variables**
   - **Dashboard links**

1. Click **Start \<playlist name\>**.

The playlist displays each dashboard for the time specified in the **Interval** field, set when creating or editing a playlist. After a playlist starts, you can start or stop it it using the [controls](#playlist-controls) at the top of your screen.

### Playlist modes

<!-- prettier-ignore-start -->

| Mode             | Description                        |
| ---------------------------------- | -------------------- |
| Normal mode                        | <ul><li>The main menu and navigation bar remain visible.</li><li>Dashboard controls are hidden.</li><li>Playlist controls are displayed at the top of the screen.</li></ul> |
| Normal mode (with auto fit panels) | <ul><li>The main menu and navigation bar remain visible.</li><li>Dashboard controls are hidden.</li><li>Playlist controls are displayed at the top of the screen.</li><li>Dashboard panels automatically adjust to optimize space on screen.</li></ul> |
| Kiosk mode                         | <ul><li>The main menu, navigation bar, and dashboard controls are hidden.</li><li>You can disable the playlist manually by pressing the `Esc` key after the playlist has started. Doing so causes the playlist controls to be displayed at the top of the screen briefly.</li></ul> |
| Kiosk mode (with auto fit panels)  | <ul><li>The main menu, navigation bar, and dashboard controls are hidden.</li><li>You can disable the playlist manually by pressing the `Esc` key after the playlist has started. Doing so causes the playlist controls to be displayed at the top of the screen briefly.</li><li>Dashboard panels automatically adjust to optimize space on screen.</li></ul> |

<!-- prettier-ignore-end -->

### Playlist controls

You can control a playlist in **Normal** mode after it's started, using the buttons at the top of your screen. Press the `Esc` key to stop the playlist.

- **Next (double-right arrow)** - Advances to the next dashboard.
- **Back (double-left arrow)** - Returns to the previous dashboard.
- **Stop playlist** - Ends the playlist, and exits to the current dashboard.

## Create a playlist

You can create a playlist to present dashboards in a sequence, with a set order and time interval between dashboards. Be sure that all the dashboards you want to appear in your playlist are added before you create the playlist.

1. Click **Dashboards** in the main menu.
1. Click **Playlists**.
1. Click **New playlist**.
1. In the **Name** field, enter a descriptive name.
1. In the **Interval** field, enter the time interval each dashboard is displayed before moving on to the next dashboard.
1. In the **Add dashboards** section, add dashboards to the playlist using the **Add by title** and **Add by tag** drop-down options.

   Added dashboards are displayed in a list in the **Dashboards** section of the page, in the order you added them. This is also the play order of the dashboards.

1. Click **Save**.

## Edit a playlist

You can edit a playlist including adding, removing, and rearranging the order of dashboards.

1. Click **Dashboards** in the main menu.
1. Click **Playlists**.
1. Find the playlist you want to update and click **Edit playlist**. Do one or more of the following:
   - Edit - Update the name and time interval.
   - Add dashboards - Search for dashboards by title or tag to add them to the playlist.
   - Rearrange dashboards - Click and drag the dashboards into your desired order.
   - Remove dashboards - Click the **X** next to the name of the dashboard you want to remove from the playlist.

1. Click **Save**.

## Share a playlist in a view mode

You can share a playlist by copying the link address on the view mode you prefer, and pasting the URL to your destination.

1. Click **Dashboards** in the main menu.
1. Click **Playlists**.
1. Click the share icon of the playlist you want to share.
1. Select the view mode you prefer.
1. Click **Copy** next to the **Link URL** to copy it to your clipboard.
1. Paste the URL to your destination.

## Delete a playlist

When you no longer need a playlist, follow these steps to delete it:

1. Click **Dashboards** in the main menu.
1. Click **Playlists**.
1. Find the playlist you want to remove.
1. Click **Delete playlist**.

# Reporting settings

You can configure organization-wide report settings and branding options in **Dashboards > Reporting**.
These settings are applied to all the reports for the current organization.

To access the settings, go to **Dashboards > Reporting** and click the **Report settings** button.
This opens the **Report template settings** drawer, where you can make changes.

{{< admonition type="note" >}}
The redesigned reporting experience, including the report settings drawer, is generally available in [Grafana Cloud](/docs/grafana-cloud/).

In self-managed Grafana, enable the `newShareReportDrawer` [feature toggle](ref:feature-toggles) in your Grafana configuration file.
{{< /admonition >}}

You can also navigate these settings from the **Schedule report** drawer that opens when you create a report directly from a dashboard.

## Attachment settings

The options in this section control the branding and theming of the report attachments.

### PDF

- **Company logo** - Company logo displayed in the report PDF.
  Configure it by specifying a URL or uploading a file.
  The maximum file size is 3 MB.
  If not set, defaults to the Grafana logo. If the specified URL isn't valid, the logo image appears as broken.

- **Theme** - Theme of the PDF attached to the report.
  The selected theme is also applied to the PDFs generated when you click **Preview PDF** during report creation or select the **Export as PDF** option on a dashboard.

  If **Current** is selected, the PDF in the report is in the instance theme of the report creator, but the preview and exported PDFs are in the user's instance theme.
  Defaults to **Light**.

### PDF header

{{< admonition type="note" >}}
The PDF header options are available in [Grafana Enterprise](ref:grafana-enterprise) and [Grafana Cloud](/docs/grafana-cloud/).

In self-managed Grafana Enterprise, enable the `reportingHeaderSettings` [feature toggle](ref:feature-toggles) in your Grafana configuration file.
{{< /admonition >}}

The PDF header repeats at the top of every page of the report PDF.
In the **PDF** section, use the following switches to choose what it contains:

- **Show header** - Displays the report name and the date the PDF was generated.
  Defaults to on.
- **Show dashboard title** - Displays the title of the dashboard that the page was rendered from.
  Defaults to on.
- **Show data time range** - Displays the absolute time range of the data in the report.
  Defaults to on.

When you turn a switch off, that item no longer appears on any page, and the panels get the space instead.
If you turn all three off, Grafana removes the header area unless the report has **Show template variables** selected.

These switches apply to the report PDF and to the separate PDF of table data.
They also apply to the PDFs you generate with **Preview PDF** during report creation and with **Export as PDF** on a dashboard.

### PDF footer

{{< admonition type="note" >}}
The PDF footer options are available in [Grafana Enterprise](ref:grafana-enterprise), as well as in [Grafana Cloud](/docs/grafana-cloud/).

In self-managed Grafana Enterprise, enable the `reportingFooterSettings` [feature toggle](ref:feature-toggles) in your Grafana configuration file.
{{< /admonition >}}

The PDF footer repeats at the bottom of every page of the report PDF.
By default, it shows the page number on the left and your company logo on the right.

In the **Footer** section, you can replace that default with your own sequence of items.
You can also style each text item individually.
For example, you can add a confidentiality label next to the page number, or center a copyright line on the page.

- **Font family** - Font for every text item in the footer.
  Select **Inter (Default)**, **Helvetica**, **Arial**, **Times New Roman**, **Georgia**, **Verdana**, **Trebuchet MS**, or **Courier New**.
  Individual items can't set a different font.

#### Footer items

Grafana prints footer items from left to right, in the order they appear in the list.
The list might already contain the default items of a page number, a flex spacer, and a logo.
Remove the ones you don't want before you add your own.

You can add up to 10 items, and you can add the same item type more than once.
If you remove every item, reports fall back to the default footer of a page number and the company logo.

<!-- prettier-ignore-start -->

| Item | Description | Options |
| ---- | ----------- | ------- |
| Page number | Prints the current page and the total page count, such as `Page 2/8`. | [Text styling](#text-styling-options) |
| Date | Prints the date that the PDF was generated. | <ul><li>Date format of `dd/MM/yyyy`, `MM/dd/yyyy`, or `MM/yyyy`, defaulting to `dd/MM/yyyy`</li><li>[Text styling](#text-styling-options)</li></ul> |
| Fixed text | Prints the text that you enter, such as a copyright notice or a document classification. The text is limited to 50 characters, and the characters `<` and `>` aren't allowed. | [Text styling](#text-styling-options) |
| Logo | Prints the logo from the **Company logo** setting in the [PDF](#pdf) section. | Size of **Small (24px)**, **Medium (36px)**, **Large (48px)**, or **Extra large (64px)**, defaulting to **Medium (36px)** |
| Flex spacer | Prints nothing and pushes the items on either side of it apart. Use spacers to align items to the left, center, and right of the page. | None |

<!-- prettier-ignore-end -->

To place a date on the left, a label in the center, and the page number on the right, arrange five items in this order:
**Date**, **Flex spacer**, **Fixed text**, **Flex spacer**, **Page number**.

#### Text styling options

Page number, date, and fixed text items each have their own styling options.
To open them, click the gear icon next to the item.

- **Size** - Font size of the item in pixels.
  Select 8, 10, 12, 14, 16, or 18.
- **Weight** - Select **Normal** or **Bold**.
- **Style** - Select **Normal** or **Italic**.
- **Color** - Select a color from the color picker or enter a hex color code, such as `#FF0000`.

Clear an option to return the item to its default styling.
The default font size scales with the **Zoom** setting of the report, so text stays readable at every zoom level.

#### Customize the PDF footer

To customize the PDF footer, follow these steps:

1. Go to **Dashboards > Reporting** and click **Report settings**.
1. Under **Attachment settings**, go to the **Footer** section.
1. Remove any items you don't want to keep by clicking the trash icon next to each one.
1. Click **+ Add footer item** and select an item type.
1. Configure the item.

   For a fixed text item, enter the text.
   For a date item, select a date format.
   For a logo item, select a size.

1. Repeat the previous two steps for each item you want in the footer.
1. Click **Save**.

While you're editing the footer, you can also:

- Apply text styling options to a page number, date, or fixed text item. For more information, refer to [Text styling options](#text-styling-options)
- Reorder the footer.
  Use the up and down arrows next to each item.

The next report that Grafana generates uses the updated footer.
To check it sooner, click **Preview PDF** during report creation.

The footer applies to the report PDF, and to the PDFs you generate with **Preview PDF** during report creation and with **Export as PDF** on a dashboard.
The separate PDF of table data isn't affected by these settings.
It always prints a page number and the company logo.

### Embedded Image

- **Theme** - Theme of the dashboard image embedded in the email.
  If **Current** is selected, the image in the report is in the instance theme of the report creator. If the report creator doesn't have a theme set, then the team, organization, or server theme is used. For more information refer to [Change Grafana UI theme](ref:change-ui-theme).
  Defaults to **Dark**.

<!-- vale Grafana.WordList = NO -->

## Email branding

<!-- vale Grafana.WordList = YES -->

- **Company logo** - Company logo displayed in the report email. Configure it by specifying a URL or uploading a file. The maximum file size is 3 MB. If not set, defaults to the Grafana logo. If the specified URL isn't valid, the logo image appears as broken.
- **Email footer** - Select **Sent by** to add a footer link to the report email, or **None** to omit it.
- **Footer link text** - Text of the link in the report email footer. Defaults to `Grafana`.
- **Footer link URL** - Link of the report email footer.

Currently, the API does not allow for the simultaneous upload of files with identical names for both the email logo and report logo.
You can still upload the same file for each logo separately in two distinct steps.

# Manage dashboards

On the **Dashboards** page, you can perform dashboard management tasks such as:

- [Browsing](#browse-dashboards) and [creating](#create-a-dashboard-folder) dashboard folders
- [Working with team folders](../../../administration/team-management/team-folders/)
- [Managing folder permissions](#folder-permissions)
- [Adding generative AI features to dashboards](#set-up-generative-ai-features-for-dashboards)

For more information about creating dashboards, refer to [Build dashboards](ref:build-dashboards).

## Browse dashboards

On the **Dashboards** page, you can browse and manage folders and dashboards. This includes the options to:

- Create folders and dashboards.
- Move dashboards between folders.
- Delete multiple dashboards and folders.
- Navigate to a folder.
- Manage folder permissions. For more information, refer to [Dashboard permissions](ref:dashboard-permissions).

The page lists all the dashboards to which you have access, grouped into folders. Dashboards without a folder are displayed at the top level alongside folders.

### Shared with me

The **Shared with me** section displays folders and dashboards that are directly shared with you. These folders and dashboards aren't shown in the main list because you don't have access to one or more of their parent folders.

If you have permission to view all folders, you won't see a **Shared with me**.

### Team folders

If you belong to teams that own folders, the **Dashboards** page displays a **My team folders** section near the top of the page.

For more information, refer to [Manage team folders](/docs/grafana/<GRAFANA_VERSION>/administration/team-management/team-folders/).

## Create a dashboard folder

Folders help you organize and group dashboards, which is useful when you have many dashboards or multiple teams using the same Grafana instance.

> **Before you begin:** Ensure you have permission to create root-level folders, or Edit or Admin access to a parent folder to create subfolders. For more information about dashboard permissions, refer to [Dashboard permissions](ref:dashboard-permissions).

**To create a dashboard folder:**

1. Click **Dashboards** in the primary menu.
1. Do one of the following:
   - On the **Dashboards** page, click **New** and select **New folder** in the drop-down.
   - Click an existing folder and on the folder’s page, click **New** and select **New folder** in the drop-down.

1. Enter a unique name.

   Folder names can't include underscores (\_) or percentage signs (%), as it interferes with the search functionality.

   Also, alerts can't be placed in folders with slashes (\ /) in the name. If you want to place alerts in the folder, don't use slashes in the folder name.

1. Click **Create**

When you nest folders, you can do so up to four levels deep.

When you save a dashboard, you can optionally select a folder to save the dashboard in.

**To edit the name of a folder:**

1. Click **Dashboards** in the primary menu.
1. Navigate to the folder by selecting it in the list, or searching for it.
1. Click the **Edit title** icon (pencil) in the header and update the name of the folder.

The new folder name is automatically saved.

### Folder permissions

You can assign permissions to a folder. Dashboards in the folder inherit any permissions that you've assigned to the folder. You can assign permissions to organization roles, teams, and users.

**To modify permissions for a folder:**

1. Click **Dashboards** in the primary menu.
1. Navigate to the folder by selecting it in the list, or searching for it.
1. On the folder's page, click **Folder actions** and select **Manage permissions** in the drop-down.
1. Update the permissions as desired.

Changes are saved automatically.

For more information about dashboard permissions, refer to [Dashboard permissions](ref:dashboard-permissions).

## Dashboard and folder deletion

**Folders** - When you delete folders, the action is immediate and can't be reversed.

**Dashboards** - Deleted dashboards are stored in the deletion history and can be [restored](#restore-dashboards) if needed.

## Restore deleted dashboards

When you delete dashboards, they’re stored in the deletion history for up to 12 months, allowing you to restore them if needed.
After 12 months, Grafana permanently removes them.

You can access the list of deleted dashboards from the **Dashboards** page by clicking the **Recently deleted** button, or by navigating to **Dashboards > Recently deleted**.

{{% admonition type="note" %}}
You can restore dashboards that you deleted. Users with admin rights can restore any dashboard.
{{% /admonition %}}

To restore one or more dashboards, follow these steps:

1. In the main menu, click **Dashboards > Recently deleted** or click the **Recently deleted** button from the **Dashboards** page.
2. Select the dashboards you want to restore.
3. Click **Restore** and select a folder to which the dashboards will be moved:
   - If you select one dashboard and its original folder still exists, that folder is preselected.
   - If you select multiple dashboards from the _same_ folder, that folder is preselected.
   - If you select dashboards from _different_ folders, you’ll need to choose one target folder for all of them.
   - If the original folder no longer exists, you’ll need to select a new target folder.
4. Click **Restore**.

### Limitations

Restoring dashboards has the following limitations:

- **Permissions aren't preserved** - Dashboard-specific permissions are not restored. After restoration, you'll need to manually reconfigure any custom permissions that were previously set on the dashboard.
- **Folder-level permissions apply** - Restored dashboards inherit the permissions of the target folder you select during restoration.
- **Version history is reset** - The dashboard's version history is not preserved. After restoration, the dashboard starts with version 1, and all previous versions are lost.

## Set up generative AI features for dashboards

You can use generative AI to help you with the following tasks:

- **Generate panel and dashboard titles and descriptions**: Generate a title and description based on the data you’ve added for your panel or dashboard. This is useful when you want to visualize your data quickly and don’t want to spend time coming up with a title or description.
- **Generate dashboard save changes summary**: Generate a summary of the changes you’ve made to a dashboard when you save it. This is great for easily tracking the history of a dashboard.

To access these features, install and configure Grafana’s Large Language Model (LLM) app plugin. For more information, refer to the [Grafana LLM plugin documentation](ref:grafana-llm-plugin-documentation).

When enabled, the **✨ Auto generate** option displays next to the **Title** and **Description** fields in your panels and dashboards, or when you press the **Save** button.

# Search dashboards and folders

You can search for dashboards and dashboard folders by name.

When you search for dashboards, you can also do it by panel title. Whether you search by name or panel title, the system returns all dashboards available within the Grafana instance, even if you do not have permission to view the contents of the dashboard.

## Search by name

Begin typing any part of the dashboard or folder name in the search bar. The search returns results for any partial string match in real-time, as you type.

The search is:

- Real-time
- _Not_ case sensitive
- Functional across stored _and_ file based dashboards and folders.

{{< admonition type="note" >}}
You can use your keyboard arrow keys to navigate the results and press `Enter` to open the selected dashboard or folder.
{{< /admonition >}}

The following images show:

Searching by dashboard name from the **Dashboards** page.

{{< figure src="/media/docs/grafana/dashboards/search-for-dashboard.png" width="700px" >}}

Searching by folder name from the **Dashboards** page.

{{< figure src="/media/docs/grafana/dashboards/search-folder.png" width="700px" >}}

Searching by dashboard name inside a folder.

{{< figure src="/media/docs/grafana/dashboards/search-in-folder.png" width="700px" >}}

{{< admonition type="note" >}}
When you search within a folder, its subfolders are not part of the results returned. You need to be on the **Dashboards** page (or the root level) to search for subfolders by name.
{{< /admonition >}}

## Search dashboards using panel title

You can search for a dashboard by the title of a panel that appears in a dashboard.
If a panel's title matches your search query, the dashboard appears in the search results.

This feature is available by default in Grafana Cloud and in Grafana OSS v9.1 and higher, you access this feature by enabling the `panelTitleSearch` feature toggle.
For more information about enabling panel title search, refer to [Enable the panelTitleSearch feature toggle.](#enable-the-paneltitlesearch-feature-toggle)

The following image shows the search results when you search using panel title.

{{< figure src="/static/img/docs/v91/dashboard-features/search-by-panel-title.png" width="700px" >}}

### Enable the panelTitleSearch feature toggle

Complete the following steps to enable the `panelTitleSearch` feature toggle.

**Before you begin:**

- If you are running Grafana Enterprise with RBAC, enable [service accounts](ref:service-accounts).

**To enable the panelTitleSearch feature toggle:**

1. Open the Grafana [configuration file](ref:config-file).

1. Locate the [feature_toggles](ref:feature-toggles) section.

1. Add the following parameter to the `feature_toggles` section:

   ```
   [feature_toggles]
   # enable features, separated by spaces
   enable = panelTitleSearch
   ```

1. Save your changes and restart the Grafana server.

## Filter dashboard search results by tag(s)

Tags are a great way to organize your dashboards, especially as the number of dashboards grow. You can add and manage tags in dashboard `Settings`.

When you select multiple tags, Grafana shows dashboards that include all selected tags.

To filter dashboard search result by a tag, complete one of the following steps:

- To filter dashboard search results by tag, click a tag that appears in the right column of the search results.

  You can continue filtering by clicking additional tags.

- To see a list of all available tags, click the **Filter by tags** dropdown menu and select a tag.

  All tags will be shown, and when you select a tag, the dashboard search will be instantly filtered.

{{< admonition type="note" >}}
When using only a keyboard, press the `tab` key and navigate to the **Filter by tag** drop-down menu, press the down arrow key `▼` to activate the menu and locate a tag, and press `Enter` to select the tag.
{{< /admonition >}}

# Externally shared dashboards

{{< admonition type="note" >}}
This feature was previously called **Public dashboards**.
{{< /admonition >}}

Externally shared dashboards allow you to share your Grafana dashboard with anyone. This is useful when you want to make your dashboard available to the world without requiring access to your Grafana organization.

If you change a dashboard, ensure that you save the changes before sharing.

In order to create public dashboards, you need to be an Admin, have the `dashboards.public:write` permission, or the `fixed:dashboards.public:write` [RBAC role](ref:fixed-role-definitions).

{{< admonition type="warning" >}}
Sharing your dashboard externally could result in a large number of queries to the data sources used by your dashboard.
This can be mitigated by using the Enterprise [caching](ref:caching) and/or rate limiting features.
{{< /admonition >}}

## Shared dashboards list

You can see a list of all your externally shared dashboards in one place by navigating to **Dashboards > Shared dashboards**. For each dashboard in the list, the page displays:

- Link to view the externally shared version of the dashboard
- Link to the shared dashboard configuration
- Options to pause or revoke access to the external dashboard

You can also click the name of the dashboard to navigate to the dashboard internally.

## Important notes about sharing your dashboard externally

- Anyone with the URL can access the dashboard.
- Externally shared dashboards are read-only.
- Arbitrary queries **cannot** be run against your data sources through externally shared dashboards. Externally shared dashboards can only execute the queries stored on the original dashboard.

## Share externally with specific people

{{< admonition type="note">}}
This feature was previously called **email sharing**.

Available in [Grafana Enterprise](ref:grafana-enterprise) and [Grafana Cloud](/docs/grafana-cloud).
{{< /admonition >}}

<!-- {{< docs/private-preview product="Sharing externally with specific people" >}}-->

{{< admonition type="note" >}}
Sharing externally with specific people is currently in [private preview](https://grafana.com/docs/release-life-cycle/#private-preview). Please contact support to have this feature enabled.

This feature will incur a cost once it is promoted to general availability.
{{< /admonition >}}

To share a dashboard with specific external users, you can send them a link by email. Use this option when you only want to share your dashboard with specific people. When you share dashboards by email, recipients receive a one-time use link that's valid for **one hour**. Once the link is used, the viewer has access to the shared dashboard for **30 days**.

<!--When you share a dashboard with an email link, your organization is billed per user, regardless of how many dashboards are shared. Grafana bills monthly per user until access is revoked.-->

To share a dashboard with specific people, follow these steps:

1. Click **Dashboards** in the main menu.
1. Click the dashboard you want to share.
1. Click the **Share** drop-down list in the top-right corner and select **Share externally**.

   The **Share externally** drawer opens.

1. In the **Link access** drop-down list, select **Only specific people**.
1. Click the checkbox confirming that you understand payment is required to add users.
1. Click **Accept**.
1. In the **Invite** field, enter the email address of the person you want to invite and click **Invite** and repeat this process for all the people you want to invite.

   You can only invite one person at a time.

1. (Optional) Set the following options:
   - **Enable time range** - Allow people accessing the link to change the time range. This configuration screen shows the default time range of the dashboard.
   - **Display annotations** - Allow people accessing the link to view the dashboard annotations. If the dashboard uses tag-based annotation queries, annotations from other dashboards in your organization might also be visible. Refer to [Limitations](#limitations).
1. (Optional) Click **Copy external link** and send the copied URL to any external user.
1. Click the **X** at the top-right corner to close the share drawer.

Once you've shared a dashboard externally, a **Public** label is displayed in the header of the dashboard.

### Viewers requesting access

If a viewer without access tries to navigate to the shared dashboard, they'll be asked to request access by providing their email. They'll receive an email with a new one-time use link if the email they provided has already been invited to view the shared dashboard and hasn't been revoked.

### Revoke access for a viewer

You can revoke access to the entire dashboard using the steps in [Update access to an external dashboard link](#update-access-to-an-external-dashboard-link), but you can also revoke access to the dashboard for specific people.

To revoke access for a viewer, follow these steps:

1. Click **Dashboards** in the main menu.
1. Click the dashboard you want to share.
1. Click the **Share** drop-down list in the top-right corner and select **Share externally**.
1. In the **Share externally** drawer that opens, click the menu icon (three dots) next to the email address of the viewer for whom you'd like to revoke access.
1. Click **Revoke access**.
1. Click the **X** at the top-right corner to close the share drawer.

The viewer immediately no longer has access to the dashboard, nor can they use any existing one-time use links they may have.

### Re-invite a viewer

To re-invite a viewer, follow these steps:

1. Click **Dashboards** in the main menu.
1. Click the dashboard you want to share.
1. Click the **Share** drop-down list in the top-right corner and select **Share externally**.
1. In the **Share externally** drawer that opens, click the menu icon (three dots) next to the email address of the viewer you'd like to invite again.
1. Click **Resend invite**.
1. Click the **X** at the top-right corner to close the share drawer.

The viewer receives an email with a new one-time use link. This invalidates all previously issued links for that viewer.

### View shared dashboard users

To see a list of users who have accessed your externally shared dashboard by way of an emailed link, follow these steps:

1. Click **Administration** in in the main menu.
1. Select **Users and access** > **Users**.
1. On the **Users** page, click the **Shared dashboard users** tab.

On this screen, you can see:

- The earliest time a user has been active in a dashboard
- When they last accessed a shared dashboard
- The dashboards they have access to
- Their role

You can also revoke a user's access to all shared dashboards on from this tab.

### Access limitations

One-time use links use browser cookies, so when a viewer is granted access through one of these links, they'll only have access on the browser they used to claim the link.

A single viewer can't generate multiple valid one-time use links for a dashboard. When a new one-time use link is issued for a viewer, all previous ones are invalidated.

If a Grafana user has read access to the parent dashboard, they can view the externally shared dashboard without needing to have access granted.

## Share externally to anyone with a link

To share your dashboard so that anyone with the link can access it, follow these steps:

1. Click **Dashboards** in the main menu.
1. Click the dashboard you want to share.
1. Click the **Share** drop-down list in the top-right corner and select **Share externally**.

   The **Share externally** drawer opens.

1. In the **Link access** drop-down list, select **Anyone with the link**.
1. Click the checkbox confirming that you understand the entire dashboard will be public.
1. Click **Accept**.
1. (Optional) Set the following options:
   - **Enable time range** - Allow people accessing the link to change the time range. This configuration screen shows the default time range of the dashboard.
   - **Display annotations** - Allow people accessing the link to view the dashboard annotations. If the dashboard uses tag-based annotation queries, annotations from other dashboards in your organization might also be visible. Refer to [Limitations](#limitations).
1. Click the **X** at the top-right corner to close the share drawer.

Now anyone with the link can access the dashboard until you pause or revoke access to it.

Once you've shared a dashboard externally, a **Public** label is displayed in the header of the dashboard.

### Update access to an external dashboard link

You can update the access to externally shared dashboard links by following these steps:

1. Click **Dashboards** in the main menu.
1. Click the dashboard you want to share.
1. Click the **Share** drop-down list in the top-right corner and select **Share externally**.
1. In the **Share externally** drawer that opens, do one of the following:
   - Click **Pause access** so that people can't access the dashboard, but the link is maintained.
   - Click **Resume access** so that people can access the dashboard again.
   - Click **Revoke access** so that people can't access the dashboard unless a new external link is generated. Confirm that you want to revoke the link.
1. Click the **X** at the top-right corner to close the share drawer.

## Assess shared dashboard usage

{{< admonition type="note" >}}
Available in [Grafana Enterprise](ref:grafana-enterprise) and [Grafana Cloud](/docs/grafana-cloud).
{{< /admonition >}}

You can check usage analytics about your externally shared dashboard by clicking the insights icon in the dashboard header:

![Dashboard insights icon](/media/docs/grafana/dashboards/screenshot-dashboard-insights-icon-11.2.png)

Learn more about the kind of information provided in the [dashboard insights documentation](ref:dashboard-insights-documentation).

## Supported data sources

Externally shared dashboards _should_ work with any data source that has the properties `backend` and `alerting` both set to true in its `plugin.json`. However, this can't always be
guaranteed because plugin developers can override this functionality. The following lists include data sources confirmed to work with externally shared dashboards and data sources that should work, but have not been confirmed as compatible.

### Confirmed

{{< column-list >}}

- ClickHouse
- CloudWatch
- Elasticsearch
- Infinity
- InfluxDB
- Loki
- Microsoft SQL Server
- MongoDB
- MySQL
- Oracle Database
- PostgreSQL
- Prometheus
- Redis
- SQLite

{{< /column-list >}}

### Unsupported

- DynamoDB
- Dynatrace
- Graphite
- Google Sheets
- Tempo
- Zabbix

### Unconfirmed

{{< column-list >}}

- Altinity plugin for ClickHouse
- Amazon Athena
- Amazon Redshift
- Amazon Timestream
- Apache Cassandra
- AppDynamics
- Azure Data Explorer Datasource
- Azure Monitor
- CSV
- DB2 Datasource
- Databricks
- Datadog
- Dataset
- Druid
- GitHub
- Google BigQuery
- Grafana for YNAB
- Honeycomb
- Jira
- Mock
- Neo4j Datasource
- New Relic
- OPC UA (Unified Architecture)
- Open Distro for Elasticsearch
- OpenSearch
- OpenTSDB
- Orbit
- SAP HANA®
- Salesforce
- Sentry
- ServiceNow
- Snowflake
- Splunk
- Splunk Infrastructure Monitoring
- Sqlyze data source
- TDengine
- Vertica
- Wavefront
- X-Ray
- kdb+
- simple grpc data source

{{< /column-list >}}

## Limitations

- Panels that use frontend data sources will fail to fetch data.
- Variables and queries including variables are not supported.
- Exemplars will be omitted from the panel.
- Only annotations that query the `-- Grafana --` data source and use the query type `Annotations & Alerts` are supported.
- Organization annotations are not supported.
- Tag-based annotation queries, using the `-- Grafana --` data source with **Filter by Tags**, return matching annotations from _all dashboards_ in the organization, not only the shared dashboard. This means annotations from dashboards that are not shared externally can be visible to anyone viewing the shared dashboard. This is by design. If you use tag-based annotation queries on a shared dashboard, ensure you are comfortable with the annotations from other dashboards in your organization being exposed to external viewers.
- Grafana Live and real-time event streams are not supported.
- Library panels are not supported.
- Data sources using Reverse Proxy functionality are not supported.

## Custom branding

If you're a Grafana Enterprise customer, you can use custom branding to change the appearance of an externally shared dashboard footer. For more information, refer to [Custom branding](ref:custom-branding).

# Troubleshoot dashboards

Use the following strategies to help you troubleshoot common dashboard problems.

## Dashboard is slow

- Are you trying to render dozens (or hundreds or thousands) of time series on a graph? This can cause the browser to lag. Try using functions like `highestMax` (in Graphite) to reduce the number of returned series.
- Sometimes series names can be very large. This causes larger response sizes. Try using `alias` to reduce the size of the returned series names.
- Are you querying many time series or a long time range? Both of these conditions can cause Grafana or your data source to pull in a lot of data, which may slow the dashboard down. Try reducing one or both of these.
- There could be high load on your network infrastructure. If the slowness isn't consistent, this may be the problem.

## Dashboard refresh rate issues

By default, Grafana queries your data source every 30 seconds. However, setting a low refresh rate on your dashboards puts unnecessary stress on the backend. In many cases, querying this frequently isn't necessary because the data source isn't sending data often enough for there to be changes every 30 seconds.

We recommend the following:

- Only enable auto-refreshing on dashboards, panels, or variables if necessary. Users can refresh their browser manually.
- If you require auto-refreshing, then set the refresh rate to a longer time period that makes sense, such as once a minute, every 10 minutes, or every hour.
- Check the time range of your dashboard. If your dashboard has a longer time range, such as a week, then you really don't need automated refreshing and you should disable it.

## Handling or rendering null data is wrong or confusing

Some applications publish data intermittently; for example, they only post a metric when an event occurs. By default, Grafana graphs connect lines between the data points, but this can be deceptive.

The graph in the following image has:

- Points and 3-point radius enabled to highlight where data points are actually present.
- **Connect null values** set to **Always**.

{{< figure src="/static/img/docs/troubleshooting/grafana_null_connected.png" max-width="1200px" alt="Graph with null values connected" >}}

The graph in this next image shows bars instead of lines and has the **No value** option under **Standard options** set to **0**.

{{< figure src="/static/img/docs/troubleshooting/grafana_null_zero.png" max-width="1200px" alt="Graph with null values not connected" >}}

As you can see, there's a significant difference in the visualizations.

## Tooltip doesn't stay open when clicked

If you click a data point expecting the tooltip to stay open so you can scroll or copy its content, but the tooltip immediately disappears or resets, the **Refresh live dashboards** setting is likely the cause.

When you enable **Refresh live dashboards** on a dashboard, Grafana continuously re-renders panels as data arrives.
If the refresh interval is too low, each re-render dismisses any pinned tooltip before you can interact with it.

To fix this, do one of the following:

- **Disable live refresh**: Open the dashboard settings, go to **General**, and toggle off the **Refresh live dashboards** switch. Tooltips remain open on click after the next render cycle.
- **Increase the refresh interval**: If you need live updates, open the dashboard settings, go to **General**, and increase the **Auto refresh** value. This provides enough time between dashboard re-renders for the tooltip to remain open.

# Use dashboards

This topic provides an overview of dashboard features and shortcuts, and describes how to use dashboard search.

{{< youtube id="vTiIkdDwT-0" >}}

## Dashboard feature overview

The dashboard user interface provides a number of features that you can use to customize the presentation of your data.

The following image and descriptions highlight all dashboard features.
Hover your cursor over a number to display information about the dashboard element.

{{< image-map key="annotated-dashboard" >}}

## Keyboard shortcuts

Grafana has a number of keyboard shortcuts available. Press `?` on your keyboard to display all keyboard shortcuts available in your version of Grafana.

- `Ctrl+S`: Saves the current dashboard.
- `f`: Opens the dashboard finder / search.
- `d+k`: Toggle kiosk mode (hides the menu).
- `d+e`: Expand all rows.
- `d+s`: Dashboard settings.
- `Ctrl+K`: Opens the command palette.
- `Esc`: Exits panel when in full screen view or edit mode. Also returns you to the dashboard from dashboard settings.

### Focused panel

By hovering over a panel with the mouse you can use some shortcuts that will target that panel.

- `e`: Toggle panel edit view
- `v`: Toggle panel full screen view
- `pu`: Copy panel share link
- `pe`: Open share panel embed configuration
- `ps`: Open share panel snapshot configuration
- `pd`: Duplicate panel
- `pr`: Remove panel

## Kiosk mode

In kiosk mode, the main menu and top navigation bar of a dashboard are hidden.
This can be useful if you want to display as much information as possible on the screen or use the dashboard to present information to a wider audience.

- **Turn on kiosk mode**: Click your user icon and select **Enable kiosk mode**.
- **Turn off kiosk mode**: Press `Esc`.

## Set dashboard time range

Grafana provides several ways to manage the time ranges of the data being visualized, for dashboard, panels and also for alerting.

This section describes supported time units and relative ranges, the common time controls, dashboard-wide time settings, and panel-specific time settings.

### Time units and relative ranges

Grafana supports the following time units: `s (seconds)`, `m (minutes)`, `h (hours)`, `d (days)`, `w (weeks)`, `M (months)`, `Q (quarters)` and `y (years)`.

The minus operator enables you to step back in time, relative to the current date and time, or `now`. If you want to display the full period of the unit (day, week, month, etc...), append `/<time unit>` to the end. To view fiscal periods, use `fQ (fiscal quarter)` and `fy (fiscal year)` time units.

The plus operator enables you to step forward in time, relative to now. For example, you can use this feature to look at predicted data in the future.

The following table provides example relative ranges:

| Example relative range | From:       | To:         |
| ---------------------- | ----------- | ----------- |
| Last 5 minutes         | `now-5m`    | `now`       |
| The day so far         | `now/d`     | `now`       |
| This week              | `now/w`     | `now/w`     |
| This week so far       | `now/w`     | `now`       |
| This month             | `now/M`     | `now/M`     |
| This month so far      | `now/M`     | `now`       |
| Previous Month         | `now-1M/M`  | `now-1M/M`  |
| This year so far       | `now/Y`     | `now`       |
| This Year              | `now/Y`     | `now/Y`     |
| Previous fiscal year   | `now-1y/fy` | `now-1y/fy` |

{{< admonition type="note" >}}

Grafana Alerting does not support the following syntaxes at this time:

- now+n for future timestamps.
- now-1n/n for "start of n until end of n" because this is an absolute timestamp.

{{< /admonition >}}

### Common time range controls

The dashboard and panel time controls have a common UI.

![Common time controls](/media/docs/grafana/dashboards/screenshot-common-time-controls-11.2.png)

The following sections define common time range controls.

#### Current time range

The current time range, also called the _time picker_, shows the time range currently displayed in the dashboard or panel you are viewing.

Hover your cursor over the field to see the exact time stamps in the range and their source (such as the local browser).

![Time picker](/media/docs/grafana/dashboards/screenshot-time-picker-11.2.png)

Click the current time range to change it. You can change the current time using a _relative time range_, such as the last 15 minutes, or an _absolute time range_, such as `2020-05-14 00:00:00 to 2020-05-15 23:59:59`.

![Current time range](/media/docs/grafana/dashboards/screenshot-current-time-range-11.2.png)

#### Relative time range

Select the relative time range from the **Relative time ranges** list. You can filter the list using the input field at the top. Some examples of time ranges include:

- Last 30 minutes
- Last 12 hours
- Last 7 days
- Last 2 years
- Yesterday
- Day before yesterday
- This day last week
- Today so far
- This week so far
- This month so far

You can enter a custom relative time range into the search at the top to quickly select, such as `13h` to select a time range for the last 13 hours.

#### Absolute time range

You can set an absolute time range in the following ways:

- Type values into the **From** and **To** fields. You can type exact time values or relative values, such as `now-24h`, and then click **Apply time range**.
- Click in the **From** or **To** field. Grafana displays a calendar. Click the day or days you want to use as the current time range and then click **Apply time range**.

This section also displays recently used absolute ranges.

#### Semi-relative time range

{{< admonition type="note" >}}

Grafana Alerting does not support semi-relative time ranges.

{{< /admonition >}}

You can also use the absolute time range settings to set a semi-relative time range. Semi-relative time range dashboards are useful when you need to monitor the progress of something over time, but you also want to see the entire history from a starting point.

Set a semi-relative time range by setting the start time to an absolute timestamp and the end time to a “now” that is relative to the current time. For example:

**Start time:** `2023-05-01 00:00:00`

**End time:** `now`

If you wanted to track the progress of something during business hours, you could set a time range that covers the current day, but starting at 8am, like so:

**Start time:** `now/d+8h`

**End time:** `now`

This is equivalent to the **Today so far** time range preset, but it starts at 8:00am instead of 12:00am by appending +8h to the periodic start time.

Using a semi-relative time range, as time progresses, your dashboard will automatically and progressively zoom out to show more history and fewer details. At the same rate, as high data resolution decreases, historical trends over the entire time period will become more clear.

#### Copy and paste time range

You can copy and paste the time range from a dashboard to **Explore** and vice versa, or from one dashboard to another.
Click the **Copy time range to clipboard** icon to copy the current time range to the clipboard. Then paste the time range into **Explore** or another dashboard.

<img class="no-shadow" src="/media/docs/grafana/dashboards/screenshot-copy-paste-time-range.png" max-width="900">

You can also copy and paste a time range using the keyboard shortcuts `t+c` and `t+v` respectively.

#### Zoom out

- Click the **Zoom out** icon to view a larger time range in the dashboard or panel visualizations
- Double click on the panel graph area (time series family visualizations only)
- Type the `t-` keyboard shortcut

#### Zoom in

- Click and drag horizontally in the panel graph area to select a time range (time series family visualizations only)
- Type the `t+` keyboard shortcut

#### Refresh dashboard

Click the **Refresh dashboard** icon to immediately run every query on the dashboard and refresh the visualizations. Grafana cancels any pending requests when you trigger a refresh.

By default, Grafana does not automatically refresh the dashboard. Queries run on their own schedule according to the panel settings. However, if you want to regularly refresh the dashboard, click the down arrow next to the **Refresh dashboard** icon, and then select a refresh interval.

Selecting the **Auto** interval schedules a refresh based on the query time range and browser window width. Short time ranges update frequently, while longer ones update infrequently. There is no need to refresh more often then the pixels available to draw any updates.

### Control the time range using a URL

{{< docs/shared lookup="dashboards/time-range-URLs.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Filter dashboard data

Filters help you narrow the data displayed in a dashboard.

After you've added a filter in the dashboard settings, you can create label/value filter pairs on the dashboard.
These filters are applied to all metric queries that use the specified data source and to all panels on the dashboard.

To learn more, refer to the [Filter and Group by documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/filter-group-by/).

# Add variables

You can create your own variables within any dashboard, and these variables are specific to the dashboard in which you create them.
For information on variables that you can use across dashboards, refer to the [Cross-dashboard variables documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/cross-dashboard-variables/).

The following table lists the types of variables you can create:

<!-- prettier-ignore-start -->

| Variable type     |Description                                                                                                                                                   |
| ----------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Query             | Query-generated list of values such as metric names, server names, sensor IDs, data centers, and so on. [Add a query variable](#add-a-query-variable).       |
| Custom            | Define the variable options manually using a comma-separated list. [Add a custom variable](#add-a-custom-variable).                                          |
| Text box          | Display a free text input field with an optional default value. [Add a text box variable](#add-a-text-box-variable).                                         |
| Constant          | Define a hidden constant. [Add a constant variable](#add-a-constant-variable).                                                                               |
| Data source       | Quickly change the data source for an entire dashboard. [Add a data source variable](#add-a-data-source-variable).                                           |
| Interval          | Interval variables represent time spans. [Add an interval variable](#add-an-interval-variable).                                                              |
| Switch            | Display a switch that allows you to toggle between two configurable values for enabled and disabled states. [Add a switch variable](#add-a-switch-variable). |

<!-- prettier-ignore-end -->

<!-- vale Grafana.WordList = NO -->
<!-- vale Grafana.Spelling = NO -->

{{< admonition type="note" >}}
In Grafana 13.1, **Ad hoc filters** were renamed **Filter and Group by** and extended by adding grouping for Prometheus and Loki data sources.
However, in the dashboard schema, they're still referred to as `"kind": "AdhocVariable"`.

For information on the **Filter and Group by** feature, refer to the [documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/filter-group-by/).
{{< /admonition >}}

<!-- vale Grafana.WordList = YES -->
<!-- vale Grafana.Spelling = YES -->

## Enter general options for any variable

To create a variable, follow these steps:

{{< docs/list >}}

{{< shared id="add-variable" >}}

1. Navigate to the dashboard you want to update.
1. Click **Edit**.
1. Click the plus sign **Add new element** icon in the sidebar and select **Variable**.

   You can also click the **Add variable** plus sign icon at the top of the dashboard.

1. Choose a variable type from the list.

   You can update the variable type by clicking **Change type** in the sidebar.

1. Enter a **Name** for the variable.
1. (Optional) In the **Label** field, enter the display name for the variable drop-down list.

   If you leave this empty, then Grafana uses the variable name.

1. (Optional) In the **Description** field, enter a description of the variable. The description appears as an info icon tooltip next to the variable name on the dashboard.

   Descriptions support links. You can use Markdown-style links like `[link text](https://example.com)`, or paste bare URLs like `https://example.com`. Only `http` and `https` URLs render as clickable links—other protocols display as plain text.

1. Choose a **Display** option:
   - **Above dashboard** - The variable drop-down list displays above the dashboard with the variable **Name** or **Label** value. This is the default.
   - **Above dashboard, label hidden** - The variable drop-down list displays above the dashboard, but without showing the name of the variable.
   - **Controls menu** - The variable displays in the dashboard controls menu instead of above the dashboard. The dashboard controls menu appears as a button in the dashboard toolbar.
   - **Hidden** - No variable drop-down list displays on the dashboard.

{{< /shared >}}

1. Click one of the following links to complete the steps for adding your selected variable type:
   - [Query](#add-a-query-variable)
   - [Custom](#add-a-custom-variable)
   - [Textbox](#add-a-text-box-variable)
   - [Constant](#add-a-constant-variable)
   - [Data source](#add-a-data-source-variable)
   - [Interval](#add-an-interval-variable)
   - [Switch](#add-a-switch-variable)

{{< /docs/list >}}

<!-- vale Grafana.Spelling = YES -->

### Variable best practices

- Variable drop-down lists display in the order in which they're listed in the **Variables** section of the sidebar, so put the variables that you change often at the top, so they're shown first.
- By default, variables don't have a default value. This means that the topmost value in the drop-down list is always preselected. If you want to pre-populate a variable with an empty value, you can use the following workaround in the variable settings:
  1. Select the **Include All Option** checkbox.
  1. In the **Custom all value** field, enter a value like `.+`.

## Add a query variable

Query variables enable you to write a data source query that can return a list of metric names, tag values, or keys. For example, a query variable might return a list of server names, sensor IDs, or data centers. The variable values change as they dynamically fetch options with a data source query.

Query variables are generally only supported for strings. If your query returns numbers or any other data type, you might need to convert them to strings to use them as variables. For the Azure data source, for example, you can use the [`tostring`](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/tostringfunction) function for this purpose.

Query expressions can contain references to other variables and in effect create linked variables. Grafana detects this and automatically refreshes a variable when one of its linked variables change.

{{< admonition type="note" >}}
Query expressions are different for each data source. For more information, refer to the documentation for your [data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/).
{{< /admonition >}}

1. [Enter general options](#enter-general-options-for-any-variable).
1. Click **Open variable editor**.
1. Set the following options to configure your query in the **Query** tab:

   | Option             | Description                                                                                                                                                                                                                                                                                                                                            |
   | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
   | Target data source | Select a data source in the drop-down list. You can also click **Open advanced data source picker** to see more options, including adding a data source (Admins only). For more information about data sources, refer to [Add a data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/#add-a-data-source).                       |
   | Query type         | If applicable, select an option in the drop-down list and fill in the query fields accordingly. For more information, refer to the [Query type](#query-type) section following these steps.                                                                                                                                                            |
   | Regex              | (Optional) Type a regular expression in the field to filter or capture specific parts of the names returned by your data source query. To see examples, refer to [Filter variables with a regular expression](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/advanced-variables/#filter-variables-with-regex). |
   | Apply regex to     | Select **Variable value** or **Display text** to choose where the regular expression pattern is applied. The default is **Variable value**.                                                                                                                                                                                                            |
   | Sort               | Select the sort order for values to display in the drop-down list. The default option, **Disabled**, means that the order of options returned by your data source query is used.                                                                                                                                                                       |

1. In the **Static options** tab, set the following options:
   - Select an option in the **Static options sort** drop-down list to control where in the list of variable values the static options appear.
   - Under **value**, click **+ Add new option** or **Paste from clipboard** to add static options.

1. Click **Preview** in the top of the dialog box to test the variable.

   Grafana displays a list of the current variable values.

1. In the **Preview of values** section, Review them to ensure they match what you expect.
1. Click **Apply** to close the query variable editor dialog box.
1. In the **Refresh** drop-down list, select an option to control when the variable should update:
   - **On dashboard load** - Queries the data source every time the dashboard loads. This slows down dashboard loading, because the variable query needs to be completed before the dashboard can be initialized.
   - **On time range change** - Queries the data source every time the dashboard loads and when the dashboard time range changes. Use this option if your variable options query contains a time range filter or is dependent on the dashboard time range.

1. (Optional) Set [Selection options](#configure-variable-selection-options) if needed:
   - **Multi-value** - Enables the selection of select multiple values at the same time.
   - **Include All value** - Enables an option to include all variables.
   - **Allow custom values** - Enables users to add custom values to the list. Only applies to CSV custom values.

1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Exit edit**.

### Query type

The query section of the editor varies according to your data source.
Some data sources have custom query editors.

Each data source defines how it extracts variable values.
The typical implementation uses every string value returned from the data source response as a variable value.
Make sure to double-check the documentation for the data source.

Some data sources let you provide custom "display names" for the values.
For instance, the PostgreSQL, MySQL, and Microsoft SQL Server plugins handle this by looking for fields named `__text` and `__value` in the result.
Other data sources may look for `text` and `value` or use a different approach.
Always remember to double-check the documentation for the data source.

If you need more room in a single input field query editor, then hover your cursor over the lines in the lower right corner of the field and drag downward to expand.

## Add a custom variable

Use a _custom_ variable for a value that doesn't change, such as a number or a string.

For example, if you have server names or region names that never change, then you might want to create them as custom variables rather than query variables. Because they don't change, you might use them in [chained variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/advanced-variables/#chained-variables) rather than other query variables. That would reduce the number of queries Grafana must send when you update chained variables.

1. [Enter general options](#enter-general-options-for-any-variable).
1. Click **Open variable editor**.
1. Configure one of the following options:
   - **CSV** - Enter a flat list of values for the variable in a comma-separated list. You can include numbers, strings, or key/value pairs separated by a space and a colon. For example, `key1 : value1,key2 : value2`.
   - **JSON** - Provide a JSON array of objects where each object can have any number of properties that you can reference. For more information refer, to [Configure multi-property variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/advanced-variables/#multi-property-variables).

1. Click **Preview** to test the variable.
1. In the **Preview of values** section, Grafana displays a list of the current variable values. If you've entered a JSON array, the preview is a table that includes all the value properties. Review them to ensure they match what you expect.
1. Click **Apply** to close the query editor dialog box.
1. (Optional) Set [Selection options](#configure-variable-selection-options) if needed:
   - **Multi-value** - Enables the selection of multiple values at the same time.
   - **Include All value** - Enables an option to include all variables.
   - **Allow custom values** - Enables users to add custom values to the list. Only applies to CSV custom values.

1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Exit edit**.

## Add a textbox variable {#add-a-text-box-variable}

_Textbox_ variables display a free text input field with an optional default value. This is the most flexible variable, because you can enter any value. Use this type of variable if you have metrics with high cardinality or if you want to update multiple panels in a dashboard at the same time.

For more information about cardinality, refer to [What are cardinality spikes and why do they matter?](https://grafana.com/blog/2022/02/15/what-are-cardinality-spikes-and-why-do-they-matter/)

1. [Enter general options](#enter-general-options-for-any-variable).
1. (Optional) Under **Textbox options**, enter the default value for the variable in the **Value** field.

   If you don't enter anything in this field, then Grafana displays an empty text box for users to type text into.

1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Exit edit**.

## Add a constant variable

_Constant_ variables enable you to define a hidden constant. This is useful for metric path prefixes for dashboards you want to share. When you export a dashboard, Grafana converts constant variables to import options.

Constant variables are _not_ flexible. Each constant variable only holds one value, and you can't update it unless you update the variable settings.

Constant variables are useful when you have complex values that you need to include in queries but don't want to retype in every query. For example, if you had a server path called `i-0b6a61efe2ab843gg`, then you could replace it with a variable called `$path_gg`.

1. [Enter general options](#enter-general-options-for-any-variable).
1. Under **Constant options**, enter the variable value in the **Value** field.

   You can enter letters, numbers, and symbols. You can even use wildcards if you use [raw format](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/variable-syntax/#raw).

1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Exit edit**.

## Add a data source variable

_Data source_ variables enable you to quickly change the data source for an entire dashboard. They're useful if you have multiple instances of a data source, perhaps in different environments.

1. [Enter general options](#enter-general-options-for-any-variable).
1. Configure the following **Data source options**:
   - **Type** - Select the target data source for the variable.
   - **Name filter** - (Optional) Enter a regular expression filter for which data source instances to choose from in the variable value drop-down list. Leave this field empty to display all instances.

1. (Optional) Set [Selection options](#configure-variable-selection-options) if needed:
   - **Multi-value** - Enables the selection of multiple values at the same time.
   - **Include All value** - Enables an option to include all variables.
   - **Allow custom values** - Enables users to add custom values to the list. Only applies to CSV custom values.

1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Exit edit**.

## Add an interval variable

Use an _interval_ variable to represents time spans such as `1m`,`1h`, `1d`. You can think of them as a dashboard-wide "group by time" command. Interval variables change how the data is grouped in the visualization. You can also use the Auto Option to return a set number of data points per time span.

You can use an interval variable as a parameter to group by time (for InfluxDB), date histogram interval (for Elasticsearch), or as a summarize function parameter (for Graphite).

1. [Enter general options](#enter-general-options-for-any-variable).
1. Configure the following **Interval options**:

   | Option       | Description                                                                                                                                                                                                                                                                                                                                                                      |
   | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | Values       | Enter the time range intervals that you want to appear in the variable drop-down list. Grafana supports the following time units: `s (seconds)`, `m (minutes)`, `h (hours)`, `d (days)`, `w (weeks)`, `M (months)`, and `y (years)`. You can also accept or edit the default values: `1m,10m,30m,1h,6h,12h,1d,7d,14d,30d`.                                                       |
   | Auto option  | (Optional) Select on the checkbox if you want to add the `auto` option to the list. This option allows you to specify how many times to divide the current time range to calculate the current `auto` time span.                                                                                                                                                                 |
   | Step count   | Select the number of times to divide the current time range to calculate the value, similar to the **Max data points** query option. For example, if the current visible time range is 30 minutes, then the `auto` interval groups the data into 30 one-minute increments. The default value is 30 steps. This option only displays hen you select the **Auto option** checkbox. |
   | Min interval | The minimum threshold below which the step count intervals doesn't divide the time. To continue the 30 minute example, if the minimum interval is set to 2m, then Grafana would group the data into 15 two-minute increments. Only displayed when you select the **Auto option** checkbox.                                                                                       |

1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Exit edit**.

### Interval variable examples

The following example shows a template variable `myinterval` in a Graphite function:

```
summarize($myinterval, sum, false)
```

The following example shows a more complex Graphite example, from the [Graphite Template Nested Requests panel](https://play.grafana.org/d/dbg1erw7kech6oe/placeholder-dashboard):

```
groupByNode(summarize(movingAverage(apps.$app.$server.counters.requests.count, 5), '$interval', 'sum', false), 2, 'sum')
```

## Add a switch variable

_Switch_ variables display a switch with two configurable values representing enabled and disabled states. This variable type is useful when you need to:

- Toggle between different query conditions
- Enable or disable specific filters
- Switch between different visualization modes
- Control boolean parameters in your data sources

1. [Enter general options](#enter-general-options-for-any-variable).
1. Configure the following **Switch options**:

   | Option          | Description                                                                                                                                                                                                                                                                                                                              |
   | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | Value pair type | Select one of the following predefined options:<ul><li>**True / False** - Uses boolean values `true` and `false`.</li><li>**1 / 0** - Uses numeric values `1` and `0`.</li><li>**Yes / No** - Uses string values `yes` and `no`.</li><li>**Custom** - Allows you to define custom values for both enabled and disabled states.</li></ul> |
   | Enabled value   | If you selected **Custom**, configure the custom values. Enter the value that represents the enabled state (for example, "on").                                                                                                                                                                                                          |
   | Disabled value  | If you selected **Custom**, enter the value that represents the disabled state (for example, "off").                                                                                                                                                                                                                                     |

1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Exit edit**.

### Switch variable examples

The following example shows a switch variable `$debug_mode` used in a Prometheus query to conditionally include debug labels:

```
up{job="my-service"} and ($debug_mode == "true" or on() vector(0))
```

The following example shows a switch variable `$show_errors` used to filter log entries:

```
{job="application"} |= ($show_errors == "1" ? "ERROR" : "")
```

You can also use switch variables in panel titles and other dashboard elements:

```
{{#if debug_mode}}Debug Mode: {{/if}}Application Metrics
```

<!-- vale Grafana.Spelling = YES -->
<!-- vale Grafana.WordList = YES -->

## Variable selection options {#configure-variable-selection-options}

**Selection Options** are a feature you can use to manage variable option selections for query and custom variables. All selection options are optional, and they're off by default.

### Multi-value variables

Interpolating a variable with multiple values selected is tricky as it's not straight forward how to format the multiple values into a string that's valid in the given context where you use the variable. Grafana tries to solve this by allowing each data source plugin to inform the templating interpolation engine what format to use for multiple values.

{{< admonition type="note" >}}
The **Custom all value** option on the variable must be blank for Grafana to format all values into a single string. If it's left blank, then Grafana concatenates (adds together) all the values in the query. Something like `value1,value2,value3`. If you use a custom `all` value, then instead the value is something like `*` or `all`.
{{< /admonition >}}

#### Multi-value variables with a Graphite data source

Graphite uses glob expressions. In this case, Grafana interpolates a variable with multiple values as `{host1,host2,host3}` if the current variable value is _host1_, _host2_, and _host3_.

#### Multi-value variables with a Prometheus or InfluxDB data source

InfluxDB and Prometheus use regular expressions, so Grafana interpolates the same variable as `(host1|host2|host3)`. Grafana also escapes every value for regular expressions. If not, a value with a regular expression control character would break the regular expression.

#### Multi-value variables with an Elastic data source

Elasticsearch uses Lucene query syntax, so Grafana formats the same variable as `("host1" OR "host2" OR "host3")`. In this case, every value must be escaped so that the value only contains Lucene control words and quotation marks.

#### Variable indexing

If you have a multi-value variable that's formatted as an array, you can use array positions to reference the values rather than the actual values.
You can use this functionality in dashboard panels to filter data, and maintain the array.

To reference variable values this way, use the following syntax:

```text
${query0.0}
```

The preceding syntax references the first, or `0`, position in the array.

In the following example, there's an array of three values, `1t`, `2t`, and `3t`, and rather than referencing those values, the panel query references the second value in the array using the syntax `${query0.1}`:

{{< figure src="/media/docs/grafana/dashboards/screenshot-indexed-variables-v12.1.png" max-width="750px" alt="Panel query using variable indexing to reference a value" >}}

#### Troubleshoot multi-value variables

Automatic escaping and formatting can cause problems and it can be tricky to grasp the logic behind it. Especially for InfluxDB and Prometheus where regular expression syntax requires you to use the variable in a regular expression operator context.

If you don't want Grafana to do this automatic regular expression escaping and formatting, then you must do one of the following:

- Turn off the **Multi-value** or **Include All option** options.
- Use the [raw variable format](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/variable-syntax/#raw).

### Include All option

Grafana adds an **All** option to the variable drop-down list. If a user selects this option, then all variable options are selected.

If you enable this option but also set a variable's **Display** option to **Hidden**, the variable defaults to selecting only the first value&mdash;not **All**&mdash;unless you explicitly select **All** while the variable is visible and then save the dashboard.
To set **All** as the default for a hidden variable, complete the following steps:

1. Temporarily set **Display** to **Above dashboard**.
1. On the dashboard, select the **All** option from the variable drop-down list.
1. Save the dashboard to persist the selection.
1. Return to **Dashboard options** > **Variables**, select the variable, and set **Display** back to **Hidden**.
1. Save the dashboard again.

### Custom all value

This option is only visible if you select the **Include All option**.

Enter regular expressions, globs, or Lucene syntax in the **Custom all value** field to define the value of the `All` option.

By default the `All` value includes all options in combined expression. This can become very long and can have performance problems. Sometimes it can be better to specify a custom all value, like a wildcard regular expression.

In order to have custom regular expression, globs, or Lucene syntax in the **Custom all value** option, it's never escaped so you have to think about what's a valid value for your data source.

# Advanced variable configurations

The advanced variable configurations described on this page help you build dashboards that adapt to complex data structures and user selections:

- **Multi-property variables**: Map one logical value to multiple source-specific values.
- **Chained variables**: Update the options in one variable based on the selections in another.
- **Regular expression filtering**: Filter the values returned by a variable query.

These configurations are useful for dashboards that span multiple environments, data sources, regions, clusters, namespaces, or other hierarchical data.

## Multi-property variables

If a multi-source dashboard uses multiple values for the same logical concept, such as an environment identified as `dev` in one source and `development` in another, use a multi-property variable to reference those values from one variable. This lets you map related identifiers together and reference the property you need in panels and queries.

To do so, configure a JSON array in which each object can have any number of properties.
Then, you can reference any of the properties as you use those variables.

This applies to the following variable types:

- Custom
- Query
  - Infinity
  - PostgreSQL

<!-- add links to data source docs -->

### Multi-property custom variables

To create a custom variable with multiple properties, define a JSON array, like this:

```json
[
  { "value": "1", "text": "Development", "aws": "dev", "azure": "development", "google": "googledev" },
  { "value": "2", "text": "Staging", "aws": "stag", "azure": "staging", "google": "googlestag" },
  { "value": "3", "text": "Production", "aws": "prod", "azure": "production", "google": "googleprod" }
]
```

This is how those values are displayed in a preview:

{{< figure src="/media/docs/grafana/dashboards/screenshot-multipropvar-preview-vals-v12.4.png" max-width="600px" alt="Custom variable configuration and preview of values" >}}

Then you can use `${varName.someProperty}` syntax to reference any property in your dashboard panels or metrics:

{{< figure src="/media/docs/grafana/dashboards/screenshot-multipropvar-used-12.4.png" max-width="750px" alt="Multi-property variable used in a text panel" >}}

You can even deeply nest properties and still access them using familiar variable syntax. In the following example, each user has an address property with all the elements of an address nested within it:

{{< figure src="/media/docs/grafana/dashboards/screenshot-multipropvar-nested-v12.4.png" max-width="650px" alt="Nested variable configuration" >}}

### Multi-property query variables

Because query configuration is different for each data source, there's no one way to set up a JSON array to create a multi-property query variable.
For PostgreSQL, refer to [PostgreSQL template variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/postgres/template-variables/).
For other data sources, refer to the relevant [Data sources documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/) for preinstalled data source plugins.
For configuration information on all other data source plugins, refer to the [Plugins documentation](https://grafana.com/docs/plugins/).

## Chained variables

_Chained variables_, also called _linked variables_ or _nested variables_, are query variables with one or more other variables in their variable query.

Chained variables create dependent relationships where the options available in one variable depend on the selections made in another.
This pattern enables cascading filters that help users drill down through hierarchical data structures.

Cascading filter patterns are common in multi-tier filtering scenarios such as region > cluster > namespace > pod.
Each level narrows the available options based on the previous selection, creating an intuitive navigation experience through your infrastructure or data hierarchy.

Technically, there is no limit to how many variables you can chain together, but the more links you have, the greater the query load.

### Grafana Play dashboard examples

<!-- If you update or replace the Play dashboards linked in this section, you must also update the text examples to match them -->

The following Grafana Play dashboards contain chained variables. To view the variables and their settings, click **Edit** in the top-right corner of the dashboard, click the **Dashboard options** icon in the toolbar, and expand the **Variables** section. Both examples are expanded in the following section.

- [Templating - Nested Variables Drilldown](https://play.grafana.org/d/testdata-nested-variables-drilldown/templating-nested-variables-drilldown?orgId=1)
- [InfluxDB Templated](https://play.grafana.org/d/dbg1erw7kech6oe/placeholder-dashboard)

While the examples are data source-specific, the concepts can be applied broadly.

#### Templating - Nested Variables Drilldown example explained

In this example, there are several data centers. Each data center has a different set of servers, and each server has a different set of pods. The example uses the TestData data source, whose variable queries use dot-separated paths with `*` wildcards, similar to Graphite metric paths.

You could make separate variables for data centers, servers, and pods, but then you have to know which server belongs to which data center, and which pod runs on which server. A better solution is to use one variable to filter the next. In this example, when you change the value of the `datacenter` variable, it changes the drop-down options returned by the `server` variable. When you change the value of the `server` variable, it changes the drop-down options returned by the `pod` variable. All three variables use the **Multi-value** option and **Include all option**, so you can select some or all options at any time.

##### `datacenter` variable

The query for this variable returns all the data centers that exist.

```
*
```

The values returned are `A` through `H`, and `All`.

##### `server` variable

The query for this variable returns all servers for the currently selected data centers.

```
$datacenter.*
```

If you select `A`, then the query changes to:

```
A.*
```

The query returns all servers in data center `A`: `AA`, `AB`, and so on, through `AH`.

If you select `B`, then the query changes to:

```
B.*
```

The query returns all servers in data center `B`: `BA`, `BB`, and so on, through `BH`.

Because `datacenter` is a multi-value variable, you can select more than one data center. If you select both `A` and `B`, then the query changes to:

```
{A,B}.*
```

The query returns the servers in both data centers, `AA` through `AH` and `BA` through `BH`.

##### `pod` variable

The query for this variable returns all pods for the currently selected data centers and servers.

```
$datacenter.$server.*
```

If you select data center `A` and server `AB`, then the query changes to:

```
A.AB.*
```

The query returns all pods on server `AB`: `ABA`, `ABB`, and so on, through `ABH`.

The dashboard includes a text panel that displays the current values of `$datacenter`, `$server`, and `$pod`, so you can see how each selection narrows the options for the next variable.

#### InfluxDB example explained

In this example, you have several data centers. Each data center has a different subset of hosts. It's based on the [InfluxDB Templated](https://play.grafana.org/d/dbg1erw7kech6oe/placeholder-dashboard) dashboard.

In this example, when the user changes the value of the `datacenter` variable, it changes the drop-down options returned by the `host` variable. The `host` variable uses the **Multi-value** option and **Include all option**, allowing users to select some or all options presented at any time. The `datacenter` does not use either option, so you can only select one data center at a time.

##### `datacenter` variable

The query for this variable returns all the data centers that exist.

```
SHOW TAG VALUES WITH KEY = "datacenter"
```

The values returned are `America`, `Africa`, `Asia`, and `Europe`.

##### `host` variable

The query for this variable returns all hosts for the currently selected data center.

```
SHOW TAG VALUES WITH KEY = "hostname" WHERE "datacenter" =~ /^$datacenter$/
```

If the user selects `America`, then the query changes to:

```
SHOW TAG VALUES WITH KEY = "hostname" WHERE "datacenter" =~ /^America/
```

The query returns all servers associated with `America`, including `server1`, `server2`, and so on.

If the user selects `Europe`, then the query changes to:

```
SHOW TAG VALUES WITH KEY = "hostname" WHERE "datacenter" =~ /^Europe/
```

The query returns all servers associated with `Europe`, including `server3`, `server4`, and so on.

##### More variables

{{< admonition type="note" >}}
This example is theoretical. The InfluxDB server used in the example doesn't contain CPU metrics.
{{< /admonition >}}

The dashboard stops at two levels, but you could keep going. For example, if you wanted to get CPU metrics for selected hosts, you could copy the `host` variable and extend the query so that it reads:

```
SHOW TAG VALUES WITH KEY = "cpu" WHERE "datacenter" =~ /^$datacenter$/ AND "host" =~ /^$host$/
```

This query returns the CPU metrics for the selected host.

Depending on what variable options the user selects, you could get queries like:

```bash
SHOW TAG VALUES WITH KEY = "cpu" WHERE "datacenter" =~ /^America/ AND "host" =~ /^server2/
SHOW TAG VALUES WITH KEY = "cpu" WHERE "datacenter" =~ /^Africa/ AND "host" =~ /^server/7/
SHOW TAG VALUES WITH KEY = "cpu" WHERE "datacenter" =~ /^Europe/ AND "host" =~ /^server3+server4/
```

### Best practices for chained variables

The following practices make your dashboards and variables easier to use.

#### New linked variables creation

- Chaining variables create parent/child dependencies. You can envision them as a ladder or a tree.
- The easiest way to create a new chained variable is to copy the variable that you want to base the new one on. In the variable list, click the **Duplicate variable** icon to the right of the variable entry to create a copy. You can then add on to the query for the parent variable.
- New variables created this way appear at the bottom of the list. You might need to drag it to a different position in the list to get it into a logical order.

#### Variable order

You can change the orders of variables in the dashboard variable list by clicking the up and down arrows on the right side of each entry. Grafana lists variable dropdowns left to right according to this list, with the variable at the top on the far left.

- List variables that don't have dependencies at the top, before their child variables.
- Each variable should follow the one it's dependent on.
- Remember there is no indication in the UI of which variables have dependency relationships. List the variables in a logical order to make it easier for others to use.

#### Complexity consideration

The more layers of dependency you have in variables, the longer it takes to update dashboards after you change variables.

For example, if you have a series of four linked variables (country, region, server, metric) and you change a root variable value (country), then Grafana must run queries for all the dependent variables before it updates the visualizations in the dashboard.

<!-- vale Grafana.WordList = NO -->

## Filter variables with regular expressions {#filter-variables-with-regex}

Using the **Regex** query option, you filter the list of options returned by the variable query or modify the options returned.

This section shows how to use a regular expression to filter or modify values in the variable drop-down list.

Using the **Regex** query option, you filter the list of options returned by the Variable query or modify the options returned. For more information, refer to the Mozilla guide on [Regular expressions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions).

Examples of filtering on the following list of options:

```text
backend_01
backend_02
backend_03
backend_04
```

### Filter to return only the options that end with `01` or `02`

**Regex**:

```regex
/(01|02)$/
```

Result:

```text
backend_01
backend_02
```

### Filter and modify the options using a regular expression to capture group to return part of the text

**Regex**:

```regex
/.*(01|02)/
```

Result:

```text
01
02
```

### Filter and modify: Prometheus Example

List of options:

```text
up{instance="demo.robustperception.io:9090",job="prometheus"} 1 1521630638000
up{instance="demo.robustperception.io:9093",job="alertmanager"} 1 1521630638000
up{instance="demo.robustperception.io:9100",job="node"} 1 1521630638000
```

**Regex**:

```regex
/.*instance="([^"]*).*/
```

Result:

```text
demo.robustperception.io:9090
demo.robustperception.io:9093
demo.robustperception.io:9100
```

### Filter and modify using named text and value capture groups

Using named capture groups, you can capture separate 'text' and 'value' parts from the options returned by the variable query. This allows the variable drop-down list to contain a friendly name for each value that can be selected.

For example, when querying the `node_hwmon_chip_names` Prometheus metric, the `chip_name` is a lot friendlier than the `chip` value. So the following variable query result:

```text
node_hwmon_chip_names{chip="0000:d7:00_0_0000:d8:00_0",chip_name="enp216s0f0np0"} 1
node_hwmon_chip_names{chip="0000:d7:00_0_0000:d8:00_1",chip_name="enp216s0f0np1"} 1
node_hwmon_chip_names{chip="0000:d7:00_0_0000:d8:00_2",chip_name="enp216s0f0np2"} 1
node_hwmon_chip_names{chip="0000:d7:00_0_0000:d8:00_3",chip_name="enp216s0f0np3"} 1
```

Passed through the following regular expression:

```regex
/chip_name="(?<text>[^"]+)|chip="(?<value>[^"]+)/g
```

Would produce the following drop-down list:

```text
Display Name          Value
------------          -------------------------
enp216s0f0np0         0000:d7:00_0_0000:d8:00_0
enp216s0f0np1         0000:d7:00_0_0000:d8:00_1
enp216s0f0np2         0000:d7:00_0_0000:d8:00_2
enp216s0f0np3         0000:d7:00_0_0000:d8:00_3
```

{{< admonition type="note" >}}
Only `text` and `value` capture group names are supported.
{{< /admonition >}}

# Built-in variables

Grafana has built-in variables that you can use in expressions in the query editor.
These variables are globally-scoped so you can use them in any dashboard.
This page lists them in alphabetical order and defines them.
Most variables are useful in queries, dashboard links, panel links, and data links&mdash;but some, like `$__url_time_range`, are only intended for use in links, not query editors.

## `$__dashboard`

This variable is the name of the current dashboard.

## `$__from` and `$__to`

Grafana has built-in time range variables: `$__from` and `$__to`.
They're always interpolated as epoch milliseconds by default, but you can control date formatting.

<!-- prettier-ignore-start -->

| Syntax                   | Example result           | Description                                 |
| ------------------------ | ------------------------ | ------------------------------------------- |
| `${__from}`              | 1594671549254            | Unix millisecond epoch                      |
| `${__from:date}`         | 2020-07-13T20:19:09.254Z | No arguments, defaults to ISO 8601/RFC 3339 |
| `${__from:date:iso}`     | 2020-07-13T20:19:09.254Z | ISO 8601/RFC 3339                           |
| `${__from:date:seconds}` | 1594671549               | Unix seconds epoch                          |
| `${__from:date:YYYY-MM}` | 2020-07                  | Any custom [date format](https://momentjs.com/docs/#/displaying/) that doesn't include the `:` character. Uses browser time. Use `:date` or `:date:iso` for UTC                                                                                 |

<!-- prettier-ignore-end -->

The syntax above also works with `${__to}`.

You can use this variable in URLs, as well. For example, you can send a user to a dashboard that shows a time range from six hours ago until the current time: `https://play.grafana.org/d/000000016/time-series-graphs?orgId=1&from=now-6h&to=now`

## `$__interval`

You can use the `$__interval` variable as a parameter to group by time for InfluxDB, MySQL, Postgres, and MSSQL; as a Date histogram interval for Elasticsearch; or as a _summarize_ function parameter for Graphite.

Grafana automatically calculates an interval that you can use to group by time in queries. When there are more data points than a graph can show, grouping by a larger interval makes queries more efficient. Grouping by 1 day is more efficient than grouping by 10s when you look at 3 months of data. The graph looks the same, and the query is faster. Grafana calculates `$__interval` from the time range and the graph width in pixels.

Approximate Calculation: `(to - from) / resolution`

For example, when the time range is 1 hour and the graph is full screen, then the interval might be calculated to `2m` - points are grouped in 2 minute intervals. If the time range is 6 months and the graph is full screen, then the interval might be `1d` (1 day) - points are grouped by day.

In the InfluxDB data source, the legacy variable `$interval` is the same variable. You should use `$__interval` instead.

The InfluxDB and Elasticsearch data sources have `Group by time interval` fields that are used to hard code the interval or set the minimum limit for the `$__interval` variable by using the `>` syntax, for example `>10m`.

## `$__interval_ms`

This variable is the `$__interval` variable in milliseconds, not a time interval formatted string. For example, if the `$__interval` is `20m` then the `$__interval_ms` is `1200000`.

## `$__name`

This variable is only available in the **Singlestat** panel and you can use it in the prefix or suffix fields on the Options tab. Grafana replaces the variable with the series name or alias.

{{< admonition type="note" >}}
The **Singlestat** panel is no longer available from Grafana 8.0.
{{< /admonition >}}

## `$__namespace`

This variable is the Kubernetes namespace that Grafana uses internally to identify the current organization. In Grafana Cloud, the value might be `stacks-123`. In self-managed Grafana, it's `default` for the default organization and `org-<id>` for other organizations. Use the variable in API URLs that require an organization namespace, such as `/apis/<group>/<version>/namespaces/${__namespace}/<resource>`.

{{< admonition type="note" >}}
This variable is only interpolated in dashboards. In Explore, alert rule query editors, and other views outside a dashboard, it's left as written.
{{< /admonition >}}

## `$__org`

This variable is the ID of the current organization.
`${__org.name}` is the name of the current organization.

## `$__user`

`${__user.id}` is the ID of the current user.
`${__user.login}` is the login handle of the current user.
`${__user.email}` is the email for the current user.

## `$__range`

Currently only supported for Prometheus and Loki data sources. This variable represents the range for the current dashboard. It's calculated by `to - from`. It has a millisecond and a second representation called `$__range_ms` and `$__range_s`.

## `$__rate_interval`

Currently only supported for Prometheus data sources. Use the `$__rate_interval` variable in the rate function. Refer to [Prometheus query variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/template-variables/#use-__rate_interval) for details.

## `$__rate_interval_ms`

This variable is the `$__rate_interval` variable in milliseconds, not a time-interval-formatted string. For example, if the `$__rate_interval` is `20m` then the `$__rate_interval_ms` is `1200000`.

## `$__theme`

The `$__theme` variable returns a value from the current Grafana theme, like a color, font, or border radius.
Use it to style content so it matches the rest of Grafana in both light and dark themes, without fixed color values.

Add the path to a theme value after the variable name, using the `${__theme.<PATH>}` syntax.
The following table lists common paths and their values in the dark theme:

<!-- prettier-ignore-start -->

| Syntax                                           | Example result                    |
| ------------------------------------------------ | --------------------------------- |
| `${__theme.colors.mode}`                         | `dark`                            |
| `${__theme.colors.text.primary}`                 | `rgb(204, 204, 220)`              |
| `${__theme.colors.text.secondary}`               | `rgba(204, 204, 220, 0.65)`       |
| `${__theme.colors.background.secondary}`         | `#22252b`                         |
| `${__theme.colors.border.weak}`                  | `rgba(204, 204, 220, 0.12)`       |
| `${__theme.colors.success.text}`                 | `#6ccf8e`                         |
| `${__theme.typography.fontFamilyMonospace}`      | `'Roboto Mono', monospace`        |
| `${__theme.typography.size.sm}`                  | `12px`                            |
| `${__theme.typography.fontWeightMedium}`         | `500`                             |
| `${__theme.shape.radius.default}`                | `6px`                             |
| `${__theme.shadows.z1}`                          | `0px 1px 2px rgba(1, 4, 9, 0.75)` |
| `${__theme.visualization.hues.0.shades.2.color}` | `#F2495C`                         |

<!-- prettier-ignore-end -->

The semantic colors `primary`, `secondary`, `info`, `success`, `warning`, and `error` each have the shades `main`, `shade`, `text`, `border`, `transparent`, `borderTransparent`, and `contrastText`. For example, `${__theme.colors.warning.border}`.

The following HTML in a [text visualization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/visualizations/text/) shows a status badge in the theme's success colors:

```html
<span
  style="padding: 2px 8px; color: ${__theme.colors.success.text}; border: 1px solid ${__theme.colors.success.border}; border-radius: ${__theme.shape.radius.pill}"
>
  Healthy
</span>
```

Keep the following in mind when you use `$__theme`:

- **Values follow the current theme:** The variable returns values from the theme that's active when Grafana interpolates it. For example, the text visualization updates as soon as a user switches between light and dark themes. Other places pick up the new values the next time they're interpolated, like when the dashboard refreshes.
- **Only single values resolve:** A path must end at a single value, like a color or a size. A path to a group of values, like `${__theme.colors}`, a missing path, like `${__theme.colors.nope}`, or `${__theme}` without a path remains literal text, so you can spot a typo.
- **Units are part of the value:** Most sizes include their unit, like `12px`. A few return only a number, like `${__theme.typography.fontSize}`, which returns `14`. You can add the unit yourself: `${__theme.typography.fontSize}px`.
- **Use `typography.size` for pixel font sizes:** The theme's text styles, like headings and body text, set their font sizes in `rem`, a unit relative to the page's base font size. For a font size in pixels, use `${__theme.typography.size.sm}`, or `xs`, `md`, or `lg`.
- **Use hues for visualization colors:** Use `${__theme.visualization.hues.<HUE>.shades.<SHADE>.color}`, where `<HUE>` is `0` to `5` (red, orange, yellow, green, blue, purple) and `<SHADE>` is `0` (lightest) to `4` (darkest). For example, `${__theme.visualization.hues.2.shades.3.color}` returns `#F2CC0C`.
- **Spacing isn't available:** Use `${__theme.spacing.gridSize}`, which always returns `8`, and multiply it. For example, `calc(${__theme.spacing.gridSize}px * 2)` returns `16px`.

## `$timeFilter` or `$__timeFilter`

The `$timeFilter` variable returns the active time range selection as an expression. For example, the time range interval `Last 7 days` expression is `time > now() - 7d`.

It's used in several places, including:

- The `WHERE` clause for the InfluxDB data source. Grafana adds it automatically to InfluxDB queries when in Query Editor mode. You can add it manually in Text Editor mode: `WHERE $timeFilter`.
- Log Analytics queries in the Azure Monitor data source.
- SQL queries in MySQL, Postgres, and MSSQL.
- The `$__timeFilter` variable is used in the MySQL data source.

## `$__timezone`

The `$__timezone` variable returns the active time zone selection, either `utc` or an entry from the IANA time zone database, such as `America/New_York`.

If the active time zone selection is _Browser Time_, Grafana tries to determine your browser time zone.

## `$__url_time_range`

The `$__url_time_range` variable returns the current dashboard time range as URL query parameters.
It's intended for use in data links and panel links, not in query editors.

The variable expands to a string like `from=1607687293000&to=1607687293100`, where the values are Unix millisecond timestamps matching the current time range selection.

You must include the `?` or `&` separator yourself when constructing URLs:

<!-- prettier-ignore-start -->

| Usage | Example result |
| ----- | -------------- |
| `https://example.com/d/abc?${__url_time_range}`         | `https://example.com/d/abc?from=1594671549254&to=1594672349254`         |
| `https://example.com/d/abc?orgId=1&${__url_time_range}` | `https://example.com/d/abc?orgId=1&from=1594671549254&to=1594672349254` |

<!-- prettier-ignore-end -->

To link to another dashboard while preserving the current time range, follow this pattern:

```text
https://your-grafana/d/other-dashboard?${__url_time_range}
```

To also pass a variable value from the current dashboard, follow this pattern:

```text
https://your-grafana/d/other-dashboard?{__url_time_range}&var-host= {host}
```

{{< admonition type="note" >}}
`$__url_time_range` always uses Unix millisecond epoch timestamps.
To include only the start or end of the time range with specific formatting, use [`$__from` and `$__to` variables](#__from-and-__to) instead.
{{< /admonition >}}

# Cross-dashboard variables

{{< docs/public-preview product="Cross-dashboard variables" featureFlag="`grafana.dashboardGlobalVariables`" >}}

Unlike variables that are specific to the dashboard in which you build them, you can create _cross-dashboard variables_ for use across multiple dashboards.
These variables help ensure consistent naming across your organization and eliminate the need to re-create the same variables every time you add a new dashboard.

You can scope cross-dashboard variables in two ways:

- **Globally**: Available to all dashboards in the organization.
- **By folder**: Available only to the dashboards in the same folder as the variable.

The **Dashboards > Variables** page lists cross-dashboard variables, showing which ones are available globally or by folder.
In the following image, there are variables in "Core metrics" and "Grafana Cloud" folders and then a number of variables that aren't in any folders; these ones are global:

![Cross-dashboard variables page](/media/docs/grafana/dashboards/screenshot-cross-dash-variables-v13.3.png)

When you add cross-dashboard variables to a dashboard, only the ones that share the same scope as your dashboard are available for you to select.

For example, in the preceding image, there are two variables in the "Core metrics" folder.
The following image shows a dashboard in the "Core metrics" folder.
As a result, the variables in the "Core metrics" folder are available to that dashboard, as well as all the global variables:

![Available cross-dashboard variables in dashboard sidebar](/media/docs/grafana/dashboards/screenshot-cross-dash-variables-sidebar-v13.3.png)

Variables in other folders aren't available to that dashboard.

## Create cross-dashboard variables

<!-- prettier-ignore-start -->

To create cross-dashboard variables, follow these steps:

1. Navigate to **Dashboards > Variables**.
1. Click **+ New variable**.
1. Set the following general variable properties:

   | Option        | Description                                                                                                                            |
   | ------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
   | Folder        | Set the availability of the variable. To allow the variable to be used in all of the organization's dashboards, select **Dashboards**. |
   | Variable type | Choose a variable type.                                                                                                                |
   | Name          | Enter a name for the variable.                                                                                                         |
   | Label         | (Optional) Enter the display name for the variable drop-down list. If you leave this empty, then the variable name is used.            |
   | Description   | <p>(Optional) Enter a description of the variable. When you add a description, an info icon appears next to the variable name on the dashboard. Hover your cursor over the icon to display the description.</p><p>Descriptions support links. You can use Markdown-style links like `[link text](https://example.com)`, or paste bare URLs like `https://example.com`. Only `http` and `https` URLs render as clickable links—other protocols display as plain text.</p>  |
   | Display       | Choose where the variable displays:<ul><li>**Above dashboard**: The variable drop-down list displays above the dashboard with the variable **Name** or **Label** value. This is the default.</li><li>**Above dashboard, label hidden**: The variable drop-down list displays above the dashboard, but without showing the name of the variable.</li><li>**Controls menu**: The variable displays in the dashboard controls menu instead of above the dashboard. The dashboard controls menu appears as a button in the dashboard toolbar.</li><li>**Hidden**: No variable drop-down list displays on the dashboard.</li></ul>           |

1. Click one of the following links to complete the steps for adding your selected variable type:
   - [Query](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-query-variable)
   - [Custom](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-custom-variable)
   - [Textbox](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-text-box-variable)
   - [Constant](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-constant-variable)
   - [Data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-data-source-variable)
   - [Interval](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-an-interval-variable)
   - [Switch](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/#add-a-switch-variable)

<!-- prettier-ignore-end -->

## Add cross-dashboard variables to a dashboard

You can choose which cross-dashboard variables to add to dashboards you create.
Only variables that share the same scope as your dashboard are available for you to select.

To add cross-dashboard variables to a dashboard, follow these steps:

1. Navigate to the dashboard you want to update.
1. Click **Edit**.
1. Click the **Add new element** icon in the sidebar and select **Variable**.

   You can also click the **Add** (plus sign) icon at the top of the dashboard and select **Variable**.

1. Select the **Global or folder variable** variable type.
1. Select the variables you want to add to the dashboard:

   ![Cross-dashboard variable selected and added above dashboard](/media/docs/grafana/dashboards/screenshot-cross-dash-variable-selected-v13.3.png)

   To add every global variable available, select **All global** and to add every folder variable available, select **All folder**.

1. Update variable values as needed.
1. Click **Save**.
1. Enter an optional description of your changes and click **Save**.
1. Click **Exit edit**.

After you've added cross-dashboard variables to a dashboard, you can view its settings or remove it by way of the dashboard sidebar.
To edit the variable, go to **Dashboards > Variables**.

# Manage and inspect variables

In the **Variables** section of the sidebar, you can [add](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/add-template-variables/) variables and manage existing variables. You can also inspect variables to identify any dependencies between them.

{{< figure src="/media/docs/grafana/dashboards/screenshot-manage-variables-v13.2.png" max-width="400px" alt="Variables section of the dashboard options in the sidebar" >}}

## Manage variables

You can take the following actions in the **Variables** section of the sidebar:

- **Edit**: Click **Select** on the control to open it in the sidebar so you can make updates. Then you can access the following options:
  - **Duplicate**: Duplicate a variable by clicking the clone icon the sidebar header. This creates a copy of the variable with the name of the original variable prefixed with `copy` and the number of the copy; for example, "copy1".
  - **Delete**: Delete a variable by clicking the trash icon in sidebar header.
- **Reorder**: Drag and drop controls to reorder them.
- **Change display**: Drag and drop controls between sub-sections **Above dashboard**, **Controls menu**, and **Hidden** to update the control display option. Note that links can't be hidden.

## Inspect variables

In addition to managing variables, the **Variables** section lets you quickly identify whether variables have any dependencies.
To check, click **Show dependencies** at the bottom of the list, which opens the dependencies diagram:

{{< figure src="/media/docs/grafana/dashboards/screenshot-variable-dependencies-v13.2.png" max-width="600px" alt="Dependency map showing relationships between dashboard variables" >}}

# Variable syntax

Panel titles and metric queries can refer to variables using two different syntaxes:

- `$varname`
  This syntax is easy to read, but it doesn't allow you to use a variable in the middle of a word.
  **Example:** apps.frontend.$server.requests.count
- `${var_name}` Use this syntax when you want to interpolate a variable in the middle of an expression.
- `${var_name:<format>}` This format gives you more control over how Grafana interpolates values. Refer to [Advanced variable format options](#advanced-variable-format-options) for more detail on all the formatting types.
- `[[varname]]` Do not use. Deprecated syntax, that will be removed in a future release.

Before queries are sent to your data source, the query is _interpolated_, meaning the variable is replaced with its current value.
During interpolation, the variable value might be _escaped_ to conform to the syntax of the query language and where it's used.
For example, variables in InfluxDB or Prometheus regular expression queries are automatically escaped.
Read the data source specific documentation topic for details on value escaping during interpolation.

For advanced syntax to override data source default formatting, refer to [Advanced variable format options](#advanced-variable-format-options).

## Advanced variable format options

The formatting of the variable interpolation depends on the data source, but there are some situations where you might want to change the default formatting.

For example, the default for the MySql data source is to join multiple values as comma-separated with quotes: `'server01','server02'`.
In some cases, you might want to have a comma-separated string without quotes: `server01,server02`.
You can make that happen with advanced variable formatting options listed below.

### General syntax

Syntax: `${var_name:option}`

Test the formatting options on the [Grafana Play site](https://play.grafana.org/d/cJtIfcWiz/template-variable-formatting-options?orgId=1).

If any invalid formatting option is specified, then `glob` is the default/fallback option.

An alternative syntax (that might be deprecated in the future) is `[[var_name:option]]`.

### CSV

Formats variables with multiple values as a comma-separated string.

```bash
servers = ['test1', 'test2']
String to interpolate: '${servers:csv}'
Interpolation result: 'test1,test2'
```

### Distributed - OpenTSDB

Formats variables with multiple values in custom format for OpenTSDB.

```bash
servers = ['test1', 'test2']
String to interpolate: '${servers:distributed}'
Interpolation result: 'test1,servers=test2'
```

### `doublequote`

Formats single- and multi-valued variables into a comma-separated string, escapes `"` in each value by `\"` and quotes each value with `"`.

```bash
servers = ['test1', 'test2']
String to interpolate: '${servers:doublequote}'
Interpolation result: '"test1","test2"'
```

### Glob - Graphite

Formats variables with multiple values into a glob (for Graphite queries).

```bash
servers = ['test1', 'test2']
String to interpolate: '${servers:glob}'
Interpolation result: '{test1,test2}'
```

### Join

Formats multi-valued variables with a custom delimiter. If no delimiter argument is supplied, they're combined with `,`.

```bash
servers = ["test1", "test2"]
String to interpolate: '${servers:join:&}'
Interpolation result: "test1&test2"
```

### JSON

Formats variables with multiple values as a comma-separated string.

```bash
servers = ['test1', 'test2']
String to interpolate: '${servers:json}'
Interpolation result: '["test1", "test2"]'
```

### Lucene - Elasticsearch

Formats variables with multiple values in Lucene format for Elasticsearch.

```bash
servers = ['test1', 'test2']
String to interpolate: '${servers:lucene}'
Interpolation result: '("test1" OR "test2")'
```

### `percentencode`

Formats single and multi valued variables for use in URL parameters.

```bash
servers = ['foo()bar BAZ', 'test2']
String to interpolate: '${servers:percentencode}'
Interpolation result: 'foo%28%29bar%20BAZ%2Ctest2'
```

### Pipe

Formats variables with multiple values into a pipe-separated string.

```bash
servers = ['test1.', 'test2']
String to interpolate: '${servers:pipe}'
Interpolation result: 'test1.|test2'
```

### Query parameters

Formats single- and multi-valued variables into their query parameter representation. Example: `var-foo=value1&var-foo=value2`

```bash
servers = ["test1", "test2"]
String to interpolate: '${servers:queryparam}'
Interpolation result: "var-servers=test1&var-servers=test2"
```

Use the `customqueryparam` formatter to customize how the query parameters are formatted. It accepts two optional arguments to specify the parameter name, and a value prefix.

```bash
servers = ["test1", "test2"]
String to interpolate: '${servers:customqueryparam:v-servers:x-}'
Interpolation result: "v-servers=x-test1&v-servers=x-test2"
```

### Raw

Doesn't apply any data source-specific formatting to the variable.

For example, in this case, there's a dashboard with a Prometheus data source and a multi-value variable.
Grafana typically converts the variable values as follows to accommodate Prometheus:

```bash
servers = ['test1.', 'test2']
String to interpolate: '${servers}'
Interpolation result: '(test1 | test2)'
```

Using the raw format, the values are returned without that formatting:

```bash
servers = ['test1.', 'test2']
String to interpolate: '${servers:raw}'
Interpolation result: 'test1,test2'
```

### Regex

Formats variables with multiple values into a regular expression string.

```bash
servers = ['test1.', 'test2']
String to interpolate: '${servers:regex}'
Interpolation result: '(test1\.|test2)'
```

### `singlequote`

Formats single- and multi-valued variables into a comma-separated string, escapes `'` in each value by `\'` and quotes each value with `'`.

```bash
servers = ['test1', 'test2']
String to interpolate: '${servers:singlequote}'
Interpolation result: "'test1','test2'"
```

### `sqlstring`

Formats single- and multi-valued variables into a comma-separated string, escapes `'` in each value by `''` and quotes each value with `'`.

```bash
servers = ["test'1", "test2"]
String to interpolate: '${servers:sqlstring}'
Interpolation result: "'test''1','test2'"
```

### Text

Formats single- and multi-valued variables into their text representation. For a single variable it just returns the text representation. For multi-valued variables it returns the text representation combined with `+`.

```bash
servers = ["test1", "test2"]
String to interpolate: '${servers:text}'
Interpolation result: "test1 + test2"
```

# Correlations Editor in Explore

{{< admonition type="note" >}}
The Explore editor is available in 10.1 and later versions. In the editor, transformations is available in Grafana 10.3 and later versions.
{{< /admonition >}}

Correlations allow users to build a link between any two data sources. For more information about correlations in general, please see the [correlations](/docs/grafana/<GRAFANA_VERSION>/administration/correlations/) topic in the administration page.

## Create a correlation

1. In Grafana, navigate to the Explore page.
1. Select a data source that you would like to be [the source data source](/docs/grafana/<GRAFANA_VERSION>/administration/correlations/correlation-configuration/#source-data-source-and-result-field) for a new correlation.
1. Run a query producing data in [a supported visualization](/docs/grafana/<GRAFANA_VERSION>/administration/correlations/#correlations).
1. Click **+ Add** in the top toolbar and select **Add correlation** (you can also select **Correlations Editor** from the [Command Palette](/docs/grafana/<GRAFANA_VERSION>/search/#command-palette)).
1. Explore is now in Correlations Editor mode indicated by a blue border and top bar. You can exit Correlations Editor by clicking **Exit** in the top bar.
1. You can now create the following new correlations for the visualization with links that are attached to the data that you can use to build a new query:
   - Logs: links are displayed next to field values inside log details for each log row
   - Table: every table cell is a link
1. Click on a link to add a new correlation.
   Links are associated with a field that is used as a [result field of a correlation](/docs/grafana/<GRAFANA_VERSION>/administration/correlations/correlation-configuration/).
1. In the split view that opens, use the right pane to set up [the target query source of the correlation](/docs/grafana/<GRAFANA_VERSION>/administration/correlations/correlation-configuration/#target-query).
1. Build a target query using [variables syntax](/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/variable-syntax/) with variables from the list provided at the top of the pane. The list contains sample values from the selected data row.
1. Provide a label and description (optional).
   A label will be used as the name of the link inside the visualization and can contain variables.
1. Provide transformations (optional; see below for details).
1. Click **Save** in the top toolbar to save the correlation and exit Correlations Editor mode.
   The link used to create the correlation is replaced with a data link in each row. When the link is clicked, the query you defined will run in another pane, with the variables replaced dynamically with the values from the selected row.

## Transformations

Transformations allow you to extract values that exist in a field with other data. For example, using a transformation, you can extract one portion of a log line to use in a correlation. For more details on transformations in correlations, see [Correlations](/docs/grafana/<GRAFANA_VERSION>/explore/correlations-editor-in-explore/#transformations).

After clicking one of the generated links in the editor mode, you can add transformations by clicking **Add transformation** in the Transformations dropdown menu.

You can use a transformation in your correlation with the following steps:

1. Select a field to apply the transformation to.
   Select the portion of the field that you want to use for the transformation. For example, a log line.
   Once selected, the value of this field will be used to assist you in building the transformation.
1. Select the type of the transformation.
   See [correlations](/docs/grafana/<GRAFANA_VERSION>/explore/correlations-editor-in-explore/#transformations) for the options and relevant settings.
1. Based on your selection, you might see one or more variables populate, or you might need to provide more specifications in options that are displayed.
1. Select **Add transformation to correlation** to add the specified variables to the list of available variables.

### Notes for regular expressions

For regular expressions in this dialog box, the `mapValue` referred to in other documentation is called `Variable Name` here. Grafana highlights any text that matches the expression in the field value. Use regular expression capture groups to select what portion of the match should be extracted. When a valid regular expression is provided, the variable and the value of that variable appear below the `Variable Name` field.

## Correlations examples

The following examples show how to create correlations using the Correlations Editor in Explore. If you'd like to follow these examples, make sure to set up a [test data source](/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/#testdata-data-source).

### Create a text to graph correlation

This example shows how to create a correlation using Correlations Editor in Explore.

Correlations allow you to use results of one query to run a new query in any data source. In this example, you will run a query that renders tabular data. The data will be used to run a different query that yields a graph result.

To follow this example, make sure you have set up [a test data source](/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/#testdata-data-source).

1. In Grafana, navigate to **Explore**.
1. Select the **test data source** from the dropdown menu at the top left of the page.
1. Click **+ Add** in the dropdown menu to the right and select **Add correlation**.
1. Explore is now in Correlations Editor mode, indicated by a blue border.
1. Select the following scenario from the scenario dropdown menu: **CSV File**.
1. Select the file, **population_by_state.csv**.
   Each cell is a link that you can click on to begin creating a new correlation.

   {{< figure src="/static/img/docs/correlations/screenshot-correlations-editor-source-10.2.png" max-width="600px" caption="Selecting the source of a correlation" >}}

1. Click on any cell in the `State` column to create a new correlation that attaches a data link to that entry. For example, select "California".
1. In the split view, select the same data source you selected in the left pane.
   The helper above the query editor contains all available variables you can use the target query. Variables contain all data fields (table columns) from the selected row.
1. In the **Scenario** menu, select **CSV Metric Values**.
   The `String Input` field in the Query editor provides variables with population values for each year: `${1980},${2000},${2020}`. This will generate a graph using variable values.
1. In the Query Editor **Alias** field, enter "${State}".

   {{< figure src="/static/img/docs/correlations/screenshot-correlations-editor-target-10.2.png" max-width="600px" caption="Setting up the target of a correlation" >}}

   Run a query to see that it produces a graph using sample values from the variables.

1. Click **Save** to save the correlation and exit the Correlations Editor.

   After the correlation is saved, Explore will rerun the query in the left pane. By clicking a state name, the query on the right is rerun with values from the row being inserted into the CSV, thus changing the graph. The query is rerun with updated values every time you click on a state name.

   {{< figure src="/static/img/docs/correlations/screenshot-correlations-example-link-10.2.png" max-width="600px" caption="Result of clicking on a data link" >}}

You can apply the same steps to any data source. Correlations allow you to create links in visualizations to run dynamic queries based on selected data. In this example we used data returned by a query to build a new query generating different visualization using the same data source. However, you can create correlations between any data sources to create custom exploration flows.

### Create a logs to table correlation

In this example, you will create a correlation to demonstrate how to use transformations to extract values from the log line and another field.

To follow this example, make sure you have set up [a test data source](/docs/grafana/<GRAFANA_VERSION>/datasources/testdata/#testdata-data-source).

1. In Grafana, navigate to **Explore**.
1. Select the **test data source** from the dropdown menu at the top left of the page.
1. Click **+ Add** in the dropdown menu to the right and select **Add correlation**.
1. Explore is now in Correlations Editor mode, indicated by a blue border.
1. In the **Scenario** menu, select **Logs**.
1. Expand a log line to see the correlation links. Select `Correlate with hostname`.
1. Explore opens in split view. Select the same data source you selected in the left pane.
   The helper above the query editor contains all available variables you can use the target query.
1. Expand the transformations section, and click **Add transformation**.
1. In the **Field** dropdown menu, select **message**.
   The log line shows up as example data.
1. Under **Type**, select **Logfmt**.
   This populates the list of variables.
1. Click **Add transformation to correlation**.
1. Click **Add transformation** again and under **Field**, select **hostname**.
1. Under **Type**, select **Regular expression**.
1. Under **Expression**, enter the following:
   `-([0-9]\*)`
   This selects any numbers to the right of the dash.
1. Under **Variable Name**, enter the following:
   hostNumber
   This populates the list of variables.
1. Click **Add transformation to correlation** to add it to the other variables.
1. In the data source editor, open the **Scenario** dropdown menu and select **CSV Content**.
1. In the text box below, provide the following and save the correlation:

   ```csv
   time,msg,hostNumber,status
   ${time},${msg},${hostNumber},${status}
   ```

   This closes the split view and reruns the left query. Expand any log line to see the correlation button. Clicking the correlation button opens the split view with the `time` (a field), `msg` (extracted with `logfmt` from the log line), host number (extracted with `regex` from the `hostname`) and the `status` (extracted with `logfmt` from the log line).

# Query inspector in Explore

The Query inspector in Grafana Explore gives you detailed statistics regarding your query, which helps you understand and troubleshoot issues with your queries. Query inspector also lets you inspect raw data, export data to a comma-separated values (CSV) file, export log results in TXT format, and view query requests.

## Query inspector UI

To open Query inspector:

1. Go to the Explore page.
1. Run the query you would like to inspect.
1. Click **Query inspector**.

The Query inspector pane opens on the bottom of the Explore page, where you see the following tabs:

- **Stats tab -** Shows statistics regarding the query, including the amount of time it takes to run the query, data processing time and the amount of data returned.
- **Query tab -** Provides raw request and response data and time when Grafana queries the data source.
- **JSON tab -** Allows you to view and copy the JSON data and the JSON data frame structure.
- **Data tab -** Shows the raw data returned by the query. You can download the information to a CSV file.
- **Error tab -** Shows any errors. _Only visible if the query returns an error._

## Query inspector Stats tab

You can inspect query performance in the **Stats tab**, which displays statistics that tell you how long your query takes, how many queries you send, the number of rows returned and trace IDs. This information can help you troubleshoot your queries, especially if any of the numbers are unexpectedly high or low.

1. Open the inspector.
1. Click the **Stats tab**.

Statistics display in read-only format.

## Query inspector Query tab

View raw request and response in the Query tab.

1. Open the Query inspector and click the **Query tab**.
1. Click **Refresh**.

Grafana sends the query to the server and displays the result. You can drill down on specific portions of the query, expand or collapse all of it. Click **Copy to clipboard** to copy the data to use in other applications.

## Query inspector JSON tab

View data results as JSON and as data frame JSON models in the **JSON tab**.

1. Open the Query inspector and click the **JSON tab**.
1. Choose one of the following options from the **Select source** dropdown menu:
   - **Panel data -** Displays a JSON object representing the data retrieved by the visualization from Explore.
   - **DataFrame JSON (from query) -** Displays the raw data result set without transformations and field configuration applied.

## Query inspector Data tab

View, inspect and download raw query results in the **Data tab**.

1. Open the Query inspector and click the **Data** tab.
1. Click **Data options** to view options under **Show data frame**.
1. Select a data results set from the dropdown menu.
1. For multiple queries or for queries multiple nodes, you can select **Series joined by time** from the dropdown to view the raw data from all of your queries at once, one result set per column. You can click any column heading to sort the data.
1. Toggle **Formatted data** to match the format in the panel.
1. Toggle **Download for Excel** to download a CSV file specifically formatted for Excel.
1. To download the results to a CSV file click **Download CSV** in the upper right of the Query inspector pane.

### Download log results as TXT

Based on the type of data source (Loki, for example), or when logs are present in the results set, Grafana generates a TXT file of log raw data results in your default browser download location. You can open it in the viewer of your choice.

1. Click **Query inspector**.
1. Click the **Data tab** to view log query results.
1. Click **Download logs**.

### Download trace results

Based on the data source type (Tempo, for example), Grafana generates a JSON file for trace results in one of these supported formats: Jaeger, Zipkin, or OTLP.

1. Click **Query inspector**.
1. Click the **Data tab** to view traces results.
1. Click **Download traces**.

# Get started with Explore

{{< shared id="explore-overview" >}}

Explore is your gateway for querying, analyzing, and aggregating data in Grafana. It allows you to visually explore and iterate until you develop a working query or set of queries for building visualizations and conducting data analysis. If your data source supports graph and table data, there's no need to create a dashboard, as Explore can display the results in both formats. This facilitates quick, detailed, real-time data analysis.

With Explore you can:

- Create visualizations to integrate into your dashboards.
- Create queries using mixed data sources.
- Create multiple queries within a single interface.
- Understand the shape of your data across various data sources.
- Perform real time data exploration and analysis.

Key features include:

- Query editor, based on specific data source, to create and iterate queries.
- [Query history](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/query-management/) to track and maintain your queries.
- [Query inspector](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/explore-inspector/) to help troubleshoot query performance.

{{< /shared >}}

Watch the following video to get started using Explore:

{{< youtube id="1q3YzX2DDM4" >}}

To try Explore with sample data, open it on Grafana Play:

{{< docs/play title="Explore" url="https://play.grafana.org/explore" >}}

## Before you begin

In order to access Explore, you must have either the `editor` or `administrator` basic role or the `data sources explore` role. Refer to [Role and permissions](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/) for more information on what each role can access.

Refer to [Role-based access control (RBAC)](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/) in Grafana Enterprise to understand how you can manage Explore with role-based permissions.

## Explore elements

Explore consists of a toolbar, outline, query editor, the ability to add multiple queries, a query history and a query inspector.

### Content outline

The content outline is a side navigation bar that keeps track of the queries and visualizations you create in Explore.
The outline helps you quickly navigate between queries and visualizations.
It works in a split view, with a separate outline generated for each pane.

To open the content outline, click the Outline button in the top left corner of the Explore screen.

You can then click on any panel icon in the content outline to navigate to that panel.

### Toolbar

The toolbar provides quick access to frequently used tools and settings.

- **Data source picker** - Select a data source from the dropdown menu, or use absolute time.
- **Split** - Click to compare visualizations side by side. Refer to [Split and compare](#split-and-compare) for additional detail.
- **Add** - Click to add your exploration to a dashboard. You can also use this to declare an incident,create a forecast, detect outliers and to run an investigation.
- **Time picker** - Select a time range form the time picker. You can also enter an absolute time range. Refer to [Time picker](#time-picker) for more information.
- **Run query** - Click to run your query.

#### Time picker

Use the time picker to select a time range for your query. The default is **last hour**. You can select a different option from the dropdown or use an absolute time range. You can also change the timezone associated with the query, or use a fiscal year.

1. Click **Change time settings** to change the timezone or apply a fiscal year.

Refer to [Set dashboard time range](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/use-dashboards/#set-dashboard-time-range) for more information on absolute and relative time ranges. You can also [control the time range using a URL](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/use-dashboards/#control-the-time-range-using-a-url).

#### Split and compare

The split view enables easy side-by-side comparison of visualizations or simultaneous viewing of related data on a single page.

To open the split view:

1. Click the split button to duplicate the current query and split the page into two side-by-side queries.
1. Run and re-run queries as often as needed.

You can select a different data source, or different metrics and label filters for the new query, allowing you to compare the same query across two different servers or compare the staging environment with the production environment.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-explore-split-10.1.png" max-width= "950px" caption="Screenshot of Explore screen split" >}}

You can also link the time pickers for both panels by clicking on one of the time-sync buttons attached to the time pickers. When linked, changing the time in one panel automatically updates the other, keeping the start and end times synchronized. This ensures that both split panels display data for the same time interval.

Click **Close** to quit split view.

### Query editor

The query editor is the interface where you construct the query for a data source.
Each data source's query editor provides a customized user interface that helps you write queries that take advantage of its unique capabilities.
To run queries across multiple data sources you need to select **Mixed** from the data source picker.
When you select **Mixed**, you can select a different data source for each new query that you add.

You can also reuse previously created queries with the saved queries feature.

Refer to [Query editor in Explore](http://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/explore/query-editor/) for detailed information on the query editor.

### Query history

The query history contains the list of queries that you created in Explore.
You can also add queries from the history to your saved queries.
Refer to [Query history](/docs/grafana/<GRAFANA_VERSION>/explore/query-management/#query-history) for detailed information on working with your query history.

### Query inspector

The query inspector provides detailed statistics regarding your query.
Inspector functions as a kind of debugging tool that "inspects" your query.
It provides query statistics under **Stats**, request response time under **Query**, data frame details under **{} JSON**, and the shape of your data under **Data**.
Refer to [Query inspector in Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/explore/explore-inspector/) for additional information.

## Access Explore

To access Explore:

1. Click on **Explore** in the left side menu.

   To start with an existing query from a dashboard panel, select the Explore option from the Panel menu in the upper right. This opens an Explore page with the panel's query, enabling you to tweak or iterate the query outside your dashboard.

   {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-panel-menu-10.1.png" class="docs-image--no-shadow" caption="Panel menu with Explore option" >}}

1. Select a data source from the drop-down in the upper left.

1. Using the query editor provided for the specific data source, begin writing your query. Each query editor differs based on each data source's unique elements.

Some query editors provide a **Kick start your query** option, which gives you a list of basic pre-written queries. Refer to [Use query editors](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/#use-query-editors) to see how to use various query editors. For general information on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/).

Based on specific data source, certain query editors allow you to select the label or labels to add to your query. Labels are fields that consist of key/value pairs representing information in the data. Some data sources allow for selecting fields.

1. Click **Run query** in the upper right to run your query.

## Share Explore URLs

When using Explore, the URL in the browser address bar updates as you make changes to the queries. You can share or bookmark this URL.

{{< admonition type="note" >}}
Explore may generate long URLs, which some tools, like messaging or videoconferencing applications, might truncate due to fixed message lengths. In such cases, Explore displays a warning and loads a default state.
If you encounter issues when sharing Explore links in these applications, you can generate shortened links. See [Share shortened link](#share-shortened-link) for more information.
{{< /admonition >}}

### Generate Explore URLs from external tools

Because Explore URLs have a defined structure, you can build a URL from external tools and open it in Grafana. The URL structure is:

```
http://<grafana_url>/explore?panes=<panes>&schemaVersion=<schema_version>&orgId=<org_id>
```

where:

- `org_id` is the organization ID
- `schema_version` is the schema version (should be set to the latest version which is `1`)
- `panes` is a URL-encoded JSON object of panes, where each key is the pane ID and each value is an object matching the following schema:

```
{
  datasource: string; // the pane's root datasource UID, or `-- Mixed --` for mixed datasources
  queries: {
    refId: string; // an alphanumeric identifier for this query, must be unique within the pane, i.e. "A", "B", "C", etc.
    datasource: {
      uid: string; // the query's datasource UID ie: "AD7864H6422"
      type: string; // the query's datasource type-id, i.e: "loki"
    }
    // ... any other datasource-specific query parameters
  }[]; // array of queries for this pane
  range: {
    from: string; // the start time, in milliseconds since epoch
    to: string; // the end time, in milliseconds since epoch
  }
}
```

{{< admonition type="note" >}}
The `from` and `to` also accept relative ranges defined in [Time units and relative ranges](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/use-dashboards/#time-units-and-relative-ranges).
{{< /admonition >}}

## Share shortened link

The Share shortened link capability allows you to create smaller and simpler URLs of the format `/goto/:uid` instead of using longer URLs with query parameters. To create a shortened link to the executed query, click the **Share** option in the Explore toolbar.

### Share shortened links with absolute time

{{< admonition type="note" >}}
Available in Grafana 10.3 and later versions.
{{< /admonition >}}

Shortened links have two options: relative time (e.g., from two hours ago to now) or absolute time (e.g., from 8am to 10am). By default, sharing a shortened link copies the selected time range, whether it's relative or absolute.

To create a short link with an absolute time:

1. Click the dropdown button next to the share shortened link button.
1. Select one of the options under **Time-Sync URL Links**.

This ensures that anyone receiving the link will see the same data you see, regardless of when they open it. Your selected time range will remain unaffected.

## Next steps

Now that you are familiar with Explore you can:

- [Build dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/)
- Create a wide variety of [visualizations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/)
- [Work with logs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/logs-integration/)
- [Work with traces](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/trace-integration/)
- [Create and use correlations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/correlations-editor-in-explore/)

# Logs in Explore

Explore is a powerful tool for logging and log analysis. It allows you to investigate logs from different data sources including:

- [Loki](/docs/grafana/<GRAFANA_VERSION>/datasources/loki/)
- [Elasticsearch](/docs/grafana/<GRAFANA_VERSION>/datasources/elasticsearch/)
- [Cloudwatch](/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/)
- [InfluxDB](/docs/grafana/<GRAFANA_VERSION>/datasources/influxdb/)
- [Azure Monitor](/docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/)
- [ClickHouse](https://github.com/grafana/clickhouse-datasource)

With Explore, you can efficiently monitor, troubleshoot, and respond to incidents by analyzing your logs and identifying the root causes. It also helps you to correlate logs with other telemetry signals such as metrics, traces or profiles, by viewing them side-by-side.

The results of log queries display as individual log lines and as a graph showing the logs volume for the selected time period.

## Logs volume

When working with data sources that support a full range logs volume, Explore automatically displays a graph showing the log distribution for all submitted log queries. This feature is currently supported by the Elasticsearch and Loki data sources.

{{< admonition type="note" >}}
In Loki, generating the full range log volume via a metric query can be resource-intensive, depending on the time range queried. This is especially challenging for smaller Loki installations. To mitigate this, we recommend that you use a proxy like [nginx](https://www.nginx.com/) in front of Loki with a timeout like 10ss. Log volume histogram queries can be identified by looking for queries with the HTTP header `X-Query-Tags` with value `Source=logvolhist`; these headers are added by Grafana to all log volume histogram queries.
{{< /admonition >}}

If the data source doesn't support loading the full range logs volume, the logs model calculates a time series by counting log rows and organizing them into buckets based on an automatically calculated time interval. The timestamp of the first log row is used to anchor the start of the logs volume in the results. The end of the time series is anchored to the time picker's **To** range. This way, you can still analyze and visualize log data efficiently even when the data source doesn't offer full range support.

## Logs

The following sections provide detailed explanations on how to visualize and interact with individual logs in Explore.

### Infinite scroll

<!-- vale Grafana.GoogleWill = NO -->

When you reach the bottom of the list of logs, you will see the message `Scroll to load more`. If you continue scrolling and the displayed logs are within the selected time interval, Grafana will load more logs. When the sort order is "newest first" you receive older logs, and when the sort order is "oldest first" you get newer logs.

<!-- vale Grafana.GoogleWill = YES -->

### Visualization options

You have the option to customize the display of logs and choose which columns to show. Following is a list of available options.

<!-- vale Grafana.Spelling = NO -->

| Option                                | Description                                                                                                                                                                                                                                                                                                                                                                         |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Expand / Collapse                     | Expand or collapse the controls toolbar.                                                                                                                                                                                                                                                                                                                                            |
| Scroll to bottom                      | Jump to the bottom of the logs table.                                                                                                                                                                                                                                                                                                                                               |
| Oldest Logs First / Newest logs first | Sort direction (ascending or descending).                                                                                                                                                                                                                                                                                                                                           |
| Search logs / Close search            | Click to open/close the client side string search of the displayed logs result.                                                                                                                                                                                                                                                                                                     |
| Deduplication                         | **None** does not perform any deduplication, **Exact** matches are done on the whole line except for date fields. **Numbers** matches are done on the line after stripping out numbers such as durations, IP addresses, and so on. **Signature** is the most aggressive deduplication as it strips all letters and numbers and matches on the remaining whitespace and punctuation. |
| Filter levels                         | Filter logs in display by log level: All levels, Info, Debut, Warning, Error.                                                                                                                                                                                                                                                                                                       |
| Set Timestamp format                  | Hide timestamps (disabled), Show milliseconds timestamps, Show nanoseconds timestamps.                                                                                                                                                                                                                                                                                              |
| Set line wrap                         | Disable line wrapping, Enable line wrapping, Enable line wrapping and prettify JSON.                                                                                                                                                                                                                                                                                                |
| Enable highlighting                   | Plain text, Highlight text.                                                                                                                                                                                                                                                                                                                                                         |
| Font size                             | Small font (default), Large font.                                                                                                                                                                                                                                                                                                                                                   |
| Unescaped newlines                    | Only displayed if the logs contain unescaped new lines. Click to unescape and display as new lines.                                                                                                                                                                                                                                                                                 |
| Download logs                         | Plain text (txt), JavaScript Object Notation (JSON), Comma-separated values (CSV)                                                                                                                                                                                                                                                                                                   |

<!-- vale Grafana.Spelling = YES -->

### Download log lines

This feature lets you save log data for further analysis or to share it with others in a convenient and accessible format.

In Explore there are three export options:

- **TXT** - will export the data as visible on the screen, meaning it will take into account formatting, like `line_format`.
- **JSON** - will export the raw data, regardless of the formatting, like `line_format`.
- **CSV** - will export the raw data, regardless of the formatting, like `line_format`.

Click **Download** and select `TXT`, `JSON` or `CSV` to download log results.

### Log result meta information

The following meta information displays above the retrieved log lines:

- **Number of received logs -** Indicates the total count of logs received for the current query or time range.
- **Error -** Displays any errors in your log results.
- **Common labels -** Displays common labels.
- **Total bytes processed -** Represents the cumulative size of the log data processed in bytes.

{{< admonition type="note" >}}
The availability of certain metadata may vary depending on the data source, so you might only see details related to those specific data sources.
{{< /admonition >}}

### Escaping newlines

Explore automatically detects some incorrectly escaped sequences in log lines, such as newlines (`\n`, `\r`) or tabs (`\t`). When it detects such sequences, Explore provides an **Escape newlines** option.

To automatically fix incorrectly escaped sequences that Explore has detected:

1. Click **Escape newlines** to replace the sequences.
2. Review returned log lines.

Explore replaces these sequences, changing the option from **Escape newlines** to **Remove escaping**. Assess the changes, as the parsing may not be accurate based on the input. To revert the replacements, click **Remove escaping**.

### Log level

For logs where a `level` label is specified, the value of this label is used to determine the log level and update the color of each log line accordingly.
If the log doesn't have a specified level label, Grafana treats the log line as not containing a level and omits rendering any value for it.
Refer to the following table for more information. The log level is always determined by the first match when level is defined.

{{< admonition type="tip" >}}
When using the Loki data source, if `level` is part of your log line, you can use parsers such as `json`, `logfmt`, or `regex` to extract the level information into a level label. This label is used to determine the level value, allowing the histogram to display the various log levels as separate bars.
{{< /admonition >}}

**Log levels supported and mapping of log level abbreviation and expressions:**

| Log level | Color      | Supported expressions                                   |
| :-------- | :--------- | ------------------------------------------------------- |
| critical  | purple     | emerg, emergency, fatal, alert, crit, critical, 0, 1, 2 |
| error     | red        | err, eror, error, 3                                     |
| warning   | yellow     | warn, warning, 4                                        |
| info      | blue       | info, information, informational, notice, 5, 6          |
| debug     | gray       | dbug, debug, 7                                          |
| trace     | light blue | trace                                                   |
| unknown   | gray       | \*                                                      |

### Highlight searched words

When your query includes specific words or expressions for keyword search, Explore highlights them in log lines to enhance visibility. This highlighting feature facilitates easier identification and focus on the relevant content within your logs.

{{< admonition type="note" >}}
The ability to highlight search words varies depending on data source. For some data sources, the highlighting of search words may not be available.
{{< /admonition >}}

### Log details view

In Explore, each log line has an expandable section called **Log details** that you open by clicking on the log line. The Log details view provides additional information and exploration options in the form of **Fields** and **Links** attached to the log lines, enabling a more robust interaction and analysis.

#### Fields

Within the **Log details** view, you have the ability to filter the displayed fields in two ways: a positive filter, which focuses on an specific field and a negative filter, which excludes certain fields.
These filters modify the corresponding query that generated the log line, incorporating equality and inequality expressions accordingly.

If the data source supports it, as is the case with Loki and Elasticsearch, log details will verify whether the field is already included in the current query, indicating an active state for positive filters. This enables you to toggle it off from the query or convert the filter expression from positive to negative as necessary.

Click the **eye icon** to select a subset of fields to visualize in the logs list instead of the complete log line.

Each field has a **stats icon**, which displays ad-hoc statistics in relation to all displayed logs.

For data sources that support log types, such as Loki, instead of a single view containing all fields, fields will be displayed grouped by their type: Indexed Labels, Parsed fields, and Structured Metadata.

#### Links

Grafana provides data links or correlations, allowing you to convert any part of a log message into an internal or external link. These links enable you to navigate to related data or external resources, offering a seamless and convenient way to explore additional information.

{{< figure src="/static/img/docs/explore/data-link-9-4.png" max-width="800px" caption="Data link in Explore" >}}

#### Log details modes

There are two modes available to view log details:

- **Inline** The default, displays log details below the log line.
- **Sidebar** Displays log details in a sidebar view.

No matter which display mode you are currently viewing, you can change it by clicking the mode control icon.

### Log context

Log context is a feature that displays additional lines of context surrounding a log entry that matches a specific search query. This helps in understanding the context of the log entry and is similar to the `-C` parameter in the `grep` command.

If you're using Loki for your logs, to modify your log context queries, you can use the Loki log context query editor at the top of the table. You can activate this editor by clicking the menu for the log line, and selecting **Show context**. Within the **Log Context** view, you have the option to modify your search by removing one or more label filters from the log stream. If your original query used a parser, you can refine your search by leveraging extracted label filters.

Change the **Context time window** option to look for logs within a specific time interval around your log line.

Toggle **Wrap lines** if you encounter long lines of text that make it difficult to read and analyze the context around log entries. By enabling this toggle, Grafana automatically wraps long lines of text to fit within the visible width of the viewer, making the log entries easier to read and understand.

Click **Open in split view** to execute the context query for a log entry in a split screen in the Explore view. Clicking this button opens a new Explore pane with the context query displayed alongside the log entry, making it easier to analyze and understand the surrounding context.

Use Command-click or Ctrl+click to open the log context query in a new browser to view the context model. All previously selected filters get applied.

### Copy log line

Click **Copy log line** to copy the content of a selected log line to the clipboard.

### Copy link to log line

Linking log lines in Grafana allows you to quickly navigate to specific log entries for detailed and precise analysis. Click **Copy shortlink** to generate and copy a [short URL](/docs/grafana/<GRAFANA_VERSION>/developers/http_api/short_url/) that provides direct access to the exact log entry within an absolute time range. When you open the link, Grafana automatically scrolls to the corresponding log line and highlights it in blue, making it easy to identify and focus on relevant information.

{{< admonition type="note" >}}
The short URL feature is currently only supported in Loki and other data sources that provide an `id` field.
{{< /admonition >}}

## Live tailing

Use the **Live tail** feature to view real-time logs from data sources.

1. Click **Live** in the Explore toolbar to switch to Live tail view.
1. In **Live tail view**, new logs appear at the bottom of the screen, and have a contrasting background, allowing you to easily track what's new.
1. Click **Pause** to pause live tailing and explore previous logs without interruptions. or simply scroll through the logs view.
1. Click **Clear logs** to remove all displayed logs. This action resets the log view and provides a clean slate to continue your log analysis
1. Click **Resume** to resume live tailing and continue viewing real-time logs.
1. Click **Stop** to exit live tailing and return to the standard Explore view.

The **Live tailing feature** allows you to monitor the latest logs in real-time, making it easier to track events as they occur and promptly detect issues.

{{< video-embed src="/static/img/docs/v95/explore_live_tailing.mp4" >}}

### Logs sample

If the selected data source supports log samples and both log and metric queries, you will automatically see log line samples that contribute to the visualized metrics for metric queries. **This feature is currently only supported by the Loki data source.**

### Switch from metrics to logs

If you are transitioning from a metrics data source that implements `DataSourceWithQueryExportSupport` (such as Prometheus) to a logging data source that supports `DataSourceWithQueryImportSupport` (such as Loki), Explore retains the labels from your query that exist in the logs and use them to query the log streams.

For example, after switching to the Loki data source, the Prometheus query `grafana_alerting_active_alerts{job="grafana"}` changes to `{job="grafana"}`. This will retrieve a set of logs within the specified time range, which can be searched using grep or text search.

# Query editor in Explore

The query editor is the interface where you construct the query for a data source.
Each data source's query editor provides a customized user interface that helps you write queries that take advantage of its unique capabilities.

Because of the differences between query languages, each data source query editor looks and functions differently.
Depending on your data source, the query editor might provide auto-completion features, metric names, variable suggestions, or a visual query-building interface.

{{< figure src="/media/docs/grafana/dashboards/screenshot-explore-query-editor-v13.2.png" max-width="750px" alt="The Prometheus query editor in Explore" >}}

For details on a specific data source's unique query editor features, refer to its documentation:

- For data sources included with Grafana, refer to [Built-in core data sources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/#built-in-core-data-sources), which links to each core data source's documentation.
- For data sources installed as plugins, refer to the documentation for the plugin.
  - Data source plugins in the Grafana [plugin catalog](/grafana/plugins/) link to or include their documentation in their catalog listings.
    For details about the plugin catalog, refer to [Plugin management](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/plugin-management/).
  - For links to Grafana Enterprise data source plugin documentation, refer to the [Enterprise plugins index](https://grafana.com/docs/plugins/).

## Query syntax

Each data source uses a different query languages to request data.
For details on a specific data source's unique query language, refer to its documentation.

**PostgreSQL example:**

```
SELECT hostname FROM host WHERE region IN($region)
```

**PromQL example:**

```
query_result(max_over_time(<metric>[${__range_s}s]) != <state>)
```

## Special data sources

Grafana also includes three special data sources: **Grafana**, **Mixed**, and **Dashboard**.
For details, refer to [Data sources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/)

## Saved queries

{{< admonition type="note" >}}
Saved queries is only available on Grafana Enterprise and Grafana Cloud.
{{< /admonition >}}

You can save queries that you've created so they can be reused by you and others in your organization.
This helps users across your organization find insights in Explore without having to create their own queries or know a query language.
It also helps you avoid having several users build the same queries for the same data sources multiple times.

Saved queries are supported in:

- [Dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/#create-a-dashboard)
- [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/explore/get-started-with-explore/#explore-elements)
- [Annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/annotate-visualizations/#add-new-annotation-queries)

Learn more about saved queries:

- [Saved queries dialog box](#saved-queries-dialog-box)
- [Roles, permissions, and RBAC](#roles-permissions-and-rbac)
- [How to save a query](#save-a-query)
- [Variables in saved queries](#variables-in-saved-queries)
- [Manage saved queries as code](#manage-saved-queries-as-code)
- [Known limitations](#known-limitations)

### Saved queries dialog box

The **Saved queries** dialog box gives you access to all the saved queries in your organization:

{{< figure src="/media/docs/grafana/dashboards/screenshot-saved-queries-v13.0.png" max-width="750px" alt="List of saved queries" >}}

You can access saved queries three ways:

- Press `Ctrl + K` or `Cmd + K` to open the command palette and search "Saved queries", and select the **Open saved queries** action. The **Shared queries** dialog box opens, where you can select a query and then click **Open in Explore**.
- Click **Saved queries > Replace query** in the query editor.
- At the bottom of the query editor, click **Add from saved queries**.

{{< figure src="/media/docs/grafana/dashboards/screenshot-add-saved-reuse-query-v13.2.png" max-width="750px" alt="Access saved queries" >}}

Clicking **Add from saved queries** adds an additional query, while clicking **Replace query** updates your configured query.

From the **Saved queries** dialog box, you can:

- Search for queries by data source name, query content, title, or description.
- Sort queries alphabetically or by creation date.
- Filter by data source name, author name, and tags. The tag filter uses the `OR` operator, while the others use the `AND` operator. Use the **Remember filters** switch to persist your filter selections across sessions in your local storage.
- Star queries so that they appear in the **Starred queries** filter view.
- Duplicate or delete a saved query.
- Edit a query title, description, or tags.
- Click **Edit** at the bottom of the dialog box to update the body of a query.

You can apply all the same search, filter, and sort options in the **Starred queries** filter view.

{{< admonition type="tip">}}
When you select a query with a Loki, Mimir, Tempo, or Pyroscope data source, the **Saved queries** dialog box displays a **Drilldown** button.
Click the button to open the associated Drilldown app, while maintaining the context of the query.
Learn more about these apps in the [Drilldown documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/simplified-exploration/).
{{< /admonition >}}

### Roles, permissions, and RBAC

Saved queries support role-based access controls.
By default, saved queries have two RBAC roles:

- **Writer**: Create, update, and delete all saved queries.
- **Reader**: Reuse saved queries.

If you used saved queries prior to the addition of RBAC support in Grafana v12.4, Grafana user roles are mapped as follows:

- Admin > Writer
- Editor > Writer
- Viewer > Reader

### Save a query

To save a query you've created:

1. From the query editor, open the **Saved queries** drop-down menu and click the **Save query** option:

   {{< figure src="/media/docs/grafana/dashboards/screenshot-save-query-v13.2.png" max-width="750px" alt="Save a query" >}}

1. In the **Saved queries** dialog box, enter a title for the query that makes it easier to find later.
1. (Optional) Enter a description and relevant tags.
1. Click **Save**.

### Variables in saved queries

If a saved query includes variables, you can substitute the variables in the query without modifying it by mapping them to a custom value.
This is useful in environments where variable names or available values differ between dashboards.

{{< figure src="/media/docs/grafana/dashboards/screenshot-saved-query-variable-v13.2.png" max-width="450px" alt="A saved query with substituted variables" >}}

Grafana applies your selections to the query before inserting it into the dashboard.
However, the substitutions only apply to the query when it's reused, and the original saved query remains unchanged.

### Manage saved queries as code

You can manage saved queries as code with the Grafana Terraform provider, which lets you version-control your query library and keep it consistent across instances.
For more information, refer to [Manage saved queries using Terraform](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/as-code/terraform/manage-saved-queries/).

### Known limitations

- No validation is performed when you save a query, so it's possible to save an invalid query. You should confirm the query is working properly before you save it.
- You can save a maximum of 1000 queries.
- If you have multiple queries open in Explore and you edit one of them by way of the **Edit in Explore** function in the **Saved queries** dialog box, the edited query replaces your open queries in Explore.

## Add a query

To add a query, follow these steps:

1. If you want use a data source that's not the default one, click the data source drop-down menu in the toolbar and make a selection.

   If you want to use multiple data sources, select the **Mixed** data source at this step. The query editor displays a secondary data source picker.

1. To create a query, do one of the following:
   - Write or construct a query in the query language of your data source.
   - Click **Replace** to reuse a saved query.

   {{< admonition type="note" >}}
   [Saved queries](#saved-queries) is only available on Grafana Enterprise and Grafana Cloud.
   {{< /admonition >}}

1. (Optional) To [save the query](#save-a-query) for reuse, click the **Save** in the editor pane.
1. (Optional) At the bottom of the query editor, click **Add query** or **Add from saved queries** to add more queries as needed.
1. Click the **Run query** icon in the toolbar.

Grafana queries the data source.
If the data source supports graph and table data, Explore displays the results in the **Graph**.

## Manage queries

The following table describes actions you can take for each query:

<!-- prettier-ignore-start -->
| Icon    | Description                                  |
| ------- | -------------------------------------------- |
| {{< figure src="/media/docs/grafana/panels-visualizations/replace-query-icon-v13.1.png" max-width="30px" max-height="30px" alt="Replace query icon" >}} | [Saved query](#saved-queries) options. Open the drop-down list and choose from **Save query** or **Replace query**.(Enterprise and Cloud only). |
| {{< figure src="/static/img/docs/queries/query-editor-help-7-4.png" max-width="30px" max-height="30px" alt="Help icon" >}} | Toggles query editor help. If supported by the data source, click this icon to display information on how to use the query editor or provide quick access to common queries. Click the **More query actions** menu to access this option. |
| {{< figure src="/media/docs/grafana/panels-visualizations/create-recorded-query-icon.png" max-width="30px" max-height="30px" alt="Create recorded query icon" >}} | Create [recorded queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/recorded-queries/) so you can see trends over time by taking a snapshot of a data point on a set interval (Enterprise and Cloud only). |
| {{< figure src="/static/img/docs/queries/duplicate-query-icon-7-0.png" max-width="30px" max-height="30px" alt="Duplicate icon" >}} | Copies a query. Duplicating queries is useful when working with multiple complex queries that are similar and you want to either experiment with different variants or do minor alterations. Click the **More query actions** menu to access this option. |
| {{< figure src="/static/img/docs/queries/hide-query-icon-7-0.png" max-width="30px" max-height="30px" alt="Hide icon" >}} | Hides a query. Grafana does not send hidden queries to the data source. |
| {{< figure src="/static/img/docs/queries/remove-query-icon-7-0.png" max-width="30px" max-height="30px" alt="Remove icon">}} | Removes a query. Removing a query permanently deletes it, but sometimes you can recover deleted queries by reverting to previously saved versions of the panel. |
<!-- prettier-ignore-end -->

# Query management in Explore

Grafana Explore provides a variety of tools to help manage your queries.

{{< admonition type="note" >}}
For help with debugging queries, Explore allows you to investigate query requests and responses, as well as query statistics, via the Query inspector. Refer to [Query inspector in Explore](/docs/grafana/<GRAFANA_VERSION>/explore/explore-inspector/) for more information.
{{< /admonition >}}

## Query history

Query history contains the list of queries that you created in Explore. This history is stored in the Grafana database and isn't shared with other users. The retention period for a query history is **two weeks**. Queries older than two weeks are automatically deleted.

{{< admonition type="note" >}}
Starred queries aren't subject to the two-week retention period and aren't deleted.
{{< /admonition >}}

To view your query history:

1. Go to the Explore page.
1. Click **Query history**.

The Query history pane opens at the bottom of the page, and contains the following tabs:

- **Query history tab-** Contains a history of all your queries, with options for searching and managing them.
- **Starred tab -** Contains all of your starred queries.
- **Settings tab-** Provides customizable options for your query history.

### Query history tab

The Query history depicts a history of your queries for the past two weeks, unless the query is starred, which means it doesn't get deleted. For each individual query, you can:

- Run and re-run the query.
- Create and/or edit a comment.
- Copy a query to the clipboard.
- Copy a shortened link with the query to the clipboard.
- Delete a query.
- Star a query.
- Add a query from your history to your [saved queries](ref:saved-queries).

By default, query history shows you newest queries first. Click the sort box in the upper right to change to **Oldest first** to older queries first. You can search your queries using keywords.

### Query history Starred tab

All starred queries are displayed in the **Starred** tab. This gives quick access to key or favorite queries without having to rewrite them.

You also have the option to switch the data source and run a starred query.

#### Filter query history

Filter query history in both the **Query history** and **Starred** tabs by data source name:

1. Click the **Filter queries for specific data source(s)** field.
1. Select the data source in the dropdown by which you want to filter your history. You can select multiple data sources.

{{< admonition type="note" >}}
Queries with the **Mixed** data source appear only when filtering for "Mixed" and not when filtering by individual data source.
{{< /admonition >}}

You can also filter queries by date using the vertical slider:

- Drag the bottom circle to adjust the start date.
- Drag the top circle to adjust the end date.

#### Search in query history

Use **Search queries** in both the **Query history** and **Starred** tabs to search your query history and comments using keywords.

1. Click in the **Search queries** field.
1. Type the keyword(s) or term you are want to search for in search field.

### Query history Settings tab

You can customize your query history in the **Settings** tab.

Toggle **Change the default active tab from "Query history" to "Starred"** to make the **Starred tab** the default active tab.

{{< admonition type="note" >}}
Query history settings are global, and applied to both panels in split mode.
{{< /admonition >}}

<!-- All queries that have been starred in the Query history tab are displayed in the Starred tab. This allows you to access your favorite queries faster and to reuse these queries without typing them from scratch. -->

# Traces in Explore

You can use Explore to query and visualize traces from tracing data sources. Supported data sources include:

- [Tempo](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/)
- [Jaeger](/docs/grafana/<GRAFANA_VERSION>/datasources/jaeger/)
- [Zipkin](/docs/grafana/<GRAFANA_VERSION>/datasources/zipkin/)
- [X-Ray](https://grafana.com/grafana/plugins/grafana-x-ray-datasource)
- [Azure Monitor](/docs/grafana/latest/datasources/azure-monitor/)
- [ClickHouse](https://github.com/grafana/clickhouse-datasource)
- [New Relic](/docs/plugins/grafana-newrelic-datasource/latest/)
- [Infinity](/docs/plugins/yesoreyeram-infinity-datasource/latest/)

Here are some references to learn more about traces and how you can use them:

- [Introduction to tracing](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/)
- [Trace structure](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/trace-structure/#trace-structure)
- [Traces and telemetry](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/telemetry/)
- [Using traces to find solutions to problems](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/solutions-with-traces/)

## Query editors

You can query and search tracing data using a data source's query editor. Note that data sources in Grafana have unique query editors.

For information on how to use the query editor to create queries for tracing data sources, refer to the documentation for each individual data source.

## Trace view

The trace view provides an overview of a request as it travels through your system. The following sections provide detail on various elements of the trace view.

{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view.png" class="docs-image--no-shadow" max-width= "900px" caption="Trace view" >}}

### Header

The trace view header includes the following:

- **Header title** - Shows the name of the root span and trace ID.
- **Search** - Highlights spans containing the searched text.
- **Metadata** - Various metadata about the trace.

{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-header.png" class="docs-image--no-shadow" max-width= "750px" caption="Trace view header" >}}

### Minimap

**Minimap** displays a condensed view of the trace timeline. Drag your mouse over the minimap to zoom into a smaller time range. This also updates the main timeline, making it easier to view shorter spans
When zoomed in, hovering over the minimap displays **Reset selection**, which resets the zoom.

{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-minimap.png" class="docs-image--no-shadow" max-width= "900px" caption="Trace view minimap example" >}}

### Timeline

Timeline shows list of spans within the trace. Each span row consists of the following components:

- **Expand children** - Expands or collapses all the children spans of the selected span.
- **Service name** - Name of the service logged the span.
- **Operation name** - Name of the operation that this span represents.
- **Span duration bar** - Visual representation of the operation duration within the trace.

Click anywhere on the span row to reveal span details.

{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-timeline.png" class="docs-image--no-shadow" max-width= "900px"  caption="Trace view timeline" >}}

The **Span duration bar** highlights the trace’s critical path with a darker segment. This darker segment means the span lies on the trace’s critical path, following the CRISP (Critical Path for Service Performance) algorithm.

Use the highlight to focus on spans that drive end‑to‑end latency. Spans outside the critical path do not increase the trace duration.

{{< figure src="/media/docs/grafana/explore/traces-panel-critical-path.png" class="docs-image--no-shadow" max-width= "900px"  caption="Critical path span" >}}

You can also focus on this path by clicking the **Show critical path only** button in the **Span filters** options.

### Span details

Traces are composed of one or more spans.
A span is a unit of work within a trace that has a start time relative to the beginning of the trace, a duration and an operation name for the unit of work.
It usually has a reference to a parent span, unless it’s the first span, the root span, in a trace.
It frequently includes key/value attributes that are relevant to the span itself, for example the HTTP method used in the request, as well as other metadata such as the service name, sub-span events, or links to other spans.

You can expand any span in a trace and view the details, including the span and resource attributes.

For more information about spans and traces, refer to [Introduction to tracing](https://grafana.com/docs/tempo/latest/introduction/) in the Tempo documentation.

Span details include span attributes, resource attributes, events, and links.

#### Span and resource attributes

**Span attributes** are key-value pairs that provide metadata about a specific span. They give context to the operation being performed, such as information about the request, response, or any relevant operational details. For example, if the span deals with calling another service via HTTP, an attribute could include the HTTP URL (maybe as the span attribute key `http.url`) and the HTTP status code returned (as the span attribute `http.status_code`).

{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-span-span-attributes.png" class="docs-image--no-shadow" max-width= "900px"  caption="Trace view span attributes" >}}

**Resource attributes** are key-value pairs that describe the environment or entity that is producing the trace. They capture static information about the origin of traces, like the application name or the service version.

{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-span-resource-attributes.png" class="docs-image--no-shadow" max-width= "900px"  caption="Trace view span resource attributes" >}}

Span attributes are specific to a particular operation, while resource attributes are associated with the whole trace or the entire service emitting the spans. Refer to [Span and resource attributes](/docs/tempo/<TEMPO_VERSION>/operations/best-practices/#span-and-resource-attributes) for more detail.

#### Events

Events are log-like records attached to a span that represent an occurrence during its execution. They record notable moments or occurrences within the span's lifecycle, such as errors, warnings, or checkpoints. If an error occurs during an operation, an event can be added to the span to indicate what went wrong and when. Events include a timestamp, name, and key-value pairs attributes that provide additional context or details about the event.

{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-span-events.png" class="docs-image--no-shadow" max-width= "900px"  caption="Trace view span events" >}}

#### Links

Links show relationships between spans that are not in a direct parent-child relationship. They represent associations between spans that happen concurrently or across separate trace trees, linking traces that originated from separate sources but are logically connected, such as background job processing initiated from a web request. You might use links when a trace passes through an asynchronous queue or when correlating traces from different services.

### Span filters

Span filters allow you to refine the spans displayed in the trace timeline viewer.
The more filters you add, the more specific the filtered spans become.
Click on a trace to access Span filters.

![Screenshot of span filtering](/media/docs/tempo/screenshot-grafana-tempo-span-filters-v10-1.png)

You can add one or more of the following filters:

- **Service name** - Filter by selecting a service name from the dropdown.
- **Span name** - Filter by selecting a span name from the dropdown.
- **Duration** - Filter by duration. Accepted units include ns, us, ms, s, m, h.
- **Tags** - Filter by tags, process tags, or log fields in your span.

To only show the spans you have matched, toggle **Show matches only**.

Refer to [Span filters](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/span-filters/) for more in depth information.

### Trace to logs

You can navigate from a span in a trace view directly to logs relevant for that span.
This feature is available for the Tempo, Jaeger, and Zipkin data sources.
Refer to each individual data source's documentation for configuration instructions.

Click the document icon to open a split view in Explore with the configured data source and query relevant logs for the span.

{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-trace-to-logs.png" class="docs-image--no-shadow" max-width= "900px" caption="Trace to logs" >}}

### Trace to metrics

You can navigate from a span in a trace view directly to metrics relevant for that span.
This feature is available for the Tempo, Jaeger, and Zipkin data sources.

Refer to each individual data source's documentation for configuration instructions.
For Tempo, refer to [Trace to metrics configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/).

### Trace to profiles

Using Trace to profiles, you can use Grafana’s ability to correlate different signals by adding the functionality to link between traces and profiles.

For Tempo, refer to [Trace to profiles](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/) for configuration instructions.

{{< figure src="/static/img/docs/tempo/profiles/tempo-trace-to-profile.png" max-width="900px" class="docs-image--no-shadow" alt="Selecting a link in the span queries the profile data source" >}}

### Trace correlations

You can use [correlations](/docs/grafana/<GRAFANA_VERSION>/administration/correlations/) to define custom links that appear in the trace view based on trace and span information.

For Tempo, refer to [Trace correlations](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/trace-correlations/) for configuration instructions.

{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-correlations.png" max-width="900px" class="docs-image--no-shadow" alt="Using correlations for a trace" >}}

## Node graph

You can also expand the node graph for a displayed trace. If the data source supports it, this displays spans of the trace as nodes in the graph, or provides additional context, such as a service graph based on the current trace.

Refer to [Node graph](/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/node-graph/) for additional information.

{{< admonition type="note" >}}
The node graph requires data to be returned from the data source in a specific format to display correctly. Refer to [Data API](/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/node-graph/#data-api), [Nodes data frame structure](/docs/grafana/latest/panels-visualizations/visualizations/node-graph/#nodes-data-frame-structure) and [Node graph data requirements](/docs/grafana/latest/panels-visualizations/visualizations/node-graph/#data-requirements) for additional information and configuration instructions.
{{< /admonition >}}

{{< figure src="/media/docs/tempo/screenshot-grafana-node-graph.png" class="docs-image--no-shadow" max-width= "900px"  caption="Node graph" >}}

## Service graph

A service graph visualizes rates, error rates, and durations (RED), along with service relationships.
After the requirements are configured, this pre-configured view is immediately available.

For additional information refer to the following documentation:

- [Service Graph and Service Graph view](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/)
- [Service graph view](/docs/tempo/<TEMPO_VERSION>/metrics-generator/service-graph-view/) in Tempo documentation

{{< figure src="/static/img/docs/grafana-cloud/apm-overview.png" class="docs-image--no-shadow" max-width= "900px" caption="Service graph view" >}}

# Configure data links and actions

_Data links_ allow you to link to other panels, dashboards, and external resources and _actions_ let you trigger basic, unauthenticated, API calls.
In both cases, you can carry out these tasks while maintaining the context of the source panel.

## Data links

With data links, you can create links that include the series name or even the value under the cursor. For example, if your visualization shows four servers, you can add a data link to one or two of them.

The link itself is accessible in different ways depending on the visualization. For the time series visualization you need to click a data point or line:

![Time series visualization with a data link displayed](/media/docs/grafana/panels-visualizations/screenshot-time-series-data-link-v10.3.png)

For visualizations like stat, gauge, or bar gauge you can click anywhere on the visualization to open the context menu:

![Stat visualization with a data link displayed](/media/docs/grafana/panels-visualizations/screenshot-stat-data-link-v10.3.png)

If there's only one data link in the visualization, clicking anywhere on the visualization opens the link rather than the context menu.

### Supported visualizations

You can configure data links for the following visualizations:

{{< column-list >}}

- [Bar chart](ref:bar-chart)
- [Bar gauge](ref:bar-gauge)
- [Candlestick](ref:candlestick)
- [Canvas](ref:canvas)
- [Gauge](ref:gauge)
- [Geomap](ref:geomap)
- [Heatmap](ref:heatmap)
- [Histogram](ref:histogram)
- [Pie chart](ref:pie-chart)
- [Stat](ref:stat)
- [State timeline](ref:state-timeline)
- [Status history](ref:status-history)
- [Table](ref:table)
- [Time series](ref:time-series)
- [Trend](ref:trend)
- [XY chart](ref:xy-chart)

{{< /column-list >}}

## Actions

Using actions, you can trigger processes like starting or shutting down a server, directly from a dashboard panel. [API settings](ref:api-settings) are configured in the **Add action** dialog box. You can also pass variables in the API editor.

### Supported visualizations

You can configure actions for the following visualizations:

{{< column-list >}}

- [Bar chart](ref:bar-chart)
- [Candlestick](ref:candlestick)
- [State timeline](ref:state-timeline)
- [Status history](ref:status-history)
- [Table](ref:table)
- [Time series](ref:time-series)
- [Trend](ref:trend)
- [XY chart](ref:xy-chart)

{{< /column-list >}}

## Data link and action variables {#data-link-variables}

Variables in data links and actions let you send people to a detailed dashboard or trigger an API call with preserved data filters. For example, you could use variables to specify a label, time range, series, or variable selection.

To see a list of available variables, enter `$` in the data link or action **URL** field.

{{< admonition type="note" >}}
These variables changed in 6.4 so if you have a previous version of Grafana, then use the version picker to select docs for a previous version of Grafana.
{{< /admonition >}}

Azure Monitor, [CloudWatch](ref:cloudwatch), and [Google Cloud Monitoring](ref:google-cloud-monitoring) have pre-configured data links called _deep links_.

You can also use template variables in your data links or actions URLs. For more information, refer to [Templates and variables](ref:templates-and-variables).

### Time range panel variables

These variables allow you to include the current time range in the data link or action URL:

| Variable           | Description                                                          |
| ------------------ | -------------------------------------------------------------------- |
| `__url_time_range` | Current dashboard's time range (for example, `?from=now-6h&to=now`). |
| `__from`           | A specific start time.                                               |
| `__to`             | A specific end time.                                                 |

When you create data links and actions using time range variables like `__url_time_range` in the URL, you have to form the query parameter syntax yourself; that is, you must format the URL by appending query parameters using the question mark (`?`) and ampersand (`&`) syntax. These characters aren't automatically generated.

For more information about these variables, refer to the [Built-in variables documentation](ref:built-in-variables).

### Series variables

Series-specific variables are available under `__series` namespace:

| Variable        | Description            |
| --------------- | ---------------------- |
| `__series.name` | Series name to the URL |

### Field variables

Field-specific variables are available under `__field` namespace:

| Variable                 | Description                                                                                         |
| ------------------------ | --------------------------------------------------------------------------------------------------- |
| `__field.name`           | The name of the field                                                                               |
| `__field.displayName`    | The display name of the field. Use this for renamed field names.                                    |
| `__field.labels.<LABEL>` | Label's value to the URL. If your label contains dots, then use `__field.labels["<LABEL>"]` syntax. |

### Value variables

Value-specific variables are available under `__value` namespace:

| Variable          | Description                                                                       |
| ----------------- | --------------------------------------------------------------------------------- |
| `__value.time`    | Value's timestamp (Unix ms epoch) to the URL (for example, `?time=1560268814105`) |
| `__value.raw`     | Raw value                                                                         |
| `__value.numeric` | Numeric representation of a value                                                 |
| `__value.text`    | Text representation of a value                                                    |
| `__value.calc`    | Calculation name if the value is result of calculation                            |

Using value-specific variables in data links and actions can show different results depending on the set option of Tooltip mode.

When you create data links and actions using time range variables like `__value.time` in the URL, you have to form the query parameter syntax yourself; that is, you must add the question mark (`?`) and ampersand (`&`). These characters aren't automatically generated.

### Data variables

To access values and labels from other fields use:

| Variable                          | Description                                |
| --------------------------------- | ------------------------------------------ |
| `__data.fields[i]`                | Value of field `i` (on the same row)       |
| `__data.fields["NameOfField"]`    | Value of field using name instead of index |
| `__data.fields[1].labels.cluster` | Access labels of another field             |

### Template variables

When linking to another dashboard that uses template variables, select variable values for whoever clicks the link.

`${var-myvar:queryparam}` - where `var-myvar` is the name of the template variable that matches one in the current dashboard that you want to use.

| Variable state           | Result in the created URL           |
| ------------------------ | ----------------------------------- |
| selected one value       | `var-myvar=value1`                  |
| selected multiple values | `var-myvar=value1&var-myvar=value2` |
| selected `All`           | `var-myvar=All`                     |

The `queryparam` format generates the ampersand (`&`) _within_ a single multi-value variable, as shown in the previous table.
However, it doesn't automatically insert an ampersand _between_ different template variables.
When you combine multiple template variables in one URL, you must add the ampersand yourself:

```text
/d/xxxx/dashboard-b?${var-server:queryparam}&${var-host:queryparam}
```

In this example, the ampersand between `${var-server:queryparam}` and `${var-host:queryparam}` is required.
Without it, the resulting URL is malformed.

If you want to add all of the current dashboard's variables to the URL, use `${__all_variables}`.

When you link to another dashboard, ensure that:

- The target dashboard has the same variable name. If it doesn't (for example, `server` in the source dashboard and `host` in the target), you must align them or explicitly map values (for example, `&var-host=${server}`).
- You use the variable _name_, and not the label. Labels are only used as display text and aren't recognized in URLs.

For example, if you have a variable with the name `var-server` and the label `ChooseYourServer`, you must use `var-server` in the URL, as shown in the following table:

| Correct link                                   | Incorrect link                                           |
| ---------------------------------------------- | -------------------------------------------------------- |
| `/d/xxxx/dashboard-b?orgId=1&var-server=web02` | `/d/xxxx/dashboard-b?orgId=1&var-ChooseYourServer=web02` |

## Add data links or actions {#add-a-data-link}

The following tasks describe how to configure data links and actions.

{{< tabs >}}
{{< tab-content name="Add data links" >}}
To add a data link, follow these steps:

1. Navigate to the panel to which you want to add the data link.
1. Hover over any part of the panel to display the menu icon in the upper-right corner.
1. Click the menu icon and select **Edit** to open the panel editor.
1. Scroll down to the **Data links and actions** section and expand it.
1. Click **+ Add link**.
1. In the dialog box that opens, enter a **Title**.

   This is a human-readable label for the link displayed in the UI. This is a required field.

1. Enter the **URL** to which you want to link.
1. (Optional) To add a data link variable, click in the **URL** field and enter `$` or press Ctrl+Space or Cmd+Space to see a list of available variables.
1. If you want the link to open in a new tab, toggle the **Open in a new tab** switch.
1. If you want the data link to open with a single click on the visualization, toggle the **One click** switch.

   Only one data link can have **One click** enabled at a time. **One click** is only supported for some visualizations.

1. Click **Save** to save changes and close the dialog box.
1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Back** and then **Exit edit**.

   {{< /tab-content >}}
   {{< tab-content name="Add actions" >}}

   {{< admonition type="note">}}
   Actions aren't supported for all visualizations. For the list of supported visualizations, refer to [Supported visualizations](#supported-visualizations-1).
   {{< /admonition >}}

   To add an action, by follow these steps:

1. Navigate to the panel to which you want to add the action.
1. Hover over any part of the panel to display the menu icon in the upper-right corner.
1. Click the menu icon and select **Edit** to open the panel editor.
1. Scroll down to the **Data links and actions** section and expand it.
1. Click **+ Add action**.
1. In the dialog box that opens, define the API call settings:

   | Option               | Description                                                                                                                                                                                                                                 |
   | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | Title                | A human-readable label for the action that's displayed in the UI.                                                                                                                                                                           |
   | Confirmation message | A descriptive prompt to confirm or cancel the action.                                                                                                                                                                                       |
   | Connection           | Specify how the action's HTTP request is sent. Choose from: **Direct from browser** or routed through a configured data source.                                                                                                             |
   | Method               | Select from **POST** or **GET**.                                                                                                                                                                                                            |
   | URL                  | The request URL.</p><p>To add a variable, click in the **URL** field and enter `$` or press Ctrl+Space or Cmd+Space to see a list of available variables.                                                                                   |
   | Variables            | **Key** and **Name** pairs with a type selection. Click the **+** icon to add as many variables as you need. To add a variable to the request, prefix the key with `$`. You can set the values for the variables when performing an action. |
   | Query parameters     | **Key** and **Value** pairs. Click the **+** icon to add as many key/value pairs as you need.                                                                                                                                               |
   | Headers              | Comprised of **Key** and **Value** pairs and a **Content-Type**.</p><p>Click the **+** icon to add as many key/value pairs as you need.                                                                                                     |
   | Content-Type         | Select from the following: **application/json**, **text/plain**, **application/XML**, and **application/x-www-form-urlencoded**.                                                                                                            |
   | Body                 | The body of the request.                                                                                                                                                                                                                    |

1. Click **Save** to save changes and close the dialog box.
1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Back** and then **Exit edit**.
   {{< /tab-content >}}
   {{< /tabs >}}

If you add multiple data links or actions, you can control the order in which they appear in the visualization. To reorder items, drag the data link or action to the desired position.

## Panel-to-panel filtering

You can use data links to link back to the dashboard you're on. This enables "panel-to-panel filtering," where clicking a data point in one panel updates the dashboard variables and filters the rest of the dashboard.

To preserve the context of the current dashboard:

- **Time range:** You must explicitly include the current time range in the link.
- **Variables:** You must enable **Include all variables** to preserve current selections.
- **Ordering:** Place **Include all variables** before the specific variable you define in the link.

Filters on the current dashboard are automatically preserved.
Learn more in [Create dashboard URL variables > Filters](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard-url-variables/#filters).

# Configure a legend

A panel includes a legend that you can use to identify and interpret data displayed in a visualization. Each legend option adds context and clarity to the data illustrated in a visualization.

## Supported visualizations

Legends are supported for the following visualizations:

- [Bar chart](ref:bar-chart)
- [Candlestick](ref:candlestick)
- [Histogram](ref:histogram)
- [Pie chart](ref:pie-chart)
- [State timeline](ref:state-timeline)
- [Status history](ref:status-history)
- [Time series](ref:time-series)
- [Trend](ref:trend)
- [XY chart](ref:xy-chart)

[Geomaps](ref:geomaps) and [heatmaps](ref:heatmaps) also have legends, but they only provide the choice to display or not display a legend and don't support other legend options.

## Legend options

You can find the following options under the **Legend** section in the panel edit pane.

{{< admonition type="note" >}}
Not all of the options listed apply to all visualizations with legends.
{{< /admonition >}}

### Visibility

Set whether the legend is displayed or not. Use the switch to toggle a legend on or off.

### Mode

Set the format in which the legend is displayed. Choose from:

- **List**
- **Table**

When you format a legend as a table, other information about the legend, such as associated [values](#values) or where it's located in the visualization, might be displayed as well.

### Placement

Set where on the visualization a legend is displayed. Choose from:

- **Bottom**
- **Right**

### Width

If you set the legend placement to **Right**, the **Width** option becomes available. Leave the field empty to allow Grafana to automatically set the legend width or enter a value in the field.

### Limit

Limit how many series items are shown by default.
When the series count exceeds the limit, click the link to show all items to see the full list of series.

Legend limits are supported for the following visualizations: time series, bar chart, bar gauge, candlestick, histogram, pie chart, state timeline, status history, trend, and XY chart.

### Values

You can add more context to a visualization by adding series data values or [calculations](ref:calculations) to a legend. You can add as many values as you'd like. After you apply your changes, you can scroll the legend to see all values.

![Legend showing values](/media/docs/grafana/panels-visualizations/screenshot-legend-values-10.3.png)

### Series visibility

Toggle the **Series visibility** switch on to add the corresponding filter control next to or above the legend.
This lets you narrow the visible series by name or by label without editing the query, which is useful when a panel renders many series and you want to focus on a subset.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-series-visibility-v13.1.png" max-width="400px" alt="Series visibility filter switch" >}}

After toggling the switch on, click the **Series visibility** icon to open a tooltip that has one or two sections, depending on your dataset:

- **By name**: Lists each unique series name. Select one or more names to limit the visualization to those series.
- **By labels**: Lists each label key with its values. Select label values to filter series that match.

The tooltip also provides the following options:

- **Select all and Deselect all**: Toggle every value in a section.
- **Clear all**: Reset the filter.
- **Pin to sidebar**: Dock the filter alongside the panel so it stays open while you explore.

{{< video-embed src="/media/docs/grafana/panels-visualizations/screenrecord-series-visibility-v13.1.mp4" >}}

{{< admonition type="note" >}}
The series visibility filter is currently only supported in the time series visualization.
{{< /admonition >}}

## Change a series color

By default, Grafana sets the colors of your series data, but you can change them through the panel legend. To change the series data color, follow these steps:

1. Navigate to the panel you want to update.
1. In the legend, click the color bar associated with the series.
1. Select a pre-set color in the **Colors** tab or set a custom color in the **Custom** tab, using the picker or RGB values.
1. Save the dashboard.

![Change legend series color](/static/img/docs/legend/legend-series-color-7-5.png)

## Isolate series data in a visualization

Visualizations can often be visually complex, and include many data series. You can simplify the view by removing series data from the visualization through the legend, which isolates the data you want to see. When you do this, Grafana automatically creates a new override in the **Override** section.

To isolate a series, follow these steps:

1. Navigate to the panel you want to update.
1. In the legend, click the label of the series you want to isolate.

   The system removes all other series data from view.

1. To incrementally add series data back to an isolated series, press the **Ctrl** or **Command** key and click the label of the series you want to add.
1. To save your changes so that they appear to all viewers of the panel, save the dashboard.

To revert back to the default view that includes all data, click any series label twice.

## Sort series

When you format a legend as a table and add values to it, you can sort series in the table by those values. To do so, follow these steps:

1. Navigate to the panel you want to update.
1. Hover over any part of the panel you want to work on to display the menu on the top right corner.
1. Click the menu and select **Edit**.
1. Scroll to the **Legend** section of the panel edit pane.
1. Under **Mode**, select **Table**.
1. Under **Values**, select the value or calculation that you want to show.

   The legend table now displays values.

1. Click the calculation name header in the legend table to sort the values in the table in ascending or descending order.

![Legend formatted as a table showing sorted values](/media/docs/grafana/panels-visualizations/screenshot-legend-sorted-10.3-v2.png)

{{< admonition type="note" >}}
This feature is only supported for the following visualizations: bar chart, histogram, time series.
{{< /admonition >}}

# Configure field overrides

Overrides allow you to customize visualization settings for specific fields or series. When you add an override rule, it targets a particular set of fields and lets you define multiple options for how that field is displayed.

For example, you can override the default unit measurement for all fields that include the text "bytes" by adding an override using the **Fields with name matching regex** matcher and then the **Standard options > Unit** setting to the override rule:

![Field with unit override](/media/docs/grafana/panels-visualizations/screenshot-unit-override-v10.3.png)

After you've set them, your overrides appear in both the **All** and **Overrides** tabs of the panel editor pane:

![All and Overrides tabs of panel editor pane](/media/docs/grafana/panels-visualizations/screenshot-all-overrides-tabs-v11.png)

## Supported visualizations

You can configure field overrides for the following visualizations:

{{< column-list >}}

- [Bar chart](ref:bar-chart)
- [Bar gauge](ref:bar-gauge)
- [Candlestick](ref:candlestick)
- [Canvas](ref:canvas)
- [Gauge](ref:gauge)
- [Geomap](ref:geomap)
- [Heatmap](ref:heatmap)
- [Histogram](ref:histogram)
- [Pie chart](ref:pie-chart)
- [Stat](ref:stat)
- [State timeline](ref:state-timeline)
- [Status history](ref:status-history)
- [Table](ref:table)
- [Time series](ref:time-series)
- [Trend](ref:trend)
- [XY chart](ref:xy-chart)

{{< /column-list >}}

## Override rules

You can choose from five types of override rules, which are described in the following sections.

### Fields with name

Select a field from the list of all available fields. Properties you add to this type of rule are only applied to this single field.

### Fields with name matching regex

Specify fields to override with a regular expression. Properties you add to this type of rule are applied to all fields where the field name matches the regular expression. This override doesn't rename the field; to do this, use the [Rename by regex transformation](ref:rename-by-regex-transformation).

### Fields with type

Select fields by type, such as string, numeric, or time. Properties you add to this type of rule are applied to all fields that match the selected type.

### Fields returned by query

Select all fields returned by a specific query, such as A, B, or C. Properties you add to this type of rule are applied to all fields returned by the selected query.

### Fields with values

Select all fields returned by your defined reducer condition, such as **Min**, **Max**, **Count**, **Total**. Properties you add to this type of rule are applied to all fields returned by the selected condition.

## Examples

The following examples demonstrate how you can use override rules to alter the display of fields in visualizations.

### Example 1: Format temperature

The following result set is a data frame that consists of two fields: time and temperature.

|        time         | temperature |
| :-----------------: | :---------: |
| 2020-01-02 03:04:00 |    45.0     |
| 2020-01-02 03:05:00 |    47.0     |
| 2020-01-02 03:06:00 |    48.0     |

You can apply field options to each field (column) of this structure to alter the way its values are displayed. For example, you can set the following override rule:

- Rule: **Fields with type**
- Field: temperature
- Override property: **Standard options > Unit**
  - Selection: **Temperature > Celsius**

This results in the following table:

|        time         | temperature |
| :-----------------: | :---------: |
| 2020-01-02 03:04:00 |   45.0 °C   |
| 2020-01-02 03:05:00 |   47.0 °C   |
| 2020-01-02 03:06:00 |   48.0 °C   |

In addition, the decimal place isn't required, so you can remove it by adding another override property that changes the **Standard options > Decimals** setting from **auto** to `0`. That results in the following table:

|        time         | temperature |
| :-----------------: | :---------: |
| 2020-01-02 03:04:00 |    45 °C    |
| 2020-01-02 03:05:00 |    47 °C    |
| 2020-01-02 03:06:00 |    48 °C    |

### Example 2: Format temperature and humidity

The following result set is a data frame that consists of four fields: time, high temp, low temp, and humidity.

| time                | high temp | low temp | humidity |
| ------------------- | --------- | -------- | -------- |
| 2020-01-02 03:04:00 | 45.0      | 30.0     | 67       |
| 2020-01-02 03:05:00 | 47.0      | 34.0     | 68       |
| 2020-01-02 03:06:00 | 48.0      | 31.0     | 68       |

Use the following override rule and properties to add the **Celsius** unit option and remove the decimal place:

- Rule: **Fields with type**
- Field: temperature
- Override property: **Standard options > Unit**
  - Selection: **Temperature > Celsius**
- Override property: **Standard options > Decimals**
  -Change setting from **auto** to `0`

This results in the following table:

| time                | high temp | low temp | humidity |
| ------------------- | --------- | -------- | -------- |
| 2020-01-02 03:04:00 | 45 °C     | 30 °C    | 67 °C    |
| 2020-01-02 03:05:00 | 47 °C     | 34 °C    | 68 °C    |
| 2020-01-02 03:06:00 | 48 °C     | 31 °C    | 68 °C    |

The temperature fields are displaying correctly, but the humidity has incorrect units. You can fix this by applying a **Misc > Percent (0-100)** override to the humidity field. This results in the following table:

| time                | high temp | low temp | humidity |
| ------------------- | --------- | -------- | -------- |
| 2020-01-02 03:04:00 | 45 °C     | 30 °C    | 67%      |
| 2020-01-02 03:05:00 | 47 °C     | 34 °C    | 68%      |
| 2020-01-02 03:06:00 | 48 °C     | 31 °C    | 68%      |

## Add a field override

To add a field override, follow these steps:

1. Navigate to the panel to which you want to add the data link.
1. Hover over any part of the panel to display the menu icon in the upper-right corner.
1. Click the menu icon and select **Edit** to open the panel editor.
1. At the bottom of the panel editor pane, click **Add field override**.
1. Select the fields to which the override will be applied:
   - **Fields with name**
   - **Fields with name matching regex**
   - **Fields with type**
   - **Fields returned by query**
   - **Fields with values**
1. Click **Add override property**.
1. Select the field option that you want to apply.
1. Continue to add overrides to this field by clicking **Add override property**.
1. Add as many overrides as you need.
1. When you're finished, click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Back** and then **Exit edit**.

## Edit a field override

To edit a field override, follow these steps:

1. Navigate to the panel to which you want to add the data link.
1. Hover over any part of the panel to display the menu icon in the upper-right corner.
1. Click the menu icon and select **Edit** to open the panel editor.
1. In the panel editor pane, click the **Overrides** tab.
1. Locate the override you want to change.
1. Perform any of the following tasks:
   - Edit settings on existing overrides or field selection parameters.
   - Delete existing override properties by clicking the **X** next to the property.
   - Delete an override entirely by clicking the trash icon at the top-right corner.
1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Back** and then **Exit edit**.

The changes you make take effect immediately.

# Configure panel options

There are settings common to all visualizations, which you set in the **Panel options** section of the panel editor pane. The following sections describe these options as well as how to set them.

## Panel options

Set the following options to provide basic information about a panel and define basic display elements:

<!-- prettier-ignore-start -->

| Option                                       | Description                                                                           |
| -------------------------------------------- | ------------------------------------------------------------------------------------- |
| Title                                        | Text entered in this field appears at the top of your panel in the panel editor and in the dashboard. You can use [variables you have defined](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/) or [built-in variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/built-in-variables). |
| [Description](#description-field-formatting) | Add a description to a panel to share with users any important information about it, such as its purpose. Text entered in this field appears in a tooltip in the upper-left corner of the panel. Conversely, toggle on the **As subtitle** switch to display the description just below the panel title instead. You can use [variables you have defined](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/) or dashboard-scoped [built-in variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/built-in-variables) (such as `$__from`, `$__to`, `$__dashboard`, `$__org`, and `$__user`). Query-scoped built-in variables (such as `$__interval` and `$__interval_ms`) are not available in this field. |
| Transparent background                       | Toggle this switch on and off to control whether or not the panel has the same background color as the dashboard. | 
| Panel links | Add [links to the panel](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/manage-dashboard-links/#panel-links) to create shortcuts to other dashboards, panels, and external websites. Access panel links by clicking the icon next to the panel title. |
| Repeat options                               | Set whether to repeat the panel for each value in the selected variable. For more information, refer to [Configure repeating panels](#configure-repeating-panels).|
| Show/hide rules                               | Set whether to show or hide panels based on various rules. For more information, refer to [Show/hide rules](#showhide-rules).|

<!-- prettier-ignore-end -->

You can use generative AI to populate the **Title** and **Description** fields with the [Grafana LLM plugin](https://grafana.com/docs/grafana-cloud/alerting-and-irm/machine-learning/configure/llm-plugin/), which is currently in public preview.
To enable this, refer to [Set up generative AI features for dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/manage-dashboards/#set-up-generative-ai-features-for-dashboards).

### Description field formatting

Grafana renders the **Description** field as [GitHub Flavored Markdown](https://github.github.com/gfm/) (GFM). You can use common Markdown and GFM features, including:

- **Emphasis**: `**bold**`, `_italic_`, and `~~strikethrough~~`
- **Structure**: headings, paragraphs, blockquotes, horizontal rules, bulleted and numbered lists
- **Code**: `` `inline code` `` and fenced code blocks
- **Links and media**: `[link text](https://example.com)` and images `![alt](https://example.com/image.png)`
- **Tables**: GFM-style pipe tables

The description appears in a tooltip, so short copy and basic formatting work best.
To break text into separate paragraphs, use an empty line between them, or use a list.

Grafana sanitizes the rendered HTML to reduce security risk and strips out or restricts arbitrary HTML, scripts, and many embedded elements, similar to Markdown in the [Text](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/text/) visualization.

## Configure repeating panels

You can configure Grafana to dynamically add panels to a dashboard.
A dynamic panel is a panel that the system creates based on the value of a multi-value variable.
Variables dynamically change your queries across all panels in a dashboard.

To see an example of repeating panels, refer to [this dashboard with repeating panels](https://play.grafana.org/d/000000025/templating-repeated-panels?orgId=1).

For information about repeating rows and tabs, refer to [Configure repeat options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/#configure-repeat-options).

**Before you begin:**

- Ensure that the query includes a multi-value variable.

To configure repeating panels, follow these steps:

{{< docs/list >}}

1. Go to the dashboard you want to update and navigate to the panel you want to repeat.

   If the dashboard is large, open the **Content outline** and use it to navigate to the part of the dashboard you want to update.

1. Click **Edit**.
1. Click the panel to open the sidebar.

{{% shared-snippet path="/docs/grafana/latest/visualizations/dashboards/build-dashboards/create-dashboard/_index.md" id="configure-repeat" %}}

1. To propagate changes to all panels, reload the dashboard.

{{< /docs/list >}}

You can stop a panel from repeating by selecting **Disable repeating** in the **Repeat by variable** drop-down list.

### Limitations of repeating panels

Panels generated by the repeat feature have the following limitations:

- **Generated panels can't be edited**: Only the source panel is editable and changes you make in the source panel editor apply to all generated copies.
- **Layout changes don't persist**: Each time the dashboard loads or refreshes, Grafana regenerates repeated panels from the source panel. Any layout changes you make to the generated copies are overwritten on reload.
- **You can't mix repeated panels**: You can't mix repeated panels with other panels on the same row when using the horizontal repeat direction.

If you need each panel to have a unique, stable layout position or individual settings, disable the repeat option and create separate panels manually instead.

## Show/hide rules

You can configure panels to be shown or hidden based on rules.
For example, you can set a panel to be hidden if there's no data returned by a query or to only be shown if a specific variable value is present.

{{% shared-snippet path="/docs/grafana/latest/visualizations/dashboards/build-dashboards/create-dashboard/_index.md" id="show-hide-1" %}}

### Query result rule

{{% shared-snippet path="/docs/grafana/latest/visualizations/dashboards/build-dashboards/create-dashboard/_index.md" id="show-hide-2" %}}

### Template variable rule

Show or hide a panel dynamically based on a variable value.
You can select any variable that's configured for the dashboard and choose from the following operators for maximum flexibility:

- Equals
- Not equals
- Matches (regular expression values)
- Not matches (regular expression values)

You can [add more variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/dashboard-controls/#add-variables) if you need to without leaving the dashboard.

### Time range less than rule

Show or hide a panel if the dashboard time range is shorter than the selected time range.
This ensures that as you change the time range of the dashboard, you only see data relevant to that time period.

{{% shared-snippet path="/docs/grafana/latest/visualizations/dashboards/build-dashboards/create-dashboard/_index.md" id="show-hide-3" %}}

### Configure show/hide rules

To configure a show/hide rule, follow these steps:

1. Navigate to the dashboard you want to update.
1. Click **Edit**.
1. Click the panel you want to update to open the sidebar, or click the **Dashboard options** icon to open it.

   If the dashboard is large, open the **Content outline** and use it to navigate to the part of the dashboard you want to update.

1. Expand the **Show / hide rules** section.
1. Select **Show** or **Hide** to set whether the panel is shown or hidden based on the rules outcome.
1. Click **+ Add rule**.
1. Select a rule type:
   - **Query result**: Show or hide a panel based on query results. Choose from **Has data** and **No data**.
   - **Template variable**: Show or hide the panel dynamically based on the variable value. Select a variable and operator, and enter a value.
   - **Time range less than**: Show or hide the panel if the dashboard time range is shorter than the selected time range. Select a time range from **5 minutes** to **5 years**.

1. If you've configured multiple rules, the **Match rules** option displays. Select one of the following:
   - **Match all**: The panel is shown or hidden only if _all_ the rules are matched.
   - **Match any**: The panel is shown or hidden if _any_ of the rules are matched.

1. When you've finished setting rules, save the dashboard.
1. (Optional) Enter a description of the changes you've made.
1. Click **Save** again.
1. Click **Exit edit**

Hidden panels aren't visible when the dashboard is in view mode.
In edit mode, hidden dashboard panels are displayed with an icon or overlay indicating this.

# Configure standard options

**Standard options** in the panel editor pane let you change how field data is displayed in your visualizations. Options that you apply don't change the data, they just change how Grafana _displays_ the data.

When you set a standard option, the change is applied to all fields or series. For example, if you set the **Unit** option to **Percentage**, all fields with numeric values are displayed as percentages.

For more granular control over the display of fields, refer to [Configure overrides](ref:configure-overrides).

## Supported visualizations

You can configure standard options for the following visualizations:

{{< column-list >}}

- [Bar chart](ref:bar-chart)
- [Bar gauge](ref:bar-gauge)
- [Candlestick](ref:candlestick)
- [Canvas](ref:canvas)
- [Gauge](ref:gauge)
- [Geomap](ref:geomap)
- [Histogram](ref:histogram)
- [Pie chart](ref:pie-chart)
- [Stat](ref:stat)
- [State timeline](ref:state-timeline)
- [Status history](ref:status-history)
- [Table](ref:table)
- [Time series](ref:time-series)
- [Trend](ref:trend)
- [XY chart](ref:xy-chart)

{{< /column-list >}}

## Standard options

This section explains all available standard options.

To set these options, expand the **Standard options** section in the panel editor pane. Most field options won't affect the visualization until you click outside of the field option box you're editing or press Enter.

{{< admonition type="note" >}}
Not all of the options listed apply to all visualizations with standard options.
{{< /admonition >}}

### Unit

This option lets you choose which unit a field should use. Click in the **Unit** field, then drill down until you find the unit you want. The unit you select is applied to all fields except time.

#### Custom units

You can also use the **Unit** drop-down to specify custom units, custom prefixes or suffixes, and date time formats.

To set a custom unit, enter the unit you want to use and then select it in the drop-down. It'll be the last option listed. For example, if you enter a unit called "Hearts", the drop-down will then include the option **Custom unit: Hearts**.

You can further define a custom unit with specific syntax. For example, to set a custom currency unit called "Gems", enter `currency:Gems` in the field. The drop-down will include the option **Custom unit: currency:Gems**:

![A custom currency unit called Gems in the Unit drop-down](/media/docs/grafana/panels-visualizations/custom_unit_currency_v11.0.png)

The following table lists the special syntax options for custom units:

| Custom unit                        | Description                                                                                                                                                                                                                                                                                                                                             |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `suffix:<suffix>`                  | Custom unit that should go after value.                                                                                                                                                                                                                                                                                                                 |
| `prefix:<prefix>`                  | Custom unit that should go before value.                                                                                                                                                                                                                                                                                                                |
| `time:<format>`                    | Custom date time formats type, such as `time:YYYY-MM-DD`. Refer to [formats](https://momentjs.com/docs/#/displaying/) for the format syntax and options.                                                                                                                                                                                                |
| `si:<base scale><unit characters>` | Custom SI units, such as `si: mF`. You can specify both a unit and the source data scale. For example, if your source data is represented as milli-something, prefix the unit with the `m` SI scale character.                                                                                                                                          |
| `count:<unit>`                     | Custom count unit.                                                                                                                                                                                                                                                                                                                                      |
| `currency:<unit>`                  | Custom currency unit.                                                                                                                                                                                                                                                                                                                                   |
| `currency:financial:<unit>`        | Full format currency unit without abbreviations. Displays complete numeric values instead of scaled abbreviations (K: Thousand, M: Million, B: Billion, T: Trillion). For example, `currency:financial:$` displays `500,555` instead of `$501K`. Add `:suffix` to place the symbol after the number: `currency:financial:€:suffix` displays `500,555€`. |

You can also paste a native emoji in the **Unit** drop-down and select it as a custom unit:

![A thumbs up emoji as a custom unit](/media/docs/grafana/panels-visualizations/custom_unit_thumbsup_v11.0.png)

![A time series visualization using custom thumbs up emoji units](/media/docs/grafana/panels-visualizations/thumbsup_panel_v11.0.png)

##### Time format units

All **Date & time** format units in Grafana (such as **Datetime ISO** or **Datetime US**) expect input values to be in milliseconds since the Unix epoch (January 1, 1970). If your data source provides timestamps in seconds, these will be incorrectly interpreted as dates very close to January 1, 1970.

To display timestamps that are in seconds since epoch, multiply your timestamp values by 1000 using a transformation following these steps:

1. In the panel editor, click the **Transformations** tab.
1. Click **Add transformation**.
1. Select the **Add field from calculation** transformation.
1. Set the following options:
   - **Mode** - **Binary operation**
   - **Operation**
     - Select your timestamp field
     - Select the asterisk (`*`) for multiply by
     - Enter 1000 in the **Field or Number** field
   - Toggle the **Replace all fields** switch on if you want to see the calculated field.

#### Control unit scaling

By default, Grafana automatically scales the unit based on the magnitude of the value. For example, if you have values of 0.14kW and 3000kW, Grafana displays them as 140W and 3MW, respectively. You can use custom units to control this behavior by setting a prefix, suffix, or custom SI unit.

#### String units

Sometimes Grafana is too aggressive in interpreting strings and displaying them as numbers. To configure Grafana to show the original string value, select **Misc > String** in the **Unit** drop-down.

### Min

Set the minimum value used in percentage threshold calculations. Leave this field empty to automatically calculate the minimum.

### Max

Set the maximum value used in percentage threshold calculations. Leave this field empty to automatically calculate the maximum.

### Field min/max

By default, the calculated **Min** and **Max** are based on the minimum and maximum of all series and fields. When you enable **Field min/max**, Grafana calculates the min or max of each field individually, based on the minimum or maximum value of the field.

### Decimals

Specify the number of decimals Grafana includes in the rendered value. If you leave this field empty, Grafana automatically truncates the number of decimals based on the value. For example 1.1234 displays as 1.12 and 100.456 displays as 100.

To display all decimals, set the unit to **String**.

### Display name

Set the display title of all fields. You can use [variables](ref:variables) in the field title.

When multiple stats, fields, or series are displayed, this field controls the title in each stat. You can use expressions like `${__field.name}` to use only the series name or the field name in the title.

The following table shows examples of the different field names generated using various expressions. In this example, there's a field with a name of "Temp" and labels of {"Loc"="PBI", "Sensor"="3"}:

| Expression syntax            | Example                 | Renders to                     | Explanation                                                                                                                                                                                                        |
| ---------------------------- | ----------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `${__field.displayName}`     | Same as syntax          | `Temp {Loc="PBI", Sensor="3"}` | Displays the field name, and labels in `{}` if they are present. If there is only one label key in the response, then for the label portion, Grafana displays the value of the label without the enclosing braces. |
| `${__field.name}`            | Same as syntax          | `Temp`                         | Displays the name of the field (without labels).                                                                                                                                                                   |
| `${__field.labels}`          | Same as syntax          | `Loc="PBI", Sensor="3"`        | Displays the labels without the name.                                                                                                                                                                              |
| `${__field.labels.X}`        | `${__field.labels.Loc}` | `PBI`                          | Displays the value of the specified label key.                                                                                                                                                                     |
| `${__field.labels.__values}` | Same as Syntax          | `PBI, 3`                       | Displays the values of the labels separated by a comma (without label keys).                                                                                                                                       |

If the value is an empty string after rendering the expression for a particular field, then the default display method is applied.

### Color scheme

The **Color scheme** options let you set single or multiple colors for your entire visualization.

The color options and their effect on a visualization depend on the visualization you're working with and some visualizations have different color options.

Select one of the following schemes:

| Color scheme                          | Description                                                                                                                                                                                                                                                                                                                                                                                                 |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Single color                          | Specifies a single color.                                                                                                                                                                                                                                                                                                                                                                                   |
| Shades of a color                     | Grafana selects shades of a single color.                                                                                                                                                                                                                                                                                                                                                                   |
| From thresholds (by value)            | The color is taken from the matching [threshold](ref:threshold). For some visualizations, you also need to choose if the color is set by the **Last**, **Min**, or **Max** value of the field or series.                                                                                                                                                                                                    |
| Classic palette                       | Grafana automatically assigns a color for each field or series based on its order. If the order of a field changes in your query, the color also changes. Useful for graphs, pie charts, and other categorical data visualizations.                                                                                                                                                                         |
| Classic palette (by series name)      | Grafana automatically assigns colors based on the name of the series. Useful when the series names to be visualized can change based on the available data.                                                                                                                                                                                                                                                 |
| Multiple continuous colors (by value) | Grafana automatically assigns colors based on the percentage of a value relative to the min and the max of the field or series. For some visualizations, you also need to choose if the color is set by the **Last**, **Min**, or **Max** value of the field or series. Select from: **Green-Yellow-Red**, **Red-Yellow-Green**, **Blue-Yellow-Red**, **Yellow-Red**, **Blue-Purple**, and **Yellow-Blue**. |
| Single continuous color (by value)    | Grafana automatically assigns shades of one color based on the percentage of a value relative to the min and the max of the field or series. For some visualizations, you also need to choose if the color is set by the **Last**, **Min**, or **Max** value of the field or series. Select from: **Blues**, **Reds**, **Greens**, and **Purples**.                                                         |

You can also use the legend to open the color picker by clicking the legend series color icon. Setting color this way automatically creates an override rule that set's a specific color for a specific series.

### No value

Enter what Grafana should display if the field value is empty or null. The default value is a hyphen (-).

# Configure thresholds

In dashboards, a threshold is a value or limit you set for a metric that's reflected visually when it's met or exceeded. Thresholds are one way you can conditionally style and color your visualizations based on query results.

Using thresholds, you can color grid lines and regions in a time series visualization:
![Time series visualization with green, blue, and purple threshold lines and regions](/media/docs/grafana/panels-visualizations/screenshot-thresholds-lines-regions-v10.4.png)

You can color the background or value text in a stat visualization:
![Stat visualization with three values in green and orange](/media/docs/grafana/panels-visualizations/screenshot-thresholds-value-v10.4.png)

You can define regions and region colors in a state timeline:
![State timeline with green, blue, and pink region thresholds](/media/docs/grafana/panels-visualizations/screenshot-thresholds-state-timeline-v10.4.png)

You can also use thresholds to:

- Color lines in a time series visualization
- Color the gauge and threshold markers in a gauge
- Color markers in a geomap
- Color cell text or background in a table

{{< docs/play title="Threshold example" url="https://play.grafana.org/d/000000167/" >}}

## Supported visualizations

You can set thresholds in the following visualizations:

{{< column-list >}}

- [Bar chart](ref:bar-chart)
- [Bar gauge](ref:bar-gauge)
- [Candlestick](ref:candlestick)
- [Canvas](ref:canvas)
- [Gauge](ref:gauge)
- [Geomap](ref:geomap)
- [Histogram](ref:histogram)
- [Stat](ref:stat)
- [State timeline](ref:state-timeline)
- [Status history](ref:status-history)
- [Table](ref:table)
- [Time series](ref:time-series)
- [Trend](ref:trend)

{{< /column-list >}}

## Default thresholds

On visualizations that support thresholds, Grafana has the following default threshold settings:

- 80 = red
- Base = green
- Mode = Absolute
- Show thresholds = Off (for some visualizations); for more information, see the [Show thresholds](#show-thresholds) option.

## Thresholds options

You can set the following options to further define how thresholds look.

### Threshold value

This number is the value that triggers the threshold. You can also set the color associated with the threshold in this field.

The **Base** value represents minus infinity. By default, it's set to the color green, which is generally the “good” color.

### Thresholds mode

There are two threshold modes:

- **Absolute** thresholds are defined by a number. For example, 80 on a scale of 1 to 150.
- **Percentage** thresholds are defined relative to minimum or maximum. For example, 80 percent.

### Show thresholds

{{< admonition type="note" >}}
This option is supported for the bar chart, candlestick, time series, and trend visualizations.
{{< /admonition>}}

Set if and how thresholds are shown with the following options.

| Option                               | Example                                                                                                                                                                                              |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Off                                  |                                                                                                                                                                                                      |
| As lines                             | {{< figure max-width="500px" src="/media/docs/grafana/panels-visualizations/screenshot-thresholds-lines-v10.4.png" alt="Visualization with threshold as a line" >}}                                  |
| As lines (dashed)                    | {{< figure max-width="500px" src="/media/docs/grafana/panels-visualizations/screenshot-thresholds-dashed-lines-v10.4.png" alt="Visualization with threshold as a dashed line" >}}                    |
| As filled regions                    | {{< figure max-width="500px" src="/media/docs/grafana/panels-visualizations/screenshot-thresholds-regions-v10.4.png" alt="Visualization with threshold as a region" >}}                              |
| As filled regions and lines          | {{< figure max-width="500px" src="/media/docs/grafana/panels-visualizations/screenshot-thresholds-lines-regions-v10.4.png" alt="Visualization with threshold as a region and line" >}}               |
| As filled regions and lines (dashed) | {{< figure max-width="500px" src="/media/docs/grafana/panels-visualizations/screenshot-thresholds-dashed-lines-regions-v10.4.png" alt="Visualization with threshold as a region and dashed line" >}} |

## Add a threshold

You can add as many thresholds to a visualization as you want. Grafana automatically sorts thresholds values from highest to lowest.

1. Navigate to the panel you want to update.
1. Hover over any part of the panel you want to work on to display the menu on the top right corner.
1. Click the menu and select **Edit**.
1. Scroll to the **Thresholds** section or enter `thresholds` in the search bar at the top of the panel edit pane.
1. Click **+ Add threshold**.
1. Enter a new threshold value or use the up and down arrows at the right side of the field to increase or decrease the value incrementally.
1. Click the colored circle to the left of the threshold value to open the color picker, where you can update the threshold color.
1. Under **Thresholds mode**, select either **Absolute** or **Percentage**.
1. Under **Show thresholds**, set how the threshold is displayed or turn it off.
1. Click **Save** in the top-right corner.
1. Enter an optional description of your changes and click **Save**.
1. Click **Back** and then **Exit edit**.

To delete a threshold, navigate to the panel that contains the threshold and click the trash icon next to the threshold you want to remove.

# Configure tooltips

When you hover your cursor over a visualization, Grafana can display tooltips that contain more information about a data point, like the exact time of a result. You can customize tooltips to control how many series they include and the order of those values. You can also copy the content from tooltips to use elsewhere. Learn more about configuring tooltips in [Tooltip options](#tooltip-options).

## Supported visualizations

You can configure tooltips for the following visualizations:

{{< column-list >}}

- [Bar chart](ref:bar-chart)
- [Candlestick](ref:candlestick)
- [Heatmap](ref:heatmap)
- [Pie chart](ref:pie-chart)
- [State timeline](ref:state-timeline)
- [Status history](ref:status-history)
- [Time series](ref:time-series)
- [Trend](ref:trend)
- [XY chart](ref:xy-chart)

{{< /column-list >}}

Some visualizations, for example [candlestick](ref:candlestick) and [flame graph](ref:flame-graph), have tooltips, but they aren't configurable. These visualizations don't have a **Tooltip** section in the panel editor pane. [Geomaps](ref:geomaps) provide you the option to have tooltips triggered upon click or hover under the **Map controls** options in the panel editor pane.

<!-- if we add documentation for treemap, some info will need to be added in the paragraph above -->

## Tooltip options

You can find the following options under the **Tooltip** section in the panel edit pane.

{{< admonition type="note" >}}
Not all of the options listed apply to all visualizations with tooltips.
{{< /admonition >}}

### Tooltip mode

Choose how tooltips behave with the following options:

- **Single** - The tooltip only the single series that you're hovering over in the visualization.
- **All** - The tooltip shows all series in the visualization. Grafana highlights the series that you are hovering over in bold in the series list in the tooltip.
- **Hidden** - Tooltips aren't displayed when you interact with the visualization.

You can use a [field override](ref:field-override) to hide individual series from the tooltip.

### Values sort order

When you set the **Tooltip mode** to **All**, the **Values sort order** option is displayed. This option controls the order in which values are listed in a tooltip. Choose from the following:

- **None** - Grafana automatically sorts the values displayed in a tooltip.
- **Ascending** - Values in the tooltip are listed from smallest to largest.
- **Descending** - Values in the tooltip are listed from largest to smallest.

### Hide zeros

When you set the **Tooltip mode** to **All**, the **Hide zeros** option is displayed. This option controls whether or not series with `0` values are shown in the list in the tooltip.

### Hover proximity

Set the hover proximity (in pixels) to control how close the cursor must be to a data point to trigger the tooltip to display.

![Adding a hover proximity limit for tooltips](/media/docs/grafana/gif-grafana-10-4-hover-proximity.gif)

### Max width

Set the maximum width of the tooltip box.

### Max height

Set the maximum height of the tooltip box. The default is 600 pixels.

### Show histogram (Y axis)

For the heatmap visualization only, when you set the **Tooltip mode** to **Single**, the **Show histogram (Y axis)** option is displayed. This option controls whether or not the tooltip includes a histogram representing the y-axis.

### Show color scale

For the heatmap visualization only, when you set the **Tooltip mode** to **Single**, the **Show color scale** option is displayed. This option controls whether or not the tooltip includes the color scale that's also represented in the legend. When the color scale is included in the tooltip, it shows the hovered value on the scale:

![Heatmap with a tooltip displayed showing the hovered value reflected in the color scale](/media/docs/grafana/panels-visualizations/screenshot-heatmap-tooltip-color-scale-v11.0.png)

# Configure value mappings

In addition to field overrides, value mapping is a technique you can use to change how data appears in a visualization.

For example, the mapping applied in the following image causes the visualization to display the text `Cold`, `Good`, and `Hot` in blue, green, and red for ranges of temperatures rather than actual temperature values. Using value mappings this way can make data faster and easier to understand and interpret.

![Value mappings applied to a gauge visualization](/media/docs/grafana/panels-visualizations/screenshot-value-mappings-v10.4.png)

Value mappings bypass unit formatting set in the **Standard options** section of panel editor, like color or number of decimal places displayed. When value mappings are present in a panel, Grafana displays a summary of them in the **Value mappings** section of the editor panel.

## Supported visualizations

You can configure value mappings for the following visualizations:

{{< column-list >}}

- [Bar chart](ref:bar-chart)
- [Bar gauge](ref:bar-gauge)
- [Candlestick](ref:candlestick)
- [Canvas](ref:canvas)
- [Gauge](ref:gauge)
- [Geomap](ref:geomap)
- [Histogram](ref:histogram)
- [Pie chart](ref:pie-chart)
- [Stat](ref:stat)
- [State timeline](ref:state-timeline)
- [Status history](ref:status-history)
- [Table](ref:table)
- [Time series](ref:time-series)
- [Trend](ref:trend)

{{< /column-list >}}

## Types of value mappings

Grafana supports the following value mapping types:

### Value

A **Value** mapping maps specific values to text and a color. For example, you can configure a mapping so that all instances of the value `10` appear as **Perfection!** rather than the number. Use **Value** mapping when you want to format a single value.
![The value 10 mapped to the text Perfection!](/media/docs/grafana/panels-visualizations/screenshot-map-value-v10.4.png)

### Range

A **Range** mapping maps numerical ranges to text and a color. For example, if a value is within a certain range, you can configure a range value mapping to display **Low** or **High** rather than the number. Use **Range** mapping when you want to format multiple, continuous values.
![Ranges of numbers mapped to the text Low and High with colors yellow and red](/media/docs/grafana/panels-visualizations/screenshot-map-range-v10.4.png)

### Regex

A **Regex** mapping maps regular expressions to text and a color. For example, if a value is `www.example.com`, you can configure a regular expression value mapping so that Grafana displays **www** and truncates the domain. Use the **Regex** mapping when you want to format the text and color of a regular expression value.
![A regular expression used to truncate full URLs to the text wwww](/media/docs/grafana/panels-visualizations/screenshot-map-regex-v10.4.png)

{{< admonition type="note" >}}
The character `.` doesn't match newline characters `\n` and `\r` by default.
If your data contains multiline strings, use `[\s\S]` in place of `.` to match any character including newlines.
For more detailed information, refer to the [Multiline regular expression example](#multiline-regular-expression-example).
{{< /admonition >}}

### Special

A **Special** mapping maps special values like `Null`, `NaN` (not a number), and boolean values like `true` and `false` to text and color. For example, you can configure a special value mapping so that `null` values appear as **N/A**. Use the **Special** mapping when you want to format uncommon, boolean, or empty values.
![The value null mapped to the text N/A](/media/docs/grafana/panels-visualizations/screenshot-map-special-v10.4.png)

## Examples

Refer to the following examples to learn more about value mapping.

### Time series example

The following image shows a time series visualization with value mappings. Value mapping colors aren't applied to this visualization, but the display text is shown on the axis.

![Value mappings time series example](/static/img/docs/value-mappings/value-mappings-summary-example-8-0.png)

### Stat example

The following image shows a stat visualization with value mappings and text colors applied. You can hide the sparkline so it doesn't interfere with the values.

![Value mappings stat example](/static/img/docs/value-mappings/value-mappings-stat-example-8-0.png)

### Bar gauge example

The following image shows a bar gauge visualization with value mappings. Note that the value mapping colors are applied to the text, but not to the gauges.

![Value mappings bar gauge example](/static/img/docs/value-mappings/value-mappings-bar-gauge-example-8-0.png)

### Table example

The following image shows a table visualization with value mappings. If you want value mapping colors displayed on the table, then set the cell display mode to **Color text** or **Color background**.

![Value mappings table example](/static/img/docs/value-mappings/value-mappings-table-example-8-0.png)

### Multiline regular expression example

If your data source returns multiline strings—such as Windows event log entries that contain `\r\n` line breaks—a regular expression pattern using `.` won't match across those line breaks. Use `[\s\S]` to match any character including newlines.

| Pattern   | Matches                        |
| --------- | ------------------------------ |
| `.*`      | Single-line text only          |
| `[\s\S]*` | Single-line and multiline text |

For example, if you wanted to map any multiline log entry that starts with `ERROR` to the display text **Error**, using `ERROR.*` would only match text up to the first newline and would fail to map the full entry.

Instead, you should use the pattern:

```text
ERROR[\s\S]*
```

## Add a value mapping

1. Navigate to the panel you want to update.
1. Hover over any part of the panel you want to work on to display the menu on the top right corner.
1. Click the menu and select **Edit**.
1. Scroll to the **Value mappings** section and expand it.
1. Click **Add value mappings**.
1. Click **Add a new mapping** and then select one of the following:
   - **Value** - Enter a single value to match.
   - **Range** - Enter the beginning and ending values of a range to match.
   - **Regex** - Enter a regular expression pattern to match.
   - **Special** - Select a special value to match.

1. (Optional) Enter display text.
1. (Optional) Set the color.
1. (Optional) Set an icon (canvas visualizations only).
1. Click **Update** to save the value mapping.

After you've added a mapping, the **Edit value mappings** button replaces the **Add value mappings** button. Click the edit button to add or update mappings.

1. Click **Save dashboard**.
1. Click **Back** and then **Exit edit**.

# Panel editor

In the panel editor, you can update all the elements of a visualization including the data source, queries, time range, and visualization display options.

![Panel editor](/media/docs/grafana/panels-visualizations/screenshot-panel-editor-2-v12.4.png)

This following sections describe the areas of the Grafana panel editor.

## Panel header

The header section lists the dashboard in which the panel appears and the following controls:

- **Back** - Return to the dashboard with changes applied, but not yet saved.
- **Discard** - Discard changes you have made to the panel since you last saved the dashboard.
- **Save** - Save your changes to the dashboard.

## Visualization preview

The visualization preview section contains the following options:

- **Table view** - Convert any visualization to a table so you can see the data. Table views are helpful for troubleshooting. This view only contains the raw data. It doesn't include transformations you might have applied to the data or the formatting options available in the [table](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/visualizations/table/) visualization.
- **Time range controls** - **Default** is either the browser local timezone or the timezone selected at a higher level.
- **Refresh** - Query the data source.

## Data section

The data section contains tabs where you enter queries, transform your data, and create alert rules (if applicable).

- **Queries**
  - Select your data source. You can also set or update the data source in existing dashboards using the drop-down menu in the **Queries** tab.
  - **Saved queries**:
    - **Save query** - To [save the query](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/#save-a-query) for reuse, click the **Save query** button (or icon).
    - **Replace query** - Reuse a saved query.
  - **+ Add query** - Add an additional query.
  - **+ Add from saved queries** - Add an additional query by reusing a saved query.

  {{< admonition type="note" >}}
  [Saved queries](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/#saved-queries) is only available on Grafana Enterprise and Grafana Cloud.
  {{< /admonition >}}

- **Transformations** - Apply data transformations. For more information, refer to [Transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/transform-data/).
- **Alert** - Write alert rules. For more information, refer to [the overview of Grafana Alerting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/).

## Panel display options

The display options section contains tabs where you configure almost every aspect of your data visualization.

When you first open the panel editor, Grafana analyzes your query results and suggests visualizations that suit the shape of the data. You can select a suggested visualization or click **All visualizations** to browse the full list of available panel types. After you've selected a visualization, the display options section provides further configuration.

For more information about individual visualizations, refer to [Visualizations options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/visualizations/).

### Panel styles

While visualization suggestions help you choose _which_ panel type to use, panel styles help you decide _how_ that panel should look. The **Panel styles** section of the panel editor sidebar contains preconfigured options for the currently selected visualization. It appears after you've selected a visualization and the panel has data.

![Panel styles example for time series visualization](/media/docs/grafana/panels-visualizations/screenshot-visualization-presets-v13.0.png)

Each style is displayed as a live preview card that shows how it changes the visualization. Clicking a style applies a combination of display options and field configuration to the panel. For example, changing a style might switch a time series visualization from a line chart with a gradient fill to a stacked bar chart, or update the color scheme and graph mode of a stat visualization.

Panel styles merge their settings with the panel's existing defaults as follows:

- Only the fields the style defines are changed.
- Styles can modify thresholds and color settings—any preset that modifies thresholds displays a badge on the preview card.
- Styles don't affect field overrides

You can further customize the panel after applying a style.

Panel styles are available for the following visualizations:

{{< column-list >}}

- Bar chart
- Bar gauge
- Candlestick
- Gauge
- Heatmap
- Histogram
- Pie chart
- Stat
- State timeline
- Status history
- Time series
- Trend
- XY chart

{{< /column-list >}}

However, plugin authors can add panel styles support to their own panel plugins as well.

### Copy and paste panel styles

You can duplicate the styles of one panel in another panel with the same visualization type.
Use the **Copy styles** and **Paste styles** options to do so:

{{< video-embed src="/media/docs/grafana/panels-visualizations/screenrecord-copy-paste-styles.mp4" >}}

To copy and paste a panel style, follow these steps:

1. Navigate to the dashboard you want to update.
1. Click **Edit**.
1. Navigate to the panel with the style you want to copy.
1. Hover over any part of the panel you want to work on to display the menu on the top right corner.
1. Click the menu and select **Styles > Copy styles**.
1. Navigate to the panel on which you want to copy the style.
1. Click the panel menu and select **Styles > Paste styles**.
1. Save your changes.

{{< admonition type="note" >}}
Copying and pasting styles doesn't copy query information or field overrides.
{{< /admonition >}}

# The panel inspect view

The panel inspect view, which you can open from the panel menu, helps you understand and troubleshoot your panels.
You can inspect the raw data for any Grafana panel, export that data to a comma-separated values (CSV) file, view query requests, review errors and notices, and export panel and data JSON.
You can also resize the drawer.

The panel inspector includes the following tabs:

- **Data**: Shows the raw data returned by the query with transformations applied. Field options such as overrides and value mappings are not applied by default.
- **Stats**: Shows how long your query takes and how much it returns.
- **Query**: Shows you the requests to the server sent when Grafana queries the data source.
- **Error and notices**: Shows errors and notices. Only visible when query returns an error or notice.
- **JSON**: Allows you to view and copy the panel JSON, panel data JSON, and data frame structure JSON. This is useful if you are provisioning or administering Grafana.

Not all panel types include all tabs.
For example, dashboard list panels don't have raw data to inspect, so they don't display the **Stats**, **Data**, or **Query** tabs.

## Download raw query results

Grafana generates a CSV file that contains your data, including any transformations to that data.
You can choose to view the data before or after the panel applies field options or field option overrides.

1. Edit the panel that contains the query data you want to download.
1. In the query editor, click **Query inspector**.
1. Click the **Data** tab.

   If your panel contains multiple queries or queries multiple nodes, then you have additional options.
   - **Select result**: Choose which result set data you want to view.
   - **Transform data**
   - **Join by time**: View raw data from all your queries at once, one result set per column. Click a column heading to reorder the data.

1. To see data before the system applies field overrides, click the **Formatted data** toggle.
1. To download a CSV file specifically formatted for Excel, click the **Download for Excel** toggle .
1. Click **Download CSV**.

## Inspect query performance

The **Stats** tab displays statistics that tell you how long your query takes, how many queries you send, and the number of rows returned.
This information can help you troubleshoot your queries, especially if any of the numbers are unexpectedly high or low.

1. Edit the panel that contains the query with performance you want to inspect.
1. In the query editor, click **Query inspector**.
1. Click **Stats**.

Statistics display in read-only format.

## Inspect query request and response data

Inspect query request and response data when you want to troubleshoot a query that returns unexpected results, or fails to return expected results.

1. Edit the panel that contains the query you want to export.
1. In the query editor, click **Query inspector**.
1. Click **Refresh**.

   The panel populates with response data.

1. Make adjustments, as necessary and re-run the query.
1. To download the query request and response data, click the **Copy to clipboard** icon and paste the results into another application.

## Inspect errors and notices

Panels with errors or notices display an icon in the top-left corner, before the panel title.
The icon is dynamic, reflecting the highest severity notice on a panel.

To inspect errors and notices, follow these steps:

1. Navigate to the panel you want to view.
1. Click the icon to display a tooltip that shows all the panel errors and notices, sorted by severity:

   ![Errors and notices tooltip displayed on a panel](/media/docs/grafana/dashboards/screenshot-errors-and-notices-tooltip-v13.3.png)

1. Click **Inspect** to open the drawer where you can see the full text of errors and notices.

# Panel overview

A Grafana panel is a visual representation of data composed of a [query](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/) and a [visualization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/visualizations/). Within panels, you can apply [transformations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/transform-data/), which process the results of a query before they're passed on for visualization. You can also further customize a panel by formatting data and configuring visualization options.

Each panel has a query editor specific to the data source selected in the panel. The query editor allows you to build a query that returns the data you want to visualize.

Panels offer a wide variety of formatting and styling options, from applying colors based on field values to creating custom units. Each visualization also comes with options specific to it that give you further control over how your data is displayed. Panels can also be dragged, dropped, and resized to rearrange them on the dashboard.

To get started adding panels, ensure that you have configured a data source:

- For details about using data sources, refer to [Data sources](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/).
- For more information about managing data sources as an administrator, refer to [Data source management](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/).

  {{< admonition type="note" >}}
  [Data source management](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/) is only available in [Grafana Enterprise](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/introduction/grafana-enterprise/) and [Grafana Cloud](https://grafana.com/docs/grafana-cloud/).
  {{< /admonition >}}

## Panel feature overview

The following image and descriptions highlight the panel features:

![Annotated panel with time series visualization](/media/docs/grafana/panels-visualizations/screenshot-panel-overview-ann-v11.0.png)

1. **Panel title** - You can create your own panel titles or have Grafana create them for you using [generative AI features](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/manage-dashboards/#set-up-generative-ai-features-for-dashboards).
1. **Panel description** - You can create your own panel descriptions or have Grafana create them for you using [generative AI features](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/manage-dashboards/#set-up-generative-ai-features-for-dashboards)
1. **Links** - Add [panel links](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/manage-dashboard-links/#panel-links) to other dashboards, panels, or external sites.
1. **Panel menu** - In the [panel menu](#panel-menu), access actions such as **View**, **Edit**, **Inspect**, and **Remove**.
1. **Legend** - Change series colors, y-axis, and series visibility directly from the [legend](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-legend/).
1. **Tooltips** - View [tooltips](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-tooltips/) to get more information about data points.

## Panel menu

To access the panel editor, hover the cursor over the top-right corner of any panel.
Click the panel menu icon that appears and select **Edit**.

The panel menu also gives you access to several actions.
If an option is only available in view mode or edit mode, that's indicated:

- **View**: View the panel in full screen. Opens the [View panel sidebar](#view-mode-panel-controls) and displays controls for supported visualizations, if enabled.
- **Edit**: Open the panel editor to edit panel and visualization options.
- **Share**: Share the panel as a link, embed, or snapshot.
- **Explore**: Open the panel in **Explore**, where you can focus on your query.
- **Inspect**: Open the **Inspect** drawer, where you can review the panel data, stats, metadata, JSON, and query.
  - **Data**: Open the **Inspect** drawer in the **Data** tab.
  - **Query**: Open the **Inspect** drawer in the **Query** tab.
  - **Panel JSON**: Open the **Inspect** drawer in the **JSON** tab.
- **Time settings**: Opens the **Panel time settings** drawer where you can set panel-specific time options. For more information, refer to [Panel time settings](#panel-time-settings).
- **Styles**: Edit mode only. Copy and paste styles from one panel to another in the same dashboard. For more information, refer to [Copy and paste panel styles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/panel-editor-overview/#copy-and-paste-panel-styles).
- **Assistant**: View mode only. Access Grafana Assistant help options. This option is only available on Grafana Cloud.
- **Metrics drilldown**: Open the panel in the **Drilldown > Metrics** feature for further exploration. For more information, refer to [Metrics drilldown](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/simplified-exploration/metrics/).
- **Extensions**: View mode only. Access other actions provided by installed applications, such as declaring an incident. This option is only available in view mode and only appears if you have app plugins installed that contribute an [extension](https://grafana.com/developers/plugin-tools/key-concepts/ui-extensions) to the panel menu.
- **More**: Access other panel actions.
  - **Duplicate**: Edit mode only. Make a copy of the panel. Duplicated panels query data separately from the original panel. You can use the special `Dashboard` data source to [share the same query results across panels](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/share-query/) instead.
  - **Copy**: Copy the panel to the clipboard.
  - **New library panel**: Edit mode only. Create a panel that can be imported into other dashboards.
  - **New alert rule**: Open the alert rule configuration page in **Alerting**, where you can [create a Grafana-managed alert](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/#create-alerts-from-panels) based on the panel queries.
  - **Hide/Show legend**: Hide or show the panel legend.
  - **Get help**: Send a snapshot or panel data to Grafana Labs Technical Support.
- **Remove**: Edit mode only. Remove the panel from the dashboard.

## View mode panel controls

{{< admonition type="note" >}}
View panel sidebar is currently in public preview. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.

This feature is enabled by default. To turn it off, disable the `viewPanelPane` feature toggle in your Grafana configuration file or contact Support.
{{< /admonition >}}

When you open a panel in full-screen view mode, Grafana displays a **View panel** sidebar with controls for adjusting the panel.
These controls make it easier to view specific data or identify patterns and correlations.
Because these controls exist in view mode, you don't need edit permissions to use them, and the changes you make don't affect the saved dashboard.

After you close the sidebar, it stays closed when you view a panel again.
Select **View panel controls** in the toolbar to reopen it and turn automatic opening back on.

The sidebar includes the following controls:

- **Quick toggles**: Adjust common visualization options, such as legend visibility and basic graph styles.
- **Fan-out by series or label**: Split one graph into multiple graphs per series or label value. This is useful when a single graph has many series and you need to spot which series or labels share an anomaly.

{{< figure src="/media/docs/grafana/dashboards/screenshot-view-panel-sidebar-2-v13.2.png" max-width="750px" alt="Panel in view mode with sidebar" >}}

{{< admonition type="note" >}}
This feature is supported for time series visualizations.
If it's added to more visualizations, available quick toggles and fan-out support will depend on the visualization type.
{{< /admonition >}}

### Fan-out by series or label

Fan-out splits one panel into multiple panels so you can compare series or label values side by side.
This is useful when a single graph has many series and you need to spot which series or labels share an anomaly.

In the **Fan-out by series or label** section, choose one of the following:

| Option        | Description                                                                           |
| ------------- | ------------------------------------------------------------------------------------- |
| **Disabled**  | Show the original single panel.                                                       |
| **By series** | Create one panel per series.                                                          |
| **Labels**    | Create one panel per value of the selected label (for example, `method` or `status`). |

If the panel data has no labels, Grafana shows **Data has no labels** under **Labels**.

{{< figure src="/media/docs/grafana/dashboards/screenshot-view-panel-sidebar-v13.2.png" max-width="750px" caption="Panel in view mode with fan-out applied to series" >}}

When you select a fan-out mode, Grafana updates the dashboard URL with a `fanout` query parameter so you can share the split view.
Leaving view mode removes that parameter.

## Keyboard shortcuts

Grafana has a number of keyboard shortcuts available specifically for panels. Press `?` on your keyboard to display all keyboard shortcuts available in your version of Grafana.

By hovering over a panel with the mouse you can use some shortcuts that target that panel.

- `e`: Toggle panel edit view
- `v`: Toggle panel full screen view
- `pu`: Copy panel share link
- `pe`: Share embed
- `ps`: Share snapshot
- `px`: Open panel in **Explore**
- `pc`: Copy panel
- `pv`: Paste panel
- `pd`: Duplicate Panel
- `i`: Inspect
- `pl`: Hide or show legend
- `pr`: Remove Panel

## Panel time settings

You can configure the following settings to control the time range for a panel:

| Option                         | Description                                                                                                                                                                                                                                                     |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Panel time range               | Overrides the dashboard time range. Use one of the preset values or enter a custom value like `5m` or `2h`.                                                                                                                                                     |
| Time shift                     | Adds a time shift relative to the dashboard or panel time range. Use one of the preset values or enter a custom value like `5m` or `2h`.                                                                                                                        |
| Time comparison                | Compares data with an earlier time range, such as the day or week before. Applied after **Time shift** when used together.                                                                                                                                      |
| Comparison tooltip delta color | Colors the change between the original and comparison values in the tooltip. An increase is green with **Standard**, red with **Inverted**, or the series color with **Same as value**. Only shown for time series visualizations with **Time comparison** set. |
| Hide panel time range          | Don't show the panel time range in the panel header.                                                                                                                                                                                                            |

## Pan and zoom panel time range

You can pan the panel time range left and right, and zoom it and in and out.
This, in turn, changes the dashboard time range.

This feature is supported for the following visualizations:

- Candlestick
- Heatmap
- State timeline
- Status history
- Time series

### Zoom in

Click and drag on the panel to zoom in on a particular time range.

The following screen recordings show this interaction in the time series and candlestick visualizations:

Time series

{{< video-embed src="/media/docs/grafana/panels-visualizations/recording-ts-time-zoom-in-mouse.mp4" >}}

Candlestick

{{< video-embed src="/media/docs/grafana/panels-visualizations/recording-candle-panel-time-zoom-in-mouse.mp4" >}}

### Zoom out

Double-click anywhere on the panel to zoom out the time range.

The range doubles with each double-click, adding equal time to each side of the range.
For example, if the original time range is from 9:00 to 9:59, the time range changes as follows with each double-click:

- Next range: 8:30 - 10:29
- Next range: 7:30 - 11:29

The following screen recordings demonstrate the preceding example in the time series and heatmap visualizations:

Time series

{{< video-embed src="/media/docs/grafana/panels-visualizations/recording-ts-time-zoom-out-mouse.mp4" >}}

Heatmap

{{< video-embed src="/media/docs/grafana/panels-visualizations/recording-heatmap-panel-time-zoom-out-mouse.mp4" >}}

### Pan

Click and drag the x-axis area of the panel to pan the time range.

The time range shifts by the distance you drag.
For example, if the original time range is from 9:00 to 9:59 and you drag 30 minutes to the right, the time range changes to 9:30 to 10:29.

The following screen recordings show this interaction in the time series visualization:

Time series

{{< video-embed src="/media/docs/grafana/panels-visualizations/recording-ts-time-pan-mouse.mp4" >}}

## Add a panel

To add a panel to an empty dashboard, click or drag the panel onto the dashboard:

![Empty dashboard state](/media/docs/grafana/dashboards/screenshot-empty-dashboard-v13.2.png)

To add a panel to an existing dashboard, follow these steps:

1. Click **Edit** in the top-right corner of the dashboard.
1. Click the **Add new element** icon (blue plus sign).

   {{< figure src="/media/docs/grafana/dashboards/screenshot-add-element-icon-v13.1.png" max-width="250px" alt="Add element icon" >}}

1. Click or drag a panel onto the dashboard.

## Panel configuration

To configure panels, refer to the following subtopics:

- [Configure panel options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-panel-options/)
- [Configure standard options](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-standard-options/)
- [Configure a legend](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-legend/)
- [Configure tooltips](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-tooltips/)
- [Configure data links](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-data-links/)
- [Configure value mappings](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-value-mappings/)
- [Configure thresholds](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-thresholds/)
- [Configure field overrides](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-overrides/)

# Calculation types

The following table contains a list of calculations you can perform in Grafana. You can find these calculations in the **Transform** tab and in the bar gauge, gauge, stat, and table visualizations.

| Calculation        | Description                                               |
| :----------------- | :-------------------------------------------------------- |
| 1st % - 99th %     | 1st - 99th percentile value.                              |
| All nulls          | True when all values are null                             |
| All unique values  | Array with all unique values                              |
| All values         | Array with all values                                     |
| All zeros          | True when all values are 0                                |
| Change count       | Number of times the field's value changes                 |
| Count              | Number of values in a field                               |
| Delta              | Cumulative change in value, only counts increments        |
| Difference         | Difference between first and last value of a field        |
| Difference percent | Percentage change between first and last value of a field |
| Distinct count     | Number of unique values in a field                        |
| First              | First value in a field                                    |
| First\* (not null) | First, not null value in a field (also excludes NaNs)     |
| Last               | Last value in a field                                     |
| Last\* (not null)  | Last, not null value in a field (also excludes NaNs)      |
| Max                | Maximum value of a field                                  |
| Median             | Median value of all values in a field                     |
| Mean               | Mean value of all values in a field                       |
| Min                | Minimum value of a field                                  |
| Min (above zero)   | Minimum, positive value of a field                        |
| Range              | Difference between maximum and minimum values of a field  |
| StdDev             | Standard deviation of all values in a field               |
| Step               | Minimal interval between values of a field                |
| Total              | Sum of all values in a field                              |
| Variance           | Variance of all values in a field                         |

# Write expression queries

Server-side expressions enable you to manipulate data returned from queries with math and other operations. Expressions create new data and do not manipulate the data returned by data sources.

## About expressions

Server-side expressions allow you to manipulate data returned from queries with math and other operations. Expressions create new data and do not manipulate the data returned by data sources, aside from some minor data restructuring to make the data acceptable input for expressions.

### Using expressions

Expressions are most commonly used for [Grafana Alerting](ref:grafana-alerting). The processing is done server-side, so expressions can operate without a browser session. However, expressions can also be used with backend data sources and visualization.

{{< admonition type="note" >}}
Expressions do not work with legacy dashboard alerts.
{{< /admonition >}}

Expressions are meant to augment data sources by enabling queries from different data sources to be combined or by providing operations unavailable in a data source.

{{< admonition type="note" >}}
When possible, you should do data processing inside the data source. Copying data from storage to the Grafana server for processing is inefficient, so expressions are targeted at lightweight data processing.
{{< /admonition >}}

Expressions work with data source queries that return time series or number data. They also operate on [multiple-dimensional data](ref:multiple-dimensional-data). For example, a query that returns multiple series, where each series is identified by labels or tags.

An individual expression takes one or more queries or other expressions as input and adds data to the result. Each individual expression or query is represented by a variable that is a named identifier known as its RefID (e.g., the default letter `A` or `B`).

To reference the output of an individual expression or a data source query in another expression, this identifier is used as a variable.

### Types of expressions

Expressions work with two types of data.

- A collection of time series.
- A collection of numbers, where each number is an item.

Each collection is returned from a single data source query or expression and represented by the RefID. Each collection is a set, where each item in the set is uniquely identified by its dimensions which are stored as [labels](ref:labels) or key-value pairs.

### Data source queries

Server-side expressions only support data source queries for backend data sources. The data is generally assumed to be labeled time series data. In the future we intend to add an assertion of the query return type (number or time series) data so expressions can handle errors better.

Data source queries, when used with expressions, are executed by the expression engine. When it does this, it restructures data to be either one time series or one number per data frame. So for example if using a data source that returns multiple series on one frame in the table view, you might notice it looks different when executed with expressions.

Currently, the only non-time series format (number) is supported when you're using data frames and you have a table response that returns a data frame with no time, string columns, and one number column:

| Loc | Host | Avg_CPU |
| --- | ---- | ------- |
| MIA | A    | 1       |
| NYC | B    | 2       |

The example above will produce a number that works with expressions. The string columns become labels and the number column the corresponding value. For example `{"Loc": "MIA", "Host": "A"}` with a value of 1.

### Operations

You can use the following operations in expressions: math, reduce, resample, and SQL.

#### Math

Math is for free-form math formulas on time series or number data. Math operations take numbers and time series as input and change them to different numbers and time series.

Data from other queries or expressions are referenced with the RefID prefixed with a dollar sign, for example `$A`. If the variable has spaces in the name, then you can use a brace syntax like `${my variable}`.

Numeric constants may be in decimal (`2.24`), octal (with a leading zero like `072`), or hex (with a leading 0x like `0x2A`). Exponentials and signs are also supported (e.g., `-0.8e-2`).

##### Operators

The arithmetic (`+`, binary and unary `-`, `*`, `/`, `%`, exponent `**`), relational (`<`, `>`, `==`, `!=`, `>=`, `<=`), and logical (`&&`, `||`, and unary `!`) operators are supported.

How the operation behaves with data depends on if it is a number or time series data.

With binary operations, such as `$A + $B` or `$A || $B`, the operator is applied in the following ways depending on the type of data:

- If both `$A` and `$B` are a number, then the operation is performed between the two numbers.
- If one variable is a number, and the other variable is a time series, then the operation between the value of each point in the time series and the number is performed.
- If both `$A` and `$B` are time series data, then the operation between each value in the two series is performed for each time stamp that exists in both `$A` and `$B`. The Resample operation can be used to line up time stamps. (**Note:** in the future, we plan to add options to the Math operation for different behaviors).

Summary:

- Number OP number = number
- Number OP series = series
- Series OP series = series

Because expressions work with multiple series or numbers represented by a single variable, binary operations also perform a union (join) between the two variables. This is done based on the identifying labels associated with each individual series or number.

So if you have numbers with labels like `{host=web01}` in `$A` and another number in `$B` with the same labels then the operation is performed between those two items within each variable, and the result will share the same labels. The rules for the behavior of this union are as follows:

- An item with no labels will join to anything.
- If both `$A` and `$B` each contain only one item (one series, or one number), they will join.
- If labels are exact match they will join.
- If labels are a subset of the other, for example and item in `$A` is labeled `{host=A,dc=MIA}` and item in `$B` is labeled `{host=A}` they will join.
- Currently, if within a variable such as `$A` there are different tag _keys_ for each item, the join behavior is undefined.

The relational and logical operators return 0 for false 1 for true.

##### Math Functions

While most functions exist in the own expression operations, the math operation does have some functions similar to math operators or symbols. When functions can take either numbers or series, than the same type as the argument will be returned. When it is a series, the operation of performed for the value of each point in the series.

###### abs

abs returns the absolute value of its argument which can be a number or a series. For example `abs(-1)` or `abs($A)`.

###### is_inf

is_inf takes a number or a series and returns `1` for `Inf` values (negative or positive) and `0` for other values. For example `is_inf($A)`.

{{< admonition type="note" >}}
If you need to specifically check for negative infinity for example, you can do a comparison like `$A == infn()`.
{{< /admonition >}}

###### is_nan

is_nan takes a number or a series and returns `1` for `NaN` values and `0` for other values. For example `is_nan($A)`. This function exists because `NaN` is not equal to `NaN`.

###### is_null

is_null takes a number or a series and returns `1` for `null` values and `0` for other values. For example `is_null($A)`.

###### is_number

is_number takes a number or a series and returns `1` for all real number values and `0` for other values (which are `null`, `Inf+`, `Inf-`, and `NaN`). For example `is_number($A)`.

###### log

Log returns the natural logarithm of of its argument which can be a number or a series. If the value is less than 0, NaN is returned. For example `log(-1)` or `log($A)`.

###### inf, infn, nan, and null

The inf, infn, nan, and null functions all return a single value of the name. They primarily exist for testing. Example: `null()`.

###### round

Round returns a rounded integer value. For example, `round(3.123)` or `round($A)`. (This function should probably take an argument so it can add precision to the rounded value).

###### ceil

Ceil rounds the number up to the nearest integer value. For example, `ceil(3.123)` returns 4.

###### floor

Floor rounds the number down to the nearest integer value. For example, `floor(3.123)` returns 3.

#### Reduce

Reduce takes one or more time series returned from a query or an expression and turns each series into a single number. The labels of the time series are kept as labels on each outputted reduced number.

**Fields:**

- **Function -** The reduction function to use
- **Input -** The variable (refID (such as `A`)) to resample
- **Mode -** Allows control behavior of reduction function when a series contains non-numerical values (null, NaN, +\-Inf)

##### Reduction Functions

###### Count

Count returns the number of points in each series.

###### Mean

Mean returns the total of all values in each series divided by the number of points in that series. In `strict` mode if any values in the series are null or nan, or if the series is empty, NaN is returned.

###### Min and Max

Min and Max return the smallest or largest value in the series respectively. In `strict` mode if any values in the series are null or nan, or if the series is empty, NaN is returned.

###### Sum

Sum returns the total of all values in the series. If series is of zero length, the sum will be 0. In `strict` mode if there are any NaN or Null values in the series, NaN is returned.

##### Last

Last returns the last number in the series. If the series has no values then returns NaN.

##### Reduction Modes

###### Strict

In Strict mode the input series is processed as is. If any values in the series are non-numeric (null, NaN or +\-Inf), NaN is returned.

###### Drop Non-Numeric

In this mode all non-numeric values (null, NaN or +\-Inf) in the input series are filtered out before executing the reduction function.

###### Replace Non-Numeric

In this mode all non-numeric values are replaced by a pre-defined value.

#### Resample

Resample changes the time stamps in each time series to have a consistent time interval. The main use case is so you can resample time series that do not share the same timestamps so math can be performed between them. This can be done by resample each of the two series, and then in a Math operation referencing the resampled variables.

**Fields:**

- **Input -** The variable of time series data (refID (such as `A`)) to resample
- **Resample to -** The duration of time to resample to, for example `10s`. Units may be `s` seconds, `m` for minutes, `h` for hours, `d` for days, `w` for weeks, and `y` of years.
- **Downsample -** The reduction function to use when there are more than one data point per window sample. See the reduction operation for behavior details.
- **Upsample -** The method to use to fill a window sample that has no data points.
  - **pad** fills with the last know value
  - **backfill** with next known value
  - **fillna** to fill empty sample windows with NaNs

## Write an expression

{{< admonition type="note" >}}
The steps in this section describe the classic, generally available panel query editor.
For information about the new panel query editor experience, currently in public preview, refer to the [Query and transform data documentation](https://grafana.com/docs/grafana/v13.1/visualizations/panels-visualizations/query-transform-data/).
{{< /admonition >}}

If your data source supports them, then Grafana displays the **Expression** button and shows any existing expressions in the query editor list.

For more information about expressions, refer to [About expressions](#about-expressions).

1. Open the panel.
1. Below the query, click **Expression**.
1. In the **Operation** field, select the type of expression you want to write.

   For more information about expression operations, refer to [About expressions](#about-expressions).

1. Write the expression.
1. Click **Apply**.

## Special cases

When any queried data source returns no series or numbers, the expression engine returns `NoData`. For example, if a request contains two data source queries that are merged by an expression, if `NoData` is returned by at least one of the data source queries, then the returned result for the entire query is `NoData`.

For more information about how [Grafana Alerting](ref:grafana-alerting) processes `NoData` results, refer to [No data and error handling](ref:no-data-and-error-handling).

In the case of using an expression on multiple queries, the expression engine requires that all of the queries return an identical timestamp. For example, if using math to combine the results of multiple SQL queries which each use `SELECT NOW() AS "time"`, the expression will only work if all queries evaluate `NOW()` to an identical timestamp; which does not always happen. To resolve this, you can replace `NOW()` with an arbitrary time, such as `SELECT 1 AS "time"`, or any other valid UNIX timestamp.

# Share query results with another panel

Grafana lets you use the query result from one panel for any other panel in the dashboard. Sharing query results across panels reduces the number of queries made to your data source, which can improve the performance of your dashboard.

The Dashboard data source lets you select a panel in your dashboard that contains the queries ‌you want to share the results for. Instead of sending a separate query for each panel, Grafana sends one query and other panels use the query results to construct visualizations.

This strategy can drastically reduce the number of queries being made when you for example have several panels visualizing the same data.

1. [Create a dashboard](ref:create-a-dashboard).
1. Create a panel.
1. Change the panel title to "Source panel". You'll use this panel as a source for the other panels.
1. Define the [query](ref:query) or queries that you want share.

   If you don't have a data source available, use the **Grafana** data source, which returns a random time series that you can use for testing.

1. Add a new panel and select the **Dashboard** data source in the query editor.
1. In the **Use results from panel list**, select the first panel you created.

All queries defined in the source panel are now available to the new panel. Queries defined in the source panel can be shared with multiple panels.

You can click on any of the queries to go to the panel where they are defined.

# SQL expressions

SQL Expressions are server-side expressions that manipulate and transform the results of data source queries using MySQL-like syntax. They allow you to easily query and transform your data after the data source returns it, using SQL, which provides a familiar and powerful syntax that can handle everything from simple filters to highly complex, multi-step transformations.

In Grafana, a server-side expression is a way to transform or calculate data after Grafana retrieves it from the data source, but before Grafana sends it to the frontend for visualization. Grafana evaluates these expressions on the server, not in the browser or at the data source.

For general information on Grafana expressions, refer to [Write expression queries](ref:expressions).

{{< admonition type="note" >}}
The screenshots and steps on this page reflect the classic, generally available panel query editor.
For information about the new panel query editor experience, currently in public preview, refer to the [Query and transform data documentation](https://grafana.com/docs/grafana/v13.1/visualizations/panels-visualizations/query-transform-data/).
{{< /admonition >}}

![Example of a SQL expression](/media/docs/sql-expressions/sql-expressions-example-1.png)

To see examples of filtering, aggregation, and JOIN operations with SQL expressions, explore this demo dashboard on Grafana Play:

{{< docs/play title="SQL Expressions Showcase" url="https://play.grafana.org/d/mapr4nb/" >}}

## Before you begin

- Grafana enables SQL expressions by default. You don't need to set a feature toggle or change your configuration to use them.
- You must query a backend data source. SQL expressions don't work with frontend-only data sources.

## Transform data with SQL expressions

SQL expressions allow you to:

- Shape, transform, and modify query results without changing the original query.
- JOIN data from multiple tables.
- Create alerts or recording rules based on transformed data.
- Perform final-stage modifications to datasets, including:
  - Show, hide, or rename columns.
  - Filter rows based on conditions.
  - Aggregate data (for example: sum, average, count).
- Write subqueries and Common Table Expressions (**CTEs**) to support more complex logic:
  - **Subqueries** are nested queries used for filtering, calculations, or transformations.
  - **CTEs** are temporary named result sets that help make complex queries more readable and reusable.

A key capability of SQL expressions is the ability to JOIN data from multiple tables. This allows users to combine and transform data in a predictable, user-friendly way—even for complex use cases. You can JOIN data from many data source queries, subject to the [input cell limit](#known-limitations).

To work with SQL expressions, you must use data from a backend data source. In Grafana, a backend data source refers to a data source plugin or integration that communicates with a database, service, or API through the Grafana server, rather than directly from the browser (frontend).

Frontend-only data sources, such as the **[Dashboard](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/#dashboard)** data source, aren't supported. Use one of the compatible data sources listed below for queries that you want to reference in a SQL expression.

## Compatible data sources

The following are compatible data sources:

**Full support:** Grafana supports all query types for each of these data sources.

- Elasticsearch
- MySQL
- Loki
- Graphite
- Google Sheets
- Amazon Athena
- PostgreSQL

**Partial support:** The following data sources have limited or conditional support. Some support multiple query types depending on the service. For example, Azure Monitor can query multiple services, each with its own query format. In some cases, you can also switch the query type within a panel.

- InfluxDB
- Infinity
- Azure Monitor
- TestData
- Tempo
- Prometheus
- CloudWatch
- GitHub
- BigQuery

## Create SQL expressions

To create a SQL expression, complete the following steps:

1. Navigate to **Dashboards** in the left-side menu.
1. Select a dashboard and open a dashboard panel.
1. Click the ellipsis in the upper right and select **Edit** .
1. Click **+ Expression**.
1. Select **SQL** from the drop-down.

After you have added a SQL expression, you can select from other data source queries by referencing the RefIDs of the queries in your SQL expression as if they were tables in a SQL database.

{{< admonition type="note" >}}
The **RefID** is a unique identifier assigned to each query within a Grafana panel that serves as a reference name for that query's data.
{{< /admonition >}}

![Using the RefID](/media/docs/sql-expressions/using-the-RefID.png)

## Workflow to build SQL expressions

Use the following workflow to create a SQL expression:

1. **Build your base queries.** Create the individual query and give it a meaningful name. Create the queries (A, B, etc.) that provide the data you want to combine or transform using SQL Expressions.
1. **Hide your base queries.** Click the **👁️ Eye icon** next to each base query to hide them from visualization. This keeps your panel clean while still making the data available to the SQL Expression.
1. **Switch to table view**. Set the panel visualization to **Table** to inspect and review the structure and output of your SQL expression as you build and refine it.
1. **Add a SQL Expression**. Add a new query and add select SQL Expression as its type.  
   **Inspect inputs**. Start with simple test queries to understand the shape of your input frames.

   ```sql
   SELECT * FROM A LIMIT 10
   ```

   This lets you see the available columns and sample rows from `query A`. Repeat this for each input query you want to use (for example, `SELECT * FROM B LIMIT 10`).

1. **Inspect your data**. Repeat this for each input query to understand the column structure and data types you're working with.

   ```sql
   SELECT * FROM <B, C, D, etc> LIMIT 10
   ```

1. **Construct the SQL expression.** After you understand your data, you can write your SQL expression to join, filter, or otherwise transform the data.
1. **Validate and iterate**. Click **Refresh** every time you update your SQL query to re-evaluate and see the updated result.

When selecting a visualization type, **ensure your SQL expression returns data in the required shape**. For example, time series panels require a column with a time field (such as a timestamp) and a numeric value column (such as `__value__`). If the output is not shaped correctly, your visualization may appear empty or fail to render.

The SQL expression workflow in Grafana has the following behaviors:

- **Unhidden queries are visualized automatically.** If an input query is not hidden, Grafana will attempt to render it alongside your SQL expression. This can clutter the output, especially in table visualizations.

- **SQL expression results may not be immediately visible.** You might need to use the data frame selector (dropdown at the bottom of the table panel) to switch between the raw query and the SQL expression result.

- **Non-tabular or incorrectly shaped data will not render in certain panels.** Visualizations such as graphs or gauges require properly structured data. Mismatched formats will result in rendering issues or missing data.

## The SQL expression editor

The SQL expression editor provides several tools to help you write and run queries.

- **Run query**: Runs your SQL expression and refreshes the result. You can also press Ctrl+Enter, or Cmd+Enter on macOS, while your cursor is in the editor.
- **Format query**: Formats your SQL for readability while preserving template variables.
- **Copy query**: Copies the current SQL to your clipboard.

The editor autocompletes SQL keywords, [supported functions](#supported-functions), and the RefIDs of your other queries as table names. Column autocomplete is experimental and requires the `sqlExpressionsColumnAutoComplete` feature toggle. When enabled, the editor suggests column names by querying your data sources for their fields.

{{< admonition type="note" >}}
An enhanced editor built on CodeMirror is available on an experimental basis behind the `sqlExpressionsCodeMirror` feature toggle. It adds function signature help and context-aware column suggestions, such as resolving table aliases.
{{< /admonition >}}

### Schema inspector

When you enable the query service API on your instance, the editor includes a **Schema inspector** side panel. Click the eye icon next to the editor to show or hide it.

The Schema inspector has a tab for each source query and lists the following details for every column, so you can reference the correct field names and types as you write your SQL:

- **Field**: The column name.
- **Type**: The MySQL data type.
- **Nullable**: Whether the column can contain null values.
- **Sample values**: Example values from the query result.

## SQL expressions examples

1. Create the following Prometheus query:

   ```promql
   sum(
     rate(go_cpu_classes_gc_total_cpu_seconds_total{namespace=~".*(namespace).*5."}[$__rate_interval])
   ) by (namespace)
   ```

   The panel displays the CPU usage by Go garbage collection (GC) over time, broken down by namespace.

   ![Example using a Prometheus query](/media/docs/sql-expressions/sql-expressions-prom-query-example.png)

2. Add the SQL expression `SELECT * from A`. After you add a SQL expression that selects from RefID A, Grafana converts it to a table response:

   ![Add the SQL expression](/media/docs/sql-expressions/add-the-sql-expression.png)

## SQL conversion rules

When you reference a RefID within a SQL statement (for example, `SELECT * FROM A`), the system invokes a distinct SQL conversion process.

The SQL conversion path:

- The query result appears as a single data frame, without labels, and maps directly to a tabular format.
- If the frame type is present and is either numeric, wide time series, or multi-frame time series (for example: labeled formats), Grafana automatically converts the data into a table structure.
- If a column name repeats, later columns get a number, for example `host_2` and `host_3`. The match ignores case.

## Data source response formats

Grafana supports three types of data source response formats:

1. **Single Table-like Frame**:  
   This refers to data returned in a standard tabular structure that organizes all values into rows and columns, similar to what you'd get from a SQL query.
   - **Example**: Any query against a SQL data source (for example, PostgreSQL, MySQL) with the format set to Table.

2. **Dataplane: Time Series Format**:  
   This format represents time series data with timestamps and associated values. It is typically returned from monitoring data sources.
   - **Example**: Prometheus or Loki Range Queries (queries that return a set of values over time).

3. **Dataplane: Numeric Long Format**:  
   This format represents point-in-time (instant) metric queries that return a single value (or a set of values) at a specific moment.
   - **Example**: Prometheus or Loki Instant Queries (queries that return the current value of a metric).

<!-- vale Grafana.Spelling = NO -->

For more information on these formats, refer to the [Grafana Dataplane documentation](https://grafana.com/developers/dataplane).

<!-- vale Grafana.Spelling = YES -->

The following non-tabular formats are automatically converted to a tabular format (`FullLong`) when used in SQL expressions:

- **Time Series Wide**: Label keys become column names.
- **Time Series Multi**: Label values become the values in each row (or null if a label is missing).
- **Numeric Wide**: The `__value__` column contains the numeric metric value.
- **Numeric Multi**: If a display name exists, it will appear in the `__display_name__` column.

During conversion:

- Label keys become column names.
- If a label key matches a generated column name, such as `time` or `__value__`, its column is named `label_<LABEL_KEY>`, for example `label_time`. The match ignores case.
- Label values populate the corresponding rows (null if a label is missing).
- The `__value__` column contains the numeric metric.
- If available, the `__display_name__` column contains a human-readable name.
- The `__metric_name__` column stores the raw metric identifier.
- For time series data, Grafana includes a `time` column with timestamps

## Supported functions

Grafana maintains a complete list of supported SQL keywords, operators, and functions in the SQL expressions query validation code.

For the most up-to-date reference of all supported SQL functionality, refer to the `allowedNode` and `allowedFunction` definitions in the Grafana [codebase](https://github.com/grafana/grafana/blob/main/pkg/expr/sql/parser_allow.go).

## Alerting and recording rules

SQL expressions integrates alerting and recording rules, allowing you to define complex conditions and metrics using standard SQL queries. The system processes your query results and automatically creates alert instances or recorded metrics based on the returned data structure.

For SQL Expressions to work properly with alerting and recording rules, your query must return:

- Exactly one numeric column - **_required_**. This contains the value that triggers alerts or gets recorded. The query fails if it returns no numeric column or more than one.
- Unique string column combinations - **_required_**. Each row must have a unique combination of string column values.
- One or more string columns - _optional_. These become **labels** for the alert instances or metrics. Examples: `service`, `region`.

Consider the following query results:

```sql
error_count,service,region
25,auth-service,us-east
0,payment-service,us-west
15,user-service,eu-west
```

This query returns:

- the numeric column `error_count` (values: 25, 0, 15)
- the string columns `service` and `region`

For alert rules, this creates three alert instances:

- First instance with labels {service=auth-service, region=us-east} and value 25 (triggers alert - high error count)
- Second instance with labels {service=payment-service, region=us-west} and value 0 (no alert - zero errors)
- Third instance with labels {service=user-service, region=eu-west} and value 15 (triggers alert - elevated error count)

For recording rules, creates one metric with three series:

- First series: error_count_total{service=auth-service, region=us-east} 25
- Second series: error_count_total{service=payment-service, region=us-west} 0
- Third series: error_count_total{service=user-service, region=eu-west} 15

Following are some best practices for alerting and recording rules:

- Keep numeric values meaningful (for example: error counts, request duration).
- Use clear, descriptive column names - these become your labels.
- Keep string values short and consistent.
- Avoid too many unique label combinations, as this can result in high cardinality.
- Always use `GROUP BY` to avoid duplicate label errors.
- Aggregate numeric values logically (for example: `SUM(error_count)`).

## Known limitations

- Grafana supports certain data sources. Refer to [compatible data sources](#compatible-data-sources) for a current list.
- A SQL expression can only reference data source queries. It can't reference other expressions, and you can't use the output of a SQL expression as the input to another expression.
- You can use only one SQL expression per panel or alert.
- Each SQL expression query must include a time range.
- SQL expressions aren't supported on 32-bit ARM builds of Grafana.
- Autocomplete is available, but column and field autocomplete requires the experimental `sqlExpressionsColumnAutoComplete` feature toggle.

### Query limits

Grafana enforces the following limits on SQL expressions. Administrators can configure them in the `[expressions]` section of the Grafana configuration. For more information, refer to [Configure Grafana](ref:configure-expressions).

- **Input cells**: The total number of cells, that is rows multiplied by columns, across all referenced queries can't exceed `sql_expression_cell_limit`. The default is `100000`. If the input exceeds this limit, the query fails.
- **Output cells**: The number of cells returned can't exceed `sql_expression_output_cell_limit`. The default is `100000`. If the output exceeds this limit, Grafana truncates the result and returns a warning.
- **Query length**: The SQL text can't exceed `sql_expression_query_length_limit` characters. The default is `10000`.
- **Timeout**: Grafana cancels a SQL expression that runs longer than `sql_expression_timeout`. The default is `10s`.

For each limit, a value of `0` means no limit.

### Regular expression limitations in SQL expressions

SQL expressions run on an embedded SQL engine that evaluates regular expressions with Go's standard `regexp` package, which uses RE2 syntax instead of the SQL engine's full MySQL-compatible regular expressions. As a result, some regular expression features aren't available.

SQL expressions that use regular expression functions have limitations such as:

- Lack of back-references.
<!-- vale Grafana.Spelling = NO -->
- No lookahead or lookbehind assertions.
<!-- vale Grafana.Spelling = YES -->
- Differences in handling carriage return (`\r`) characters.

There may be other minor differences as well.

### Schema changes and missing data

SQL expressions have known limitations that may cause queries to fail or return unexpected results. These constraints are inherent to how the feature works, so keep them in mind when you build queries.

These limitations affect the following situations:

- **Error responses**: When a data source query returns an error, SQL expressions cannot interpret the result.

- **No data responses**: If a query returns no rows, the SQL expression engine cannot infer a schema.

- **Dynamic schema responses**: If the set of columns or labels changes between query executions, SQL expressions may fail because it treats column changes as schema changes.

An embedded SQL engine powers SQL expressions, treating each query result as a table. Grafana derives the schema of that table from the columns that the underlying data source returns.

Unlike traditional SQL databases, where schemas are usually fixed, many Grafana data sources (for example, Prometheus) can return results with varying label sets or no data at all.

When this happens:

- A missing column appears to the SQL engine as if it doesn’t exist.
- A completely empty result provides no schema for subsequent SQL operations.
- Error responses break the assumption that the query returns tabular data.

As a result, SQL expressions can’t gracefully handle changes in schema or no-data conditions, since these cases violate the static schema model that SQL relies on.

#### Workarounds

You can mitigate these issues in the following ways:

- Avoid `SELECT *`: Explicitly select only the columns you expect to exist.

- Ensure a consistent schema: If possible, configure your query to always return columns, even when no data is present.

#### Example: Handling Prometheus no data

When joining results from the same Prometheus query across different data source instances, you can use this pattern:

<!-- vale Grafana.Spelling = NO -->

```sql
-- Prometheus query
sum by (cluster) (up{job=~".*zruler.*"})
or on (cluster) (
  (0/0) *
  (
    label_replace(vector(1), "cluster", "fake", "", "")
  )
)

-- SQL expression
SELECT
    COALESCE(a.time, b.time) AS time,
    COALESCE(a.cluster, b.cluster) AS cluster,
    COALESCE(a.up, 0) + COALESCE(b.up, 0) AS unified_up
FROM (
    SELECT time, cluster, __value__ AS up
    FROM A
    WHERE cluster != 'fake'
    ORDER BY time
    LIMIT 5
) a
FULL OUTER JOIN (
    SELECT time, cluster, __value__ AS up
    FROM B
    WHERE cluster != 'fake'
    ORDER BY time
    LIMIT 5
) b ON a.time = b.time;
```

<!-- vale Grafana.Spelling = YES -->

This approach ensures that a schema exists even when one query returns no data.

## Grafana Assistant integration

[Grafana Assistant](ref:assistant) brings AI-powered assistance to your SQL expressions workflow. Assistant knows SQL expressions, so it can explain a query, fix syntax errors, and suggest improvements using the correct MySQL-dialect syntax, table references, and column conventions.

You can also ask Assistant to build a panel using natural language. When you ask it to combine or correlate data across multiple queries, it prefers SQL expressions over join transformations and writes the SQL expression directly into your panel.

{{< admonition type="note" >}}
Grafana Assistant is generally available on Grafana Cloud and in public preview for Grafana OSS and Grafana Enterprise. On Grafana Enterprise (version 13.1 and later), Assistant comes pre-installed, so you only need to connect your Grafana Cloud account to start using it. On Grafana OSS, install the Assistant app from the plugin catalog and connect it to a Grafana Cloud stack. For more information, refer to the [Grafana Assistant documentation](ref:assistant).
{{< /admonition >}}

When Grafana Assistant is available on your instance, the SQL expression editor shows Assistant buttons to the right of the **Run query** button. When you use a button, Assistant opens in the sidebar with your SQL query, any query errors, and your schema column metadata already included as context, so it can reference the actual field names and types in your data. From there you get the full Assistant experience, including follow-up questions, conversation history, and access to the Assistant's built-in tools.

- **Explain query**: Describes what your query does, including its interpreted meaning.
- **Improve query**: Suggests performance, reliability, and readability enhancements, and fixes syntax errors. If you haven't written a query yet, this appears as **Generate suggestion** and proposes a starting query, such as a join, aggregation, filter, percentile, or time-based window function, using your available queries.

# Transform data

> **Note:** The screenshots and steps on this page reflect the classic, generally available panel query editor. For information about the new panel query editor experience, currently in public preview, refer to the [Query and transform data documentation](https://grafana.com/docs/grafana/v13.1/visualizations/panels-visualizations/query-transform-data/).

Transformations are a powerful way to manipulate data returned by a query before the system applies a visualization. Using transformations, you can:

- Rename fields
- Join time series/SQL-like data
- Perform mathematical operations across queries
- Use the output of one transformation as the input to another transformation

{{< docs/learning-journeys title="Transform data in a Grafana Cloud dashboard" url="https://grafana.com/docs/learning-journeys/data-transformation/" >}}

For users that rely on multiple views of the same dataset, transformations offer an efficient method of creating and maintaining numerous dashboards.

You can also use the output of one transformation as the input to another transformation, which results in a performance gain.

To see transformations in action, explore this demo dashboard on Grafana Play:

{{< docs/play title="Transforms - Rows to fields" url="https://play.grafana.org/d/T5dVQV47z/" >}}

> **Note:** Sometimes the system cannot graph transformed data. When that happens, click the `Table view` toggle above the visualization to switch to a table view of the data. This can help you understand the final result of your transformations.

## Transformation types

Grafana provides a number of ways that you can transform data. For a complete list of transformations, refer to [Transformation functions](#transformation-functions).

## Order of transformations

When there are multiple transformations, Grafana applies them in the order they are listed. Each transformation creates a result set that then passes on to the next transformation in the processing pipeline.

The order in which Grafana applies transformations directly impacts the results. For example, if you use a Reduce transformation to condense all the results of one column into a single value, then you can only apply transformations to that single value.

## Dashboard variables in transformations

All text input fields in transformations accept [variable syntax](ref:dashboard-variable):

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-transformation-variables-v11.6.png" alt="Transformation with a mock variable in a text field" >}}

When you use dashboard variables in transformations, the variables are automatically interpolated before the transformations are applied to the data.

For an example, refer to [Use a dashboard variable](#use-a-dashboard-variable) in the **Filter fields by name** transformation.

## Add a transformation function to data

The following steps guide you in adding a transformation to data. This documentation does not include steps for each type of transformation. For a complete list of transformations, refer to [Transformation functions](#transformation-functions).

1. Navigate to the panel where you want to add one or more transformations.
1. Hover over any part of the panel to display the actions menu on the top right corner.
1. Click the menu and select **Edit**.
1. Click the **Transform** tab.
1. Click a transformation.
   A transformation row appears where you configure the transformation options. For more information about how to configure a transformation, refer to [Transformation functions](#transformation-functions).
   For information about available calculations, refer to [Calculation types][].
1. To apply another transformation, click **Add transformation**.
   This transformation acts on the result set returned by the previous transformation.
   {{< figure src="/static/img/docs/transformations/transformations-7-0.png" class="docs-image--no-shadow" max-width= "1100px" alt="Transform tab in the panel editor" >}}

## Debug a transformation

To see the input and the output result sets of the transformation, click the bug icon on the right side of the transformation row.

The input and output results sets can help you debug a transformation.

{{< figure src="/static/img/docs/transformations/debug-transformations-7-0.png" class="docs-image--no-shadow" max-width= "1100px" alt="The debug transformation screen with the debug icon highlighted" >}}

## Disable a transformation

You can disable or hide one or more transformations by clicking on the eye icon on the top right side of the transformation row. This disables the applied actions of that specific transformation and can help to identify issues when you change several transformations one after another.

{{< figure src="/static/img/docs/transformations/screenshot-example-disable-transformation.png" class="docs-image--no-shadow" max-width= "1100px" alt="A transformation row with the disable transformation icon highlighted" >}}

## Filter a transformation

If your panel uses more than one query, you can filter these and apply the selected transformation to only one of the queries. To do this, click the filter icon on the top right of the transformation row. This opens a drop-down with a list of queries used on the panel. From here, you can select the query you want to transform.

You can also filter by annotations (which includes exemplars) to apply transformations to them. When you do so, the list of fields changes to reflect those in the annotation or exemplar tooltip.

The filter icon is always displayed if your panel has more than one query or source of data (that is, panel or annotation data) but it may not work if previous transformations for merging the queries’ outputs are applied. This is because one transformation takes the output of the previous one.

## Set a static refId for a transformation

Some transformations build the refId of their output from the refIds of their input. For example, a **Merge series/tables** transformation applied to queries A and B outputs a frame with the refId `merge-A-B`. If you add, remove, or rename a query, or a query returns no data, that refId changes, and a later [Filter data by query refId](#filter-data-by-query-refid) transformation stops selecting the frame. To keep the refId stable, give the transformation a static refId. The transformation row calls this the transformation name.

You can set a static refId on the following transformations:

- [Merge series/tables](#merge-seriestables)
- [Join by field](#join-by-field)
- [Join by labels](#join-by-labels)
- [Series to rows](#series-to-rows)
- [Reduce](#reduce), in any mode except **Reduce fields**

Other transformations don't offer this setting.

To set a static refId:

1. Click the name to the left of the transformation type in the transformation row.
1. Enter a name.
1. Press Enter or click outside the field.

To go back to the generated refId, clear the field.

When you haven't set a static refId, the row shows the refId the transformation generates, in italics. If the transformation doesn't output exactly one frame, the row shows **(Auto)** instead.

A static refId can't:

- Contain a variable, because the refId would then change with the variable's value.
- Match the static refId of another transformation in the panel.
- Match a refId that a query or an earlier transformation already uses.

If a name breaks one of these rules, Grafana shows an error and doesn't save it.

## Delete a transformation

We recommend that you remove transformations that you don't need. When you delete a transformation, you remove the data from the visualization.

**Before you begin:**

- Identify all dashboards that rely on the transformation and inform impacted dashboard users.

**To delete a transformation**:

1. Open a panel for editing.
1. Click the **Transform** tab.
1. Click the trash icon next to the transformation you want to delete.

{{< figure src="/static/img/docs/transformations/screenshot-example-remove-transformation.png" class="docs-image--no-shadow" max-width= "1100px" alt="A transformation row with the remove transformation icon highlighted" >}}

## Transformation functions

You can perform the following transformations on your data.

### Add field from calculation

Use this transformation to add a new field calculated from two other fields. Each transformation allows you to add one new field.

- **Mode** - Select a mode:
  - **Reduce row** - Apply selected calculation on each row of selected fields independently.
  - **Binary operation** - Apply basic binary operations (for example, sum or multiply) on values in a single row from two selected fields.
  - **Unary operation** - Apply basic unary operations on values in a single row from a selected field. The available operations are:
    - **Absolute value (abs)** - Returns the absolute value of a given expression. It represents its distance from zero as a positive number.
    - **Natural exponential (exp)** - Returns _e_ raised to the power of a given expression.
    - **Natural logarithm (ln)** - Returns the natural logarithm of a given expression.
    - **Floor (floor)** - Returns the largest integer less than or equal to a given expression.
    - **Ceiling (ceil)** - Returns the smallest integer greater than or equal to a given expression.
  - **Cumulative functions** - Apply functions on the current row and all preceding rows.
    - **Total** - Calculates the cumulative total up to and including the current row.
    - **Mean** - Calculates the mean up to and including the current row.
  - **Window functions** - Apply window functions. The window can either be **trailing** or **centered**.
    With a trailing window the current row will be the last row in the window.
    With a centered window the window will be centered on the current row.
    For even window sizes, the window will be centered between the current row, and the previous row.
    - **Mean** - Calculates the moving mean or running average.
    - **Stddev** - Calculates the moving standard deviation.
    - **Variance** - Calculates the moving variance.
  - **Row index** - Insert a field with the row index.
- **Field name** - Select the names of fields you want to use in the calculation for the new field.
- **Calculation** - If you select **Reduce row** mode, then the **Calculation** field appears. Click in the field to see a list of calculation choices you can use to create the new field. For information about available calculations, refer to [Calculation types][].
- **Operation** - If you select **Binary operation** or **Unary operation** mode, then the **Operation** fields appear. These fields allow you to apply basic math operations on values in a single row from selected fields. You can also use numerical values for binary operations.
  - **All number fields** - Set the left side of a **Binary operation** to apply the calculation to all number fields.
- **As percentile** - If you select **Row index** mode, then the **As percentile** switch appears. This switch allows you to transform the row index as a percentage of the total number of rows.
- **Alias** - (Optional) Enter the name of your new field. If you leave this blank, then the field will be named to match the calculation.
  > **Note:** If a variable is used in this transformation, the default alias will be interpolated with the value of the variable. If you want an alias to be unaffected by variable changes, explicitly define the alias.
- **Replace all fields** - (Optional) Select this option if you want to hide all other fields and display only your calculated field in the visualization.

In the example below, we added two fields together and named them Sum.

{{< figure src="/static/img/docs/transformations/add-field-from-calc-stat-example-7-0.png" class="docs-image--no-shadow" max-width= "1100px" alt="A stat visualization including one field called Sum" >}}

### Concatenate fields

Use this transformation to combine all fields from all frames into one result.

For example, if you have separate queries retrieving temperature and uptime data (Query A) and air quality index and error information (Query B), applying the concatenate transformation yields a consolidated data frame with all relevant information in one view.

Consider the following:

**Query A:**

| Temp | Uptime  |
| ---- | ------- |
| 15.4 | 1230233 |

**Query B:**

| AQI | Errors |
| --- | ------ |
| 3.2 | 5      |

After you concatenate the fields, the data frame would be:

| Temp | Uptime  | AQI | Errors |
| ---- | ------- | --- | ------ |
| 15.4 | 1230233 | 3.2 | 5      |

This transformation simplifies the process of merging data from different sources, providing a comprehensive view for analysis and visualization.

### Config from query results

Use this transformation to select a query and extract standard options, such as **Min**, **Max**, **Unit**, and **Thresholds**, and apply them to other query results. This feature enables dynamic visualization configuration based on the data returned by a specific query.

#### Options

- **Config query** - Select the query that returns the data you want to use as configuration.
- **Apply to** - Select the fields or series to which the configuration should be applied.
- **Apply to options** - Specify a field type or use a field name regex, depending on your selection in **Apply to**.

#### Field mapping table

Below the configuration options, you'll find the field mapping table. This table lists all fields found in the data returned by the config query, along with **Use as** and **Select** options. It provides control over mapping fields to config properties, and for multiple rows, it allows you to choose which value to select.

#### Example

Input[0] (From query: A, name: ServerA)

| Time          | Value |
| ------------- | ----- |
| 1626178119127 | 10    |
| 1626178119129 | 30    |

Input[1] (From query: B)

| Time          | Value |
| ------------- | ----- |
| 1626178119127 | 100   |
| 1626178119129 | 100   |

Output (Same as Input[0] but now with config on the Value field)

| Time          | Value (config: Max=100) |
| ------------- | ----------------------- |
| 1626178119127 | 10                      |
| 1626178119129 | 30                      |

Each row in the source data becomes a separate field. Each field now has a maximum configuration option set. Options such as **Min**, **Max**, **Unit**, and **Thresholds** are part of the field configuration. If set, they are used by the visualization instead of any options manually configured in the panel editor options pane.

#### Value mappings

You can also transform a query result into value mappings. With this option, every row in the configuration query result defines a single value mapping row. See the following example.

Config query result:

| Value | Text   | Color |
| ----- | ------ | ----- |
| L     | Low    | blue  |
| M     | Medium | green |
| H     | High   | red   |

In the field mapping specify:

| Field | Use as                 | Select     |
| ----- | ---------------------- | ---------- |
| Value | Value mappings / Value | All values |
| Text  | Value mappings / Text  | All values |
| Color | Value mappings / Color | All values |

Grafana builds value mappings from your query result and applies them to the real data query results. You should see values being mapped and colored according to the config query results.

> **Note:** When you use this transformation for thresholds, the visualization continues to use the panel's base threshold.

### Convert field type

Use this transformation to modify the field type of a specified field.

This transformation has the following options:

- **Field** - Select from available fields
- **as** - Select the FieldType to convert to
  - **Numeric** - attempts to make the values numbers
  - **String** - will make the values strings
  - **Time** - attempts to parse the values as time
    - The input will be parsed according to the [Moment.js parsing format](https://momentjs.com/docs/#/parsing/)
    - It will parse the numeric input as a Unix epoch timestamp in milliseconds.
      You must multiply your input by 1000 if it's in seconds.
    - Will show an option to specify a DateFormat as input by a string like yyyy-mm-dd or DD MM YYYY hh:mm:ss
    - The **Timezone** option determines how Grafana interprets input strings without timezone information. If not set, Grafana uses the browser timezone or your configured default timezone.
  - **Boolean** - will make the values booleans
  - **Enum** - will make the values enums
    - Will show a table to manage the enums
  - **Other** - attempts to parse the values as JSON

For example, consider the following query that could be modified by selecting the time field as Time and specifying Date Format as YYYY.

#### Sample Query

| Time       | Mark  | Value |
| ---------- | ----- | ----- |
| 2017-07-01 | above | 25    |
| 2018-08-02 | below | 22    |
| 2019-09-02 | below | 29    |
| 2020-10-04 | above | 22    |

The result:

#### Transformed Query

| Time                | Mark  | Value |
| ------------------- | ----- | ----- |
| 2017-01-01 00:00:00 | above | 25    |
| 2018-01-01 00:00:00 | below | 22    |
| 2019-01-01 00:00:00 | below | 29    |
| 2020-01-01 00:00:00 | above | 22    |

This transformation allows you to flexibly adapt your data types, ensuring compatibility and consistency in your visualizations.

### Extract fields

Use this transformation to select a source of data and extract content from it in different formats. This transformation has the following fields:

- **Source** - Select the field for the source of data.
- **Format** - Choose one of the following:
  - **JSON** - Parse JSON content from the source.
  - **Key+value pairs** - Parse content in the format 'a=b' or 'c:d' from the source.
  - **RegExp** - Parse content using a regular expression with [named capturing group(s)](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Regular_expressions/Named_capturing_group) like `/(?<NewField>.*)/`.
    {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-regexp-detail-v11.3-2.png" class="docs-image--no-shadow" max-width= "1100px" alt="Example of a regular expression" >}}
  - **Auto** - Discover fields automatically.
- **Replace All Fields** - (Optional) Select this option to hide all other fields and display only your calculated field in the visualization.
- **Keep Time** - (Optional) Available only if **Replace All Fields** is true. Keeps the time field in the output.

Consider the following dataset:

#### Dataset Example

| Timestamp           | json_data     |
| ------------------- | ------------- |
| 1636678740000000000 | {"value": 1}  |
| 1636678680000000000 | {"value": 5}  |
| 1636678620000000000 | {"value": 12} |

You could prepare the data to be used by a [Time series panel][] with this configuration:

- Source: json_data
- Format: JSON
  - Field: value
  - Alias: my_value
- Replace all fields: true
- Keep time: true

This will generate the following output:

#### Transformed Data

| Timestamp           | my_value |
| ------------------- | -------- |
| 1636678740000000000 | 1        |
| 1636678680000000000 | 5        |
| 1636678620000000000 | 12       |

This transformation allows you to extract and format data in various ways. You can customize the extraction format based on your specific data needs.

### Lookup fields from resource

Use this transformation to enrich a field value by looking up additional fields from an external source.

This transformation has the following fields:

- **Field** - Select a text field from your dataset.
- **Lookup** - Choose from **Countries**, **USA States**, and **Airports**.

This transformation currently supports spatial data.

For example, if you have this data:

#### Dataset Example

| Location  | Values |
| --------- | ------ |
| AL        | 0      |
| AK        | 10     |
| Arizona   | 5      |
| Arkansas  | 1      |
| Somewhere | 5      |

With this configuration:

- Field: location
- Lookup: USA States

You'll get the following output:

#### Transformed Data

| Location  | ID  | Name     | Lng         | Lat       | Values |
| --------- | --- | -------- | ----------- | --------- | ------ |
| AL        | AL  | Alabama  | -80.891064  | 12.448457 | 0      |
| AK        | AK  | Arkansas | -100.891064 | 24.448457 | 10     |
| Arizona   |     |          |             |           | 5      |
| Arkansas  |     |          |             |           | 1      |
| Somewhere |     |          |             |           | 5      |

This transformation lets you augment your data by fetching additional information from external sources, providing a more comprehensive dataset for analysis and visualization.

### Filter data by query refId

Use this transformation to hide one or more queries in panels that have multiple queries.

Grafana displays the query identification letters in dark gray text. Click a query identifier to toggle filtering. If the query letter is white, then the results are displayed. If the query letter is dark, then the results are hidden.

> **Note:** This transformation is not available for Graphite because this data source does not support correlating returned data with queries.

In the example below, the panel has three queries (A, B, C). We removed the B query from the visualization.

{{< figure src="/static/img/docs/transformations/filter-by-query-stat-example-7-0.png" class="docs-image--no-shadow" max-width= "1100px" alt="A stat visualization with results from two queries, A and C" >}}

### Filter data by values

Use this transformation to selectively filter data points directly within your visualization. This transformation provides options to include or exclude data based on one or more conditions applied to a selected field.

This transformation is very useful if your data source does not natively filter by values. You might also use this to narrow values to display if you are using a shared query.

The available conditions for all fields are:

- **Is Null** - Match if the value is null.
- **Is Not Null** - Match if the value is not null.
- **Equal** - Match if the value is equal to the specified value.
- **Not Equal** - Match if the value is not equal to the specified value.
- **Regex** - Match a regex expression.

The available conditions for string fields are:

- **Contains substring** - Match if the value contains the specified substring (case insensitive).
- **Does not contain substring** - Match if the value doesn't contain the specified substring (case insensitive).

The available conditions for number fields are:

- **Greater** - Match if the value is greater than the specified value.
- **Lower** - Match if the value is lower than the specified value.
- **Greater or equal** - Match if the value is greater or equal.
- **Lower or equal** - Match if the value is lower or equal.
- **In between** - Match a range between a specified minimum and maximum, min and max included.

The available conditions for time fields are:

- **In between** - Match a range between a specified minimum and maximum. The min and max values will pre-populate with variables to filter by selected time.

Consider the following dataset:

#### Dataset Example

| Time                | Temperature | Altitude |
| ------------------- | ----------- | -------- |
| 2020-07-07 11:34:23 | 32          | 101      |
| 2020-07-07 11:34:22 | 28          | 125      |
| 2020-07-07 11:34:21 | 26          | 110      |
| 2020-07-07 11:34:20 | 23          | 98       |
| 2020-07-07 10:32:24 | 31          | 95       |
| 2020-07-07 10:31:22 | 20          | 85       |
| 2020-07-07 09:30:57 | 19          | 101      |

If you **Include** the data points that have a temperature below 30°C, the configuration will look as follows:

- Filter Type: 'Include'
- Condition: Rows where 'Temperature' matches 'Lower Than' '30'

And you will get the following result, where only the temperatures below 30°C are included:

#### Transformed Data

| Time                | Temperature | Altitude |
| ------------------- | ----------- | -------- |
| 2020-07-07 11:34:22 | 28          | 125      |
| 2020-07-07 11:34:21 | 26          | 110      |
| 2020-07-07 11:34:20 | 23          | 98       |
| 2020-07-07 10:31:22 | 20          | 85       |
| 2020-07-07 09:30:57 | 19          | 101      |

You can add more than one condition to the filter. For example, you might want to include the data only if the altitude is greater than 100. To do so, add that condition to the following configuration:

- Filter type: 'Include' rows that 'Match All' conditions
- Condition 1: Rows where 'Temperature' matches 'Lower' than '30'
- Condition 2: Rows where 'Altitude' matches 'Greater' than '100'

When you have more than one condition, you can choose if you want the action (include/exclude) to be applied on rows that **Match all** conditions or **Match any** of the conditions you added.

In the example above, we chose **Match all** because we wanted to include the rows that have a temperature lower than 30°C _AND_ an altitude higher than 100. If we wanted to include the rows that have a temperature lower than 30°C _OR_ an altitude higher than 100 instead, then we would select **Match any**. This would include the first row in the original data, which has a temperature of 32°C (does not match the first condition) but an altitude of 101 (which matches the second condition), so it is included.

Conditions that are invalid or incompletely configured are ignored.

This versatile data filtering transformation lets you to selectively include or exclude data points based on specific conditions. Customize the criteria to tailor your data presentation to meet your unique analytical needs.

### Filter fields by name

Use this transformation to selectively remove parts of your query results. There are three ways to filter field names:

- [Using a regular expression](#use-a-regular-expression)
- [Manually selecting included fields](#manually-select-included-fields)
- [Using a dashboard variable](#use-a-dashboard-variable)

#### Use a regular expression

When you filter using a regular expression, field names that match the regular expression are included.

For example, from the input data:

| Time                | dev-eu-west | dev-eu-north | prod-eu-west | prod-eu-north |
| ------------------- | ----------- | ------------ | ------------ | ------------- |
| 2023-03-04 23:56:23 | 23.5        | 24.5         | 22.2         | 20.2          |
| 2023-03-04 23:56:23 | 23.6        | 24.4         | 22.1         | 20.1          |

The result from using the regular expression 'prod.\*' would be:

| Time                | prod-eu-west | prod-eu-north |
| ------------------- | ------------ | ------------- |
| 2023-03-04 23:56:23 | 22.2         | 20.2          |
| 2023-03-04 23:56:23 | 22.1         | 20.1          |

The regular expression can include an interpolated dashboard variable by using the ${variableName} syntax.

#### Manually select included fields

Click and uncheck the field names to remove them from the result. Fields that are matched by the regular expression are still included, even if they're unchecked.

#### Use a dashboard variable

Enable 'From variable' to let you select a dashboard variable that's used to include fields. By setting up a [dashboard variable][] with multiple choices, the same fields can be displayed across multiple visualizations.

{{< figure src="/static/img/docs/transformations/filter-name-table-before-7-0.png" class="docs-image--no-shadow" max-width= "1100px" alt="A table visualization with time, value, Min, and Max columns" >}}

Here's the table after we applied the transformation to remove the Min field.

{{< figure src="/static/img/docs/transformations/filter-name-table-after-7-0.png" class="docs-image--no-shadow" max-width= "1100px" alt="A table visualization with time, value, and Max columns" >}}

Here is the same query using a Stat visualization.

{{< figure src="/static/img/docs/transformations/filter-name-stat-after-7-0.png" class="docs-image--no-shadow" max-width= "1100px" alt="A stat visualization with value and Max fields" >}}

This transformation provides flexibility in tailoring your query results to focus on the specific fields you need for effective analysis and visualization.

### Format string

Use this transformation to customize the output of a string field. This transformation has the following fields:

- **Upper case** - Formats the entire string in uppercase characters.
- **Lower case** - Formats the entire string in lowercase characters.
- **Sentence case** - Formats the first character of the string in uppercase.
- **Title case** - Formats the first character of each word in the string in uppercase.
- **Pascal case** - Formats the first character of each word in the string in uppercase and doesn't include spaces between words.
- **Camel case** - Formats the first character of each word in the string in uppercase, except the first word, and doesn't include spaces between words.
- **Snake case** - Formats all characters in the string in lowercase and uses underscores instead of spaces between words.
- **Kebab case** - Formats all characters in the string in lowercase and uses dashes instead of spaces between words.
- **Trim** - Removes all leading and trailing spaces from the string.
- **Substring** - Returns a substring of the string, using the specified start and end positions.

This transformation provides a convenient way to standardize and tailor the presentation of string data for better visualization and analysis.

### Format time

Use this transformation to customize the output of a time field. Output can be formatted using [Moment.js format strings](https://momentjs.com/docs/#/displaying/). For example, if you want to display only the year of a time field, the format string 'YYYY' can be used to show the calendar year (for example, 1999 or 2012).

**Before Transformation:**

| Timestamp           | Event        |
| ------------------- | ------------ |
| 1636678740000000000 | System Start |
| 1636678680000000000 | User Login   |
| 1636678620000000000 | Data Updated |

**After applying 'YYYY-MM-DD HH:mm:ss':**

| Timestamp           | Event        |
| ------------------- | ------------ |
| 2021-11-12 14:25:40 | System Start |
| 2021-11-12 14:24:40 | User Login   |
| 2021-11-12 14:23:40 | Data Updated |

This transformation lets you tailor the time representation in your visualizations, providing flexibility and precision in displaying temporal data.

> **Note:** This transformation is available in Grafana 10.1+ as an alpha feature.

### Group by

Use this transformation to group the data by a specified field (column) value and process calculations on each group. Click to see a list of calculation choices. For information about available calculations, refer to [Calculation types][].

Here's an example of original data.

| Time                | Server ID | CPU Temperature | Server Status |
| ------------------- | --------- | --------------- | ------------- |
| 2020-07-07 11:34:20 | server 1  | 80              | Shutdown      |
| 2020-07-07 11:34:20 | server 3  | 62              | OK            |
| 2020-07-07 10:32:20 | server 2  | 90              | Overload      |
| 2020-07-07 10:31:22 | server 3  | 55              | OK            |
| 2020-07-07 09:30:57 | server 3  | 62              | Rebooting     |
| 2020-07-07 09:30:05 | server 2  | 88              | OK            |
| 2020-07-07 09:28:06 | server 1  | 80              | OK            |
| 2020-07-07 09:25:05 | server 2  | 88              | OK            |
| 2020-07-07 09:23:07 | server 1  | 86              | OK            |

This transformation goes in two steps. First you specify one or multiple fields to group the data by. This will group all the same values of those fields together, as if you sorted them. For instance if we group by the Server ID field, then it would group the data this way:

| Time                | Server ID      | CPU Temperature | Server Status |
| ------------------- | -------------- | --------------- | ------------- |
| 2020-07-07 11:34:20 | **server 1**   | 80              | Shutdown      |
| 2020-07-07 09:28:06 | **server 1**   | 80              | OK            |
| 2020-07-07 09:23:07 | **server 1**   | 86              | OK            |
| 2020-07-07 10:32:20 | server 2       | 90              | Overload      |
| 2020-07-07 09:30:05 | server 2       | 88              | OK            |
| 2020-07-07 09:25:05 | server 2       | 88              | OK            |
| 2020-07-07 11:34:20 | **_server 3_** | 62              | OK            |
| 2020-07-07 10:31:22 | **_server 3_** | 55              | OK            |
| 2020-07-07 09:30:57 | **_server 3_** | 62              | Rebooting     |

All rows with the same value of Server ID are grouped together. Optionally, you can add a count of how may values fall in the selected group.

After choosing which field you want to group your data by, you can add various calculations on the other fields, and apply the calculation to each group of rows. For instance, we could want to calculate the average CPU temperature for each of those servers. So we can add the _mean_ calculation applied on the CPU Temperature field to get the following:

| Server ID | CPU Temperature (mean) |
| --------- | ---------------------- |
| server 1  | 82                     |
| server 2  | 88.6                   |
| server 3  | 59.6                   |

If you had added the count stat to the group by transformation, there would be an extra column showing that the count of each server from above was 3.

| Server ID | CPU Temperature (mean) | Server ID (count) |
| --------- | ---------------------- | ----------------- |
| server 1  | 82                     | 3                 |
| server 2  | 88.6                   | 3                 |
| server 3  | 59.6                   | 3                 |

And we can add more than one calculation. For instance:

- For field Time, we can calculate the _Last_ value, to know when the last data point was received for each server
- For field Server Status, we can calculate the _Last_ value to know what is the last state value for each server
- For field Temperature, we can also calculate the _Last_ value to know what is the latest monitored temperature for each server

We would then get:

| Server ID | CPU Temperature (mean) | CPU Temperature (last) | Time (last)         | Server Status (last) |
| --------- | ---------------------- | ---------------------- | ------------------- | -------------------- |
| server 1  | 82                     | 80                     | 2020-07-07 11:34:20 | Shutdown             |
| server 2  | 88.6                   | 90                     | 2020-07-07 10:32:20 | Overload             |
| server 3  | 59.6                   | 62                     | 2020-07-07 11:34:20 | OK                   |

This transformation allows you to extract essential information from your time series and present it conveniently.

### Grouping to matrix

Use this transformation to combine three fields—which are used as input for the **Column**, **Row**, and **Cell value** fields from the query output—and generate a matrix. The matrix is calculated as follows:

**Original data**

| Server ID | CPU Temperature | Server Status |
| --------- | --------------- | ------------- |
| server 1  | 82              | OK            |
| server 2  | 88.6            | OK            |
| server 3  | 59.6            | Shutdown      |

We can generate a matrix using the values of 'Server Status' as column names, the 'Server ID' values as row names, and the 'CPU Temperature' as content of each cell. The content of each cell will appear for the existing column ('Server Status') and row combination ('Server ID'). For the rest of the cells, you can select which value to display between: **Null**, **True**, **False**, or **Empty**.

**Output**

| Server IDServer Status | OK   | Shutdown |
| ---------------------- | ---- | -------- |
| server 1               | 82   |          |
| server 2               | 88.6 |          |
| server 3               |      | 59.6     |

Use this transformation to construct a matrix by specifying fields from your query results. The matrix output reflects the relationships between the unique values in these fields. This helps you present complex relationships in a clear and structured matrix format.

### Group to nested tables

Use this transformation to group the data by a specified field (column) value and process calculations on each group. Records are generated that share the same grouped field value, to be displayed in a nested table.

To calculate a statistic for a field, click the selection box next to it and select the **Calculate** option:

{{< figure src="/static/img/docs/transformations/nested-table-select-calculation.png" class="docs-image--no-shadow" max-width= "1100px" alt="A select box showing the Group and Calculate options for the transformation." >}}

Once **Calculate** has been selected, another selection box will appear next to the respective field which will allow statistics to be selected:

{{< figure src="/static/img/docs/transformations/nested-table-select-stat.png" class="docs-image--no-shadow" max-width= "1100px" alt="A select box showing available statistic calculations once the calculate option for the field has been selected." >}}

For information about available calculations, refer to [Calculation types][].

Here's an example of original data:

| Time                | Server ID | CPU Temperature | Server Status |
| ------------------- | --------- | --------------- | ------------- |
| 2020-07-07 11:34:20 | server 1  | 80              | Shutdown      |
| 2020-07-07 11:34:20 | server 3  | 62              | OK            |
| 2020-07-07 10:32:20 | server 2  | 90              | Overload      |
| 2020-07-07 10:31:22 | server 3  | 55              | OK            |
| 2020-07-07 09:30:57 | server 3  | 62              | Rebooting     |
| 2020-07-07 09:30:05 | server 2  | 88              | OK            |
| 2020-07-07 09:28:06 | server 1  | 80              | OK            |
| 2020-07-07 09:25:05 | server 2  | 88              | OK            |
| 2020-07-07 09:23:07 | server 1  | 86              | OK            |

This transformation has two steps. First, specify one or more fields by which to group the data. This groups all the same values of those fields together, as if you sorted them. For instance, if you group by the Server ID field, Grafana groups the data this way:

| Server ID |                                                                                                                                                                                                                                                                                                     |
| --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| server 1  | <table><th><tr><td>Time</td><td>CPU Temperature</td><td>Server Status</td></tr></th><tbody><tr><td>2020-07-07 11:34:20</td><td>80</td><td>Shutdown</td></tr><tr><td>2020-07-07 09:28:06</td><td>80</td><td>OK</td></tr><tr><td>2020-07-07 09:23:07</td><td>86</td><td>OK</td></tr></tbody></table>  |
| server 2  | <table><th><tr><td>Time</td><td>CPU Temperature</td><td>Server Status</td></tr></th><tbody><tr><td>2020-07-07 10:32:20</td><td>90</td><td>Overload</td></tr><tr><td>2020-07-07 09:30:05</td><td>88</td><td>OK</td></tr><tr><td>2020-07-07 09:25:05</td><td>88</td><td>OK</td></tr></tbody></table>  |
| server 3  | <table><th><tr><td>Time</td><td>CPU Temperature</td><td>Server Status</td></tr></th><tbody><tr><td>2020-07-07 11:34:20</td><td>62</td><td>OK</td></tr><tr><td>2020-07-07 10:31:22</td><td>55</td><td>OK</td></tr><tr><td>2020-07-07 09:30:57</td><td>62</td><td>Rebooting</td></tr></tbody></table> |

After choosing the field by which you want to group your data, you can add various calculations on the other fields and apply the calculation to each group of rows. For instance, you might want to calculate the average CPU temperature for each of those servers. To do so, add the **mean calculation** applied on the CPU Temperature field to get the following result:

| Server ID | CPU Temperatute (mean) |                                                                                                                                                                                                                                            |
| --------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| server 1  | 82                     | <table><th><tr><td>Time</td><td>Server Status</td></tr></th><tbody><tr><td>2020-07-07 11:34:20</td><td>Shutdown</td></tr><tr><td>2020-07-07 09:28:06</td><td>OK</td></tr><tr><td>2020-07-07 09:23:07</td><td>OK</td></tr></tbody></table>  |
| server 2  | 88.6                   | <table><th><tr><td>Time</td><td>Server Status</td></tr></th><tbody><tr><td>2020-07-07 10:32:20</td><td>Overload</td></tr><tr><td>2020-07-07 09:30:05</td><td>OK</td></tr><tr><td>2020-07-07 09:25:05</td><td>OK</td></tr></tbody></table>  |
| server 3  | 59.6                   | <table><th><tr><td>Time</td><td>Server Status</td></tr></th><tbody><tr><td>2020-07-07 11:34:20</td><td>OK</td></tr><tr><td>2020-07-07 10:31:22</td><td>OK</td></tr><tr><td>2020-07-07 09:30:57</td><td>Rebooting</td></tr></tbody></table> |

#### Display options

| Option                            | Description                                                                                                                |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Show field names in nested tables | Show or hide the column headers inside each nested table. On by default                                                    |
| Expand nested rows by default     | Expand all nested rows automatically when the panel loads, instead of starting with all columns collapsed. Off by default. |

### Create heatmap

Use this transformation to prepare histogram data for visualizing trends over time. Similar to the heatmap visualization, this transformation converts histogram metrics into temporal buckets.

#### X Bucket

This setting determines how the x-axis is split into buckets.

- **Size** - Specify a time interval in the input field. For example, a time range of '1h' creates cells one hour wide on the x-axis.
- **Count** - For non-time-related series, use this option to define the number of elements in a bucket.

#### Y Bucket

This setting determines how the y-axis is split into buckets.

- **Linear**
- **Logarithmic** - Choose between log base 2 or log base 10.
- **Symlog** - Uses a symmetrical logarithmic scale. Choose between log base 2 or log base 10, allowing for negative values.

Assume you have the following dataset:

| Timestamp           | Value |
| ------------------- | ----- |
| 2023-01-01 12:00:00 | 5     |
| 2023-01-01 12:15:00 | 10    |
| 2023-01-01 12:30:00 | 15    |
| 2023-01-01 12:45:00 | 8     |

- With X Bucket set to 'Size: 15m' and Y Bucket as 'Linear', the histogram organizes values into time intervals of 15 minutes on the x-axis and linearly on the y-axis.
- For X Bucket as 'Count: 2' and Y Bucket as 'Logarithmic (base 10)', the histogram groups values into buckets of two on the x-axis and use a logarithmic scale on the y-axis.

### Histogram

Use this transformation to generate a histogram based on input data, allowing you to visualize the distribution of values.

- **Bucket size** - The range between the lowest and highest items in a bucket (xMin to xMax).
- **Bucket offset** - The offset for non-zero-based buckets.
- **Combine series** - Create a unified histogram using all available series.

**Original data**

Series 1:

| A   | B   | C   |
| --- | --- | --- |
| 1   | 3   | 5   |
| 2   | 4   | 6   |
| 3   | 5   | 7   |
| 4   | 6   | 8   |
| 5   | 7   | 9   |

Series 2:

| C   |
| --- |
| 5   |
| 6   |
| 7   |
| 8   |
| 9   |

**Output**

| xMin | xMax | A   | B   | C   | C   |
| ---- | ---- | --- | --- | --- | --- |
| 1    | 2    | 1   | 0   | 0   | 0   |
| 2    | 3    | 1   | 0   | 0   | 0   |
| 3    | 4    | 1   | 1   | 0   | 0   |
| 4    | 5    | 1   | 1   | 0   | 0   |
| 5    | 6    | 1   | 1   | 1   | 1   |
| 6    | 7    | 0   | 1   | 1   | 1   |
| 7    | 8    | 0   | 1   | 1   | 1   |
| 8    | 9    | 0   | 0   | 1   | 1   |
| 9    | 10   | 0   | 0   | 1   | 1   |

Visualize the distribution of values using the generated histogram, providing insights into the data's spread and density.

### Join by field

Use this transformation to merge multiple results into a single table, enabling the consolidation of data from different queries.

This is especially useful for converting multiple time series results into a single wide table with a shared time field.

#### Inner join (for Time Series or SQL-like data)

An inner join merges data from multiple tables where all tables share the same value from the selected field. This type of join excludes data where values do not match in every result.

Use this transformation to combine the results from multiple queries (combining on a passed join field or the first time column) into one result, and drop rows where a successful join cannot occur. This is not optimized for large Time Series datasets.

In the following example, two queries return Time Series data. It is visualized as two separate tables before applying the inner join transformation.

**Query A:**

| Time                | Job     | Uptime    |
| ------------------- | ------- | --------- |
| 2020-07-07 11:34:20 | node    | 25260122  |
| 2020-07-07 11:24:20 | postgre | 123001233 |
| 2020-07-07 11:14:20 | postgre | 345001233 |

**Query B:**

| Time                | Server   | Errors |
| ------------------- | -------- | ------ |
| 2020-07-07 11:34:20 | server 1 | 15     |
| 2020-07-07 11:24:20 | server 2 | 5      |
| 2020-07-07 11:04:20 | server 3 | 10     |

The result after applying the inner join transformation looks like the following:

| Time                | Job     | Uptime    | Server   | Errors |
| ------------------- | ------- | --------- | -------- | ------ |
| 2020-07-07 11:34:20 | node    | 25260122  | server 1 | 15     |
| 2020-07-07 11:24:20 | postgre | 123001233 | server 2 | 5      |

This works in the same way for non-Time Series tabular data as well.

**Students**

| StudentID | Name     | Major            |
| --------- | -------- | ---------------- |
| 1         | John     | Computer Science |
| 2         | Emily    | Mathematics      |
| 3         | Michael  | Physics          |
| 4         | Jennifer | Chemistry        |

**Enrollments**

| StudentID | CourseID | Grade |
| --------- | -------- | ----- |
| 1         | CS101    | A     |
| 1         | CS102    | B     |
| 2         | MATH201  | A     |
| 3         | PHYS101  | B     |
| 5         | HIST101  | B     |

The result after applying the inner join transformation looks like the following:

| StudentID | Name    | Major            | CourseID | Grade |
| --------- | ------- | ---------------- | -------- | ----- |
| 1         | John    | Computer Science | CS101    | A     |
| 1         | John    | Computer Science | CS102    | B     |
| 2         | Emily   | Mathematics      | MATH201  | A     |
| 3         | Michael | Physics          | PHYS101  | B     |

The inner join only includes rows where there is a match between the "StudentID" in both tables. In this case, the result does not include "Jennifer" from the "Students" table because there are no matching enrollments for her in the "Enrollments" table.

If a query returns data that doesn't contain the join field, then no row can match in every query, so the result is empty. To join only some of your queries, add a [Filter data by query refId](#filter-data-by-query-refid) transformation before the join.

#### Outer join (for Time Series data)

An outer join includes all data from an inner join and rows where values do not match in every input. While the inner join joins Query A and Query B on the time field, the outer join includes all rows that don't match on the time field.

In the following example, two queries return table data. It is visualized as two tables before applying the outer join transformation.

**Query A:**

| Time                | Job     | Uptime    |
| ------------------- | ------- | --------- |
| 2020-07-07 11:34:20 | node    | 25260122  |
| 2020-07-07 11:24:20 | postgre | 123001233 |
| 2020-07-07 11:14:20 | postgre | 345001233 |

**Query B:**

| Time                | Server   | Errors |
| ------------------- | -------- | ------ |
| 2020-07-07 11:34:20 | server 1 | 15     |
| 2020-07-07 11:24:20 | server 2 | 5      |
| 2020-07-07 11:04:20 | server 3 | 10     |

The result after applying the outer join transformation looks like the following:

| Time                | Job     | Uptime    | Server   | Errors |
| ------------------- | ------- | --------- | -------- | ------ |
| 2020-07-07 11:04:20 |         |           | server 3 | 10     |
| 2020-07-07 11:14:20 | postgre | 345001233 |          |        |
| 2020-07-07 11:34:20 | node    | 25260122  | server 1 | 15     |
| 2020-07-07 11:24:20 | postgre | 123001233 | server 2 | 5      |

In the following example, a template query displays time series data from multiple servers in a table visualization. The results of only one query can be viewed at a time.

{{< figure src="/static/img/docs/transformations/join-fields-before-7-0.png" class="docs-image--no-shadow" max-width= "1100px" alt="A table visualization showing results for one server" >}}

I applied a transformation to join the query results using the time field. Now I can run calculations, combine, and organize the results in this new table.

{{< figure src="/static/img/docs/transformations/join-fields-after-7-0.png" class="docs-image--no-shadow" max-width= "1100px" alt="A table visualization showing results for multiple servers" >}}

#### Outer join (for SQL-like data)

A tabular outer join combining tables so that the result includes matched and unmatched rows from either or both tables.

| StudentID | Name     | Major            |
| --------- | -------- | ---------------- |
| 1         | John     | Computer Science |
| 2         | Emily    | Mathematics      |
| 3         | Michael  | Physics          |
| 4         | Jennifer | Chemistry        |

Can now be joined with:

| StudentID | CourseID | Grade |
| --------- | -------- | ----- |
| 1         | CS101    | A     |
| 1         | CS102    | B     |
| 2         | MATH201  | A     |
| 3         | PHYS101  | B     |
| 5         | HIST101  | B     |

The result after applying the outer join transformation looks like the following:

| StudentID | Name     | Major            | CourseID | Grade |
| --------- | -------- | ---------------- | -------- | ----- |
| 1         | John     | Computer Science | CS101    | A     |
| 1         | John     | Computer Science | CS102    | B     |
| 2         | Emily    | Mathematics      | MATH201  | A     |
| 3         | Michael  | Physics          | PHYS101  | B     |
| 4         | Jennifer | Chemistry        | NULL     | NULL  |
| 5         | NULL     | NULL             | HIST101  | B     |

Combine and analyze data from various queries with table joining for a comprehensive view of your information.

#### Join on multiple fields

A join drops any query that doesn’t contain the field you’re joining on. So if you try to add another **Join by field** transformation using a different field, some of the queries you need may already have been removed by the first join.

Use the **Keep unjoined** option to maintain those queries instead of dropping them, so that a later **Join by field** transformation can join them on a different field. Enable it on every join except the last one.

For example, one query returns measurements with two IDs, and the query run on each ID returns that ID's name:

**Query A:**

| GroupID | ProductID | Value |
| ------- | --------- | ----- |
| g1      | p1        | 1     |

**Query B:**

| GroupID | GroupName |
| ------- | --------- |
| g1      | Alpha     |

**Query C:**

| ProductID | ProductName |
| --------- | ----------- |
| p1        | Widget      |

Add two transformations:

1. **Join by field** on `GroupID`, with **Keep unjoined** enabled. Query C has no `GroupID`, so it's passed through rather than dropped.
1. **Join by field** on `ProductID`. The result of the first join still has a `ProductID` column, so it joins with query C.

The result looks like this:

| ProductID | GroupID | Value | GroupName | ProductName |
| --------- | ------- | ----- | --------- | ----------- |
| p1        | g1      | 1     | Alpha     | Widget      |

The field being joined on moves to the first column, so the column order changes at each step. Add an **Organize fields by name** transformation afterwards if you need a specific order.

Leave **Keep unjoined** disabled unless you're chaining joins. When it's disabled, a query that doesn't contain the selected field is treated as a failed input to the join. As a result, an inner join correctly returns no rows if a query doesn't contain that field.

### Join by labels

Use this transformation to join multiple results into a single table.

This is especially useful for converting multiple time series results into a single wide table with a shared **Label** field.

- **Join** - Select the label to join by between the labels available or common across all time series.
- **Value** - The name for the output result.

#### Example

##### Input

series1{what="Temp", cluster="A", job="J1"}

| Time | Value |
| ---- | ----- |
| 1    | 10    |
| 2    | 200   |

series2{what="Temp", cluster="B", job="J1"}

| Time | Value |
| ---- | ----- |
| 1    | 10    |
| 2    | 200   |

series3{what="Speed", cluster="B", job="J1"}

| Time | Value |
| ---- | ----- |
| 22   | 22    |
| 28   | 77    |

##### Config

value: "what"

##### Output

| cluster | job | Temp | Speed |
| ------- | --- | ---- | ----- |
| A       | J1  | 10   |       |
| A       | J1  | 200  |       |
| B       | J1  | 10   | 22    |
| B       | J1  | 200  | 77    |

Combine and organize time series data effectively with this transformation for comprehensive insights.

### Labels to fields

Use this transformation to convert time series results with labels or tags into a table, including each label's keys and values in the result. Display labels as either columns or row values for enhanced data visualization.

Given a query result of two time series:

- Series 1: labels Server=Server A, Datacenter=EU
- Series 2: labels Server=Server B, Datacenter=EU

In "Columns" mode, the result looks like this:

| Time                | Server   | Datacenter | Value |
| ------------------- | -------- | ---------- | ----- |
| 2020-07-07 11:34:20 | Server A | EU         | 1     |
| 2020-07-07 11:34:20 | Server B | EU         | 2     |

In "Rows" mode, the result has a table for each series and show each label value like this:

| label      | value    |
| ---------- | -------- |
| Server     | Server A |
| Datacenter | EU       |

| label      | value    |
| ---------- | -------- |
| Server     | Server B |
| Datacenter | EU       |

#### Value field name

If you selected Server as the **Value field name**, then you would get one field for every value of the Server label.

| Time                | Datacenter | Server A | Server B |
| ------------------- | ---------- | -------- | -------- |
| 2020-07-07 11:34:20 | EU         | 1        | 2        |

#### Merging behavior

The labels to fields transformer is internally two separate transformations. The first acts on single series and extracts labels to fields. The second is the [merge](#merge) transformation that joins all the results into a single table. The merge transformation tries to join on all matching fields. This merge step is required and cannot be turned off.

To illustrate this, here is an example where you have two queries that return time series with no overlapping labels.

- Series 1: labels Server=ServerA
- Series 2: labels Datacenter=EU

This will first result in these two tables:

| Time                | Server  | Value |
| ------------------- | ------- | ----- |
| 2020-07-07 11:34:20 | ServerA | 10    |

| Time                | Datacenter | Value |
| ------------------- | ---------- | ----- |
| 2020-07-07 11:34:20 | EU         | 20    |

After merge:

| Time                | Server  | Value | Datacenter |
| ------------------- | ------- | ----- | ---------- |
| 2020-07-07 11:34:20 | ServerA | 10    |            |
| 2020-07-07 11:34:20 |         | 20    | EU         |

Convert your time series data into a structured table format for a clearer and more organized representation.

### Limit

Use this transformation to restrict the number of rows displayed, providing a more focused view of your data. This is particularly useful when dealing with large datasets.

Below is an example illustrating the impact of the **Limit** transformation on a response from a data source:

| Time                | Metric      | Value |
| ------------------- | ----------- | ----- |
| 2020-07-07 11:34:20 | Temperature | 25    |
| 2020-07-07 11:34:20 | Humidity    | 22    |
| 2020-07-07 10:32:20 | Humidity    | 29    |
| 2020-07-07 10:31:22 | Temperature | 22    |
| 2020-07-07 09:30:57 | Humidity    | 33    |
| 2020-07-07 09:30:05 | Temperature | 19    |

Here is the result after adding a Limit transformation with a value of '3':

| Time                | Metric      | Value |
| ------------------- | ----------- | ----- |
| 2020-07-07 11:34:20 | Temperature | 25    |
| 2020-07-07 11:34:20 | Humidity    | 22    |
| 2020-07-07 10:32:20 | Humidity    | 29    |

Using a negative number, you can keep values from the end of the set. Here is the result after adding a Limit transformation with a value of '-3':

| Time                | Metric      | Value |
| ------------------- | ----------- | ----- |
| 2020-07-07 10:31:22 | Temperature | 22    |
| 2020-07-07 09:30:57 | Humidity    | 33    |
| 2020-07-07 09:30:05 | Temperature | 19    |

This transformation helps you tailor the visual presentation of your data to focus on the most relevant information.

### Merge series/tables

Use this transformation to combine the results from multiple queries into a single result, which is particularly useful when using the table panel visualization. This transformation merges values into the same row if the shared fields contain the same data.

Here's an example illustrating the impact of the **Merge series/tables** transformation on two queries returning table data:

**Query A:**

| Time                | Job     | Uptime    |
| ------------------- | ------- | --------- |
| 2020-07-07 11:34:20 | node    | 25260122  |
| 2020-07-07 11:24:20 | postgre | 123001233 |

**Query B:**

| Time                | Job     | Errors |
| ------------------- | ------- | ------ |
| 2020-07-07 11:34:20 | node    | 15     |
| 2020-07-07 11:24:20 | postgre | 5      |

Here is the result after applying the Merge transformation.

| Time                | Job     | Errors | Uptime    |
| ------------------- | ------- | ------ | --------- |
| 2020-07-07 11:34:20 | node    | 15     | 25260122  |
| 2020-07-07 11:24:20 | postgre | 5      | 123001233 |

This transformation combines values from Query A and Query B into a unified table, enhancing the presentation of data for better insights.

### Organize fields by name

Use this transformation to provide the flexibility to rename, reorder, or hide fields returned by a single query in your panel. This transformation is applicable only to panels with a single query. If your panel has multiple queries, consider using an "Outer join" transformation or removing extra queries.

#### Transforming fields

Grafana displays a list of fields returned by the query, allowing you to perform the following actions:

- **Set field order mode** - If the mode is **Manual**, you can change the field order by hovering the cursor over a field and dragging the field to its new position. If it's **Auto**, use the **OFF**, **ASC**, and **DESC** options to order by any labels on the field or by the field name. For any field that is sorted **ASC** or **DESC**, you can drag the option to set the priority of the sorting.
- **Change field order** - Hover over a field, and when your cursor turns into a hand, drag the field to its new position.
- **Hide or show a field** - Use the eye icon next to the field name to toggle the visibility of a specific field.
- **Rename fields** - Type a new name in the "Rename <field>" box to customize field names.

#### Example:

##### Original Query Result

| Time                | Metric      | Value |
| ------------------- | ----------- | ----- |
| 2020-07-07 11:34:20 | Temperature | 25    |
| 2020-07-07 11:34:20 | Humidity    | 22    |
| 2020-07-07 10:32:20 | Humidity    | 29    |

##### After Applying Field Overrides

| Time                | Sensor      | Reading |
| ------------------- | ----------- | ------- |
| 2020-07-07 11:34:20 | Temperature | 25      |
| 2020-07-07 11:34:20 | Humidity    | 22      |
| 2020-07-07 10:32:20 | Humidity    | 29      |

This transformation lets you to tailor the display of query results, ensuring a clear and insightful representation of your data in Grafana.

### Partition by values

Use this transformation to streamline the process of graphing multiple series without the need for multiple queries with different 'WHERE' clauses.

This is particularly useful when dealing with a metrics SQL table, as illustrated below:

| Time                | Region | Value |
| ------------------- | ------ | ----- |
| 2022-10-20 12:00:00 | US     | 1520  |
| 2022-10-20 12:00:00 | EU     | 2936  |
| 2022-10-20 01:00:00 | US     | 1327  |
| 2022-10-20 01:00:00 | EU     | 912   |

With the **Partition by values** transformation, you can issue a single query and split the results by unique values in one or more columns (fields) of your choosing. The following example uses 'Region':

'SELECT Time, Region, Value FROM metrics WHERE Time > "2022-10-20"'

| Time                | Region | Value |
| ------------------- | ------ | ----- |
| 2022-10-20 12:00:00 | US     | 1520  |
| 2022-10-20 01:00:00 | US     | 1327  |

| Time                | Region | Value |
| ------------------- | ------ | ----- |
| 2022-10-20 12:00:00 | EU     | 2936  |
| 2022-10-20 01:00:00 | EU     | 912   |

This transformation simplifies the process and enhances the flexibility of visualizing multiple series within the same time series visualization.

### Prepare time series

Use this transformation to address issues when a data source returns time series data in a format that isn't compatible with the desired visualization. This transformation allows you to convert time series data between wide and long formats, providing flexibility in data frame structures.

#### Available options

##### Wide time series

Select this option to transform the time series data frame from the long format to the wide format. If your data source returns time series data in a long format and your visualization requires a wide format, this transformation simplifies the process.

A wide time series combines data into a single frame with one shared, ascending time field. Time fields do not repeat and multiple values extend in separate columns.

**Example: Converting from long to wide format**

| Timestamp           | Variable | Value |
| ------------------- | -------- | ----- |
| 2023-01-01 00:00:00 | Value1   | 10    |
| 2023-01-01 00:00:00 | Value2   | 20    |
| 2023-01-01 01:00:00 | Value1   | 15    |
| 2023-01-01 01:00:00 | Value2   | 25    |

**Transformed to:**

| Timestamp           | Value1 | Value2 |
| ------------------- | ------ | ------ |
| 2023-01-01 00:00:00 | 10     | 20     |
| 2023-01-01 01:00:00 | 15     | 25     |

##### Multi-frame time series

Multi-frame time series break data into multiple frames that all contain two fields: a time field and a numeric value field. Time is always ascending. String values are represented as field labels.

##### Long time series

A long time series combines data into one frame, with the first field being an ascending time field. The time field might have duplicates. String values are in separate fields, and there might be more than one.

**Example: Converting to long format**

| Value1 | Value2 | Timestamp           |
| ------ | ------ | ------------------- |
| 10     | 20     | 2023-01-03 00:00:00 |
| 30     | 40     | 2023-01-02 00:00:00 |
| 50     | 60     | 2023-01-01 00:00:00 |
| 70     | 80     | 2023-01-01 00:00:00 |

**Transformed to:**

| Timestamp           | Value1 | Value2 |
| ------------------- | ------ | ------ |
| 2023-01-01 00:00:00 | 70     | 80     |
| 2023-01-01 01:00:00 | 50     | 60     |
| 2023-01-02 01:00:00 | 30     | 40     |
| 2023-01-03 01:00:00 | 10     | 20     |

### Reduce

Use this transformation to apply a calculation to each field in the data frame and return a single value. This transformation is particularly useful for consolidating multiple time series data into a more compact, summarized format. Time fields are removed when applying this transformation.

Consider the input:

**Query A:**

| Time                | Temp | Uptime  |
| ------------------- | ---- | ------- |
| 2020-07-07 11:34:20 | 12.3 | 256122  |
| 2020-07-07 11:24:20 | 15.4 | 1230233 |

**Query B:**

| Time                | AQI | Errors |
| ------------------- | --- | ------ |
| 2020-07-07 11:34:20 | 6.5 | 15     |
| 2020-07-07 11:24:20 | 3.2 | 5      |

The reduce transformer has two modes:

- **Series to rows** - Creates a row for each field and a column for each calculation.
- **Reduce fields** - Keeps the existing frame structure, but collapses each field into a single value.

For example, if you used the **First** and **Last** calculation with a **Series to rows** transformation, then
the result would be:

| Field  | First  | Last    |
| ------ | ------ | ------- |
| Temp   | 12.3   | 15.4    |
| Uptime | 256122 | 1230233 |
| AQI    | 6.5    | 3.2     |
| Errors | 15     | 5       |

The **Reduce fields** with the **Last** calculation,
results in two frames, each with one row:

**Query A:**

| Temp | Uptime  |
| ---- | ------- |
| 15.4 | 1230233 |

**Query B:**

| AQI | Errors |
| --- | ------ |
| 3.2 | 5      |

This flexible transformation simplifies the process of consolidating and summarizing data from multiple time series into a more manageable and organized format.

### Rename by regex

Use this transformation to rename parts of the query results using a regular expression and replacement pattern.

You can specify a regular expression, which is only applied to matches, along with a replacement pattern that support back references. For example, let's imagine you're visualizing CPU usage per host and you want to remove the domain name. You could set the regex to '/^([^.]+).\*/' and the replacement pattern to '$1', 'web-01.example.com' would become 'web-01'.

> **Note:** The Rename by regex transformation was improved in Grafana v9.0.0 to allow global patterns of the form '/<stringToReplace>/g'. Depending on the regex match used, this may cause some transformations to behave slightly differently. You can guarantee the same behavior as before by wrapping the match string in forward slashes '(/)', e.g. '(._)' would become '/(._)/'.

In the following example, we are stripping the 'A-' prefix from field names. In the before image, you can see everything is prefixed with 'A-':

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-rename-by-regex-before-v11.0.png" class="docs-image--no-shadow" max-width= "1100px" alt="A time series with full series names" >}}

With the transformation applied, you can see we are left with just the remainder of the string.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-rename-by-regex-after-v11.0.png" class="docs-image--no-shadow" max-width= "1100px" alt="A time series with shortened series names" >}}

This transformation lets you to tailor your data to meet your visualization needs, making your dashboards more informative and user-friendly.

### Rows to fields

Use this transformation to convert rows into separate fields. This can be useful because fields can be styled and configured individually. It can also use additional fields as sources for dynamic field configuration or map them to field labels. The additional labels can then be used to define better display names for the resulting fields.

This transformation includes a field table which lists all fields in the data returned by the configuration query. This table gives you control over what field should be mapped to each configuration property (the **Use as** option). You can also choose which value to select if there are multiple rows in the returned data.

This transformation requires:

- One field to use as the source of field names.

  By default, the transform uses the first string field as the source. You can override this default setting by selecting **Field name** in the **Use as** column for the field you want to use instead.

- One field to use as the source of values.

  By default, the transform uses the first number field as the source. But you can override this default setting by selecting **Field value** in the **Use as** column for the field you want to use instead.

Useful when visualizing data in:

- Gauge
- Stat
- Pie chart

#### Map extra fields to labels

If a field does not map to config property Grafana will automatically use it as source for a label on the output field-

**Example:**

| Name    | DataCenter | Value |
| ------- | ---------- | ----- |
| ServerA | US         | 100   |
| ServerB | EU         | 200   |

**Output:**

| ServerA (labels: DataCenter: US) | ServerB (labels: DataCenter: EU) |
| -------------------------------- | -------------------------------- |
| 10                               | 20                               |

The extra labels can now be used in the field display name provide more complete field names.

If you want to extract config from one query and apply it to another you should use the config from query results transformation.

#### Example

**Input:**

| Name    | Value | Max |
| ------- | ----- | --- |
| ServerA | 10    | 100 |
| ServerB | 20    | 200 |
| ServerC | 30    | 300 |

**Output:**

| ServerA (config: max=100) | ServerB (config: max=200) | ServerC (config: max=300) |
| ------------------------- | ------------------------- | ------------------------- |
| 10                        | 20                        | 30                        |

As you can see each row in the source data becomes a separate field. Each field now also has a max config option set. Options like **Min**, **Max**, **Unit** and **Thresholds** are all part of field configuration and if set like this will be used by the visualization instead of any options manually configured in the panel editor options pane.

This transformation enables the conversion of rows into individual fields, facilitates dynamic field configuration, and maps additional fields to labels.

### Series to rows

Use this transformation to combine the result from multiple time series data queries into one single result. This is helpful when using the table panel visualization.

The result from this transformation will contain three columns: Time, Metric, and Value. The Metric column is added so you easily can see from which query the metric originates from. Customize this value by defining Label on the source query.

In the example below, we have two queries returning time series data. It is visualized as two separate tables before applying the transformation.

**Query A:**

| Time                | Temperature |
| ------------------- | ----------- |
| 2020-07-07 11:34:20 | 25          |
| 2020-07-07 10:31:22 | 22          |
| 2020-07-07 09:30:05 | 19          |

**Query B:**

| Time                | Humidity |
| ------------------- | -------- |
| 2020-07-07 11:34:20 | 24       |
| 2020-07-07 10:32:20 | 29       |
| 2020-07-07 09:30:57 | 33       |

Here is the result after applying the Series to rows transformation.

| Time                | Metric      | Value |
| ------------------- | ----------- | ----- |
| 2020-07-07 11:34:20 | Temperature | 25    |
| 2020-07-07 11:34:20 | Humidity    | 22    |
| 2020-07-07 10:32:20 | Humidity    | 29    |
| 2020-07-07 10:31:22 | Temperature | 22    |
| 2020-07-07 09:30:57 | Humidity    | 33    |
| 2020-07-07 09:30:05 | Temperature | 19    |

This transformation facilitates the consolidation of results from multiple time series queries, providing a streamlined and unified dataset for efficient analysis and visualization in a tabular format.

### Sort by

Use this transformation to sort each frame within a query result based on a specified field, making your data easier to understand and analyze. By configuring the desired field for sorting, you can control the order in which the data is presented in the table or visualization.

Use the **Reverse** switch to inversely order the values within the specified field. This functionality is particularly useful when you want to quickly toggle between ascending and descending order to suit your analytical needs.

For example, in a scenario where time-series data is retrieved from a data source, the **Sort by** transformation can be applied to arrange the data frames based on the timestamp, either in ascending or descending order, depending on the analytical requirements. This capability ensures that you can easily navigate and interpret time-series data, gaining valuable insights from the organized and visually coherent presentation.

### Spatial

Use this transformation to apply spatial operations to query results.

- **Action** - Select an action:
  - **Prepare spatial field** - Set a geometry field based on the results of other fields.
    - **Location mode** - Select a location mode (these options are shared by the **Calculate value** and **Transform** modes):
      - **Auto** - Automatically identify location data based on default field names.
      - **Coords** - Specify latitude and longitude fields.
      - **Geohash** - Specify a geohash field.
      - **Lookup** - Specify Gazetteer location fields.
  - **Calculate value** - Use the geometry to define a new field (heading/distance/area).
    - **Function** - Choose a mathematical operation to apply to the geometry:
      - **Heading** - Calculate the heading (direction) between two points.
      - **Area** - Calculate the area enclosed by a polygon defined by the geometry.
      - **Distance** - Calculate the distance between two points.
  - **Transform** - Apply spatial operations to the geometry.
    - **Operation** - Choose an operation to apply to the geometry:
      - **As line** - Create a single line feature with a vertex at each row.
      - **Line builder** - Create a line between two points.

This transformation allows you to manipulate and analyze geospatial data, enabling operations such as creating lines between points, calculating spatial properties, and more.

### Time series to table transform

Use this transformation to convert time series results into a table, transforming a time series data frame into a **Trend** field which can then be used with the [sparkline cell type][]. If there are multiple time series queries, each will result in a separate table data frame. These can be joined using join or merge transforms to produce a single table with multiple sparklines per row.

{{< figure src="/static/img/docs/transformations/table-sparklines.png" class="docs-image--no-shadow" max-width= "1100px" alt="A table panel showing multiple values and their corresponding sparklines." >}}

For each generated **Trend** field value, a calculation function can be selected. This value is displayed next to the sparkline and will be used for sorting table rows.

{{< figure src="/static/img/docs/transformations/timeseries-table-select-stat.png" class="docs-image--no-shadow" max-width= "1100px" alt="A select box showing available statistics that can be calculated." >}}

### Transpose

Use this transformation to pivot the data frame, converting rows into columns and columns into rows. This transformation is particularly useful when you want to switch the orientation of your data to better suit your visualization needs.
If you have multiple types, it will default to string type. You can select how empty cells should be represented.

**Before Transformation:**

| env  | January | February |
| ---- | ------- | -------- |
| prod | 1       | 2        |
| dev  | 3       | 4        |

**After applying transpose transformation:**

| Field    | prod | dev |
| -------- | ---- | --- |
| January  | 1    | 3   |
| February | 2    | 4   |

{{< figure src="/media/docs/grafana/transformations/screenshot-grafana-11-2-transpose-transformation.png" class="docs-image--no-shadow" max-width= "1100px" alt="Before and after transpose transformation" >}}

### Trendline

Use this transformation to create a new data frame containing values predicted by a statistical model. This is useful for finding a trend in chaotic data. It works by fitting a mathematical function to the data, using either linear or polynomial regression. The data frame can then be used in a visualization to display a trendline.

There are two different models:

- **Linear** - Fits a linear function to the data.
  {{< figure src="/static/img/docs/transformations/linear-regression.png" class="docs-image--no-shadow" max-width= "1100px" alt="A time series visualization with a straight line representing the linear function" >}}
- **Polynomial** - Fits a polynomial function to the data.
  {{< figure src="/static/img/docs/transformations/polynomial-regression.png" class="docs-image--no-shadow" max-width= "1100px" alt="A time series visualization with a curved line representing the polynomial function" >}}

> **Note:** This transformation was previously called regression analysis.

### Smoothing

Use this transformation to reduce noise in time series data through adaptive smoothing. This transformation creates smoother, cleaner visualizations while preserving all original time points and important trends and patterns in your data.

The smoothing transformation uses the ASAP (Automatic Smoothing for Attention Prioritization) algorithm internally to generate a smoothed curve, which is then interpolated back onto all original time points. This ensures your visualization maintains continuous lines without gaps while reducing noise.

#### Available options

- **Resolution** - Controls smoothing intensity (1-1000). Lower values create more aggressive smoothing, while higher values preserve more detail. The output preserves all original time points.

#### When to use smoothing

This transformation is useful for:

- Noisy time series data that obscures underlying trends
- Clearer trend analysis and pattern recognition

#### Example

Consider noisy sensor data with thousands of points:

**Before smoothing:**

| Time                 | Temperature |
| -------------------- | ----------- |
| 2020-07-07 10:00:00  | 23.1        |
| 2020-07-07 10:00:01  | 23.3        |
| 2020-07-07 10:00:02  | 22.9        |
| 2020-07-07 10:00:03  | 23.2        |
| ... (thousands more) | ...         |

**After smoothing (Resolution: 100):**

| Time                | Temperature (smoothed) |
| ------------------- | ---------------------- |
| 2020-07-07 10:00:00 | 23.1                   |
| 2020-07-07 10:00:01 | 23.1                   |
| 2020-07-07 10:00:02 | 23.0                   |
| 2020-07-07 10:00:03 | 23.0                   |
| ... (same count)    | ...                    |

The transformation preserves all original time points while reducing noise, resulting in smoother curves that maintain continuous lines without gaps.

[Table panel]: ref:table-panel
[Calculation types]: ref:calculation-types
[sparkline cell type]: ref:sparkline-cell-type
[configuration file]: ref:configuration-file
[Time series panel]: ref:time-series-panel
[feature toggle]: ref:feature-toggle
[dashboard variable]: ref:dashboard-variable

# Troubleshoot queries

This page provides information to solve common dashboard problems.

## I get different results when I rearrange my functions

Function order is very important. Just like in math, the order that you place your functions can affect the result.

## Inspect your query request and response

The most common problems are related to the query and response from your data source. Even if it looks
like a bug or visualization issue in Grafana, it is almost always a problem with the data source query or
the data source response. Start by inspecting your panel query and response.

For more information, refer to [Inspect request and response data](ref:inspect).

## My query is slow

How many data points is your query returning? A query that returns lots of data points will be slow. Try this:

- In **Query options**, limit the **Max data points** returned.
- In **Query options**, increase the **Min interval** time.
- In your query, use a `group by` function.

# Alert list

Alert lists allow you to display a list of important alerts that you want to track. You can configure the alert list to show the current state of your alert, such as firing, pending, or normal. Learn more about alerts in [Grafana Alerting overview](ref:grafana-alerting-overview).

![An alert list visualization](/media/docs/grafana/panels-visualizations/screenshot-alert-list-v11.3.png)

On each dashboard load, this visualization queries the alert list, always providing the most up-to-date results.
It shows only alerts that are currently active, and doesn't support historical alerts or dashboard time range filtering.

{{< docs/play title="Alert List" url="https://play.grafana.org/d/bdodlcyou483ke/" >}}

## Configure an alert list

Once you’ve [created a dashboard](ref:create-dashboard), the following video shows you how to configure an alert list visualization:

{{< youtube id="o4rK7_AXZ9Y" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Options

Use the following options to refine your alert list visualization.

<!-- prettier-ignore-start -->

| Option     | Description                                                                                               |
| ---------- | --------------------------------------------------------------------------------------------------------- |
| View mode  | Choose between **List** to display alerts in a detailed list format with comprehensive information, or **Stat** to show alerts as a summarized single-value statistic.  |
| Group mode | Choose between **Default grouping** to show alert instances grouped by their alert rule, or **Custom grouping** to show alert instances grouped by a custom set of labels. |
| Group by | When **Custom grouping** is selected, choose label keys to group alert instances. |
| Max items | Sets the maximum number of alerts to list when **Group mode** is **Default grouping**. This option is hidden for **Custom grouping**. By default, Grafana sets this value to 20. |
| [Sort order](#sort-order) | Select how to order the alerts displayed. |
| Alerts linked to this dashboard | Toggle the switch on to only show alerts from the dashboard the alert list is in. |

<!-- prettier-ignore-end -->

#### Sort order

Select how to order the alerts displayed. Choose from:

- **Alphabetical (asc)** - Alphabetical order.
- **Alphabetical (desc)** - Reverse alphabetical order.
- **Importance** - By importance according to the following values, with 1 being the highest:
  - alerting: 1
  - firing: 1
  - no_data: 2
  - pending: 3
  - ok: 4
  - paused: 5
  - inactive: 5
- **Time (asc)** - Oldest active alert instances first.
- **Time (desc)** - Newest active alert instances first.

### Filter options

These options allow you to limit alerts shown to only those that match the query, folder, or tags you choose.

<!-- prettier-ignore-start -->

| Option     | Description                                                                                               |
| ---------- | --------------------------------------------------------------------------------------------------------- |
| Alert name | Filter alerts by name. |
| Alert instance label | Filter alert instances using [label](ref:alert-label) querying. For example,`{severity="critical", instance=~"cluster-us-.+"}`. |
| Datasource | Filter alerts from the selected data source. Selecting a specific data source excludes Grafana-managed alert rules, even if they match your other filters. To include Grafana-managed alerts, select the **-- Grafana --** data source or leave this field empty. |
| Folder | Filter Grafana-managed alert rules by alert rule folder. This option is available only when **Datasource** is empty or set to **-- Grafana --**. It doesn't filter by dashboard folder. |
| Show alerts with 0 instances | Filter for alert rules with no instances. Alert rules with 0 (zero) instances are hidden by default. You can choose to show them by toggling this switch. Because these rules have no instances, they remain hidden if the **Alert instance label** filter is configured. |

<!-- prettier-ignore-end -->

{{< admonition type="note" >}}
Grafana distinguishes between Grafana-managed alert rules and data source-managed alert rules.
The **Datasource** filter only matches data source-managed alerts unless you select **-- Grafana --**, so leave **Datasource** empty if you want to see alerts from both kinds of rules.
{{< /admonition >}}

### Alert state filter options

Choose which alert states to display in this visualization.

<!-- prettier-ignore-start -->

| Option     | Description                                                                                               |
| ---------- | --------------------------------------------------------------------------------------------------------- |
| Alerting / Firing | Shows alerts that are currently active and triggering an alert condition. |
| Pending | Shows alerts that are in a transitional state, waiting for conditions to be met before triggering. |
| No Data | Shows alerts where the data source is not returning any data, which could indicate an issue with data collection. |
| Recovering | Shows alerts in a recovering state after the alert condition is resolved. |
| Normal | Shows alerts that are in a normal or resolved state, where no alert condition is currently met. |
| Error | Shows alerts where an error has occurred, typically related to an issue in the alerting process. |

<!-- prettier-ignore-end -->

# Annotations list

The annotations list shows a list of available annotations you can use to view annotated data. Various options are available to filter the list based on tags and on the current dashboard.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-annotations-list-viz-v12.0.png" max-width="750px" alt="The annotations list visualization" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Annotation query options

The following options control the source query for the list of annotations:

<!-- prettier-ignore-start -->

| Option     | Description                                                                                               |
| ---------- | --------------------------------------------------------------------------------------------------------- |
| [Query filter](#query-filter) | Specify which annotations are included in the list.  |
| [Time range](#time-range) | Specify whether the list should be limited to the current time range. |
| Tags | Filter the annotations by tags. You can add multiple tags to refine the list. Optionally, leave the tag list empty and filter in view mode by selecting tags that are listed as part of the results on the panel itself. |
| Limit | Limit the number of results returned. The default is `10`. |

<!-- prettier-ignore-end -->

#### Query filter

Use the **Query filter** option to create a list of annotations from all dashboards in your organization or the current dashboard in which this panel is located.
Choose from:

- **All dashboards** - List annotations from all dashboards in the current organization.
- **This dashboard** - Limit the list to the annotations on the current dashboard.

#### Time range

Specify whether the list should be limited to the current time range.
Choose from:

- **None** - No time range limit for the annotations query.
- **This dashboard** - Limit the list to the time range of the dashboard where the annotations list is available.

### Display options

These options control additional metadata included in the annotations list display:

<!-- prettier-ignore-start -->

| Option     | Description                                                                                               |
| ---------- | --------------------------------------------------------------------------------------------------------- |
| Show user | Show or hide which user created the annotation.  |
| Show time | Show or hide the time the annotation creation time. |
| Show tags | Show or hide the tags associated with an annotation. Note that you can use the tags to filter the annotations list. |

<!-- prettier-ignore-end -->

### Link behavior options

Use the following options to control the behavior of annotation links in the list:

<!-- prettier-ignore-start -->

| Option     | Description                                                                                               |
| ---------- | --------------------------------------------------------------------------------------------------------- |
| Link target | Set how to view the annotated data. The default is **Panel**. Choose from:<ul><li>**Panel** - The link takes you directly to a full-screen view of the panel with the corresponding annotation.</li><li>**Dashboard** - Focuses the annotation in the context of a complete dashboard.</li></ul> |
| Time before | Set the time range before the annotation. Use duration string values like `1h` for one hour and `10m` for 10 minutes. The default is `10m`. |
| Time after | Set the time range after the annotation. The default is `10m`. |

<!-- prettier-ignore-end -->

# Bar chart

A bar chart is a visual representation that uses rectangular bars, where the length of each bar represents each value.
You can use the bar chart visualization when you want to compare values over different categories or time periods. The visualization can display the bars horizontally or vertically, and can be customized to group or stack bars for more complex data analysis.

![Bar chart visualizations](/media/docs/grafana/panels-visualizations/screenshot-bar-charts-v11.3.png)

You can use the bar chart visualization if you need to show:

- Population distribution by age or location
- CPU usage per application
- Sales per division
- Server cost distribution

## Configure a bar chart

The following video shows you how to create and configure a bar chart visualization:

{{< youtube id="qyKE9-71KkE" >}}

{{< docs/play title="Grafana Bar Charts and Pie Charts" url="https://play.grafana.org/d/ktMs4D6Mk/" >}}

## Supported data formats

To create a bar chart visualization, you need a dataset containing one string or time field (or column) and at least one numeric field, though preferably more than one to make best use of the visualization.

The text or time field is used to label the bars or values in each row of data and the numeric fields are represented by proportionally sized bars.

### Example 1

| Group | Value1 | Value2 | Value3 |
| ----- | ------ | ------ | ------ |
| uno   | 5      | 3      | 2      |

![Bar chart single row example](/media/docs/grafana/panels-visualizations/screenshot-grafana-11.1-barchart-example1.png 'Bar chart single row example')

If you have more than one text or time field, by default, the visualization uses the first one, but you can change this in the x-axis option as described in the [Bar chart options](#bar-chart-options) section.

### Example 2

If your dataset contains multiple rows, the visualization displays multiple bar chart groups where each group contains multiple bars representing all the numeric values for a row.

| Group | Value1 | Value2 | Value3 |
| ----- | ------ | ------ | ------ |
| uno   | 5      | 3      | 2      |
| dos   | 10     | 6      | 4      |
| tres  | 20     | 8      | 2      |

![Bar chart multiple row example](/media/docs/grafana/panels-visualizations/screenshot-grafana-11.1-barchart-example2.png 'Bar chart multiple row example')

While the first field can be time-based and you can use a bar chart to plot time-series data, for large amounts of time-series data, we recommend that you use the [time series visualization](ref:time-series) and configure it to be displayed as bars.

We recommend that you only use one dataset in a bar chart because using multiple datasets can result in unexpected behavior.

<!-- vale Grafana.WordList = NO -->
<!-- vale Grafana.Spelling = NO -->

## Apply filters from the bar chart

In bar charts, you can apply filters directly from the visualization.
Filters are available when the x-axis field is filterable and the data source supports dashboard drilldown filters, such as Prometheus or Loki.

To display the filter button, hover your cursor over the bar that has the value for which you want to filter and click the bar:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-adhoc-filter-icon-bar-v12.2.png" max-width="300px" alt="The filter button in a bar chart tooltip">}}

For more information about applying filters this way, refer to [Cross-filtering](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/filter-group-by/#dashboard-drilldown-with-filters).

<!-- vale Grafana.Spelling = YES -->
<!-- vale Grafana.WordList = YES -->

## Transform override property

Use the **Graph styles > Transform** [override property](#field-overrides) to transform series values without affecting the values shown in the tooltip, context menu, or legend. Choose from the following transform options:

- **Constant** - Show the first value as a constant line.
- **Negative Y transform** - Flip the results to negative values on the y-axis.

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Bar chart options

Use these options to refine your visualization.

<!-- prettier-ignore-start -->

| Option                           | Description                                                                                      |
| -------------------------------- | ------------------------------------------------------------------------------------------------------- |
| X Axis                           | Specify which field is used for the x-axis.          |
| Orientation                      | Choose from: <ul><li>**Auto** - Grafana decides the bar orientation based on the panel dimensions.</li><li>**Horizontal** - Will make the X axis the category axis.</li><li>**Vertical** - Will make the Y axis the category axis.</li></ul> |
| Rotate x-axis tick labels        | When the graph is vertically oriented, this setting rotates the labels under the bars. This setting is useful when bar chart labels are long and overlap.  |
| X-axis tick label max length | Sets the maximum length of bar chart labels. Labels longer than the maximum length are truncated, and appended with `...`. This option is available when **Rotate x-axis tick labels** isn't `0`. |
| X-axis labels minimum spacing | Sets the minimum spacing between x-axis labels. Depending on your choice, you can select the **RTL** checkbox to require space from the right side. Choose from: <ul><li>**None** - All tick marks are shown.</li><li>**Small** - 100 pixels of space is required between labels.</li><li>**Medium** - 200 pixels of space is required between labels.</li><li>**Large** - 300 pixels of space is required between labels.</li></ul>  |
| Show values                      | This controls whether values are shown. Values are shown on top or to the left of bars. Choose from: <ul><li>**Auto** Values will be shown if there is space.</li><li>**Always** Always show values.</li><li>**Never** Never show values.</li></ul>                                                               |
| Stacking                         | Controls bar chart stacking. Choose from: <ul><li>**Off**: Bars will not be stacked.</li><li>**Normal**: Bars will be stacked on each other.</li><li>**100%**: Bars will be stacked on each other, and the height of each bar is the percentage of the total height of the stack.</li></ul> |
| Group width                      | Controls the width of groups. 1 = Max width, 0 = Min width. This option is hidden when stacking is enabled or when the chart has only one numeric field.       |
| Bar width                        | Controls the width of bars. 1 = Max width, 0 = Min width.    |
| Bar radius                       | Controls the radius of the bars. Choose from: <ul><li>0 = Minimum radius</li><li>0.5 = Maximum radius</li></ul>     |
| Highlight full area on hover     | Controls if the entire surrounding area of the bar is highlighted when you hover over the bar. This option is available only when **Stacking** is **Off**.    |
| Color by field    | Use the color value for a sibling field to color each bar value. |
| Line width                       | Controls line width of the bars. This is a field option and can be set in field defaults or overrides.    |
| Fill opacity                     | Controls the fill opacity of bars. This is a field option and can be set in field defaults or overrides.                    |
| [Gradient mode](#gradient-mode)  | Set the mode of the gradient fill. Fill gradient is based on the line color. To change the color, use the standard color scheme field option. Gradient appearance is influenced by the **Fill opacity** setting. This is a field option and can be set in field defaults or overrides.                                                                               |

<!-- prettier-ignore-end -->

#### Gradient mode

Set the mode of the gradient fill. Fill gradient is based on the line color. To change the color, use the standard color scheme field option. Gradient appearance is influenced by the **Fill opacity** setting. Choose from:

- **None** - No gradient fill. This is the default setting.
- **Opacity** - Transparency of the gradient is calculated based on the values on the y-axis. Opacity of the fill is increasing with the values on the Y-axis.
- **Hue** - Gradient color is generated based on the hue of the line color.
- **Scheme** - The bar receives a gradient color defined by the **Standard options > Color scheme** selection.
  - **From thresholds** - If the **Color scheme** selection is **From thresholds (by value)**, then each bar is the color of the defined threshold.

    {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-colors-by-thresholds-v11.3.png" alt="Color scheme From thresholds" caption="Color scheme: From thresholds" >}}

  - **Gradient color schemes** - The following image shows a bar chart with the **Green-Yellow-Red (by value)** color scheme option selected.

    {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-colors-by-value-v11.3.png" alt="Color scheme Green-Yellow-Red" caption="Color scheme: Green-Yellow-Red" >}}

### Tooltip options

{{< docs/shared lookup="visualizations/tooltip-options-1.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Legend options

{{< docs/shared lookup="visualizations/legend-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Text size

Enter a **Value** to change the size of the text on your bar chart.

### Axis

Use the following field settings to refine how your axes display.

For guidance on configuring more than one y-axis, refer to [Multiple y-axes](#multiple-y-axes).

Some field options will not affect the visualization until you click outside of the field option box you are editing or press Enter.

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Placement | Select the placement of the Y-axis. Choose from: <ul><li>**Auto** - Grafana automatically assigns Y-axis to the series. When there are two or more series with different units, then Grafana assigns the left axis to the first unit and right to the following units.</li><li>**Left** - Display all Y-axes on the left side.</li><li>**Right** - Display all Y-axes on the right side.</li><li>**Hidden** - Hide all axes. To selectively hide axes, [add a field override](ref:add-a-field-override) that targets specific fields.</li></ul> |
| Label | Set a Y-axis text label. If you have more than one Y-axis, then you can assign different labels with an override. |
| Width | Set a fixed width of the axis. By default, Grafana dynamically calculates the width of an axis.<br></br>By setting the width of the axis, data whose axes types are different can share the same display proportions. This makes it easier to compare more than one graph’s worth of data because the axes are not shifted or stretched within visual proximity of each other. |
| Show grid lines | Set whether grid lines are displayed in the chart. Choose from: <ul><li>**Auto** - Grafana automatically determines whether grid lines are displayed.</li><li>**On** - Grid lines are always displayed.</li><li>**Off** - Grid lines are never displayed</li></ul> |
| Color | Choose whether the axis color is the **Text** or **Series** color. |
| Show border | Toggle the switch to hide or display the border. |
| Scale | Set how the y-axis is split. Choose from: <ul><li>**Linear**</li><li>**Logarithmic** - Choose between log base 2 or log base 10.</li><li>**Symlog** - Uses a symmetrical logarithmic scale. Choose between log base 2 or log base 10, allowing for negative values.</li></ul> |
| Centered zero | Set the y-axis so it's centered on zero. |
| [Soft min and soft max](#soft-min-and-soft-max) | Set a **Soft min** or **soft max** option for better control of Y-axis limits. By default, Grafana sets the range for the Y-axis automatically based on the dataset. |

<!-- prettier-ignore-end -->

#### Soft min and soft max

Set a **Soft min** or **soft max** option for better control of Y-axis limits. By default, Grafana sets the range for the Y-axis automatically based on the dataset.

**Soft min** and **soft max** settings can prevent blips from turning into mountains when the data is mostly flat, and hard min or max derived from standard min and max field options can prevent intermittent spikes from flattening useful detail by clipping the spikes past a defined point.

You can set standard min/max options to define hard limits of the Y-axis. For more information, refer to [Standard options definitions](ref:standard-options-definitions).

{{< docs/shared lookup="visualizations/multiple-y-axes.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+3" >}}

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Bar gauge

Bar gauges simplify your data by reducing every field to a single value. You choose how Grafana calculates the reduction. This visualization can show one or more bar gauges depending on how many series, rows, or columns your query returns.

{{< figure src="/static/img/docs/v66/bar_gauge_cover.png" max-width="1025px" alt="Bar gauge panel" >}}

The bar gauge visualization displays values as bars with various lengths or fills proportional to the values they represent. They differ from traditional bar charts in that they act as gauges displaying metrics between ranges. One example is a thermometer displaying body temperature in a bar filling up.

You can use a bar gauge visualization when you need to show:

- Key performance indicators (KPIs)
- System health
- Savings goals
- Attendance
- Process completion rates

## Configure a bar gauge visualization

The following video shows you how to create and configure a bar gauge visualization:

{{< youtube id="7PhDysObEXA" >}}

{{< docs/play title="Bar Gauge" url="https://play.grafana.org/d/vmie2cmWz/" >}}

## Supported data formats

To create a bar gauge visualization, you need a dataset querying at least one numeric field. Every numeric field in the dataset is displayed as a bar gauge. Text or time fields aren't required but if they're present, they're used for labeling.

### Example 1

| Label | Value1 | Value2 | Value3 |
| ----- | ------ | ------ | ------ |
| Row1  | 5      | 3      | 2      |

![Bar gauge with single row of data](/media/docs/grafana/panels-visualizations/screenshot-grafana-12.1-bargauge-example1.png)

The minimum and maximum range for the bar gauges is automatically pulled from the largest and smallest numeric values in the dataset. You can also manually define the minimum and maximum values as indicated in the [Standard options](#standard-options) section.

You can also define the minimum and maximum from the dataset provided.

### Example 2

| Label | Value | Max | Min |
| ----- | ----- | --- | --- |
| Row1  | 3     | 6   | 1   |

![Bar gauge with single row of data including maximum and minimum](/media/docs/grafana/panels-visualizations/screenshot-grafana-12.1-bargauge-example2.png)

If you don’t want to show gauges for the min and max values, you can configure only one field to be displayed as described in the [Value options](#value-options) section.

![Bar gauge, single row of data with max and min displaying value](/media/docs/grafana/panels-visualizations/screenshot-grafana-12.1-bargauge-example3.png)

Even if the min and max aren’t displayed, the visualization still pulls the range from the data set.

### Example 3

The bar gauge visualization also supports multiple records (rows) in the dataset.

| Label | Value1 | Value2 | Value3 |
| ----- | ------ | ------ | ------ |
| Row1  | 5      | 3      | 2      |
| Row2  | 10     | 6      | 4      |
| Row3  | 20     | 8      | 2      |

![Bar gauge with multiple rows of data displaying last row](/media/docs/grafana/panels-visualizations/screenshot-grafana-12.1-bargauge-example4.png)

By default, the visualization is configured to [calculate](#value-options) a single value per column or series and to display only the last set of data. However, it derives the minimum and maximum from the full dataset even if those values aren’t visible. In this example, that means only the last row of data is displayed in the gauges and the minimum and maximum values are defined as 2 and 20, pulled from the whole dataset.

If you want to show one gauge per cell you can change the **Show** setting from **Calculate** to **All values** and each bar is labeled by concatenating the text column with each value's column name.

![Bar gauge with multiple rows of data displaying all the values](/media/docs/grafana/panels-visualizations/screenshot-grafana-12.1-bargauge-example5.png)

For more information on these settings, refer to [Value options](#value-options).

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value options

Use the following options to refine how your visualization displays the value:

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Show | Set how Grafana displays your data. Choose from:<ul><li>**Calculate** - Show a calculated value based on all rows.</li><li>**All values** - Show a separate value for every row. If you select this option, then you can also limit the number of rows to display.</li></ul> |
| Calculation | If you chose **Calculate** as your **Show** option, select a reducer function that Grafana will use to reduce many fields to a single value. For a list of available calculations, refer to [Calculation types](ref:calculation-types). |
| Limit | If you chose **All values** as your **Show** option, enter the maximum number of rows to display. The default is 25 and the maximum is 5,000. |
| Fields | Select the fields display in the panel. |

<!-- prettier-ignore-end -->

### Bar gauge options

Adjust how the gauge is displayed.

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Orientation | Choose a stacking direction:<ul><li>**Auto** - Grafana determines the best orientation.</li><li>**Horizontal** - Bars stretch horizontally, left to right.</li><li>**Vertical** - Bars stretch vertically, bottom to top.</li></ul> |
| Display mode | Choose a display mode:<ul><li>**Gradient** - Threshold levels define a gradient.</li><li>**Retro LCD** - The bar is split into sections that are lit or unlit.</li><li>**Basic** - Single color based on the matching threshold.</li></ul> |
| Value display | Choose a value display mode:<ul><li>**Value color** - Value color is determined by value.</li><li>**Text color** - Value color is default text color.</li><li>**Hidden** - Values are hidden.</li></ul> |
| Name placement | Set the name placement mode when the bar gauge orientation is **Auto** or **Horizontal**. Choose from:<ul><li>**Auto** - Grafana determines the best placement.</li><li>**Top** - Names are placed on top of each bar gauge.</li><li>**Left** - Names are placed to the left of each bar gauge.</li><li>**Hidden** - Names are hidden.</li></ul> <p>When the bar gauge is in the vertical orientation, choose from **Auto** (names are always placed at the bottom of each bar) or **Hidden**.</p>|
| Text mode | Choose what text to render in the bar gauge:<ul><li>**Auto** - Grafana determines the text to show.</li><li>**Value and name** - Renders both the value and series name.</li><li>**Value** - Renders only the value.</li><li>**Name** - Renders only the series name.</li><li>**None** - Renders the bar gauge with no labels.</li></ul> Has no effect when **Name placement** is set to **Hidden**. |
| Show unfilled area | Select if you want to render the unfilled region of the bars as gray. Not applicable to **Retro LCD** display mode. |
| Bar size | Choose a bar size mode:<ul><li>**Auto** - Grafana determines the best bar size.</li><li>**Manual** - Manually configure the bar size.</li></ul> |
| Min width | Limit the minimum width of the bar column when the gauge is oriented vertically or is in **Auto** mode. Automatically shows the x-axis scroll bar when there's a large amount of data.<p>This option only applies when the **Bar size** mode is set to **Manual**.</p> |
| Min height | Limit the minimum height of the bar row when the bar gauge is oriented horizontally or is in **Auto** mode. Automatically shows the y-axis scroll bar when there's a large amount of data. <p>This option only applies when the **Bar size** mode is set to **Manual**.</p> |
| Max height | Limit the maximum height of the bar row when the bar gauge is oriented horizontally or is in **Auto** mode. Automatically shows the y-axis scroll bar when there's a large amount of data. <p>This option only applies when the **Bar size** mode is set to **Manual**.</p> |

<!-- prettier-ignore-end -->

### Legend options

The legend is hidden by default.

{{< docs/shared lookup="visualizations/legend-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Text size options

Set the sizes of the following text elements in pixels:

- **Title** - Bar name
- **Value** - Bar value

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Candlestick

The candlestick visualization allows you to visualize data that includes a number of consistent dimensions focused on price movements, such as stock prices. The candlestick visualization includes an [Open-High-Low-Close (OHLC) mode](#open-high-low-close), as well as support for additional dimensions based on time series data.

Candlestick visualizations build upon the foundation of the [time series visualization](ref:time-series-visualization) and include many common configuration settings.

You can use a candlestick if you want to visualize, at a glance, how a price moved over time, whether it went up, down, or stayed the same, and how much it fluctuated:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-candlestick-v11.6.png" max-width="750px" alt="A candlestick visualization" >}}

Each candlestick is represented as a rectangle, referred to as the _candlestick body_. The candlestick body displays the opening and closing prices during a time period. Green candlesticks represent when the price appreciated while the red candlesticks represent when the price depreciated. The lines sticking out the candlestick body are referred to as _wicks_ or _shadows_, which represent the highest and lowest prices during the time period.

Use a candlestick when you need to:

- Monitor and identify trends in price movements of specific assets such as stocks, currencies, or commodities.
- Analyze any volatility in the stock market.
- Provide data analysis to help with trading decisions.

## Configure a candlestick

Once you’ve created a [dashboard](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/create-dashboard/), the following video shows you how to configure a candlestick visualization:

{{< youtube id="IOFKBgbf3aM" >}}

{{< docs/play title="Candlestick" url="https://play.grafana.org/d/candlestick/candlestick" >}}

## Supported data formats

The candlestick visualization works best with price movement data for an asset. The data must include:

- **Timestamps** - The time at which each price movement occurred.
- **Opening price** - The price of the asset at the beginning of the time period.
- **Closing price** - The price of the asset at the end of the time period.
- **Highest price** - The highest price the asset reached during the time period.
- **Lowest price** - The lowest price the asset reached during the time period.

### Example

| Timestamps          | Open  | High  | Low   | Close |
| ------------------- | ----- | ----- | ----- | ----- |
| 2024-03-13 10:05:00 | 0.200 | 0.205 | 0.201 | 0.203 |
| 2024-03-14 10:10:10 | 0.204 | 0.205 | 0.201 | 0.200 |
| 2024-03-15 10:15:10 | 0.204 | 0.205 | 0.201 | 0.200 |
| 2024-03-16 10:20:11 | 0.203 | 0.203 | 0.202 | 0.203 |
| 2024-03-17 10:25:11 | 0.203 | 0.203 | 0.202 | 0.203 |
| 2024-03-18 10:30:12 | 0.202 | 0.202 | 0.201 | 0.201 |

The data is converted as follows:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-candles-volume-v11.6.png" max-width="750px" alt="A candlestick visualization showing the price movements of specific asset." >}}

## Pan and zoom panel time range

{{< docs/shared lookup="visualizations/panel-pan-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Candlestick options

The following options let you control which data is displayed in the visualization and how it appears:

| Option                                                 | Description                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Mode                                                   | Controls which dimensions are used for the visualization. Choose from:<ul><li>**Candles** - Uses the open, high, low, and close dimensions.</li><li>**Volume** - Uses only the volume dimension.</li><li>**Both** - The default behavior, which displays both candles and volume values.</li></ul>                                            |
| Candle style                                           | Controls the appearance of the candles. Choose from:<ul><li>**Candles** - The default display style, which creates candle-style markers between the open and close dimensions.</li><li>**OHLC Bars** - Displays values for the four core dimensions, open, high, low, and close.</li></ul> This option is hidden when **Mode** is **Volume**. |
| [Color strategy](#color-strategy)                      | Controls how colors are applied to dimensions. Choose from:<ul><li>**Since Open** - This mode uses the **Up color** if the intra-period price movement is positive.</li><li>**Since Prior Close** - The color of the candle is based on the inter-period price movement or change in value.</li></ul>                                         |
| Up color/Down color                                    | These options control which colors are used when the price movement is up or down. Note that the **Color strategy** selection determines if intra-period or inter-period price movement is used to select the candle or OHLC bar color.                                                                                                       |
| [Open, High, Low, Close, Volume](#open-high-low-close) | The candlestick visualization attempts to map fields from your data to these dimensions, as appropriate dimension.                                                                                                                                                                                                                            |
| [Additional fields](#additional-fields)                | The **Include** and **Ignore** options allow the candlestick visualization to display other included data such as simple moving averages, Bollinger bands and more, using the same styles and configurations available in the [time series](ref:time-series-visualization) visualization.                                                     |

#### Color strategy

The **Color strategy** option controls how colors are applied to dimensions. Choose from:

- **Since Open** - The default behavior. This mode uses the **Up color** if the intra-period price movement is positive. In other words, if the value on close is greater or equal to the value on open, the **Up color** is used.
- **Since Prior Close** - An alternative display method where the color of the candle is based on the inter-period price movement or change in value. In other words, if the value on open is greater than the previous value on close, the **Up color** is used. If the value on open is lower than the previous value on close, the **Down color** is used.

  This option also triggers the _hollow candlestick_ visualization mode. Hollow candlesticks indicate that the intra-period movement is positive (value is higher on close than on open), while filled candlesticks indicate the intra-period change is negative (value is lower on close than on open). To learn more, refer to the [explanation of the differences](https://thetradingbible.com/how-to-read-hollow-candlesticks).

#### Open, High, Low, Close, Volume {#open-high-low-close}

The candlestick visualization attempts to map fields from your data to the appropriate dimension:

- **Open** - Starting value of the given period.
- **High** - Highest value of the given period.
- **Low** - Lowest value of the given period.
- **Close** - Final (end) value of the given period.
- **Volume** - Sample count in the given period (for example, number of trades).

In **Volume** mode, the **High** and **Low** field pickers are hidden. In **Candles** mode, the **Volume** field picker is hidden. **Both** mode shows all pickers.

{{< admonition type="note" >}}
The candlestick visualization legend doesn't display these values.
{{< /admonition >}}

If your data can't be mapped to these dimensions for some reason (for example, because the column names aren't the same), you can map them manually using the **Open**, **High**, **Low**, and **Close** fields under the **Candlestick** options in the panel editor:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-olhc-options-v11.6.png" max-width="400px" alt="Open, High, Low, and Close fields in the panel editor" >}}

#### Additional fields

The candlestick visualization is based on the time series visualization, and it can visualize additional data dimensions beyond open, high, low, close, and volume.
The **Include** and **Ignore** options allow it to visualize other included data such as simple moving averages, Bollinger bands and more, using the same styles and configurations available in the [time series](ref:time-series-visualization) visualization.

### Tooltip options

Tooltip options control the information overlay that appears when you hover over data points in the visualization.

| Option                                  | Description                                                                                                                    |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| [Tooltip mode](#tooltip-mode)           | When you hover your cursor over the visualization, Grafana can display tooltips. Choose how tooltips behave.                   |
| [Values sort order](#values-sort-order) | This option controls the order in which values are listed in a tooltip.                                                        |
| [Hover proximity](#hover-proximity)     | Set the hover proximity (in pixels) to control how close the cursor must be to a data point to trigger the tooltip to display. |
| Max width                               | Set the maximum width of the tooltip box.                                                                                      |
| Max height                              | Set the maximum height of the tooltip box. The default is 600 pixels.                                                          |

#### Tooltip mode

When you hover your cursor over the visualization, Grafana can display tooltips. Choose how tooltips behave.

- **Single -** The hover tooltip shows only a single series, the one that you are hovering over on the visualization.
- **All -** The hover tooltip shows all series in the visualization. Grafana highlights the series that you are hovering over in bold in the series list in the tooltip.
- **Hidden -** Do not display the tooltip when you interact with the visualization.

Use an override to hide individual series from the tooltip.

#### Values sort order

When you set the **Tooltip mode** to **All**, the **Values sort order** option is displayed. This option controls the order in which values are listed in a tooltip. Choose from the following:

- **None** - Grafana automatically sorts the values displayed in a tooltip.
- **Ascending** - Values in the tooltip are listed from smallest to largest.
- **Descending** - Values in the tooltip are listed from largest to smallest.

#### Hover proximity

Set the hover proximity (in pixels) to control how close the cursor must be to a data point to trigger the tooltip to display.
The following screen recording shows this option in a time series visualization:

![Adding a hover proximity limit for tooltips](/media/docs/grafana/gif-grafana-10-4-hover-proximity.gif)

### Legend options

{{< docs/shared lookup="visualizations/legend-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Annotation options

{{< docs/shared lookup="visualizations/annotations/options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Graph styles options

The options under the **Graph styles** section let you control the general appearance of [additional fields](#additional-fields) in the visualization, excluding [color](#standard-options).

<!-- prettier-ignore-start -->

| Option                                      | Description                                                                                    |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| [Style](#style)                             | Choose whether to display your time-series data as **Lines**, **Bars**, or **Points**. |
| [Line interpolation](#line-interpolation)   | Choose how the graph interpolates the series line. |
| Line width                                  | Set the thickness of the series lines or the outline for bars using the **Line width** slider. |
| Fill opacity                                | Set the series area fill color using the **Fill opacity** slider. |
| [Gradient mode](#gradient-mode)             | Choose a gradient mode to control the gradient fill, which is based on the series color. |
| [Line style](#line-style)                   | Choose a solid, dashed, or dotted line style. |
| [Connect null values](#connect-null-values) | Choose how null values, which are gaps in the data, appear on the graph. |
| [Disconnect values](#disconnect-values)     | Choose whether to set a threshold above which values in the data should be disconnected. |
| [Show points](#show-points)                 | Set whether to show data points to lines or bars. |
| Point size                                  | Set the size of the points, from 1 to 40 pixels in diameter. |
| [Stack series](#stack-series)               | Set whether Grafana displays series on top of each other. |
| [Bar alignment](#bar-alignment)             | Set the position of the bar relative to a data point. |
| Bar width factor                            | Set the width of the bar relative to minimum space between data points. A factor of 0.5 means that the bars take up half of the available space between data points. A factor of 1.0 means that the bars take up all available space. |

<!-- prettier-ignore-end -->

#### Style

Choose whether to display your time-series data as **Lines**, **Bars**, or **Points**.
You can use overrides to combine multiple styles in the same graph. Choose from the following:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-candle-style-v12.0.png" max-width="750px" alt="Graph styles" >}}

#### Line interpolation

Choose how the graph interpolates the series line:

- **Linear** - Points are joined by straight lines.
- **Smooth** - Points are joined by curved lines that smooths transitions between points.
- **Step before** - The line is displayed as steps between points. Points are rendered at the end of the step.
- **Step after** - The line is displayed as steps between points. Points are rendered at the beginning of the step.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-candle-interpolation-v12.0.png" max-width="750px" alt="Line interpolation styles" >}}

#### Gradient mode

Choose a gradient mode to control the gradient fill, which is based on the series color. To change the color, use the standard color scheme field option. For more information, refer to [Color scheme](ref:color-scheme).

- **None** - No gradient fill. This is the default setting.
- **Opacity** - An opacity gradient where the opacity of the fill increases as y-axis values increase.
- **Hue** - A subtle gradient that's based on the hue of the series color.
- **Scheme** - A color gradient defined by your [Color scheme](ref:color-scheme). This setting is used for the fill area and line. For more information about scheme, refer to [Scheme gradient mode](#scheme-gradient-mode).

Gradient appearance is influenced by the **Fill opacity** setting. The following image shows the **Fill opacity** set to 50.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-candle-gradient-v12.0.png" max-width="750px" alt="Gradient modes" >}}

##### Scheme gradient mode

In **Scheme** gradient mode, the line or bar receives a gradient color defined from the selected **Color scheme** option in the visualization's **Standard** options.

The following image shows a line chart with the **Green-Yellow-Red (by value)** color scheme option selected:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-candle-scheme-grad-1-v12.0.png" max-width="600px" alt="Gradient color scheme" >}}

If the **Color scheme** is set to **From thresholds (by value)** and **Gradient mode** is set to **Scheme**, then the line or bar color changes as it crosses the defined thresholds:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-candle-scheme-grad-2-v12.0.png" max-width="600px" alt="Gradient color scheme with thresholds" >}}

#### Line style

Choose a solid, dashed, or dotted line style:

- **Solid** - Display a solid line. This is the default setting.
- **Dash** - Display a dashed line. When you choose this option, a list appears for you to select the length and gap (length, gap) for the line dashes. Dash spacing is 10, 10 by default.
- **Dots** - Display dotted lines. When you choose this option, a list appears for you to select the gap (length = 0, gap) for the dot spacing. Dot spacing is 0, 10 by default.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-candle-line-style-v12.0.png" max-width="750px" alt="Line styles" >}}

{{< docs/shared lookup="visualizations/connect-null-values.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="visualizations/disconnect-values.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

To change the color, use the standard [color scheme](ref:color-scheme) field option.

#### Show points

Set whether to show data points as lines or bars. Choose from the following:

- **Auto** - Grafana determines a point's visibility based on the density of the data. If the density is low, then points appear.
- **Always** - Show the points regardless of how dense the dataset is.
- **Never** - Don't show points.

#### Stack series

Set whether Grafana stacks or displays series on top of each other. Be cautious when using stacking because it can create misleading graphs. To read more about why stacking might not be the best approach, refer to [The issue with stacking](https://www.data-to-viz.com/caveat/stacking.html). Choose from the following:

- **Off** - Turns off series stacking. When **Off**, all series share the same space in the visualization.
- **Normal** - Stacks series on top of each other.
- **100%** - Stack by percentage where all series add up to 100%.

##### Stack series in groups

The stacking group option is only available as an override. For more information about creating an override, refer to [Configure field overrides](ref:configure-field-overrides).

1. Edit the panel and click **Overrides**.
1. Create a field override for the **Stack series** option.
1. In stacking mode, click **Normal**.
1. Name the stacking group in which you want the series to appear.

   The stacking group name option is only available when you create an override.

#### Bar alignment

Set the position of the bar relative to a data point. In the examples below, **Show points** is set to **Always** which makes it easier to see the difference this setting makes. The points don't change, but the bars change in relationship to the points. Choose from the following:

- **Before** ![Bar alignment before icon](/static/img/docs/time-series-panel/bar-alignment-before.png)
  The bar is drawn before the point. The point is placed on the trailing corner of the bar.
- **Center** ![Bar alignment center icon](/static/img/docs/time-series-panel/bar-alignment-center.png)
  The bar is drawn around the point. The point is placed in the center of the bar. This is the default.
- **After** ![Bar alignment after icon](/static/img/docs/time-series-panel/bar-alignment-after.png)
  The bar is drawn after the point. The point is placed on the leading corner of the bar.

### Axis options

{{< docs/shared lookup="visualizations/axis-options-2.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Canvas

Canvases combine the power of Grafana with the flexibility of custom elements.
They are extensible visualizations that allow you to add and arrange elements wherever you want within unstructured static and dynamic layouts.
This lets you design custom visualizations and overlay data in ways that aren't possible with standard Grafana visualizations, all within the Grafana UI.

{{< video-embed src="/static/img/docs/canvas-panel/canvas-beta-overview-9-2-0.mp4" max-width="750px" alt="Canvas beta overview" >}}

If you've used popular UI and web design tools, then designing canvases will feel very familiar.
With all of these dynamic elements, there's almost no limit to what a canvas can display.

{{< admonition type="note" >}}
We'd love your feedback on the canvas visualization. Please check out the [open GitHub issues](https://github.com/grafana/grafana/issues?page=1&q=is%3Aopen+is%3Aissue+label%3Aarea%2Fpanel%2Fcanvas) and [submit a new feature request](https://github.com/grafana/grafana/issues/new?assignees=&labels=type%2Ffeature-request,area%2Fpanel%2Fcanvas&title=Canvas:&projects=grafana-dataviz&template=1-feature_requests.md) as needed.
{{< /admonition >}}

## Plugin information

Starting with Grafana 13.3, the canvas visualization loads as a standalone plugin by default, instead of as a core plugin. The standalone plugin can get updates between Grafana releases. Existing dashboards that use canvas continue to work.

Grafana bundles the standalone plugin, so you don't need to install it. If you use a slim Docker image, install the plugin from **Administration > Plugins and data > Plugins** first.

In Grafana 13.3 and later 13.x releases, you can go back to the core plugin. To do so, add the following to your [configuration file](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/) and restart Grafana:

```ini
[plugin.canvas]
as_external = false
```

Grafana 14.0 will remove the core plugin. After you upgrade to Grafana 14.0, the standalone plugin will be the only canvas plugin, and the `as_external` setting will have no effect.

{{< admonition type="note" >}}
On Grafana Cloud, the canvas plugin is managed by Grafana and updates automatically.
{{< /admonition >}}

## Configure a canvas visualization

The following video shows you how to create and configure a canvas visualization:

{{< youtube id="b7AYKoFcPpY" >}}

## Supported data formats

The canvas visualization is unique in that it doesn't have any specific data requirements. You can even start adding and configuring visual elements without providing any data. However, any data you plan to consume should be accessible through supported Grafana data sources and structured in a way that ensures smooth integration with your custom elements.

If your canvas is going to update in real time, your data should support refreshes at your desired intervals without degrading the user experience.

You can tie [Elements](#elements) and [Connections](#connections) to data through options like text, colors, and background pattern images, etc. available in the canvas visualization.

## Elements

Elements are the basic building blocks of a canvas and they help you visualize data with different shapes and options. You can rotate and move elements around the canvas. When you move elements, snapping and alignment guides help you create more precise layouts.

Add elements in the [Layer](#layer-options) section of canvas options.

{{< admonition type="note" >}}
Element snapping and alignment only works when the canvas is not zoomed in.
{{< /admonition >}}

### Element types

When you select an element that you've added to a canvas, you can access [configuration options](#selected-element-options) for it that are dependent on the element type.

The following sections describe the different elements available.

{{< column-list >}}

- [Metric value](#metric-value)
- [Text](#text)
- [Ellipse](#basic-shapes)
- [Rectangle](#basic-shapes)
- [Icon](#icon)
- [Server](#server)
- [Triangle](#basic-shapes)
- [Cloud](#basic-shapes)
- [Parallelogram](#basic-shapes)
- [Button](#button)

{{< /column-list >}}

#### Basic shapes

A basic shape element can display text (both fixed and field data) and its background color can be changed based on data thresholds. You can add the following basic shapes to a canvas:

- Cloud
- Ellipse
- Parallelogram
- Rectangle
- Triangle

#### Metric value

The metric value element lets you select the data you want to display on a canvas. This element has a unique “edit” mode that can be triggered either through the context menu “Edit” option or by double clicking. When in edit mode you can select which field data that you want to display.

#### Text

The text element lets you add text to the canvas. The element also supports an editing mode, triggered via either double clicking or the edit menu option in the context menu.

#### Icon

The icon element lets you add a supported icon to the canvas. Icons can have their color set based on thresholds or value mappings.

##### Add a custom icon

You can add a custom icon by referencing an SVG file. To add a custom icon, follow these steps:

1. Under **Icon > SVG Path**, if it's not already selected, select **Fixed** as your file source.
1. Click **Select a value** in the field below.
1. In the dialog box that opens, click the **URL** tab.
1. Enter the URL in the field below the **URL** tab.

   {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-canvas-custom-image-v11.3.png" max-width="250px" alt="Add a custom image URL" >}}

1. Click **Select**.
1. (Optional) Add a background image to your icon with the **Background (icon)** option by following the steps to [add a custom image](#add-custom-images-to-elements).

If you don't have an SVG file, you can use a rectangle element instead of an icon and set its background image to an image file type. To add a custom image for another element type, follow the steps to [add a custom image](#add-custom-images-to-elements).

#### Server

The server element lets you easily represent a single server, a stack of servers, a database, or a terminal. Server elements support status color, bulb color, and a bulb blink rate all configurable by fixed or field values.

{{< figure src="/media/docs/grafana/canvas-server-element-9-4-0.png" max-width="650px" alt="Canvas server element" >}}

#### Button

The button element lets you add a basic button to the canvas. Button elements are available when **Experimental element types** is enabled. They support triggering basic, unauthenticated API calls. [API settings](#button-api-options) are found in the button element editor. You can also pass template variables in the API editor.

{{< admonition type="note" >}}
A button click will only trigger an API call when [inline editing](#inline-editing) is disabled.
{{< /admonition >}}

{{< video-embed src="/media/docs/grafana/2023-20-10-Canvas-Button-Element-Enablement-Video.mp4" max-width="650px" alt="Canvas button element demo" >}}

{{< docs/play title="Canvas Visualization: Buttons" url="https://play.grafana.org/d/c9ea65f5-ed5a-45cf-8fb7-f82af7c3afdf/" >}}

##### Button API options

The following options let you configure basic, unauthenticated API calls:

<!-- prettier-ignore-start -->
| Option  | Description  |
| ------- | ------------ |
| Endpoint | Enter the endpoint URL. |
| Method | Choose from **GET**, **POST**, and **PUT**. |
| Content-Type | Select an option in the drop-down list. Choose from: JSON, Text, JavaScript, HTML, XML, and x-www-form-urlencoded. |
| Query parameters | Enter as many **Key**, **Value** pairs as you need. |
| Header parameters | Enter as many **Key**, **Value** pairs as you need. |
| Success message | Enter a message that will appear if the API call is successful. If you leave the field empty, the default message "API call was successful" appears. |
| Payload | Enter the body of the API call. |

<!-- prettier-ignore-end -->

### Add custom images to elements

You can add custom background images to all elements except **Button** by referencing an image URL.
The image must be hosted at a URL that allows requests from your Grafana instance.

To upload a custom image, follow these steps:

1. Under **Background (\<ELEMENT TYPE\>)**, if it's not already selected, select **Fixed** as your image source.

   {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-canvas-custom-image-src-v11.3.png" max-width="300px" alt="Custom image source selection" >}}

1. Click **Select a value** in the field below.
1. In the dialog box that opens, click the **URL** tab.
1. Enter the URL in the field below the **URL** tab.

   {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-canvas-custom-image-v11.3.png" max-width="250px" alt="Add a custom image URL" >}}

1. Click **Select**.

## Connections

When building a canvas, you can connect elements together to create more complex visualizations. You can also create connections to the background of the canvas.

To create a connection, follow these steps:

1. In the panel edit pane, expand the **Canvas** options section.
1. Toggle on the **Inline editing** switch.
1. Hover the cursor over an element you want to connect to display the connection anchors:

   ![Element with connection anchors displayed](/media/docs/grafana/panels-visualizations/screenshot-connection-anchors-v11.3.png)

1. Drag the cursor from a connection anchor on that element to one on another element.

To remove a connection, click the connection and then press the `Delete` or `Backspace` key.

### Connection adjustments

You can adjust connections, adding angles to them, to fit the canvas you're working in. When you move connected elements, the connection resizes to fit the space.

- To adjust a connection, click it to display the midpoint controls and move those as needed.
- To make a connection a straight line again, move the midpoint back until the midpoint controls disappear.

If you move a connection so that it's almost a right angle or a straight line, the connection snaps into that angle or into a straight line.

### Connection styles

You can set the size, color, direction, and style of connections based on fixed or field values. To do so, enter into panel edit mode, select the connection, and modify the connection's properties in the panel editor. For more information on connection styles, refer to [Selected connection options](#selected-connection-options).

{{< youtube id="0iO2gqv0XNA" >}}

## Canvas editing

You can make changes to a canvas visualization while in the context of the dashboard, or in dashboard mode. The following sections describe the editing options available in dashboard mode.

### Inline editor

You can edit your canvas inline while in dashboard mode. The inline editor menu displays the options relevant to the part of the canvas that you've selected. You can also move the editor window around.

{{< video-embed src="/static/img/docs/canvas-panel/canvas-inline-editor-9-2-0.mp4" max-width="750px" alt="Inline editor demo" >}}

### Context menu

The context menu lets you perform common tasks quickly and efficiently. Supported functionality includes opening and closing the inline editor, duplicating an element, deleting an element, and more.

The context menu is triggered by a right click action over the panel or over a given canvas element. When right clicking the panel, you are able to set a background image and easily add elements to the canvas.

{{< figure src="/static/img/docs/canvas-panel/canvas-panel-context-menu-9-3-0.png" max-width="350px" alt="Canvas panel context menu" >}}

When right clicking an element, you are able to edit, delete, duplicate, and modify the element's layer positioning.

{{< figure src="/static/img/docs/canvas-panel/canvas-context-menu-9-2-0.png" max-width="250px"  alt="Canvas element context menu" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Canvas options

#### Inline editing

The inline editing toggle lets you lock or unlock the canvas. When turned off the canvas becomes “locked”, freezing elements in place and preventing unintended modifications.

{{< video-embed src="/static/img/docs/canvas-panel/canvas-inline-editing-toggle-9-2-0.mp4" max-width="750px" alt="Inline editing toggle demo" >}}

#### Experimental Element types

Toggle the switch to include experimental element types in the available selections, including **Button**, **Wind Turbine**, **Drone Top**, **Drone Front**, and **Drone Side**. This option is on by default.

### Tooltip options

The **Tooltip mode** setting controls the display of tooltips when hovering over canvas elements that are connected to data, data links, or actions.
The options are:

- **Enabled** - Show a tooltip when the cursor hovers over an element.
- **Disabled** - Tooltips are not shown on hover.

The **Disable for one-click elements** setting allows hiding tooltips on elements that have one-click functionality enabled. This prevents tooltips from interfering with one-click interactions while still allowing tooltips on other elements.

### Layer options

The **Layer** options let you add elements to the canvas and control its appearance:

- [Elements](#elements-layer)
- [Background](#background-canvas)
- [Border](#border-canvas)

#### Elements (layer)

Use the **Add item** button to open the [element type](#element-types) drop-down list. Elements are listed in the reverse order that you add them to the canvas:

![Canvas elements added in the Layer options](/media/docs/grafana/panels-visualizations/screenshot-canvas-elements-v11.3.png)

By default, elements have numbered names, like "Element 1", and those numbers correspond to the order in which the elements were added, but you can [change the default names](#rename-an-element).

You can also take the following actions on elements:

- Change the order of elements by clicking and holding the row of the element while moving it up and down in the element list.
- Duplicate or remove elements by clicking the icons on the element row.
- Access the element editing options by clicking the element row. This displays the [Selected element](#selected-element-options) section of options. Click **Clear selection** to remove the element from focus and stop displaying that section of options.

##### Rename an element

To update the name of an element, follow these steps:

1. Hover the cursor over the element name so the **Edit layer name** (pencil) icon is displayed.
1. Click the **Edit layer name** icon.
1. Enter a new name.
1. Click outside of the name field.

#### Background (canvas)

Use the following options to control the background of the canvas:

| Option     | Description                                                                                               |
| ---------- | --------------------------------------------------------------------------------------------------------- |
| Color      | Set the background color.                                                                                 |
| Image      | Use one of the provided background images or [add your own custom image](#add-custom-images-to-elements). |
| Image size | Control the size of the image or set it as a tile.                                                        |

#### Border (canvas)

Use the following options to control the border of the canvas:

| Option | Description                                                                                     |
| ------ | ----------------------------------------------------------------------------------------------- |
| Width  | Set the border width in pixels.                                                                 |
| Color  | Set the border color. This option is only displayed when the border width is greater than zero. |
| Radius | Add rounded corners to the border and control the degree of curve.                              |

### Selected element options

The following options allow you to control the appearance of the element you've selected. To access an element so that you can edit it, expand the **Layer** section and select the desired element.

| Option                                            | Description                                                                                     |
| ------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| [Element type](#element-type)                     | Change the selected element type.                                                               |
| [Element](#element)                               | Control the appearance of text on the element. This section is named based on the element type. |
| [Layout](#layout)                                 | Control the placement of elements on the canvas.                                                |
| [Background (element)](#background-element)       | Set the background of the element.                                                              |
| [Border (element)](#border-element)               | Set the border of the element.                                                                  |
| [Data links and actions](#data-links-and-actions) | Configure data links and actions for elements.                                                  |

#### Element type

You can change element type by making a new selection in the drop-down list:

![Cursor on the element type selection drop-down](/media/docs/grafana/panels-visualizations/screenshot-element-type-select-v11.3.png)

#### Element

This section is named based on the element type. Control the appearance of text on the element with the following options:

<!-- prettier-ignore-start -->

| Option         | Description                                               |
| -------------- | --------------------------------------------------------- |
| Style          | Buttons only. Select an option in the **Variant** drop-down list to indicate what kind of action the button initiates. Choose from **primary**, **secondary**, **success**, and **destructive**. |
| Text           | Select a **Source**. Choose from **Fixed** or **Field**. If you selected **Fixed**, enter text in the **Value** field. If you selected **Field**, choose the field. |
| Text color     | Choose a text color.     |
| Align text     | Set the horizontal alignment of text within the element. Choose from **Left**, **Center**, and **Right**.  |
| Vertical align | Set the vertical alignment of the text within the element. Choose from **Top**, **Middle**, and **Bottom**. |
| Text size      | Set the text size. Leave the field empty to allow Grafana to automatically set the text size. |
| API      | Buttons only. Configure API options. For more information, refer to [Button API options](#button-api-options).   |

<!--prettier-ignore-end -->

<!-- prettier-ignore-start -->

Icons don't have text, so they have different options:

| Option         | Description                                               |
| -------------- | --------------------------------------------------------- |
| SVG Path     | Choose whether the icon SVG file source is **Fixed** or **Field**. If you selected **Fixed**, choose a provided option or [add a custom icon](#add-a-custom-icon). If you selected **Field**, choose a field. |
| Fill color     | Choose a fill color for the icon.   |

<!--prettier-ignore-end -->

#### Layout

Control the placement of elements on the canvas with the following options:

<!-- prettier-ignore-start -->

| Option          | Description     |
| --------------- | --------------- |
| Quick placement | Select an alignment option to automatically place the element. Choose from:<ul><li>Align left</li><li>Align horizontal centers</li><li>Align right</li><li>Align top</li><li>Align vertical centers</li><li>Align bottom</li></ul> |
| Constraints     | Set element constraints. Choose from: **Left**, **Right**, **Left & Right**, **Center**, and **Scale**.<br></br>Use the **Scale** option to ensure that elements are automatically resized when the panel size changes. |
| Position        | Use these settings to manually set the position of an element. Set any or all of the following options: **top**, **left**, **width**, **height**, and **rotation**. |

<!-- prettier-ignore-end -->

#### Background (element)

Use the following options to set the background of the element:

- **Color** - Set the background color.
- **Image** - Use one of the provided background images or [add your own custom image](#add-custom-images-to-elements).

This option doesn't apply to the button element.

#### Border (element)

Use the following options to set the border of the element:

- **Width** - Set the border width in pixels.
- **Color** - Set the border color. This option is only displayed when the border width is greater than zero.
- **Radius** - Add rounded corners to the element border and control the degree of curve.

#### Data links and actions

Canvases support [data links](ref:data-links) and actions for all elements.

If you add multiple data links or actions to an element, you can control the order in which they appear in the element tooltip.
To do this, click and drag the link or action to the desired position.

The following tasks describe how to configure data links and actions.

{{< tabs >}}
{{< tab-content name="Add data links" >}}
To add a data link, follow these steps:

1. Enable inline editing.
1. Click the element to which you want to add the data link.
1. In either the inline editor or panel editor, expand the **Selected element** editor.
1. Scroll down to the **Data links and actions** section and expand it.
1. Click **+ Add link**.
1. In the dialog box that opens, enter a **Title**.

   This is a human-readable label for the link displayed in the UI. This is a required field.

1. Enter the **URL** or variable to which you want to link.

   To add a data link variable, click in the **URL** field and enter `$` or press Ctrl+Space or Cmd+Space to see a list of available variables. This is a required field.

1. If you want the link to open in a new tab, toggle the **Open in a new tab** switch.
1. If you want the data link to open with a single click on the element, toggle the **One click** switch.

   Only one data link or action can have **One click** enabled at a time.

1. Click **Save** to save changes and close the dialog box.
1. Disable inline editing.
   {{< /tab-content >}}
   {{< tab-content name="Add actions" >}}
   To add an action, by follow these steps:

1. Enable inline editing.
1. Click the element to which you want to add the data link.
1. In either the inline editor or panel editor, expand the **Selected element** editor.
1. Scroll down to the **Data links and actions** section and expand it.
1. Click **+ Add action**.
   In the dialog box that opens, set the action options:

   | Option               | Description                                                                                                                                                                                                                                 |
   | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | Title                | A human-readable label for the action that's displayed in the UI.                                                                                                                                                                           |
   | Confirmation message | A descriptive prompt to confirm or cancel the action.                                                                                                                                                                                       |
   | One click            | If you want the action to be triggered by a single click on the element, toggle the switch.</p><p>Only one data link or action can have **One click** enabled at a time.                                                                    |
   | Connection           | Specify how the action's HTTP request is sent. Choose from: **Direct from browser** or routed through a configured data source.                                                                                                             |
   | Method               | Select from **POST** or **GET**.                                                                                                                                                                                                            |
   | URL                  | The request URL or variable to which you want to link.</p><p>To add a variable, click in the **URL** field and enter `$` or press Ctrl+Space or Cmd+Space to see a list of available variables.                                             |
   | Variables            | **Key** and **Name** pairs with a type selection. Click the **+** icon to add as many variables as you need. To add a variable to the request, prefix the key with `$`. You can set the values for the variables when performing an action. |
   | Query parameters     | **Key** and **Value** pairs. Click the **+** icon to add as many key/value pairs as you need.                                                                                                                                               |
   | Headers              | Comprised of **Key** and **Value** pairs and a **Content-Type**.</p><p>Click the **+** icon to add as many key/value pairs as you need.                                                                                                     |
   | Content-Type         | Select from the following: **application/json**, **text/plain**, **application/xml**, and **application/x-www-form-urlencoded**.                                                                                                            |
   | Body                 | The body of the request.                                                                                                                                                                                                                    |

1. Click **Save** to save changes and close the dialog box.
1. Disable inline editing.
   {{< /tab-content >}}
   {{< /tabs >}}

### Selected connection options

You can style the selected connection using the following options:

- **Color** - Set the connection color.
- **Size** - Control the size of the connection by entering a number in the **Value** field.
- **Radius** - Add curve to the connection by entering a value to represent the degree.
- **Direction** - Control the appearance of the arrow head. Choose your source from **Fixed** or **Field**. The default value is **Forward** regardless of the source type.

  If the direction source is **Fixed**, choose from:
  - **Forward** - The arrow head points in the direction in which the connection was drawn.
  - **Reverse** - The arrow head points in the opposite direction of which the connection was drawn.
  - **Both** - Adds arrow heads to both ends of the connection.
  - **None** - Removes the arrow head.

  If the direction source is **Field**, select a field that contains numeric values:
  - **Positive values** - Display forward arrows.
  - **Negative values** - Display reverse arrows.
  - **Zero** - Display no arrow heads.

- **Line style** - Choose from the following line styles: **Solid**, **Dashed**, and **Dotted**.

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Dashboard list

Dashboard lists allow you to display dynamic links to other dashboards. You can configure the list to use starred dashboards, recently viewed dashboards, a search query, and dashboard tags.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-dashboard-list-v11.6.png" max-width="750px" alt="A dashboard list visualization" >}}

On each dashboard load, this panel queries the dashboard list, always providing the most up-to-date results.

Dashboards that have a description show an info icon next to the dashboard link; hover over the icon to view the description.

You can use a dashboard list visualization to display a list of important dashboards that you want to track.

## Configure a dashboard list visualization

Once you’ve created a [dashboard](ref:dashboard), the following video shows you how to configure a dashboard list visualization:

{{< youtube id="MserjWGWsh8" >}}

{{< docs/play title="Dashboard List Visualization" url="https://play.grafana.org/d/fdlojrg7daebka/" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Dashboard list options

Use the following options to refine your dashboard list visualization.

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Include current time range | Propagate the time range of the current dashboard to the dashboard list links. When you click a link, the linked dashboard opens with the indicated time range already set. |
| Include current template variable values | Include template variables that are being used as query parameters in the dashboard list link. When you click the link, any matching templates in the linked dashboard are set to the values from the link. Learn more in [Dashboard URL variables](ref:dashboard-url-variables). |
| Starred | Display starred dashboards in alphabetical order. |
| Recently viewed | Display recently viewed dashboards in view order, with the most recently viewed dashboards first. |
| Search | Display dashboards returned by search. Results use the **Query**, **Folder**, and **Tags** filters. An empty **Query** matches all dashboard names. Variable interpolation is supported for **Query** and **Tags**. For example, `$my_var` or `${my_var}`. |
| Show headings | Headings for enabled sections are displayed. Sections are:<ul><li>**Starred dashboards**</li><li>**Recently viewed dashboards**</li><li>**Search**</li> |
| Show folder names | Display the name of the folder where the dashboard is located. |
| Max items | Set the maximum number of items to list per section. If you enter "10" and enable **Starred** and **Recently viewed** dashboards, the panel displays up to 20 total dashboards, 10 in each section. |
| [Query](#query) | Search by dashboard name. This option is only applied when the **Search** switch is toggled on. |
| [Folder](#folder) | Only dashboards from the selected folder are displayed in the dashboard list. This option is only applied when the **Search** switch is toggled on. |
| [Tags](#tags) | Search by tags. This option is only applied when the **Search** switch is toggled on.  |

<!-- prettier-ignore-end -->

#### Query

Use this field to search by dashboard name. Query terms are case-insensitive and partial values are accepted.
For example, if you have dashboards called "Indoor Temps" and "Outdoor temp", entering the word "temp" returns both results.
This option is only applied when the **Search** switch is toggled on.

#### Folder

Only dashboards from the selected folder are included in search results and displayed in the dashboard list.
To include all dashboards in search results, select the top-level **Dashboards** folder.
This option is only applied when the **Search** switch is toggled on.

#### Tags

Enter tags by which you want to search. Note that tags don't appear as you type, and they're case sensitive.
When you specify multiple tags, dashboards must match all tags.

When multiple tags, query strings, and folders appear, the dashboard list displays dashboards matching _all_ conditions.

This option is only applied when the **Search** switch is toggled on.

# Flame graph

Flame graphs let you visualize [profiling](https://grafana.com/docs/pyroscope/latest/introduction/what-is-profiling/) data. Using this visualization, a [profile](https://grafana.com/docs/pyroscope/latest/view-and-analyze-profile-data/profiling-types/) can be represented as a [flame graph](#flame-graph-mode), [top table](#top-table-mode), or [call tree](#call-tree-mode), shown individually or two at a time, side by side.

For example, if you want to understand which parts of a program consume the most resources, such as CPU time, memory, or I/O operations, you can use a flame graph to visualize and analyze where potential performance issues are:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-flamegraph-dark-v12.0.png" max-width="750px" alt="A flame graph visualization for a system profile with both flame graph and top table mode." >}}

You can use a flame graph visualization if you need to:

- Identify any performance bottlenecks to find where code optimizations may be needed.
- Diagnose the root cause of any performance degradation.
- Analyze the behavior of complex systems, including distributed systems or microservices architectures.

To learn more about how Grafana Pyroscope visualizes flame graphs, refer to [Flame graphs: Visualizing performance data](https://grafana.com/docs/pyroscope/latest/view-and-analyze-profile-data/flamegraphs/).

## Configure a flame graph visualization

Once you’ve created a [dashboard](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/create-dashboard/), the following video shows you how to configure a flame graph visualization:

{{< youtube id="VEvK0JkPlOY" >}}

{{< docs/play title="Flame Graphs" url="https://play.grafana.org/d/cdl34qv4zzg8wa/" >}}

## Supported data formats

To render a flame graph, you must format the data frame data using a _nested set model_.

A nested set model ensures each item of a flame graph is encoded by its nesting level as an integer value, its metadata, and by its order in the data frame. This means that the order of items is significant and needs to be correct. The ordering is a depth-first traversal of the items in the flame graph which recreates the graph without needing variable-length values in the data frame like in a children's array.

Required fields:

| Field name | Type           | Description                                                                                                                 |
| ---------- | -------------- | --------------------------------------------------------------------------------------------------------------------------- |
| level      | number         | The nesting level of the item, which represents how many items are between this item and the top item of the flame graph.   |
| value      | number         | The absolute or cumulative value of the item. This translates to the width of the item in the graph.                        |
| label      | string or enum | Label to be shown for the particular item.                                                                                  |
| self       | number         | Self value, which is usually the cumulative value of the item minus the sum of cumulative values of its immediate children. |

Diff profiles can also include optional `valueRight` and `selfRight` fields. When present, the tooltip and top table show baseline, comparison, and diff values.

### Example

The following table is an example of the type of data you need for a flame graph visualization and how it should be formatted:

| level | value    | self   | label                                     |
| ----- | -------- | ------ | ----------------------------------------- |
| 0     | 16.5 Bil | 16.5 K | total                                     |
| 1     | 4.10 Bil | 4.10 k | test/pkg/agent.(\*Target).start.func1     |
| 2     | 4.10 Bil | 4.10 K | test/pkg/agent.(\*Target).start.func1     |
| 3     | 3.67 Bil | 3.67 K | test/pkg/distributor.(\*Distributor).Push |
| 4     | 1.13 Bil | 1.13 K | compress/gzip.(\*Writer).Write            |
| 5     | 1.06 Bil | 1.06 K | compress/flat.(\*compressor).write        |

## Flame graph mode

A flame graph takes advantage of the hierarchical nature of profiling data. It condenses data into a format that allows you to easily see which code paths are consuming the most system resources, such as CPU time, allocated objects, or space when measuring memory. Each block in the flame graph represents a function call in a stack and its width represents its value.

Grayed-out sections are a set of functions that represent a relatively small value and they are collapsed together into one section for performance reasons.

{{< figure src="/static/img/docs/flame-graph-panel/flame-graph-mode-dark.png" max-width="700px" alt="A flame graph visualization for a system profile with flame graph mode." >}}

You can hover over a specific function to view a tooltip that shows you additional data about that function, like the function's value, percentage of total value, and the number of samples with that function.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-flamegraph-tooltip-v11.6.png" max-width="700px" alt="A flame graph visualization with a hover tooltip." >}}

### Menu actions

You can click a function to show a drop-down menu with additional actions:

- [Focus block](#focus-block)
- [Copy function name](#copy-function-name)
- [Sandwich view](#sandwich-view)
- [Grouping](#grouping)

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-flamegraph-menu-v12.0.png" max-width="700px" alt="A flame graph visualization with drop-down actions." >}}

#### Focus block

When you click **Focus block**, the block, or function, is set to 100% of the flame graph's width and all its child functions are shown with their widths updated relative to the width of the parent function. This makes it easier to drill down into smaller parts of the flame graph.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-flamegraph-focus-v11.6.png" max-width="700px" alt="A flame graph visualization with focus block action selected." >}}

#### Copy function name

When you click **Copy function name**, the full name of the function that the block represents is copied.

#### Sandwich view

The sandwich view allows you to show the context of the clicked function. It shows all the function's callers on the top and all the callees at the bottom. This shows the aggregated context of the function so if the function exists in multiple places in the flame graph, all the contexts are shown and aggregated in the sandwich view.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-flamegraph-sandwich-v11.6.png" max-width="700px" alt="A flame graph visualization with sandwich view selected.">}}

#### Grouping

Under the **Grouping** section of the menu, the following options let you expand and collapse groups of functions:

- **Expand group** - Expands the grouped function you've clicked. Displayed if you click a function that's been automatically grouped in the flame graph.
- **Expand all groups** - Expands all grouped functions in the flame graph. Always displayed when you click the graph.
- **Collapse group** - Collapses the expanded function you've clicked. Displayed if you click a function in the flame graph that's been manually expanded.
- **Collapse all groups** - Collapses all expanded functions in the flame graph. Displayed if there are any expanded functions when you click the graph.

### Status bar

The status bar shows metadata about the flame graph and currently applied modifications, like what part of the graph is in focus or what function is shown in sandwich view. Click the **X** in the status bar pill to remove that modification.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-flamegraph-status-v11.6.png" max-width="730px" alt="A flame graph visualization's status bar.">}}

## Top table mode

The top table shows the functions from the profile in table format. The table has three columns: **Symbol**, **Self**, and **Total**. The table is sorted by self time by default, but can be reordered by total time or symbol name by clicking the column headers. Each row represents aggregated values for the given function if the function appears in multiple places in the profile.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-flamegraph-toptable-v12.0.png" max-width="700px" alt="Table view">}}

There are also action buttons on the left-most side of each row. The first button searches for the function name while second button shows the sandwich view of the function.

## Call tree mode

The call tree shows the profile's call hierarchy as an expandable tree. Each row represents a function and has **Self** and **Total** values: **Self** is the resources spent in the function itself, and **Total** is the resources spent in the function together with everything it calls.

{{< figure src="/media/docs/pyroscope/screenshot-flamegraph-call-tree.png" max-width="700px" alt="A flame graph visualization showing the toolbar and a Call tree view alongside the flame graph." >}}

To explore the tree, click the expand icon on a row to reveal its child functions, and click it again to collapse them. Expanding a branch lets you follow a single call path from a parent function down to the functions it calls.

Use the call tree when you want to read the call hierarchy directly, one path at a time, instead of scanning block widths in the flame graph or aggregated rows in the top table. It works well as a substitute for the callers side of [Sandwich view](#sandwich-view) when you want to trace how a function is reached in the call hierarchy.

## Toolbar

The following table lists the features of the toolbar:

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| [Search](#search) | Use the search field to find functions with a particular name. All the functions in the flame graph that match the search will remain colored while the rest of the functions appear in gray. |
| Reset | Reset the flame graph back to its original state from a focus block or sandwich view. The reset icon is only displayed when the flame graph is in one of those two states. |
| [Change color scheme](#change-color-scheme) | Switch between **By value** and **By package name** to visually tie functions from the same package together. |
| Grouping | Expand or collapse all groups to show all instances of a function or show the function grouped. |
| Text align | Align text either to the left or to the right to show more important parts of the function name when it does not fit into the block. |
| Visualization picker | Switch between **Single** and **Split** view. In **Single** view, one pane shows the **Top table**, **Flame graph**, or **Call tree**. In **Split** view, two panes appear side by side, and each pane can independently show the **Top table**, **Flame graph**, or **Call tree**. |

<!-- prettier-ignore-end -->

### Search

You can use the search field to find functions with a particular name. All the functions in the flame graph that match the search will remain colored while the rest of the functions are grayed-out.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-flamegraph-search-v12.0.png" max-width="700px" alt="Searching for a function name in a flame graph visualization.">}}

### Change color scheme

You can switch between **By value** and **By package name** to visually tie functions from the same package together.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-flamegraph-color-v11.6.png" max-width="700px" alt="Different color scheme" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Standard options

**Standard options** in the panel editor pane let you change how field data is displayed in your visualizations.
When you set a standard option, the change is applied to all fields or series.
For more granular control over the display of fields, refer to [Configure field overrides](ref:configure-field-overrides).

You can customize the following standard options:

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Unit | This option lets you choose which unit a field should use. For more information on unit options as well as creating custom units, refer to the [unit configuration documentation](ref:units). |
| Decimals | Specify the number of decimals Grafana includes in the rendered value. |

<!-- prettier-ignore-end -->

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Frequently asked questions

{{< qa-list >}}
{{< qa question="What can I learn from a flame graph?" >}}
A flame graph helps you understand how your application spends time or consumes resources during execution.
By visualizing the call hierarchy and the relative cost of each function, you can identify which parts of your application contribute most to overall resource usage and explore how functions relate to one another.
{{< /qa >}}
{{< qa question="How do I read a flame graph?" >}}
A flame graph represents a hierarchy of function calls collected during profiling.
Each block represents a function, and its position in the graph shows where it appears in the call stack.
The width of a block indicates how much time or resources were spent in that function, making wider blocks a good starting point for identifying performance bottlenecks.
You can click a function to inspect its call hierarchy, use Sandwich view to examine its callers and callees, switch to the Call tree to expand the call hierarchy one path at a time, or switch to the Top table to see aggregated profiling data in a tabular format.
{{< /qa >}}
{{< qa question="How does a flame graph help identify performance bottlenecks?" >}}
A flame graph makes performance bottlenecks easier to spot by showing the relative cost of each function in the call stack.
Wider blocks represent functions that consume more time or resources, allowing you to quickly identify costly functions and then drill into their callers and callees to understand where optimization efforts will have the greatest impact.
{{< /qa >}}
{{< /qa-list >}}

# Gauge

Gauges are single-value visualizations that allow you to quickly visualize where a value falls within a defined or calculated min and max range. With repeat options, you can display multiple gauges, each corresponding to a different series, column, or row.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-gauge-visualization-v13.0.png" alt="A gauge visualization">}}

You can use gauges if you need to track:

- Service level objectives (SLOs)
- How full a piece of equipment is
- How fast a vehicle is moving within a set of limits
- Network latency
- Equipment state with set point and alarm thresholds
- CPU consumption (0-100%)
- RAM availability

## Configure a gauge visualization

The following video provides beginner steps for creating gauge panels. You'll learn the data requirements and caveats, special customizations, and much more:

{{< youtube id="S3qJ1cuWHkI" >}}

{{< docs/play title="Grafana Gauge Visualization" url="https://play.grafana.org/d/KIhkVD6Gk/" >}}

## Supported data formats

To create a gauge visualization you need a dataset containing at least one numeric field. These values are identified by the field name. Additional text fields aren't required but can be used for identification and labeling.

### Example - One row, multiple values

The gauge visualization can support multiple fields in a dataset. <!-- In this case, multiple gauges are displayed. -->

| value1 | value2 | value3 |
| ------ | ------ | ------ |
| 5      | 3      | 10     |

![Gauge visualization with multiple numeric values in a single row](/media/docs/grafana/panels-visualizations/screenshot-gauge-1-row-many-vals-v13.0.png)

When there are multiple values in the dataset, the visualization displays multiple gauges and automatically defines the minimum and maximum. In this case, those are 3 and 10. Because the minimum and maximum values are defined, each gauge is shaded in to show that value in relation to the minimum and maximum.

### Example - Multiple rows and values

The gauge visualization can display datasets with multiple rows of data or even multiple datasets.

| Identifier | value1 | value2 | value3 |
| ---------- | ------ | ------ | ------ |
| A          | 5      | 3      | 10     |
| B          | 6      | 9      | 15     |
| C          | 1      | 4      | 8      |

![Gauge visualization with multiple rows and columns of numeric values showing the last row](/media/docs/grafana/panels-visualizations/screenshot-gauge-multi-rows-values-v13.0.png)

By default, the visualization [calculates](#value-options) a single value per column or series and displays only the last row of data. However, it derives the minimum and maximum from the full dataset, even if those values aren't visible.

In this example, that means only the last row of data, row C, is displayed in the gauges and the minimum and maximum values are 1 and 15. The value 1 is displayed because it's in the last row, while 15 is not.

If you want to show one gauge per table cell, under **Value options**, change the **Show** setting from **Calculate** to **All values**. Each gauge label is a concatenation of the text column with each value's column name.

![Gauge visualization with multiple rows and columns of numeric values showing all the values](/media/docs/grafana/panels-visualizations/screenshot-multi-rows-values-show-all-v13.0.png)

### Example - Control min and max

You can control the minimum and maximum values by defining them directly in your dataset.
This is an alternative to setting it manually in the field options.

| value | max | min |
| ----- | --- | --- |
| 5     | 10  | 2   |

![Gauge visualization with numeric values defining max and minimum](/media/docs/grafana/panels-visualizations/screenshot-gauge-defined-min-max-v13.0.png)

In this example, to hide the min and max fields, select the "value" field in **Value options > Fields**.

![Gauge visualization with numeric values defining max and minimum but hidden](/media/docs/grafana/panels-visualizations/screenshot-gauge-hidden-min-max-v13.0.png)

Even when minimum and maximum values aren't displayed, the visualization still pulls the range from them.

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value options

Use the following options to refine how your visualization displays the value:

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Show | Set how Grafana displays your data. Choose from:<ul><li>**Calculate** - Show a calculated value based on all rows.</li><li>**All values** - Show a separate value for every row. If you select this option, then you can also limit the number of rows to display.</li></ul> |
| Calculation | If you chose **Calculate** as your **Show** option, select a reducer function that Grafana will use to reduce many fields to a single value. For a list of available calculations, refer to [Calculation types](ref:calculation-types). |
| Limit | If you chose **All values** as your **Show** option, enter the maximum number of rows to display. The default is 25 and the maximum is 5,000. |
| Fields | Select the fields display in the panel. |

<!-- prettier-ignore-end -->

### Gauge options

Adjust how the gauge is displayed.

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Style | Choose a gauge shape:<ul><li>**Circle** - Fills clockwise around a circle, starting from the 12 o'clock position.</li><li>**Arc** - Fills from left to right around an arc.</li></ul> |
| Orientation | Choose a stacking direction:<ul><li>**Auto** - Gauges display in rows and columns.</li><li>**Horizontal** - Gauges display top to bottom.</li><li>**Vertical** - Gauges display left to right.</li></ul> |
| Gauge size | Choose a gauge size mode:<ul><li>**Auto** - Grafana determines the best gauge size.</li><li>**Manual** - Manually configure the gauge size.</li></ul>This option only applies when **Orientation** is set to **Horizontal** or **Vertical**. |
| Min width | Set the minimum width of vertically oriented gauges. If you set a minimum width, the x-axis scroll bar is automatically displayed when there's a large amount of data. This option only applies when **Gauge orientation** is **Vertical** and **Gauge size** is set to **Manual**. |
| Min height | Set the minimum height of horizontally oriented gauges. If you set a minimum height, the y-axis scroll bar is automatically displayed when there's a large amount of data. This option only applies when **Gauge orientation** is **Horizontal** and **Gauge size** is set to **Manual**. |
| Bar width factor | Set a factor between 0.1 and 1 to control the width of the gauge bar relative to the total gauge area. |
| Segments | Enter a value between 1 and 100 to break the gauge into equal segments. Segments are always fully filled, even if the gauge's current value falls within a segment. |
| Bar style | Choose the shape of the gauge bar. Choose from **Flat** or **Rounded**. Only applies if **Segments** is set to `1`. Note: All segmented gauges are flat. |
| Segment spacing | Enter a value between 0 and 1 to set the factor that controls the size of the gap between each segment. This option is only applies when the **Segments** value is more than `1`. |
| Text mode | Choose what text to render in the gauge:<ul><li>**Auto** - Grafana determines the text to show.</li><li>**Value and name** - Renders both the value and series name.</li><li>**Value** - Renders only the value.</li><li>**Name** - Renders only the series name.</li><li>**None** - Renders the gauge with no labels.</li></ul> |
| [Neutral value](#neutral-value) | Set the starting value from which every gauge will be filled. |
| [Show sparkline](#show-sparkline) | Toggle on the switch to render a sparkline containing the series data for the gauge. This option is on by default. |
| [Show thresholds](#show-threshold-markers) | Control whether a threshold band is shown outside the inner gauge value band. |
| Show labels | Control whether threshold and neutral labels are shown outside of the gauge. |
| Endpoint marker | Choose how to mark the current value endpoint. Choose from **Point**, **Glow**, or **None**. **Glow** is available only in dark mode. This option only applies when the **Bar style** is **Rounded**. |
| Effects | Other styling choices you can apply to your gauge include: <ul><li>**Gradient** - Color transitions are represented with gradients for color schemes where the gauge color differs by value. This effect is on by default.</li><li>**Bar glow** - Adds a glowing shadow outside the gauge bar.</li><li>**Center glow** - The color representing the current gauge value is visible in the center of the gauge.</li></ul> |

<!-- prettier-ignore-end -->

#### Neutral value

By default, the starting value of a gauge is the minimum value, not zero.
You can change the starting value from which every gauge will be filled when you don't want it to be the minimum value.
This option is especially useful in cases where the range of data values includes negative numbers.
If you leave this empty, Grafana uses the configured minimum value.

For example, you want to display the storage of several batteries and the power range is from -2.5 kW (discharging) to 2.5 kW.
When the minimum value is used as the starting point for the gauges, the visualization looks like this:

![Gauge using min as neutral](/media/docs/grafana/panels-visualizations/screenshot-gauge-neutral-min-v13.0.png)

If you enter a neutral value of `0`, the visualization looks like this and is easier to reason about:

![Gauge with zero as neutral](/media/docs/grafana/panels-visualizations/screenshot-gauge-neutral-0-v13.0.png)

#### Show sparkline

If you want to visualize the list of values as a time series along with the calculated gauge value, toggle on the **Show sparkline** switch.
Each gauge displays the sparkline inside the circle or arc:

![Gauge with a sparkline](/media/docs/grafana/panels-visualizations/screenshot-gauge-w-sparkline-v12.4.png)

#### Show thresholds {#show-threshold-markers}

Control whether a threshold band is shown outside the inner gauge value band.

![Gauge viz with multiple rows and columns of numeric values showing all the values and thresholds defined for 0-6-11](/media/docs/grafana/panels-visualizations/screenshot-gauge-w-thresholds-v13.0.png)

### Text size options {#text-size}

Adjust the sizes of the gauge text.

- **Title** - Enter a numeric value for the gauge title size.
- **Value** - Enter a numeric value for the gauge value size.

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Geomap

Geomaps allow you to view and customize the world map using geospatial data. It's the ideal visualization if you have data that includes location information and you want to see it displayed in a map.

You can configure and overlay [map layers](#layer-type), like heatmaps and networks, and blend included basemaps or your own custom maps. This helps you to easily focus on the important location-based characteristics of the data.

{{< figure src="/static/img/docs/geomap-panel/geomap-example-8-1-0.png" max-width="750px" alt="Geomap visualization" >}}

When a geomap is in focus, in addition to typical mouse controls, you can pan around using the arrow keys or zoom in and out using the plus (`+`) and minus (`-`) keys or icons.

Geomaps are also useful when you have location data that’s changing in real time and you want to visualize where an element is moving, using auto-refresh.

You can use a geomap visualization if you need to:

- Track your fleet of vehicles and associated metrics
- Show the locations and statuses of data centers or other connected assets in a network
- Display geographic trends in a heatmap
- Visualize the relationship of your locations' HVAC consumption or solar production with the sun's location

{{< admonition type="note" >}}
We'd love your feedback on the geomap visualization. Please check out the [open Github issues](https://github.com/grafana/grafana/issues?page=1&q=is%3Aopen+is%3Aissue+label%3Aarea%2Fpanel%2Fgeomap) and [submit a new feature request](https://github.com/grafana/grafana/issues/new?assignees=&labels=type%2Ffeature-request,area%2Fpanel%2Fgeomap&title=Geomap:&projects=grafana-dataviz&template=1-feature_requests.md) as needed.
{{< /admonition >}}

## Configure a geomap visualization

The following video provides beginner steps for creating geomap visualizations. You'll learn the data requirements and caveats, special customizations, preconfigured displays and much more:

{{< youtube id="HwM8AFQ7EUs" >}}

{{< docs/play title="Geomap Examples" url="https://play.grafana.org/d/panel-geomap/" >}}

## Supported data formats

To create a geomap visualization, you need datasets containing fields with location information.

The supported location formats are:

- Latitude and longitude
- Geohash
- Lookup codes: country, US states, or airports

To learn more, refer to [Location mode](#location-mode).

Geomaps also support additional fields with various data types to define things like labels, numbers, heat sizes, and colors.

### Example - Latitude and longitude

If you plan to use latitude and longitude coordinates, the dataset must include at least two fields (or columns): one called `latitude` (you can also use`lat`), and one called `longitude` (also `lon` or `lng`). When you use this naming convention, the visualization automatically detects the fields and displays the elements. The order of the fields doesn't matter as long as there is one latitude and one longitude.

| Name            | latitude  | longitude | value |
| --------------- | --------- | --------- | ----- |
| Disneyland      | 33.8121   | -117.9190 | 4     |
| DisneyWorld     | 28.3772   | -81.5707  | 10    |
| EuroDisney      | 48.867374 | 2.784018  | 3     |
| Tokyo Disney    | 35.6329   | 139.8804  | 70    |
| Shanghai Disney | 31.1414   | 121.6682  | 1     |

If your latitude and longitude fields are named differently, you can specify them, as indicated in the [Location mode](#location-mode) section.

### Example - Geohash

If your location data is in geohash format, the visualization requires at least one field (or column) containing location data.

If the field is named `geohash`, the visualization automatically detects the location and displays the elements. The order of the fields doesn't matter and the data set can have multiple other numeric, text, and time fields.

| Name      | geohash      | trips |
| --------- | ------------ | ----- |
| Cancun    | d5f21        | 8     |
| Honolulu  | 87z9ps       | 0     |
| Palm Cove | rhzxudynb014 | 1     |
| Mykonos   | swdj02ey9gyx | 3     |

If your field containing geohash location data is not named as above, you can configure the visualization to use geohash and specify which field to use, as explained in the [Location mode](#location-mode) section.

### Example - Lookup codes

The geomap visualization can identify locations based on country, airport, or US state codes.

For this configuration, the dataset must contain at least one field (or column) containing the location code.

If the field is named `lookup`, the visualization automatically detects it and displays points based on country codes.

| Year | lookup | gdp       |
| ---- | ------ | --------- |
| 2016 | MEX    | 104171935 |
| 2016 | DEU    | 94393454  |
| 2016 | FRA    | 83654250  |
| 2016 | BRA    | 80921527  |
| 2016 | CAN    | 79699762  |

The other location types&mdash; airport codes or US state codes&mdash;aren't automatically detected.

If you want to use other codes or give the field a custom name, you can follow the steps in the [Location mode](#location-mode) section.

## Configuration options

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Map view options

The map view controls the initial view of the map when the dashboard loads.

#### Initial View

The initial view configures how the geomap renders when the panel is first loaded.

- **View** - Sets the center for the map when the panel first loads. Refer to the table following this list for view selections.
- **Zoom** - Sets the initial zoom level.
- **Use current map settings** - Use the settings of the current map to set the center.

<!-- prettier-ignore-start -->
| View selection | Description |
|---|---|
| Fit to data | fits the map view based on the data extents of Map layers and updates when data changes.<ul><li>**Data** - option allows selection of extent based on data from "All layers", a single "Layer", or the "Last value" from a selected layer.</li><li>**Layer** - can be selected if fitting data from a single "Layer" or the "Last value" of a layer.</li><li>**Padding** - sets padding in relative percent beyond data extent (not available when looking at "Last value" only).</li><li>**Max zoom** - sets the maximum zoom level when fitting data.</li> |
| (0°, 0°) |  |
| Coordinates | sets the map view based on: **Latitude** and **Longitude**. |
<!-- prettier-ignore-end -->

Default Views are also available including:

<!-- prettier-ignore-start -->
|               |               |        |        |           |
| ------------- | ------------- | ------ | ------ | --------- |
| North America | South America | Europe | Africa | West Asia |
| South Asia | South-East Asia | East Asia | Australia | Oceania |
<!-- prettier-ignore-end -->

#### Share view

The **Share view** option allows you to link the movement and zoom actions of multiple map visualizations within the same dashboard. The map visualizations that have this option enabled act in tandem when one of them is moved or zoomed, leaving the other ones independent.

{{< admonition type="note" >}}
You might need to reload the dashboard for this feature to work.
{{< /admonition >}}

#### No map repeating

The **No map repeating** option prevents the base map tiles from repeating horizontally when you pan across the world. This constrains the view to a single instance of the world map and avoids visual confusion when displaying global datasets. Enabling this option requires the map to reinitialize.

#### Sync view to dashboard variable

Stores the current map's view extents in a dashboard variable of your choosing.
This is particularly useful for dynamically querying data based on the map's current extents.

To use the option, follow these steps:

1. Create a **Custom** type dashboard variable.
2. Return to panel configuration and toggle on this switch.
3. Select the newly created variable from the **Variable name** drop-down list.
4. Use the variable in your queries.

The variable contains comma-separated coordinates (EPSG:4326): `minLon,minLat,maxLon,maxLat`

### Map layers options

Geomaps support showing multiple layers. Each layer determines how you visualize geospatial data on top of the base map.

There are three options that you need to set for all maps:

- [Layer type](#layer-type)
- [Data](#data)
- [Location mode](#location-mode)

Other options are dependent on your map layer type and are described within the layer type section.

The layer controls allow you to create layers, change their name, reorder and delete layers.

- **Add layer** creates an additional, configurable data layer for the geomap. When you add a layer, you are prompted to select a layer type. You can change the layer type at any point during panel configuration. See the **Layer Types** section above for details on each layer type.
- **Edit layer name (pencil icon)** renames the layer.
- **Trash Bin** deletes the layer.
- **Reorder (six dots/grab handle)** allows you to change the layer order. Data on higher layers will appear above data on lower layers. The visualization will update the layer order as you drag and drop to help simplify choosing a layer order.

You can add multiple layers of data to a single geomap in order to create rich, detailed visualizations.

#### Layer type

Geomap has nine data layer types and five basemap layer types.

- [Markers](#markers-layer) renders a marker at each data point.
- [Heatmap](#heatmap-layer) visualizes a heatmap of the data.
- [GeoJSON](#geojson-layer) renders static data from a GeoJSON file.
- [Night / Day](#night--day-layer) renders a night / day region.
- [Route](#route-layer) render data points as a route.
- [Photos](#photos-layer) renders a photo at each data point.
- [Network](#network-layer) visualizes a network graph from the data.
- [OpenStreetMap](#openstreetmap-layer) adds a map from a collaborative free geographic world database.
- [CARTO basemap](#carto-basemap-layer) adds a layer from CARTO Raster basemaps.
- [ArcGIS MapServer](#arcgis-mapserver-layer) adds a layer from an ESRI ArcGIS MapServer.
- [XYZ Tile layer](#xyz-tile-layer) adds a map from a generic tile layer.
- [MapLibre Style layer](#maplibre-style-layer) adds a map from a MapLibre/Mapbox style URL.

There are also two experimental, alpha layer types.

- **Icon at last point (Alpha)** renders an icon at the last data point.
- **Dynamic GeoJSON (Alpha)** styles a GeoJSON file based on query results. It includes options for **GeoJSON URL**, **ID Field**, **Data style**, and **Default style**.

To enable experimental layers. Set `enable_alpha` to `true` in your configuration file:

```
[panels]
enable_alpha = true
```

To enable the experimental layers using Docker, run the following command:

```
docker run -p 3000:3000 -e "GF_PANELS_ENABLE_ALPHA=true" grafana/grafana:<VERSION>
```

#### Data

Geomaps need a source of geographical data gathered from a data source query which can return multiple datasets. By default Grafana picks the first dataset, but this drop-down allows you to pick other datasets if the query returns more than one.

#### Location Mode

There are four options to map the data returned by the selected query:

- **Auto** automatically searches for location data. Use this option when your query is based on one of the following names for data fields.
  - geohash: “geohash”
  - latitude: “latitude”, “lat”
  - longitude: “longitude”, “lng”, “lon”
  - lookup: “lookup”
- **Coords** specifies that your query holds coordinate data. You will get prompted to select numeric data fields for latitude and longitude from your database query.
- **Geohash** specifies that your query holds geohash data. You will be prompted to select a string data field for the geohash from your database query.
- **Lookup** specifies that your query holds location name data that needs to be mapped to a value. You will be prompted to select the lookup field from your database query and a gazetteer. The gazetteer is the directory that is used to map your queried data to a geographical point.

#### Markers layer

The markers layer allows you to display data points as different marker shapes such as circles, squares, triangles, stars, and more.

![Markers Layer](/static/img/docs/geomap-panel/geomap-markers-8-1-0.png)

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Data | Configure the data settings for the layer. For more information, refer to [Data](#data). |
| Location | Configure the data settings for the layer. For more information, refer to [Location Mode](#location-mode). |
| Size | Configures the size of the markers. The default is `Fixed size`, which makes all marker sizes the same regardless of the data; however, there is also an option to size the markers based on data corresponding to a selected field. `Min` and `Max` marker sizes have to be set such that the markers can scale within this range. |
| Symbol | Allows you to choose the symbol, icon, or graphic to aid in providing additional visual context to your data. Choose from assets that are included with Grafana such as simple symbols or the Unicon library. You can also specify a URL containing an image asset. The image must be a scalable vector graphic (SVG). |
| Symbol vertical align | Configures the vertical alignment of the symbol relative to the data point. Note that the symbol's rotation angle is applied first around the data point, then the vertical alignment is applied relative to the rotation of the symbol. |
| Symbol horizontal align | Configures the horizontal alignment of the symbol relative to the data point. Note that the symbol's rotation angle is applied first around the data point, then the horizontal alignment is applied relative to the rotation of the symbol. |
| Color | Configures the color of the markers. The default `Fixed color` sets all markers to a specific color. There is also an option to have conditional colors depending on the selected field data point values and the color scheme set in the `Standard options` section. |
| Fill opacity | Configures the transparency of each marker. |
| Rotation angle | Configures the rotation angle of each marker in degrees. The default is `Fixed value`, which makes all markers rotate to the same angle regardless of the data; however, there is also an option to set the rotation of the markers based on data corresponding to a selected field. |
| Text label | Configures a text label for each marker. |
| Show legend | Allows you to toggle the legend for the layer. |
| Display tooltip | Allows you to toggle tooltips for the layer. |
<!-- prettier-ignore-end -->

#### Heatmap layer

The heatmap layer clusters various data points to visualize locations with different densities.
To add a heatmap layer:

Click on the drop-down menu under Data Layer and choose `Heatmap`.

Similar to `Markers`, you are prompted with various options to determine which data points to visualize and how you want to visualize them.

![Heatmap Layer](/static/img/docs/geomap-panel/geomap-heatmap-8-1-0.png)

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Data | Configure the data settings for the layer. For more information, refer to [Data](#data). |
| Location | Configure the data settings for the layer. For more information, refer to [Location mode](#location-mode). |
| Weight values | Configures the size of the markers. The default is `Fixed size`, which makes all marker sizes the same regardless of the data; however, there is also an option to size the markers based on data corresponding to a selected field. `Min` and `Max` marker sizes have to be set such that the markers can scale within this range. |
| Radius | Configures the size of the heatmap clusters. |
| Blur | Configures the amount of blur on each cluster. |
| Opacity | Configures the opacity of each cluster. |
| Display tooltip | Allows you to toggle tooltips for the layer. |
<!-- prettier-ignore-end -->

#### GeoJSON layer

The GeoJSON layer allows you to select and load a static GeoJSON file from the filesystem.

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| GeoJSON URL | Provides a choice of GeoJSON files that are included with Grafana. You can also enter a URL manually, which supports variables. |
| Default style | Controls which styles to apply when no rules above match.<ul><li>**Color** - configures the color of the default style</li><li>**Opacity** - configures the default opacity</li></ul> |
| Style Rules | Apply styles based on feature properties <ul><li>**Rule** - allows you to select a _feature_, _condition_, and _value_ from the GeoJSON file in order to define a rule. The trash bin icon can be used to delete the current rule.</li><li>**Color** - configures the color of the style for the current rule</li><li>**Opacity** - configures the transparency level for the current rule</li> |
| Display tooltip | Allows you to toggle tooltips for the layer. |
<!-- prettier-ignore-end -->

Styles can be set within the "properties" object of the GeoJSON with support for the following geometries:

**Polygon, MultiPolygon**

- **"fill"** - The color of the interior of the polygon(s)
- **"fill-opacity"** - The opacity of the interior of the polygon(s)
- **"stroke-width"** - The width of the line component of the polygon(s)

**Point, MultiPoint**

- **"marker-color"** - The color of the point(s)
- **"marker-size"** - The size of the point(s)

**LineString, MultiLineString**

- **"stroke"** - The color of the line(s)
- **"stroke-width"** - The width of the line(s)

#### Night / Day layer

The Night / Day layer displays night and day regions based on the current time range.

{{< figure src="/static/img/docs/geomap-panel/geomap-day-night-9-1-0.png" max-width="600px" alt="Geomap panel Night / Day" >}}

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Data | Configures the dataset for the layer. For more information, refer to [Data](#data). |
| Show | Selects which panel time range endpoint drives the night and day regions. Choose **From** or **To**. |
| Night region color | Picks the color for the night region. |
| Display sun | Toggles the sun icon. |
| Opacity | Set the opacity from `0` (transparent) to `1` (opaque). |
| Display tooltip | Allows you to toggle tooltips for the layer. |
<!-- prettier-ignore-end -->

[Extensions for OpenLayers - DayNight](https://viglino.github.io/ol-ext/examples/layer/map.daynight.html)

#### Route layer

The Route layer renders data points as a route.

{{< figure src="/media/docs/grafana/geomap-route-layer-basic-9-4-0.png" max-width="600px" alt="Geomap panel Route" >}}

The layer can also render a route with arrows.

{{< figure src="/media/docs/grafana/geomap-route-layer-arrow-size-9-4-0.png" max-width="600px" alt="Geomap panel Route arrows with size" >}}

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Data | configure the data settings for the layer. For more information, refer to [Data](#data). |
| Location | configure the data settings for the layer. For more information, refer to [Location mode](#location-mode). |
| Size | sets the route thickness. Fixed value by default. When field data is selected you can set the Min and Max range in which field data can scale. |
| Color | sets the route color. Set to `Fixed color` by default. You can also tie the color to field data. |
| Fill opacity | configures the opacity of the route. |
| Text label | configures a text label for each route. |
| Arrow | sets the arrow styling to display along route, in order of data. Choose from: **None**, **Forward**, and **Reverse** |
| Display tooltip | allows you to toggle tooltips for the layer. |
<!-- prettier-ignore-end -->

[Extensions for OpenLayers - Flow Line Style](http://viglino.github.io/ol-ext/examples/style/map.style.gpxline.html)

#### Photos layer

The Photos layer renders a photo at each data point.

{{< figure src="/static/img/docs/geomap-panel/geomap-photos-9-3-0.png" max-width="600px" alt="Geomap panel Photos" >}}

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Data | Configure the data settings for the layer. For more information, refer to [Data](#data). |
| Location | Configure the data settings for the layer. For more information, refer to [Location mode](#location-mode). |
| Image Source field | Allows you to select a string field containing image data in either of the following formats:<ul><li>**Image URLs**</li><li>**Base64 encoded** - Image binary ("data:image/png;base64,...")</li></ul> |
| Kind | Sets the frame style around the images. Choose from: **Square**, **Circle**, **Anchored**, and **Folio**. |
| Crop | Toggles whether the images are cropped to fit. |
| Shadow | Toggles a box shadow behind the images. |
| Border | Sets the border size around images. |
| Border color | Sets the border color around images. |
| Radius | Sets the overall size of images in pixels. |
| Display tooltip | Allows you to toggle tooltips for the layer. |
<!-- prettier-ignore-end -->

[Extensions for OpenLayers - Image Photo Style](http://viglino.github.io/ol-ext/examples/style/map.style.photo.html)

#### Network layer

The Network layer renders a network graph. This layer supports the same [data format supported by the node graph visualization](ref:data-format) with the addition of [geospatial data](#location-mode) included in the nodes data. The geospatial data is used to locate and render the nodes on the map.

{{< figure src="/media/docs/grafana/screenshot-grafana-10-1-geomap-network-layer-v2.png" max-width="750px" alt="Geomap network layer" >}}

You can convert node graph data to a network layer:
{{< video-embed src="/media/docs/grafana/screen-recording-10-1-geomap-network-layer-from-node-graph.mp4" max-width="750px" alt="Node graph to Geomap network layer" >}}

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Data | Configure the data settings for the layer. For more information, refer to [Data](#data). |
| Location | Configure the data settings for the layer. For more information, refer to [Location mode](#location-mode). |
| Arrow | Sets the arrow direction to display for each edge, with forward meaning source to target. Choose from: **None**, **Forward**, **Reverse** and **Both**. |
| Show legend | Allows you to toggle the legend for the layer. **Note:** The legend currently only supports node data. |
| Display tooltip | Allows you to toggle tooltips for the layer. |
<!-- prettier-ignore-end -->

##### Node styles options

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Size | Configures the size of the nodes. The default is `Fixed size`, which makes all node sizes the same regardless of the data; however, there is also an option to size the nodes based on data corresponding to a selected field. `Min` and `Max` node sizes have to be set such that the nodes can scale within this range. |
| Symbol | Allows you to choose the symbol, icon, or graphic to aid in providing additional visual context to your data. Choose from assets that are included with Grafana such as simple symbols or the Unicon library. You can also specify a URL containing an image asset. The image must be a scalable vector graphic (SVG). |
| Color | Configures the color of the nodes. The default `Fixed color` sets all nodes to a specific color. There is also an option to have conditional colors depending on the selected field data point values and the color scheme set in the `Standard options` section. |
| Fill opacity | Configures the transparency of each node. |
| Rotation angle | Configures the rotation angle of each node in degrees. The default is `Fixed value`, which makes all nodes rotate to the same angle regardless of the data; however, there is also an option to set the rotation of the nodes based on data corresponding to a selected field. |
| Text label | Configures a text label for each node. |
<!-- prettier-ignore-end -->

##### Edge styles options

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Size | Configures the line width of the edges. The default is `Fixed size`, which makes all edge line widths the same regardless of the data; however, there is also an option to size the edges based on data corresponding to a selected field. `Min` and `Max` eges sizes have to be set such that the edges can scale within this range. |
| Color | Configures the color of the edges. The default `Fixed color` sets all edges to a specific color. There is also an option to have conditional colors depending on the selected field data point values and the color scheme set in the `Standard options` section. |
| Fill opacity | Configures the transparency of each edge. |
| Text label | Configures a text label for each edge. |
<!-- prettier-ignore-end -->

#### OpenStreetMap layer

A map from a collaborative free geographic world database.

{{< figure src="/static/img/docs/geomap-panel/geomap-osm-9-1-0.png" max-width="600px" alt="Geomap panel OpenStreetMap" >}}

- **Opacity** from 0 (transparent) to 1 (opaque)
- **Display tooltip** - allows you to toggle tooltips for the layer.

[About OpenStreetMap](https://www.openstreetmap.org/about)

#### CARTO basemap layer

A CARTO layer is from CARTO Raster basemaps.

- **Theme**
  - Auto
  - Light
    {{< figure src="/static/img/docs/geomap-panel/geomap-carto-light-9-1-0.png" max-width="600px" alt="Geomap panel CARTO light example" >}}
  - Dark
    {{< figure src="/static/img/docs/geomap-panel/geomap-carto-dark-9-1-0.png" max-width="600px" alt="Geomap panel CARTO dark example" >}}
- **Show labels** shows the Country details on top of the map.
- **Opacity** from 0 (transparent) to 1 (opaque)
- **Display tooltip** - allows you to toggle tooltips for the layer.

[About CARTO](https://carto.com/about-us/)

#### ArcGIS MapServer layer

An ArcGIS layer is a layer from an ESRI ArcGIS MapServer.

- **Server Instance** to select the map type.
  - World Street Map
    {{< figure src="/static/img/docs/geomap-panel/geomap-arcgis-wsm-9-1-0.png" max-width="600px" alt="Geomap panel ArcGIS World Street Map" >}}
  - World Imagery
    {{< figure src="/static/img/docs/geomap-panel/geomap-arcgis-wi-9-1-0.png" max-width="600px" alt="Geomap panel ArcGIS World Imagery" >}}
  - World Physical
    {{< figure src="/static/img/docs/geomap-panel/geomap-arcgis-wp-9-1-0.png" max-width="600px" alt="Geomap panel ArcGIS World Physical" >}}
  - Topographic
    {{< figure src="/static/img/docs/geomap-panel/geomap-arcgis-topographic-9-1-0.png" max-width="600px" alt="Geomap panel ArcGIS Topographic" >}}
  - USA Topographic
    {{< figure src="/static/img/docs/geomap-panel/geomap-arcgis-usa-topographic-9-1-0.png" max-width="600px" alt="Geomap panel ArcGIS USA Topographic" >}}
  - World Ocean
    {{< figure src="/static/img/docs/geomap-panel/geomap-arcgis-ocean-9-1-0.png" max-width="600px" alt="Geomap panel ArcGIS World Ocean" >}}
  - Custom MapServer (see [XYZ](#xyz-tile-layer) for formatting)
    - URL template
    - Attribution
- **Opacity** from 0 (transparent) to 1 (opaque)
- **Display tooltip** - allows you to toggle tooltips for the layer.

##### More Information

- [ArcGIS Services](https://services.arcgisonline.com/arcgis/rest/services)
- [About ESRI](https://www.esri.com/en-us/about/about-esri/overview)

#### XYZ Tile layer

The XYZ Tile layer is a map from a generic tile layer.

{{< figure src="/static/img/docs/geomap-panel/geomap-xyz-9-1-0.png" max-width="600px" alt="Geomap panel xyz example" >}}

- **URL template** - Set a valid tile server URL, with {z}/{x}/{y} for example: `https://tile.openstreetmap.org/{z}/{x}/{y}.png`. Dashboard variables are supported, for example, `https://example.com/maps/${version}/{z}/{x}/{y}.png`.
- **Attribution** sets the reference string for the layer if displayed in [map controls](#show-optional-attribution). Dashboard variables are supported.
- **Min zoom** - Minimum zoom level. Tiles aren't loaded below this level.
- **Max zoom** - Maximum zoom level provided by the server. Beyond this level, the map is rendered by scaling up existing tiles.
- **Opacity** from 0 (transparent) to 1 (opaque)

##### More information

- [Tiled Web Map Wikipedia](https://en.wikipedia.org/wiki/Tiled_web_map)
- [List of OpenStreetMap Tile Servers](https://wiki.openstreetmap.org/wiki/Tile_servers)

#### MapLibre Style layer

The MapLibre Style Layer is a map defined using a MapLibre/Mapbox `style.json` URL. The style contains the URL to the tiles, layer definitions, and more. Typically, they're based on vector tiles as opposed to raster tiles.

- **URL template** - Set a valid style URL. For example: `https://demotiles.maplibre.org/style.json`
- **Public access token** - An API token for mapbox maps. Only works for `mapbox://` URLs. Refer to [mapbox access tokens documentation](https://docs.mapbox.com/help/dive-deeper/access-tokens/) for more information. In other cases, you might have to include the token in the URL. For example: `https://example.com/map/style.json?key=XXX`.

### Basemap layer options

A basemap layer provides the visual foundation for a mapping application. It typically contains data with global coverage. Several base layer options
are available each with specific configuration options to style the base map.

Basemap layer types can also be added as layers. You can specify an opacity.

There are five basemap layer types to choose from in a geomap.

- [OpenStreetMap](#openstreetmap-layer) adds a map from a collaborative free geographic world database.
- [CARTO basemap](#carto-basemap-layer) adds a layer from CARTO Raster basemaps.
- [ArcGIS MapServer](#arcgis-mapserver-layer) adds a layer from an ESRI ArcGIS MapServer.
- [XYZ Tile layer](#xyz-tile-layer) adds a map from a generic tile layer.
- [MapLibre Style layer](#maplibre-style-layer) adds a map from a MapLibre/Mapbox style URL.

The default basemap layer uses the CARTO map. You can define custom default base layers in the `.ini` configuration file.

![Basemap layer options](/static/img/docs/geomap-panel/geomap-baselayer-8-1-0.png)

#### Configure the default base layer with provisioning

You can configure the default base map using config files with Grafana’s provisioning system. For more information on all the settings, refer to the [provisioning docs page](ref:provisioning-docs-page).

Use the JSON configuration option `default_baselayer_config` to define the default base map. There are currently five base map options to choose from: `carto`, `esri-xyz`, `osm-standard`, `xyz`, `maplibre`. Here are some provisioning examples for each base map option.

- **carto** loads the CartoDB tile server. You can choose from `auto`, `dark`, and `light` theme for the base map and can be set as shown below. The `showLabels` tag determines whether or not Grafana shows the Country details on top of the map. Here is an example:

```ini
[geomap]
default_baselayer_config = `{
  "type": "carto",
  "config": {
    "theme": "auto",
    "showLabels": true
  }
}`
```

- **esri-xyz** loads the ESRI tile server. There are already multiple server instances implemented to show the various map styles: `world-imagery`, `world-physical`, `topo`, `usa-topo`, and `ocean`. The `custom` server option allows you to configure your own ArcGIS map server. Here are some examples:

{{< tabs >}}
{{< tab-content name="World imagery" >}}

```ini
[geomap]
default_baselayer_config = `{
  "type": "esri-xyz",
  "config": {
    "server": "world-imagery"
  }
}`
```

{{< /tab-content >}}
{{< tab-content name="Custom" >}}

```ini
[geomap]
default_baselayer_config = `{
  "type": "esri-xyz",
  "config": {
    "server": "custom",
    "url": "[tile server url]",
    "attribution": "[tile server attribution]"
  }
}`
```

{{< /tab-content >}}
{{< /tabs >}}

- **osm-standard** loads the OpenStreetMap tile server. There are no additional configurations needed and the `config` fields can be left blank. Here is an example:

```ini
default_baselayer_config = `{
  "type": "osm-standard",
  "config": {}
}`
```

- **xyz** loads a custom tile server defined by the user. Set a valid tile server `url`, with {z}/{x}/{y} for this option in order to properly load a default base map. Here is an example:

```ini
default_baselayer_config = `{
  "type": "xyz",
  "config": {
    "attribution": "OpenStreetMap",
    "url": "https://tile.openstreetmap.org/{z}/{x}/{y}.png"
  }
}`
```

- **maplibre** loads a custom tile server defined by the user. Set a valid style `url` for this option to properly load a default base map. Here's an example:

```ini
default_baselayer_config = `{
  "type": "maplibre",
  "config": {
    "url": "https://demotiles.maplibre.org/style.json"
  }
}`
```

`enable_custom_baselayers` allows you to enable or disable custom open source base maps that are already implemented. The default is `true`.

### Map controls options

The map controls section contains various options for map information and tool overlays.

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| [Show zoom control](#show-zoom-control) | Displays zoom controls in the upper left corner. |
| [Mouse wheel zoom](#mouse-wheel-zoom) | Enables the mouse wheel to be used for zooming in or out. |
| [Show optional attribution](#show-optional-attribution) | Displays attribution for basemap layers whose license doesn't require it. |
| [Show scale](#show-scale) | Displays scale information in the bottom left corner in meters (m) or kilometers (km). |
| [Show measure tools](#show-measure-tools) | Displays measure tools in the upper right corner. This includes the [Length](#length) and [Area](#area) options. |
| [Show debug](#show-debug) | Displays debug information in the upper right corner. |
| [Tooltip](#tooltip) | Controls display of tooltips. |
<!-- prettier-ignore-end -->

#### Show zoom control

Displays zoom controls in the upper left corner. This control can be useful when using systems that don't have a mouse.

{{< figure src="/static/img/docs/geomap-panel/geomap-map-controls-zoom-9-1-0.png" max-width="300px" alt="Geomap panel zoom" >}}

#### Mouse wheel zoom

Enables the mouse wheel to be used for zooming in or out.

#### Show optional attribution

Displays attribution for basemap layers whose license doesn't require it. When a layer license requires attribution, Grafana always displays it and this option has no effect.

{{< figure src="/static/img/docs/geomap-panel/geomap-map-controls-attribution-9-1-0.png" max-width="400px" alt="Geomap panel attribution" >}}

#### Show scale

Displays scale information in the bottom left corner in meters (m) or kilometers (km).

{{< figure src="/static/img/docs/geomap-panel/geomap-map-controls-scale-9-1-0.png" max-width="400px" alt="Geomap panel scale" >}}

#### Show measure tools

Displays measure tools in the upper right corner. Measurements appear only when this control is open.

{{< figure src="/static/img/docs/geomap-panel/geomap-map-controls-measure-9-1-0.png" max-width="400px" alt="Geomap panel measure" >}}

- **Click** to start measuring
- **Continue clicking** to continue measurement
- **Double-click** to end measurement

When you change measurement type or units, the previous measurement is removed from the map. If the control is closed and then re-opened, the most recent measurement is displayed. A measurement can be modified by clicking and dragging on it.

##### Length

Get the spherical length of a geometry. This length is the sum of the great circle distances between coordinates. For multi-part geometries, the length is the sum of the length of each part. Geometries are assumed to be in 'EPSG:3857'.

- **Metric (m/km)**
- **Feet (ft)**
- **Miles (mi)**
- **Nautical miles (nmi)**

{{< figure src="/static/img/docs/geomap-panel/geomap-map-controls-measure-length-9-1-0.png" max-width="400px" alt="Geomap panel measure length" >}}

##### Area

Get the spherical area of a geometry. This area is calculated assuming that polygon edges are segments of great circles on a sphere. Geometries are assumed to be in 'EPSG:3857'.

- **Square Meters (m²)**
- **Square Kilometers (km²)**
- **Square Feet (ft²)**
- **Square Miles (mi²)**
- **Acres (acre)**
- **Hectare (ha)**

{{< figure src="/static/img/docs/geomap-panel/geomap-map-controls-measure-area-9-1-0.png" max-width="550px" alt="Geomap panel measure area" >}}

#### Show debug

Displays debug information in the upper right corner. This can be useful for debugging or validating a data source.

- **Zoom** displays current zoom level of the map.
- **Center** displays the current **longitude**, **latitude** of the map center.

{{< figure src="/static/img/docs/geomap-panel/geomap-map-controls-debug-9-1-0.png" max-width="400px" alt="Geomap panel debug" >}}

#### Tooltip

Tooltips are supported for the **Markers**, **Heatmap**, **Photos** layers.
For these layer types, choose from the following tooltip options:

- **None** displays tooltips only when a data point is clicked.
- **Details** displays tooltips when a mouse pointer hovers over a data point.

When a data point on the geomap represents one row&mdash;that is, only a single row of response data is relevant to that point&mdash;the tooltip displays a grid with the row's names and values:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-single-row-marker-v12.1.png" max-width="750px" alt="A data point with one row of associated data" >}}

When a data point represents more than one row&mdash;that is, different rows but with the same geographical information&mdash;then each row appears as a single entry:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-multiple-row-marker-v12.1.png" max-width="750px" alt="A data point with multiple rows of associated data" >}}

The text displayed in each tooltip row is associated with the first field value in each data row.
Click it to expand and display the full details of the data row.

{{< admonition type="note" >}}
The data appearing in each detail row is determined by the underlying query and transformations applied to the query's results, and can't be directly controlled using tooltip options.
{{< /admonition >}}

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Heatmap

Heatmaps allow you to view [histograms](ref:histograms) over time. While histograms display the data distribution that falls in a specific value range, heatmaps allow you to identify patterns in the histogram data distribution over time. For more information about heatmaps, refer to [Introduction to histograms and heatmaps](ref:intro-histograms-heatmaps).

For example, if you want to understand the temperature changes for the past few years, you can use a heatmap visualization to identify trends in your data:

{{< figure src="/static/img/docs/heatmap-panel/temperature_heatmap.png" max-width="1025px" alt="A heatmap visualization showing the random walk distribution over time" >}}

{{< docs/play title="Grafana Heatmaps" url="https://play.grafana.org/d/heatmap-calculate-log/" >}}

You can use a heatmap visualization if you need to:

- Visualize a large density of your data distribution.
- Condense large amounts of data through various color schemes that are easier to interpret.
- Identify any outliers in your data distribution.
- Provide statistical analysis to see how values or trends change over time.

## Configure a heatmap visualization

Once you’ve created a [dashboard](ref:dashboards), the following video shows you how to configure a heatmap visualization:

{{< youtube id="SGWBzQ54koE" >}}

## Supported data formats

Heatmaps support time series data.

### Example

The table below is a simplified output of random walk distribution over time:

| Time                | Walking (km) |
| ------------------- | ------------ |
| 2023-06-25 21:13:09 | 10           |
| 2023-08-25 21:13:10 | 8            |
| 2023-08-30 21:13:10 | 10           |
| 2023-10-08 21:13:11 | 12           |
| 2023-12-25 21:13:11 | 14           |
| 2024-01-05 21:13:12 | 13           |
| 2024-02-22 21:13:13 | 10           |

The data is converted as follows:

{{< figure src="/static/img/docs/heatmap-panel/heatmap.png" max-width="1025px" alt="A heatmap visualization showing the random walk distribution over time" >}}

## Pan and zoom panel time range

{{< docs/shared lookup="visualizations/panel-pan-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Heatmap options

The following options control how data in the heatmap is calculated and grouped.

<!-- prettier-ignore-start -->

| Options | Description |
| ------- | ----------- |
| Calculate from data | This setting determines if the data is already a calculated heatmap (from the data source/transformer), or one that should be calculated in the panel. |
| X Bucket | This setting determines how the x-axis is split into buckets. You can specify a time interval in the **Size** input. For example, a time range of `1h` makes the cells 1-hour wide on the x-axis. You can also set an interval based on **Count**.  |
| Y Bucket | This setting determines how the y-axis is split into buckets. Choose from **Size** or **Count**. |
| Y Bucket scale | When **Calculate from data** is **Yes**. Select one of the following y-axis value scales:<ul><li>**Linear** - Linear scale.</li><li>**Logarithmic** - Choose a **Log base** of **2** or **10**.</li><li>**Symlog** - Symlog scale. Choose a **Log base** of **2** or **10** and enter a value for the **Linear threshold**.</li></ul> |
| Y Bucket scale | When **Calculate from data** is **No**. Set the y-axis scale for pre-bucketed data in the wide, one-field-per-bucket data structure (not available when the response uses the heatmap-cells data frame type). Choose from:<ul><li>**Auto** - Uses the default scale behavior.</li><li>**Linear** - Linear scale.</li><li>**Log** - Logarithmic scale. Choose a **Log base** of **2** or **10**.</li><li>**Symlog** - Symmetrical logarithmic scale. Choose a **Log base** of **2** or **10** and enter a value for the **Linear threshold**.</li></ul> |

<!-- prettier-ignore-end -->

### Y-Axis options

The following options define the display of the y-axis.

<!-- prettier-ignore-start -->

| Options | Description |
| ------- | ----------- |
| Placement | Set where the y-axis is displayed. Choose from: **Left**, **Right**, or **Hidden**. |
| Unit | Unit configuration. |
| Decimals | This setting determines decimal configuration. |
| Min/Max value | These settings configure the axis range. |
| Axis width | This setting configures the width for the axis. |
| Axis label | This setting configures the axis value. |
| Tick alignment | Sets the alignment of the tick marks on the visualization. Choose from: **Auto**, **Top (LE)**, **Middle**, and **Bottom (GE)**. This option is only displayed when your **Calculate from data** setting is **No** and **Y Bucket scale** is set to **Auto**. |
| Reverse| When selected, the axis appears in reverse order. |

<!-- prettier-ignore-end -->

{{< docs/shared lookup="visualizations/multiple-y-axes.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+3" >}}

### Colors options

The color spectrum controls the mapping between value count (in each bucket) and the color assigned to each bucket. The leftmost color on the spectrum represents the minimum count and the color on the right most side represents the maximum count. Some color schemes are automatically inverted when using the light theme.

You can also change the color mode to Opacity. In this case, the color will not change but the amount of opacity will change with the bucket count

#### Mode

Use the following options to define the heatmap colors.

- **Scheme** - Bucket value represented by cell color.
  - **Scheme** - If the mode is **Scheme**, then select a color scheme.
- **Opacity** - Bucket value represented by cell opacity. Opaque cell means maximum value.
  - **Color** - Cell base color.
  - **Scale** - Scale for mapping bucket values to the opacity.
    - **Exponential** - Power scale. Cell opacity calculated as `value ^ k`, where `k` is a configured **Exponent** value. If exponent is less than `1`, you will get a logarithmic scale. If exponent is greater than `1`, you will get an exponential scale. In case of `1`, scale will be the same as linear.
      - **Exponent** - Value of the exponent, greater than `0`.
    - **Linear** - Linear scale. Bucket value maps linearly to the opacity.

#### Steps

Set a value between `1` and `128`.

#### Reverse

Toggle the switch to reverse the color scheme. This option only applies the **Scheme** color mode.

#### Start/end color scale from value

By default, Grafana calculates cell colors based on minimum and maximum bucket values. With **Start color scale from value** and **End color scale at value**, you can overwrite those values. Consider a bucket value as a z-axis, with the start and end values as z-min and z-max.

- **Start color scale from value** - Minimum value used for cell color calculation. The placeholder **Auto (min)** uses the series minimum value. If the bucket value is less than this value, then it's mapped to the minimum color.
- **End color scale at value** - Maximum value used for cell color calculation. The placeholder **Auto (max)** uses the series maximum value. If the bucket value is greater than this value, then it's mapped to the maximum color.

### Cell display options

Use these settings to control the display of heatmap cells.

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Value name | Sets the label for the cell value column when **Calculate from data** is **No**. |
| Unit | Unit configuration. |
| Decimals | This setting determines decimal configuration. |
| Cell gap | Set how much space there is between cells. |
| Hide cells with values <= | Enter a value. |
| Hide cells with values >= | Enter a value. |

<!-- prettier-ignore-end -->

### Tooltip options

Tooltip options control the information overlay that appears when you hover over data points in the visualization.

| Option                                | Description                                                                                                                                                                     |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Tooltip mode](#tooltip-mode)         | When you hover your cursor over the visualization, Grafana can display tooltips. Choose how tooltips behave.                                                                    |
| Show histogram (Y axis)               | When you set the **Tooltip mode** to **Single**, this option is displayed. This option controls whether or not the tooltip includes a histogram representing the y-axis.        |
| [Show color scale](#show-color-scale) | This option controls whether or not the tooltip includes the color scale that's also represented in the legend.                                                                 |
| Max width                             | Set the maximum width of the tooltip box.                                                                                                                                       |
| Max height                            | Set the maximum height of the tooltip box. This option is shown when **Tooltip mode** is **All**, or when exemplar data is present. If unset, the tooltip isn't scroll-limited. |

#### Tooltip mode

When you hover your cursor over the visualization, Grafana can display tooltips. Choose how tooltips behave.

- **Single -** The hover tooltip shows only a single series, the one that you are hovering over on the visualization.
- **All -** The hover tooltip shows all series in the visualization. Grafana highlights the series that you are hovering over in bold in the series list in the tooltip.
- **Hidden -** Do not display the tooltip when you interact with the visualization.

Use an override to hide individual series from the tooltip.

#### Show color scale

When you set the **Tooltip mode** to **Single**, this option is displayed. This option controls whether or not the tooltip includes the color scale that's also represented in the legend. When the color scale is included in the tooltip, it shows the hovered value on the scale:

![Heatmap with a tooltip displayed showing the hovered value reflected in the color scale](/media/docs/grafana/panels-visualizations/screenshot-heatmap-tooltip-color-scale-v11.0.png)

### Legend options

Legend options control how the color scale legend is displayed on the visualization.
For more information about the legend, refer to [Configure a legend](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-legend/).

| Option      | Description                                                                                                                                                                                                                                                                                                                                        |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Show legend | Toggle the switch to turn the legend on or off.                                                                                                                                                                                                                                                                                                    |
| Placement   | Choose where to display the legend. **Bottom** places the legend below the graph. **Right** places the legend to the right of the graph.                                                                                                                                                                                                           |
| Width       | Control how wide the legend is when placed on the right side of the visualization. The value may be a basic pixel width or a more complex [CSS width property](https://developer.mozilla.org/en-US/docs/Web/CSS/Reference/Properties/width) such as `max(200px, 20%)`. This option is only displayed if you set the legend placement to **Right**. |

### Annotation options

{{< docs/shared lookup="visualizations/annotations/options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Exemplars

Set the color used to show exemplar data.

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Histogram

Histograms calculate the distribution of values and present them as a bar chart. Each bar represents a bucket; the y-axis and the height of each bar represent the count of values that fall into each bucket, and the x-axis represents the value range.

For example, if you want to understand the distribution of people's heights, you can use a histogram visualization to identify patterns or insights in the data distribution:

{{< figure src="/static/img/docs/histogram-panel/histogram-example-v8-0.png" max-width="1025px" alt="A histogram visualization showing the distribution of people's heights" >}}

You can use a histogram visualization if you need to:

- Visualize and analyze data distributions over a specific time range to see how frequently certain values occur.
- Identify any outliers in your data distribution.
- Provide statistical analysis to help with decision-making

## Configure a histogram visualization

After you've created a [dashboard](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/create-dashboard/), the following video shows you how to configure a histogram visualization:

{{< youtube id="QfJ480j9-KM" >}}

{{< docs/play title="Histogram Examples" url="https://play.grafana.org/d/histogram_tests/" >}}

## Supported data formats

Histograms support time series and any table results with one or more numerical fields.
If a data source returns pre-calculated histogram buckets, Grafana renders the supplied frequencies instead of calculating buckets in the panel.

### Examples

The following tables are examples of the type of data you need for a histogram visualization and how it should be formatted.

#### Time-series table

| Time                | Walking (km) |
| ------------------- | ------------ |
| 2024-03-25 21:13:09 | 37.2         |
| 2024-03-25 21:13:10 | 37.1         |
| 2024-03-25 21:13:10 | 37.0         |
| 2024-03-25 21:13:11 | 37.2         |
| 2024-03-25 21:13:11 | 36.9         |
| 2024-03-25 21:13:12 | 36.7         |
| 2024-03-25 21:13:13 | 36.3         |

The data is converted as follows:

{{< figure src="/static/img/docs/histogram-panel/histogram-example-time-series.png" max-width="1025px" alt="A histogram visualization showing the random walk distribution." >}}

#### Basic numerical table

| Gender | Height (in) | Weight (lbs) |
| ------ | ----------- | ------------ |
| Male   | 73.8        | 242          |
| Male   | 68.8        | 162          |
| Male   | 74.1        | 213          |
| Male   | 71.7        | 220          |
| Male   | 69.9        | 206          |
| Male   | 67.3        | 152          |
| Male   | 68.8        | 184          |

The data is converted as follows:

{{< figure src="/static/img/docs/histogram-panel/histogram-example-height-weight.png" max-width="1025px" alt="A histogram visualization showing the male height and weight distribution" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Histogram options

Use the following options to refine your histogram visualization.
When the query data is already a histogram frame, the bucket options are read-only and Grafana shows frequencies calculated by the query.

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Bucket count | Specifies the number of bins used to group your data in the histogram, affecting the granularity of the displayed distribution. Leave this empty for automatic bucket count of 30. |
| Bucket size | The size of the buckets. Leave this empty for automatic bucket sizing (~10% of the full range). |
| [Bucket offset](#bucket-offset) | If the first bucket should not start at zero. A non-zero offset has the effect of shifting the aggregation window. |
| Combine series | This will merge all series and fields into a combined histogram. |
| Stacking | Controls how multiple series are displayed in the histogram. Choose from the following:<ul><li>**Off** - Series are not stacked, but instead shown side by side.</li><li>**Normal** - Series are stacked on top of each other, showing cumulative values.</li><li>**100%** - Series are stacked to fill 100% of the chart, showing the relative proportion of each series.</li></ul> |
| Line width | Controls line width of the bars. |
| Fill opacity | Controls the fill opacity bars. |
| [Gradient mode](#gradient-mode) | Set the mode of the gradient fill. Fill gradient is based on the line color. |

<!-- prettier-ignore-end -->

#### Bucket offset

If the first bucket should not start at zero, a non-zero offset has the effect of shifting the aggregation window.

For example, 5-sized buckets that are 0-5, 5-10, 10-15 with a default 0 offset would become 2-7, 7-12, 12-17 with an offset of 2; offsets of 0, 5, or 10, in this case, would effectively do nothing.

Typically, this option would be used with an explicitly defined bucket size rather than automatic. For this setting to affect, the offset amount should be greater than 0 and less than the bucket size; values outside this range have the same effect as values within this range.

#### Gradient mode

Set the mode of the gradient fill. Fill gradient is based on the line color. To change the color, use the standard [color scheme](ref:color-scheme) field option.

Gradient display is influenced by the **Fill opacity** setting.

Choose from the following:

- **None** - No gradient fill. This is the default setting.
- **Opacity** - Transparency of the gradient is calculated based on the values on the Y-axis. The opacity of the fill is increasing with the values on the Y-axis.
- **Hue** - Gradient color is generated based on the hue of the line color.
- **Scheme** - The selected [color palette](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-standard-options/#color-scheme) is applied to the histogram bars.

### Tooltip options

{{< docs/shared lookup="visualizations/tooltip-options-3.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Legend options

{{< docs/shared lookup="visualizations/legend-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Logs

_Logs_ are structured records of events or messages generated by a system or application&mdash;that is, a series of text records with status updates from your system or app. They generally include timestamps, messages, and context information like the severity of the logged event.

The logs visualization displays these records from data sources that support logs, such as Elastic, Influx, and Loki. The logs visualization shows, by default, the timestamp, a colored string representing the log status, the log line body, as well as collapsible log events that help you analyze the information generated.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-logs-v12.3.png" max-width="750px" alt="Logs visualization" >}}

When the log line details are open, you have access to the following additional options:

- Explain log line in Assistant
- Scroll to log line
- Show or hide log line
- Copy to clipboard
- Copy shortlink
- Switch to condensed view
- Anchor to the right

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-logs-options-v12.3.png" max-width="750px" alt="Log icons" >}}

You can access the first three options from the log line details or from the log line menu:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-log-line-menu-v12.3.png" max-width="300px" alt="The log line menu" >}}

{{< docs/play title="Logs Panel" url="https://play.grafana.org/d/6NmftOxZz/" >}}

Typically, you use logs with a graph visualization to display the log output of a related process. If you have an incident in your application or systems, such as a website disruption or code failure, you can use the logs visualization to help you figure out what went wrong, when, and even why.

## Configure a log visualization

The following video provides a walkthrough of creating a logs visualization. You'll also learn how to customize some settings and log visualization caveats:

{{< youtube id="jSSi_x-fD_8" >}}

## Supported data formats

The logs visualization works best with log-type datasets such as queries from data sources like Loki, Elastic, and InfluxDB.

The preferred dataplane log format uses a `timestamp` time field and a `body` string field. It can also include optional `severity`, `id`, and `labels` fields.

You can also build legacy log-formatted data from other data sources as long as the first field is a time type followed by string, number, and time fields. The leading time field is used to sort and timestamp the logs and if the data contains other time-type fields, they’re included as elements of the logged record.

The second field is used as the log record title regardless of whether it’s a time, numeric, or string field. Usually the second field is a text field containing multiple string elements, but if the message level (or `lvl`) is present, the visualization uses the values in it to add colors to the record, as described in [Log levels integration](ref:log-levels).

Subsequent fields are collapsed inside of each log record.

To limit the number of log lines rendered in the visualization, you can use the **Max data points** setting in the panel **Query options**. If that option isn't set, then the data source typically enforces its own default limit.

### Example

| Time                | TitleMessage         | Element1 | Element2 | Element3            |
| ------------------- | -------------------- | -------- | -------- | ------------------- |
| 2023-02-01 12:00:00 | title=Log1 lvl=info  | 1        | server2  | 2023-02-01 11:00:00 |
| 2023-02-01 11:30:00 | title=Log1 lvl=error | 1        | server2  | 2023-02-01 11:00:00 |
| 2023-02-01 11:00:00 | title=Log1 lvl=trace | 1        | server2  | 2023-02-01 11:00:00 |

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-logs-example2-v12.3.png" max-width="750px" alt="Logs example" >}}

## Log level

Grafana determines the log level from the first matching `level`, `detected_level`, `lvl`, or `loglevel` label and updates color accordingly. If the log doesn't have one of these labels, Grafana tries to find out if its content matches any of the supported expressions. If Grafana can't determine a log level, it visualizes the line with **unknown** log level. Refer to [supported log levels and mappings of log level abbreviation and expressions](ref:log-levels).

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Logs options

Use these settings to refine your visualization:

<!-- prettier-ignore-start -->

|      Option     |   Description   |
| --------------- | --------------- |
| Show timestamps | Show or hide the time column. This is the timestamp associated with the log line as reported from the data source. The default is off. |
| Timestamp resolution | Choose the timestamp precision: **Milliseconds** or **Nanoseconds**. |
| Display log level | Show or hide the detected log level. |
| Font size | Choose from **Default** or **Small**. |
| Wrap lines | Turn line wrapping on or off. |
| Prettify JSON | Toggle the switch on to pretty print all JSON logs. This setting does not affect logs in any format other than JSON. |
| Enable columns for displayed fields | Show selected displayed fields as columns. |
| Enable logs highlighting | Use a predefined syntax coloring grammar to highlight relevant parts of the log lines. |
| Enable log details | Toggle the switch on to see an extendable area with log details including labels and detected fields. Each field or label has a stats icon to display free-form statistics in relation to all displayed logs. The default setting is on. |
| Log Details panel mode | Choose to display the log details in a sidebar panel or inline, below the log line. |
| Enable infinite scrolling | Request more results by scrolling to the bottom of the logs list. |
| Show controls | Display controls to jump to the last or first log line and filter by log level. |
| Display download control | Display a control to download log lines when **Show controls** is enabled. |
| Font size | Select between the default font size and small font size. |
| Unique labels | Show or hide the unique labels column, which shows only non-common labels. |
| Deduplication | Hide log messages that are duplicates of others shown, according to your selected criteria. Choose from: <ul><li>**None** - Don't deduplicate log lines.</li><li>**Exact** - Ignoring ISO datetimes.</li><li>**Numbers** - Ignoring only those that differ by numbers such as IPs or latencies.</li><li>**Signature** - Removing successive lines with identical punctuation and white space.</li></ul> |
| Order | Set whether to show results **Newest first** or **Oldest first**. |

<!-- prettier-ignore-end -->

# News

The news visualization displays an RSS feed. By default, it displays articles from the Grafana Labs blog, but you can change this by entering a different RSS feed URL.

{{< figure src="/static/img/docs/news/news-visualization.png" max-width="1025px" alt="A news visualization showing the latest Grafana news feed" >}}

{{< admonition type="note" >}}
In version 8.5, we discontinued the "Use Proxy" option for Grafana news visualizations. As a result, RSS feeds that are not configured for request by Grafana's frontend (with the appropriate [CORS headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS)) may not load.
{{< /admonition >}}

You can use the news visualization to provide regular news and updates to your users.

{{< docs/play title="News Panel" url="https://play.grafana.org/d/cdodkwspaaa68b/" >}}

## Refresh the news feed

The news visualization doesn't have its own refresh interval.
It requests the feed when the dashboard first loads and each time the dashboard refreshes.

To update the articles, do one of the following:

- **Refresh the dashboard manually:** Click the **Refresh dashboard** icon.
- **Set an auto-refresh interval:** Click the drop-down next to the **Refresh dashboard** icon and select an interval. For more information on setting these intervals, refer to [Refresh dashboard](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/use-dashboards/#refresh-dashboard).

The refresh interval applies to the whole dashboard, so pick one that suits your other visualizations as well as your feed.

## Configure a news visualization

After you've created a [dashboard](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/), enter the URL of an RSS in the **URL** field in the **News** section. This visualization type doesn't accept any other queries, and you shouldn't expect to be able to filter or query the RSS feed data in any way using this visualization.

If you're having trouble loading an RSS feed, you can try re-hosting the feed on a different server or using a CORS proxy. A CORS proxy is a tool that allows you to bypass CORS restrictions by making requests to the RSS feed on your behalf. You can find more information about using CORS proxies online.

If you're unable to display an RSS feed using the news visualization, you can try using the community RSS/Atom data source plugin [RSS/Atom data source](https://grafana.com/grafana/plugins/volkovlabs-rss-datasource/) in combination with the Dynamic text community panel [Dynamic text](https://grafana.com/grafana/plugins/marcusolsson-dynamictext-panel/). This allows you to display the RSS feed in a different way.

## Supported data formats

The news visualization supports RSS and Atom feeds.

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### News options

Use the following options to refine your news visualization:

- **URL** - The URL of the RSS or Atom feed. If you leave the field empty, the Grafana Labs blog is used.
- **Show image** - Controls if the news social image displays beside the text content.

# Node graph

Node graphs are useful when you need to visualize elements that are related to each other. This is done by displaying circles&mdash;or _nodes_&mdash;for each element you want to visualize, connected by lines&mdash;or _edges_. By default, the visualization uses a [layered layout](#layout-algorithm) that positions the nodes into a network of connected circles.

Node graphs display useful information about each node, as well as the relationships between them, allowing you to visualize complex infrastructure maps, hierarchies, or execution diagrams.

![Node graph visualization](/media/docs/grafana/panels-visualizations/screenshot-node-graph-v11.3.png 'Node graph')

The appearance of nodes and edges can also be customized in several ways including color, borders, and line style.

You can use a node graph visualization if you need to show:

- Solution topologies
- Networks
- Infrastructure
- Organizational charts
- Critical path diagrams
- Family trees
- Mind maps

## Configure a node graph visualization

The following video provides beginner steps for creating node panel visualizations. You'll learn the data requirements and caveats, special customizations, and much more:

{{< youtube id="VrvsMkRwoKw" >}}

{{< docs/play title="Node graph panel" url="https://play.grafana.org/d/bdodfbi3d57uoe/" >}}

## Supported data formats

To create node graphs, you need two datasets: one containing the records for the displayed elements (nodes) and one dataset containing the records for the connections between those elements (edges).

### Nodes dataset

The nodes dataset must contain one alphanumeric ID field that gives each element a unique identifier. The visualization also accepts other options fields for titles, subtitles, main and secondary stats, arc information for how much of the circle border to paint, details, colors, icons, node size, and indicators for element highlighting. For more information and naming conventions for these fields, refer to the [Nodes data frame structure](#nodes-data-frame-structure) section.

#### Example

| id    | title | subtitle | mainstat | secondarystat | color | icon | highlighted |
| ----- | ----- | -------- | -------- | ------------- | ----- | ---- | ----------- |
| node1 | PC    | Windows  | AMD      | 16gbRAM       | blue  |      | true        |
| node2 | PC    | Linux    | Intel    | 32gbRAM       | green | eye  | false       |
| node3 | Mac   | MacOS    | M3       | 16gbRAM       | gray  | apps | false       |
| node4 | Alone | SoLonely | JustHere | NotConnected  | red   |      | false       |

If the icon field contains a value, it’s displayed instead of the title and subtitle. For a list of of available icons, refer to [Icons Overview](https://developers.grafana.com/ui/latest/index.html?path=/story/iconography-icon--icons-overview).

### Edges dataset

Similar to the nodes dataset, the edges dataset needs one unique ID field for each relationship, followed by two fields containing the source and the target nodes of the edge; that is, the nodes the edge connects. Other optional fields are main and secondary stats, context menu elements, line thickness, highlight indications, line colors, and configurations to turn the connection into a dashed line. For more information and naming conventions for these fields, refer to the [Edges data frame structure](#edges-data-frame-structure) section.

#### Example

| id    | source | target | mainstat | seconddarystat | thickness | highlighted | color  |
| ----- | ------ | ------ | -------- | -------------- | --------- | ----------- | ------ |
| edge1 | node1  | node2  | TheMain  | TheSub         | 3         | true        | cyan   |
| edge2 | node3  | node2  | Main2    | Sub2           | 1         | false       | orange |

If a node lacks edge connections, it’s displayed on its own outside of the network.

### Data requirements

A node graph requires a specific shape of the data to be able to display its nodes and edges. This means not every data source or query can be visualized with this graph. If you want to use this as a data source developer see the section about data API.

A node graph consists of _nodes_ and _edges_.

- A _node_ is displayed as a circle. A node might represent an application, a service, or anything else that is relevant from an application perspective.
- An _edge_ is displayed as a line that connects two nodes. The connection might be a request, an execution, or some other relationship between the two nodes.

Both nodes and edges can have associated metadata or statistics. The data source defines what information and values is shown, so different data sources can show different type of values or not show some values.

#### Nodes

{{< admonition type="note" >}}
Node graphs show up to 200 visible nodes by default. If the graph has more nodes, Grafana shows a warning at the top of the panel and hides some nodes for performance reasons. You can expand hidden parts of the graph by clicking the "Hidden nodes" markers in the graph.
{{< /admonition >}}

Usually, nodes show two statistical values inside the node and two identifiers just below the node, usually name and type. Nodes can also show another set of values as a color circle around the node, with sections of different color represents different values that should add up to 1.

For example, you can have the percentage of errors represented by a red portion of the circle.
Additional details can be displayed in a context menu which is displayed when you click on the node.
There also can be additional links in the context menu that can target either other parts of Grafana or any external link.

![Node context menu](/media/docs/grafana/panels-visualizations/screenshot-node-links-v11.3.png 'Node context menu')

#### Edges

Edges can also show statistics when you hover over the edge. Similar to nodes, you can open a context menu with additional details and links by clicking on the edge.

The first data source supporting this visualization is X-Ray data source for its Service map feature. For more information, refer to the [X-Ray plugin documentation](https://grafana.com/grafana/plugins/grafana-x-ray-datasource).

## Node graph navigation

You can use pan, zoom, and other functions to navigate a node graph.

### Pan

You can pan the view by clicking outside any node or edge and dragging your mouse.

### Zoom

Use the buttons in the lower right corner to zoom in or out. You can also use the mouse wheel or touchpad scroll, together with either Ctrl or Cmd key to do so.

### Switch layouts

Switch quickly between displaying the visualization in graph or grid [layout](#layout-algorithm).

Click a node and select either **Show in Grid layout** or **Show in Graph layout**, depending on the current layout of the visualization:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-node-graph-grid-menu.png" max-width="750px" alt="Node graph in grid layout with node menu open" >}}

In grid layout, you can sort nodes by clicking on the stats inside the legend.
The marker next to the stat name shows which stat is currently used for sorting and the sorting direction:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-node-graph-legend-sort.png" max-width="550px" alt="Node graph legend sorting" >}}

Switching between grid and other layouts this way only changes the layout temporarily.
The visualization maintains the layout algorithm selected in the panel editor, and reverts to it when the dashboard refreshes.

For more information about layouts, refer to [Layout algorithm](#layout-algorithm).

<!-- if you have the panel in grid layout and switch it to graph, is it switching to layered? -->

### Hidden nodes

The number of nodes shown at a given time is limited to maintain a reasonable visualization performance. Nodes that are not currently visible are hidden behind clickable markers that show an approximate number of hidden nodes that are connected by a particular edge. You can click on the marker to expand the graph around that node.

![Node graph exploration](/media/docs/grafana/panels-visualizations/node-graph-exploration-8.0-2.png 'Node graph exploration')

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Node graph options

Use the following options to refine your node graph visualization.

#### Zoom mode

Choose how the node graph should handle zoom and scroll events:

- **Cooperative** - Allows you to scroll the visualization normally.
- **Greedy** - Reacts to all zoom gestures.

#### Layout algorithm

Choose how the visualization layout is generated:

- **Layered** - Default. Creates a predictable and orderly layout, especially useful for service graphs.
- **Force** - Uses a physics-based force layout algorithm that's useful with a large number of nodes (500+).
- **Grid** - Arranges nodes into a grid format to provide a better overview of the most interesting nodes in the graph. This layout shows nodes in a grid without edges and can be sorted by the stats shown inside the node or by the ones represented by the a colored border of the nodes.

  {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-node-graph-grid.png" max-width="650px" alt="Node graph in grid layout" >}}

  For more information about using the graph in grid layout, refer to [Switch layouts](#switch-layouts).

{{< admonition type="note" >}}
Layered layout can be slow when the graph has more than 500 nodes. For large graphs, use **Force** layout.
{{< /admonition >}}

### Nodes options

The **Nodes** options section provides configurations for node behaviors.

- **Main stat unit** - Choose which unit the main stat displays in the graph's nodes.
- **Secondary stat unit** - Choose which unit the secondary stat displays in the graph's nodes.
- **Arc sections** - Configure which fields define the size of the colored circle around the node and select a color for each. You can add multiple fields.

{{< admonition type="note" >}}
Defining arc sections overrides the automatic detection of `arc__*` and `color` fields described in the **Optional fields** section of [Nodes data frame structure](#nodes-data-frame-structure).
{{< /admonition >}}

### Edges options

The **Edges** options section provides configurations for node edges behaviors.

- **Main stat unit** - Choose which unit the main stat displays in the graph's edges.
- **Secondary stat unit** - Choose which unit the secondary stat displays in the graph's edges.

### Data links

{{< docs/shared lookup="visualizations/datalink-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

In node graphs, some data fields may have pre-configured data links. To add a different data link in those cases, use a [field override](#field-overrides).

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Data API

This visualization needs a specific shape of the data to be returned from the data source to correctly display it.

Node graphs, at minimum, require a data frame describing the edges of the graph. By default, node graphs compute the nodes and any stats based on this data frame. Optionally a second data frame describing the nodes can be sent when you need to show more node-specific metadata. Grafana recognizes node graph frames when `frame.meta.preferredVisualisationType = 'nodeGraph'`, when the frame name or `refId` is `nodes` or `edges`, or when a frame contains an `id` field.

### Edges data frame structure

Required fields:

| Field name | Type   | Description                    |
| ---------- | ------ | ------------------------------ |
| id         | string | Unique identifier of the edge. |
| source     | string | Id of the source node.         |
| target     | string | Id of the target.              |

Optional fields:

| Field name      | Type          | Description                                                                                                                                                                                                                                                               |
| --------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| mainstat        | string/number | First stat shown in the overlay when hovering over the edge. It can be a string showing the value as is or it can be a number. If it is a number, any unit associated with that field is also shown                                                                       |
| secondarystat   | string/number | Same as mainStat, but shown right under it.                                                                                                                                                                                                                               |
| detail\_\_\*    | string/number | Any field prefixed with `detail__` will be shown in the header of context menu when clicked on the edge. Use `config.displayName` for more human readable label.                                                                                                          |
| thickness       | number        | The thickness of the edge. Default: `1`                                                                                                                                                                                                                                   |
| highlighted     | boolean       | Sets whether the edge should be highlighted. Useful, for example, to represent a specific path in the graph by highlighting several nodes and edges. Default: `false`                                                                                                     |
| color           | string        | Sets the default color of the edge. It can be an acceptable HTML color string. Default: `#999`                                                                                                                                                                            |
| strokeDasharray | string        | Sets the pattern of dashes and gaps used to render the edge. If unset, a solid line is used as edge. For more information and examples, refer to the [`stroke-dasharray` MDN documentation](https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/stroke-dasharray). |

{{< admonition type="caution" >}}
Starting with 10.5, `highlighted` is deprecated.
It will be removed in a future release.
Use `color` to indicate a highlighted edge state instead.
{{< /admonition >}}

### Nodes data frame structure

Required fields:

| Field name | Type   | Description                                                                                  |
| ---------- | ------ | -------------------------------------------------------------------------------------------- |
| id         | string | Unique identifier of the node. This ID is referenced by edge in its source and target field. |

Optional fields:

| Field name    | Type          | Description                                                                                                                                                                                                                                                                                                                                                              |
| ------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| title         | string        | Name of the node visible in just under the node.                                                                                                                                                                                                                                                                                                                         |
| subtitle      | string        | Additional, name, type or other identifier shown under the title.                                                                                                                                                                                                                                                                                                        |
| mainstat      | string/number | First stat shown inside the node itself. It can either be a string showing the value as is or a number. If it is a number, any unit associated with that field is also shown.                                                                                                                                                                                            |
| secondarystat | string/number | Same as mainStat, but shown under it inside the node.                                                                                                                                                                                                                                                                                                                    |
| arc\_\_\*     | number        | Any field prefixed with `arc__` will be used to create the color circle around the node. All values in these fields should add up to 1. You can specify color using `config.color.fixedColor`.                                                                                                                                                                           |
| detail\_\_\*  | string/number | Any field prefixed with `detail__` will be shown in the header of context menu when clicked on the node. Use `config.displayName` for more human readable label.                                                                                                                                                                                                         |
| color         | string/number | Can be used to specify a single color instead of using the `arc__` fields to specify color sections. It can be either a string which should then be an acceptable HTML color string or it can be a number in which case the behavior depends on `field.config.color.mode` setting. This can be for example used to create gradient colors controlled by the field value. |
| icon          | string        | Name of the icon to show inside the node instead of the default stats. Only Grafana [built in icons](https://developers.grafana.com/ui/latest/index.html?path=/story/iconography-icon--icons-overview) are allowed.                                                                                                                                                      |
| nodeRadius    | number        | Radius value in pixels. Used to manage node size.                                                                                                                                                                                                                                                                                                                        |
| highlighted   | boolean       | Sets whether the node should be highlighted. Useful for example to represent a specific path in the graph by highlighting several nodes and edges. Default: `false`                                                                                                                                                                                                      |
| fixedX        | number        | Fixed x-coordinate for the node. If you use `fixedX`, all nodes must provide a value.                                                                                                                                                                                                                                                                                    |
| fixedY        | number        | Fixed y-coordinate for the node. If you use `fixedY`, all nodes must provide a value.                                                                                                                                                                                                                                                                                    |

<!-- `isInstrumented` is not included here as it's only used internally at this time -->

# Pie chart

A pie chart is a graph that displays data as segments of a circle proportional to the whole, making it look like a sliced pie. Each slice corresponds to a value or measurement.

![Pie chart visualizations](/media/docs/grafana/panels-visualizations/screenshot-pie-chart-v11.4.png)

The pie chart visualization is ideal when you have data that adds up to a total and you want to show the proportion of each value compared to other slices, as well as to the whole of the pie.

You can use a pie chart if you need to compare:

- Browser share distribution in the market
- Incident causes per category
- Network traffic sources
- User demographics

## Configure a pie chart visualization

The following video guides you through the creation steps and common customizations of pie chart visualizations and is great for beginners:

{{< youtube id="A_lDhM9w4_g" >}}

{{< docs/play title="Grafana Bar Charts and Pie Charts" url="https://play.grafana.org/d/ktMs4D6Mk/" >}}

## Supported data formats

The pie chart is different from other visualizations in that it will only display one pie, regardless of the number of datasets, fields, or records queried in it.

To create a pie chart visualization, you need a dataset containing a set of numeric values either in rows, columns, or both.

### Example - One row

The easiest way to provide data for a pie chart visualization is in a dataset with a single record (or row) containing the fields (or columns) that you want in the pie, as in the following example. The default settings of the pie chart visualization automatically display each column as a slice of the pie.

| Value1 | Value2 | Value3 | Optional |
| ------ | ------ | ------ | -------- |
| 5      | 3      | 2      | Sums10   |

![Pie chart visualization with multiple values in a single row](/media/docs/grafana/panels-visualizations/screenshot-grafana-12.1-pie-example1.png)

### Example - Multiple rows

If you need to use numeric data that's in multiple rows, the default **Show** parameter of the visualization [Value options](#value-options) is set to **Calculate** and use data from the last row.

| Value | Label  |
| ----- | ------ |
| 5     | Value1 |
| 3     | Value2 |
| 2     | Value3 |

![Pie chart visualization with multiple row values showing the last one](/media/docs/grafana/panels-visualizations/screenshot-grafana-12.1-pie-example2.png)

By default, the visualization is configured to [calculate](#value-options) a single value per column or series and to display only the last row of data.

To allow values in multiple rows to be displayed, change the **Show** setting in the [Value options](#value-options) from **Calculate** to **All values**.

![Pie chart visualization with multiple row values showing all values](/media/docs/grafana/panels-visualizations/screenshot-grafana-12.1-pie-example3.png)

### Example - Multiple rows and columns

If your dataset contains multiple rows and columns with numeric data, by default only the last row's values are summed.

| Value1 | Value2 | Value3 | Optional |
| ------ | ------ | ------ | -------- |
| 5      | 3      | 2      | Sums10   |
| 10     | 6      | 4      | Sums20   |
| 20     | 8      | 2      | Sums30   |

![Pie chart visualization with multiple rows and columns showing the last one](/media/docs/grafana/panels-visualizations/screenshot-grafana-12.1-pie-example4.png)

If you want to display all the cells, change the **Show** setting in the [Value options](#value-options) from **Calculate** to **All values**. This also labels the elements by concatenating all the text fields (if you have any) with the column name.

![Pie chart visualization with multiple rows and columns showing the all values](/media/docs/grafana/panels-visualizations/screenshot-grafana-12.1-pie-example5.png)

If you want to display only the values from a given field (or column), once the **Show** setting in the [Value options](#value-options) is set to **All values**, set the **Fields** option to the column you wish to sum in the display. The value labels are also concatenated as indicated before.

![Pie chart visualization with multiple rows and columns showing values from one column](/media/docs/grafana/panels-visualizations/screenshot-grafana-12.1-pie-example6.png)

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value options

Use the following options to refine the value in your visualization.

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Show | Set how much information to show. Choose from:<ul><li>**Calculate** - Reduces each value to a single value per series.</li><li>**All values** - Displays every value from a single series.</li></ul> |
| Calculation | If you chose **Calculate** as your **Show** option, select a calculation to reduce each series. For information about available calculations, refer to [Calculation types](ref:calculation-types). |
| Limit | If you chose **All values** as your **Show** option, enter a value to limit the number of values displayed. |
| Fields | Select which field or fields to display in the visualization. Each field name is available on the list, or you can select one of the following options:<ul><li>**Numeric Fields** - All fields with numerical values.</li><li>**All Fields** - All fields that are not removed by transformations.</li></ul> |

<!-- prettier-ignore-end -->

### Pie chart options

Use these options to refine how your visualization looks.

#### Pie chart type

Select the pie chart display style. Choose from **Pie** or **Donut**.

![Pie chart types](/media/docs/grafana/panels-visualizations/screenshot-pie-chart-types.png)

#### Slice sorting

By default, the pie chart is sorted so that the slices decrease in size clockwise around the circle.
You can configure the sorting of the slices, and by extension the legend, with the following options:

- **Descending** - The slices decrease in size, clockwise (default).
- **Ascending** - The slices increase in size, clockwise.
- **None** - No sorting is applied. The original order of the data is maintained.

#### Labels

Select labels to display on the pie chart. You can select more than one.

- **Name** - The series or field name.
- **Percent** - The percentage of the whole.
- **Value** - The raw numerical value.

Labels are displayed in white over the body of the chart. You might need to select darker chart colors to make them more visible. Long names or numbers might be clipped.

The following example shows a pie chart with **Name** and **Percent** labels displayed:

{{< figure src="/static/img/docs/pie-chart-panel/pie-chart-labels-7-5.png" alt="Pie chart labels" max-width="350px" >}}

### Tooltip options

{{< docs/shared lookup="visualizations/tooltip-options-1.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Legend options

Use these settings to define how the legend appears in your visualization. For more information about the legend, refer to [Configure a legend](ref:configure-legends).

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Visibility | Toggle the switch to turn the legend on or off. |
| Mode | Use these settings to define how the legend appears in your visualization. Choose from:<ul><li>**List** - Displays the legend as a list. This is a default display mode of the legend.</li><li>**Table** - Displays the legend as a table.</li></ul> |
| Placement | Select where to display the legend. Choose **Bottom** or **Right**. |
| Width | Control how wide the legend is when placed on the right side of the visualization. This option is only displayed if you set the legend placement to **Right**. |
| Legend values | Select values to display in the legend. You can select more than one:<ul><li>**Percent** - The percentage of the whole.</li><li>**Value** - The raw numerical value.</li></ul> |

<!-- prettier-ignore-end -->

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Stat

{{< admonition type="note" >}}
This visualization replaces the Singlestat visualization, which was deprecated in Grafana 7.0 and removed in Grafana 8.0.
{{< /admonition >}}

A stat visualization displays your data in single values of interest&mdash;such as the latest or current value of a series&mdash;with an optional graph sparkline. A graph sparkline, which is only available in stat visualizations, is a small time-series graph shown in the background of each value in the visualization.

For example, if you're monitoring the utilization of various services, you can use a stat visualization to show their latest usage:

![A stat panel showing latest usage of various services](/media/docs/grafana/panels-visualizations/screenshot-stat-visualization-v11.3.png)

Use a stat visualization when you need to:

- Monitor key metrics at a glance, such as the latest health of your application, number of high priority bugs in your application, or total number of sales.
- Display aggregated data, such as the average response time of your services.
- Highlight values above your normal thresholds to quickly identify if any metrics are outside your expected range.

{{< docs/play title="Stat Visualizations in Grafana" url="https://play.grafana.org/d/Zb3f4veGk/" >}}

## Configure a stat visualization

Once you've [created a dashboard](ref:create-dashboard), the following video shows you how to configure a stat visualization:

{{< youtube id="yNRnLyVntUw" start="1048" >}}

Alternatively, refer to this blog post on [how to easily retrieve values from a range in Grafana using a stat visualization](https://grafana.com/blog/2023/10/18/how-to-easily-retrieve-values-from-a-range-in-grafana-using-a-stat-panel/).

## Supported data formats

The stat visualization supports a variety of formats for displaying data. Supported formats include:

- **Single values** - The most common format and can be numerical, strings, or boolean values.
- **Time-series data** - [Calculation types](ref:calculation-types) can be applied to your time-series data to display single values over a specified time range.

### Examples

The following tables are examples of the type of data you need for a stat visualization and how it should be formatted.

#### Single numerical values

| Number of high priority bugs |
| ---------------------------- |
| 80                           |
| 52                           |
| 59                           |
| 40                           |

The data is visualized as follows, with the last value displayed, along with a sparkline and [percentage change](#value-options):

![A stat panel showing the latest number of high priority bugs](/media/docs/grafana/panels-visualizations/screenshot-stat-single-value-v11.3.png)

#### Time-series data

| Time                | Cellar | Living room | Porch | Bedroom | Guest room | Kitchen |
| ------------------- | ------ | ----------- | ----- | ------- | ---------- | ------- |
| 2024-03-20 06:34:40 | 12.3   | 18.3        | 18.8  | 15.9    | 9.29       | 9.61    |
| 2024-03-20 06:41:40 | 16.8   | 17.1        | 21.5  | 14.1    | 10.5       | 17.5    |
| 2024-03-20 06:48:40 | 16.7   | 18.0        | 21.0  | 9.51    | 13.6       | 20.1    |
| 2024-03-20 06:55:40 | 14.3   | 18.7        | 16.5  | 9.11    | 14.8       | 12.5    |
| 2024-03-20 07:02:40 | 12.8   | 15.2        | 21.1  | 15.6    | 7.98       | 13.0    |

The data is visualized as follows, with the mean value displayed for each room, along with the room name, sparkline, and unit of measurement:

![A stat panel showing some statistics for each room in square meters](/media/docs/grafana/panels-visualizations/screenshot-stat-multiple-values-v11.3.png)

By default, a stat displays one of the following:

- Just the value for a single series or field.
- Both the value and name for multiple series or fields.

You can use the [**Text mode**](#text-mode) to control how the text is displayed.

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value options

Use the following options to refine how your visualization displays its values:

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Show | Display a single value per column or series, or show values for each row. Choose from: <ul><li>**Calculate** - Display a calculated value based on all rows.</li><li>**All values** - Show a separate stat for every row. If you select this option, then you can also limit the number of rows to display.</li> |
| Calculation | This option is displayed when you select **Calculate** as your **Show** option. Select a reducer function that Grafana will use to reduce many fields to a single value. For a list of available calculations, refer to [Calculation types](ref:calculation-types). |
| Limit | This option is displayed when you select **All values** as your **Show** option. Set the maximum number of rows to display. The default is 25 and the maximum is 5,000. |
| Fields | Select the fields displayed in the visualization. |

<!-- prettier-ignore-end -->

### Stat styles

The stat visualization automatically adjusts the layout depending on available width and height in the dashboard, but you can also use the following options to further style the visualization.

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Orientation | Select the stat tile layout. Choose from: <ul><li>**Auto** - Grafana selects the ideal orientation.</li><li>**Horizontal** - Stat tiles are arranged left to right.</li><li>**Vertical** - Stat tiles are arranged top to bottom.</li></ul> |
| [Text mode](#text-mode) | You can use the **Text mode** option to control what text the visualization renders. If the value is not important, only the name and color is, then change the **Text mode** to **Name**. The value will still be used to determine color and is displayed in a tooltip. |
| [Wide layout](#wide-layout) | Set whether wide layout is enabled or not. Wide layout is enabled by default. This option is only applicable when **Text mode** is set to **Value and name**. |
| Color mode | Select a color mode. Choose from: <ul><li>**None** - No color applied to the value.</li><li>**Value** - Applies color to the value and graph area.</li><li>**Background Gradient** - Applies color to the value, graph area, and background, with a slight background gradient.</li><li>**Background Solid** - Applies color to the value, graph area, and background, with a solid background color.</li></ul> |
| Graph mode | Select a graph sparkline mode. Choose from: <ul><li>**None** - Hides the graph sparkline and only shows the value.</li><li>**Area** - Shows the graph sparkline in the background of the value. This requires that your query returns a time column.</li></ul> The graph sparkline is automatically hidden if the panel becomes too small.|
| Text alignment | Select an alignment mode. Choose from: <ul><li>**Auto** - If only a single value is shown (no repeat), then the value is centered. If multiple series or rows are shown, then the value is left-aligned.</li><li>**Center** - Stat value is centered.</li></ul> |
| Show percent change | Set whether percent change is displayed or not. Disabled by default. This option is applicable when the **Show** setting, under **Value options**, is set to **Calculate**. |
| Percent change color mode | This option is only displayed when **Show percent change** is enabled. Choose from: <ul><li>**Standard** - Green if the percent change is positive, red if the percent change is negative.</li><li>**Inverted** - Red if the percent change is positive, green if the percent change is negative.</li><li>**Same as Value** - Use the same color as the value.</li></ul> |
<!-- prettier-ignore-end -->

#### Text mode

You can use the Text mode option to control what text the visualization renders. If the value is not important, only the name and color is, then change the **Text mode** to **Name**. The value will still be used to determine color and is displayed in a tooltip.

- **Auto** - If the data contains multiple series or fields, show both name and value. Grafana also shows both name and value when the panel has no title or the field has a display name.
- **Value** - Show only value, never name. Name is displayed in the hover tooltip instead.
- **Value and name** - Always show value and name.
- **Name** - Show name instead of value. Value is displayed in the hover tooltip.
- **None** - Show nothing (empty). Name and value are displayed in the hover tooltip.

#### Wide layout

Set whether wide layout is enabled or not. Wide layout is enabled by default.

- **On** - Wide layout is enabled.
- **Off** - Wide layout is disabled.

This option is only applicable when **Text mode** is set to **Value and name**. When wide layout is enabled, the value and name are displayed side-by-side with the value on the right, if the panel is wide enough. When wide layout is disabled, the value is always rendered underneath the name.

### Text size

Adjust the sizes of the stat text.

- **Title** - Enter a numeric value for the stat title size.
- **Value** - Enter a numeric value for the stat value size.
- **Percent change** - Enter a numeric value for the percent change size when **Show percent change** is enabled.

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# State timeline

A state timeline visualization displays data in a way that shows state changes over time. In a state timeline, the data is presented as a series of bars or bands called _state regions_. State regions can be rendered with or without values, and the region length indicates the duration or frequency of a state within a given time range.

For example, if you're monitoring the CPU usage of a server, you can use a state timeline to visualize the different states, such as “LOW,” “NORMAL,” “HIGH,” or “CRITICAL,” over time. Each state is represented by a different color and the lengths represent the duration of time that the server remained in that state:

![A state timeline visualization showing CPU usage](/media/docs/grafana/panels-visualizations/screenshot-state-timeline-v11.4.png)

The state timeline visualization is useful when you need to monitor and analyze changes in states or statuses of various entities over time. You can use one when you need to:

- Monitor the status of a server, application, or service to know when your infrastructure is experiencing issues over time.
- Identify operational trends over time.
- Spot any recurring issues with the health of your applications.

## Configure a state timeline

{{< youtube id="a9wZHM0mdxo" >}}

{{< docs/play title="Grafana State Timeline & Status History" url="https://play.grafana.org/d/qD-rVv6Mz/6-state-timeline-and-status-history?orgId=1" >}}

## Supported data formats

The state timeline visualization works best if you have data capturing the various states of entities over time, formatted as a table. The data must include:

- **Timestamps** - Indicate when each state change occurred. This could also be the start time for the state change. You can also add an optional timestamp to indicate the end time for the state change.
- **Entity name/identifier** - Represents the name of the entity you're trying to monitor.
- **State value** - Represents the state value of the entity you're monitoring. These can be string, numerical, or boolean states.

Each state ends when the next state begins or when there is a `null` value.

### Example 1

The following example has a single time column and includes null values:

| Timestamps          | Server A | Server B |
| ------------------- | -------- | -------- |
| 2024-02-29 8:00:00  | Up       | Up       |
| 2024-02-29 8:15:00  | null     | Up       |
| 2024-02-29 8:30:00  | Down     | null     |
| 2024-02-29 8:45:00  |          | Up       |
| 2024-02-29 9:00:00  | Up       |          |
| 2024-02-29 9:15:00  | Up       | Down     |
| 2024-02-29 9:30:00  | Up       | Down     |
| 2024-02-29 10:00:00 | Down     | Down     |
| 2024-02-29 10:30:00 | Warning  | Down     |

The data is converted as follows, with the [null and empty values visualized as gaps](#connect-null-values) in the state timeline:

{{< figure src="/static/img/docs/state-timeline-panel/state-timeline-with-null-values.png" max-width="1025px" alt="A state timeline visualization with null values showing the status of two servers" >}}

### Example 2

The following example has two time columns and doesn't include any null values:

| Start time          | End time            | Server A | Server B |
| ------------------- | ------------------- | -------- | -------- |
| 2024-02-29 8:00:00  | 2024-02-29 8:15:00  | Up       | Up       |
| 2024-02-29 8:15:00  | 2024-02-29 8:30:00  | Up       | Up       |
| 2024-02-29 8:45:00  | 2024-02-29 9:00:00  | Down     | Up       |
| 2024-02-29 9:00:00  | 2024-02-29 9:15:00  | Down     | Up       |
| 2024-02-29 9:30:00  | 2024-02-29 10:00:00 | Down     | Down     |
| 2024-02-29 10:00:00 | 2024-02-29 10:30:00 | Warning  | Down     |

The data is converted as follows:

{{< figure src="/static/img/docs/state-timeline-panel/state-timeline-with-two-timestamps.png" max-width="1025px" alt="A state timeline visualization with two time columns showing the status of two servers" >}}

If your query results aren't in a table format like the preceding examples, especially for time-series data, you can apply specific [transformations](https://stackoverflow.com/questions/68887416/grafana-state-timeline-panel-with-values-states-supplied-by-label) to achieve this.

### Time series data

You can also create a state timeline visualization using time series data. To do this, add [thresholds](#thresholds), which turn the time series into discrete colored state regions.

![State timeline with time series](/media/docs/grafana/panels-visualizations/screenshot-state-timeline-time-series-v11.4.png)

## Pan and zoom panel time range

{{< docs/shared lookup="visualizations/panel-pan-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### State timeline options

Use these options to refine the visualization.

<!-- prettier-ignore-start -->

| Option | Description                                                                                     |
| ------ | ----------------------------------------------------------------------------------------------- |
| Merge equal consecutive values  | Controls whether Grafana merges identical values if they are next to each other. |
| Show values  | Controls whether values are rendered inside the state regions. Choose from **Auto**, **Always**, and **Never**. **Auto** renders values if there is sufficient space. |
| Align values | Controls value alignment inside state regions. Choose from **Left**, **Center**, and **Right**. |
| Row height | Controls bar height as a fraction of each row. `1` fills the row, and lower values leave more space between rows. |
| [Page size](#page-size-enable-pagination) | The **Page size** option lets you paginate the state timeline visualization to limit how many series are visible at once.  |
| Line width | Controls line width of state regions. This is a field option and can be set in field defaults or overrides. |
| Fill opacity | Controls the fill opacity of state regions. This is a field option and can be set in field defaults or overrides. |
| [Connect null values](#connect-null-values) | Choose how null values, which are gaps in the data, appear on the graph. |
| [Disconnect values](#disconnect-values) | Choose whether to set a threshold above which values in the data should be disconnected. |

<!-- prettier-ignore-end -->

#### Page size (enable pagination)

The **Page size** option lets you paginate the state timeline visualization to limit how many series are visible at once. This is useful when you have many series. With paginated results, the visualization displays a subset of all series on each page:

{{< video-embed src="/media/docs/grafana/panels-visualizations/screen-recording-grafana-11-2-state-timeline-pagination-dark.mp4" >}}

{{< docs/shared lookup="visualizations/connect-null-values.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

{{< docs/shared lookup="visualizations/disconnect-values.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Legend options

{{< docs/shared lookup="visualizations/legend-options-2.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Tooltip options

{{< docs/shared lookup="visualizations/tooltip-options-3.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Annotation options

{{< docs/shared lookup="visualizations/annotations/options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Axis options

{{< docs/shared lookup="visualizations/axis-options-3.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Status history

A status history visualization displays data in a way that shows periodic states over time. In a status history, each field or series is rendered as a horizontal row, with multiple boxes showing the different statuses. This provides you with a centralized view for the status of a component or service.

For example, if you're monitoring the health status of different services, you can use a status history to visualize the different statuses, such as “True” or "False," over time. Each status is represented by a different color:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-status-history-v11.6.png" max-width="800px" alt="A status history panel showing the health status of different sensors" >}}

{{< admonition type="note" >}}
A status history is similar to a [state timeline](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/state-timeline/), but has different [configuration options](#status-history-options). Unlike state timelines, status histories don't merge consecutive values.
{{< /admonition >}}

Use a status history when you need to:

- Monitor the status of a server, application, or service to know when your infrastructure is experiencing issues over time.
- Identify operational trends over time.
- Spot any recurring issues with the health of your applications.

## Configure a status history

Once you've [created a dashboard](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/create-dashboard/), you can use the following state timeline video as a reference for how to configure a status history:

{{< youtube id="a9wZHM0mdxo" >}}

{{< docs/play title="Grafana State Timeline & Status History" url="https://play.grafana.org/d/qD-rVv6Mz/6-state-timeline-and-status-history?orgId=1" >}}

## Supported data formats

The status history visualization works best if you have data capturing the various status of entities over time, formatted as a table. The data must include:

- **Timestamps** - Indicate when each status change occurred. This could also be the start time for the status change. You can also add an optional timestamp to indicate the end time for the status change.
- **Entity name/identifier** - Represents the name of the entity you're trying to monitor.
- **Status value** - Represents the state value of the entity you're monitoring. These can be string, numerical, or boolean states.

### Examples

The following tables are examples of the type of data you need for a status history visualization and how it should be formatted.

#### Single time column with null values

| Timestamps         | Backend_01 | Backend_02 |
| ------------------ | ---------- | ---------- |
| 2024-02-29 8:00:00 | OK         | WARN       |
| 2024-02-29 8:15:00 | WARN       |            |
| 2024-02-29 8:18:00 |            | WARN       |
| 2024-02-29 8:30:00 | BAD        |            |
| 2024-02-29 8:36:00 |            | OK         |
| 2024-02-29 8:45:00 | OK         |            |

The data is converted as follows, with the null and empty values visualized as gaps in the status history:

{{< figure src="/static/img/docs/status-history-panel/status_history_with_null.png" max-width="1025px" alt="A status history panel with null values showing the status of two servers" >}}

#### Two time columns without null values

| Start time         | End time           | Backend_01 | Backend_02 |
| ------------------ | ------------------ | ---------- | ---------- |
| 2024-02-29 8:00:00 | 2024-02-29 8:15:00 | OK         | OK         |
| 2024-02-29 8:15:00 | 2024-02-29 8:30:00 | OK         | OK         |
| 2024-02-29 8:30:00 | 2024-02-29 8:45:00 | OK         | OK         |
| 2024-02-29 8:45:00 | 2024-02-29 9:00:00 | BAD        | WARN       |
| 2024-02-29 9:00:00 | 2024-02-29 9:15:00 | OK         | WARN       |

The data is converted as follows:

{{< figure src="/static/img/docs/status-history-panel/status_history.png" max-width="1025px" alt="A status history panel with two time columns showing the status of two servers" >}}

## Pan and zoom panel time range

{{< docs/shared lookup="visualizations/panel-pan-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Status history options

Use these options to refine the visualization.

<!-- prettier-ignore-start -->

| Option | Description                                                                                     |
| ------ | ----------------------------------------------------------------------------------------------- |
| Show values  | Controls whether values are rendered inside the status cells. Choose from **Auto**, **Always**, and **Never**. **Auto** renders values if there is sufficient space. |
| Row height  | Controls the height of boxes. `1` creates the largest boxes, and `0` creates the smallest boxes with more gap. |
| Column width | Controls the width of boxes. `1` creates the widest boxes, and `0` creates the narrowest boxes with more gap. |
| Page size (enable pagination) | The **Page size** option lets you paginate the status history visualization to limit how many series are visible at once. This is useful when you have many series. |
| Line width | Controls line width of status cells. This is a field option and can be set in field defaults or overrides. |
| Fill opacity | Controls the fill opacity of status cells. This is a field option and can be set in field defaults or overrides. |

<!-- prettier-ignore-end -->

### Legend options

{{< docs/shared lookup="visualizations/legend-options-2.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Tooltip options

{{< docs/shared lookup="visualizations/tooltip-options-3.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Annotation options

{{< docs/shared lookup="visualizations/annotations/options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Axis options

{{< docs/shared lookup="visualizations/axis-options-3.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Table

Tables are a highly flexible visualization designed to display data in columns and rows.
The table visualization can take multiple datasets and provide the option to switch between them.
With this versatility, it's the preferred visualization for viewing multiple data types, aiding in your data analysis needs.

![Basic table visualization](/media/docs/grafana/panels-visualizations/screenshot-basic-table-v11.3.png)

You can use a table visualization to show datasets such as:

- Common database queries like logs, traces, metrics
- Financial reports
- Customer lists
- Product catalogs

Any information you might want to put in a spreadsheet can often be best visualized in a table.

Tables also provide different styles to visualize data inside the table cells, such as colored text and cell backgrounds, gauges, sparklines, data links, JSON code, and images.

{{< admonition type="note" >}}
Annotations and alerts are not currently supported for tables.
{{< /admonition >}}

## Configure a table visualization

The following video provides a visual walkthrough of the options you can set in a table visualization.
If you want to see a configuration in action, check out the video:

{{< youtube id="PCY7O8EJeJY" >}}

{{< docs/play title="Table Visualizations in Grafana" url="https://play.grafana.org/d/OhR1ID6Mk/" >}}

## Supported data formats

The table visualization supports any data that has a column-row structure.

{{< admonition type="note" >}}
If you’re using a cell type such as sparkline or JSON, the data requirements may differ in a way that’s specific to that type. For more info refer to [Cell type](#cell-type).
{{< /admonition >}}

### Example

This example shows a basic dataset in which there's data for every table cell:

```csv
Column1, Column2, Column3
value1 , value2 , value3
value4 , value5 , value6
value7 , value8 , value9
```

If a cell is missing or the table column-row structure is not complete, as in the following example, the table visualization won’t display any of the data:

```csv
Column1, Column2, Column3
value1 , value2 , value3
gap1   , gap2
value4 , value5 , value6
```

If you need to hide columns, you can do so using [data transformations](ref:data-transformation), [field overrides](#field-overrides), or by [building a query](ref:build-query) that returns only the needed columns.

## Column filtering

You can temporarily change how column data is displayed using column filtering.
For example, you can show or hide specific values.

### Turn on column filtering

To turn on column filtering, follow these steps:

1. In Grafana, navigate to the dashboard with the table with the columns that you want to filter.
1. Hover over any part of the panel to which you want to add the link to display the actions menu on the top right corner.
1. Click the menu and select **Edit**.
1. In the panel editor pane, expand the **Table** options section.
1. Toggle on the [**Column filter** switch](#table-options).

A filter icon (funnel) appears next to each column title.

{{< figure src="/static/img/docs/tables/column-filter-with-icon.png" max-width="350px" alt="Column filtering turned on" class="docs-image--no-shadow" >}}

### Filter column values

To filter column values, follow these steps:

1. Click the filter icon (funnel) next to a column title.

   Grafana displays the filter options for that column.

   {{< figure src="/media/docs/grafana/panels-visualizations/filter-column-values_12.2.png" max-width="300px" alt="Filter column values" >}}

1. Click the checkbox next to the values that you want to display or click **Select all**.
1. Enter text in the search field at the top to show those values in the display so that you can select them rather than scroll to find them.
1. Choose from several operators to display column values:
   - **Contains** - Matches a regex pattern (operator by default).
   - **Expression** - Evaluates a boolean expression. The character `$` represents the column value in the expression (for example, "$ >= 10 && $ <= 12").
   - The typical comparison operators: `=`, `!=`, `<`, `<=`, `>`, `>=`.

1. Click the checkbox above the **Ok** and **Cancel** buttons to add or remove all displayed values to and from the filter.

### Clear column filters

Columns with filters applied have a blue filter displayed next to the title.

{{< figure src="/static/img/docs/tables/filtered-column.png" max-width="100px" alt="Filtered column" class="docs-image--no-shadow" >}}

To remove the filter, click the blue filter icon and then click **Clear filter**.

<!-- vale Grafana.WordList = NO -->
<!-- vale Grafana.Spelling = NO -->

### Apply filters from the table

In tables, you can apply filters directly from the visualization with one click.

To display the filter icons, hover your cursor over the cell that has the value for which you want to filter:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-table-adhoc-filter-v12.2.png" max-width="500px" alt="Table with filter icon displayed on a cell" >}}

For more information about applying filters this way, refer to [Cross-filtering](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/filter-group-by/#dashboard-drilldown-with-filters).

<!-- vale Grafana.Spelling = YES -->
<!-- vale Grafana.WordList = YES -->

## Sort columns

Click a column title to change the sort order from default to descending to ascending.
Each time you click, the sort order changes to the next option in the cycle.
You can sort multiple columns by holding the `Cmd` or `Ctrl` key
and clicking the column name.

{{< figure src="/static/img/docs/tables/sort-descending.png" max-width="350px" alt="Sort descending" class="docs-image--no-shadow" >}}

## Dataset selector

If the data queried contains multiple datasets, a table displays a drop-down list at the bottom, so you can select the dataset you want to visualize.
This option is only available when you're editing the panel.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-table-multi-dataset-v11.3.png" max-width="650px" alt="Table visualization with multiple datasets" >}}

## Nested tables

A table can display sub-tables inside expandable rows. You can add these nested tables using the [Group to nested tables transformation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/#group-to-nested-tables), which groups rows by one or more fields, and can summarize nested row data by applying calculations.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-nested-table-collapsed-v13.1.png" max-width="650px" alt="Table with all rows collapsed" >}}

Click the expand icon on a row to toggle the visibility of its nested table:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-nested-table-expanded-v13.1.png" max-width="650px" alt="Table with two rows expanded showing nested sub-tables" >}}

To sort nested and top-level rows in nested tables, click a column title to change the sort order from default to descending to ascending.
Each time you click, the sort order changes to the next option in the cycle.
You can sort multiple columns by holding the `Cmd` or `Ctrl` key and clicking the column name.

{{< figure src="/static/img/docs/tables/sort-descending.png" max-width="350px" alt="Sort descending" class="docs-image--no-shadow" >}}

To control the display of fields inside a nested table&mdash;for example, to apply thresholds, units, or a different cell type&mdash;use [field overrides](#field-overrides) with the **Target fields** option set to **Nested**.
For more information, refer to [Apply overrides to nested table fields](#apply-overrides-to-nested-table-fields).

## Configuration options

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Table options

<!-- prettier-ignore-start -->
| Option               | Description                                               |
| -------------------- | --------------------------------------------------------- |
| Show table header    | Show or hide column names imported from your data source. |
| Frozen columns       | Freeze columns starting from the left side of the table. Enter a value to set how many columns are frozen. |
| Cell height          | Set the height of the cell. Choose from **Small**, **Medium**, or **Large**. |
| Max row height       | Define the maximum height for a row in the table. This can be useful when **Wrap text** is enabled for one or more columns. |
| Enable pagination    | Toggle the switch to control how many table rows are visible at once. When switched on, the page size automatically adjusts to the height of the table. This option doesn't affect queries. |
| Minimum column width | Define the lower limit of the column width, in pixels. By default, the minimum width of the table column is 150 pixels. For small-screen devices, such as mobile phones or tablets, reduce the value to `50` to allow table-based panels to render correctly in dashboards. |
| Column width         | Define a column width, in pixels, rather than allowing the width to be set automatically. By default, Grafana calculates the column width based on the table size and the minimum column width. |
| Column alignment     | Set how Grafana should align cell contents. Choose from: **Auto** (default), **Left**, **Center**, or **Right**.  |
| Column filter        | Temporarily change how column data is displayed. For example, show or hide specific values. For more information, refer to [Column filtering](#column-filtering). |
| Wrap text            | Enables text wrapping for cell content. |
| Wrap header text     | Enables text wrapping for column headers. |
<!-- prettier-ignore-end -->

### Table footer options

The table footer displays the results of calculations (and reducer functions) on fields.
The footer is only displayed after you select an option in the **Calculation** drop-down list:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-table-footer-selector-v12.2.png" max-width="300px" alt="The footer calculation selector, open" >}}

There are several calculations you can choose from including minimum, maximum, first, last, and total.
For the full list of options, refer to [Calculations](ref:calculations).

In the table footer:

- You can apply multiple calculations at once.
- The calculations and reducer functions apply to all fields in the table, by default. To control which fields have a calculation or function applied, add the table footer in an override instead.
- If you enable a mathematical function for a non-numeric field, nothing for that function is displayed for that field.

In the following image, multiple calculations&mdash;**Mean**, **Max**, and **Last**&mdash;have been applied:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-tablefooter-v12.2.png" max-width="750px" alt="Table with footer displaying mean, max, and last" >}}

You can also see in the previous image that the mathematical functions, **Mean** and **Max**, haven't been applied to the text field in the table.
Only the **Last** function has been applied to that field.

{{< admonition type="note">}}
Calculations applied to cell types like **Markdown + HTML** might have unexpected results.
{{< /admonition>}}

### Cell options

Cell options allow you to control how data is displayed in a table.
The options are differ based on the cell type that you select and are outlined within the descriptions of each cell type.
The following table provides short descriptions for each cell type and links to a longer description and the cell type options.

#### Cell type

By default, Grafana automatically chooses display settings.
You can override these settings by choosing one of the following cell types to control the default display for all fields.
Additional configuration is available for some cell types.

If you want to apply a cell type to only some fields instead of all fields, you can do so using the **Cell options > Cell type** field override.

<!-- prettier-ignore-start -->
| Cell type                                 | Description                                                                                                                |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| [Auto](#auto)                             | A basic text and number cell. |
| [Colored text](#colored-text)             | If thresholds, value mappings, or color schemes are set, then the cell text is displayed in the appropriate color. |
| [Colored background](#colored-background) | If thresholds, value mappings, or color schemes are set, then the cell background is displayed in the appropriate color. |
| [Data links](#data-links)                 | The cell text reflects the titles of the configured data links.|
| [Gauge](#gauge)                           | Values are displayed as a horizontal bar gauge. You can set the [Gauge display mode](#gauge-display-mode) and the [Value display](#value-display) options. |
| [Sparkline](#sparkline)                   | Shows values rendered as a sparkline. |
| [JSON View](#json-view)                   | Shows values formatted as code. |
| [Pill](#pill)                             | Displays each item in a comma-separated string in a colored block. |
| [Markdown + HTML](#markdown--html)        | Displays rich markdown or HTML content. |
| [Image](#image)                           | Displays an image when the value is a URL or a base64 encoded image. |
| [Actions](#actions)                       | The cell displays a button that triggers a basic, unauthenticated API call when clicked. |
<!-- prettier-ignore-end -->

#### Auto

This is a basic text and number cell.

It has the following cell options:

{{< docs/shared lookup="visualizations/cell-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

#### Colored text

If thresholds, value mappings, or color schemes are set, the cell text is displayed in the appropriate color.

![Table with colored text cell type](/media/docs/grafana/panels-visualizations/screenshot-table-colored-text-v11.3-2.png)

The colored text cell type has the following options:

{{< docs/shared lookup="visualizations/cell-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

#### Colored background

If thresholds, value mappings, or color schemes are set, the cell background is displayed in the appropriate color.

![Table with colored background cell type](/media/docs/grafana/panels-visualizations/screenshot-table-colored-bkgrnd-v11.3-2.png)

You can also set background cell color by row:

![Table with background cell color applied to row](/media/docs/grafana/panels-visualizations/screenshot-table-colored-row-v11.3.png)

The colored background cell type has the following options:

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Background display mode | Choose between **Basic** and **Gradient**. |
| Apply to entire row | Toggle the switch on to apply the background color that's configured for the cell to the whole row. When more than one field has this option enabled (for example, when it's set through the panel-level cell type or through overrides that target multiple fields), the row uses the color from the first (leftmost) matching column. |
| Cell value inspect | <p>Enables value inspection from table cells. When the switch is toggled on, clicking the inspect icon in a cell opens the **Inspect value** drawer which contains two tabs: **Plain text** and **Code editor**.</p><p>Grafana attempts to automatically detect the type of data in the cell and opens the drawer with the associated tab showing. However, you can switch back and forth between tabs.</p> |
| Tooltip from field | Toggle on the **Tooltip from field** switch to use the values from another field (or column) in a tooltip. For more information, refer to the [Tooltip from field](#tooltip-from-field). |
| Styling from field | Toggle on the **Styling from field** switch to apply the styling from another field (or column). The referenced field must contain CSS properties formatted in JSON object syntax (for example, `{"name":"John"}`). For more information, refer to the [Styling from field](#styling-from-field). |
<!-- prettier-ignore-end -->

<!-- The cell value inspect, tooltip from field, and styling from field descriptions above should be copied from docs/sources/shared/visualizations/cell-options.md -->

#### Data links

If you've configured data links, when the cell type is **Auto**, the cell text becomes clickable.
If you change the cell type to **Data links**, the cell text reflects the titles of the configured data links. To control the application of data link text more granularly, use a **Cell option > Cell type > Data links** field override.

#### Gauge

With this cell type, cells can be displayed as a graphical gauge, with several different presentation types.

The gauge cell type has the following options:

<!-- prettier-ignore-start -->
| Option             | Description                                                                                                    |
| ------------------ | -------------------------------------------------------------------------------------------------------------- |
| Gauge display mode | Controls the type of gauge used. For more information, refer to the [Gauge display mode](#gauge-display-mode). |
| Value display      | Controls how the value is displayed. For more information, refer to the [Value display](#value-display). |
| Tooltip from field | Toggle on the **Tooltip from field** switch to use the values from another field (or column) in a tooltip. For more information, refer to [Tooltip from field](#tooltip-from-field). |
| Styling from field | Toggle on the **Styling from field** switch to apply the styling from another field (or column). The referenced field must contain CSS properties formatted in JSON object syntax (for example, `{"name":"John"}`). For more information, refer to the [Styling from field](#styling-from-field). |
<!-- prettier-ignore-end -->

{{< admonition type="note" >}}
The maximum and minimum values of the gauges are configured automatically from the smallest and largest values in your whole dataset.
If you don't want the max/min values to be pulled from the whole dataset, you can configure them for each column using [field overrides](#field-overrides).
{{< /admonition >}}

##### Gauge display mode

You can set three gauge display modes.

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Basic | Shows a simple gauge with the threshold levels defining the color of gauge. {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-gauge-mode-basic-v11.3.png" alt="Table cell with basic gauge mode" >}} |
| Gradient | The threshold levels define a gradient. {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-gauge-mode-gradient-v11.3.png" alt="Table cell with gradient gauge mode" >}} |
| Retro LCD | The gauge is split up in small cells that are lit or unlit. {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-gauge-mode-retro-v11.3.png" alt="Table cell with retro LCD gauge mode" >}} |
<!-- prettier-ignore-end -->

##### Value display

Labels displayed alongside of the gauges can be set to be colored by value, match the theme text color, or be hidden.

<!-- prettier-ignore-start -->
| Option | Description |
| ------ | ----------- |
| Value color | Labels are colored by value. {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-labels-value-color-v11.3.png" alt="Table with labels in value color" >}} |
| Text color | Labels match the theme text color. {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-labels-text-color-v11.3.png" alt="Table with labels in theme color" >}} |
| Hidden | Labels are hidden. {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-labels-hidden-v11.3.png" alt="Table with labels hidden" >}} |
<!-- prettier-ignore-end -->

#### Sparkline

This cell type shows values rendered as a sparkline.
To show sparklines on data with multiple time series, use the [Time series to table transformation](ref:time-series-to-table-transformation) to process it into a format the table can show.

You can color sparklines using thresholds by setting the field's color scheme to **From thresholds (by value)** in the [standard color scheme options](ref:color-scheme), using an override, and setting **Gradient mode** to **Scheme**. When thresholds are defined, the sparkline automatically reflects threshold levels along the line.

![Table using sparkline cell type](/media/docs/grafana/panels-visualizations/screenshot-table-as-sparkline-v11.3.png)

The sparkline cell type options are described in the following table.
For more detailed information about all of the sparkline styling options (except **Hide value**), refer to the [time series graph styles documentation](ref:graph-styles).

<!-- prettier-ignore-start -->
| Option              | Description                                                                |
| ------------------- | --------------------------------------------------------------------------------------------- |
| Hide value          | Toggle the switch on or off to display or hide the cell value on the sparkline. |
| Style               | Choose whether to display your time-series data as **Lines**, **Bars**, or **Points**. You can use overrides to combine multiple styles in the same graph. |
| Line interpolation  | How the graph interpolates the series line. Choose from:<ul><li>**Linear** - Points are joined by straight lines.</li><li>**Smooth** - Points are joined by curved lines that smooths transitions between points.</li><li>**Step before** - The line is displayed as steps between points. Points are rendered at the end of the step.</li><li>**Step after** - The line is displayed as steps between points. Points are rendered at the beginning of the step.</li></ul> |
| Line width          | The thickness of the series lines or the outline for bars using the **Line width** slider. |
| Fill opacity        | The series area fill color using the **Fill opacity** slider. |
| Gradient mode       | Gradient mode controls the gradient fill, which is based on the series color. Gradient appearance is influenced by the **Fill opacity** setting. To change the color, use the standard color scheme field option. For more information, refer to [Color scheme](ref:color-scheme). Choose from:<ul><li>**None** - No gradient fill. This is the default setting.</li><li>**Opacity** - An opacity gradient where the opacity of the fill increases as y-axis values increase.</li><li>**Hue** - A subtle gradient that's based on the hue of the series color.</li><li>**Scheme** - The sparkline line color automatically reflects the configured threshold levels when the color scheme is set to **From thresholds (by value)** and thresholds are defined.</li></ul> |
| Line style          | Choose from:<ul><li>**Solid**</li><li>**Dash** - Select the length and gap for the line dashes. Default dash spacing is 10, 10.</li><li>**Dots** - Select the gap for the dot spacing. Default dot spacing is 0, 10.</li></ul> |
| Connect null values | How null values, which are gaps in the data, appear on the graph. Null values can be connected to form a continuous line or set to a threshold above which gaps in the data are no longer connected. Choose from:<ul><li>**Never** - Time series data points with gaps in the data are never connected.</li><li>**Always** - Time series data points with gaps in the data are always connected.</li><li>**Threshold** - Specify a threshold above which gaps in the data are no longer connected. This can be useful when the connected gaps in the data are of a known size or within a known range, and gaps outside this range should no longer be connected.</li></ul> |
| Show points         | Whether to show data points to lines or bars. Choose from: <ul><li>**Auto** - Grafana determines a point's visibility based on the density of the data. If the density is low, then points appear.</li><li>**Always** - Show the points regardless of how dense the dataset is.</li><li>**Never** - Don't show points.</li></ul> |
| Point size          | Set the size of the points, from 1 to 40 pixels in diameter. |
| Bar alignment       | Set the position of the bar relative to a data point. |
| Tooltip from field | Toggle on the **Tooltip from field** switch to use the values from another field (or column) in a tooltip. For more information, refer to [Tooltip from field](#tooltip-from-field). |
| Styling from field | Toggle on the **Styling from field** switch to apply the styling from another field (or column). The referenced field must contain CSS properties formatted in JSON object syntax (for example, `{"name":"John"}`). For more information, refer to the [Styling from field](#styling-from-field). |
<!-- prettier-ignore-end -->

#### JSON View

This cell type shows values formatted as code.
If a value is an object, the JSON object will appear on hover.

{{< figure src="/static/img/docs/tables/json-view.png" max-width="350px" alt="JSON view" class="docs-image--no-shadow" >}}

It has the following cell options:

{{< docs/shared lookup="visualizations/cell-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

#### Pill

The **Pill** cell type displays each item in a comma-separated string in a colored block.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-table-pill-cells-v12.2.png" max-width="750px" alt="Table using the pill cell type" >}}

The colors applied to each piece of text are maintained throughout the table.
For example, if the word "test" is first displayed in a red pill, it will always be displayed in a red pill.

The following data formats are supported for the pill cell type:

- Comma-separated values (`cows,chickens,goats`)
- JSON arrays of uniform (`(["cows","chickens","goats"])`) or mixed (`[1,2,3,"foo",42,"bar"]`) types

Toggle on the **Tooltip from field** switch to use the values from another field (or column) in a tooltip.
For more information, refer to [Tooltip from field](#tooltip-from-field).

#### Markdown + HTML

The **Markdown + HTML** cell type displays rich Markdown or HTML content, rendered using the
[GitHub-Flavored Markdown](https://github.github.com/gfm/) spec. This is useful if you need to display
customized, pre-formatted information alongside tabular data, such as formatted strings,
lists of links, or other dynamic cases.

For this cell type, you can toggle the **Dynamic height** switch, which allows the cell to resize
dynamically based on the cell content. If you use dynamic height, we strongly recommend that you
also toggle on **Pagination** to avoid performance issues in larger tables, since enabling
Dynamic height disables table {{< term "virtualization" >}}virtualization{{< /term >}}.

By default, the HTML rendered is sanitized, and un-sanitized HTML can only be rendered
in these cells if the [`disable_sanitize_html`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#disable_sanitize_html) option is set to true for your Grafana instance.

Toggle on the **Tooltip from field** switch to use the values from another field (or column) in a tooltip.
For more information, refer to [Tooltip from field](#tooltip-from-field).

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-table-markdown-v12.2.png" max-width="600px" alt="Table using the pill cell type" >}}

#### Image

If you have a field value that is an image URL or a base64 encoded image, this cell type displays it as an image.

![Table with image cell type](/media/docs/grafana/panels-visualizations/screenshot-table-cell-image-v11.3.png)

It has the following options:

<!-- prettier-ignore-start -->
| Option             | Description                                                                                                                   |
| ------------------ |  ---------------------------------------------------------------------------------------------------------------------------- |
| Alt text           | Set the alternative text of an image. The text will be available for screen readers and in cases when images can't be loaded. |
| Title text         | Set the text that's displayed when the image is hovered over with a cursor. |
| Tooltip from field | Toggle on the **Tooltip from field** switch to use the values from another field (or column) in a tooltip. For more information, refer to [Tooltip from field](#tooltip-from-field). |
<!-- prettier-ignore-end -->

#### Actions

Actions add a button to a cell that triggers a basic, unauthenticated API call when clicked. Configure actions from **Data links and actions** or with field overrides.

<!-- prettier-ignore-start -->
| Option             | Description  |
| ------------------ | ------------ |
| Endpoint           | Enter the endpoint URL. |
| Method             | Choose from **GET**, **POST**, and **PUT**. |
| Content-Type       | Select an option in the drop-down list. Choose from: JSON, Text, JavaScript, HTML, XML, and x-www-form-urlencoded. |
| Query parameters   | Enter as many **Key**, **Value** pairs as you need. |
| Header parameters  | Enter as many **Key**, **Value** pairs as you need. |
| Payload            | Enter the body of the API call. |
| Tooltip from field | Toggle on the **Tooltip from field** switch to use the values from another field (or column) in a tooltip. For more information, refer to [Tooltip from field](#tooltip-from-field). |
<!-- prettier-ignore-end -->

#### Tooltip from field

Toggle on the **Tooltip from field** switch to use the values from another field (or column) in a tooltip.

When you toggle the switch on, you can select from a drop-down list any of the fields in the table to be used as the source of the tooltip content.
All table fields are included in the drop-down list, whether visible or hidden.

When a tooltip from a field has been added to a cell, a chip is displayed in the top-right or top-left corner of the cell:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-tooltip-chip-1-v12.2.png" max-width="500px" alt="Tooltip chip" >}}

Hover your mouse over the chip to display the tooltip.

When you toggle on the switch, the **Tooltip placement** option, which controls where the tooltip box opens upon hover, is also displayed.
Select one of the following options: **Auto**, **Top**, **Right**, **Bottom**, and **Left**.

The content of the tooltip is determined by the values of the source field and can't be directly edited.
However, you can affect the display of the value using overrides like value mappings, as shown in the [Example: Tooltip from field with value mappings](#example-tooltip-from-field-with-value-mappings) section.

While you can turn on this option under **Cell options** and have it applied to all cells in the table, it's typically used as an override on a sub-set of cells instead.
This is demonstrated in the example in the following section.

##### Example: Tooltip from field using overrides

The following table has five visible fields (columns) as well as a hidden field called "Info":

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-tooltip-table-1-v12.2.png" max-width="750px" alt="Table that includes a hidden column" >}}

- The "Info" field is hidden using the **Table > Hide in table** override property.
- The following overrides have been applied to the "Short text" field:
  - The values from the "Info" field are used as tooltip text for the "Short text" cells using the **Cell options > Tooltip from field** override property.
  - The **Cell options > Tooltip placement** override property is set to control the placement of the tooltip.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-tooltip-override-2-v12.2.png" max-width="300px" alt="Override to use the Info field values as tooltips for the Short text column" >}}

Now, when you hover the cursor over the chip in the "Short text" column, the corresponding values from the "Info" column appear in the tooltip:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-tooltip-on-hover-v12.2.png" max-width="750px" alt="Info field value in the tooltip of the Short text cell upon hover" >}}

##### Example: Tooltip from field with value mappings

While the content of the tooltip is determined by the values of the source field and can't be directly edited, you can use field overrides on the source field to manipulate the display of that value.

For example, if the "Info" column is being used as the source field for the tooltip values, you could set up a value mapping.
In this case, the value "up" is mapped to the word "Good":

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-tooltip-value-map-v12.2.png" max-width="750px" alt="Info field value up being mapped to the value Good in an override" >}}

Now, when you hover the cursor over the chip in the "Short text" cell, the mapped value appears in the tooltip:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-tooltip-on-hover-2-v12.2.png" max-width="750px" alt="Info field mapped to a new value in the tooltip of the Short text cell upon hover" >}}

You can use all field overrides to affect the display of the tooltip.
For example, the **Table > Column width** or **Cell options > Cell type** overrides can change the cell width or visual display of the data.

#### Styling from field

Toggle on the **Styling from field** switch to apply the styling from another field (or column).
The referenced field must contain [CSS properties](https://developer.mozilla.org/en-US/docs/Web/API/CSSStyleProperties) formatted in JSON object syntax. For example:

```JSON
{"marginLeft":12, "text-decoration": "underline"}
```

While you can turn on this option under **Cell options** and have it applied to all cells in the table, it's typically used as an override on a sub-set of cells instead.
This is demonstrated in the following example.

The following table has six visible fields (columns) as well as a hidden field called "Style":

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-style-from-field-config-v12.3.png" max-width="750px" alt="Configuration of a table including the styling from field option" >}}

- The "Style" field has JSON objects with CSS properties. (Note that they are formatted for use in CSV format in this example.)
- The "Style" field is hidden using the **Table > Hide in table** override property.
- The "Info" field is using the **Cell options > Styling from field** override property with the "Style" field as the source.

The following image shows the "Info" field with the styling from the "Style" field applied:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-style-from-field-v12.3.png" max-width="750px" alt="Info field with styling from Style field applied" >}}

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-3.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

#### Apply overrides to nested table fields

By default, field overrides apply only to columns in the parent table.
To target columns inside a nested table, set the **Target fields** option on the override to **Nested**:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-table-override-nested-scope-v13.x.png" max-width="350px" alt="Field override configuration with the Target fields selector showing Series and Nested options" >}}

All standard override properties&mdash;including thresholds, value mappings, units, data links, and cell type&mdash;apply the same way to nested fields.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-nested-table-w-overrides-v13.1.png" max-width="650px" alt="Nested table with a threshold override applied to a column inside an expanded sub-table" >}}

# Text

{{< admonition type="note" >}}
The new text panel is currently in public preview.
Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.

To use this feature, enable the `grafana.newTextPanel` and `text.newFeatures` feature toggles in your Grafana configuration file or contact Support.

For information about the generally available experience, refer to the [latest OSS documentation](https://grafana.com/docs/grafana/latest/visualizations/panels-visualizations/visualizations/text/).
{{< /admonition >}}

Text visualizations let you include text, HTML, code blocks, or Mermaid diagrams in your dashboards.
You can use them to add contextual information and descriptions or to embed complex HTML.
With text visualizations, you can build things like lightweight data-driven tables, cards, lists, and status summaries.

The following images show different uses of the text visualization.

Here, text visualizations use Markdown and HTML modes:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-text-markdown-html-v13.3.png" max-width="750px" alt="Text visualizations in Markdown and HTML mode" >}}

In this example, one text visualization is in TypeScript mode and the other is using Handlebars templating:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-text-code-handlebars-v13.3.png" max-width="750px" alt="Text visualizations in TypeScript mode and using Handlebars templating" >}}

Finally, this image shows one text visualization using per row render mode and another using pagination:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-text-render-pagination-v13.3.png" max-width="750px" alt="Text visualizations using per row render mode and pagination" >}}

{{< docs/play title="Text Panel" url="https://play.grafana.org/d/adl33bxy1ih34b/" >}}

Use a text visualization when you need to:

- Add important links or useful annotations.
- Provide instructions or guidance on how to interpret different panels, configure settings, or take specific actions based on the displayed data.
- Announce any scheduled maintenance or downtime that might impact your dashboards.

## Text visualization editor

The editor is where you enter and preview the content of a text visualization.
It's where you set text modes, like Markdown, HTML, or a specific coding language.
Additionally, it provides formatting options and syntax highlighting, as well as variable and diagram support.

To learn more, click the following links:

- [Editor views](#views)
- [Formatting toolbar](#toolbar)
- [Modes](#modes)

### Views

The editor provides three views:

- **Preview**: See only the preview block.
- **Split**: See the authoring and preview blocks at the same time.
- **Write**: See only the authoring block.

### Toolbar

The toolbar provides formatting options for common Markdown and HTML operations, like bold or italic text and lists.
You can also insert:

- Tables
- Mermaid diagrams
- Dashboard variables

Dashboard variables autocomplete as you enter them, and they're interpolated in the content.

The options displayed in the formatting toolbar depend on whether you select Markdown or HTML.

### Modes

The visualization mode determines how embedded content appears.
Choose from:

- **Markdown**: Formats the content as [Markdown](https://en.wikipedia.org/wiki/Markdown).
- **HTML**: Renders the content as [sanitized](https://github.com/grafana/grafana/blob/main/packages/grafana-data/src/text/sanitize.ts) HTML. If you require more direct control over the output, you can set the [`disable_sanitize_html`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#disable_sanitize_html) flag which enables you to directly enter HTML.
- **Code**: Renders content inside a read-only code editor. [Variables](/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/variable-syntax/) in the content are expanded for display. The code options are: Go, HTML, JSON, Markdown, Plain text, SQL, TypeScript, XML, YAML.

The editor provides syntax highlighting in the authoring block.

{{< admonition type="note" >}}
When a text visualization has query data, an iframe only loads after you allow its source.
For more information, refer to the [Security and external documentation](#security-and-external-content).

To embed Grafana pages in an iframe, also set [`allow_embedding`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#allow_embedding) to `true` in your Grafana configuration file or environment variables.
This setting controls whether browsers can show Grafana in a frame. Other websites control this with their own settings.
{{< /admonition >}}

## Render query results as text with Handlebars templates

Use `${__value}`, `${__field}`, `${__series}`, and `${__data}` macros to reference query results in the visualization content, and [Handlebars templating](https://handlebarsjs.com/guide/) for more control.
Choose what data to display, repeat content for each row with `{{#each}}`, and add if/then logic with `{{#if}}` helpers.

![Text visualization with Handlebars support](/media/docs/grafana/panels-visualizations/screenshot-text-handlebars-v13.3.png)

## Style content with theme colors

Use the [`$__theme` variable](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/variables/built-in-variables/#__theme) in inline styles to match the current Grafana theme.
The content updates when a user switches between light and dark themes.

For example, the following HTML shows a card with the theme's background, border, and secondary text colors:

```html
<div
  style="padding: 12px; background: ${__theme.colors.background.secondary}; border: 1px solid ${__theme.colors.border.weak}; border-radius: ${__theme.shape.radius.default}"
>
  <strong>checkout</strong>
  <div style="color: ${__theme.colors.text.secondary}; font-size: ${__theme.typography.size.sm}">182 ms</div>
</div>
```

The variable works in Markdown and HTML modes, and in the output of Handlebars templates.
For example, use `{{#if}}` to choose a success or error color for each row.

## Security and external content

When a text visualization has query data, Grafana renders its content in an isolated frame.
This protects the dashboard and your Grafana session from the content of the panel.

### What the frame blocks

Inside the frame, the content can't:

- Run scripts, including inline scripts and event handlers.
- Read your Grafana session, cookies, or the rest of the dashboard.
- Call the Grafana API or send requests to other services.
- Submit forms or embed plugins with `<object>` or `<embed>`.
- Load images, stylesheets, fonts, audio, video, or iframes from a URL, unless you allow its source.

Grafana always sanitizes HTML in the frame, even when [`disable_sanitize_html`](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#disable_sanitize_html) is enabled.
Inline styles, `data:` images, and Mermaid diagrams still work.
By default, links open in a new tab.

### Review blocked resources

In most cases, when a panel tries to load content from a URL, Grafana blocks it and shows a banner at the top of the dashboard and in the panel editor.
To load the content, complete the following steps:

1. In the banner, click **Review**.
1. Review each source and the full list of URLs that it requests.
1. Select the sources that you trust.
1. Click **Allow** or **Always allow**.

You can't allow a source that requests too many URLs to list, because you can't review all of them.

### Allow and Always allow

The two options differ in where the approval applies and how long it lasts:

- **Allow**: Loads content from the source in the text visualizations listed in the review. The approval only applies to the current dashboard and ends when you close the browser tab.
- **Always allow**: Loads content from the source in any text visualization. The approval applies to any dashboard and lasts for 30 days.

Approvals apply only to you, in the current organization and browser.
They're stored in your browser, not in Grafana, and they aren't shared with other users.

### What an approval allows

When you allow a source, you allow its whole website, for example `https://example.com`, not only the URLs listed in the review.
The panel can then load images, stylesheets, fonts, audio, video, and iframes from that website.

Before you allow a source, consider the following risks:

- **Data in URLs**: Panel content can put dashboard variables and query results into URLs. The website receives that data with each request.
- **Viewer information**: The website receives your IP address, your browser details, and the time of each request. The requests don't include the address of your Grafana instance or the dashboard, unless the panel content puts it in a URL.
- **Content changes**: The website can change what it serves after you allow it. An approval doesn't lock the content that you reviewed.
- **Embedded pages**: An iframe from the website can show any content in the panel, and it can load content from other websites. However, it can't run scripts, submit forms, or read your Grafana session or the dashboard.

Only allow websites that you trust with the data on the dashboard.

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data options

Use the following options to control how data is rendered in the text visualization.

#### Render mode

Select a render mode to control how your template is applied:

- **Once**: Render the whole result.
- **Per row**: Each row in the data gets its own repeated block of content.

You can also select which data frame to display.

![Text visualization with Render mode set to per row](/media/docs/grafana/panels-visualizations/screenshot-text-render-mode-2-v13.3.png)

#### Page size

When a query returns many rows, pagination lets you move through the full result instead of showing only a limited number of rows.
Grafana automatically adjusts the number of rows on each page to fit the panel or you can set a range from 1-1000 rows.

This option only displays when you set **Render mode** to **Per row** and when there are 100 rows or more.

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

<!-- Show line numbers: Displays line numbers in the panel preview when you choose **Code** as your text mode. -->

# Time series

Time series visualizations are the default way to show the variations of a set of data values over time. Each data point is matched to a timestamp and this _time series_ is displayed as a graph. The visualization can render series as lines, points, or bars and it's versatile enough to display almost any type of [time-series data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/fundamentals/timeseries/).

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-time-series-v12.0.png" max-width="750px" alt="Time series visualization" >}}

{{< admonition type="note" >}}
When you open a dashboard with a legacy Graph visualization, Grafana migrates it automatically to time series or another supported visualization based on the old panel settings.
{{< /admonition >}}

A time series visualization displays an x-y graph with time progression on the x-axis and the magnitude of the values on the y-axis. This visualization is ideal for displaying large numbers of timed data points that would be hard to track in a table or list.

You can use the time series visualization if you need track:

- Temperature variations throughout the day
- The daily progress of your retirement account
- The distance you jog each day over the course of a year

## Configure a time series visualization

The following video guides you through the creation steps and common customizations of time series visualizations, and is great for beginners:

{{< youtube id="RKtW87cPxsw" >}}

{{< docs/play title="Time Series Visualizations in Grafana" url="https://play.grafana.org/d/000000016/" >}}

## Supported data formats

Time series visualizations require time-series data—a sequence of measurements, ordered in time, and formatted as a table—where every row in the table represents one individual measurement at a specific time. Learn more about [time-series data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/fundamentals/timeseries/).

The dataset must contain at least one numeric field, and in the case of multiple numeric fields, each one is plotted as a new line, point, or bar labeled with the field name in the tooltip.

Time series data is expected to contain unique timestamps for each data point within a series. If multiple points in the same series share the same timestamp, the visualization might not render or behave as expected.

### Example 1

In the following example, there are three numeric fields represented by three lines in the chart:

| Time                | value1 | value2 | value3 |
| ------------------- | ------ | ------ | ------ |
| 2022-11-01 10:00:00 | 1      | 2      | 3      |
| 2022-11-01 11:00:00 | 4      | 5      | 6      |
| 2022-11-01 12:00:00 | 7      | 8      | 9      |
| 2022-11-01 13:00:00 | 4      | 5      | 6      |

![Time series line chart with multiple numeric fields](/media/docs/grafana/panels-visualizations/screenshot-grafana-11.1-timeseries-example1v2.png 'Time series line chart with multiple numeric fields')

If the time field isn't automatically detected, you might need to convert the data to a time format using a [data transformation](ref:panel-data-section).

### Example 2

The time series visualization also supports multiple datasets. If all datasets are in the correct format, the visualization plots the numeric fields of all datasets and labels them using the column name of the field.

#### Query1

| Time                | value1 | value2 | value3 |
| ------------------- | ------ | ------ | ------ |
| 2022-11-01 10:00:00 | 1      | 2      | 3      |
| 2022-11-01 11:00:00 | 4      | 5      | 6      |
| 2022-11-01 12:00:00 | 7      | 8      | 9      |

#### Query2

| timestamp           | number1 | number2 | number3 |
| ------------------- | ------- | ------- | ------- |
| 2022-11-01 10:30:00 | 11      | 12      | 13      |
| 2022-11-01 11:30:00 | 14      | 15      | 16      |
| 2022-11-01 12:30:00 | 17      | 18      | 19      |
| 2022-11-01 13:30:00 | 14      | 15      | 16      |

![Time series line chart with two datasets](/media/docs/grafana/panels-visualizations/screenshot-grafana-11.1-timeseries-example2v2.png 'Time series line chart with two datasets')

### Example 3

If you want to more easily compare events between different, but overlapping, time frames, you can do this by using a time offset while querying the compared dataset:

#### Query1

| Time                | value1 | value2 | value3 |
| ------------------- | ------ | ------ | ------ |
| 2022-11-01 10:00:00 | 1      | 2      | 3      |
| 2022-11-01 11:00:00 | 4      | 5      | 6      |
| 2022-11-01 12:00:00 | 7      | 8      | 9      |

#### Query2

| timestamp(-30min)   | number1 | number2 | number3 |
| ------------------- | ------- | ------- | ------- |
| 2022-11-01 10:30:00 | 11      | 12      | 13      |
| 2022-11-01 11:30:00 | 14      | 15      | 16      |
| 2022-11-01 12:30:00 | 17      | 18      | 19      |
| 2022-11-01 13:30:00 | 14      | 15      | 16      |

![Time Series Example with second Data Set offset](/media/docs/grafana/panels-visualizations/screenshot-grafana-11.1-timeseries-example3v2.png 'Time Series Example with second Data Set offset')

When you add the offset, the resulting visualization makes the datasets appear to be occurring at the same time so that you can compare them more easily.

## Alert rules

You can [link alert rules](ref:link-alert) to time series visualizations in the form of annotations to observe when alerts fire and are resolved. In addition, you can create alert rules from the **Alert** tab within the [panel editor](ref:panel-data-section).

## Special overrides

The following overrides help you further refine a time series visualization.

### Transform override property

Use the **Graph styles > Transform** [override property](#field-overrides) to transform series values without affecting the values shown in the tooltip, context menu, or legend. Choose from the following transform options:

- **Constant** - Show the first value as a constant line.
- **Negative Y** - Flip the results to negative values on the y-axis.

### Fill below to override property

The **Graph styles > Fill below to** [override property](#field-overrides) fills the area between two series. When you configure the property, select the series for which you want the fill to stop.

The following example shows three series: Min, Max, and Value. The Min and Max series have **Line width** set to 0. Max has a **Fill below to** override set to Min, which fills the area between Max and Min with the Max line color.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-fill-below-to-v12.0.png" max-width="600px" alt="Fill below to example" >}}

{{< docs/shared lookup="visualizations/multiple-y-axes.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+2" >}}

## Pan and zoom panel time range

{{< docs/shared lookup="visualizations/panel-pan-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Tooltip options

{{< docs/shared lookup="visualizations/tooltip-options-2.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1">}}

### Legend options

{{< docs/shared lookup="visualizations/legend-options-1.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

#### Series visibility

Toggle the **Series visibility** switch on to add the control next to or above the legend.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-series-visibility-v13.1.png" max-width="400px" alt="Series visibility filter switch" >}}

This lets you narrow the visible series by name or by label.
Use the series visibility filter when a panel renders many series and you want to focus on a subset without editing the query.

After you've toggled the switch on, click the **Series visibility** icon to open a tooltip.
Depending on your dataset, you can filter:

- **By name**: Lists each unique series name. Select one or more names to limit the visualization to those series.
- **By labels**: Lists each label key with its values. Select label values to filter series that match.

The tooltip also provides the following options:

- **Select all** and **Deselect all**: Toggle every value in a section.
- **Clear all**: Reset the filter.
- **Pin to sidebar**: Dock the filter alongside the panel so it stays open while you explore.

For more information, refer to the [Configure legend documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/configure-legend/#series-visibility).

### Axis options

{{< docs/shared lookup="visualizations/axis-options-1.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Annotation options

{{< docs/shared lookup="visualizations/annotations/options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Graph styles options

The options under the **Graph styles** section let you control the general appearance of the graph, excluding [color](#standard-options).

{{< docs/shared lookup="visualizations/graph-styles-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Traces

{{< shared id="traces-viz" >}}

Traces visualizations let you follow a request as it traverses the services in your infrastructure.
The traces visualization displays traces data in a diagram that allows you to easily interpret it. Traces visualizations currently render one trace traversal based on the traceID used in TraceQL or using a variable.

{{< /shared >}}

For more information about traces and how to use them, refer to the following documentation:

- [Tracing in Explore](ref:tracing-in-explore)
- [Tempo data source](ref:tempo-data-source)
- [Getting started with Tempo](/docs/tempo/latest/getting-started)

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-traces-visualization-11.5.png" max-width= "700px" alt="Screenshot of the trace view" >}}

{{< docs/play title="Traces Panel" url="https://play.grafana.org/d/edodkfmj0tce8f/" >}}

## Add a panel with tracing visualizations

After you have tracing data available in your Grafana instance, you can add tracing panels to your Grafana dashboards.

Using a dashboard variable, `traceID`, lets you create a query to show specific traces for a given trace ID.
For more information about dashboard variables, refer to the [Variables documentation](ref:variables-documentation).

### Before you begin

To use this procedure, you need:

- A Grafana instance
- A [Tempo data source](ref:tempo-data-source) connected to your Grafana instance

### Steps {#add-the-traces-panel-query}

To view and analyze traces data in a dashboard, you need to add the traces visualization to your dashboard and define a query using the panel editor.
The query determines the data that is displayed in the visualization.
For more information on the panel editor, refer to the [Panel editor documentation](ref:panel-editor-documentation).

This procedure uses dashboard variables and templates to allow you to enter trace IDs which can then be visualized. You'll use a variable called `traceId` and add it as a template query.

1. From your Grafana instance, do one of the following:
   - On a new dashboard: Click or drag a panel onto the dashboard.
   - On an existing dashboard: Click **Edit** in the top-right corner, click the **Add new element** icon, and then click or drag a panel onto the dashboard.

1. Click **Configure visualization** to open panel edit mode.
1. In the query editor, click the data source list and select the appropriate tracing data source.
1. In the top-right corner of the panel editor, select the **All visualizations** tab, search for, and select **Traces**.
1. Under the **Panel options**, enter a **Title** for your trace panel or have Grafana create one using [generative AI features](ref:generative-ai-features).

   For more information on the panel editor, refer to the [Configure panel options documentation](ref:configure-panel-options-documentation).

1. In the query editor, click the **TraceQL** query type tab.
1. Enter `${traceId}` in the TraceQL query field to create a dashboard variable. This variable is used as the template query.

   {{< figure src="/static/img/docs/panels/traces/screenshot-traces-template-query.png" alt="Add a template query" >}}

1. Click **Back**.
1. Click the **Add new element** icon and click **Variable**.
1. Add a new variable called `traceId`, of variable type **Custom**, giving it a label if required.

   {{< figure src="/media/docs/grafana/panels-visualizations/screenshot-traces-custom-variable-v11.5.png" max-width="400px" alt="Add a template query" >}}

1. Click **Save**.
1. Enter an optional description of your changes, and click **Save**.
1. Click **Exit edit**.
1. Verify that the panel works by using a valid trace ID for the data source used for the trace panel and editing the ID in the dashboard variable.

   {{< figure src="/static/img/docs/panels/traces/screenshot-traces-traceid-panel.png" alt="Results of query in trace panel" >}}

## Add TraceQL with table visualizations

While you can add a trace visualization to a dashboard, having to manually add trace IDs as a dashboard variable is cumbersome.
It's more useful to instead be able to use TraceQL queries to search for specific types of traces and then select appropriate traces from matching results.

1. In the same dashboard where you added the trace visualization, click **Edit** in the top-right corner.
1. Click the **Add new element** icon, and then click or drag a panel onto the dashboard.
1. Click **Configure visualization**.
1. Select the same trace data source you used in the previous task.
1. In the top-right corner of the panel editor, select the **All visualizations** tab, search for, and select **Table**.
1. In the query editor, select the **TraceQL** tab.
1. Under the **Panel options**, enter a **Title** for your trace panel or have Grafana create one using [generative AI features](ref:generative-ai-features).
1. Add an appropriate TraceQL query to search for traces that you would like to visualize in the dashboard. This example uses a simple, static query. You can write the TraceQL query as a template query to take advantage of other dashboard variables, if they exist. This lets you create dynamic queries based on these variables.

   {{< figure src="/static/img/docs/panels/traces/screenshot-traces-dynamic-query.png" alt="Create a dynamic query" >}}

1. Click **Save**.
1. Enter an optional description of your changes, and click **Save**.
1. Click **Back** and **Exit edit**.

When results are returned from a query, the results are rendered in the panel’s table.

{{< figure src="/static/img/docs/panels/traces/screenshot-traces-returned-query.png" alt="Results of a returned query in the panel table" >}}

### Use a variable to add other links to traces

The results in the traces visualization include links to the **Explore** page that renders the trace. You can add other links to traces in the table that fill in the `traceId` dashboard variable when selected, so that the trace is visualized in the same dashboard.

To create a set of data links in the panel, use the following steps:

1. In the panel editor menu, under **Data links**, click **Add link**.
1. Add a **Title** for the data link.
1. Find the UUID of the dashboard by looking in your browser’s address bar when the full dashboard is being rendered. Because this is a link to a dashboard in the same Grafana instance, only the path of the dashboard is required.

   {{< figure src="/static/img/docs/panels/traces/screensnot-traces-uuid-url.png" alt="Unique identifier for the dashboard" >}}

1. In the **URL** field, make a self-reference to the dashboard that contains both of the panels. This self-reference uses the value of the selected trace in the table to fill in the dashboard variable. Use the path for the dashboard from the previous step and then fill in the value of `traceId` using the selected results from the TraceQL table.
   The trace ID is exposed using the `traceID` data field in the returned results, so use that as the value for the dashboard variable.

   {{< figure src="/static/img/docs/panels/traces/screenshot-traces-edit-link.png" alt="Edit link and add the Trace link" >}}

1. Select **Save** to save the data link.
1. Enter an optional description of your changes, and click **Save**.
1. Click **Back** and **Exit edit**.

You should now see a list of matching traces in the table visualization. While selecting the **TraceID** or **SpanID** fields will give you the option to either open the **Explore** page to visualize the trace or following the data link, selecting any other field (such as **Start time**, **Name** or **Duration**) automatically follows the data link, filling in the `traceId` dashboard variable, and then shows the relevant trace in the trace panel.

{{< figure src="/static/img/docs/panels/traces/screenshot-traces-trace-link.png" alt="Selecting the trace link" >}}

{{< figure src="/static/img/docs/panels/traces/screenshot-traces-trace-link-follow.png" caption="Follow the trace link populates the trace ID and displays the traces view" >}}

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Span filters options

The **Span filters** options control the initial state of the span filters when the visualization loads, allowing you to customize your trace analysis view.

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Filters | <p>Add free-form filters that set the initial span filter state. Use **Text search** for text queries, `duration` for duration filters, or span attributes such as `service.name` and `span.name`.</p><p>Supports variable interpolation. For example, you can set a filter value to `$var`, and the visualization replaces it with the value for the dashboard variable named `$var`.</p> |
| Show matches only | Toggle the switch on to display only spans that match the defined filter criteria. This helps simplify trace interpretation. |
| Select critical path | Toggle the switch on to highlight spans in the critical path, which helps identify performance bottlenecks and their impact on overall latency. |

<!-- prettier-ignore-end -->

Duration filters use the `duration` key with operators such as `=`, `>=`, `<=`, `>`, and `<`. In the visualization, users can toggle **Show all spans** to switch from the filtered view back to all spans.

# Trend

Trend visualizations should be used for datasets that have a sequential, numeric x-field that is not time. Some examples are function graphs, rpm/torque curves, supply/demand relationships, and elevation or heart rate plots along a race course (with x as distance or duration from start).

For example, you could represent engine power and torque versus speed where speed is plotted on the x-axis and power and torque are plotted on the y-axes:

{{< figure src="/media/docs/grafana/screenshot-trend-visualization-v12.0.png" max-width="750px" alt="Trend engine power and torque curves" >}}

Trend visualizations support all visual styles and options available in the [time series visualization](ref:time-series-visualization) with these exceptions:

- No annotations or time regions
- No shared cursor/crosshair
- No multi-timezone x-axis
- No ability to change the dashboard time range using drag-selection

Trend visualizations require at least two numeric fields. The x-field must use ascending numeric values. If the values aren't ascending, Grafana shows an error. When multiple frames or queries exist, you should use a [join transformation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/transform-data/) on the x-fields to produce a single frame.

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### X axis options

In the **X field** option, select a numeric field that contains ascending values.

### Tooltip options

{{< docs/shared lookup="visualizations/tooltip-options-2.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Legend options

{{< docs/shared lookup="visualizations/legend-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Graph styles options

The options under the **Graph styles** section let you control the general appearance of the graph, excluding [color](#standard-options). These options apply to numeric-x series.

{{< docs/shared lookup="visualizations/graph-styles-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Axis options

{{< docs/shared lookup="visualizations/axis-options-2.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+1" >}}

### Standard options

{{< docs/shared lookup="visualizations/standard-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# XY chart

XY charts provide a way to visualize arbitrary x and y values in a graph so that you can easily show the relationship between two variables. XY charts are typically used to create scatter plots. You can also use them to create bubble charts where field values determine the size of each bubble:

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-xy-charts-v11.6.png" max-width="750px" alt="An xy chart showing height weight distribution" >}}

## Supported data formats

You can use any type of tabular data with at least two numeric fields in an xy chart. The x field can be a time field. This type of visualization doesn't require time data, but it can be used.

## Configuration options

{{< docs/shared lookup="visualizations/config-options-intro.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Panel options

{{< docs/shared lookup="visualizations/panel-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### XY Chart options

The following options let you control how data is displayed in an xy chart:

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| [Series mapping](#series-mapping) | Set how series data is mapped in the chart. Choose from: **Auto** and **Manual**. Depending on your series mapping selection, the **Frame**, **X-field**, and **Y-field** options differ. For information on setting these specific fields, refer to the [Series mapping section](#series-mapping). |
| Size field | Set which field's values control the size of the points in the chart. This value is relative to the min and max of all the values in the data frame. When you select this option, you can then set the **Min point size** and **Max point size** options. Required in **Manual** mode. |
| Color field | Set which field's values control the color of the points in the chart. To use the color value options under the **Standard** options, you must set this field. Typically, this field is used when you only have one series displayed in the chart. Required in **Manual** mode. Only supports `number` fields.  |
| [Show](#show) | Set how values are represented in the visualization. Choose from: **Points**, **Lines**, or **Both**. |
| Point size | Set the size of all points in the chart, from one to one hundred pixels in diameter. The default size is five pixels. You can set an [override](ref:configure-field-overrides) to set the pixel size by series (y-field). |
| Min/Max point size | Use these options to control the minimum or maximum point size when you've set the **Size field** option. You can [override](ref:configure-field-overrides) these options for specific series. |
| Point shape | Set the shape of the points in the chart. Choose from:<ul><li>**Circle** - The default setting</li><li>**Square** </li></ul> |
| Point stroke width | The width of the point stroke in pixels. The default is one pixel. |
| Fill opacity | The opacity of the point fill. The default is 50. |
| [Line style](#line-style) | Set the style of the lines that connect points. Choose from: **Solid**, **Dash**, or **Dots**. |
| Line width | The width of the lines that connect points, in pixels. |

<!-- prettier-ignore-end -->

#### Series mapping

Set how series data is mapped in the chart. Choose from:

- **Auto** - Automatically generates series from all available data frames (or datasets). You can filter to select only one frame.
- **Manual** - Explicitly define the series by selecting from available data frames.

Depending on your series mapping selection, the **Frame**, **X-field**, and **Y-field** options differ.
These options are described in the tabs that follow:

{{< tabs >}}
{{< tab-content name="Auto series mapping options" >}}

When you select **Auto** as your series mapping mode, the following options are preconfigured, but you can also define them yourself:

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Frame | By default, an xy chart displays all data frames. You can filter to select only one frame. |
| [X field](#x-field) | Select which field or fields x represents. By default, this is the first number or time field in each data frame. For an example of this in **Auto** mode, refer to the [X field section](#x-field). |
| [Y field](#y-field) | After the x-field is set, by default, all the remaining number fields in the data frame are designated as the y-fields. You can use this option to explicitly choose which fields to use for y. For more information on how to use this in **Auto** mode, refer to the [Y field section](#y-field). |

<!-- prettier-ignore-end -->

{{< /tab-content >}}
{{< tab-content name="Manual series mapping options" >}}

When you select **Manual** as your series mode, you can add, edit, and delete series.
To manage a series, click the **Series** field; to rename the series, click the series name.

In **Manual** mode, these fields are required:

<!-- prettier-ignore-start -->

| Option | Description |
| ------ | ----------- |
| Frame | Select your data frame or dataset. You can add as many frames as you want. |
| X field | Select which field x represents. |
| Y field | Select which field y represents. |

<!-- prettier-ignore-end -->

{{< /tab-content >}}
{{< /tabs >}}

#### X field

In **Auto** series mapping mode, select which field or fields x represents. By default, this is the first number or time field in each data frame. For example, you enter the following CSV content:

| a   | b   | c   |
| --- | --- | --- |
| 0   | 0   | 0   |
| 1   | 1   | 9   |
| 2   | 2   | 4   |

In the resulting chart, the x-field is generated from the values in column "a" unless you define it differently.

#### Y field

In **Auto** series mapping mode, after the x-field is set, by default, all the remaining number fields in the data frame are designated as the y-fields.
You can use this option to explicitly choose which fields to use for y.

The series of the chart are generated from the y-fields.
To make changes to a series in an xy chart, make [overrides](ref:configure-field-overrides) to the y-field.

{{< admonition type=note >}}
Any field you use in the **Size field** or **Color field** doesn't generate a series.
{{< /admonition >}}

You can also use [overrides](ref:configure-field-overrides) to exclude y-fields individually.
To do so, add an override with the following properties for each y-field you want removed:

- Override type: **Fields with name**
- Override property: **Series > Hide in area**
- Area: **Viz**

#### Show

Set how values are represented in the visualization. Choose from:

- **Points** - Display values as points. When you select this option, the **Point size** option is also displayed.
- **Lines** - Add a line between values. When you select this option, the [Line style](#line-style) and **Line width** options are also displayed.
- **Both** - Display both points and lines.

#### Line style

Set the style of the lines that connect points. To change the color, use the standard [Color scheme](ref:color-scheme) field option.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-line-style-options-v11.6.png" max-width="400px" alt="Line style options" >}}

- **Solid** - Display a solid line. This is the default setting.
- **Dash** - Display a dashed line. When you choose this option, a drop-down list is displayed where you can select the length and gap setting for the line dashes. By default, the length and gap are set to `10, 10`.
- **Dots** - Display dotted lines. When you choose this option, a drop-down list is displayed where you can select dot spacing. By default, the dot spacing is set to `0, 10` (the first number represents dot length, which is always zero).

### Tooltip options

Tooltip options control the information overlay that appears when you hover over data points in the visualization.

<!-- prettier-ignore-start -->

| Option       | Description |
| ------------ | ----------- |
| Tooltip mode | When you hover your cursor over the visualization, Grafana can display tooltips. Choose how they behave:<ul><li>**Single** - The hover tooltip shows only a single series, the one that you are hovering over on the visualization.</li><li>**Hidden** - Do not display the tooltip when you interact with the visualization.</li></ul> |
| Max width    | Set the maximum width of the tooltip box. |

<!-- prettier-ignore-end -->

The **All**, **Values sort order**, **Hover proximity**, and **Max height** tooltip options aren't available for XY charts.

### Legend options

{{< docs/shared lookup="visualizations/legend-options-1.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Axis options

{{< docs/shared lookup="visualizations/axis-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Standard options

**Standard options** in the panel editor pane let you change how field data is displayed in your visualizations. When you set a standard option, the change is applied to all fields or series. For more granular control over the display of fields, refer to [Configure field overrides](ref:configure-field-overrides).

You can customize the following standard options:

- **Field min/max** - Enable **Field min/max** to have Grafana calculate the min or max of each field individually, based on the minimum or maximum value of the field.
- **Color scheme** - Set single or multiple colors for your entire visualization. To learn more about color schemes, refer to [Configure standard options](ref:configure-standard-options).

### Data links and actions

{{< docs/shared lookup="visualizations/datalink-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Value mappings

{{< docs/shared lookup="visualizations/value-mappings-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Thresholds

{{< docs/shared lookup="visualizations/thresholds-options-2.md" source="grafana" version="<GRAFANA_VERSION>" >}}

### Field overrides

{{< docs/shared lookup="visualizations/overrides-options.md" source="grafana" version="<GRAFANA_VERSION>" >}}

# Whatsnew

# What's new in Grafana v7.0

This topic includes the release notes for Grafana v7.0. For all details, read the full [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

This major release of Grafana is the next step in our Observability story. It includes powerful new features for manipulating, transforming, and doing math on data. Grafana Enterprise has the first version of Usage analytics, which will help Grafana Admins better manage large, corporate Grafana ecosystems.

The Grafana 7.0 stable release is scheduled for the 18th of May. In the meantime, if you want to know more about what we've been up to and what is coming, sign up for our online GrafanaCon conference.

[{{< figure src="/assets/img/blog/GrafanaCONline.jpg" max-width="800px" lightbox="false" caption="GrafanaCONline May 13-29" >}}](/about/events/grafanacon/2020/?source=blog)

The main highlights are:

- [**New Panel Editor** Redesign based on community feedback.](#new-panel-editor-and-unified-data-model)
- [**Explore** New tracing UI and support for visualizing Jaeger and Zipkin traces.](#new-tracing-ui)
- [**Enterprise** Usage insights, Presence indicator, and Auth improvements.](#grafana-enterprise)
- [**Transformations** Transformations and simple Math operations for all data sources.](#transformations)
- [**Field overrides** Automatically configure panels with data from queries.](#field-options-and-overrides)
- [**Table** New Table panel.](#table-panel)
- [**Plugins** New plugins platform.](#plugins-platform)
- [**Tutorials** New tutorials section.](#new-tutorials)
- [**Cloudwatch** Support for Cloudwatch Logs in Explore and the Logs panel.](#cloudwatch-logs)
- [**Breaking change** PhantomJS removed.](#breaking-change---phantomjs-removed)
- [**Time zones** Time zone support](#time-zone-support)

## New panel editor and unified data model

We have redesigned the UI for editing panels. The first visible change is that we have separated panel display settings to a right-hand side pane that you can collapse or expand depending on what your focus is on. With this change we are also introducing our new unified option model and UI for defining data configuration and display options. This unified data configuration system powers a consistent UI for setting data options across visualizations, as well as making all data display settings data driven and overridable.

This new option architecture and UI will make all panels have a consistent set of options and behaviors for attributes like unit, min, max, thresholds, links, decimals. Not only that but all these options will share a consistent UI for specifying override rules and is extensible for custom panel specific options.

In previous versions of Grafana, each visualization had slightly different ways to define their options. One immediate benefit is after setting options like units or thresholds in a panel, you can seamlessly switch between visualization types and keep those options. This will bring increased ease of use and more consistency for users and plugin developers.

We have yet to migrate all core panels to this new architecture so in 7.0 there will be some inconsistencies in the UI between panels. This will be fixed soon in future releases as we update all the core panels and help the community update the community panel plugins.

Learn more about this feature in [Panel editor](../../panels-visualizations/panel-editor-overview/).

## New tracing UI

This release adds major support for distributed tracing, including a telemetry mode to complement the existing support for metrics and logs. Traces allow you to follow how single requests travel through a distributed system. We are starting with an integrated trace viewer and two new built-in data sources: Jaeger and Zipkin.

You can use the new trace view in Explore either directly to search for a particular trace or you can configure Loki to detect trace IDs in the log lines and link directly to a trace timeline pulled from Jaeger or Zipkin data source.

In the future we will add more workflows and integrations so that correlating between metrics, logs and traces is even easier.

{{< figure src="/static/img/docs/v70/tracing_ui.png" max-width="1024px" caption="Tracing UI" >}}

## Transformations

The data you want to visualize can come from many different places and it is usually not in exactly the right form. Users can now transform non-time series data into tables (e.g., JSON files or even simple lookup tables) in seconds without any customization or additional overhead. They can then combine non-time series data with any other data in Grafana; data from an external database or a panel that already exists in one of their current dashboards.

By chaining a simple set of point and click [transformations](../../panels-visualizations/query-transform-data/transform-data/#transformation-functions), users will be able join, pivot, filter, re-name and do calculations to get the results they need. Perfect for operations across queries or data sources missing essential data transformations.

[Transformations](../../panels-visualizations/query-transform-data/transform-data/) also adds the ability to do math across queries. Lots of data sources do not support this natively, so being able to do it in Grafana is a powerful feature.

For users with large dashboards or with heavy queries, being able to reuse the query result from one panel in another panel can be a huge performance gain for slow queries (e.g log or sql queries). From the data source menu in the query editor, you can choose the `--dashboard--` option and then choose the query result from another panel on the same dashboard.

The [Google Sheets data source](/grafana/plugins/grafana-googlesheets-datasource) that was published a few weeks ago works really well together with the transformations feature.

We are also introducing a new shared data model for both time series and table data that we call [DataFrame](https://grafana.com/developers/plugin-tools/key-concepts/data-frames#data-frames-as-time-series). A DataFrame is like a table with columns but we refer to columns as fields. A time series is a DataFrame with two fields (time & value).

**Transformations shipping in 7.0**

- **Reduce:** Reduce all rows or data points to a single value using a function like max, min, mean or last.
- **Filter by name:** Removes part of the query results using a regex pattern. The pattern can be inclusive or exclusive.
- **Filter data by query** Filter data by query. This is useful if you are sharing the results from a different panel that has many queries and you want to only visualize a subset of that in this panel.
- **Organize fields:** Allows the user to re-order, hide, or rename fields / columns. Useful when data source doesn't allow overrides for visualizing data.
- **Labels to fields:** Groups series by time and returns labels or tags as fields. Useful for showing time series with labels in a table where each label key becomes a separate column.
- **Outer join:** Joins many time series/tables by a field. This can be used to outer join multiple time series on the _time_ field to show many time series in one table.
- **Add field from calculation:** This is a powerful transformation that allows you perform many different types of math operations and add the result as a new field. Can be used to calculate the difference between two series or fields and add the result to a new field. Or multiply one field with another and add the result to a new field.

Learn more about this feature in [Transformations](../../panels-visualizations/query-transform-data/transform-data/#transformation-functions).

## Field options and overrides

With Grafana 7.0 we are introducing a new, unified data configuration system that powers a consistent UI for setting data options across visualizations as well as making all data display settings data driven and overridable. This new option architecture and UI will make all panels have a consistent set of options and behaviors for attributes like `unit`, `min`, `max`, `thresholds`, `links`, `decimals` or `value mappings`. Not only that but all these options will share a consistent UI for specifying override rules and is extensible for custom panel specific options.

Up until now the overrides were available only for Graph and Table panel(via Column Styles), but with 7.0 they work consistently across all visualization types and plugins.

This feature enables even more powerful visualizations and fine grained control over how the data is displayed.

Learn more about this feature in [Field overrides](../../panels-visualizations/configure-overrides/).

## Inspect panels and export data to CSV

{{< figure src="/static/img/docs/v70/panel_edit_export_raw_data.png" max-width="800px" class="docs-image--right" caption="Panel Edit - Export raw data to CSV" >}}

Another new feature of Grafana 7.0 is the panel inspector. Inspect allows you to view the raw data for any Grafana panel as well as export that data to a CSV file. With Panel inspect you will also be able to perform simple raw data transformations like join, view query stats or detailed execution data.

Learn more about this feature in [Panel Inspector](../../panels-visualizations/panel-inspector/#inspect-query-request-and-response-data).

<div class="clearfix"></div>

## Table panel

Grafana 7.0 comes with a new table panel (and deprecates the old one). This new table panel supports horizontal scrolling and column resize. Paired with the new `Organize fields` transformation detailed above you can reorder, hide & rename columns. This new panel also supports new cell display modes, like showing a bar gauge inside a cell.

{{< youtube J29wILRh3QQ >}}
<br />

## Auto grid mode for Stat panel and Gauge

This new 7.0 feature is for the gauge and stat panels. Before, stat and gauge only supported horizontal or vertical stacking: The auto layout mode just selected vertical or horizontal stacking based on the panel dimensions (whatever was highest). But in 7.0 the auto layout for these two panels will allow dynamic grid layouts where Grafana will try to optimize the usage of space and lay out each sub-visualization in a grid.

{{< youtube noq1rLGvsrU >}}
<br />

## Cloudwatch Logs

Grafana 7.0 adds logging support to one of our most popular cloud provider data sources. Autocomplete support for Cloudwatch Logs queries is included for improved productivity. There is support for deep linking to the CloudWatch Logs Insights console for log queries, similar to the deep linking feature for Cloudwatch metrics. Since CloudWatch Logs queries can return time series data, for example through the use of the `stats` command, alerting is supported too.

## Plugins platform

The [platform for plugins](/developers/plugin-tools/) has been completely re-imagined and provides ready-made components and tooling to help both inexperienced and experienced developers get up and running more quickly. The tooling, documentation, and new components will improve plugin quality and reduce long-term maintenance. We are already seeing that a high quality plugin with the Grafana look and feel can be written in much fewer lines of code than previously.

Learn more about developing plugins in the new framework in [Plugin tools](/developers/plugin-tools/).

### Front end plugins platform

In Grafana 7.0 we are maturing our panel and front-end datasource plugins platform.

Plugins can use the same React components that the Grafana team uses to build Grafana. Using these components means the Grafana team will support and improve them continually and make your plugin as polished as the rest of Grafana’s UI. The new [`@grafana/ui` components library](https://developers.grafana.com/ui) is documented with Storybook (visual documentation) and is available on NPM.

The `@grafana/data`, `@grafana/runtime`, `@grafana/e2e packages` (also available via NPM) aim to simplify the way plugins are developed. We want to deliver a set of [reliable APIs](/docs/grafana/v7.0/packages_api/) for plugin developers.

With [@grafana/toolkit](https://www.npmjs.com/package/@grafana/toolkit) we are delivering a simple CLI that helps plugin authors quickly scaffold, develop and test their plugins without worrying about configuration details. A plugin author no longer needs to be a grunt or webpack expert to build their plugin.

### Support for backend plugins

Grafana now officially supports [backend plugins](https://grafana.com/developers/plugin-tools/key-concepts/backend-plugins/) and the first type of plugin to be introduced is a backend component for data source plugins. You can optionally add a backend component to your data source plugin and implement the query logic there to automatically enable alerting in Grafana for your plugin. In the 7.0 release, we introduce the [Grafana Plugin SDK for Go](https://grafana.com/developers/plugin-tools/key-concepts/backend-plugins/grafana-plugin-sdk-for-go) that enables and simplifies building a backend plugin in [Go](https://golang.org/).

Plugins can be monitored with the new metrics and health check capabilities. The new Resources capability means backend components can return non-time series data like JSON or static resources like images and opens up Grafana for new use cases.

With this release, we are deprecating the unofficial first version of backend plugins which will be removed in a future release.

To learn more, start with the [overview](https://grafana.com/developers/plugin-tools/key-concepts/backend-plugins/). Next, in this [tutorial](https://grafana.com/developers/plugin-tools/tutorials/build-a-data-source-backend-plugin) you'll learn how to build a backend for a data source plugin and enable it for use with [Grafana Alerting](../../alerting/). Make sure to keep an eye out for additional documentation and tutorials that will be published after the Grafana v7.0 release.

## New tutorials

To help you get started with Grafana, we’ve launched a brand new tutorials platform. We’ll continue to expand the platform with more tutorials, but here are some of the ones you can try out now:

- [Grafana fundamentals](/tutorials/grafana-fundamentals/)
- [Create users and teams](/tutorials/create-users-and-teams/)
- [Build a panel plugin](/tutorials/build-a-panel-plugin/)
- [Build a data source plugin](/tutorials/build-a-data-source-plugin/)

## Rollup indicator for Metrictank queries

{{< figure src="/static/img/docs/v70/metrictank_rollup_metadata.png" max-width="800px" class="docs-image--right" caption="Metrictank rollup metadata" >}}

Depending on the cardinality of the data and the time range MetricTank may return rolled up (aggregated) data. This can be as subtle as potentially only 1 or 2 graphs out of nine being rolled up. The new rollup indicator is visible in the panel title and you can also inspect extensive metadata and stats about the Metrictank query result and its rollups.

<div class="clearfix"></div>

## Breaking change - PhantomJS removed

[PhantomJS](https://phantomjs.org/), have been used for rendering images of dashboards and panels and have been included with Grafana since Grafana v2.0. Since then we’ve had a lot of related bugs and security related issues, mainly due to the fact that PhantomJS have struggled with supporting modern web technologies. Throughout the years, maintaining PhantomJS support in Grafana has been a nightmare. Removing support for PhantomJS has been a high priority for the Grafana project and got stressed even more when the PhantomJS maintainer in March 2018 [announced](https://github.com/ariya/phantomjs/issues/15344) the end of the project.

Since then we have been working towards removing PhantomJS. In October 2019, when Grafana v6.4 was released, we [announced](/blog/2019/10/02/grafana-v6.4-released/#phantomjs-deprecation) the deprecation of PhantomJS. Grafana v7.0 removes all PhantomJS support which means that Grafana distribution no longer will include a built-in image renderer.

As a replacement for PhantomJS we’ve developed the [Grafana Image Renderer plugin](/grafana/plugins/grafana-image-renderer) which is a plugin that runs on the backend and handles rendering panels and dashboards as PNG images using headless Chromium/Chrome. The [Grafana Image Renderer plugin](/grafana/plugins/grafana-image-renderer) can either be installed as a Grafana plugin running in its own process side-by-side with Grafana. or runs as an external HTTP service, hosted using Docker or as a standalone application.

Read more about [Image Rendering](../../setup-grafana/image-rendering/) in the documentation for further instructions.

## Query history in Explore out of beta

The Query history feature lets you view and interact with the queries that you have previously run in Explore. You can add queries to the Explore query editor, write comments, create and share URL links, star your favorite queries, and much more. Starred queries are displayed in the Starred tab, so it is easier to reuse queries that you run often without typing them from scratch.

It was released as a beta feature in Grafana 6.7. The feedback has been really positive and it is now out of beta for the 7.0 release. Learn more about query history in [Explore](../../explore/).

## Stackdriver data source supports Service Monitoring

[Service monitoring](https://cloud.google.com/service-monitoring) in Google Cloud Platform (GCP) enables you to monitor based on Service Level Objectives (SLOs) for your GCP services. The new SLO query builder in the Stackdriver data source allows you to display SLO data in Grafana. Read more about it in the [Stackdriver data source documentation](../../datasources/google-cloud-monitoring/#slo-service-level-objective-queries).

## Time zone support

You can now override the [time zone](../../dashboards/manage-dashboards/#dashboard-time-settings) used to display date and time values in a dashboard. One benefit of this is that you can specify the local time zone of the service or system that you are monitoring which can be helpful when monitoring a system or service that operates across several time zones.

## Alerting and deep linking for Azure Log Analytics

The Azure Monitor data source supports multiple Azure services. Log Analytics queries in the data source now have alerting support too (Azure Monitor and Application Insights already had alerting support).

A new feature is [deep linking from the graph panel to the Log Analytics query editor in the Azure Portal](../../datasources/azure-monitor/#deep-link-from-grafana-panels-to-the-log-analytics-query-editor-in-azure-portal). Click on a time series in the panel to see a context menu with a link to View in Azure Portal. Clicking that link opens the Azure Log Analytics query editor in the Azure Portal and runs the query from the Grafana panel.

## Grafana Enterprise

Grafana Enterprise focuses on solving problems for large companies and Grafana installations. And in Grafana 7.0 we are finally
solving one of the most common problems of using Grafana at scale.

This includes problems like:

- There are too many dashboards, how do I find the right one?
- How to find popular dashboards
- How to find dashboards with errors
- How to identify dashboards that are not being used
- Who created or last viewed this dashboard?

{{< figure src="/static/img/docs/v70/dashboard_insights_users.png" max-width="1024px" caption="Dashboard Insights Users" >}}

### Usage insights and Presence indicator

This release includes a series of features that build on our new usage analytics engine. This “Grafana about Grafana” feature will help our large customers get better insight into the behavior and utilization of their users, dashboards, and data sources. The improved [dashboard search](../../dashboards/assess-dashboard-usage/#sort-dashboards-by-using-insights-data) allows you to sort dashboards by usage and errors. When a user opens a dashboard, they will see a [presence indicator](../../dashboards/assess-dashboard-usage/#presence-indicator) of who else is viewing the same dashboard. And finally [Dashboard insights](../../dashboards/assess-dashboard-usage/#dashboard-insights) allows you to view recent dashboard usage.

{{< figure src="/static/img/docs/v70/presence_indicator.jpg" max-width="1024px" caption="Grafana Enterprise - Presence indicator" >}}

### SAML Role and Team Sync

SAML support in Grafana Enterprise is improved by adding Role and Team Sync. Read more about how to use these features in the [SAML team sync documentation](../../setup-grafana/configure-access/configure-authentication/saml/#configure-team-sync).

### Okta OAuth Team Sync

Okta gets its own provider which adds support for Team Sync. Read more about it in the [Okta documentation](../../setup-grafana/configure-access/configure-authentication/okta/).

## Changelog

Check out [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md) for a complete list of new features, changes, and bug fixes.

# What's new in Grafana v7.1

This topic includes the release notes for the Grafana v7.1. For all details, read the full [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

The main highlights are:

- [**Flux and InfluxDB 2.x support in the Influx Datasource**](#influx-data-source)
- [**Query history search**](#query-history-search)
- [**Unification of Explore modes**](#explore-modes-unified)
- [**Elasticsearch- link to another data source from Explore**](#internal-links-for-elasticsearch)
- [**Merge on time transform for the new table panel**](#transformations)
- [**Stat panel text mode**](#stat-panel-text-mode)
- [**Time range picker update**](#time-range-picker-update)
- [**Provisioning of apps**](#provisioning-of-apps)
- [**Azure Monitor Datasource**](#azure-monitor-data-source)
- [**Deep linking for Google Cloud Monitoring (formerly named Google Stackdriver) datasource**](#deep-linking-for-google-cloud-monitoring-formerly-named-google-stackdriver-data-source)
- [**Grafana Enterprise features**](#grafana-enterprise-features)
  - [**Secret management with HashiCorp Vault**](#support-for-hashicorp-vault)
  - [**Monthly schedules in reports**](#support-for-monthly-schedules-in-reports)

## Influx data source

Support for Flux and Influx v2 has been added. The InfluxData blog post, [How to Build Grafana Dashboards with InfluxDB, Flux and InfluxQL](https://www.influxdata.com/blog/how-grafana-dashboard-influxdb-flux-influxql/) explains the changes in depth.

## Query history search

In Grafana v 7.1 we are introducing search functionality in Query history. You can search across queries and your comments. It is especially useful in combination with a time filter and data source filter. Read more about [Query history here](../../explore/#query-history).

{{< figure src="/static/img/docs/v71/query_history_search.gif" max-width="800px" caption="Query history search" >}}

## Explore modes unified

Grafana 7.1 includes a major change to Explore: it removes the query mode selector.

Many data sources tell Grafana whether a response contains time series data or logs data. Using this information, Explore chooses which visualization to use for that data. This means that you don't need to switch back and forth between Logs and Metrics modes depending on the type of query that you want to make.

## Internal links for Elasticsearch

The new internal linking feature for Elasticsearch allows you to link to other data sources from your logs. You can now create links in Elastic configuration that point to another data source (similar to an existing feature in Loki). An example would be using a traceID field from your logs to be able to link to traces in a tracing data source like Jaeger.

## Transformations

We have added a new **Merge on time** transform that can combine many time series or table results. Unlike the join transform, this combines the result into one table even when the time values do not align / match.

The new table panel introduced in 7.0 was missing a few features that the old table panel had. This feature, along with ad hoc filtering, means that the new table panel has achieved feature parity with the old table panel.

## Ad hoc filtering in the new table panel

[Ad hoc filtering](../../dashboards/variables/add-template-variables/#add-ad-hoc-filters), a way to automatically add filters to queries without having to define template variables is now supported in the new Table panel.

## Stat panel text mode

The [stat panel](../../panels-visualizations/visualizations/stat/#text-mode) has a new **Text mode** option to control what text to show.

By default, the Stat panel displays:

- Just the value for a single series or field.
- Both the value and name for multiple series or fields.

You can use the Text mode option to control what text the panel renders. If the value is not important, only name and color is, then change the `Text mode` to **Name**. The value will still be used to determine color and is displayed in a tooltip.

{{< figure src="/static/img/docs/v71/stat-panel-text-modes.png" max-width="1025px" caption="Stat panel" >}}

## Provisioning of apps

Grafana v7.1 adds support for provisioning of app plugins. This allows app plugins to be configured and enabled/disabled using configuration files. For more information about provisioning of app, refer to [provisioning plugin](../../administration/provisioning/#plugins).

## Azure Monitor data source

Support for multiple dimensions has been added to all services in the Azure Monitor datasource. This means you can now group by more than one dimension with time series queries. With the Kusto based services, Log Analytics and Application Insights Analytics, you can also select multiple metrics as well as multiple dimensions.

Additionally, the Raw Edit mode for Application Insights Analytics has been replaced with a new service in the drop-down for the data source and is called Insights Analytics. The new query editor behaves in the same way as Log Analytics.

## Deep linking for Google Cloud Monitoring (formerly named Google Stackdriver) data source

A new feature in Grafana 7.1 is [deep linking from Grafana panels to the Metrics Explorer in Google Cloud Console](../../datasources/google-cloud-monitoring/#deep-linking-from-grafana-panels-to-the-metrics-explorer-in-google-cloud-console). Click on a time series in the panel to see a context menu with a link to View in Metrics explorer in Google Cloud Console. Clicking that link opens the Metrics explorer in the Monitoring Google Cloud Console and runs the query from the Grafana panel there.

## Time range picker update

With 7.1 we are updating the dashboard's time range picker to allow time zone selection. You no longer need to go to dashboard settings to change the dashboard's time zone.

The time zone picker itself also got UX improvements. Now you can search for the timezone using country or city name, time zone abbreviations, or UTC offsets.

## Grafana Enterprise features

General features are included in the Grafana Enterprise edition software.

### Support for HashiCorp Vault

You can now use HashiCorp Vault to get secrets for configuration and provisioning of Grafana Enterprise. For more information about HashiCorp Vault, refer to [vault](../../setup-grafana/configure-security/configure-database-encryption/integrate-with-hashicorp-vault/).

### Support for monthly schedules in reports

With Grafana Enterprise 7.1, you can generate reports on a [monthly schedule](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/create-reports/#2-schedule).

## Changelog

Check out [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md) for a complete list of new features, changes, and bug fixes.

# What's new in Grafana v7.2

This topic includes the release notes for the Grafana v7.2. For all details, read the full [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

The main highlights are:

- [**New date formatting options added**](#new=date-formatting-options-added)
- [**Field options are out of beta!**](#field-options-are-out-of-beta)
  - [**Added table column filters**](#added-table-column-filters)
  - [**New field override selection options**](#new-field-override-selection-options)
- [**New transformations and enhancements**](#new-transformations-and-enhancements)
- [**Drag to reorder queries**](#drag-to-reorder-queries)
- [**Inspect queries in Explore**](#inspect-queries-in-explore)
- [**$\_\_rate_interval for Prometheus**](#__rate_interval-for-prometheus)
- [**Toggle parsed log fields**](#toggle-parsed-log-fields)
- [**Sensitive alert channel settings are now encrypted**](#sensitive-alert-channel-settings-are-now-encrypted)
- [**Grafana Enterprise features**](#grafana-enterprise-features)
  - [**Report time range**](#report-time-range)
  - [**Organization-wide report settings**](#organization-wide-report-settings)
  - [**Report grid layout**](#report-grid-layout)
- [**What's new in other parts of the Grafana ecosystem**](#whats-new-in-other-parts-of-the-grafana-ecosystem)
  - [**ADX (Azure Data Explorer) plugin**](#adx-azure-data-explorer-plugin)
  - [**X-Ray data source plugin**](#x-ray-data-source-plugin)

## New date formatting options added

You can now customize how dates are formatted in Grafana. Custom date formats apply to the time range picker, graphs, and other panel visualizations.

This screenshot shows both a custom full date format with a 12 hour clock and am/pm suffix. The graph is also showing the same 12-hour clock format and a customized month and day format compared to the Grafana default `MM/DD` format.

{{< figure src="/static/img/docs/v72/date_formats.png" max-width="800px" caption="Custom date time formats" >}}

Date formats are set for a Grafana instance by adjusting [server-wide settings](../../setup-grafana/configure-grafana/#date_formats) in the Grafana configuration file. We hope to add org- and user-level settings in the future.

```
[date_formats]
full_date = MMM Do, YYYY @ hh:mm:ss a
interval_second = hh:mm:ss a
interval_minute = hh:mm a
interval_hour = MMM DD hh:mm a
interval_day = MMM DD
interval_month = YYYY-MM
interval_year = YYYY
```

There is also experimental support to use the browser location and language to dynamically change the current date format for each user. This feature is disabled by default.

The [Configuration](../../setup-grafana/configure-grafana/#date_formats) topic has been updated as a result of this change.

## Field options are out of beta!

After lots of testing and user feedback, we removed the beta label from the configuration options in the Field and Override tabs. This release also includes the following feature enhancements.

### Added table column filters

You can now dynamically apply value filters to any table column. This option can be enabled for all columns or one specific column using an override rule.

{{< figure src="/static/img/docs/v72/table_column_filters.png" max-width="800px" caption="Table column filters" >}}

[Filter table columns](../../panels-visualizations/visualizations/table/#filter-table-columns) has been added as a result of this feature.

### New field override selection options

You can now add override rules that use a regex matcher to choose which fields to apply rules to.

## New transformations and enhancements

Grafana 7.2 includes the following transformation enhancements:

- A new [Group By](../../panels-visualizations/query-transform-data/transform-data/#group-by) transformation that allows you to group by multiple fields and add any number of aggregations for other fields.
- The [Labels to fields](../../panels-visualizations/query-transform-data/transform-data/#labels-to-fields) transformation now allows you to pick one label and use that as the name of the value field.
- You can drag transformations to reorder them. Remember that transformations are processed in the order they are listed in the UI, so think before you move something!

{{< figure src="/static/img/docs/v72/transformations.gif" max-width="800px" caption="Group by and reordering of transformations" >}}

## Drag to reorder queries

The up and down arrows, which were previously the only way to change query order, have been removed. Instead, there is now a grab icon that allows you to drag and drop queries in a list to change their order.

{{< figure src="/static/img/docs/v72/drag-queries.gif" max-width="800px" caption="Drag to reorder queries" >}}

The [Queries](../../panels-visualizations/#add-a-query) topic has been updated as a result of this change.

## Inspect queries in Explore

You can enjoy all the details query inspector gave you in dashboards now in Explore as well. You can open query inspector tab with the button next to query history. See [Query inspector in Explore](../../explore/#query-inspector) for more details.

## \$\_\_rate_interval for Prometheus

You can now use the new variable `$__rate_interval` in Prometheus for rate functions mainly. `$__rate_interval` in general is one scrape interval larger than `$__interval` but is never smaller than four times the scrape interval (which is 15s by default). See the [Prometheus data source](../../datasources/prometheus/#using-__rate_interval-variable) for more details.

## Toggle parsed log fields

With this awesome contribution from one of our community members, you can now toggle parsed fields in Explore if your logs are structured in `json` or `logfmt`.

{{< figure src="/static/img/docs/v72/explore-toggle-parsed-fields.gif" max-width="800px" caption="Toggling parsed fields in Explore" >}}

The [Toggle parsed fields](../../explore/#toggle-detected-fields) section has been added to [Explore](../../explore/) as a result of this feature.

## Sensitive alert channel settings are now encrypted

Alert notification channels now store sensitive settings and secrets, such as API tokens and passwords, encrypted in the database.

## Grafana Enterprise features

These features are included in the Grafana Enterprise edition software.

### Report and export dashboards in grid layout

A new layout option is available when rendering reports: the grid layout. With this option, your report uses the panel layout from your dashboard, so that what you see is what you get. Learn more about the [grid layout](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/create-reports/#5-attachments) in the documentation.

The grid layout is also available for the [Export dashboard as PDF](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/share-dashboards-panels/#export-a-dashboard-as-pdf) feature.

{{< figure src="/static/img/docs/enterprise/reports_grid_landscape_preview.png" max-width="500px" class="docs-image--no-shadow" alt="Report in grid layout" >}}

### Report time range

You can now generate a report with a different time range from the dashboard it is based on. This means that you no longer have to apply workarounds, such as copying dashboards or carefully aligning report generation with the end of the month, to generate reports that cover the period you want.

For more information, refer to [Reports time range](../../dashboards/share-dashboards-panels/#report-time-range).

### Organization-wide report settings

You can now configure organization-wide report settings, such as report branding, in the Settings tab on the Reporting page. Settings are applied to all the reports of your current organization.

{{< figure src="/static/img/docs/enterprise/reports_settings.png" max-width="500px" class="docs-image--no-shadow" caption="Reports settings" >}}

For more information, refer to [Report settings](../../dashboards/share-dashboards-panels/#report-settings).

## Changelog

Check out [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md) for a complete list of new features, changes, and bug fixes.

## What's new in other parts of the Grafana ecosystem

### ADX (Azure Data Explorer) plugin

In collaboration with Microsoft, we have improved the usability of our ADX datasource plugin by adding a visual query builder. The goal is to make it easier for users, regardless of their previous knowledge of writing KQL (Kusto Query Language) queries, to query and visualize their data.

{{< figure src="/static/img/docs/v72/adx-ds.png" max-width="800px" caption="ADX visual query builder" >}}

### X-Ray data source plugin

We are pleased to announce our very first version of our data source plugin for AWS X-Ray. You can use this plugin to visualize traces, look at analytics tables, and see insight summaries. For more information, refer to the [X-Ray data source](/grafana/plugins/grafana-x-ray-datasource) plugin page.

# What's new in Grafana v7.3

This topic includes the release notes for Grafana v7.3. For all details, read the full [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md) or the [Patch release notes](#patch-release-notes).

The main highlights are:

- [**Google Cloud Monitoring:** Out of the box dashboards](#cloud-monitoring-out-of-the-box-dashboards)
- [**Shorten URL for dashboards and Explore**](#shorten-url-for-dashboards-and-explore)
- [**Table improvements and new image cell mode**](#table-improvements-and-new-image-cell-mode)
- [**New color scheme option**](#new-color-scheme-option)
- [**SigV4 Authentication for Amazon Elasticsearch Service**](#sigv4-authentication-for-aws-users)
- [**CSV exports for Excel**](#csv-exports-for-excel)

## Table improvements and new image cell mode

The table has been updated with improved hover behavior for cells that have longer content than what fits the current column width. As you can see
in the animated gif below the cell will automatically expand to show you full content of the cell.

{{< figure src="/static/img/docs/v73/table_hover.gif" max-width="900px" caption="Table hover" >}}

Another new feature that can be seen in the image above is the new image cell display mode. If you have a field value that is an image URL or a base64 encoded image you can configure the table to display it as an image.

## New color scheme option

{{< figure src="/static/img/docs/v73/color_scheme_dropdown.png" max-width="450px" caption="Color scheme" class="pull-right" >}}

A new standard field color scheme option has been added. This new option will provide a unified way for all new panels to specify how colors should be assigned. For more information, refer to [Apply color to series and fields](../../panels-visualizations/configure-standard-options/#color-scheme).

- **Single color**: Specifies a single color. Useful in an override rule.
- **From thresholds**: Informs Grafana to take color from the matching threshold.
- **Classic palette**: Assigns a color by looking up a color in a palette by series index. Useful for Graphs and pie charts, and other categorical data visualizations in Grafana.
- **Green-Yellow-Red (by value)**: A continuous color scheme where Grafana will interpolate a color based on the value assigned to the green, yellow, and red components. The value must be within the min & max limits.
- **Blue-Yellow-Red (by value)**: Same as above but different colors.
- **Blues (by value)**: Same as above but color scheme go from panel background to blue.

<div class="clearfix"></div>

As you can see this adds new continuous color schemes where Grafana will interpolate colors. A great use of these new color schemes is the table panel where you can color the background and get a heatmap like effect.

{{< figure src="/static/img/docs/v73/table_color_scheme.png" max-width="900px" caption="table color scheme" >}}

Another thing to highlight is that all these new color schemes are theme aware and adapt to the current theme. For example here is how the new monochrome color scheme look like in the light theme:

{{< figure src="/static/img/docs/v73/table_color_scheme_mono_light.png" max-width="900px" caption="table color monochrome scheme" >}}

As this new option is a standard field option it works in every panel. Here is another example from the [Bar Gauge](../../panels-visualizations/visualizations/bar-gauge/) panel.

{{< figure src="/static/img/docs/v73/bar_gauge_gradient_color_scheme.png" max-width="900px" caption="bar gauge color scheme" >}}

## CSV exports for Excel

In v7.0, we introduced a new table panel and inspect mode with Download CSV enabled. However, CSV export to Excel was removed. Due to a large number of inquiries and requests, this [community contribution from tomdaly](https://github.com/grafana/grafana/pull/27284) brought the feature back.

For more information, refer to [Download raw query results](../../panels-visualizations/panel-inspector/#download-raw-query-results).

## Google Cloud monitoring out-of-the-box dashboards

The updated Google Cloud monitoring data source is shipped with pre-configured dashboards for five of the most popular Google Cloud Platform (GCP) services:

- BigQuery
- Cloud Load Balancing
- Cloud SQL
- Google Compute Engine `GCE`
- Google Kubernetes Engine `GKE`

To import the pre-configured dashboards, go to the configuration page of your Google Cloud Monitoring data source and click on the `Dashboards` tab. Click `Import` for the dashboard you would like to use. To customize the dashboard, we recommend to save the dashboard under a different name, because otherwise the dashboard will be overwritten when a new version of the dashboard is released.

For more details, see the [Google Cloud Monitoring docs](../../datasources/google-cloud-monitoring/#out-of-the-box-dashboards)

## Shorten URL for dashboards and Explore

This is an amazing new feature that was created in cooperation with one of our community members. The new share shortened link capability allows you to create smaller and simpler URLs of the format `/goto/:uid` instead of using longer URLs that can contain complex query parameters. In Explore, you can create a shortened link by clicking on the share button in Explore toolbar. In the dashboards, a shortened url option is available through the share panel or dashboard button.

## SigV4 authentication for AWS users

You can now configure your Elasticsearch data source to access your Amazon Elasticsearch Service domain directly from Grafana.

For more details, refer to the [Elasticsearch docs](../../datasources/elasticsearch/#aws-signature-version-4-authentication).

## Chaining pipeline aggregation in Elasticsearch

Thanks to a contribution from a community member, it's now possible to chain multiple pipeline aggregations together and use the results of one pipeline aggregation as the input of another. This unleashes the full power of Elasticsearch's pipeline aggregations in Grafana, allowing users to perform high order derivatives or use a pipeline aggregation result as a variable for a Bucket Script Aggregation.

## Grafana Enterprise features

These features are included in the Grafana Enterprise edition software.

### Auditing

Auditing tracks important changes to your Grafana instance to help you manage and mitigate suspicious activity and meet compliance requirements. Grafana logs events (as JSON) to file or directly to [loki](/oss/loki/).

Example of a login event:

```json
{
  "timestamp": "2020-10-22T10:18:00.838094347Z",
  "user": {
    "userId": 1,
    "orgId": 1,
    "isAnonymous": false
  },
  "action": "login-grafana",
  "result": {
    "statusType": "success",
    "statusCode": 200
  },
  "requestUri": "/login",
  "ipAddress": "127.0.0.1:41324",
  "userAgent": "Chrome/86.0.4240.111",
  "grafanaVersion": "7.3.0"
}
```

For more details, see the [Auditing docs](../../setup-grafana/configure-security/audit-grafana/).

### Data source usage insights

Data source usage insights allows you to gain insight into how a data source is being used and how well it works. There is a new tab in the data source settings page called insights that will show you information about how the data source has been used in the past 30 days.

Insights:

- Queries per day
- Errors per day
- Average load duration per day (ms)

### SAML single logout

SAML’s single logout (SLO) capability allows users to log out from all applications associated with the current identity provider (IdP) session established via SAML SSO. For more information, refer to the [docs](../../setup-grafana/configure-access/configure-authentication/saml/#single-logout).

### SAML IdP-initiated single sign on

IdP-initiated single sign on (SSO) allows the user to log in directly from the SAML identity provider (IdP). It is disabled by default for security reasons. For more information, refer to the [docs](../../setup-grafana/configure-access/configure-authentication/saml/#idp-initiated-single-sign-on-sso).

## Changelog

Check out [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md) for a complete list of new features, changes, and bug fixes.

## Patch release notes

- [Grafana 7.3.0 release notes](../../release-notes/release-notes-7-3-0/)
- [Grafana 7.3.1 release notes](../../release-notes/release-notes-7-3-1/)
- [Grafana 7.3.2 release notes](../../release-notes/release-notes-7-3-2/)
- [Grafana 7.3.3 release notes](../../release-notes/release-notes-7-3-3/)
- [Grafana 7.3.4 release notes](../../release-notes/release-notes-7-3-4/)
- [Grafana 7.3.5 release notes](../../release-notes/release-notes-7-3-5/)
- [Grafana 7.3.6 release notes](../../release-notes/release-notes-7-3-6/)
- [Grafana 7.3.7 release notes](../../release-notes/release-notes-7-3-7/)

# What's new in Grafana v7.4

This topic includes the release notes for Grafana v7.4. For all details, read the full [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

Check out the [New Features in 7.4](https://play.grafana.org/d/nP8rcffGk/1-new-features-in-v7-4?orgId=1) dashboard on Grafana Play!

## Grafana OSS features

These features are included in the Grafana open source edition.

### Time series panel visualization (Beta)

Grafana 7.4 adds a beta version of the next-gen graph visualization. The new graph panel, the _Time series_ visualization, is high-performance visualization based on the uPlot library. This new graph visualization uses the new panel architecture introduced in Grafana 7.0 and integrates with field options, overrides, and transformations.

The Time series beta panel implements the majority of the functionalities available in the current Graph panel. Our plan is to have close to full coverage of the features in Grafana 8.0, coming later this year.

Apart from major performance improvements, the new Time series panel implements new features like line interpolation modes, support for more than two Y-axes, soft min and max axis limits, automatic points display based on data density, and gradient fill modes.

{{< figure src="/static/img/docs/v74/timeseries_panel.png" max-width="900px" caption="Time series panel" >}}

The following documentation topics were added for this feature:

- [Time series panel](../../panels-visualizations/visualizations/time-series/)
- [Graph time series as lines](../../panels-visualizations/visualizations/time-series/#line-style)
- [Graph time series as bars](../../panels-visualizations/visualizations/time-series/#bar-alignment)
- [Graph time series as points](../../panels-visualizations/visualizations/time-series/#show-points)
- [Change axis display](../../panels-visualizations/visualizations/time-series/#axis-options)

### Node graph panel visualization (Beta)

_Node graph_ is a new panel type that can visualize directed graphs or network in dashboards, but also in Explore. It uses directed force layout to effectively position the nodes so it can help with displaying complex infrastructure maps, hierarchies, or execution diagrams.

All the information and stats shown in the Node graph beta are driven by the data provided in the response from the data source. The first data source that is using this panel is AWS X-Ray, for displaying their service map data.

For more details about how to use the X-Ray service map feature, see the [X-Ray plugin documentation](/grafana/plugins/grafana-x-ray-datasource).

For more information, refer to [Node graph panel](../../panels-visualizations/visualizations/node-graph/).

### New transformations

The following transformations were added in Grafana 7.4.

#### Sort by transformation

The _Sort by_ transformation allows you to sort data before sending it to the visualization.

For more information, refer to [Sort by](../../panels-visualizations/query-transform-data/transform-data/#sort-by).

#### Filter data by value transform

The new _Filter data by value_ transformation allows you to filter your data directly in Grafana and remove some data points from your query result.

This transformation is very useful if your data source does not natively filter by values. You might also use this to narrow values to display if you are using a shared query.

For more information, refer to [Filter data by value](../../panels-visualizations/query-transform-data/transform-data/#filter-data-by-value).

### New override option

On the Overrides tab, you can now set properties for fields returned by a specific query.

For more information, refer to [About field overrides](../../panels-visualizations/configure-overrides/).

### Exemplar support

Grafana graphs now support Prometheus _exemplars_. They are displayed as diamonds in the graph visualization.

> **Note:** Support for exemplars will be added in version Prometheus 2.25+.

{{< figure src="/static/img/docs/v74/exemplars.png" max-width="900px" caption="Exemplar example" >}}

For more information, refer to [Exemplars](../../fundamentals/exemplars/).

### Trace to logs

You can now navigate from a span in a trace view directly to logs relevant for that span. This feature is available for the Tempo, Jaeger, and Zipkin data sources.

The following topics were updated as a result of this feature:

- [Explore](../../explore/trace-integration/)
- [Jaeger](../../datasources/jaeger/#configure-trace-to-logs)
- [Tempo](../../datasources/tempo/#configure-trace-to-logs)
- [Zipkin](../../datasources/zipkin/#configure-trace-to-logs)

### Server-side expressions

_Server-side expressions_ is an experimental feature that allows you to manipulate data returned from backend data source queries. Expressions allow you to manipulate data with math and other operations when the data source is a backend data source or a **--Mixed--** data source.

The main use case is for [multi-dimensional](../../fundamentals/timeseries-dimensions/) data sources used with the upcoming next generation alerting, but expressions can be used with backend data sources and visualization as well.

> **Note:** Queries built with this feature might break with minor version upgrades until Grafana 8 is released. This feature does not work with the current Grafana Alerting.

For more information, refer to [Write expression queries](../../panels-visualizations/query-transform-data/expression-queries/). [About queries](../../panels-visualizations/query-transform-data/#about-queries) was also updated as a result of this feature.

### Alert notification query label interpolation

You can now provide detailed information to alert notification recipients by injecting alert label data as template variables into an alert notification. Labels that exist from the evaluation of the alert query can be used in the alert rule name and in the alert notification message fields using the `${Label}` syntax. The alert label data is automatically injected into the notification fields when the alert is in the alerting state. When there are multiple unique values for the same label, the values are comma-separated.

{{< figure src="/static/img/docs/alerting/alert-notification-template-7-4.png" max-width="700px" caption="Variable support in alert notifications" >}}

### Content security policy support

We have added support for [Content Security Policy (CSP)](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP), a layer of security that helps detect and mitigate certain types of attacks, including Cross Site Scripting (XSS) and data injection attacks.

CSP support is disabled by default, to enable it you must set `content_security_policy = true` in the Grafana configuration. If enabling it, you should also review, and potentially tweak, the CSP header template, via the configuration setting `content_security_policy_template`.

You can lock down what can be done in the frontend code. Lock down what can be loaded, what JavaScript is executed. Not compatible with some plugins.

[content_security_policy](../../setup-grafana/configure-grafana/#content_security_policy) and [content_security_policy_template](../../setup-grafana/configure-grafana/#content_security_policy_template) were added to [Configuration](../../setup-grafana/configure-grafana/) as a result of this change.

### Hide users in UI

You can now use the `hidden_users` configuration setting to hide specific users in the UI. For example, this feature can be used to hide users that are used for automation purposes.

[Configuration](../../setup-grafana/configure-grafana/#hidden_users) has been updated for this feature.

### Elasticsearch data source updates

Grafana 7.4 includes the following enhancements

- Added support for serial differencing pipeline aggregation.
- Added support for moving function pipeline aggregation.
- Added support to the terms aggregation for ordering by percentiles and extended stats.
- Updated date histogram auto interval handling for alert queries.

> **Note:** We have deprecated browser access mode. It will be removed in a future release.

For more information, refer to the [Elasticsearch docs](../../datasources/elasticsearch/).

### Azure Monitor updates

The Azure Monitor query type was renamed to Metrics and Azure Logs Analytics was renamed to Logs to match the service names in Azure and align the concepts with the rest of Grafana.

[Azure Monitor](../../datasources/azure-monitor/) was updated to reflect this change.

### MQL support added for Google Cloud Monitoring

You can now use Monitoring Query Language (MQL) for querying time-series data. MQL provides an expressive, text-based interface to retrieve, filter, and manipulate time-series data.

Unlike the visual query builder, MQL allows you to control the time range and period of output data, create new labels to aggregate data, compute the ratio of current values to past values, and so on.

MQL uses a set of operations and functions. Operations are linked together using the common pipe mechanism, where the output of one operation becomes the input to the next. Linking operations makes it possible to build up complex queries incrementally.

Once query type Metrics is selected in the Cloud Monitoring query editor, you can toggle between the editor modes for visual query builder and MQL. For more information, refer to the [Google Cloud Monitoring docs](../../datasources/google-cloud-monitoring/#import-pre-configured-dashboards).

Many thanks to [mtanda](https://github.com/mtanda) this contribution!

## Curated dashboards for Google Cloud Monitoring

Google Cloud Monitoring data source ships with pre-configured dashboards for some of the most popular GCP services. These curated dashboards are based on similar dashboards in the GCP dashboard samples repository. In this release, we have expanded the set of pre-configured dashboards.

{{< figure src="/static/img/docs/google-cloud-monitoring/curated-dashboards-7-4.png" max-width= "650px" alt="Google Cloud Monitoring pre-configured dashboards" >}}

If you want to customize a dashboard, we recommend that you save it under a different name. Otherwise the dashboard will be overwritten when a new version of the dashboard is released.

For more information, refer to the [Google Cloud Monitoring docs](../../datasources/google-cloud-monitoring/#import-pre-configured-dashboards).

### Query Editor Help

The feature previously referred to as DataSource Start Pages or Cheat Sheets has been renamed to Query Editor Help, and is now supported in panel query editors (depending on the data source), as well as in Explore.

[Queries](../../panels-visualizations/query-transform-data/#manage-queries) was updated as a result of this feature.

For more information on adding a query editor help component to your plugin, refer to [Add a query editor help component](https://grafana.com/developers/plugin-tools/how-to-guides/data-source-plugins/add-query-editor-help).

### Variable inspector

The variables list has an additional column indicating whether variables are referenced in queries and panel names or not. The dependencies graph provides an easy way to check variable dependencies. You can click on a variable name within the graph to make updates to the variable as needed.

For more information, refer to [Inspect variables and their dependencies](../../dashboards/variables/inspect-variable/).

## Grafana Enterprise features

These features are included in the Grafana Enterprise edition.

### Licensing changes

When determining a user’s role for billing purposes, a user who has the ability to edit and save dashboards is considered an Editor. This includes any user who is an Editor or Admin at the Org level, and who has granted Admin or Edit permissions via [Dashboard permissions](../../administration/user-management/manage-dashboard-permissions/).

After the number of Viewers or Editors has reached its license limit, only Admins will see a banner in Grafana indicating that the license limit has been reached. Previously, all users saw the banner.

Grafana Enterprise license tokens update automatically on a daily basis, which means you no longer need to manually update your license, and the process for adding additional users to a license is smoother than it was before.

Refer to [Licensing restrictions](../../administration/enterprise-licensing/#license-restrictions) for more information.

### Export usage insights to Loki

You can now export usage insights logs to Loki and query them from Grafana. Usage insights logs include dashboard visits, data source views, queries and errors, and more.

For more information, refer to [Export logs of usage insights](../../setup-grafana/configure-security/export-logs/).

### New audit log events

New log out events are logged based on when a token expires or is revoked, as well as [SAML Single Logout](../../setup-grafana/configure-access/configure-authentication/saml/#single-logout). A `tokenId` field was added to all audit logs to help understand which session was logged out of.

Also, a counter for audit log writing actions with status (success / failure) and logger (loki / file / console) labels was added.

[Auditing](../../setup-grafana/configure-security/audit-grafana/) was updated to reflect these changes.

### Reports support Unicode

You can now select a font, other than the default, for Unicode-based scripts. As a result, an automatically generated PDF of a dashboard, which contains for example Chinese or Cyrillic text, can display them. Because the size of a report increases as additional fonts are added, this feature is not on by default.

[Reporting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/create-reports/#rendering-configuration) was updated as a result of this change.

### Request security

Request security introduces ways to limit requests from the Grafana server, and it targets requests that are generated by users.

For more information, refer to [Request security](../../setup-grafana/configure-security/configure-request-security/).

## Breaking changes

The following Grafana 7.4 changes might break previous functionality.

### Plugin compatibility

We have upgraded AngularJS from version 1.6.6 to 1.8.2. Due to this upgrade some old angular plugins might stop working and will require a small update. This is due to the deprecation and removal of pre-assigned bindings. So if your custom angular controllers expect component bindings in the controller constructor you need to move this code to an $onInit function. For more details on how to migrate AngularJS code open the migration guide and search for pre-assigning bindings.

In order not to break all angular panel plugins and data sources we have some custom angular inject behavior that makes sure that bindings for these controllers are still set before constructor is called so many old angular panels and data source plugins will still work.

### Fixes Constant variable persistence confusion

In order to minimize the confusion with Constant variable usage, we've removed the ability to make Constant variables visible. This change will also migrate all existing visible Constant variables to Textbox variables because which we think this is a more appropriate type of variable for this use case.

## Changelog

Check out [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md) for a complete list of new features, changes, and bug fixes.

# What's new in Grafana v7.5

This topic includes the release notes for Grafana v7.5. For all details, read the full [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## Grafana OSS features

These features are included in the Grafana open source edition.

### Pie chart panel visualization (beta)

Grafana 7.5 adds a beta version of the next-generation pie chart panel.

![Pie chart panel](/static/img/docs/pie-chart-panel/pie-chart-panel-7-5.png)

For more information, refer to [Pie chart panel](../../panels-visualizations/visualizations/pie-chart/).

### Alerting for Loki

Grafana 7.5 comes with alerting support for Loki. With LogQL you can wrap a log query with the functions that allow for creating metrics out of the logs, such as "rate()". Metric queries can then be used to calculate things such as the rate of error messages. [When combined with log parsers](https://www.youtube.com/watch?v=H9z2V0Ib1q0), they can be used to calculate metrics from a value within the log line, such latency or request size.

With alerting support for Loki, you can now create alerts on Loki metrics queries.

[Alerting](../../alerting/) was updated as a result of this change.

![Loki alerting](/static/img/docs/alerting/alerting-for-loki-7-5.png)

### Loki label browser

A new Loki logs browser lets you construct the queries step by step: you choose labels that you like to consider, such as "job", then you select the values that you like, such as "my-app1". Note that you can select values from more than one label, and that they get facetted. This means only possible label combinations are selectable. When you're done, you can run the query either as a logs or a metrics query (a metrics query returns the log volume in a chart).

### Changed default HTTP method for new Prometheus data sources

For new Prometheus data sources, we have changed the default HTTP method to POST. POST allows for much larger query bodies than using the GET method. This is necessary when sending queries from graphs with a lot of targets, for example, many hosts in a dashboard variable. The POST method also makes the Query Inspector data easier to read since the query is in plain text whereas the GET query is URL encoded.

> **Note:** This is not going to affect provisioned data sources or already created data sources.

[Prometheus data source](../../datasources/prometheus/) was updated as a result of this change.

### Word highlighting for Elasticsearch

When searching for text in Elasticsearch logs, matching words in the log line returned by the query are now highlighted.

![Elastic logs highlighting](/static/img/docs/elasticsearch/elastic-word-highlighting-7-5.png)

### Better format definition for trace data

In Grafana 7.5, we changed how data for the trace view is sent from the data source. The required data frame has a clear format, which is more aligned with how data is generally represented in Grafana. This makes it easier for third-party developers to implement data sources leveraging the trace view.

For more information, refer to [trace data API docs](../../explore/trace-integration/#data-api).

### Paste in SSL certs for Postgres data source

Previously, when users wanted to configure the Postgres data source to connect with SSL certification, they needed to put the certification on the server, and configure the data source with file path.

Instead of the file path, users can now paste the SSL certification content in the UI. This allows them to configure the certification even when they do not have access to the server.

> **Note:** It remains as limitation for the Grafana Cloud, as users may not have access to the server configuration.

[Postgres data source](../../datasources/postgres/) and [Provisioning](../../administration/provisioning/) were updated as a result of this change.

### Deprecation notice for some Azure Monitor queries

In the upcoming Grafana 8.0 release, Application Insights and Insights Analytics query types within the Azure Monitor data source will be deprecated and be made read-only in favor of querying Application Insights from Metrics and Logs.

Grafana 7.5 includes a deprecation notice for these queries, and some documentation to help users prepare for the upcoming changes.

For more information, refer to [Deprecating Application Insights and Insights Analytics](../../datasources/azure-monitor/#application-insights-and-insights-analytics--removed-).

### Cloudwatch data source enhancements

- Support for region eu-south-1 has been added to the CloudWatch data source. New metrics have also been added to the namespaces AWS/Timestream, AWS/RDS (RDS Proxy metrics), AWS/NetworkFirewall, AWS/GroundStation, and AWS/DDoSProtection. Many thanks to [relvira](https://github.com/relvira), [ilyastoli](https://github.com/ilyastoli), and [rubycut](https://github.com/rubycut) for contributing!
- Added a page limit to the List Metrics API call to improve speed and reduce memory consumption. You can change this limit by entering a higher value in [list_metrics_page_limit](../../setup-grafana/configure-grafana/#list-metrics-page-limit) in the Grafana configuration file.
- You can now enable or disable authentication providers and assume a role other than default by changing the [allowed_auth_providers](../../setup-grafana/configure-grafana/#allowed_auth_providers) and [assume_role_enabled](../../setup-grafana/configure-grafana/#assume_role_enabled) options in the Grafana configuration file. By default, the allowed authentication providers are _AWS SDK Default_, _Access and secret key_, and _Credentials File_, and role is _Assume role (ARN)_.
- You can now specify a custom endpoint in the CloudWatch data source configuration page. This field is optional, and if it is left empty, then the default endpoint for CloudWatch is used. By specifying a regional endpoint, you can reduce request latency.

  [AWS Cloudwatch data source](../../datasources/aws-cloudwatch/) was updated as a result of this change.

### Increased API limit for CloudMonitoring Services

In previous versions, when querying metrics for Service Level Objectives (SLOs) in the CloudMonitoring data source, only the first 100 services were listed in the **Service** field list. To overcome this issue, the API limit for listing services has been increased to 1000.

### Tempo as a backend data source

We have converted Tempo to a backend data source and dropped support for tempo-query's (Jaeger) response. To configure it, you can now point to the port that is set in the Tempo configuration file.

```yaml
server:
  http_listen_port: 3101
```

[Azure Monitor data source](../../datasources/azure-monitor/) was updated as a result of this change.

## Enterprise features

These features are included in the Grafana Enterprise edition.

### Query caching

When caching is enabled, Grafana temporarily stores the results of data source queries. When you or another user submit the same query again, the results return from the cache instead of from the data source (such as Splunk or ServiceNow).

Query caching advantages:

- Faster dashboard load times, especially for popular dashboards.
- Reduced API costs.
- Reduced likelihood that APIs will rate-limit or throttle requests.

Caching currently works for all backend data sources. You can enable the cache globally or per data source, and you can configure the cache duration per data source. The cache is currently in-memory.

For more information, refer to [Query caching](../../administration/data-source-management/#query-and-resource-caching).

### Use template variable in reports

If you have created dashboards with template variables, then you can choose which values are selected when rendering a report. This makes it easier to tailor reports to their audience or generate multiple reports from the same dashboard.

Enable this feature in configuration settings using the `templateVariables` flag.

For more information, refer to [Reporting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/create-reports/#customize-template-variables).

### Active user limits

If a Grafana instance has exceeded its licensed number of active users, then non-active users who try to sign in to Grafana will be prevented from doing so. Active users are users who have logged in to Grafana in the past 30 days. The total number of users registered in Grafana does not affect this rule. This enforcement is applied separately for Viewers and for Editor/Admins, so if you reach your active Viewer limit, new Editor/Admins will still be able to sign in. This rule also includes a 10% buffer, meaning that you need to exceed your limit by 10% before users are prevented from signing in.

Here is an example:

A Grafana Enterprise instance includes 100 Viewers and 50 Editor/Admins. Over the course of the last 30 days, 110 Viewers and 20 Editor/Admins have signed in to Grafana.

All of the Viewers who have signed in over the past 30 days will retain the ability to sign in.

When a previously-inactive Viewer (someone who has not signed in over the past 30 days) tries to sign in, they will see a message and be prevented from signing in until the number of active users dips back below 110. New Editor/Admins are not affected by this; they can continue to sign in until the number of active Editors/Admins reaches 55.

If you try to sign in to a fourth device or browser, then you will be prevented from doing so; the limit of concurrent sessions is three.

If you sign in to a fourth device or browser, then you will be signed out of the session that is least current.
Concurrent session limits
Each Grafana Enterprise user will be limited to three concurrent user sessions. When a user opens a fourth session, then the longest-inactive session will be automatically signed out.

A new session is created when you sign in to Grafana from a different device or a different browser. Multiple windows and tabs in the same browser are all part of the same session, so having many Grafana tabs open will not cause any issues.

For more information on Grafana Enterprise licensing and restrictions, refer to [License restrictions](../../administration/enterprise-licensing/#license-restrictions).

## Breaking changes

There are no known breaking changes in this release.

## Updated configuration

```
[server]
read_timeout = 0
```

Sets the maximum time using a duration format (5s/5m/5ms) before timing out read of an incoming request and closing idle connections.
`0` means there is no timeout for reading the request.

# What's new in Grafana v8.0

This topic includes the release notes for Grafana v8.0. For all details, read the full [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## Grafana OSS features

These features are included in the Grafana open source edition.

### Grafana v8.0 alerts

The new alerts in Grafana 8.0 are an opt-in feature that centralizes alerting information for Grafana managed alerts and alerts from Prometheus-compatible data sources in one UI and API. You can create and edit alerting rules for Grafana managed alerts, Mimir alerts, and Loki alerts as well as see alerting information from prometheus-compatible data sources in a single, searchable view. For more information, on how to create and edit alerts and notifications, refer to [Grafana Alerting](../../alerting/).

As part of the new alert changes, we have introduced a new data source, Alertmanager, which includes built-in support for Prometheus Alertmanager. It is presently in alpha and it not accessible unless alpha plugins are enabled in Grafana settings. For more information, refer to [Alertmanager data source](../../datasources/alertmanager/).

> **Note:** Out of the box, Grafana still supports old Grafana alerts. They are legacy alerts at this time, and will be deprecated in a future release.

To learn more about the differences between new alerts and the legacy alerts, refer to [What's New with Grafana 8 Alerts](/docs/grafana/v10.0/alerting/difference-old-new/).

### Library panels

Library panels allow users to build panels that can be used in multiple dashboards. Any updates made to that shared panel will then automatically be applied to all the dashboards that have that panel. For instructions on how to create, add, unlink and manage library panels, refer to [Library panels](../../dashboards/build-dashboards/manage-library-panels/).

### Real-time streaming

Data sources can now send real-time updates to dashboards over a websocket connection. This can be used with the [MQTT data source](https://github.com/grafana/mqtt-datasource).

In addition to data source integration, events can be sent to dashboards by posting metrics to the new live endpoint: `/api/live/push` endpoint.

These metrics will be broadcast to all dashboards connected to that stream endpoint.

For more information about real-time streaming, refer to [Grafana Live documentation](../../setup-grafana/set-up-grafana-live/).

### Prometheus metrics browser

The Prometheus metrics browser allows you to quickly find metrics and select relevant labels to build basic queries. If supported by your Prometheus instance, each metric will show its HELP and TYPE as a tooltip. For more information, refer to [Metrics browser documentation](../../datasources/prometheus/#metrics-browser).

### Bar chart visualization (beta)

The Bar chart panel is a new visualization that supports categorical data. It only supports one data frame and it needs to have at least one string field that will be used as the category for an X or Y axis and one or more numerical fields.

{{< figure src="/static/img/docs/bar-chart-panel/bar-chart-example-v8-0.png" max-width="1025px" caption="Bar chart example" >}}

To use it with time series you first have to add a **Reduce** transform.

For more information, refer to [Bar chart visualization](../../panels-visualizations/visualizations/bar-chart/).

### State timeline visualization (beta)

The State timeline visualization shows discrete state changes over time. Each field or series is rendered as a unique horizontal band. This panel works well with string or boolean states, but it can also be used with time series data. When used with time series data, the thresholds are used to turn the numerical values into discrete state regions.

This panel also takes advantage of the new value mapping features that allow you to color string and boolean values.

Example with string values:
{{< figure src="/static/img/docs/v8/state_timeline_strings.png" max-width="800px" caption="state timeline with string states" >}}

With time series data and thresholds:
{{< figure src="/static/img/docs/v8/state_timeline_time_series.png" max-width="800px" caption="state timeline with time series" >}}

For more information, refer to [State timeline visualization](../../panels-visualizations/visualizations/state-timeline/).

### Status history visualization (beta)

A sister panel to the state timeline is the new Status history panel visualization. It can display periodic states in a grid view. It supports both numerical, string, or boolean states. You can assign colors using value mappings, thresholds, or gradient color. For more information, refer to [Status history](../../panels-visualizations/visualizations/status-history/).
schemes.

![Status grid visualization](/static/img/docs/status-grid/status-grid-8-0.png)

### Histogram visualization (beta)

This hidden feature of the old Graph panel is now a standalone visualization. It combines a histogram transformation and bar chart visualization into a single, integrated, easy-to-use panel. There is also a new standalone histogram transformation that can be paired with any visualization.

{{< figure src="/static/img/docs/histogram/histogram-8-0.png" max-width="1025px" caption="Histogram example" >}}

For more information, refer to [Histogram](../../panels-visualizations/visualizations/histogram/)

### Time series visualization updates

The Time series is out of beta! We are removing the `Beta` tag and graduating the Time series visualization to a stable state.

- **Time series** is now the default visualization option, replacing the **Graph (old)**.
- The Time series panel now supports stacking. For more information, refer to [Graph stacked time series](../../panels-visualizations/visualizations/time-series/#stack-series).
- You can now add alerts in the Time series panel, just like the old Graph panel.
- Updated [connect null values](../../panels-visualizations/visualizations/time-series/#connect-null-values) options.
- We added support for a shared crosshair and a tooltip that’s now smarter when it comes to data display in the tooltip.
- Various performance improvements.

[Time series panel](../../panels-visualizations/visualizations/time-series/) topics have been updated as a result of these changes.

### Node graph visualization updates

You can now expand the node graph for the displayed trace when using the Trace to logs feature. Depending on the data source, this can show spans of the trace as nodes in the graph, or as some additional context like service map based on the current trace.

We also added a grid view and the ability to explore hidden nodes.

[Tracing in Explore](../../explore/trace-integration/) and [Node graph](../../panels-visualizations/visualizations/node-graph/) were updated as a result of these changes.

### Pie chart visualization updates

The Pie chart is out of beta! We are removing the `beta` tag and graduating the Pie chart visualization to a stable state.

### Panel editor updates

Lots of panel editor improvements, heavily informed by user research and community feedback.

- All options are now shown in a single pane.
- You can now search panel options.
- The Value mappings feature has been completely redesigned. For more information, refer to [Configure value mappings](../../panels-visualizations/configure-value-mappings/).
- New **Table view** option is always available.

The [Panels](../../panels-visualizations/) section has been updated to reflect these changes.

### Look and feel update

Grafana 8 comes with a refreshed look and feel, including themes changed to be more accessible. The improved Grafana UI brings a number of adjustments and tweaks that make the application even more fun to use. Enjoy the new home dashboard design!

Under the hood, the new theme architecture enables us to bring more sophisticated themes control in the future.

### Download logs

When you inspect a panel, you can now download log results as a text (.txt) file.

[Download log results](../../panels-visualizations/panel-inspector/) was added as a result of this feature.

### Inspector in Explore

The new Explore inspector helps you understand and troubleshoot your queries. You can inspect the raw data, export that data to a comma-separated values (CSV) file, export log results in text format, and view query requests.

[Inspector in Explore](../../explore/explore-inspector/) was added as a result of this feature.

### Explore log improvements

Log navigation in Explore has been significantly improved. We added pagination to logs, so you can click through older or newer logs as needed.

[Logs in Explore](../../explore/logs-integration/) was updated as a result of these changes.

![Navigate logs in Explore](/static/img/docs/explore/navigate-logs-8-0.png)

### Plugin catalog

You can now use the Plugin catalog app to easily manage your plugins from within Grafana. Install, update, and uninstall plugins without requiring a server restart.

[Plugin catalog](../../administration/plugin-management/#access-the-plugin-catalog) was added as a result of this feature.

### Performance improvements

Grafana 8.0 includes many performance enhancements.

#### Initial startup and load performance

We reduced the Grafana initial download size massively, approximately 40%. This means that on slower or mobile connections, the initial login page or home dashboard will load much faster.

All panels that have migrated from Flot to uPlot will also render two to three times faster because the library is much more efficient. Right now, this includes the Time series, Stat, Timeline, Histogram, and Barchart panel visualizations.

#### Operational and runtime performance

These improvements affect any subsequent data updates or interactions, including:

- Streaming performance
- General speed of interaction, such as zooming, tooltips, synchronized cursors, and panel updates while editing

### Data source updates

The following data source updates are included with this Grafana release.

#### Azure Monitor data source

Azure Resource Graph is now supported in the Azure Monitor data source. This is a service in Azure that is designed to extend Azure Resource Management by providing efficient and performant exploration of your Azure resources.

The Azure Monitor data source now supports Managed Identity for users hosting Grafana in Azure to simplify and secure configuring Azure Monitor in Grafana.

Also, in addition to querying Log Analytics Workspaces, you can now query the logs for any individual [supported resource](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported), or for all resources in a subscription or resource group.

> **Note:** In Grafana 7.5 we started the deprecation for separate Application Insights queries, in favor of querying Application Insights resources through Metrics and Logs. In Grafana 8.0 new Application Insights and Insights Analytics queries cannot be made, and existing queries have been made read-only. For more details, refer to the [Deprecating Application Insights](../../datasources/azure-monitor/#application-insights-and-insights-analytics--removed-).

[Azure Monitor data source](../../datasources/azure-monitor/) was updated as a result of these changes.

#### Elasticsearch data source

[Elasticsearch data source](../../datasources/elasticsearch/) and [Provisioning](../../administration/provisioning/) were updated as a result of these changes.

##### Use semver strings to identify Elasticsearch version

We changed how the configured Elasticsearch version is handled. You can now specify via provisioning the full semver string version of your instance (such as “7.12.1”) instead of the old version format based on numbers. There’s no manual intervention needed, the old options will be correctly recognized.

##### Generic support for template variables

You can now use a different interpolation method to use template variables in a more extensive way. You can now use template variables in every query editor field that allows free input.

![Elasticsearch template variables](/static/img/docs/elasticsearch/input-templates-8-0.png)

##### Allow omitting field for metrics that support inline scripts

Metric aggregations can be specified without a field if a script is provided. You can now deselect fields for metrics aggregation when they support scripting.

Previously this was only possible when adding a new metric without selecting a field, because once selected, the field could not have been removed.

![Elasticsearch omit fields](/static/img/docs/elasticsearch/omit-fields-8-0.png)

##### Allow setting a custom limit for log queries

You can now set a custom line limit for logs queries instead of accepting the previously hard-coded 500. We also simplified the query editor to only show relevant fields when issuing logs queries.

![Elasticsearch custom log limit](/static/img/docs/elasticsearch/custom-log-limit-8-0.png)

##### Guess field type from first non-empty value

Response values were always interpreted as strings in Elasticsearch responses, which caused issues with some visualization types that applied logic based on numeric values. We now apply some heuristics to detect value types from the first non-empty value in each response.

#### Google Cloud Monitoring data source

In a prior release, Cloud Monitoring added _preprocessing_ support in their query editor. This capability has been added to the Cloud Monitoring data source in Grafana.

Whenever a metric is selected in the query editor, a suitable preprocessing option is automatically selected for you. To avoid breaking changes, preprocessing is not enabled by default on existing queries. If you want to use preprocessing for existing queries, you have to manually select one in the query editor.

[Google Cloud Monitoring data source](../../datasources/google-cloud-monitoring/#pre-processing) was updated as a result of this change.

#### Graphite data source

[Graphite data source](../../datasources/graphite/) was updated as a result of these changes.

##### Variable metric names expand

Values for dashboard variables can be now populated using the [Graphite expand API](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-expand). Expand API is used when the metric query is wrapped in expand() function.

This way, values can contain not only the last matching node from the metric query, but also the full path of the metric. It can also be narrowed down to a specific node with a regular expression.

##### Map Graphite queries to Loki

Graphite queries are now automatically transformed to Loki queries according to user-defined rules when the data source changes in Explore.

#### Jaeger data source

You can now use more parameters to find traces.

[Jaeger data source](../../datasources/jaeger/) was updated as a result of this change.

### Authentication updates

This Grafana release includes the following authentication updates.

#### JWT

JWT is a new authentication option in Grafana.

#### Added JWT authentication support

You can now configure Grafana to accept a JWT token provided in the HTTP header.

[JWT authentication](../../setup-grafana/configure-access/configure-authentication/jwt/) was added and [Configuration](../../setup-grafana/configure-grafana/#authjwt) was updated as a result of this feature.

#### OAuth

[Generic OAuth authentication](../../setup-grafana/configure-access/configure-authentication/generic-oauth/) has been updated as a result of these changes.

##### Added OAuth support for empty scopes

You can now configure generic OAuth with empty scopes. This allows OAuth Identity Providers that don't use or support scopes to work with Grafana authentication.

##### Added OAuth support for strict parsing of role_attribute_path

You can now configure generic OAuth with strict parsing of the `role_attribute_path`. By default, if the `role_attribute_path` property does not return a role, then the user is assigned the `Viewer` role. You can disable the role assignment by setting `role_attribute_strict = true`. It denies user access if no role or an invalid role is returned.

#### Singlestat panel deprecated

Support for Singlestat panel has been discontinued. When you upgrade to version 8.0, all existing Singlestat panels automatically becomes Stat panels.
Stat panel is available as plugin.

### Grafana license update

Grafana has updated its license from Apache 2.0 to the GNU Affero General Public License (AGPL). Please see the related [blog post](/blog/2021/04/20/grafana-loki-tempo-relicensing-to-agplv3/), [Q&A](/blog/2021/04/20/qa-with-our-ceo-on-relicensing/) and [license](https://github.com/grafana/grafana/blob/main/LICENSE) for more details.

## Enterprise features

These features are included in the Grafana Enterprise edition.

### Role-based access control

You can now add or remove detailed permissions from Viewer, Editor, and Admin org roles, to grant users just the right amount of access within Grafana. Available permissions include the ability to view and manage Users, Reports, and the Access Control API itself. Grafana will support more and more permissions over the coming months.

[Role-based access control docs](../../administration/roles-and-permissions/access-control/) were added as a result of this feature.

### Data source query caching

Grafana caches the results of backend data source queries so that multiple users viewing the same dashboard or panel do not make multiple submissions of the same query to the data source (like Splunk or Snowflake) itself.

This results in faster average load times for dashboards and fewer duplicate queries overall to data sources, which reduces cost and the risk of throttling, reaching API limits, or overloading your data sources.

You can enable caching per data source, and time-to-live (TTL) can be configured globally and per data source. Query caching can be set up with Redis, Memcached, or a simple in-memory cache.

For more information, refer to the [Data source query caching docs](../../administration/data-source-management/#query-and-resource-caching).

### Reporting updates

When creating a report, you can now choose to export Table panels as .csv files attached to your report email. This makes it easier for recipients to view and work with that data.

You can also link back to the dashboard directly from the email, for users who want to see the data live in Grafana. This release also includes some improvements to the Reports list view.

For more information, refer to [Reporting docs](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/create-reports/).

### License restrictions clarification in the docs

The Grafana Enterprise documentation has been updated to describe more specifically how licensed roles are counted, how they can be updated, and where you can see details about dashboard and folder permissions that affect users' licensed roles.

For more information, refer to [License restrictions docs](../../administration/enterprise-licensing/#license-restrictions).

## Breaking changes

The following breaking changes are included in this release.

### Variables

- Removed the **Value groups/tags** feature from variables. Any tags will be removed.
- Removed the `never` refresh option for query variables. Existing variables will be migrated and any stored options will be removed.

Documentation was updated to reflect these changes.

### Elasticsearch: Use application/x-ndjson content type for multi-search requests

For multi-search requests, we now use the correct application/x-ndjson content type instead of the incorrect application/json. Although this should be transparent to most of the users, if you are running Elasticsearch behind a proxy, then be sure that your proxy correctly handles requests with this content type.

### Prometheus: Update default HTTP method to POST for existing data sources

The default HTTP method for Prometheus data source is now POST, previously it was GET. The POST APIs are there since January 2018 (Prometheus 2.1.0) and they have fewer limitations than the GET APIs. Users with Prometheus instance with version < 2.1.0 that use the default HTTP method should update their HTTP method to GET.

# What's new in Grafana v8.1

> **Note:** This topic will be updated frequently between now and the final release.

Grafana 8.1 builds upon our promise of a composable, open observability platform with new panels and extends functionality launched in Grafana 8.0. We’ve got new Geomap and Annotations panels, and some great updates to the Time Series panel. We’ve also got new transformations and updates to data sources. For our enterprise customers, there are additions to fine grained access control, updates to the reporting schedule and query caching, and more. Read on to learn more.

In addition to what is summarized here, you might also be interested in our announcement blog post. For all the technical details, check out the complete [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## Grafana OSS features

These features are included in the Grafana open source edition.

## Geomap panel

Grafana 8.1 introduces the foundation for our new map panel. This new panel leverages [OpenLayers](https://openlayers.org/) and gives us a flexible solution for extending the way we use the new Geomap panel moving forward. The new Geomap panel includes multiple base layer styles (map layer options), as well as a more open data layer. The data layer can use coordinates and geo-hashed data in addition to a lookup table.

The Geomap panel is also able to share views across multiple Geomap panels on the same dashboard, making it straightforward to visualize and explore multiple types of geospatial data using the same map zoom and focus settings. For more information, refer to [Geomap panel](../../panels-visualizations/visualizations/geomap/).
{{< figure src="/static/img/docs/geomap-panel/geomap_with_heatmap.png" max-width="850px" caption="Geomap panel: Heatmap" >}}

## Annotation panel

The new Annotations panel shows a list of available annotations you can use to create lists of annotated data available within your organization. Various options are available to filter the list based on the tags and on the current dashboard. This panel makes it easy to find and filter annotated data within and across multiple dashboards.

{{< figure src="/static/img/docs/annotations-panel/annolist.png" max-width="900px" caption="Annotations panel" >}}

### Time series panel updates

The time series panel has been updated with the ability to color series and line by thresholds or gradient color scales. This allows users to create panels where the line color can change dynamically based on thresholds or using gradient color scales. It adds a layer of visibility to your data, making it easier to view the changes across thresholds at a glance quickly.

Color scheme **From thresholds**:
{{< figure src="/static/img/docs/time-series-panel/gradient_mode_scheme_thresholds_line.png" max-width="1200px" caption="Colors scheme: From thresholds" >}}

Color scheme: **Green-Yellow-Red (by value)**
{{< figure src="/static/img/docs/time-series-panel/gradient_mode_scheme_line.png" max-width="1200px" caption="Color scheme: Green-Yellow-Red" >}}

For more on how to configure Graphs with by value color schemes read [Graph and color schemes](../../panels-visualizations/visualizations/time-series/).

You can also create annotations in the new Time series panel bringing it closer to parity with the legacy Graph panel. To learn more, refer to the [time series panel](../../panels-visualizations/visualizations/time-series/).

### Transformations improvements

Grafana 8.1 includes some significant enhancements to transformations, including two new transformations designed around providing dynamic configuration to your panels and visualizations.

#### Config from query (Beta)

This transformation enables panel config (Threshold, Min, Max, etc.) to be derived from query results. For more information, refer to [Config from query results](../../panels-visualizations/query-transform-data/transform-data/#config-from-query-results).

#### Rows to fields (Beta)

This transformation enables rows in returned data to be converted into separate fields. Prior to this enhancement, you could style and configure fields individually, but not rows. For more information, refer to [Rows to fields transform].

Example, Input:

| Name    | Value | Max |
| ------- | ----- | --- |
| ServerA | 10    | 100 |
| ServerB | 20    | 200 |
| ServerC | 30    | 300 |

Output:

| ServerA (config: max=100) | ServerB (config: max=200) | ServerC (config: max=300) |
| ------------------------- | ------------------------- | ------------------------- |
| 10                        | 20                        | 30                        |

As you can see each row in the source data becomes a separate field. Each field now also has a max config option set. Options like **Min**, **Max**, **Unit** and **Thresholds** are all part of field configuration and if set like this will be used by the visualization instead of any options manually configured in the panel editor options pane.

For more on how to use this transformation, refer to [Rows to fields transform](../../panels-visualizations/query-transform-data/transform-data/#rows-to-fields).

#### Contextual & Inline Help

Additional inline help will be available for Transformations. We can now share examples of how to use specific transformations and point users directly to the appropriate place in the docs for more information.

### Data source updates

The following data source updates are included with this Grafana release.

#### MySQL Data Source

We have added timezone support. As a result, you can now specify the time zone used in the database session, such as `Europe/Berlin` or `+02:00`.

### Trace to logs improvements

We changed the default behavior from creating a 1-hour span Loki query to the only query at the exact time the trace span started for the duration of it. For more fine grained control over this, you can shift this time in the tracing data source settings. Also, it is now possible to shift the start time and end time of the Loki query by the set amount. For more information, refer to [Trace to logs](../../datasources/tempo/#trace-to-logs).

### Prettify JSON for Logs in Explore

Added the ability to format JSON to make it easier to view, review and find relevant data in JSON format logs. This is a regular JSON log.

{{< figure src="/static/img/docs/panels/pretty-json/regular-log-line.png" max-width="1200px" caption="Regular log line" >}}

And here is the prettified JSON log.

{{< figure src="/static/img/docs/panels/pretty-json/prettified-json.png" max-width="1200px" caption="Prettified JSON" >}}

For more on how to prettify JSON logs, refer to [Visualization](../../panels-visualizations/visualizations/) and [Display](../../panels-visualizations/visualizations/logs/) options.

### Plugin catalog - Updated UX and extended features

We’ve made some changes to the plugins UI to help make it easier to discover and manage your plugins. Enterprise users can now also manage enterprise plugins from within the catalog.

#### Documentation updates

New panel summaries and preview on the top level [Visualizations](../../panels-visualizations/visualizations/) page to help users pick or learn about specific visualizations more easily.

### Upcoming changes to the Select component

The `@grafana/ui` exposes a `Select` component, and its variants `MultiSelect`, `AsyncSelect`, and `AsyncMultiSelect`. We have made some internal changes to these components to make the behavior and positioning more consistent in all scenarios.

To test the changes, you can use the `menuShouldPortal` property:

```jsx
<Select menuShouldPortal {...otherProps} />
```

Tests are most likely to be affected. There are some tips for fixing these in the original pull request at [https://github.com/grafana/grafana/pull/36398](https://github.com/grafana/grafana/pull/36398).

We’d love as much feedback as possible about this change, because we are considering making this the default behavior in a future release of Grafana.

### High availability setup support for Grafana Live

We have added an experimental HA setup support for Grafana Live with Redis. This resolves the limitation when clients were connected to different Grafana instances and those instances had no shared state. For additional information, refer to [Configure Grafana Live HA setup](../../setup-grafana/set-up-grafana-live/#configure-grafana-live-ha-setup).

## Enterprise features

These features are included in the Grafana Enterprise edition.

### New permissions for role-based access control

Role-based access control remains in beta. You can now grant or revoke permissions for Viewers, Editors, or Admins to use Explore mode, configure LDAP or SAML settings, or view the admin/stats page. These new permissions enhance the existing permissions that can be customized, namely permissions to access Users, Orgs, LDAP settings, and Reports in Grafana.

Fine grained access control allows you to customize roles and permissions in Grafana beyond the built-in Viewer, Editor, and Admin roles. As of 8.1, you can modify some of the permissions for any of these built-in roles. This is helpful if you’d like users to have more or fewer access permissions than a given role allows for by default. For an overview of role-based access control and a complete list of available permissions, refer to the [Fine grained access control](../../administration/roles-and-permissions/access-control/) documentation.

### New and improved reporting scheduler

We’ve enhanced the scheduler for Reports to be more flexible, so you can send reports at just the right time. When scheduling a report, you can now choose to send a report at custom intervals such as every 4 hours or every 2 weeks. You can also send a report for a limited time period by providing a start and end date, or send a report only on weekdays or on the last day of each month. This change accompanies some other recent improvements to Reporting, like the ability to choose template variables for reports and an improved UX for authoring reports. To learn more, refer to the [reporting](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/create-reports/) documentation.

### Encrypt data in the query cache

Query caching was released in Grafana 8.0 and allows you to temporarily store the results of data source queries in a cache, so that Grafana reads repeated queries from there instead of from the data source itself. This reduces load on data sources, improves dashboard load times, and can save money for data sources that charge per query. To learn more about query caching see its [overview](../../administration/data-source-management/#query-and-resource-caching) page. To find out how to turn on encryption, refer to the [caching configuration](../../setup-grafana/configure-grafana/enterprise-configuration/#caching) documentation.

You can now encrypt the query data cached by Grafana. This improves the security of query data, especially when your cache (like Redis) is shared with other services.

### White labeling for the Grafana loading logo

You can now customize Grafana’s loading logo, which displays while Grafana is loading in a user’s browser. White labeling in Grafana Enterprise allows you to customize the look and feel of Grafana to match your product’s or company’s brand. This makes Grafana a more integrated part of your observability stack and keep Grafana consistent with other visualizations displayed in public.

To find out how you can configure it along with other Grafana UI elements, like the corner logo and application footer, refer to the [White labeling](../../setup-grafana/configure-grafana/configure-custom-branding/) topic of the Grafana Enterprise docs.

### Oauth2 - Team Sync to Group Mapping

With Team Sync you can map your Generic OAuth groups to teams in Grafana so that the users are automatically added to the correct teams.

# What's new in Grafana v8.2

Grafana 8.2 continues to build on the foundation of Grafana 8.0 & 8.1. Grafana 8.2 also marks the start of our work to bring Grafana closer to all users with a focus on increasing Grafana’s accessibility, part of its continuing mission to democratize metrics _for everyone_.

The plugin catalog is now on by default in Grafana 8.2. Using the plugin catalog you can now find and install official and community plugins without having to leave or restart Grafana. We’ve also updated the time picker to include configurable fiscal quarters. This update makes it easier to use Grafana to produce reports more closely aligned with common review and forecasting cycles.

Grafana Enterprise includes a revamped Stats and Licensing page, new role-based access control permissions, and improvements that make usage insights and reporting easier to access.

We’ve summarized what’s new in the release here, but you might also be interested in the announcement blog post. If you’d like all the details you can check out the [release notes](/docs/grafana/next/release-notes/release-notes-8-2-0/) and complete [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## OSS

### Community Contributions

Grafana 8.2 includes a number of important community contributions including support for OAuth role mapping with GitLab accounts ([#30025](https://github.com/grafana/grafana/pull/30025)), a new wide-to-long function ([#38670](https://github.com/grafana/grafana/pull/38670)) included in the [prepare time series transformation](/docs/grafana/latest/panels/transformations/types-options/#prepare-time-series). A number of additions to the Azure Monitor data source were submitted, including an overview dashboard([#38801](https://github.com/grafana/grafana/pull/38801)) and support for parsing numeric fields in the Azure Resource Graph([#38728](https://github.com/grafana/grafana/pull/38728)). Contributions also included [regular-expression based value mapping](/docs/grafana/next/panels/value-mappings/#map-a-regular-expression) ([#38931](https://github.com/grafana/grafana/pull/38931)) and improvements to our systemd unit for Grafana installations ([#38109](https://github.com/grafana/grafana/pull/38109)). This list is by no means exhaustive or comprehensive. We greatly appreciate _all_ the contributions submitted for inclusion in Grafana.

### Accessibility

We’ve taken our first, measured but important step towards improving the accessibility of Grafana, with much needed tweaks to keyboard navigation, accessibility labeling for UI elements for Grafana viewers, and an [accessibility statement](/accessibility/) for Grafana laying out what you can expect from a Grafana focused on making our project accessible to all.

### Dashboards

The biggest change to dashboards in Grafana 8.2 is the inclusion of a configurable fiscal year in the time picker. This option enables fiscal quarters as time ranges, which can be helpful for business-focused and executive dashboards in addition to many other common use cases. Please see the [time range controls documentation](/docs/grafana/latest/dashboards/time-range-controls/) for more information.

{{< figure src="/static/img/docs/time-range-controls/fiscal_year-8-2.png" max-width="1200px" caption="Fiscal Year Time Range Settings" >}}

### Plugins management

We have continued to improve how you manage your plugins within Grafana. The new [plugins catalog](/docs/grafana/v8.0/administration/configuration/#plugin_admin_enabled) is now enabled by default. You can use the plugin catalog to find, install, and uninstall your plugins directly from within Grafana without needing the Grafana CLI or to restart Grafana.

{{< figure src="/static/img/docs/plugins/plugins-catalog-browse-8-1.png" max-width="1200px" caption="Plugins Catalog Browsers" >}}

### Grafana Alerting

We’ve continued to bolster the new, unified alerting system launched in Grafana 8. This update includes the addition of a UI to edit the Cortex/Loki namespace, edit the alert group name, and edit the alert group evaluation interval. We've also added a Test button to test an alert notification contact point. There's even more to explore here including custom grouping for alert manager notifications and several small but significant changes to improve creation editing and managing alert rules. Please see the [alerting documentation](/docs/grafana/v8.2/alerting/unified-alerting/) for more details and information on how you can enable the unified alerting system in your instance of Grafana.

### Image Renderer performance improvements and measurement

You can use Grafana’s image renderer to generate images of panels and dashboards. Grafana uses these images for alert notifications, PDF exports (Grafana Enterprise), and reports sent by Grafana (Grafana Enterprise). We’ve added additional metrics to the image renderer to help you diagnose its performance, and [included guidance in our documentation](/docs/grafana/next/image-rendering/#rendering-mode) to help you configure it for the best mix of performance and resource usage. Tests show that we have reduced image load time from the 95th percentile of 10 seconds to less than 3 seconds under normal load.

## Grafana Enterprise

### Brand-new license and stats screen

We’ve revamped the Stats and License sections of Grafana for administrators. The new combined screen makes it easier to understand a license’s term and user counts, and find out early when you need to renew or expand a license. It’s also easier to parse Grafana statistics like the number of dashboards, data sources, and alerts in a given instance. This screen also includes an interactive list of dashboard and folder permissions, which can affect your users’ licensed roles in Grafana. Learn more about Grafana Enterprise on our [website](/products/enterprise/grafana/), and more about licenses in particular in our [docs](/docs/grafana/latest/enterprise/license/license-restrictions/).

{{< figure src="/static/img/docs/enterprise/8_2_stats_licensing_screen.png" max-width="1200px" caption="Stats and licensing" >}}

### New role-based access control permissions

Role-based access control now covers data source and provisioning permissions. You can decide which roles (Viewers, Editors, and Admins) can manage data sources and data source permissions in Grafana, and which roles can reload provisioning configuration for dashboards, data sources, and other provisioned resources. We’ll continue adding role-based access control to more Grafana services, like dashboards and API Keys, in upcoming releases. Learn more about role-based access control in our [release post](/blog/2021/06/23/new-in-grafana-enterprise-8.0-fine-grained-access-control-for-reporting-and-user-management/) and our [docs](/docs/grafana/latest/enterprise/access-control/).

{{< figure src="/static/img/docs/enterprise/8_2_data_source_permissions.png" max-width="1200px" caption="Stats and licensing" >}}

### Export usage insights logs as server logs

Usage Insights Logs contain valuable information about user dashboard visits, queries, and front-end errors that are otherwise impossible to track in Grafana. You can now export those logs alongside your regular server logs to identify problematic dashboards and data sources and improve users’ experience with Grafana. Previously, these metrics could only be exported directly to Loki. Learn more in the [documentation about exporting logs](/docs/grafana/latest/enterprise/usage-insights/export-logs/)

{{< figure src="/static/img/docs/enterprise/8_2_export_usage_insights.png" max-width="1200px" caption="Stats and licensing" >}}

### Create a report from the dashboard Share dialogue

Reports offer a powerful way to deliver insights directly to your email inboxes. Now you can create a report directly from any dashboard, using the Share button. This is especially useful when combined with role-based access control, which you can use to grant Editors or Viewers the ability to create reports in Grafana. To learn more, see the [reporting documentation](/docs/grafana/latest/enterprise/reporting/).

{{< figure src="/static/img/docs/enterprise/enterprise-report-from-share-8-2.png" max-width="1200px" caption="Create a report from the dashboard share dialogue" >}}

# What's new in Grafana v8.3

Grafana 8.3 is an exciting release for Grafana Labs. This release includes the new Candlestick Panel, a new visualization suggestions engine and, for enterprise users, Recorded Queries.

For Open Source users it also marks the first time Grafana Alerting, formerly unified alerting, is enabled by default for new Grafana installations. Grafana Alerting in 8.3 is the flexible, single pane of glass for all your alerts. Included in this release is expanded provisioning support for notifiers, contact points, and alert rules, alongside auditing and role-based access control for our Enterprise customers.

We’ve summarized what’s new in the release here, but you might also be interested in the announcement blog post as well. If you’d like all the details you can check out the complete [CHANGELOG.md](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## Grafana OSS

### Community Contributions

Grafana 8.3 includes many community contributions including the ability to interpolate variables in tags ([#42318](https://github.com/grafana/grafana/pull/42318)), fly-out sub-menu support ([#41647](https://github.com/grafana/grafana/pull/41647)), the addition of AWS RoboMaker & AWS Global Accelerator metrics to the AWS CloudWatch data source ([#41450](https://github.com/grafana/grafana/pull/41450), [#41236](https://github.com/grafana/grafana/pull/41236)), as well as updates to the news panel ([#41193](https://github.com/grafana/grafana/pull/41193)), the plugin catalog ([#40917](https://github.com/grafana/grafana/pull/40917)), and many more. The Grafana Labs team is deeply grateful for all those who take the time to contribute and participate in the development of Grafana.

### Accessibility

We’ve continued to make progress on improving Grafana’s accessibility. In Grafana 8.3 we’ve updated the main menu with improved keyboard navigation. We’ve also improved Grafana’s behavior when navigating through elements with the tab key (tab stops) and finished our work to make Grafana’s viewer roles compatible with assistive technologies such as screen readers. You can read our accessibility statement here and reach out to us with accessibility issues using our community Slack or our community forums.

### Dashboards and Visualizations

#### Panel Suggestions

Grafana will now make suggestions for visualization types based on the current data surfaced by a query or queries in a panel. This makes seeing available, possible interpretations of your data more straightforward and can be a useful jumping-off point when building panels and dashboards with specific goals in mind.

{{< figure src="/static/img/docs/panels/panel_suggestions_8_3.png" max-width="1200px" caption="Panel suggestionss" >}}

#### Candlestick Panel (Beta)

Grafana 8.3 includes a new Candlestick panel that is so much more. Beyond candlesticks and open, high, low, close, behaviors, it includes customizable up/down colors, bar color determined by intra-period or inter-period movement of the data, volume histogram with matching colors, and the ability to detach or create a separate volume histogram to allow for more flexible dashboard design. The candlestick panel is based upon our time series panel meaning it also has support for additional data with the same configuration options and styles.

{{< figure src="/static/img/docs/candlestick-panel/candlestick-panel-8-3.png" max-width="1200px" caption="Candlestick panel" >}}

#### Geomap updates

The Geomap panel has been updated to support multiple data layers. The panel can now be configured with multiple markers, heatmaps & GeoJSON layers. The layers can be individually configured, named, and re-ordered to create rich, multidimensional visualizations. Please see the documentation for details on how you can set up and configure these new configuration options.

{{< figure src="/static/img/docs/geomap-panel/geomap_8_3.png" max-width="1200px" caption="Geomap panel updates" >}}

### Grafana Alerting

Grafana Alerting is now the default alerting experience for all new Open Source installations of Grafana 8.3. Grafana Alerting in 8.3 includes the ability to test contact points and notification routing. Grafana 8.3 also adds the ability to configure and use external, Prometheus-style alert managers from within the Grafana Alerting workflow.

{{< figure src="/static/img/docs/alerting/alerting_8_0.png" max-width="1200px" caption="Grafana Alerting" >}}

### Support for AWS CloudWatch Metrics Insights

Grafana and Amazon Managed Grafana now support AWS Metrics Insights – a fast, flexible, SQL-based query engine that enables you to identify trends and patterns across millions of operational metrics in real time.

You can use Metrics Insights in the AWS CloudWatch plugin. Using this new feature is as simple as selecting the Metric Query type. The Metric Query type has two different modes: a Builder mode and a Code editor mode.

The example below demonstrates using the new Metrics Insight capability to view the top 5 instances with the highest average CPU Utilization, ordered by maximum, in descending order. The code editor has built-in autocompletion support that gives suggestions throughout the composition of the query.

{{< figure src="/static/img/docs/cloudwatch/cloudwatch-code-editor-autocomplete-8.3.0.gif" max-width="1200px" caption="Grafana Alerting" >}}

## Grafana Enterprise

### Recorded Queries

Recorded queries turn “point in time” data into time series.

Recorded queries allow you to export the results of certain non-time series queries to the Enterprise backend in order to store data over time and allow customers to construct their own time series.

This new feature is especially helpful for Enterprise customers using plugins because many new plugins, like ServiceNow and Jira, don’t return time series so customers weren’t able to plot historical data over time. With recorded queries, now they can! For more information

### Assign role-based permissions directly to users with the new role picker (beta)

Sometimes the Viewer, Editor, and Admin roles just don’t fit what a certain user needs to do in Grafana. Now you can assign role-based roles directly to users, so they can create reports, use Explore mode, create data sources, and perform other specific actions in Grafana. The role picker can be access from the Grafana Admin user management page.

{{< figure src="/static/img/docs/enterprise/enterprise_role_picker_8_3.png" max-width="1200px" caption="Grafana Enterprise Role Picker" >}}

### Use role-based access control for Organizations and Licensing (beta)

We’ve added new permissions to role-based access control to help you specify actions that users can perform. Now you can assign permissions to manage Organizations and License functions in Grafana, in addition to Users, Data Sources, Reports, and other resources. Role-based access control remains in beta and we will continue to add new permissions until all of Grafana’s endpoints are covered. For a complete list of the actions you can permit using role-based access control, see the [reference](/docs/grafana/next/enterprise/access-control/fine-grained-access-control-references/).

### Get your encryption key from a Key Management Service

Grafana’s database contains secrets, like the credentials used to query data sources, send alert notifications and perform other functions within Grafana. These secrets are encrypted using keys, which are usually stored in Grafana’s configuration file. Now you can get your encryption key from Amazon KMS, Azure Key Vault, or Hashicorp Vault. This allows you to centrally manage your Grafana encryption key and reduce the chances it will leak.

In order to support this, we’ve upgraded Grafana Enterprise to use envelope encryption, which complements the KMS integration by adding a layer of indirection to the encryption process. Instead of encrypting all secrets with a single key, Grafana uses a set of keys called data encryption keys (DEKs) to encrypt them. These data encryption keys are themselves encrypted with a single key encryption key (KEK). With envelope encryption, you can store a KEK in your KMS, and still quickly encrypt and decrypt data using DEKs stored within the Grafana database.

### Pay the same for all users, regardless of their permissions

Are you tired of managing user permissions because your license only allows a certain number of Viewers and Editors or Admins? So were we. We’ve added support for combined user pricing, where all users cost the same and fall into the same license bucket in Grafana Enterprise. This is a specific license option and must be updated in your contract. To learn more, refer to our [licensing docs](/docs/grafana/latest/enterprise/license/license-restrictions/). To switch to combined user pricing, contact your Grafana Labs account team.

{{< figure src="/static/img/docs/enterprise/enterprise_users_8_3.png" max-width="1200px" caption="Grafana Enterprise User Pricing" >}}

### Author dashboards faster with resource caching

Your query editor just became faster. [Query caching](/docs/grafana/latest/enterprise/query-caching/) improves query performance and sometimes reduces cost, by reducing the number of repetitive queries performed against data sources. Resource caching does the same thing but for resource calls, like retrieving the list of applications in the AppDynamics editor, the list of metrics from Datadog, or the list of values in a template variable dropdown. This makes for a zippier user experience for everyone writing queries in Grafana.

### Review audit logs for more services, like alerting

[Audit logs](/docs/grafana/latest/enterprise/auditing/) are a record of the actions users perform in Grafana, which you can investigate in case of a security incident or to understand Grafana usage better. We’ve added audit logs for new actions performed against plugins, data sources, library elements, and Grafana’s new alerting service. This ensures that if a user makes a change anywhere in Grafana Enterprise, you’ll have a record of it. For details, refer to the [Auditing docs](/docs/grafana/latest/enterprise/auditing/).

# What's new in Grafana v8.4

We’re excited to announce Grafana v8.4, with a variety of improvements that focus on Grafana’s usability, performance, and security. Read on to learn about Alerting enhancements like a WeCom contact point, improved Alert panel and custom mute timings, as well as visualization improvements and details to help you share playlists more easily. In Grafana Enterprise, we’ve made caching more powerful to save you time and money while loading dashboards, boosted database encryption to keep secrets safe in your Grafana database, and made usability improvements to Recorded Queries, which allow you to track any data point over time.

We’ve summarized what’s new in the release here, but you might also be interested in the announcement blog post. If you’d like all the details you can check out the complete [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## OSS

### Ease of use

#### Share playlists

You can now share links to your playlists the same way as with dashboards, to easily set up the same playlist on multiple devices or kiosks.

{{< figure src="/static/img/docs/dashboards/share-playlists-8-4.png" max-width="1200px" caption="Share playlist" >}}

#### `$__interval` and `$__interval_ms` in panel title

We’ve expanded the functionality of this existing and popular feature. You can now use `$__interval` and `$__interval_ms` in panel titles. This new function displays the interval that’s used in a specific panel without requiring edit mode.

{{< figure src="/static/img/docs/panels/interval-8-4.png" max-width="1200px" caption="Time intervals" >}}

#### Accessibility improvements

We’re continuing to improve accessibility across Grafana, particularly focusing on keyboard navigation and screen readers.

- Navigation: We’ve improved our keyboard navigation support in our main navbar, added focus states, and removed keyboard traps.
- General components (tooltips, color pickers, modals, dropdowns, and so forth): we’ve ensured they’re keyboard navigable, improved focus trapping, and improved screen reader support.
- Time series panel: charts are one of our main areas of limited accessibility. As of 8.4, you can also move a panel and make range selections using your keyboard. - Press arrow keys to move the cursor around. - Hold Shift to increase cursor speed. - Hold space to start rage selection.
  You can read our accessibility statement [here](/accessibility/) and reach out to us with accessibility issues using our community Slack or our community forums.

#### New option to configure OpenTelemetry

Grafana is used to visualize traces and metrics, but Grafana itself can be traced as well. For example, users running a Grafana instance can export all the traces of endpoints and database requests to Jaeger, which helps you view all Grafana traffic.

We’re currently using OpenTracing for this, but since the repository is being deprecated, we’ve started our work to migrate to OpenTelemetry and remove OpenTracing. This release is the first step toward this goal. It also adds the option to configure OpenTelemetry instead of OpenTracing.

#### Rotate your encryption key

In Grafana 8.3, we upgraded Grafana to use envelope encryption, which adds a layer of indirection to the encryption process. Instead of encrypting all secrets in the database with a single key, Grafana uses a set of keys, called data encryption keys (DEKs), to encrypt them. These data encryption keys are themselves encrypted with a single key-encryption key (KEK).

As of 8.4, you can rotate your KEK and quickly re-encrypt your DEKs in case a key is compromised.

Envelope encryption is not enabled by default in version 8.4. You can enable it by adding the term `envelopeEncryption` to the list of feature toggles in your [Grafana configuration](/docs/grafana/next/administration/configuration/#feature_toggles?mdm=email), or by sending a request to support if you use Grafana Cloud.

{{< figure src="/static/img/docs/encryption/rotate-encryption-8-4.png" max-width="1200px" caption="Rotate encryption" >}}

### Alerting

#### Support for mute timings

[Mute timings](/docs/grafana/next/alerting/unified-alerting/notifications/mute-timings/) have been a popular request from the community. They are a powerful addition to the new alerting feature set and allow you to suppress specific alerts on a recurring interval or schedule, [contrary to](/docs/grafana/next/alerting/unified-alerting/notifications/mute-timings/#mute-timings-vs-silences) Silences.

Paired with the existing Silences, this gives you even more control over when alerts should be sent and contact points notified.

#### Custom grouping for the Alert Panel

The new Alert Panel displays your alerts and associated alert instances, and supports grouping by one or more custom labels. You can also display all alert instances in an ungrouped list by choosing the custom grouping mode without any configured labels.

Traditionally alerts in the Alert Panel were grouped by the alert rule that created them. When you are monitoring a complex resource like an industrial pump, you typically have multiple alerts defined for that resource to observe different metrics. The new custom grouping feature allows you to view all alert instances for individual resource by specifying a label such as “pump identifier”.

{{< figure src="/static/img/docs/alerting/unified/custom-grouping-8-4.png" max-width="1200px" caption="Custom grouping in alerting" >}}

#### WeCom contact point

Starting with Grafana 8.4, you can configure a [WeCom contact point](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/manage-contact-points/) , to send alert notifications to WeCom.

### New panel options

#### Bar chart

We’ve expanded the bar chart so that you can:

- Use time for the x-axis.
- Color bars using a field property (that is, build success).
- Use labels effectively:
  - Skip values when there are too many labels.
  - Rotate labels.

{{< figure src="/static/img/docs/bar-chart-panel/bar-chart-8-4.png" max-width="1200px" caption="Updated bar chart" >}}

#### Geomap

Geomap now supports tooltips with data-links across multiple layers.

{{< figure src="/static/img/docs/geomap-panel/geomap-tooltips-multiple-layers-8-4.png" max-width="1200px" caption="Assign SAML users role" >}}

### OpenAPI v2 specification

The HTTP API details are now [specified](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/papagian/api-spec/public/api-merged.json) using OpenAPI v2.

The grafana server serves a [SwaggerUI](https://swagger.io/tools/swagger-ui/) editor via the /swagger-ui endpoint that enables users to make requests to the HTTP API via the browser. This is disabled by default and it’s enabled when the `swaggerUi` feature toggle is set.

## Grafana Enterprise

### Security improvements

#### Role-based access control works for teams

Occasionally, Viewer, Editor, and Admin roles don’t fit what a certain user needs to do in Grafana. Now you can assign role-based roles directly to users so they can create reports, use Explore mode, create data sources, and perform other specific actions in Grafana. Role-based access control is currently in beta.

In Grafana 8.4, you can assign roles to teams, which apply to all members of that team. This is a convenient way to grant certain permissions to a group of users all at once. It also makes permissions easier to manage when you synchronize groups from an SSO provider, like Google Oauth or Okta, to teams in Grafana.

In 8.4 you can also control access to Team and API key functionality itself, like viewing or editing API keys and adding members to certain teams.

Enable role-based access control by adding the term `accesscontrol` to the list of feature toggles in your [Grafana configuration](/docs/grafana/next/administration/configuration/#feature_toggles?mdm=email), or by sending a request to support if you use Grafana Cloud. Learn more about role-based access control in the [role-based access control section of the docs](/docs/grafana/next/enterprise/access-control/).

{{< figure src="/static/img/docs/enterprise/8-4-fine-grain-access-control.png" max-width="1200px" caption="Assign SAML users role" >}}

#### Assign SAML users different roles in different Organizations

You can use Grafana SAML integration to map organizations in your SAML service to [Organizations](../../setup-grafana/configure-access/configure-authentication/saml/#configure-organization-mapping) in Grafana so that users who authenticate using SAML have the right permissions. Previously, you could only choose a single role (Viewer, Editor, or Admin) for users, which would apply to all of their Organizations. Now, you can map a given SAML user or org to different roles in different Organizations, so that, for example, they can be a Viewer in one Organization and an Admin in another.

Additionally, you can now grant multiple SAML organizations access to Grafana, using the `allowed_organizations` attribute. Previously, you could only map one.

{{< figure src="/static/img/docs/enterprise/8-4-SAML-auth.png" max-width="1200px" caption="Assign SAML users role" >}}

Learn more in our [SAML docs](../../setup-grafana/configure-access/configure-authentication/saml/).

### Performance improvements

#### Recorded queries is more stable and usable

We’ve made stability and usability improvements to Recorded Queries, and removed the feature flag so you can get started with Recorded Queries out of the box.

#### Measure query cache hit rate and clear the cache

[Query caching](/blog/2021/09/02/reduce-costs-and-increase-performance-with-query-caching-in-grafana-cloud/) significantly reduces load times and costs of Grafana dashboards, by temporarily storing query results in a cache. Now you can measure the hit rate of your query cache, to see how many queries (and therefore how much time and money) it is saving. You can also use these measurements to tune the cache time to live (TTL), to balance performance and up-to-date data.

You can also now manually clear the cache for a given data source in case data becomes stale, so that the next set of queries run against the data source itself.

To learn more, refer to [query caching in the Grafana Enterprise docs](/docs/grafana/next/enterprise/query-caching/)

{{< figure src="/static/img/docs/enterprise/8-4-query-caching.png" max-width="1200px" caption="Grafana Enterprise query caching" >}}

# What's new in Grafana v8.5

We’re excited to announce Grafana v8.5, with a variety of improvements that focus on Grafana’s usability, performance, and security.

We’ve summarized what’s new in the release here, but you might also be interested in the announcement blog post as well. If you’d like all the details you can check out the complete [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## OSS

### Alerting - group names for Grafana-managed alert rules

It’s been tricky to work with more than a small number of Grafana-managed alert rules without groups in namespaces. They’ve also been inconsistent with the [Prometheus Alert Generator Compliance Specification](https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md), which made working with Grafana-managed and Prometheus-managed alerts a confusing experience. With this release, you can still see flattened Grafana-managed alerts in the “List” tab, or use the new “Grouped” tab to work with groups:

Choose useful group names, and move alert rules between groups.

{{< figure src="/static/img/docs/alerting/unified/rule-grouping-8-5.png" max-width="400px" caption="Rule group" >}}

Rules in a group are evaluated together, so you can also set the interval for the entire group.

{{< figure src="/static/img/docs/alerting/unified/rule-grouping-details-8-5.png" max-width="350px" caption="Rule group details" >}}

### Analytics

You can now enable Grafana version update checking and Grafana plugins version update checking separately using [a new configuration option to separately enable plugin version update checks](/docs/grafana/latest/administration/configuration/#check_for_plugin_updates). The [prior version update checking configuration](/docs/grafana/latest/administration/configuration/#check_for_updates) now only controls Grafana version update checks.

When enabled, a check runs every 10 minutes. It will notify, via the UI, when a new version is available. The checks will not prompt any auto-updates of the Grafana software, nor will it send any sensitive information.

Plugin update available:

{{< figure src="/static/img/docs/analytics/github-plugin-update-available-8-5.png" max-width="450px" caption="Grafana-update-available" >}}

Grafana update available:

{{< figure src="/static/img/docs/analytics/github-plugin-version-8-5.png" max-width="450px" caption="Grafana-update-available" >}}

### Dashboard Panels

In addition to RSS feeds, the News panel now supports Atom feeds, allowing you to display a wider range of data and information in Grafana.

#### Scrolling in the Bar gauge panel

The Bar gauge panel now supports scrolling to support displaying large datasets while maintaining the readability of labels. You can set a min width or height for the bars (depending on the chart’s orientation), allowing the content to overflow in the container and become scrollable.

{{< figure src="/static/img/docs/bar-gauge-panel/vertical-view-8-5.png" max-width="400px" caption="Vertical view" >}}

{{< figure src="/static/img/docs/bar-gauge-panel/horizontal-8-5.png" max-width="400px" caption="Horizontal view" >}}

### Transformations

#### Template variable substitution

We’ve added support to substitute template variables to transformations. This allows dashboards to be more interactive with transformations when a user inputs calculations and `$__interval` and `$__interval_ms`.

#### Grouping to matrix

A new transformation is available that helps you structure data in a matrix format, using the Grafana table plugin.

### Expanding the navigation bar (Beta)

Available by switching on the ‘newNavigation’ feature toggle.
You can expand the navigation bar for a better overview of Grafana’s features and installed integrations.
This feature is currently in a beta version and we would appreciate your feedback. Sign up for a call with the Grafana team - it only takes 30 minutes, and you'll receive a $40 USD gift card as a token of appreciation for your time.
US, UK, Canada, Australia, France, Germany, or South Africa: sign up [here](https://www.userinterviews.com/projects/Hz3DyNNwWA/apply);
Everywhere else in the world: sign up [here](https://www.userinterviews.com/projects/Hz3DyNNwWA/apply).

{{< figure src="/static/img/docs/navigation/new-navigation-8-5.png" max-width="400px" caption="New nav panel" >}}

### Notifications list for error alerts (Beta)

Available by switching on the ‘persistNotifications’ feature toggle.
In order to support debugging issues in Grafana, error alerts that appear when viewing a dashboard now include a trace ID, and these alerts can be accessed under Profile / Notifications.

{{< figure src="/static/img/docs/navigation/nav-profile-notification-8-5.png" max-width="200px" caption="New nav panel" >}}

### Service accounts (beta)

Service accounts are a major evolution for machine access within Grafana. You can create multiple API tokens per service account with independent expiration dates, and temporarily disable a service account without deleting it. These benefits make Service Accounts a more flexible way for Terraform and other apps to authenticate with Grafana. Service accounts also work with [fine-grained access control](/docs/grafana/latest/enterprise/access-control/) in [Grafana Enterprise](/docs/grafana/latest/enterprise/): you can improve security by granting service accounts specific roles to limit the functions they can perform. Service accounts are available in beta; you can try them out by enabling the `service-accounts` [feature toggle](/docs/grafana/v8.5/administration/service-accounts/enable-service-accounts) or, if you use Grafana Cloud, [reaching out to our support team](/orgs/raintank/tickets#) for early access. Learn more about Service Accounts in our [docs](/docs/grafana/latest/administration/service-accounts).

{{< figure src="/static/img/docs/service-accounts/configure-8-5.png" max-width="400px" caption="Configure service accounts" >}}

### Observability

#### Trace to Logs for Splunk

With Trace to Logs, you can view relevant logs for a trace or span with one click. You can now link to Splunk logs from your tracing datasource. In your tracing datasource, configure Trace to Logs by selecting the Splunk datasource and relevant query options like tags to include in the query.

### Experimental Explore to Dashboard workflow

Allows users to create panels directly from within explore.

All queries in Explore get copied over to the new panel, the panel type is automatically selected based on queries’ response
With multiple queries, it will view the response from the first, non hidden query to determine the visualization type

This feature is behind the `explore2Dashboard` feature toggle and is enabled by default.

## Grafana Enterprise

### Security

#### Fine-Grained Access Control comes to Alerting (beta)

Check the Grafana Enterprise / Security section below for more details, including how to enable this beta feature; we’ve implemented
[fine-grained access control](/docs/grafana/latest/enterprise/access-control/) for alerting rules, notification policies, and contact points in [Grafana Enterprise](/docs/grafana/latest/enterprise/). You can turn on fine-grained access control using the `accesscontrol` [feature toggle](/docs/grafana/latest/enterprise/access-control/#enable-fine-grained-access-control), or by [reaching out to our support team](/orgs/raintank/tickets#) for early access if you use Grafana Cloud. For more information on fine-grained access control, visit our [docs](/docs/grafana/latest/enterprise/access-control/).

{{< figure src="/static/img/docs/enterprise/fine-grained-access-control-8-5.png" max-width="400px" caption="Fine grained access control" >}}

#### Control access to dashboards, folders, and annotations (beta)

You can now use fine-grained access control to manage which specific users, teams, and roles can create, read, update, or delete dashboards, folders, or annotations. These are the latest services to incorporate fine-grained access control, which helps you dial in the specific access your users should have in Grafana. Fine-grained access control is currently in beta, but general availability is just around the corner, planned for our 9.0 release. You can turn on fine-grained access control using the `accesscontrol` [feature toggle](/docs/grafana/latest/enterprise/access-control/#enable-fine-grained-access-control), or by [reaching out to our support team](/orgs/raintank/tickets#) for early access if you use Grafana Cloud. For more information on fine-grained access control, visit our [docs](/docs/grafana/latest/enterprise/access-control/).

{{< figure src="/static/img/docs/enterprise/configure-role-access-8-5.png" max-width="400px" caption="Configure role access" >}}

#### Configure Azure Key Vault using Managed Identities

You can already keep secrets in Grafana’s database (like data source credentials) safer by retrieving your database encryption key from a Key Management Service, like AWS KMS or Azure Key Vault. In Grafana v8.5, you can use an Azure Managed Identity to integrate with Azure Key Vault. This simplifies the Key Vault integration and keeps it consistent with Grafana data sources, like Azure Data Explorer.

### Configure reports more easily

Reports are a great way to share Grafana dashboards by email with users who don’t regularly sign in to Grafana. In 8.5, we’ve revamped the Report authoring UI to make it quicker and easier for you to create reports. View report details at a glance in list view, consider one configuration step at a time, and save reports for later. Also, Grafana will now emit a log every time a report is sent, so you can confirm its status or learn about send errors. Learn more about Reporting in our [docs](/docs/grafana/latest/enterprise/reporting/).

{{< figure src="/static/img/docs/enterprise/report-new-report-8-5.png" max-width="350px" caption="New report" >}}

# What's new in Grafana v9.0

As tradition goes, GrafanaCon — our yearly community event for Grafana open source users — is also where we launch the latest software release of Grafana. Keeping up with tradition, we are excited to be announcing Grafana v9.0: a release that elevates Grafana’s ease of use, discovery of data through new and improved visualizations and a default Grafana Alerting experience.

A big focus for Grafana is making observability and data visualization and analytics easier and more accessible for everyone. For popular data sources like Prometheus and Loki, writing and understanding queries can be hard. This is why we are excited to announce that Grafana 9 comes with new visual query builders for both these data sources. These visual query builders will lower the barrier of entry and they help anyone to compose, understand and learn how the underlying query languages.

The release also includes a brand-new powerful and fast heatmap visualization, a more accessible navigation menu, improvements to dashboard search, advanced security and authentication features, and more.

We’ve summarized what’s new in the release here. If you’d like all the details you can check out the complete [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## Prometheus query builder

Prometheus queries are not the easiest to write or understand. The PromQL query language is very complex and advanced. Even relatively simple queries are hard to compose and read for new users. This is why we are introducing a new UI query builder that allows anyone to compose and learn how Prometheus queries work.

{{< figure src="/static/img/docs/prometheus/prom_query_builder_light.png" max-width="750px" caption="Prometheus query builder" >}}

### Metric and label filters

This new query builder allows you to easily search and select a metric as well as building label filters. You can start either the selecting a metric or a label filter as they both act as filters on each other. The metric selector allow you to search on multiple parts of the metric name at the same time, just separate each search string with a whitespace.

### Operations

All functions, aggregations and binary operations are added via the + Operation button. Operations are presented in the order they are executed, not in the inverted order they are written in the text query. This makes queries a lot easier to read and reason about as you now clearly see what function parameter is going to which function as well as information about parameter names and function documentation integrated into the builder.

### Range vector

The query builder will automatically manage and add the range selector. It will be shown as a parameter to the operations that require a range vector (rate, delta, increase, etc).

### Binary operations

Simple binary operations like multiplication or division by a scalar are represented via simple operation boxes like other functions. The builder also supports binary operations on nested query expressions.

### Switch between code and builder

You can switch between the text edit mode (Code) and the UI builder mode while having your text changes preserved as the visual builder model is derived from the text query using a full featured PromQL query parser.

## Loki query builder

Loki also has complex and powerful query language in LogQL. In many ways it’s more complex and has more syntax to remember than PromQL. Luckily the new query builder we are introducing in Grafana v 9.0 will help you write and understand Loki queries without knowing any of the syntax. Take a look at this example log query.

{{< figure src="/static/img/docs/loki/loki_query_builder_log_query_light_theme.png" max-width="750px" caption="Loki query builder" >}}

In the query builder you can add and edit label filters, line filters, parsers and functions. The image above shows a log query but you can also do metric queries.

{{< figure src="/static/img/docs/loki/loki_query_builder_log_query_light_theme.png" max-width="750px" caption="Loki query builder" >}}

The Loki query builder supports all the features listed for the prometheus query builder above, including support for nested binary operations, explain mode and switching between text editor and builder while preserving changes.

## New heatmap panel

We are replacing the old heatmap panel with a new modern panel that is using the new panel option architecture.

{{< figure src="/static/img/docs/heatmap-panel/heatmap_panel.png" max-width="500px" caption="New heatmap panel" >}}

New new heatmap panel has a number enhancements compared to the old version.

- Multiple orders of magnitude faster
- Supports showing exemplars (traces) overlay
- Supports Prometheus sparse histograms
- Supports changing number of colors steps
- For unbucketed data, it performs smarter auto bucket sizing
- Supports filtering out bucket values close to but not exactly zero

The new heatmap by default assumes that the data is pre-bucketed. So if your query returns time series each series is seen as separate bucket (y axis tick). The panel is so much faster than the old one so it can render many time series with thousands of data points each without issue.

{{< figure src="/static/img/docs/heatmap-panel/heatmap_with_time_series_light_theme.png" max-width="500px" caption="Heatmap panel with time series" >}}

## Grafana Alerting is the default in Grafana 9

Grafana Alerting is now on by default if you upgrade from an earlier version of Grafana. If you have been using legacy alerting in an earlier version of Grafana and you upgrade to Grafana 9 your alert rules will be automatically migrated and the legacy alerting interface will be replaced by the Grafana Alerting interface.

Grafana Alerting, called unified alerting in Grafana 8, has been available since June, 2021 now provides feature parity with legacy alerting and many additional benefits. To find out more on the process to revert back to legacy alerts if needed, click [here](https://grafana.com/docs/grafana/v10.2/alerting/set-up/migrating-alerts/#opt-out). Note that if you do revert back (by setting the Grafana config flag GF_UNIFIED_ALERTING_ENABLED to false), that we expect to remove legacy alerting in the next major Grafana release, Grafana 10.

### Alert state history for Grafana managed alerts

Alert state history for Grafana managed alerts indicates the cause of the state. Alert rules can be configured to use the "Alerting" or "OK" states when rule evaluations result in Error or NoData conditions. This change tracks the difference between, for example, Alerting, Alerting (NoData), and Alerting (Error). This change also updates the UI to handle the new serialized states: filtering, sorting, and coloring.

### Images in notifications for Grafana managed alerts

Grafana Alerting now provides the ability to capture an image for Grafana managed alerts and deliver it via a notification. This is configurable via the contact points tab of Grafana Alerting . This functionality was available in legacy dashboard alerting, but was missing in Grafana Alerting. It requires the alert to be related to a specific dashboard panel using the “Dashboard UID” and “Panel ID” annotations.

## Envelope encryption is generally available and enabled by default

Grafana now uses envelope encryption to encrypt secrets in the database. Envelope encryption adds a layer of indirection to the encryption process. Instead of encrypting all secrets in the database with a single key, Grafana uses a set of keys called data encryption keys (DEKs) to encrypt them. These data encryption keys are themselves encrypted with a single key encryption key (KEK). This allows you to rotate your KEK and quickly re-encrypt your DEKs in case a key is compromised. In Grafana Enterprise, you can also store your KEK in an external key management service (KMS) like AWS KMS or Azure Key Vault, for extra security. Learn more about envelope encryption in the [database encryption documentation](../../setup-grafana/configure-security/configure-database-encryption/#envelope-encryption).

{{< figure src="/static/img/docs/enterprise/dashboards/envelope-encryption.png" max-width="750px" caption="Envelope encryption" >}}

## Dashboard previews (opt-in beta)

The dashboard previews initially came to life during our internal [Grafana Labs Hackathon](/blog/2022/01/14/all-about-the-grafana-labs-hackathon-2.0/?), and are now ready for an opt-in beta release. The previews provide an overview of all available dashboards and help you quickly find the right dashboard when the dashboard names aren’t enough.

{{< figure src="/static/img/docs/dashboards/dashboard-previews.png" max-width="750px" caption="Dashboard previews" >}}

## Panel title search (opt-in beta)

Grafana 9 has a feature toggle `panelTitleSearch`. When enabled, it tells Grafana to use a new search engine. Instead of using SQL queries, the new search uses an in-memory full-text index. That provides a better search experience and additionally allows searching through panel titles.

## Expanding the navigation bar

You can expand the navigation bar for a better overview of Grafana’s features and installed integrations. This is the first improvement to Grafana’s navigation, with more to come in future releases.

{{< figure src="/static/img/docs/expanding-nav-bar-9-0.png" max-width="750px" caption="Expanding nav bar" >}}

## Starred dashboards in the navigation bar

As part of the upcoming improvements to Grafana’s navigation, you can now directly access your starred dashboards from the navigation bar. This is currently an opt-in feature that can be accessed by enabling the savedItems feature flag.

{{< figure src="/static/img/docs/dashboards/starred-dashboards-nav-bar.png" max-width="750px" caption="Starred dashboards" >}}

## Explore to Dashboard

It is now possible to create panels and/or dashboards directly from Explore. When clicking on the “Add to dashboard” button in the Explore toolbar users can now create a panel in a new dashboard or in an existing one. The generated panel contains all the pane’s queries and a default visualization automatically picked from the current results shown in Explore. Dashboards are not automatically saved so that users can go through the current save flow. Optionally, dashboards can be opened in a new tab to preserve Explore’s state.

{{< figure src="/static/img/docs/dashboards/explore-to-dashboard-9-0.png" max-width="500px" caption="Explore to dashboard" >}}

## Command palette

Using cmd+k (macOS) or ctrl+k (Linux/Windows), users can pull up a palette of commands that allow easier navigation and other tasks. <!-- Currently available commands:-->

## List of errors with trace IDs

In order to support debugging issues in Grafana, error alerts that appear in Grafana include a trace ID, and these alerts can be accessed under your Profile. This is currently an opt-in feature that can be accessed by enabling two feature flags:

- `tracing` - enables the tracing id in error messages
- `persistNotifications` - enables the alerts page under Profile

{{< figure src="/static/img/docs/dashboards/errors-with-trace-ids.png" max-width="500px" caption="Errors with trace IDs" >}}

## Query History migration

Query history is a panel that keeps track of all queries that are run in Explore. So far the history was saved in browser local storage causing some problems, for example running out of storage space or not being able to use the same query history in different browsers. Not anymore! We’re migrating all entries from browser local storage to the database. Migration will happen behind the senses when query history panel is opened. This feature can be disabled using config entry query_history in custom.ini.

{{< figure src="/static/img/docs/panels/query-history-9-0.png" max-width="900px" caption="Query history" >}}

## Enterprise

### Access control

### RBAC is generally available

Fine-grained access control is now called role-based access control, or RBAC. It is now enabled by default in Grafana Enterprise and Grafana Cloud, so you can create and define custom roles like Annotations Writer, Users Viewer, and Dashboard Permissions Writer for users, teams, or service accounts. Learn more about RBAC in the [docs](/docs/grafana/next/enterprise/access-control/).

{{< figure src="/static/img/docs/enterprise/rbac-ga.png" max-width="755px" caption="RBAC is GA" >}}

## Reporting

### Add multiple dashboards to a single report

If you have information spread across several dashboards that you’d like to send out to a group of recipients periodically, you can now add multiple dashboards to a single report and send it all in one email. Learn more about Reporting in the [Reporting docs](/docs/grafana/latest/enterprise/reporting/#reporting).

### Embed an image of a dashboard in a report

Reporting is all about convenience - getting info to eyeballs as quickly as possible. Now you can embed an image of a dashboard directly within a report email, so your recipients can see it right when they open the message instead of opening an attached PDF. This is especially handy for SLOs, financial results, or other at-a-glance data. Learn more about Reporting in the [Reporting docs](/docs/grafana/latest/enterprise/reporting/#reporting).

{{< figure src="/static/img/docs/dashboards/report-embedded-image.png" max-width="750px" caption="Embed report" >}}

## Breaking changes

This is a partial list of notable breaking changes. For the complete list, see our [Release Notes](../../release-notes/release-notes-9-0-0/).

### Role-based access control: changes for general release

Fine-grained access control is now called "Role-based access control". As part of the Grafana 9.0 release, the service is generally available, and there are several breaking changes:

- Built-in roles are now called basic roles. They now consist of permissions, not roles.
- The Terraform builtin_role_assignment resource is going away.
- Grafana provisioning has a new schema. Please refer to the documentation for more information.
- Basic roles no longer support permission inheritance. Previously, when permissions of a Viewer basic role were modified, it was propagated to the Editor and Admin basic roles. With the Grafana 9.0 release, this is not the case anymore.
- Several role-based access control actions have been renamed. All the database entries that use legacy action names will be migrated to use the new names, but provisioning files and scripts will have to be updated by the user. This change also means that if Grafana is downgraded from 9.0 to a lower version, some role-based access control permissions will not be resolved correctly.

### Loki: logs data format changed

In the Loki data source, the data format used to represent Loki logs-data has been changed to a more efficient format. (NOTE: this change applies to logs data only, it does not apply to numeric data)
The logs are represented by a single dataframe with a "labels" field added, instead of separate dataframes for every label combination. Displaying logs data in explore, or in a dashboard using the logs panel will continue to work without changes. But, when displaying logs data in other dashboard panels, for example in a table visualization, changes will be visible, and configurations might need to be adjusted. For example, if the "Labels to fields" transformation was used, it has to be replaced with an “Extract fields” transformation, where the “labels” field is chosen as the source.

### Loki: NaN values representation changed in numeric data

In the Loki data source, when grafana receives numeric data from Loki, it may contain NaN (not a number) values. For consistency and performance reasons we changed how we represent such values in Grafana. In previous versions, the behavior was different between alerting queries and other queries (like dashboard queries or explore queries). Alerting queries kept NaN values unchanged, but other queries converted these values to “null”. Starting with grafana 9.0.0, we will always keep these values unchanged. In other words, queries in dashboards and explore will behave the same as alerting queries in this regard.

### Elasticsearch: Support for versions after their end of life was removed

Support for Elasticsearch versions that are after their end of life ( based on https://www.elastic.co/support/eol ) was removed. This means that versions older than Elasticseach 7.10.0 will not be supported in Grafana 9.0.0.

### Elasticsearch: Support for browser access mode removed

In the Elasticsearch data source, browser access mode was deprecated in grafana 7.4.0 and removed in 9.0.0. If you used this mode, please switch to server access mode on the data source configuration page.

### Prometheus: NaN values representation changed in numeric data

In the Prometheus data source, when grafana receives numeric data from Prometheus, it may contain NaN (not a number) values. For consistency and performance reasons we changed how we represent such values in Grafana. In previous versions, the behavior was different between alerting queries and other queries (like dashboard queries or explore queries). Alerting queries kept NaN values unchanged, but other queries converted these values to “null”. Starting with grafana 9.0.0, we will always keep NaN values unchanged for all queries.

<!-- ### InfluxDB: Support for browser access mode removed (should this stay??)

In the InfluxDB data source, browser access mode was deprecated in grafana 8.0.0 and we are removing this feature in 9.0.0. If you are using this mode, you need to [switch to server access mode](../../datasources/influxdb/##influxql-classic-influxdb-query) on the data source configuration page or you can do this via provisioning. -->

### Transformations: Allow more complex regex expressions in rename by regex

The rename by regex transformation has been improved to allow global patterns of the form `/<stringToReplace>/g`. Depending on the regex match used, this may cause some transformations to behave slightly differently. You can guarantee the same behavior as before by wrapping the match string in forward slashes (`/`), for example, `(.*)` would become `/(.*)/`. ([Github Issue #48179](https://github.com/grafana/grafana/pull/48179))

### Clock Panel

We have updated [clock panel](/grafana/plugins/grafana-clock-panel/) to version `2.0.0` to make it compatible with Grafana 9. The previous version `1.3.1` will cause the Grafana 9 to [crash](https://github.com/grafana/clock-panel/issues/106) when being used in a dashboard, we encourage you to update the panel before migrating to Grafana 9.

### Polystat Panel

We have updated [polystat panel](/grafana/plugins/grafana-polystat-panel/) to version `1.2.10` to make it compatible with Grafana 9. The previous versions `1.2.8` and below will render empty in Grafana 9. We encourage you to update the panel before or immediately after migrating to Grafana 9.

### Plugins: Most relevant breaking changes

- `getColorForTheme` was removed, use `theme.visualization.getColorByName` instead [PR #49519](https://github.com/grafana/grafana/pull/49519)
- `TextDisplayOptions` was removed, use `VizTextDisplayOptions` instead [PR #49705](https://github.com/grafana/grafana/pull/49705)
- We have changed the internals of `backendSrv.fetch()` to throw an error when the response is an incorrect JSON. Make sure to handle possible errors on the callsite where using `backendSrv.fetch()` (or any other `backendSrv` methods) [PR #47493](https://github.com/grafana/grafana/pull/47493)
- We have removed the deprecated `getFormStyles` function from [grafana-ui](https://www.npmjs.com/package/@grafana/ui). Use `GrafanaTheme2` and the `useStyles2` hook instead [PR #49945](https://github.com/grafana/grafana/pull/49945)
- We have removed the deprecated `/api/tsdb/query` metrics endpoint. Use `/api/ds/query` instead [PR #49916](https://github.com/grafana/grafana/pull/49916)

You can find the complete list of breaking changes in the links below. Please check them out for more details and see if there is anything affecting your plugin

- /docs/grafana/next/release-notes/release-notes-9-0-0-beta1/
- /docs/grafana/next/release-notes/release-notes-9-0-0-beta2/
- /docs/grafana/next/release-notes/release-notes-9-0-0-beta3/
- /docs/grafana/next/release-notes/release-notes-9-0-0

### Envelope encryption enabled by default

Grafana v8.5 added a new kind of encryption called "envelope encryption" for secrets stored in the Grafana database, such as data source credentials, alerting notification channel credentials, oauth tokens, etc., behind a feature toggle named `envelopeEncryption`.

In v9.0, the `envelopeEncryption` feature toggle is replaced in favor of `disableEnvelopeEncryption`, and Grafana uses the envelope encryption mechanism by default.

Therefore, any secret created or updated in Grafana v9.0 won't be decryptable by any previous Grafana version, unless the Grafana version is at least v8.5 and the feature toggle `envelopeEncryption` is enabled. If not properly configured, this can cause issues in high availability setups, progressive rollouts, or any situation that involves rolling back to a previous Grafana version.

We strongly recommend that you enable `envelopeEncryption` for older versions to deploy envelope encryption. You can alternatively configure the `disableEnvelopeEncryption` feature toggle before upgrading to v9.0; however, this feature toggle might be removed in a future Grafana release.

For more details, and workarounds for potential issues, refer to the [documentation](../../setup-grafana/configure-security/configure-database-encryption/).

## A note on Grafana Enterprise licensing

When we release Grafana 9.0 on June 14th, Grafana will no longer enforce viewers and editor-admins differently. That means that regardless of whether your Grafana Enterprise license is tiered or combined, instead of seeing this on the Stats & Licensing page:

{{< figure src="/static/img/docs/enterprise/separate-licenses.png" max-width="500px" caption="Separate license" >}}

You will see this:

{{< figure src="/static/img/docs/enterprise/combined-licenses.png" max-width="500px" caption="Combined license" >}}

It also means that Grafana will count all users the same, regardless of their role, including org roles (Viewer, Editor, Admin) and fine-grained roles (Dashboard Editor, Reports Editor, etc.). You won’t see a separate warning banner or see users locked out if you hit your limit of viewers or editor-admins, only your total combined limit of active users.

For example, if you have a license for 10 active admins and 100 active viewers in your Grafana Enterprise license, then starting in v9.0 you will have a limit of 110 active users, and it doesn’t matter what roles those users have, they will all be counted and enforced the same.

This is a more permissive policy than before.

# What's new in Grafana v9.1

We're excited to announce Grafana v9.1, with a variety of improvements that focus on Grafana's usability, performance, and security.
Read on to learn about new options to share and embed dashboards, search and navigation enhancements, new panel options, and additional authentication features.
You can also find out more about new single sign-on and role-based access control options in Grafana Enterprise, and more.
For details, see the complete [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## OSS

### Auth and security

#### Grafana service accounts are generally available

Service accounts are an evolution in machine access within Grafana.
You can create multiple API tokens per service account with independent expiration dates, and temporarily disable a service account without deleting it.
These benefits make service accounts a more flexible way for Terraform and other apps to authenticate with Grafana.

Service accounts also work with [role-based access control](../../administration/roles-and-permissions/access-control/) in [Grafana Enterprise](../../introduction/grafana-enterprise/).
You can improve security by granting service accounts specific roles to limit the functions they can perform.
Service accounts have been in beta since Grafana v8.5.
During that time, we've improved the UI and migration path from API keys, made it possible to add service accounts to teams, and inherit team permissions.
To learn more about service accounts, see the [documentation](../../administration/service-accounts/).

{{< figure src="/static/img/docs/service-accounts/add-service-account-token-9-1.png" max-width="750px" caption="Adding a service account token" >}}

#### JWT URL embedding

You can now easily embed Grafana in other applications by adding a JWT token directly in the Grafana's URL, for example,`https://example.grafana.net/dashboard/uuid?aut_token=<jwt_token>`.
When the JWT token is passed through the request URL to Grafana, Grafana validates and authenticates the token linked to a specific user, allowing access to dashboards which that user can view.
To see JWT URL embedding in action, see the [sample project](https://github.com/grafana/grafana-iframe-oauth-sample).

> **Note:** JWT URL Embedding and `allow_embedding` are not available to Grafana Cloud users. For Grafana Cloud, use [public dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/share-dashboards-panels/shared-dashboards/). To enable that, [open a ticket with our Support team](/docs/grafana-cloud/account-management/support/).

{{< figure src="/static/img/docs/dashboards/jwt-url-embedding-9-1.png" max-width="750px" caption="A JWT token used to embed Grafana" >}}

#### Organization role mapping for GitHub OAuth2 authentication

You can now use GitHub OAuth2 to map users or teams to specific [Grafana organization roles](../../administration/roles-and-permissions/#organization-roles) by using `role_attribute_path` configuration option.
Grafana will use [JMESPath](https://jmespath.org/examples.html) for path lookup and role mapping.
For more information, see the [documentation](../../setup-grafana/configure-access/configure-authentication/github/#map-roles).

Grafana Cloud users can access this feature by [opening a support ticket in the Cloud Portal](/profile/org#support).

{{< figure src="/static/img/docs/permissions/org-role-mapping-github-9-1.png" max-width="750px" caption="Configuring GitHub OAuth2 authentication with role mapping" >}}

### Search and navigation

#### (Beta) Panel title search and search improvements

We've improved the performance of searching by panel title.
If a panel's title matches your search query, it will be displayed in the search results.
This feature will be rolled out to Grafana Cloud users over the course of several weeks, or can be accessed by enabling the `panelTitleSearch` feature toggle.

Panel title search uses our updated dashboard search approach.
Previously, Grafana used SQL database queries to find dashboards by title.
With the feature toggle enabled, Grafana can build an in-memory index of all dashboards.
To learn more about search in Grafana, see the [documentation](../../dashboards/use-dashboards/#dashboard-search).

{{< figure src="/static/img/docs/dashboards/panel-title-search-9-1.png" max-width="750px" caption="Searching for a panel title" >}}

#### Starred dashboards in the navigation bar

As part of the upcoming improvements to Grafana's navigation, you can now directly access your [starred dashboards](../../dashboards/use-dashboards/) from the navigation bar.

{{< figure src="/static/img/docs/dashboards/starred-dashboards-9-1.png" max-width="750px" caption="Accessing your starred dashboards" >}}

### Panels

#### Heatmap improvements

The beta heatmap announced in version 9.0 is now used throughout Grafana.
Its performance is improved, and it now supports [exemplars](../../fundamentals/exemplars/).
To learn more about the heatmap panel, see the [documentation](../../panels-visualizations/visualizations/heatmap/).

{{< figure src="/static/img/docs/panels/heatmap-panel-9-1.png" max-width="750px" caption="A heatmap panel" >}}

#### Geomap

You can now measure both distances and areas on Geomap visualizations by using the panel's new [measure tools](../../panels-visualizations/visualizations/geomap/#show-measure-tools).
To learn more about the Geomap panel, see the [documentation](../../panels-visualizations/visualizations/geomap/).

{{< figure src="/static/img/docs/geomap-panel/geomap-map-controls-measure-area-9-1-0.png" max-width="750px" caption="Geomap panel measuring tool - area" >}}

#### (Beta) Trace to metrics

You can now link metrics queries to your traces.
This feature can be accessed by enabling the `traceToMetrics` feature toggle.
Grafana Cloud users can access this feature by [opening a support ticket in the Cloud Portal](/profile/org#support).

In your tracing datasource configuration, select a metrics datasource, add tags, and write your queries.
Each query appears as a link on each span.
The configured tag values are dynamically added to your metrics queries.
You can link to any metric you'd like.
Metrics for span durations, counts, and errors filtered by service or span are a great starting point.
The metrics generator introduced in Tempo 1.4 pairs extremely well with the trace to metrics feature.
To learn more about the metrics generator, see the [blog post](/blog/2022/05/02/new-in-grafana-tempo-1.4-introducing-the-metrics-generator/).

{{< figure src="/static/img/docs/panels/trace-to-metrics-9-1.png" max-width="750px" caption="Linking a trace to a metrics query" >}}

#### (Beta) APM table

You can now get Application Performance Management (APM) data with Grafana.
The data is shown in a table in the Tempo datasource under the Service Graph tab.
To access the feature, enable the `tempoApmTable` feature toggle.
To receive the data for the APM table, you must also enable the metrics generator.

Grafana Cloud users can access this feature by [opening a support ticket in the Cloud Portal](/profile/org#support).

The APM table displays rate, errors, and duration (RED) metrics.
To view your top five RED span metrics, use the table summary view.
The span metrics are created using the Tempo [metrics generator](/blog/2022/05/02/new-in-grafana-tempo-1.4-introducing-the-metrics-generator/).

We also embedded several links directly into the table.
These links direct you to a Prometheus query to further investigate the data.
We also provide a link from the table directly to Tempo search, making it easier for you to investigate your APM metrics.

To learn more about the APM table, see the [documentation](../../datasources/tempo/#apm-table).

{{< figure src="/static/img/docs/panels/apm-table-9-1.png" max-width="750px" caption="An APM table in the Explore view" >}}

### Sharing

#### (Alpha) Public dashboards

Public dashboards are available as an alpha feature that can be enabled with the `publicDashboards` feature toggle.
Grafana Cloud users can access this feature by [opening a support ticket in the Cloud Portal](/profile/org#support).

You can generate a link for dashboards that you'd like to share publicly.
Anyone with the link will be able to access that dashboard, and nothing else.

The public view of a dashboard has a few restrictions:

- Arbitrary queries cannot be run against your datasources through public dashboards.
  Public dashboards can only execute the queries stored on the original dashboard.
- The public dashboard is displayed in a read-only kiosk view.
- The time range is fixed to the dashboard default time range.

To learn more, see the [documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/share-dashboards-panels/shared-dashboards/).

#### Provisioning improvements for Grafana Alerting

You can provision your Grafana Alerting resources directly from disk, which is a process you might already follow for your dashboards or data sources.
Grafana Alerting resources are provisioned once when you initially configure Grafana.

Provisioning for Grafana Alerting currently supports these resources:

- Alert rules
- Contact points
- Notification policies
- Mute timings
- Text templates

For more information, see the
[provisioning documentation](../../administration/provisioning/).

## Grafana Enterprise

### Sharing and customization

#### Reporting improvements

Reporting is better in a few specific ways in Grafana version 9.1:

- You can save a draft of a report, in case you need to make a quick update to a dashboard or make other changes before publishing the report.
- Each page of a report PDF includes the dashboard's name, which is useful for multi-page reports.
- You can send the same dashboard in a report multiple times with different time ranges.
  For example, you can share last month's numbers as compared to the numbers for this month.
  The dashboard uses the same template variables if you attach the dashboard to a report twice.

To learn more about reporting, see the [documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/create-reports/).

{{< figure src="/static/img/docs/enterprise/reporting-draft-9-1.png" max-width="750px" caption="Saving a report as a draft" >}}

#### (Beta) Configure custom branding in Grafana's UI

Custom branding (previously referred to as whitelabeling) lets you customize parts of Grafana's UI.
You can add links to the footer to your internal documentation, guides, or support, and you can update Grafana's sign-in page, logo, and other graphic elements to reflect your team or company identity.

Previously, you could only configure custom branding in Grafana's configuration files.
Now, you can experiment with customization in Grafana's Admin section in the UI, or customize branding and links using the API.
This is an early-access feature available only to self-managed customers.

{{< figure src="/static/img/docs/enterprise/custom-branding-9-1.png" max-width="750px" caption="Configuring custom branding" >}}

### Authentication and security

#### RBAC for app plugins, usage insights, and query caching

As part of our continued improvements to role-based access control (RBAC), we are rolling out RBAC across all of Grafana's features.
In v9.1, you can determine which users, teams, and roles can access app plugins like OnCall and Synthetics.

> **Note:** You can't yet define view or edit roles for app plugins.
> For example, you cannot yet grant a user view-only access to OnCall.
> This definition is planned for a future release.

You can also control who can view, edit, or administer dashboard and data source usage insights, as well as data source query caching configuration.
For more details, see the [RBAC documentation](../../administration/roles-and-permissions/access-control/).

{{< figure src="/static/img/docs/enterprise/rbac-app-plugins-9-1.png" max-width="750px" caption="Configuring role-based-access to app plugins" >}}

#### Rotate your database encryption keys using Grafana's API

In Grafana version 9.0, we [revamped](../whats-new-in-v9-0/#envelope-encryption-is-generally-available-and-enabled-by-default) the method Grafana uses to encrypt secrets, like data source credentials, so that you can rotate encryption keys and integrate with a key management system like Hashicorp Vault, AWS Key Management Service, or Azure Key Vault.

Now, you can rotate keys and re-encrypt secrets via API.
This makes it easier to configure Grafana to be secure while deploying it.
To learn more, see our guide to [configuring database encryption](../../setup-grafana/configure-security/configure-database-encryption/#configure-database-encryption).

#### Audit all actions in Grafana using verbose logging

Auditing logs helps you manage and mitigate activity and meet compliance requirements.
By default, Grafana emits an audit log with every action that changes something (like a user creating or updating a dashboard or updating another user's permissions).
If you want to record all actions on the Grafana server, including GETs and page views, you can now turn on `verbose mode`.
This results in more logs, but it can be useful to debug specific issues or make sure you catch everything happening in Grafana for security or compliance reasons.
To learn more, see the [configuration documentation](../../setup-grafana/configure-grafana/enterprise-configuration/#verbose).

{{< figure src="/static/img/docs/enterprise/verbose-audit-logs-9-1.png" max-width="750px" caption="Verbose audit logging output" >}}

#### See (and don't edit) users synced from SAML, LDAP, and OAuth identity providers

When you synchronize users from a SAML, LDAP, or OAuth provider, some user settings, such as name and email address, are synchronized from your identity provider.
Previously, you could edit those settings in the Grafana UI, but they would revert back.
To make user management clearer, you can now see which settings are synchronized from your identity provider, but you cannot edit those settings.
To learn more about authentication, see the [documentation](../../setup-grafana/configure-access/configure-authentication/).

{{< figure src="/static/img/docs/enterprise/oauth-synced-user-9-1.png" max-width="750px" caption="Non-interactive view of a user synced via OAuth" >}}

#### Support wildcards for LDAP groups in team sync

Team sync lets you set up synchronization between your authentication provider's teams and Grafana teams.
To leverage your existing Active Directory in an efficient way without having to create multiple teams, you can now use wildcards when configuring LDAP groups, so that multiple Active Directory groups can be added at once to a single team.

{{< figure src="/static/img/docs/enterprise/ldap-wildcard-teamsync-9-1.png" max-width="750px" caption="Using wildcard mapping for team sync" >}}

#### Redirect binding support for AzureAD SAML Single Logout

SAML Single Logout allows users to log out from all applications associated with the current IdP (Identity Provider) session established via SAML Single Sign-On (SSO).
To enable integration of more use cases, we have added support for an HTTP-Redirect binding for a Single Logout.

# What’s new in Grafana v10.0

Welcome to Grafana 10.0! Read on to learn about changes to search and navigation, dashboards and visualizations, and security and authentication.

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v10.0, check out our [Upgrade Guide](https://archive.grafana.com/docs/grafana/v10.0/upgrade-guide/upgrade-v10.0/).

<!-- Template below
## Feature
<!-- Name of contributor -->
<!-- [Generally available | Available in private/public preview | Experimental] in Grafana [Open Source, Enterprise, Cloud Free, Cloud Pro, Cloud Advanced]
Description. Include an overview of the feature and problem it solves, and where to learn more (like a link to the docs).
{{< admonition type="note" >}}
You must use relative references when linking to docs within the Grafana repo. Please do not use absolute URLs. For more information about relrefs, refer to [Links and references](/docs/writers-toolkit/writing-guide/references/).
{{< /admonition >}}
-->

## Breaking changes

For Grafana v10.0, we've also provided a list of [breaking changes](https://archive.grafana.com/docs/grafana/v10.0/breaking-changes/breaking-changes-v10-0/) to help you upgrade with greater confidence. For information about these along with guidance on how to proceed, refer to [Breaking changes in Grafana v10.0](https://archive.grafana.com/docs/grafana/v10.0/breaking-changes/breaking-changes-v10-0/).

## Correlations

<!-- Jay Goodson -->

_Available in public preview in all editions of Grafana._

You can now bring context from multiple data sources into the Explore experience. Correlations is an extension of our existing data links functionality and now enables you to link from any data source to any other data source.

Correlations enable you to seamlessly jump from one data source to another. You define relationships between your different data sources; when you're using Explore, simply click a button next to a related field in one data source and Grafana will run the corresponding query in the other data source.

Correlations is currently in preview. As such, we recommended you only enable it on test or development instances, rather than in production environments.

To try out Correlations, enable the `correlations` feature toggle. If you’re using Grafana Cloud and would like to enable this feature, please contact customer support.

In subsequent releases, we’ll be refining and enhancing the user interface for Correlations, to provide a more streamlined user experience.

## Scenes

<!--Dominik Prokop & Natalia Bernarte -->

_Available in public preview in all editions of Grafana._

Scenes is a new front-end library by Grafana that empowers Grafana plugin developers to effortlessly build stunning dashboard-like experiences into their Grafana app plugins. With Scenes, you can easily create apps that mirror the Grafana dashboarding experience, complete with template variable support, flexible layouts, dynamic panel rendering, and so much more.

To try it out, go to the [@grafana/scenes](https://github.com/grafana/scenes) repository.

To learn more, refer to the [Scenes documentation](https://grafana.github.io/scenes/).

## Subfolders

<!-- Zsofia K. -->

_Available in public preview in all editions of Grafana._

You can now try out creating subfolders in Grafana for organizing your dashboards and alerts. You can enable this new feature in your development environment to create, read, update, and delete subfolders, making it easier to sort resources by business units, departments, and teams.

You can also set up permissions using Role-Based Access Control (RBAC). Folder permissions cascade, being inherited from the parent folder, which simplifies access management.

The ability to add subfolders is currently in preview, with more functionality coming in subsequent releases. This includes creating subfolders using Terraform, and displaying the full folder tree when creating and moving resources through Grafana’s UI. We recommend that you enable this feature only on test or development instances, rather than in production environments.

To get started creating subfolders, enable the `nestedFolders` feature toggle. If you’re using Grafana Cloud, and would like to enable this feature, please contact customer support.

{{< figure src="/media/docs/grafana/screenshot-grafana-10.0-nested-folders-2.png" max-width="750px" caption="Subfolders in Grafana" >}}

## Dashboards and visualizations

### The Canvas panel is GA

_Generally available in all editions of Grafana._

<!-- Nathan Marrs -->

We're promoting the canvas panel out of public preview and into general availability. Over the past several months we've introduced multiple enhancements to the panel such as the ability to draw connections between elements, the ability to set the color and size of connections based on data, and the ability to add data links. We're excited to include Canvas as a first class citizen in Grafana’s core panel library. To learn more about the panel, refer to our [Canvas documentation](../../../panels-visualizations/visualizations/canvas/). Also, check out our [latest blog post about canvas](https://grafana.com/blog/2023/05/11/use-canvas-panels-to-customize-visualizations-in-grafana/).

{{< figure src="/media/docs/grafana/screenshot-grafana-10-0-canvas-service-graph.png" max-width="750px" caption="Canvas service graph" >}}

### New Trend Panel

<!-- Nathan Marrs -->

_Experimental in all editions of Grafana_

The Trends panel allows you to display trends where the x-axis is numeric and not time. This experimental panel addresses gaps that were not solved by either the Time series or XY Chart panels. For example, you can plot function graphs, rpm/torque curves, supply/demand relationships, and more. To learn more about the Trend panel, refer to the [Trend documentation](../../../panels-visualizations/visualizations/trend/).

{{< figure src="/media/docs/grafana/screenshot-grafana-10-0-trend-panel-new-colors.png" max-width="750px" caption="Trend engine power and torque curves" >}}

### New Datagrid panel

<!-- Victor Marin -->

_Experimental in all editions of Grafana_

Datagrid is a new panel that allows you to edit your data within a Grafana dashboard. Imagine having a spreadsheet-like view where you can fine-tune data pulled from a data source or create your own dataset from scratch and use it within your dashboard to update your panels in real time. That's what Datagrid provides. You can also use the Datagrid panel as a data source used by other panels to augment other data.

Currently, the Datagrid Panel supports the following features in Grafana version 10.0:

- Creating and deleting rows and columns
- Data and column header edit or delete
- Search functionality
- Column freezing
- Grid selection actions (copy/paste/delete)
- Draggable columns and rows
- Series selection when pulling data from a data source

In subsequent releases, we’ll continue adding features to the Datagrid panel to further improve the user experience.

### Drag and drop spreadsheets into Grafana

_Experimental in all editions of Grafana_

<!-- Oscar Kilhed -->

It's easier than ever to view local data in Grafana: introducing drag and drop.
The drag and drop functionality allows you to upload your csv, Excel, or numbers files by simply dragging and dropping them into the query editor of the Grafana data source.

To try out drag and drop, enable the `editPanelCSVDragAndDrop` feature toggle.

As of Grafana version 10.0, drag and drop supports the following scenarios:

- Drag and drop files into the panel editor
- Replace files in the panel editor
- Default table panel creation

The data from dragged and dropped files is stored in the dashboard JSON and file size is limited to 1MB. To learn more about drag and drop functionality, refer to the official documentation.

### Select data sources more easily

<!-- Ivan Ortega & Natalia Bernarte -->

_Generally available in all editions of Grafana._

Concepts like data sources and dashboard panels are hard to grasp and it can be a struggle to go from Grafana’s “empty state” to a working dashboard that displays data. Our latest advancements streamline the process of selecting the ideal data source in Grafana, prioritizing recent usage, and providing labels and supplementary descriptions.

With this flow, selecting a data source has been greatly simplified, providing a clear overview of available data sources and allowing you to quickly connect to a new one when needed.

{{< video-embed src="/media/docs/grafana/screen-recording-ds-picker-whats-new-10-final.mp4" max-width="750px" caption="Datasource picker flow" >}}

### Time series time region support

_Generally available in all editions of Grafana._

<!-- Nathan Marrs -->

We've implemented support for adding time regions to the Time series panel. Time regions provide a more contextualized experience, enabling you to highlight certain days of the week, such as Monday to Friday to display work weeks, right alongside your data. Time regions are also a useful way to highlight specific parts of a day like night, work hours, or whatever you want to define for each day. They allow you to quickly orient yourself in parts of the day or ignore highlighted parts of the time series.

To learn more, refer to our [time region documentation](../../../dashboards/build-dashboards/annotate-visualizations/#add-time-regions).

{{< figure src="/media/docs/grafana/screenshot-grafana-10-0-time-regions.png" max-width="750px" caption="Time regions" >}}

### Annotation filtering

_Generally available in all editions of Grafana._

<!-- Nathan Marrs -->

You can now filter dashboard annotations to apply annotations to all panels or selected panels, or use them to exclude selected panels.

To learn more, refer to our [documentation about adding annotation queries](../../../dashboards/build-dashboards/annotate-visualizations/#add-new-annotation-queries).

{{< figure src="/media/docs/grafana/screenshot-grafana-10-0-annotation-filtering.png" max-width="750px" caption="Annotation filtering" >}}

### Redesigned and improved log context

_Generally available in all editions of Grafana._

<!-- Sven Grossman -->

We've made enhancements to Grafana's log context feature, resulting in a more seamless and consistent user experience. With the updated user interface, you can expect the same level of functionality and usability in log context as you would in any other logs panel.

Notably, we've added the following new features that streamline the log context experience:

- Log details with actions, including a **Copy** button, to easily copy lines, and an eye icon to display only selected labels, allowing you to focus on specific information without leaving the log context section.
- A **Wrap Lines** toggle to automatically wrap long lines of text for easier reading and analysis of log entry context directly in log context.
- An **Open in split view** button to execute the context query for a log entry in a split screen in Explore.
- Only for Loki: A quick-filter menu that lets you easily refine the context query by selecting and removing labels.

{{< figure src="/media/docs/grafana/log-context-loki-2-whats-new-10-0-.png" max-width="750px" caption="Grafana Log Context" >}}

These improvements make working with log context in Grafana more intuitive and efficient, ultimately improving the overall user experience.

### Query multiple data sources in Explore

<!-- Piotr Jamroz -->

_Generally available in all editions of Grafana._

You can now query multiple data sources simultaneously in Explore. Select "Mixed" from the data source picker and specify a data source for each query.

If you’re using Grafana Open Source or Enterprise, you can disable this feature using the `exploreMixedDatasource` feature toggle.

## Public dashboards

<!-- Juani Cabanas & Ezequiel Victorero & Natalia Bernarte -->

_Available in public preview in all editions of Grafana._

Public dashboards allow you to share your Grafana dashboard with anyone without requiring them to log in to Grafana. This is useful when you want to make your dashboard available to the world.

With this update, we've made the following improvements:

- The time picker and annotations can be toggled on or off in public dashboard configuration.
- You can see a list of all your public dashboards in **Dashboards > Public dashboards**.
- The user interface has been improved with a new modal design, as well as paused and not found pages.
- Added support for collapsed rows, hidden queries, and zoom into panels.

To try it out, enable the `publicDashboards` feature toggle. If you’re using Grafana Cloud, and would like to enable this feature, please contact customer support.

To learn more, refer to our [public dashboards documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/share-dashboards-panels/shared-dashboards/).

### Public dashboards insights

<!-- Juani Cabanas & Ezequiel Victorero & Natalia Bernarte -->

_Available in public preview in Grafana Enterprise, Cloud Pro, and Cloud Advanced._

Public dashboards insights provide valuable information about your public dashboard usage. You can easily access and view important metrics such as the daily query count, the number of views in the last 30 days, and the number of errors in the last 30 days.

To try it out, enable the `publicDashboards` feature toggle. If you’re using Grafana Cloud, and would like to enable this feature, please contact customer support.

### Email sharing for public dashboards

<!-- Juani Cabanas & Ezequiel Victorero & Natalia Bernarte -->

_Available in public preview in Cloud Pro and Cloud Advanced._

Our email sharing feature allows you to easily share your public dashboards and make them visible only with specific individuals. When you add their email addresses, they receive a one-time link to access the dashboard. This provides you with greater control over who can view your public dashboards.

We've also added a **Public dashboard users** tab in **Administration > Users** where you can view a list of users who have accessed your public dashboards by way of email sharing.

To try it out, please contact customer support.

{{< admonition type="note" >}}

This feature will have a cost by active users after being promoted into general availability.

{{< /admonition >}}

To learn more, refer to our [public dashboards documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/share-dashboards-panels/shared-dashboards/).

## Authentication and authorization

### Configure your SAML provider in the Grafana UI

<!-- Vardan Torosyan -->

_Generally available in Grafana Enterprise, Cloud Pro, and Cloud Advanced._

You can now configure SAML using our new user interface, making the process easier and more convenient than ever before.
With the new user interface (UI), you can now configure SAML without needing to restart Grafana and you can control access to the configuration UI by using [role-based access control (RBAC)](../../../administration/roles-and-permissions/access-control/), which makes the process much faster and more efficient.

The SAML UI is available in Grafana Enterprise, Cloud Pro, and Advanced. It's user-friendly, with clear instructions and helpful prompts to guide you through the process.

For more information on how to set up SAML using the Grafana UI, refer to [Configure SAML authentication using the Grafana user interface](../../../setup-grafana/configure-access/configure-authentication/saml-ui/).

### Case-insensitive usernames and email addresses

<!-- Vardan Torosyan -->

_Generally available in all editions of Grafana._

Usernames and email addresses are now treated as case-insensitive, which means that you no longer need to worry about capitalization when logging in or creating an account.

From now on, whether you type your username or email address in uppercase, lowercase, or a combination of both, Grafana will treat them as the same. This simplifies the login process and reduces the risk of typos and identity conflicts when changing authentication providers.

To help you deal with potential user identity conflicts, we've built a [Grafana CLI user identity conflict resolver tool](https://grafana.com/blog/2022/12/12/guide-to-using-the-new-grafana-cli-user-identity-conflict-tool-in-grafana-9.3/), which is available from Grafana version 9.3.

{{< admonition type="note" >}}

If you're running Grafana with MySQL as a database, this change doesn't have any impact as MySQL users were already treated as case-insensitive.

{{< /admonition >}}

## Tracing

### Span filtering for traces

<!-- Timur Olzhabayev -->

_Available in public preview in Grafana Cloud Free, Cloud Pro, and Cloud Advanced._

You can now work much more efficiently with traces that consist of a large number of spans with span filtering.

Span filters exist above the trace view and allow you to filter the spans that are shown in the trace view. The more filters you add, the more specifically span are filtered.

Currently, you can add one or more of the following filters:

- Service name
- Span name
- Duration
- Tags (which include tags, process tags, and log fields)

Span filtering is currently in preview. As such, it's recommended to enable it only on test or development instances, rather than in production environments.

To try it out, enable the `newTraceViewHeader` feature toggle. This feature is enabled by default in Grafana Cloud.

### OpenTelemetry replacing OpenTracing

<!-- Timur Olzhabayev -->

_Generally available in all editions of Grafana._

We've started the work to migrate to OpenTelemetry in Grafana version 8.4; now we're removing OpenTracing and, for those who still have it configured, replacing it under the hood with OpenTelemetry. These changes are backwards compatible, so you don't need to change anything and the feature will continue working as it did before.

## Data sources

### Azure Monitor data source

<!-- Andreas Christou -->

_Generally available in all editions of Grafana._

The Azure Monitor data source now supports visualizing Application Insights Traces. A new query type, `Traces`, has been added to the service list. This can be used against Application Insights resources to query and visualize traces in both a tabular format and using the built-in Traces visualization.

This also includes support for a new Azure API that will correlate trace IDs against all Application Insights resources that are accessible to the principal that the data source is configured with. To support this feature, a new query builder has been added with support for querying the Application Insights resource using an `Operation ID` or visualizing and filtering the data based on the event type and a subset of the properties available on the trace.

### Prometheus dashboard performance improvements

<!-- Galen Kistler -->

_Experimental in Grafana Open Source._

The Prometheus data source now supports delta (incremental) querying, in which values from data frames are cached and leveraged to modify future requests to avoid requesting duplicate values in dashboards with now-relative (that is, any dashboard querying until "now") queries. This feature is disabled by default as it is still experimental, but can be enabled and configured in the Prometheus data source configuration.

This update will reduce network load, and speed up now-relative dashboards, especially for dashboards returning a lot of data.

### Phlare renamed to Grafana Pyroscope

<!-- Andrej Ocenus -->

_Generally available in all editions of Grafana._

We've renamed the Phlare data source _Grafana Pyroscope_ data source as part of the ongoing unification of the Phlare and Pyroscope projects. This data source supports both Phlare and Pyroscope backends. Existing instances of the data source should not be affected. When you create a new instance of the data source, the backend type will be autodetected on the configuration page, or you can select it manually.

### Data plane

<!-- Kyle Brandt -->

_Generally available in all editions of Grafana._

Data types are now being defined to create a data plane layer between producers and consumers of data. By defining data types as part of Grafana's platform, plugin and application developers can use these data types to achieve more reliable interoperability across the platform.

Learn more:

- [Data plane contract - Technical specification](https://grafana.com/developers/dataplane/)
- [Example typed dataframes and Go lib to use them in tests](https://github.com/grafana/dataplane/tree/main/examples)
- [Go library for reading and writing dataplane data](https://github.com/grafana/dataplane/tree/main/sdata)

## Alerting

_All Alerting features are generally available in all editions of Grafana._

### State history view

<!-- Brenda Muir -->

Use the improved State history view to get insight into how your alert instances behave over time. View information on when a state change occurred, what the previous state was, the current state, any other alert instances that changed their state at the same time, as well as what the query value was that triggered the change.

{{< figure src="/media/docs/alerting/state-history.png" max-width="750px" caption="State history view" >}}

### Preview notification templates

<!-- Brenda Muir -->

Preview how your notification templates will look before using them in your contact points.

{{< figure src="/media/docs/alerting/template-preview.png" max-width="750px" caption="Preview notification templates" >}}

## Security

### Trusted Types support

<!-- Tobias Skarhed -->

_Experimental in all editions of Grafana._

Use [trusted types](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/trusted-types) to reduce the risk of XSS vulnerabilities, including the sanitization of third party libraries or plugins that have not explicitly performed sanitization.

To use this feature in report-only mode:

- Enable `content_security_policy_report_only` in the configuration.
- Add `require-trusted-types-for 'script';` to the `content_security_policy_report_only_template`.

To use it in enforce mode:

- Enable `content_security_policy` in the configuration.
- Add `require-trusted-types-for 'script';` to the `content_security_policy_template`.

This is an experimental web technology with limited browser support.

### Private data source connect

<!-- Mitch Seaman -->

_Available in public preview in Grafana Cloud Pro and Advanced._

Some data sources, like MySQL databases, Prometheus instances or Elasticsearch clusters, run in private networks, like on premises networks or virtual private clouds (VPCs) running in AWS, GCP, or Azure.

To query these data sources from Grafana Cloud, you've had to open your private network to a range of IP addresses, a non-starter for many IT Security teams. The challenge is, how do you connect to your private data from Grafana Cloud, without exposing your network?

The answer is Private Data Source Connect (PDC), available now in public preview in Grafana Cloud Pro and Advanced. PDC uses SOCKS over SSH to establish a secure connection between a lightweight PDC agent you deploy on your network and your Grafana Cloud stack. PDC keeps the network connection totally under your control. It’s easy to set up and manage, uses industry-standard security protocols, and works across public cloud vendors and a wide variety of secure networks. Learn more in our [Private data source connect documentation](/docs/grafana-cloud/data-configuration/configure-private-datasource-connect).

## Plugins

### App plugins can start using react-router v6

We've added support for using `react-router` v6 in app plugins. However, we still support the use of `react-router` v5 for plugins that need to support a minimum Grafana version earlier than v10. For more information, refer to our [react-router migration guide](https://grafana.com/developers/plugin-tools/migration-guides/update-from-grafana-versions/migrate-9_x-to-10_x#update-to-react-router-v6).

# What’s new in Grafana v10.1

Welcome to Grafana 10.1! Read on to learn about changes to dashboards and visualizations, data sources, security and authentication and more. We're particularly excited about a set of improvements to visualizing logs from [Loki](https://grafana.com/products/cloud/logs/) and other logging data sources in Explore mode, and our Flame graph panel, used to visualize profiling data from [Pyroscope](https://grafana.com/blog/2023/03/15/pyroscope-grafana-phlare-join-for-oss-continuous-profiling/?pg=oss-phlare&plcmt=top-promo-banner) and other continuous profiling data sources.

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v10.1, check out our [Upgrade Guide](https://archive.grafana.com/docs/grafana/v10.1/upgrade-guide/upgrade-v10.1/).

<!-- Template below
## Feature
<!-- Name of contributor -->
<!-- _[Generally available | Available in private/public preview | Experimental] in Grafana [Open Source, Enterprise, Cloud Free, Cloud Pro, Cloud Advanced]_
Description. Include an overview of the feature and problem it solves, and where to learn more (like a link to the docs).
{{< admonition type="note" >}}
You must use relative references when linking to docs within the Grafana repo. Please do not use absolute URLs. For more information about relrefs, refer to [Links and references](/docs/writers-toolkit/writing-guide/references/).
{{< /admonition >}}
-->
<!-- Add an image, GIF or video  as below

{{< figure src="/media/docs/grafana/dashboards/WidgetVizSplit.png" max-width="750px" caption="DESCRIPTIVE CAPTION" >}}

Learn how to upload images here: https://grafana.com/docs/writers-toolkit/write/image-guidelines/#where-to-store-media-assets
-->

## Dashboards and visualizations

### Flame graph improvements

_Generally available in all editions of Grafana_

<!-- Andrej Ocenas -->

We've added four new features to the Flame graph visualization:

- **Sandwich view**: You can now show a sandwich view of any symbol in the flame graph. Sandwich view shows all the callers on the top and all the callees of the symbol on the bottom. This is useful when you want to see the context of a symbol.
- **Switching color scheme**: You can now switch color scheme between a color gradient based on the relative value of a symbol or by the package name of a symbol.
- **Switching symbol name alignment**: Symbols with long names may be hard to differentiate if they have the same prefix. This new option allows you to align the text to the left or right so you can see the part of the symbol name that's important.
- **Improved navigation**: You can highlight a symbol or enable sandwich view for a symbol from the table. Also, a new status bar on top of the flame graph displays which views are enabled.

{{< video-embed src="/media/docs/grafana/panels-visualizations/screen-recording-grafana-10.1-flamegraph-whatsnew.mp4" >}}

### Distinguish widgets from visualizations

<!-- Alexa Vargas, Juan Cabanas -->

_Experimental in all editions of Grafana_

This experimental feature introduces a clear distinction between two different categories of panel plugin types: visualization panels that consume a data source and a new type, called _widgets_, that don't require a data source.

Now, you can easily add widgets like Text, News, and Annotation list without the need to select a data source. The plugins list and library panels are filtered based on whether you've selected a widget or visualization, providing a streamlined editing experience.

To see the widget editor in Grafana OSS or Enterprise, enable the `vizAndWidgetSplit` feature toggle. If you’re using Grafana Cloud and would like to enable this feature, please contact customer support.

{{< figure src="/media/docs/grafana/dashboards/WidgetVizSplit.png" max-width="750px" caption="New widget option added to empty dashboards" >}}

### Transformations redesign

<!-- Catherine Gui -->

_Available in public preview in all editions of Grafana_

The transformations tab has an improved user experience and visual redesign. Now, transformations are categorized, and each transformation type has an illustration to help you choose the right one.

{{< figure src="/media/docs/grafana/screenshot-grafana-10-1-transformations.png" max-width="750px" caption="Transformations redesign" >}}

### Format Time transformation

<!-- Kyle Cunningham -->

_Available in public preview in all editions of Grafana_

When working with date and time data, it can be useful to have different time formats. With the new Format Time transformation, you can convert any time format to any other one supported by [Moment.js](https://momentjs.com/docs/#/displaying/). When used in conjunction with the _Group by_ transformation, this can also be used to bucket days, weeks, and other time windows together.

{{< figure src="/media/docs/grafana/format-time-10-1.gif" max-width="750px" caption="Format time transformation" >}}

### Join by fields transformation outer join (tabular) option

<!-- Brendan O'Handley -->

_Generally available in all editions of Grafana_

The Join by field transformation has a new option: outer join (tabular). This option is a true outer join for tabular (SQL-like) data. Data can now be joined on a field value that is not distinct. This is different from the previous outer join, which is optimized for time-series data where the join values are never repeated.

### Disconnect values in Time series, Trend, and State timeline visualizations

_Generally available in all editions of Grafana_

<!-- Nathan Marrs -->

You can now choose whether to set a threshold above which values in the data should be disconnected. This can be useful in cases where you have sensors that report a value at a set interval, but you want to disconnect the values when the sensor does not respond. This feature complements the existing [connect null values functionality](../../../panels-visualizations/visualizations/time-series/#connect-null-values).

To learn more, refer to our [disconnect values documentation](../../../panels-visualizations/visualizations/time-series/#disconnect-values).

{{< figure src="/media/docs/grafana/screenshot-grafana-10-1-disconnect-values-examples.png" max-width="750px" caption="Disconnect values in Time series, Trend, and State timeline visualizations" >}}

### Geomap Network layer

_Available in public preview in all editions of Grafana_

<!-- Nathan Marrs -->

You can now display network data in the Geomap visualization by using the new beta Network layer. This layer supports the same data format as the [Node graph visualization](../../../panels-visualizations/visualizations/node-graph/#data-api).

To learn more, refer to our [Geomap network layer documentation](../../../panels-visualizations/visualizations/geomap/#network-layer-beta).

{{< figure src="/media/docs/grafana/screenshot-grafana-10-1-geomap-network-layer-v2.png" max-width="750px" caption="Geomap Network layer" >}}

### Heatmap visualizations now support data links

_Generally available in all editions of Grafana_

<!-- Nathan Marrs -->

You can now add data links to Heatmap visualizations. This allows you to add links to other dashboards, panels, or external URLs that are relevant to the data in your heatmap. We're pleased to highlight that this feature was a community contribution.

To learn more, refer to both our [Heatmap documentation](../../../panels-visualizations/visualizations/heatmap/) and our [Configure data links documentation](../../../panels-visualizations/configure-data-links/).

{{< figure src="/media/docs/grafana/screenshot-grafana-10-1-heatmap-datalinks.png" max-width="750px" caption="Heatmap datalink support" >}}

### Activate draft reports

<!-- Robert Horvath -->

_Generally available in Grafana Enterprise, Cloud Free, Cloud Pro, and Cloud Advanced_

You can now use the resume and pause report functionality to activate draft reports that have all the required fields filled in.

To learn more, refer to our [Create and manage reports documentation](../../../dashboards/create-reports/).

## Data sources

### Step editor in Loki

<!-- Ivana Huckova -->

_Generally available in all editions of Grafana_

We've improved the Loki query editor by adding a new **Step** editor field. This field allows you to specify a value for the `step` parameter in Loki queries. You can use this parameter when making metric queries to Loki or when you want a matrix response from your queries.

By default, the `step` parameter is set to the value of the `$__interval` variable. This variable is calculated based on the time range and the width of the graph (in pixels). To learn more about the Loki `step` parameter, refer to our [Loki step parameter documentation](/docs/loki/latest/api/#step-versus-interval).

{{< figure src="/media/docs/grafana/data-sources/loki-step-editor.png" max-width="750px" caption="New Loki step editor" >}}

### Copy link to a Loki log line

<!-- Sven Grossmann -->

_Generally available in all editions of Grafana_

New functionality for linking of Loki log lines in Explore allows you to quickly navigate to specific log entries for precise analysis. By clicking the **Copy shortlink** button for a log line, you can generate and copy a [short URL](../../../developers/http_api/short_url/) that provides direct access to the exact log entry within an absolute time range. When you open the link, Grafana automatically scrolls to the corresponding log line and highlights it, making it easy to identify and focus on the relevant information.

{{< figure src="/media/docs/grafana/data-sources/loki-shortlink.png" max-width="750px" caption="New Loki log line linking" >}}

### TraceQL response streaming in Tempo

<!-- André Pereira -->

_Experimental in all editions of Grafana_

Grafana's Tempo data source now supports _streaming_ responses to TraceQL queries. With this feature, you can now see partial query results as they come in, so you no longer have to wait for the whole query to finish. This is perfect for big queries that take a long time to return a response.

To use this feature, enable the `traceQLStreaming` feature toggle. If you’re using Grafana Cloud and would like to enable this feature, please contact customer support.

Streaming is available for both the **Search** and **TraceQL** query types, and you'll get immediate visibility of incoming traces on the results table. This smooth integration makes data exploration a breeze and speeds up decision-making.

{{< video-embed src="/media/docs/grafana/data-sources/tempo-streaming-v2.mp4" >}}

### Tempo Search powered by TraceQL

<!-- André Pereira -->

_Generally available in all editions of Grafana_

The **Search** query type was replaced with a new editor powered by TraceQL. This new editor allows you to use the same query language for both Search and TraceQL queries. This change also brings a new UI that makes it easier to write queries and explore your data while using the powerful features offered by TraceQL.

The previous Search interface is now deprecated and will be removed in a future release. We recommend that you start using the new editor as soon as possible and migrate existing dashboards.

{{< figure src="/media/docs/grafana/data-sources/tempo-search.png" max-width="750px" caption="Tempo Search editor powered by TraceQL" >}}

### Span filtering for traces is GA

_Generally available in all editions of Grafana_

<!-- Joey Tawadrous -->

Since the last release, we've made several improvements to span filtering. Now, we're promoting span filters out of public preview and into general availability.

Span filters allow you to work much more efficiently with traces that consist of a large number of spans. Span filters exist above the trace view and allow you to filter the spans that are shown in the trace view. The more filters you add, the more specific the filtered spans.

Currently, you can add one or more of the following filters:

- Service Name
- Span Name
- Duration
- Tags (which includes tags, process tags, and log fields)

To only show the spans you've matched, you can enable the **Show matches only** toggle.

Learn more about span filtering in our [Tempo data source documentation](../../../datasources/tempo/#span-filters).

{{< figure src="/media/docs/tempo/screenshot-grafana-tempo-span-filters-v10-1.png" max-width="750px" caption="Traces span filtering" >}}

### Configuration page redesign for Loki and Elasticsearch

<!-- Matías Wenceslao Chomicki -->

_Generally available in all editions of Grafana_

The Loki and Elasticsearch data source configuration pages have been redesigned to make getting started and setting up data sources as simple and easy to understand as possible. You can now find new subsections with links to documentation pages, as well as tooltips to assist you with configuring and customizing data sources.

### Easier to use query editor for Elasticsearch

<!-- Gabor Farkas -->

_Generally available in all editions of Grafana_

The Elasticsearch query editor now allows convenient switching between logs, metrics, and raw data directly from the top, eliminating the need to go through the metric selector.

### Metrics explorer

<!-- Catherine Gui -->

_Generally available in all editions of Grafana_

The Metrics explorer is a new feature that enhances metrics browsing in the Prometheus query builder. The Metrics explorer makes it easier for you to find the right metric and get comfortable with PromQL. You can now explore metrics with additional metadata, perform fuzzy search on the metric name or description, and filter on the Prometheus type.

{{< figure src="/media/docs/grafana/screenshot-grafana-10-1-metrics-explorer.png" max-width="750px" caption="Searching in Metrics explorer" >}}

### Redshift and Athena: Async query data support

<!-- Isabella Siu, Kevin Yu, Andrés Martínez -->

_Generally available in all editions of Grafana_

Async query data support in Redshift and Athena makes queries over multiple requests (starting, checking status, and fetching the results) instead of in a single request query. This is useful for queries that can potentially run for a long time and time out. This feature was previously available behind a feature toggle and is now be generally available and enabled by default.

### Redshift and Athena: Async query caching

<!-- Isabella Siu -->

_Experimental in Grafana Enterprise, Cloud Pro, and Cloud Advanced_

This feature adds support for query caching of async queries in the Athena and Redshift data source plugins. To try this feature, enable both the `useCachingService` and `awsAsyncQueryCaching` feature toggles. If you’re using Grafana Cloud and would like to enable this experimental feature, please contact customer support.

### CloudWatch logs Monaco query editor

<!-- Isabella Siu, Kevin Yu -->

_Experimental in all editions of Grafana_

The CloudWatch logs query editor is moving from being a Slate-based editor to a Monaco-based editor. This new Monaco-based editor provides improved syntax highlighting, and auto-completion. To use the Monaco-based query editor, enable the `cloudWatchLogsMonacoEditor` feature toggle. If you’re using Grafana Cloud and would like to enable this feature, please contact customer support.

### InfluxDB backend mode

<!-- Ismail Simsek -->

_Available in public preview in all editions of Grafana_

Previously, InfluxDB backend mode was available, however, there were compatibility issues that needed to be addressed. In this release, we've addressed these issues and
promoted this feature from experimental to public preview. In the future, backend mode will be the default, and we'll deprecate frontend mode. To try backend mode, enable the `influxdbBackendMigration` feature toggle. If you’re using Grafana Cloud and would like to enable this feature, please contact customer support.

## Explore

### Logs: Choose which fields to display in a log line

<!-- Matías Wenceslao Chomicki -->

_Generally available in all editions of Grafana_

When you're browsing logs in Explore, you can now click the eye icon within a row to replace the log line's contents with the value of one or more of the log fields or labels. This is helpful for scanning through your logs.

{{< figure src="/media/docs/grafana/log-field-picker-10-1.gif" max-width="750px" caption="Log rows menu" >}}

### Logs: Improved rendering performance of log lines

<!-- Matías Wenceslao Chomicki -->

_Generally available in all editions of Grafana_

After a series of performance optimizations to log-related components, browsing log lines is faster than ever.

### Logs: See more log lines in log context

<!-- Gabor Farkas, Sven Grossmann -->

_Generally available in all editions of Grafana_

Log context allows you to view additional lines surrounding a specific log entry. With this enhancement, you can access as many log lines as needed within the log context. As you scroll through the logs, Grafana dynamically loads more log lines, ensuring a seamless and continuous viewing experience.

### Elasticsearch logs sample

<!-- Gareth Dawson -->

_Generally available in all editions of Grafana_

For Elasticsearch metric queries in Explore, you can now see the sample of log lines that contributed to the displayed results. To see these logs, click the collapsed logs sample panel under your graph or table panel. If you want to interact with the log lines or modify the log query, click the **Open logs in split view** button and the log query will be executed in the split view.

### Panel plugins in Explore

<!-- Ben Donnelly -->

_Experimental in all editions of Grafana_

Data source plugin developers can now use any plugin to visualize data in Explore. Similar to `preferredVisualizationType`, we've introduced an experimental API to render visualizations by plugin ID. In the returned data frame, set the meta option `preferredVisualisationPluginId` to the plugin ID you want to be used when showing the data for given data frame.

## Alerting

_All Alerting features are generally available in all editions of Grafana_

We’ve made a number of improvements to simplify the alert rule creation process as well as improvements to contact points and alert management. For all the details, refer to our [Alerting documentation](../../../alerting/).

### Alert rules

We’ve made the following changes to alert rules.

#### Alert instance routing preview

_This feature is for Grafana-managed alert rules only._

Preview how your alert instances will be routed if they fire while you're creating your alert rule. You can view routing for each Alertmanager you've configured to receive Grafana-managed alerts, and if required, easily make adjustments to your custom labels to change the way your alert instances are routed.

{{< figure src="/media/docs/alerting/alert-routing-preview.png" max-width="750px" caption="Alert instance routing preview" >}}

#### Alert rule types

You can switch to a data source-managed alert rule if your data source is configured to support alert rule creation (Ruler API enabled). By default, the alert rule type is Grafana-managed.

{{< figure src="/media/docs/alerting/alert-rule-types.png" max-width="750px" caption="Alert rule types" >}}

#### UI improvements

- **Alert evaluation behavior**: New UI components for creating a folder and adding an evaluation group. along with improved text and validation.
- **Alert Rule list page**: The process of creating recording rules (**More** drop-down) is now separate from Grafana-managed and data source-managed alert rules (**+New alert rule**)
  .
- **Annotations display**: Adding a summary, description, and runbook URL as annotations are now optional. The dashboard and panel names are now also linked directly, making them easier to access.
- **View YAML button**: Displays alert rule configuration in YAML format on the Grafana-managed alert rules form, as well as in the Grafana-managed provisioned and non-provisioned Alert Rule detail view.
- **Queries and expressions**: Several improvements have been made to the display of queries and expressions, including making **Add expression** a drop-down and moving **Conditions** to the header.
- **Min interval option**: Improves control over query costs and performance by allowing you to adjust the minimum resolution of the data used in your alerting queries.
- **In-app guidance for alert rule creation**: Learn about how to create your alert rules interactively with in-app guidance for additional context and links out to our Alerting documentation.
- **Support for toggling common labels**: Toggle between showing or hiding labels for each individual alert instance.

### Contact points

We’ve made the following changes to contact points.

#### Additional contact points for external Alertmanager

We've added support for the following contact points when using an external Alertmanager:

- WeChat
- Amazon SNS
- Telegram
- Cisco Webex Teams

#### Contact point provisioning file export

This update facilitates file provisioning and maintenance for contact points. The feature implements the provisioning API export endpoints for exporting contact points as well as adding export buttons to the contact point list in the UI.

### Notification policies

We’ve made the following changes to notification policies.

#### Notification policy provisioning file export

This update facilitates file provisioning and maintenance for notification policies. The feature implements the provisioning API export endpoints for exporting notification policies as well as adding an export button to the root notification policy in the UI.

### Alert management

We’ve made the following changes to alert management.

#### Support for time zones in mute timings

We've added support for different time zones and locations as well as a visual selector for week days, made improvements to loading and error handling, and provided better validation for time ranges.

{{< figure src="/media/docs/alerting/timezone-support.png" max-width="600px" caption="Time zone support" >}}

#### Label colors for alert instances

Labels are colored according to the label key, which makes it easier to track and view labels across alert instances.

## Authentication and authorization

### OAuth role mapping enforcement

<!-- Jo Guerreiro, AuthNZ -->

_Generally available in all editions of Grafana_

This change impacts `GitHub`, `GitLab`, `Okta`, and `Generic` OAuth.

Previously, if no organization role mapping was found for a user when they connected using OAuth, Grafana didn’t update the user’s organization role.

Now, on every login, if the `role_attribute_path` property doesn't return a role, then the user is assigned the role specified by the `auto_assign_org_role` option or the default role for the organization, which is Viewer by default.

To avoid overriding manually set roles, enable the `skip_org_role_sync` option in the Grafana configuration for your OAuth provider before affected users log in for the first time.

### Prevent manual role updates for externally synced roles

<!-- Ieva Vasiljeva, AuthNZ -->

_Generally available in all editions of Grafana_

This change impacts all instances that use an external authentication provider and have role mapping enabled.

Previously, it was possible to manually update a user's organization role (Viewer, Editor, Admin, or Grafana Admin) even if this role was managed by an external authentication provider.
This means that roles could be manually set for the duration of a user's session, but were overridden by the external authentication provider the next time the user logged in.
If the `onlyExternalOrgRoleSync` feature toggle was enabled, only then were manual role updates for externally managed roles not allowed.

Now, you can no longer manually update externally managed organization roles.
We've removed the `onlyExternalOrgRoleSync` feature toggle, and have defaulted to locking the organization role of users authenticated by an external provider.

If you prefer to manage your users' organization roles manually, enable the `skip_org_role_sync` option in the Grafana configuration for your authentication provider.

For context on the previous work done leading up to this change, refer to the [Grafana v9.5 What's new](../../whats-new-in-v9/whats-new-in-v9-5/#auth-lock-organization-roles-synced-from-auth-providers).

### GitLab OIDC support

<!-- Jo Guerreiro, AuthNZ -->

_Generally available in all editions of Grafana_

Grafana now supports GitLab OIDC through the `GitLab` OAuth provider in addition to the existing `GitLab` OAuth2 provider. This allows you to use GitLab OIDC to authenticate users in Grafana.

This change also allows Grafana to reduce the access scope to only the required scopes for authentication and authorization, instead
of full read API access.

To learn how to migrate your GitLab OAuth2 setup to OIDC, refer to our [GitLab authentication documentation](../../../setup-grafana/configure-access/configure-authentication/gitlab/).

### Google OIDC and Team Sync support

<!-- Jo Guerreiro, AuthNZ -->

_Generally available in all editions of Grafana_

Grafana now supports Google OIDC through the `Google` OAuth provider in addition to the existing `Google` OAuth2 provider. This allows you to use Google OIDC to authenticate users in Grafana, which in turn, lets Grafana reduce the access scope to only the required scopes for authentication and authorization.

This release also adds support for Google OIDC in Team Sync. You can now easily add users to teams by using their Google groups.

To learn how to migrate your Google OAuth2 setup to OIDC and how to set up Team Sync, refer to our [Google authentication documentation](../../../setup-grafana/configure-access/configure-authentication/google/).

## Plugins

### Angular deprecation changes

<!-- Giuseppe Guerra, Plugins Platform -->

_Generally available in all editions of Grafana_

We've made the following updates to increase awareness of the [Angular deprecation](https://archive.grafana.com/docs/grafana/v12.2/developers/angular_deprecation/) and its consequences in future releases of Grafana:

#### UI changes

- Added an **Angular** badge next to affected plugins in the plugins catalog.
- Added an alert at the top of a plugin's page in the plugins catalog when browsing Angular plugins.
- Added an alert at the top of the query editor when editing panels that use Angular data source plugins.

#### Other changes

- Angular Plugins will not be loaded if [angular_support_enabled](../../../setup-grafana/configure-grafana/#angular_support_enabled) is set to `false`.

Learn more in our [Angular support deprecation documentation](https://archive.grafana.com/docs/grafana/v12.2/developers/angular_deprecation/).

### Deprecated provisioning of data sources with invalid UIDs

<!-- Giuseppe Guerra, Plugins Platform -->

_Generally available in all editions of Grafana_

Grafana now logs an error when provisioning data sources with invalid UIDs. A valid UID is a combination of a-z, A-Z, 0-9 (alphanumericals), `-` (dashes), and `_` (underscores), with a maximum length of 40 characters.

Provisioning data sources with invalid UIDs will be removed in future versions of Grafana, and will return an error instead.

## Subfolders: folder selection

<!-- Zsofia Komaromi -->

_Available in public preview in all editions of Grafana_

When saving or moving a dashboard, you can now see the full folder tree when selecting the destination folder.

To get started creating subfolders, enable the `nestedFolders` feature toggle. We recommend that you enable this feature only on test or development instances, rather than in production environments.

{{< figure src="/media/docs/grafana/screenshot-grafana-10.1-subfolders-folder-picker.png" max-width="750px" caption="Selecting a folder in Grafana" >}}

# What’s new in Grafana v10.2

Welcome to Grafana 10.2! Read on to learn about changes to dashboards and visualizations, data sources, security and authentication, and more. We’re particularly excited about the addition of generative AI features for dashboards, a new kind of basic role, and improvements to visualization transformations.

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v10.2, check out our [Upgrade Guide](https://archive.grafana.com/docs/grafana/v10.2/upgrade-guide/upgrade-v10.2/).

<!-- Template below

> Add on-prem only features here. Features documented in the Cloud What's new will be copied from those release notes.

## Feature
<!-- Name of contributor -->
<!-- _[Generally available | Available in private/public preview | Experimental] in Grafana [Open Source, Enterprise]_
Description. Include an overview of the feature and problem it solves, and where to learn more (like a link to the docs).
{{< admonition type="note" >}}
Use full URLs for links. When linking to versioned docs, replace the version with the version interpolation placeholder (for example, <GRAFANA_VERSION>, <TEMPO_VERSION>, <MIMIR_VERSION>) so the system can determine the correct set of docs to point to. For example, "https://grafana.com/docs/grafana/latest/administration/" becomes "https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/".
{{< /admonition >}}
-->
<!-- Add an image, GIF or video  as below

{{< figure src="/media/docs/grafana/dashboards/WidgetVizSplit.png" max-width="750px" caption="DESCRIPTIVE CAPTION" >}}

Learn how to upload images here: https://grafana.com/docs/writers-toolkit/write/image-guidelines/#where-to-store-media-assets
-->

## Share your dashboard with the world: Public dashboards are generally available

<!-- Thanos Karachalios -->

_Generally available in all editions of Grafana_

Public dashboards allow you to share your visualizations and insights with a broader audience without the requirement of a login. You can effortlessly use our current sharing model and create a public dashboard URL to share with anyone using the generated public URL link. To learn more, refer to the [Public dashboards documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/share-dashboards-panels/shared-dashboards), as well as the following video demo:

{{< youtube id="XHwwRCdxHMg?rel=0" >}}

## Navigate lengthy, mixed data in Explore with Content Outline

<!-- Thanos Karachalios -->

_Generally available in all editions of Grafana_

Introducing Content Outline in Grafana Explore. It's easy to lose track of your place when you're running complex mixed queries or switching between logs and traces. Content outline is our first step towards seamless navigation from log lines to traces and back to queries, ensuring quicker searches while preserving context. Experience efficient, contextual investigations with this update in Grafana Explore. To learn more, refer to the [Content outline documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/#content-outline), as well as the following video demo.

{{< youtube id="cE7J1-S4ZCQ?rel=0" >}}

## Correlations

Grafana Correlations is a new public preview feature you can use to establish links from any data source query to any other, carrying forward data like namespace, host, or label values. This is extremely powerful for performing root cause analysis with a diverse set of data sources. For more information, refer to [the documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/correlations/).

### Create Correlations the easy way in Grafana Explore

<!-- Kristina Durivage -->

_Available in public preview in all editions of Grafana_

Creating correlations has just become easier. Try out our new correlations editor in Explore by selecting the **+ Add > Add correlation** option from the top bar or from the command palette. The editor shows all possible places where you can place data links and guides you through building and testing target queries. For more information, refer to [the documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/correlations/).

To try out **Correlations**, enable the `correlations` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/#preview-feature-toggles).

{{< figure src="/media/docs/grafana/correlations-explore-editor-10-2.png" max-width="750px" caption="Create a correlation with variables from within Grafana Explore" >}}

### Create correlations for provisioned data sources

<!-- Piotr Jamróz -->

_Available in public preview in all editions of Grafana_

In previous versions of Grafana, if a data source was provisioned, the only way to add correlations to it was also with provisioning. Now, that's no longer the case, and you can easily create new correlations mixing both methods—using the **Administration** page or provisioning. For more information, refer to [the documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/correlations/).

To try out **Correlations**, enable the `correlations` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/#preview-feature-toggles).

## Dashboards and visualizations

### Use AI to generate titles, descriptions, and change summaries

_Available in public preview in all editions of Grafana_

<!-- Nathan Marrs -->
<!-- Cloud -->

You can now use generative AI to assist you in your Grafana dashboards. So far generative AI can help you with the following tasks:

- **Generate panel and dashboard titles and descriptions** - You can now generate a title and description for your panel or dashboard based on the data you've added to it. This is useful when you want to quickly visualize your data and don't want to spend time coming up with a title or description.
- **Generate dashboard save changes summary** - You can now generate a summary of the changes you've made to a dashboard when you save it. This is great for effortlessly tracking the history of a dashboard.

To enable these features, you must first enable the `dashgpt` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/#experimental-feature-toggles). Then install and configure Grafana's LLM app plugin. For more information, refer to the [Grafana LLM app plugin documentation](https://grafana.com/docs/grafana-cloud/alerting-and-irm/machine-learning/configure/llm-plugin/).

When enabled, look for the **✨ Auto generate** option next to the **Title** and **Description** fields in your panels and dashboards, or when you press the **Save** button.

{{< youtube id="tgBussaYJww?rel=0" >}}

### Find your dashboard faster with the new Dashboards browse screen

<!-- Yaelle Chaudy for Frontend Platform -->

_Generally available in all editions of Grafana_

The new browse screen for dashboards features a more compact design, making it easier to navigate, search for, and manage your folders and dashboards. The new interface also has many performance improvements, especially for instances with a large number of folders and dashboards.

To make using folders easier and more consistent, there's no longer a special **General** folder. Dashboards without a folder, or dashboards previously in the **General** folder, are now shown at the root level.

To try it out, go to the **Dashboards** section of your Grafana instance.

{{< youtube id="-S_Jjuwj74k?rel=0" >}}

### Create interactive buttons in canvas visualizations

_Available in public preview in all editions of Grafana_

<!-- Nathan Marrs -->
<!-- Cloud -->

You can now add buttons to your canvas visualizations. Buttons can be configured to call an API endpoint. This pushes Grafana's capabilities to new heights, allowing you to create interactive dashboards that can be used to control external systems.

To learn more, refer to our [Canvas button element documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/canvas/#button), as well as the following video demo.

{{< youtube id="T6fg1TpfBUg?rel=0" >}}

### Zoom in on the y-axis of the time series and candlestick visualizations

_Generally available in all editions of Grafana_

<!-- Nathan Marrs -->
<!-- Cloud -->

You can now zoom in on the y-axis of your time series and candlestick visualizations. This is useful when you want to focus on a specific range of values. To zoom in on the y-axis on supported visualizations, hold the Shift key while clicking and dragging; double-click to reset the zoom.

{{< video-embed src="/media/docs/grafana/screen-recording-10-2-y-axis-zoom-demo.mp4" max-width="750px" caption="Y-axis zooming demo" >}}

### Calculate visualization min/max individually per field

<!-- Oscar Kilhed -->

_Generally available in all editions of Grafana_

When visualizing multiple fields with a wide spread of values, calculating the min or max value of the visualization based on all fields can hide useful details.

{{< figure src="/media/docs/grafana/panels-visualizations/globalminmax.png" max-width="300px" caption="Stat visualization with min/max calculated from all fields" >}}

In this example in the stat visualization, it's hard to get an idea of how the values of each series relate to the historical values of that series. The threshold of 10% is exceeded by the A-series even though the A-series is below 10% of its historical maximum.

Now, you can automatically calculate the min or max of each visualized field based on the lowest and highest value of the individual field. This setting is available in the standard options of most visualizations.

{{< figure src="/media/docs/grafana/panels-visualizations/localminmax.png" max-width="300px" caption="Stat visualization with min/max calculated per field" >}}

In this example, using the same data, with the min and max calculated for each individual field, we get a much better understanding of how the current value relates to the historical values. The A-series no longer exceeds the 10% threshold; in fact, it's now clear that it's at a historical low.

This isn't only useful in the stat visualization&mdash;gauge, bar gauge, and status history visualizations, table cells formatted by thresholds, and gauge table cells all benefit from this addition.

### Data visualization quality of life improvements

_Generally available in all editions of Grafana_

<!-- Nathan Marrs -->
<!-- Cloud -->

We've made a number of smaller improvements to the data visualization experience in Grafana.

#### Geomap marker symbol alignment options

You can now offset geomap marker symbols from the underlying data point.

{{< figure src="/media/docs/grafana/gif-grafana-10-2-geomap-marker-symbol-alignment.gif" max-width="750px" caption="Geomap marker symbol alignment" >}}

#### Gauge visualization overflow support

You can now visualize gauges in vertical and horizontal orientations with overflow. This resolves an issue where the design would break when the number of gauges exceeded the available space.

{{< figure src="/media/docs/grafana/gif-grafana-10-2-gauge-overflow.gif" max-width="750px" caption="Gauge overflow" >}}

#### Bar chart axes improvements

You can now center bar chart axes on zero and configure axes border and color settings.

{{< figure src="/media/docs/grafana/screenshot-grafana-10-2-bar-chart-axes-improvements.png" max-width="750px" caption="Bar chart improvements" >}}

## Data sources and querying

### Tempo data source

We've placed special focus on the Tempo data source over the past couple of months with new features, query performance improvements, and a better query experience.

#### Compute RED metrics over spans aggregated by attribute with the "Aggregate By" Search option

<!-- Joey Tawadrous, Jen Villa -->

_Experimental in all editions of Grafana_

Requires Tempo or Grafana Enterprise Traces (GET) v2.2 or greater.

We've added an **Aggregate By** option to the [TraceQL query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-search/#write-traceql-queries-using-search) to leverage Tempo's [metrics summary API](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/metrics-summary/). You can calculate RED metrics (total span count, percent erroring spans, and latency information) for spans of `kind=server` received in the last hour that match your filter criteria, grouped by whatever attributes you specify.

This feature is disabled by default. To enable it, use the `metricsSummary` [experimental feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/#experimental-feature-toggles).

For more information, refer to the [documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-search/#optional-use-aggregate-by), as well as the following video demo.

{{< youtube id="g97CjKOZqT4?rel=0" >}}

#### Query traces more easily with the Improved TraceQL editor

<!-- Fabrizio Casati -->

_Generally available in all editions of Grafana_

The [TraceQL query editor](https://grafana.com/docs/tempo/latest/traceql/#traceql-query-editor) has been improved to facilitate the creation of TraceQL queries. In particular, it now features improved autocompletion, syntax highlighting, and error reporting.

{{< video-embed src="/media/docs/tempo/screen-recording-grafana-10.2-traceql-query-editor-improvements.mp4" >}}

#### Group multiple spansets per trace

<!-- Joey Tawadrous -->

_Generally available in all editions of Grafana_

The [TraceQL query editor](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/#traceql-query-editor) has been improved to facilitate the grouping of multiple spans per trace in TraceQL queries. For example, when `by(resource.service.name)` is added to your TraceQL query, it will group the spans in each trace by `resource.service.name`.

{{< youtube id="fraepWra00Y?rel=0" >}}

#### Create query-type template variables for the Tempo data source

<!-- Fabrizio Casati -->

_Generally available in all editions of Grafana_

The Tempo data source now supports query-type template variables. With this update, you can create variables for which the values are a list of attribute names or attribute values seen on spans received by Tempo.

To learn more, refer to the following video demo, as well as the [Grafana Variables documentation](/docs/grafana/next/dashboards/variables/).

{{< video-embed src="/media/docs/tempo/screen-recording-grafana-10.2-tempo-query-type-template-variables.mp4" >}}

### SAP HANA®: Configure your data source with tenant database instance name and number

<!-- Miguel Palau -->

_Generally available in Grafana Enterprise and Grafana Cloud_

The SAP HANA® data source now supports tenant database connections by using the database name and/or instance number. This is helpful because these are less likely to change than the port for your database. For more information, refer to our [SAP HANA® configuration documentation](/docs/plugins/grafana-saphana-datasource/latest/#configuration).

{{< video-embed src="/media/docs/sap-hana/tenant.mp4" >}}

### Datadog: Aggregate logs to compute metrics and time series

<!-- Taewoo Kim -->

_Generally available in Grafana Enterprise and Grafana Cloud_

The Datadog data source now supports log aggregation. This feature helps aggregate logs/events into buckets and compute metrics and time series. For more information, refer to [Datadog log aggregation](/docs/plugins/grafana-datadog-datasource/latest#logs-analytics--aggregation).

{{< video-embed src="/media/docs/datadog/datadog-log-aggregation.mp4" >}}

### Datadog: Rate-limit requests from the Datadog data source

<!-- Taewoo Kim -->

_Generally available in Grafana Enterprise and Grafana Cloud_

In the Datadog data source, you can now block API requests for metric queries based on upstream rate limits. With this update, you can set a rate limit percentage at which the plugin stops sending queries.

To learn more, refer to [Datadog data source settings](/docs/plugins/grafana-datadog-datasource/latest#configure-the-data-source), as well as the following video demo.

{{< video-embed src="/media/docs/datadog/datadog-rate-limit.mp4" >}}

### Microsoft SQL Server: Support for Azure Authentication (Service principal/MSI)

<!-- Jev Forsberg, Oscar Kilhed -->

_Generally available in all editions of Grafana_

We've added support for Azure Authentication (Service principal/MSI) on our MS SQL plugin to authenticate and allow querying of content stored in SQL Managed Instance databases.

Enable this feature by setting the `managed_identity_enabled` property to `true` under the `Azure` heading in your configuration file (/conf/<your_config_file>.ini). Then take the following steps in your Microsoft SQL Server data source configuration UI:

1. Under **Authentication**, select **Azure AD Authentication** in the drop-down to reveal the **Azure Authentication Settings** section.
2. In this section, select either **Managed Identity** or **App Registration**.
3. Enter the credentials accordingly.

{{< figure src="/media/docs/grafana/data-sources/screenshot-managed-identity-mssql-ui-cropped.png" caption="Azure MSI Authentication" max-width="550px" >}}

Learn more in the [Microsoft SQL Server documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/mssql/).

## Transformations

As our work on improving the user experience of transforming data continues, we've also been adding new capabilities to transformations.

### Use dashboard variables in transformations

<!-- Oscar Kilhed, Victor Marin -->

_Experimental in all editions of Grafana_

Previously, the only transformation that supported [dashboard variables](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/) was the **Add field from calculation** transformation. We've now extended the support for variables to the **Filter by value**, **Create heatmap**, **Histogram**, **Sort by**, **Limit**, **Filter by name**, and **Join by field** transformations.

We've also made it easier to find the correct dashboard variable by displaying available variables in the fields that support them, either in the drop-down or as a suggestion when you type **$** or press Ctrl + Space.

{{< youtube id="WE0tt69ApO4?rel=0" >}}

### New modes for the Add field from calculation transformation

<!-- Victor Marin -->

_Generally available in all editions of Grafana_

The **Add field from calculation** transformation has been updated.

**Unary operation** is a new mode that lets you apply mathematical operations to a field. The currently supported operations are:

- **Absolute value (abs)** - Returns the absolute value of a given expression. It represents its distance from zero as a positive number.
- **Natural exponential (exp)** - Returns _e_ raised to the power of a given expression.
- **Natural logarithm (ln)** - Returns the natural logarithm of a given expression.
- **Floor (floor)** - Returns the largest integer less than or equal to a given expression.
- **Ceiling (ceil)** - Returns the smallest integer greater than or equal to a given expression.

{{< figure src="/media/docs/grafana/transformations/unary-operation.png" alt="Unary operation options" >}}

Also, **Row index** can now show the index as a percentage.

Learn more in the [Add field from calculation documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/#add-field-from-calculation).

### Format strings with transformations

<!-- Solomon Dubock, BI Squad -->

_Experimental in all editions of Grafana_

With the new **Format string** transformation, you can manipulate string fields to improve how they're displayed. The currently supported operations are:

- **Change case** changes the case of your string to upper case, lower case, sentence case, title case, pascal case, camel case, or snake case.
- **Trim** removes white space characters at the start and end of your string.
- **Substring** selects a part of your string field.

Learn more in the [Format string documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/#format-string).

### See which transformations will work with your data

<!-- Kyle Cunningham -->

_Available in public preview in all editions of Grafana_

We've added initial support to detect situations in which various transformations won't work appropriately based on current data. Previously, selecting the appropriate transformation and configuring it correctly required a process of trial and error or already knowing how a given transformation worked. Now, transformations that we've detected can't be used are shaded in the interface to indicate this, along with a helpful message explaining why.

{{< figure src="/media/docs/grafana/transformations/disabled-transformation.png" caption="Transformation that has been disabled because it doesn't have the necessary data" >}}

If you have the `transformationsRedesign` feature flag set, you'll be able to access this functionality right away. If you'd like to try it, enable this feature flag in your [Grafana configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#feature_toggles).

### Choose your timezome in the Format time and Convert field type transformations

<!-- Kyle Cunningham -->

_Generally available in all editions of Grafana_

We've added support for setting timezones manually when formatting times as strings using the **Format time** and **Convert field type** transformations. This allows times to be formatted relative to any timezone across the globe.

{{< figure src="/media/docs/grafana/transformations/format-timezone.png" caption="Timezone support in the Format time transformation" >}}

## Alerting

### Grafana OnCall integration for Alerting

<!-- Brenda Muir -->

_Generally available in all editions of Grafana_

Use the Grafana Alerting - Grafana OnCall integration to effortlessly connect alerts generated by Grafana Alerting with Grafana OnCall. From there, you can route them according to defined escalation chains and schedules.

To learn more, refer to the [Grafana OnCall integration for Alerting documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/integrations/configure-oncall/), as well as the following video demo.

{{< youtube id="abRn5I61hxs?rel=0" >}}

### Export alerting resources to Terraform

<!-- Yuri Tseretyan -->

_Generally available in all editions of Grafana_

Export your alerting resources, such as alert rules, contact points, and notification policies as Terraform resources. A new “Modify export” mode for alert rules enables you to edit provisioned alert rules and export a modified version.

### Additional contact points for external Alertmanager

<!-- Alexander Weaver -->

_Generally available in Grafana Open Source and Enterprise_

We've added support for the Microsoft Teams contact points when using an external Alertmanager.

## Authentication and authorization

### No basic role

_Generally available in Grafana Enterprise and Grafana Cloud_

We're excited to introduce the "No basic role," a new basic role with no permissions. A basic role in Grafana dictates the set of actions a user or entity can perform, known as permissions. This new role is especially beneficial if you're aiming for tailored, customized RBAC permissions for your service accounts or users. You can set this as a basic role through the API or UI.

Previously, permissions were granted based on predefined sets of capabilities. Now, with the "No basic role," you have the flexibility to be even more granular.

For more details on basic roles and permissions, refer to the [documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/).

### New service account permissions

[Service accounts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/service-accounts/) allow you to create tokens to access Grafana's API and dashboards. Service accounts are a powerful tool for authenticating with Grafana's API and accessing data sources. However, without proper access controls, service accounts can pose a security risk to your Grafana instance. In Grafana 10.2, we've added new tools to limit service accounts to just the resources they need to access.

#### Add dashboard and folder permissions to service accounts

<!-- Jo Guerreiro -->

_Generally available in all editions of Grafana_

In this release, we've added the ability to assign dashboard and folder permissions to service accounts.
This means that you can now create a service account that can be used to access a specific dashboard and nothing else.

This is useful if you want to limit the access service accounts have to your Grafana instance.

Learn more in our [dashboard and folder permissions documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/user-management/manage-dashboard-permissions/#manage-dashboard-permissions).

#### Add data source permissions to service accounts

<!-- Jo Guerreiro -->

_Generally available in Grafana Cloud and Grafana Enterprise_

Grafana 10.2 also introduces the ability to assign _data source_ permissions to service accounts, for Grafana CLoud and Enterprise users.
With this feature, you can create a service account that has access to a specific data source and nothing else.
This is useful in scenarios where you want to limit the access service accounts have to your Grafana instance.

For example, imagine you have a team of developers who need to access a specific data source to develop a new feature.
Instead of giving them full access to your Grafana instance, you can create a service account that has access only to that data source.
This way, you can limit the potential damage that could be caused by a compromised service account.

Learn more in our [data source permissions documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/#data-source-permissions).

{{< figure src="/media/docs/grafana/screenshot-grafana-10-2-sa-managed-permissions.png" max-width="600px" caption="Data source permissions in 10.2" >}}

### Role mapping support for Google OIDC

<!-- Jo Guerreiro -->

_Generally available in all editions of Grafana_

You can now map Google groups to Grafana organizational roles when using Google OIDC.
This is useful if you want to limit the access users have to your Grafana instance.

We've also added support for controlling allowed groups when using Google OIDC.

Refer to the [Google Authentication documentation](http://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/google/) to learn how to use these new options.

### Configure refresh token handling separately for OAuth providers

<!-- Mihaly Gyongyosi -->

_Available in public preview in all editions of Grafana_

With Grafana v9.3, we introduced a [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/) called `accessTokenExpirationCheck`. It improves the security of Grafana by checking the expiration of the access token and automatically refreshing the expired access token when a user is logged in using one of the OAuth providers.

With the current release, we've introduced a new configuration option for each OAuth provider called `use_refresh_token` that allows you to configure whether the particular OAuth integration should use refresh tokens to automatically refresh access tokens when they expire. In addition, to further improve security and provide secure defaults, `use_refresh_token` is enabled by default for providers that support either refreshing tokens automatically or client-controlled fetching of refresh tokens. It's enabled by default for the following OAuth providers: `AzureAD`, `GitLab`, `Google`.

For more information on how to set up refresh token handling, please refer to [the documentation of the particular OAuth provider.](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/).

{{< admonition type="note" >}}
The `use_refresh_token` configuration must be used in conjunction with the `accessTokenExpirationCheck` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/). If you disable the `accessTokenExpirationCheck` feature toggle, Grafana won't check the expiration of the access token and won't automatically refresh the expired access token, even if the `use_refresh_token` configuration is set to `true`.

The `accessTokenExpirationCheck` feature toggle will be removed in Grafana v10.3.
{{< /admonition >}}

### Permission validation on custom role creation and update

<!-- Mihaly Gyongyosi -->

_Generally available in Grafana Enterprise and Grafana Cloud_

With the current release, we enabled RBAC permission validation (`rbac.permission_validation_enabled` setting) by default. This means that the permissions provided in the request during custom role creation or update are validated against the list of [available permissions and their scopes](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/custom-role-actions-scopes/#action-definitions). If the request contains a permission that is not available or the scope of the permission is not valid, the request is rejected with an error message.

## Recorded queries: Record multiple metrics from a single query

<!-- Kyle Brandt, Observability Metrics -->

_Generally available in Grafana Enterprise and Grafana Cloud_

Recorded queries provide a way to take a _static_ number, (for example, the number of GitHub issues open at a given time, or the number of rows in a database table) and record it periodically as a Prometheus metric. This is great for tracking numbers over time for quick querying later. Previously, recorded queries were limited to a single series, so you needed to narrow your query down to a single number in order to record it. Now, you can record multiple metrics with a single recorded query, which makes them more powerful _and_ easier to create and manage.

<!-- TODO: add graphic or video -->

# What’s new in Grafana v10.3

Welcome to Grafana 10.3! Read on to learn about changes to navigation, visualizations and transformations, alerting, profiling, and logs.

We've also included here features released in Grafana 10.2.3, as well as breaking changes from that release. Features that were included in the 10.2.3 release are marked with an asterisk.

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v10.3, check out our [Upgrade Guide](https://archive.grafana.com/docs/grafana/v10.3/upgrade-guide/upgrade-v10.3/).

## Breaking changes

For Grafana v10.3, we've also provided a list of [breaking changes](https://archive.grafana.com/docs/grafana/v10.3/breaking-changes/breaking-changes-v10-3/) to help you upgrade with greater confidence. For information about these along with guidance on how to proceed, refer to [Breaking changes in Grafana v10.3](https://archive.grafana.com/docs/grafana/v10.3/breaking-changes/breaking-changes-v10-3/).

<!-- Template below

## Feature
<!-- Name of contributor -->
<!--_[Generally available | Available in private/public preview | Experimental] in Grafana [Open Source, Enterprise, all editions of Grafana, some combination of self-managed and Cloud]_
Description. Include an overview of the feature and problem it solves, and where to learn more (like a link to the docs).
{{< admonition type="note" >}}
Use full URLs for links. When linking to versioned docs, replace the version with the version interpolation placeholder (for example, <GRAFANA_VERSION>, <TEMPO_VERSION>, <MIMIR_VERSION>) so the system can determine the correct set of docs to point to. For example, "https://grafana.com/docs/grafana/latest/administration/" becomes "https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/".
{{< /admonition >}}

<!--Add an image, GIF or video  as below-->

<!--{{< figure src="/media/docs/grafana/dashboards/WidgetVizSplit.png" max-width="750px" caption="DESCRIPTIVE CAPTION" >}}

<!--Learn how to upload images here: https://grafana.com/docs/writers-toolkit/write/image-guidelines/#where-to-store-media-assets-->
<!---->

## Navigation updates\*

<!--Laura Benz-->

_Available in public preview in Grafana Open Source and Enterprise_

The improved navigation menu gives you a better overview by showing all levels of navigation items in a more compact design. We also implemented a better dock and improved scrolling behavior. Furthermore, we improved the structure of the nav menu and added several new items.

{{< youtube id="IhpghtVykLc" >}}

## Table data in PDF reports

<!--Agnès Toulet-->

_Available in public preview in Grafana Enterprise and Grafana Cloud_

We've improved the reporting experience with options to make all of your table data accessible in PDFs. Previously, if your dashboard included large table visualizations, you couldn't see all of the table data in your PDF report. Unlike in Grafana, you couldn't scroll in the PDF table visualization or click on the page numbers. With this new feature, you now have the option to see all the data directly in your PDF without losing your dashboard layout.

We've added two format options to the report creation form:

- **Include table data as PDF appendix** - Adds an appendix to your dashboard PDF.
- **Attach a separate PDF of table data** - Generates a separate PDF file for your table panel data.

To try out this feature, enable the `pdfTables` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/) or contact Grafana Support to have it enabled in on your Grafana Cloud stack.

{{< youtube id="1fzQQI8O838" >}}

## Dashboards and visualizations

### Moving average and trend lines using transformations

<!--Oscar Kilhed-->

_Available in public preview in all editions of Grafana_

#### Moving average\*

Sometimes your data is too noisy to quickly grasp what's going on. A common way to address this issue is to calculate the moving mean, or moving average, to filter out some of that noise. Luckily, many data sources already support calculating the moving mean, but when the support is lacking or you're not well versed in the query language, until now, you were stuck with the noise.

{{< figure src="/media/docs/grafana/transformations/noisy-sensor-data.png" caption="Noisy data can hide the general trend of your data." alt="Graph displaying noisy sensor data" max-width="300px" >}}

By selecting the **Window functions** mode and using **Mean** as the calculation for the **Add field from calculation** transformation, Grafana adds a field with the moving mean for your selected field.

{{< figure src="/media/docs/grafana/transformations/noisy-sensor-data-moving-average.png" caption="Calculating the moving mean of your data will make it easier to grasp what's going on." alt="Graph displaying the moving mean of noisy data" >}}

The **Window functions** mode also supports moving variance and moving standard deviation calculations if you need to analyze the volatility of your metric.

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/#add-field-from-calculation)

#### Trend lines\*

We're also adding some basic statistical analysis features as a way to help you visualize trends in your data. The **Regression analysis** transformation will fit a mathematical function to your data and display it as predicted data points in a separate data frame.

{{< figure src="/media/docs/grafana/transformations/trendlines.png" caption="Linear and polynomial regression trendlines" alt="Graph with trendlines" >}}

The transformation currently supports linear regression and polynomial regression to the fifth-degree.

### Canvas visualization supports pan and zoom

<!--Nathan Marrs-->

_Available in public preview in all editions of Grafana_

Canvas visualizations now support panning and zooming. This allows you to both create and navigate more complex designs.

To enable this feature, you must first enable the `canvasPanelPanZoom` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/).

{{< youtube id="CF-HFkcytRA" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/canvas/)

### Improved tooltips in visualizations\*

<!--Nathan Marrs-->

_Available in public preview in all editions of Grafana_

We've introduced enhanced tooltips as part of our standardization initiative, unifying the tooltip architecture for a consistent user experience across panels. Packed with features like color indicators, time uniformity, and improved support for long labels, these tooltips go beyond a cosmetic redesign, bringing fundamental changes to elevate your data visualization experience. Stay tuned for more updates!

To try out the new tooltips, enable the `newVizTooltips` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/). Enhanced tooltips have been implemented for the following visualizations:

- Time series
- Trend
- Heatmap
- Status history
- Candlestick
- State timeline
- XY Chart
- and more coming soon!

{{< admonition type="note" >}}
As this is an ongoing project, the dashboard shared cursor and annotations features are not yet fully supported.
{{< /admonition >}}

{{< youtube id="0Rp6FYfHu6Q" >}}

### Plot enum values in your time series and state timeline visualizations\*

<!--Nathan Marrs-->

_Generally available in all editions of Grafana_

You can now plot enum values in your time series and state timeline visualizations. This feature is useful when you want to visualize the state of a system, such as the status of a service or the health of a device. For example, you can use this feature to visualize the status of a service as `ON`, `STANDBY`, or `OFF`. To display enum values you can [use the convert field transform](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/#convert-field-type).

{{< youtube id="FG0hBFfgpps" >}}

### View percent change in stat visualizations

<!--Nathan Marrs-->

_Generally available in all editions of Grafana_

You can now view percent change in stat visualizations. This makes it easier to understand your data by showing how metrics are changing over time.

{{< youtube id="mB9FU0myZo8" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/stat/#show-percent-change)

#### Apply data transformations to annotations

<!--Nathan Marrs-->

_Generally available in all editions of Grafana_

You can now apply data transformations to annotation data. For example, you can now configure how exemplar data is displayed in tooltips.

{{< video-embed src="/media/docs/grafana/screen-recording-10-3-data-transformations-annotation-support.mp4" caption="Configure how exemplar data appears in tooltip" >}}

### New Transformations UI experience and documentation upgrades

<!--Jev Forsberg-->

_Generally available in all editions of Grafana_

We've revamped the Transformations user interface to make it cleaner, more user-friendly, and overall better for visualizing, selecting, and comprehending transformation options for your data.

#### Improved UI\*

In the past, transformations were applied through a dropdown menu, indicated solely by text names like Merge, Sort, JoinByLabels, etc. Now, we've introduced a much more user-friendly interface. A convenient drawer allows seamless access to all transformation options, each accompanied by visual/graphical representations and a brief description. These enhancements are designed to enhance the user's comprehension of their data transformation choices.

{{< figure src="/media/docs/grafana/transformations/transformations_ui_drawer_selector.png" caption="The new Transformation UI drawer" alt="Transformation UI drawer" >}}

#### In-App documentation

We've also streamlined the user experience by integrating documentation directly into the core Grafana application. Gone are the days of navigating to a separate browser page for Transformation documentation. Now, users can conveniently access documentation within the app interface, providing a more seamless and efficient way to understand and utilize various features. This enhancement aims to save time and enhance user convenience, ensuring that valuable information is readily available at their fingertips.

{{< figure src="/media/docs/grafana/transformations/transformations_internal_documentation.png" caption="Transformation documentation is now internally available inside the Grafana app itself." alt="Transformation documentation internally available" >}}

### Copy and paste time range

<!--Haris Rozajac-->

_Generally available in all editions of Grafana_

Copying and pasting time range in the time range picker is now available. For example, you can copy a time range in **Explore** and paste it into **Dashboards** and vice versa. You can also copy and paste a time range using the new keyboard shortcuts `t+c` and `t+v`, respectively.

## Profiles

### Trace to Profiles\*

<!--Joey Tawadrous-->

_Experimental in all editions of Grafana_

Using Trace to profiles, you can use Grafana’s ability to correlate different signals by adding the functionality to link between traces and profiles.

**Trace to profiles** lets you link your Grafana Pyroscope data source to tracing data. When configured, this connection lets you run queries from a trace span into the profile data.

There are two ways to configure the trace to profiles feature:

- Use a simplified configuration with default query, or
- Configure a custom query where you can use a template language to interpolate variables from the trace or span.

{{< figure src="/static/img/docs/tempo/profiles/tempo-trace-to-profile.png" caption="Trace to profiles screenshot" alt="Trace to profiles screenshot" >}}

To try out **Trace to profiles**, enable the 'traceToProfiles' feature toggle.

If you would also like to try out the **Embedded Flame Graph** feature, please enable the 'tracesEmbeddedFlameGraph' feature toggle.

Note: in order to determine that there is a profile for a given span and render the 'Profiles for this span' button or the embedded flame graph in the span details, the 'pyroscope.profile.id' key-value pair must exist in your span tags.

{{< youtube id="AG8VzfFMLxo" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#trace-to-profiles)

### FlameGraph: Collapsing similar items in the graph\*

<!--Andrej Ocenas-->

_Experimental in all editions of Grafana_

Sometimes profile stacks contain lots of levels with similar repeating items, for example long stacks of framework code that usually isn't of interest but takes up a lot of visual real estate. With this feature, instead of rendering all of the similar items we render only one and allow to expand those collapsed items on demand.

To try it out, enable the ‘traceToProfiles’ feature toggle. To enable it in your Grafana Cloud stack, contact Grafana Support.

{{< youtube id="Y1c32Cf5nSE" >}}

## Alerting

### Alerting insights\*

<!-- George Robinson -->

_Generally available in all editions of Grafana_

Use Alerting insights to monitor your alerting data, discover key trends about your organization’s alert management performance, and find patterns in why things go wrong.

### Export alerting resources to Terraform

<!-- Yuri Tseretyan-->

_Generally available in all editions of Grafana_

Export your alerting resources, such as alert rules, contact points, and notification policies as Terraform resources. A new “Modify export” mode for alert rules enables you to edit provisioned alert rules and export a modified version.

### Contact points list view redesign

<!-- Brenda Muir -->

_Generally available in all editions of Grafana_

The Contact points list view has been redesigned and split into two tabs: Contact Points and Notification Templates, making it easier to view all contact point information at a glance. You can now search for name and type of contact points and integrations, view how many notification policies each contact point is being used for, and navigate directly to the linked notification policies.

{{< youtube id="_eOhSmbYK8Q" >}}

### Create alerts from panels\*

<!-- Brenda Muir -->

_Generally available in all editions of Grafana_

Create alerts from dashboard panels. You can reuse the panel queries and create alerts based on them.

### Support for adding responders to Opsgenie alerting contact point\*

<!--Ryan Kehoe-->

_Generally available in all editions of Grafana_

The Opsgenie contact point has been extended to allow users to optionally fill out responder information for their integration. Responders tell Opsgenie who an alert should notify according to their escalation policies and routing rules.

### Recovery thresholds for alerts

<!--Ryan Kehoe-->

_Generally available in all editions of Grafana_

To reduce the noise of flapping alerts, you can set a recovery threshold different to the alert threshold.

Flapping alerts occur when a metric hovers around the alert threshold condition and may lead to frequent state changes, resulting in too many notifications being generated.

## Logs

### Logs Table UI

<!--Galen Kistler-->

_Available in public preview in all editions of Grafana_

Table view was created to help facilitate ease of use in a point and click UI, as opposed to datasource specific query language formatting options, like loki's line_format.

Tables can be configured and shared with team members via explore URLs or by adding the table to a dashboard panel.

{{< youtube id="OAZeqqNpEjc" >}}

## Data sources

### Data source Admin permission\*

<!--Ieva Vasiljeva-->

_Generally available in Grafana Enterprise and Grafana Cloud_

In addition to `Query` and `Edit` access, you can now grant users, teams, or basic roles `Admin` access to data sources. Users with `Admin` access to a data source can grant and revoke permissions to the data source, as well as to manage query caching settings for the data source. Users are automatically granted `Admin` access to data sources that they create.

### Redshift and Athena: Async query caching

<!--Isabella Siu-->

_Generally available in Grafana Enterprise, Grafana Cloud Advanced and Cloud Pro_

Introducing query caching for async queries in the Athena and Redshift data source plugins. We previously introduced async queries for the Athena and Redshift plugins, and this feature adds support for caching those queries. To use this, you must have query caching enabled for the Athena or Redshift data source you wish to cache. This feature was previously available behind a feature toggle and is now generally available and enabled by default.

{{< admonition type="note" >}}

The `useCachingService` feature toggle must also be enabled to use this feature.

{{< /admonition >}}

### Loki data source improvements: "or" filter syntax, filter by label types, derived fields by labels

<!--Sven Grossmann-->
<!--enablement videos to come?-->

_Generally available in all editions of Grafana_

Introducing several improvements to the Loki data source.

{{< youtube id="ievPSzmCrAk" >}}

#### Line filter "or" syntax\*

Loki's line filter syntax is great to find specific substrings of your log lines. If users want to find multiple different substrings it was cumbersome to use the regex `=~` operator. With this change it is possible to chain multiple strings with the existing filter operators.

Example:

```
{app="foo"} |= "foo" or "bar" != "baz" or "qux"
```

#### Filter based on label type\*

Grafana users can use the action buttons in the log details to filter for specific labels. Those would be always added as a LabelFilter expression regardless of the type of the label. Now, filtered labels will be added either to the stream selector if the label is an indexed label, or as a LabelFilter expression if the label is a parsed label or part of structured metadata.

#### Derived fields based on labels\*

Derived fields or data links are a concept to add correlations based on your log lines. Previously it was only possible to add derived fields based on a regular expression of your log line and doing it based on labels was not possible. With this change derived fields can be added either based on a regex of a log line or based on a label, parsed label or structured metadata.

The following example would add the derived field `traceID regex` based on a regular expression and another `app label` field based on the `app` label.

{{< figure src="/media/docs/grafana/2024-01-05_loki-derived-fields.png" alt="Derived fields added based on a regular expression and an app label">}}

### InfluxDB native SQL support

<!--Ismail Simsek-->

_Generally available in all editions of Grafana_

InfluxDB introduced [a new version, 3.0](https://www.influxdata.com/blog/introducing-influxdb-3-0/), in April. With this new version, InfluxDB has put [Flux in maintenance mode](https://www.influxdata.com/blog/the-plan-for-influxdb-3-0-open-source/#heading4). But with the new version we have a new querying language, [Native SQL](https://www.influxdata.com/products/sql/). With v10.3.0, Grafana has built-in support for SQL query language in InfluxDB.

All you need to do is set up your InfluxDB Cloud Account and create your InfluxDB data source on Grafana with the query language "SQL" selected.

{{< youtube id="jGclGsv5PBA" >}}

## Authentication and authorization

### Grafana Anonymous Access\*

<!--Eric Leijonmarck-->

_Generally available in Grafana Open Source and Enterprise_

We've identified a need for users who enable anonymous authentication to monitor the anonymous devices connected to their Grafana instance. This feature is part of our ongoing efforts to enhance control and transparency regarding anonymous usage.

Anonymous access now allows users, including those in open-source and enterprise self-managed environments, to view and monitor their anonymous access. They can also set a device limit, configuring a specific number of anonymous devices to connect to their instance.

Once this limit is reached, any new devices attempting to connect will be denied access until existing devices disconnect.

The anonymous devices feature improves the management and monitoring of anonymous access within your Grafana instance.

**Anonymous Device:**

When anonymous access has been enabled, any device which accesses Grafana in the last 30 days (without logging in) is considered an active anonymous device. Users can now view anonymous devices on the users page, anonymous usage statistics, including the count of devices and users over this period.

**Grafana UI:**

- Navigate to Administration -> Users to access the anonymous devices tab.

- A new statistic has been added to the Usage & Stats page, displaying active anonymous devices from the last 30 days.

{{< youtube id="B72X3_9e-ds" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/grafana/)

# What’s new in Grafana v10.4

Welcome to Grafana 10.4! This minor release contains some notable improvements in its own right, as well as early previews of functionality we intend to turn on by default in Grafana v11. Read on to learn about a quicker way to set up alert notifications, an all-new UI for configuring single sign-on, and improvements to our Canvas, Geomap, and Table panels.

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v10.4, check out our [Upgrade Guide](https://archive.grafana.com/docs/grafana/v10.4/upgrade-guide/upgrade-v10.4/).

<!-- Template below

## Feature
<!-- Name of contributor -->
<!--_[Generally available | Available in private/public preview | Experimental] in Grafana [Open Source, Enterprise, all editions of Grafana, some combination of self-managed and Cloud]_
Description. Include an overview of the feature and problem it solves, and where to learn more (like a link to the docs).
{{< admonition type="note" >}}
Use full URLs for links. When linking to versioned docs, replace the version with the version interpolation placeholder (for example, <GRAFANA_VERSION>, <TEMPO_VERSION>, <MIMIR_VERSION>) so the system can determine the correct set of docs to point to. For example, "https://grafana.com/docs/grafana/latest/administration/" becomes "https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/".
{{< /admonition >}}

<!--Add an image, GIF or video  as below-->

<!--{{< figure src="/media/docs/grafana/dashboards/WidgetVizSplit.png" max-width="750px" caption="DESCRIPTIVE CAPTION" >}}

<!--Learn how to upload images here: https://grafana.com/docs/writers-toolkit/write/image-guidelines/#where-to-store-media-assets-->
<!---->

## Dashboards and visualizations

### AngularJS plugin warnings in dashboards

<!-- #grafana-deprecate-angularjs-->

_Generally available in all editions of Grafana_

AngularJS support in Grafana was deprecated in v9 and will be turned off by default in Grafana v11. When this happens, any plugin which depended on AngularJS will not load, and dashboard panels will be unable to show data.

To help you understand where you may be impacted, Grafana now displays a warning banner in any dashboard with a dependency on an AngularJS plugin. Additionally, warning icons are present in any panel where the panel plugin or underlying data source plugin has an AngularJS dependency.

This complements the existing warnings already present on the **Plugins** page under the administration menu.

In addition, you can use our [detect-angular-dashboards](https://github.com/grafana/detect-angular-dashboards) open source tool, which can be run against any Grafana instance to generate a report listing all dashboards that have a dependency on an AngularJS plugin, as well as which plugins are in use. This tool also supports the detection of [private plugins](https://grafana.com/legal/plugins/) that are dependent on AngularJS, however this particular feature requires Grafana v10.1.0 or higher.

Use the aforementioned tooling and warnings to plan migrations to React based [visualizations](https://grafana.com/docs/grafana/latest/panels-visualizations/) and [data sources](https://grafana.com/docs/grafana/latest/datasources/) included in Grafana or from the [Grafana plugins catalog](https://grafana.com/grafana/plugins/).

To learn more, refer to the [Angular support deprecation](https://archive.grafana.com/docs/grafana/v12.2/developers/angular_deprecation/), which includes [recommended alternative plugins](https://archive.grafana.com/docs/grafana/v12.2/developers/angular_deprecation/angular-plugins/).

{{< youtube id="XlEVs6g8dC8" >}}

[Documentation](https://archive.grafana.com/docs/grafana/v12.2/developers/angular_deprecation/)

### Data visualization quality of life improvements

<!-- Nathan Marrs -->

_Generally available in all editions of Grafana_

We’ve made a number of small improvements to the data visualization experience in Grafana.

#### Geomap geojson layer now supports styling

You can now visualize geojson styles such as polygons, point color/size, and line strings. To learn more, [refer to the documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/geomap/#geojson-layer).

![Geomap marker symbol alignment](/media/docs/grafana/screenshot-grafana-10-4-geomap-geojson-styling-support.png)

#### Canvas elements now support snapping and aligning

You can precisely place elements in a canvas with ease as elements now snap into place and align with one another.

{{< video-embed src="/media/docs/grafana/screen-recording-10-4-canvas-element-snapping.mp4" caption="Canvas element snapping and alignment" >}}

#### View data links inline in table visualizations

You can now view your data links inline to help you keep your tables visually streamlined.

![Table inline datalink support](/media/docs/grafana/gif-grafana-10-4-table-inline-datalink.gif)

### Create subtables in table visualizations with Group to nested tables

<!-- Nathan Marrs -->

_Available in public preview in all editions of Grafana_

You can now create subtables out of your data using the new **Group to nested tables** transformation. To use this feature, enable the `groupToNestedTableTransformation` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/#preview-feature-toggles).

{{< video-embed src="/media/docs/grafana/screen-recording-10-4-table-group-to-nested-table-transformation.mp4" caption="Group to nested tables transformation" >}}

### Set library panel permissions with RBAC

<!-- #grafana-dashboards -->

_Generally available in Grafana Enterprise and Grafana Cloud_

We've added the option to manage library panel permissions through role-based access control (RBAC). With this feature, you can choose who can create, edit, and read library panels. RBAC provides a standardized way of granting, changing, and revoking access when it comes to viewing and modifying Grafana resources, such as dashboards, reports, and administrative settings.

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/manage-library-panels/)

### Tooltip improvements

<!--Adela Almasan-->

_Available in public preview in all editions of Grafana_

We’ve made a number of small improvements to the way tooltips work in Grafana. To try out the new tooltips, enable the `newVizTooltips` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/).

**Copy on click support**

You can now copy the content from within a tooltip by clicking on the text.

![Tooltip](/media/docs/grafana/gif-grafana-10-4-tooltip–copy.gif)

**Scrollable content**

You can now scroll the content of a tooltip, which allows you to view long lists. This is currently supported in the time series, candlestick, and trend visualizations. We'll add more improvements to the scrolling functionality in a future version.

![Tooltip](/media/docs/grafana/gif-grafana-10-4-tooltip-content-scroll.gif)

**Added tooltip options for candlestick visualization**

The default tooltip options are now also visible in candlestick visualizations.

**Hover proximity option in time series**

We've added a tooltip hover proximity limit option (in pixels), which makes it possible to reduce the number of hovered-over data points under the cursor when two datasets are not aligned in time.

![Time Series hover proximity](/media/docs/grafana/gif-grafana-10-4-hover-proximity.gif)

## Return to previous

<!-- #grafana-frontend-platform-->

_Available in public preview in all editions of Grafana_

When you're browsing Grafana - for example, exploring the dashboard and metrics related to an alert - it's easy to end up far from where you started and hard get back to where you came from. The ‘Return to previous’ button is an easy way to go back to the previous context, like the alert rule that kicked off your exploration. This first release works for Alerts, and we plan to expand to other apps and features in Grafana in future releases to make it easier to navigate around.

Return to Previous is rolling out across Grafana Cloud now. To try Return to Previous in self-managed Grafana, turn on the `returnToPrevious` [feature toggle](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/feature-toggles/) in Grafana v10.4 or newer.

{{< youtube id="-Y3qPfD2wrA" >}}

{{< admonition type="note" >}}
The term **context** refers to applications in Grafana like Incident and OnCall, as well as core features like Explore and Dashboards.

To notice a change in your context, look at Grafana's breadcrumbs. If you go from _Home > **Dashboards**_ to _Home > **Explore**_, you've changed context. If you go from _Home > **Dashboards** > Playlist > Edit playlist_ to _Home > **Dashboards** > Reporting > Settings_, you are in the same context.
{{< /admonition >}}

## Alerting

### Simplified Alert Notification Routing

<!-- #alerting -->

_Generally available in all editions of Grafana_

This feature simplifies your options for configuring where your notifications are sent when an alert rule fires. Choose an existing contact point directly from within the alert rule creation form without the need to label match notification policies.  You can also set optional muting, grouping, and timing settings directly in the alert rule.

Simplified routing inherits the alert rule RBAC, increasing control over notification routing while preventing accidental notification policy updates, ensuring critical notifications make it to their intended contact point destination.

To try out Simplified Alert Notification Routing enable the `alertingSimplifiedRouting` feature toggle.

{{< youtube id="uBBQ-_pWSNs" >}}

### Grafana Alerting upgrade with rule preview

<!-- #alerting -->

_Generally available in all editions of Grafana_

Users looking to migrate to the new Grafana Alerting product can do so with confidence with the Grafana Alerting migration preview tool. The migration preview tool allows users to view, edit, and delete migrated rules prior cutting over, with the option to roll back to Legacy Alerting.

[Documentation](https://grafana.com/docs/grafana/v10.4/alerting/set-up/migrating-alerts/#upgrade-with-preview-recommended)

### Rule evaluation spread over the entire evaluation interval

<!-- #alerting -->

_Generally available in all editions of Grafana_

Grafana Alerting previously evaluated rules at the start of the evaluation interval. This created a sudden spike of resource utilization, impacting data sources. Rule evaluation is now spread over the entire interval for smoother performance utilization of data sources.

### UTF-8 Support for Prometheus and Mimir Alertmanagers

<!-- #alerting -->

_Generally available in all editions of Grafana_

Grafana can now be used to manage both Prometheus and Mimir Alertmanagers with UTF-8 configurations. For more information, please see the
[release notes for Alertmanager 0.27.0](https://github.com/prometheus/alertmanager/releases).

## Authentication and authorization

### SSO Settings UI and Terraform resource for configuring OAuth providers

<!-- #proj-grafana-sso-config, #identity-access or Mihaly Gyongyosi (@Misi) -->

_Available in public preview in all editions of Grafana_

Configuring OAuth providers was a bit cumbersome in Grafana: Grafana Cloud users had to reach out to Grafana Support, self-hosted users had to manually edit the configuration file, set up environment variables, and then they had to restart Grafana. On Cloud, the Advanced Auth page is there to configure some of the providers, but configuring Generic OAuth hasn’t been available until now and there was no way to manage the settings through the Grafana UI, nor was there a way to manage the settings through Terraform or the Grafana API.

Our goal is to make setting up SSO for your Grafana instance simple and fast.

To get there, we are introducing easier self-serve configuration options for OAuth in Grafana. All of the currently supported OAuth providers are now available for configuration through the Grafana UI, Terraform and via the API. From the UI, you can also now manage all of the settings for the Generic OAuth provider.

We are working on adding complete support for configuring all other supported OAuth providers as well, such as GitHub, GitLab, Google, Microsoft Azure AD and Okta. You can already manage some of these settings via the new self-serve configuration options, and we’re working on adding more at the moment.

![Screenshot of the Authentication provider list page](/media/docs/grafana-cloud/screenshot-sso-settings-ui-public-prev-v10.4.png)

{{< youtube id="xXW2eRTbjDY" >}}

[Documentation](https://grafana.com/docs/grafana/next/setup-grafana/configure-access/configure-authentication/)

## Data sources

{{< admonition type="note" >}}
The following data sources are released separately from Grafana itself. They are included here for extra visibility.
{{< /admonition >}}

### PagerDuty enterprise data source for Grafana

<!-- #enterprise-datasources -->

_Generally available in Grafana Enterprise and Grafana Cloud_

PagerDuty enterprise data source plugin for Grafana allows you to query incidents data or visualize incidents using annotations.

{{< admonition type="note" >}}
Plugin is currently in a preview phase.
{{< /admonition >}}

You can find more information and how to configure the plugin in the [documentation](https://grafana.com/docs/plugins/grafana-pagerduty-datasource/latest/).

Screenshots:

{{< figure src="/media/docs/plugins/PagerDuty-incidents-annotation.png" caption="PagerDuty data source annotation editor" alt="PagerDuty data source annotation editor" >}}

{{< figure src="/media/docs/plugins/PagerDuty-incidents-real-life-example.png" caption="Incidents annotations from PagerDuty data source on a dashboard panel" alt="Incidents annotations from PagerDuty data source on a dashboard panel" >}}

{{< youtube id="dCklm2DaVqQ" >}}

### SurrealDB Data Source

<!-- #grafana-partner-datasources, @adamyeats-->

_Experimental in all editions of Grafana_

A SurrealDB data source has been [added to the Plugin Catalog](https://grafana.com/grafana/plugins/grafana-surrealdb-datasource/), enabling the integration of [SurrealDB](https://surrealdb.com/), a real-time, multi-model database, with Grafana's visualization capabilities. This datasource allows users to directly query and visualize data from SurrealDB within Grafana, using SurrealDB's query language.

The SurrealDB data source launches with just the basics today. You can write queries in SurrealQL using the built-in query editor, although many Grafana features like macros are not supported for now.

You can find more information and how to configure the plugin [on Github](https://github.com/grafana/surrealdb-datasource).

{{< figure src="/media/images/dashboards/surrealdb-dashboard-example.png" alt="Grafana dashboard using SurrealDB data source" >}}

[Documentation](https://grafana.com/grafana/plugins/grafana-surrealdb-datasource/)

## Table Visualization for Logs

<!-- #observability-logs -->

_Generally available in all editions of Grafana_

The table visualization for logs, announced in public preview for Grafana 10.3, is generally available in Cloud (all editions) and with Grafana 10.4.

New to the table visualization with 10.4:

- the ability to sort columns
- data type autodetection of fields
- autodetection and clean formatting of json fields

Try it out today!

# What's new in Grafana v11.0

<!-- vale GoogleWe = NO -->
<!-- vale We = NO -->

# What’s new in Grafana v11.0

Welcome to Grafana 11.0! This release contains some major improvements: most notably, the ability to explore your Prometheus metrics and Loki logs without writing any PromQL or LogQL, using Explore Metrics and Explore Logs. The dashboard experience is better than ever with edit mode for dashboards, AI-generated dashboard names and descriptions, and general availability for subfolders. You can also take advantage of improvements to the canvas and table visualizations, new transformations, a revamp of the Alert Rule page, and more.

{{< youtube id="96ZXShRyMB8" >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v11.0, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v11/upgrade-v11.0/).

## Breaking changes

For Grafana v11.0, we've also provided a list of [breaking changes](https://archive.grafana.com/docs/grafana/v11.0/breaking-changes/breaking-changes-v11-0/) to help you upgrade with greater confidence. For information about these along with guidance on how to proceed, refer to [Breaking changes in Grafana v11.0](https://archive.grafana.com/docs/grafana/v11.0/breaking-changes/breaking-changes-v11-0/).

<!-- Template below

## Feature
<!-- Name of contributor -->
<!--_[Generally available | Available in private/public preview | Experimental] in Grafana [Open Source, Enterprise, all editions of Grafana, some combination of self-managed and Cloud]_
Description. Include an overview of the feature and problem it solves, and where to learn more (like a link to the docs).
{{< admonition type="note" >}}
Use full URLs for links. When linking to versioned docs, replace the version with the version interpolation placeholder (for example, <GRAFANA_VERSION>, <TEMPO_VERSION>, <MIMIR_VERSION>) so the system can determine the correct set of docs to point to. For example, "https://grafana.com/docs/grafana/latest/administration/" becomes "https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/".
{{< /admonition >}}

<!--Add an image, GIF or video  as below-->

<!--{{< figure src="/media/docs/grafana/dashboards/WidgetVizSplit.png" max-width="750px" caption="DESCRIPTIVE CAPTION" >}}

<!--Learn how to upload images here: https://grafana.com/docs/writers-toolkit/write/image-guidelines/#where-to-store-media-assets-->
<!---->

## Explore Metrics and Logs

### Explore Metrics

<!-- #proj-datatrails-dev, PM: Jay Goodson, Engineering: Darren Janeczek, André Pereira, Design: Catherine Gui -->

_Public preview in all editions of Grafana_

Explore Metrics is a query-less experience for browsing Prometheus-compatible metrics. Search for or filter to find a metric. Quickly find related metrics - all in just a few clicks. You do not need to learn PromQL! With Explore Metrics, you can:

- easily slice and dice metrics based on their labels, so you can see anomalies right away
- See the right visualization for your metric based on its type (e.g. gauge vs. counter) without building it yourself
- surface other metrics relevant to the current metric
- “explore in a drawer” - expand a drawer over a dashboard with more content, so you don’t lose your place
- view a history of user steps when navigating through metrics and their filters
- easily pivot to other related telemetry - IE, logs or traces

… all without writing any queries!

To learn more, refer to [Explore Metrics](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/explore-metrics/) as well as the following video demo:

{{< youtube id="JbaPufQs5LY" >}}

### Explore Logs

<!-- Slack: #proj-lokiapp-dev, PM: Steven Dungan, Engineering: Andrew Stucky, Cyril Tovena, Design: Joan Wortman -->

_Experimental in Grafana Open Source and Enterprise_

Explore Logs is a queryless experience for exploring Loki logs - no LogQL required! The primary interaction modes are point-and-click based on log volume, similar to Explore Metrics.

Highlights:

- View log volume and log line samples when you first land in Explore Logs (no more "blank screen!")
- Explore additional labels and detected fields in a similar way, focusing on volume and distribution; add them to your "query" to refine your logs search without needing LogQL
  – See common patterns in your log lines, to easily filter out noise or focus in on anomalies
- For power users, an easy way to hop into the familiar Explore while preserving context

Explore Logs is Open Source, and experimental - some papercuts are to be expected. Give it a try and let us know what you think!

## Dashboards and visualizations

### Scenes powered Dashboards

<!-- #grafana-dashboards -->

_Available in public preview in all editions of Grafana_

For the past few months we've been working on a major update of our **Dashboards** architecture and migrated it to the Scenes library. This migration provides us with more stable, dynamic, and flexible dashboards as well as setting the foundation for what we envision the future of Grafana dashboards will be. Here are two of the improvements that are being introduced as part of this work.

#### Edit mode

It can be difficult to efficiently navigate through the visually cluttered options during the dashboard editing process. With the introduction of the edit mode, we aim to provide an easier way to discover and interact with the dashboard edit experience.

#### Fixed positioning of template variables and time picker

We moved the time picker into the dashboard canvas and now, together with template variables, it will stick to the top as you scroll through your dashboard. This has historically been a very [requested feature](https://github.com/grafana/grafana/issues/11166) that we're very happy to be able to finally roll out!

#### Known limitations

- The [variable dependency graph](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/inspect-variable/) is not yet available.
- It's no longer possible to switch a regular panel to a library panel from the edit view.

If you want to learn more, in detail, about all the improvements we've made, don't miss our blog post.

{{< youtube id="kcKwBhvrsHc" >}}

### Scenes for viewers

<!-- #grafana-dashboards, Dominik Prokop, Natalia Bernarte -->

_Generally available in all editions of Grafana_

Dashboards, when accessed by users with the Viewer role, are now using the Scenes library. Those users shouldn't see any difference in the dashboards apart from two small changes to the user interface (UI): the variables UI has slightly changed and the time picker is now part of the dashboard container.

Dashboards aren't affected for users in other roles.

This is the first step towards a more robust and dynamic dashboarding system that we'll be releasing in the upcoming months.

### Subfolders

<!-- #wg-nested-folders -->

_Generally available in all editions of Grafana_

Subfolders are here at last!

Some of you want subfolders in order to keep things tidier. It’s easy for dashboard sprawl to get out of control, and setting up folders in a nested hierarchy helps with that.

Others of you want subfolders in order to create nested layers of permissions, where teams have access at different levels that reflect their organization’s hierarchy.

We are thrilled to bring this long-awaited functionality to our community of users! Subfolders are currently being rolled out to Grafana Cloud instances and will be generally available to all Grafana users for the Grafana 11 release.

**Just a quick note**: the upgrade to enable subfolders can cause some issues with alerts in certain cases. We think these cases are pretty rare, but just in case, you’ll want to check for this:

If you've previously set up a folder that uses a forward slash in its name, and you have an alert rule in that folder, and the notification policy is set to match that folder's name, notifications will be sent to the default receiver instead of the configured receiver.

To correct this, take the following steps:

- Create a copy of the affected routes
- Rewrite the matchers for the new copy. For example, if the original matcher was `grafanafolder=folder_with/in_title`, then the new route matcher will be `grafana_folder=folder_with/_in_title`
- After rewriting the matchers, you can delete the old routes.

If you use file provisioning, you can upgrade and update the routes at the same time.

{{< youtube id="R9mehA0EssU" >}}

### Use AI to generate titles and descriptions for panels and dashboards

<!-- Ivan Ortega -->

_Generally available in all editions of Grafana_

You can now use generative AI to assist you in your Grafana dashboards. So far generative AI can help you generate **panel and dashboard titles and descriptions** - You can now generate a title and description for your panel or dashboard based on the data you've added to it. This is useful when you want to quickly visualize your data and don't want to spend time coming up with a title or description.

Make sure to enable and configure Grafana's LLM app plugin. For more information, refer to the [Grafana LLM app plugin documentation](https://grafana.com/docs/grafana-cloud/alerting-and-irm/machine-learning/llm-plugin/).

When enabled, look for the **✨ Auto generate** option next to the **Title** and **Description** fields in your panels and dashboards, or when you press the **Save** button.

![Auto-generate a panel description using AI](/media/docs/grafana/dashboards/auto-generate-description-10-2.gif)

{{< youtube id="s61WHREHuYE" >}}

### Improvements to the canvas visualization

<!-- #dataviz-squad -->

_Generally available in all editions of Grafana_

We've made a number of improvements to the canvas visualization.

#### Enhanced flowcharting functionality

With this release, we've updated the canvas visualization to include much-requested flowcharting features. These improvements are:

- Addition of widely-used elements: cloud, parallelogram, and triangle.
- Addition of midpoint controls so that the connectors no longer have to be straight lines.
- Addition of more connector styles including dashed lines as well as corner radius and direction control.
- Horizontal and vertical snapping for connectors.
- Addition of rounded corner styling for elements.
- Ability to rotate elements in the canvas.

#### Universal data link support

We've updated data links so that you can add them to almost all elements or element properties that are tied to data. Previously, you could only add data links to text elements or elements that used the `TextConfig` object. This update removes that limitation.

{{< admonition type="note" >}}
This update doesn't apply to the drone and button elements.
{{< /admonition >}}

{{< youtube id="0iO2gqv0XNA" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/canvas/)

### Infinite panning for the canvas visualization

<!-- Nathan Marrs, #grafana-dataviz -->

_Available in public preview in all editions of Grafana_

With the newly added **Infinite panning** editor option, you can now view and navigate very large canvases. This option is displayed when the **Pan and zoom** switch is enabled.

To try out this feature, you must first enable the `canvasPanelPanZoom` feature toggle.

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/canvas/)

### Colored table rows with conditional formatting

<!-- #grafana-dataviz, Kyle Cunningham -->

_Generally available in all editions of Grafana_

Grafana 11 adds the ability to color full table rows using the **Colored background** cell type of the table visualization. When you configure fields in a table to use this cell type, an option to apply the color of the cell to the entire row becomes available.

{{< figure src="/static/img/docs/tables/colored-rows.png" max-width="500px" alt="Colored row background" class="docs-image--no-shadow" >}}

This feature is useful for a wide variety of use cases including mapping status fields to colors (for example, `info`, `debug`, `warning`) and allowing rows to be colored based on threshold values. This is one of the first steps in making formatting tables more seamless, and allows for quick scanning of data using the table visualization.

To learn more, refer to the [documentation for the Colored background cell type](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/table/#color-background-gradient-or-solid).

{{< youtube id="PLfADTtCnmg" >}}

### Set threshold colors in the Config from query transformation

<!--  #grafana-dataviz" -->

_Generally available in all editions of Grafana_

You now have the ability to customize specific colors for individual thresholds when using the **Config from query results** transformer. Previously, when you added multiple thresholds, they all defaulted to the same color, red. With this addition, you gain the flexibility to assign distinct colors to each threshold.

This feature addresses a common pain point highlighted by users. With customizable threshold colors, you now have greater control over your data representation, fostering more insightful and impactful analyses across diverse datasets.

### Substring matcher added to the Filter by value transformation

<!-- #grafana-dataviz -->

_Generally available in Grafana Cloud and Open Source_

This update to the **Filter data by values** transformation simplifies data filtering by enabling partial string matching on field values thanks to two new matchers: **Contains substring** and **Does not contain substring**. With the substring matcher built into the **Filter data by values** transformation, you can efficiently filter large datasets, displaying relevant information with speed and precision. Whether you're searching for keywords, product names, or user IDs, this feature streamlines the process, saving time and effort while ensuring accurate data output.

In the **Filter data by values** transformation, simply add a condition, choose a field, choose your matcher, and then input the string to match against.

This update will be rolled out to customers over the next few weeks.

{{< video-embed src="/media/docs/grafana/substring-matcher.mp4" >}}

## Reporting

### PDF export improvements

<!-- #grafana-sharing -->

_Available in public preview in Grafana Cloud and Enterprise_

Introducing a major performance improvement for the PDF export feature.

Are you tired of waiting for your PDF to be generated or your report to be sent? We're working on a major update of the dashboard-to-PDF feature to make it faster for large dashboards. The generation time will no longer be proportional to the number of panels in your dashboard. As an example, an SLO dashboard containing around 200 panels has gone from taking more than seven minutes to be generated to only eleven seconds.

This update also fixes all [caveats](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/create-reports/#caveats) related to rendering a report with panels or rows set to repeat by a variable, like rendering repeating panels inside collapsed rows.

To try out this update, enable the `newPDFRendering` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/).

{{< youtube id="ypk1usnE7D8" >}}

## Alerting

### Keep Last State for Grafana Managed Alerting

<!-- "#alerting" -->

_Generally available in all editions of Grafana_

(Re-)introducing "Keep Last State" to Grafana managed alert rules.

You can now choose to keep the last evaluated state of an alert rule when that rule produces "No Data" or "Error" results. Simply choose the "Keep Last State" option for no data or error handling when editing a rule. Refer to the Alerting documentation on state and health of alert rules for more information.[](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/state-and-health/#state-and-health-of-alert-rules)

### Alert detail view redesign

<!-- Gilles deMey -->

_Generally available in all editions of Grafana_

The new alert rule detail view has a new look and feel with helpful metadata at the top. The namespace and group are shown in the breadcrumb navigation. This is interactive and can be used to filter rules by namespace or group. The rest of the alert detail content is split up into tabs:

**Query and conditions**

View the details of the query that is used for the alert rule, including the expressions and intermediate values for each step of the expression pipeline. A graph view is included for range queries and data sources that return time series-like data frames.

**Instances**

Explore each alert instance, its status, labels and various other metadata for multi-dimensional alert rules.

**History**

Explore the recorded history for an alert rule.

**Details**

Debug or audit using the alert rule metadata and view the alert rule annotations.

![Image shows details of an alert rule](/media/docs/alerting/alert-detail-view.png)

### RBAC for alert rule provisioning APIs

<!-- Yuri Tseretyan -->

_Generally available in all editions of Grafana_

The Alerting Provisioning HTTP API has been updated to enforce Role-Based Access Control (RBAC).

- For Grafana OSS, users with the **Editor** role can now use the API.
- For Grafana Enterprise and Grafana Cloud, users with the role **Rules Writer** and **Set Provisioning status** can access the API and limit access to alert rules that use a particular data source.
- Other roles related to provisioning, for example **Access to alert rules** provisioning API still work.

## Data sources

### Removal of old Tempo Search and Loki Search in Tempo

<!-- Joey Tawadrous -->

_Generally available in all editions of Grafana_

#### Removal of old Tempo Search tab

In Grafana v10.1, we added a Tempo search editor powered by TraceQL (search tab). We also recommended using this new editor over the older non-TraceQL powered editor.

The older non-TraceQL powered editor has been removed. Any existing queries using the older editor will be automatically migrated to the new TraceQL-powered editor.

The new TraceQL-powered editor makes it much easier to build your query by way of static filters, better input/selection validation, copy query to the TraceQL tab, query preview, dedicated status filter, and the ability to run aggregate by (metrics summary) queries.

Refer to [Query tracing data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/) to learn more.

#### Removal of Loki Search tab in Tempo

The Loki Search tab has been around since before we could natively query Tempo for traces.
This search is used by a low number of users in comparison to the TraceQL-powered editor (Search tab) or the TraceQL tab itself.

If you would like to see what logs are linked to a specific trace or service, you can use the [Trace to logs feature](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#trace-to-logs), which provides an easy way to create a custom link and set an appropriate time range if necessary.

### MSSQL: Windows Active Directory (Kerberos) authentication

<!--#grafana-partner-datasources, @adam -->

_Generally available in Grafana Open Source and Enterprise_

You can now use Windows Active Directory (or Kerberos) to authenticate to MSSQL servers from Grafana.

There are four primary ways to authenticate from Grafana to a MSSQL instance with Windows Active Directory:

1. Windows Active Directory username and password
1. Specify the path to a valid [keytab file](https://web.mit.edu/kerberos/krb5-1.12/doc/basic/keytab_def.html).
1. Specify the path to an up to date [credential cache](https://web.mit.edu/kerberos/krb5-1.12/doc/basic/ccache_def.html).
1. Specify the path to a JSON document that holds information about several credential caches and the user and database for each one.

To get started, refer to the [Getting Started documentation for MSSQL](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/getting-started/get-started-grafana-ms-sql-server/#windows-active-directory-kerberos).

## Authentication and authorization

### New strong password policy

<!-- #iam-identity-dev -->

_Available in public preview in Grafana Open Source and Enterprise_

If you manage your users using Grafana's built-in basic authorization as an identity provider, consider enabling our new strong password policy feature.

Starting with Grafana v11.0, you can enable an opinionated strong password policy feature. This configuration option validates all password updates to comply with our strong password policy.

To learn more about the strong password policy in Grafana, refer to the [documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/grafana/#strong-password-policy).

### Anonymous users are billed in Grafana Enterprise

<!-- Daniel Ken Lee -->

_Generally available in Grafana Enterprise_

We are announcing a license change to the anonymous access feature in Grafana 11. As you may already be aware, anonymous access allows users access to Grafana without login credentials. Anonymous access was an early feature of Grafana to share dashboards; however, we recently introduced [Public Dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/share-dashboards-panels/shared-dashboards) which allows you to share dashboards in a more secure manner. We also noticed that anonymous access inadvertently resulted in user licensing issues. After careful consideration, we have decided to charge for the continued use of anonymous access starting in Grafana 11.

**Affected Grafana versions**

[Anonymous authentication](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/grafana/#anonymous-authentication) is disabled by default in Grafana Cloud. This licensing change only affects Grafana Enterprise (self-managed) edition. Anonymous users will be charged as active users in Grafana Enterprise.

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/grafana/#anonymous-devices)

# What's new in Grafana v11.1

<!-- vale GoogleWe = NO -->
<!-- vale We = NO -->

# What’s new in Grafana v11.1

Welcome to Grafana 11.1! This release contains some minor improvements following on the big announcements in Grafana 11.0. We've adding cell text wrapping to table visualizations, a much-requested feature. We've also redesigned the Alerting settings page and made it possible to manage access to Silences using role-based access control (RBAC). A number of accessibility improvements are included in this release, making Grafana easier to use for everyone. And lastly, the XY chart is now generally available.

{{< youtube id="gjQX9-7Hsfs" >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v11.1, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v11/upgrade-v11.1/).

## Dashboards and visualizations

### Visualization ease of use improvements

<!-- Drew Slobodnjak, Kyle Cunningham -->

We’ve made a number of small improvements to the data visualization experience in Grafana.

#### Table cell text wrapping

_Available in public preview in all editions of Grafana_

A longstanding feature request for the table visualization is to allow wrapping of text within cells. In Grafana 11.1 we've added this capability! By default, the column with the longest text is selected for wrapping. You can also configure wrapping manually using field overrides.

Learn more about the table panel in our [table panel documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/table/).

{{< youtube id="wBLxYp2BLAU" >}}

{{< admonition type="note" >}}
Text wrapping is in [public preview](https://grafana.com/docs/release-life-cycle/#public-preview), however, it’s available to use by default. We’d love hear from you about how this new feature is working. To provide feedback, you can open an issue in the [Grafana GitHub repository](https://github.com/grafana/grafana).
{{< /admonition >}}

#### Stat visualization percent change color mode options

_Generally available in all editions of Grafana_

Sometimes less is more! The **Standard** behavior for percent change in the [stat visualization](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/stat/#show-percent-change) is to color positive percent changes green and negative percent changes red. Depending on your use case however, you may want to customize how percent change color is set. We've added the options to have percent change color mode **Inverted** (with positive red and negative green) or match the **Same as value**.

{{< youtube id="LrkR6_hcqPc" >}}

### XY chart is GA

<!-- Leon Sorokin -->

_Generally available in all editions of Grafana_

We’re promoting XY chart out of public preview and into general availability.

XY charts provide a way to visualize arbitrary x and y values in a graph so that you can easily show the relationship between two variables. XY charts are typically used to create scatter plots. You can also use them to create bubble charts where field values determine the size of each bubble:

![An xy chart showing height weight distribution](/media/docs/grafana/panels-visualizations/screenshot-xy-charts-v11.0.png)

Over the past several months we’ve introduced multiple enhancements to the visualizations like auto mode, which now handles most scenarios that previously required manual configuration. Additionally, we've added better control over point styling and further improved performance. We’re excited to include XY chart as a first class citizen in the core Grafana visualization library. To learn more about the panel, refer to [the documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/xy-chart/).

To use the new XY chart visualization, you must first enable the `autoMigrateXYChartPanel` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/).

{{< youtube id="3d3Lx-Jz_kM" >}}

## Alerting

### Re-designed settings page for Alerting

<!-- Gilles -->

_Generally available in all editions of Grafana_

The new settings page provides you with a holistic view of where Grafana-managed alert instances are forwarded.

- Manage which Alertmanagers receive alert instances from Grafana-managed rules without navigating and editing data sources.
- Manage version snapshots for the built-in Alertmanager, which allows administrators to roll back unintentional changes or mistakes in the Alertmanager configuration.
- There is also a visual diff that compares the historical snapshot with the latest configuration to see which changes were made.

{{< figure src="/media/docs/alerting/alert-settings.png" alt="Alert settings page" >}}

Learn more in the [Alertmanager configuration documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alertmanager/).

### Alerting template selector

<!-- Sonia Aguilar -->

_Generally available in all editions of Grafana_

Enables you to easily choose which templates you want to use in your alert notification messages by adding a template selector in the Contact Points form.

Select from existing templates or enter a custom one for your specific needs. You can switch between the two tabs to access the list of available templates and copy them across to the customized version.

Learn more in the [documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/).

### Add OAuth2 to HTTP settings for vanilla Alertmanager / Mimir

<!-- Gilles -->

_Generally available in all editions of Grafana_

Configure OAuth2 authentication for any Alertmanager or Mimir receiver (called Contact Points in Grafana) through the user interface. Learn more about alerting contact points in [supported contact point integrations](https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points/#list-of-supported-integrations) and find the reference for Oauth2 fields in the [Prometheus Alertmanager docs for Oauth2](https://prometheus.io/docs/alerting/latest/configuration/#oauth2).

Note that OAuth2 is not yet implemented for the Grafana built-in Alertmanager.

### Improved paused alert visibility

<!-- Tom Ratcliffe -->

_Generally available in all editions of Grafana_

Pause and resume alert rule evaluation directly from the Alert rules list and details view. This helps Improve visibility of when alert rules have been paused by displaying “Paused” as the alert rule state.

Learn more in the [alert rules documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/manage-notifications/view-alert-rules/).

### Removes requirement of datasources:query permission for reading rules

<!-- William Wernert -->

_Generally available in Grafana Enterprise and Cloud_

Fetching a rule group no longer requires the **datasources:query** permission for every data source used by the rules within that group. Now, the only requirements are **alert.rules:read** and **folders:read** for the folder the group is contained in.

Note: **datasources:query** is still required to preview an alert rule, regardless of alert rules and folders permissions.

Learn more in the [RBAC configuration documentation](https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/set-up/configure-rbac/).

### Rule-specific silences with permissions

<!-- Tom Ratcliffe -->

_Generally available in all editions of Grafana_

More easily create silences directly from the Alert rule list view or detail page.

These rule-specific silences are guaranteed to only apply to a single rule and permissions to read, create, update or delete are tied to a user’s permissions for that rule.

### Rule-specific silences with RBAC

<!-- Tom Ratcliffe -->

_Generally available in Grafana Enterprise and Cloud_

Manage silences through Role-Based Access Control (RBAC). In addition to the Grafana open source functionality in **Rule-specific silences with permissions**, you can choose who can create, edit, and read silences using the following permissions:

- Users with the **alert.silences:create permission**, scoped within a folder, are able to create silences for rules contained within that folder and its subfolders
- Users with the **alert.silences:read permission**, scoped within a folder, are able to read silences for rules contained within that folder and its subfolders, and general silences
- Users with the **alert.silences:write permission**, scoped within a folder, are able to expire and recreate silences for rules contained within that folder and its subfolders

Learn more about [managing access using roles](https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/set-up/configure-rbac/access-roles/).

### Support for AWS SNS integration in Grafana-managed alerts

<!-- Yuri Tseretyan -->

_Generally available in all editions of Grafana_

Send alerts notifications to AWS simple notifications service.

## Recent accessibility improvements

<!-- #grafana-frontend-platform,Tobias Skarhed -->

_Generally available in all editions of Grafana_

### GeoMap keyboard support

The GeoMap panel can now be used with a keyboard! Focus the map area, move around with the arrows keys and zoom in and out using + and -.

### Panel shortcut keyboard support

We have [panel shortcuts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/use-dashboards/#keyboard-shortcuts), which previously only worked on whichever panel you hovered over. It now also takes into account the keyboard focus.

{{< youtube id="Tc0pTXtCsKQ" >}}

### Heading improvements

The majority of screen reader users [find things on a web page using headings](https://webaim.org/projects/screenreadersurvey10/#finding). Recently, we have added missing headings and corrected heading levels in some places, most notably panel titles.

### Reduced motion support

Users who are affected by a lot of animations on a web site have the possibility to configure [reduced motion settings](https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-reduced-motion). Grafana now supports this, either by making animations simpler or removing them altogether. We have [implemented a rule](https://github.com/grafana/grafana/tree/main/packages/grafana-eslint-rules#no-unreduced-motion), which prohibits unhandled transitions and animations, ensuring that we can continuously support reduced motion.

# What's new in Grafana v11.2

<!-- vale GoogleWe = NO -->
<!-- vale We = NO -->

# What’s new in Grafana v11.2

Welcome to Grafana 11.2! We've made a number of improvements in this release, including a Grafana Cloud Migration Assistant in public preview, several new transformations, and a centralized page for viewing your alert history. We've also released several new data sources to help you visualize data from Zendesk, Catchpoint, and Yugabyte. Read on to learn more about these and all the new features in v11.2.

{{< youtube id="s6IYpILVDSM" >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v11.2, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v11/upgrade-v11.2/).

## Grafana Cloud Migration Assistant is in public preview

<!-- #wg-everyone-to-cloud -->

_Available in public preview in Grafana Open Source and Enterprise_

Migrating from OSS or Enterprise Grafana to Grafana Cloud has traditionally been complex, requiring technical knowledge of Grafana's HTTP API and time-consuming manual processes. The new Grafana Cloud Migration Assistant changes this by providing a user-friendly interface that automates the migration of your resources. No coding required, it securely handles the transfer in just a few easy steps.

This intuitive UI offers real-time updates on your migration status, making your migration journey faster, more efficient, and less error-prone. Initially, the Cloud Migration Assistant supports dashboards, folders, and core data sources, with plans to include alerting, app plugins, and panel plugins in future updates.

Ready to make the move? Explore our [migration guide](https://grafana.com/docs/grafana-cloud/account-management/migration-guide/) to learn more about the Cloud Migration Assistant today and begin your effortless transition to Grafana Cloud.

{{< youtube id="66W1UMHtX3U" >}}

## Navigation bookmarks

<!-- #grafana-frontend-platform -->

_Available in public preview in all editions of Grafana_

As Grafana keeps growing, we have had feedback that it can be hard to find the pages you are looking for in the navigation. That is why we have added a new section to the navigation called 'Bookmarks', so you can easily access all of your favourite pages at the top of the navigation.

This feature is being rolled out across Grafana Cloud now. To use Bookmarks in self-managed Grafana, turn on the `pinNavItems` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/) in Grafana v11.2 or newer.

{{< figure src="/media/docs/grafana/grafana-nav-bookmarks.png" caption="Bookmark pages in the nav bar for quick access." alt="Bookmark pages in the Grafana nav bar for quick access" >}}

## Dashboards and visualizations

### Transformation updates

<!-- Nathan Marrs; #grafana-dataviz -->

_Generally available in all editions of Grafana_

We've made a number of exciting updates to transformations!

**You can now use variables in some transformations**

Template variables are now supported for the **Limit**, **Sort by**, **Filter data by values**, **Grouping to matrix** ([a community contribution](https://github.com/grafana/grafana/pull/88551) ⭐️), **Heatmap**, and **Histogram** transformations. This enables dynamic transformation configurations based on panel data and dashboard variables.

**New transpose transformation**

We're excited to announce the new **Transpose** transformation, which allows you to pivot the data frame, converting rows into columns and columns into rows. This feature is particularly useful for data sources that don't support pivot queries, enabling more flexible and insightful data visualizations.

For more information, refer to the [documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/#transpose).

{{< figure src="/media/docs/grafana/transformations/screenshot-grafana-11-2-transpose-transformation.png" alt="Transpose transformation in action" >}}

This feature is [a community contribution](https://github.com/grafana/grafana/pull/88963) ❤️

**Group to nested tables is now generally available**

We're excited to announce that the **Group to nested tables** transformation is now generally available! Easily group your table data by specified fields and perform calculations on each group. With this transformation, you can enhance the depth and utility of your table visualizations.

See [the documentation for more information](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/#group-to-nested-tables).

{{< video-embed src="/media/docs/grafana/screen-recording-10-4-table-group-to-nested-table-transformation.mp4" caption="Group to nested tables transformation" >}}

**Format string is now generally available**

The **Format string** transformation is now generally available! Use this transformation to customize the output of a string field. From formatting your string data to upper, lower, title case, and more, this transformation provides a convenient way to standardize and tailor the presentation of string data for better visualization and analysis. See [the documentation for more information](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/#format-string).

**New cumulative and window calculations available in Add field from calculation**

The **Add field from calculation** transformation now supports both cumulative and window calculations. The cumulative function calculates on the current row and all preceding rows. You can calculate the total or the mean of your data up to and including the current row. With the window function you can calculate the mean, standard deviation, or variance on a specified set (window) of your data. The window can either be trailing or centered. With a trailing window the current row will be the last row in the window. With a centered window the window will be centered on the current row.

See [the documentation for more information](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/#add-field-from-calculation).

### Improvements in canvas visualizations

<!-- Adela Almasan, Nathan Marrs, #grafana-dataviz -->

_Generally available in all editions of Grafana_

#### Standardized tooltips

As a continuation of our efforts to standardize tooltips across visualizations, we've updated canvas visualization tooltips to be supported for all elements tied to data. Besides the element name and data, the tooltip now also displays the timestamp. This is a step forward from the previous implementation where tooltips were shown only if data links were configured.

#### Data links improvements

We've updated canvas visualizations so that you can add data links to canvas elements without using an override. The **Selected element** configuration now includes a **Data links** section where you can add data links to elements using the same steps as in other visualizations.

Data links in canvas elements can also be configured to open with a single click. To enable this functionality, select **Link** under the one **One-click** section in the **Selected element** data link options. If there are multiple data links for an element, the first link in the list has the one-click functionality.

As part of this improvement, we've also added the ability to control the order in which data links are displayed by dragging and dropping them. This improvement has been added to all visualizations.

{{< youtube id="zOsM8VqwYpw" >}}

In future releases, we'll add one-click functionality to data links in other Grafana visualizations.

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/canvas/#data-links)

### State timeline supports pagination

<!-- #grafana-dataviz -->

_Generally available in all editions of Grafana_

The state timeline visualization now supports pagination. The **Page size** option lets you paginate the state timeline visualization to limit how many series are visible at once. This is useful when you have many series. Previously, all the series in a state timeline were made to fit within the single window of the panel, which could make it hard to read.

{{< video-embed src="/media/docs/grafana/panels-visualizations/screen-recording-grafana-11-2-state-timeline-pagination-dark.mp4" >}}

With paginated results, the visualization displays a subset of all series on each page.

Pagination is especially useful if you're running a query on a dynamic data source. It's also helpful regardless of whether you have many data frames with just two fields (time + value) or few frames with many fields (time + many values).

This feature is [a community contribution](https://github.com/grafana/grafana/pull/89586) ❤️

{{< youtube id="mgkjWJvYoHk" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/state-timeline/#page-size-enable-pagination)

## Alerting

### Centralized alert history page

<!-- Ryan Kehoe, Sonia Alguilar -->

_Generally available in all editions of Grafana_

View a history of all alert events generated by your Grafana-managed alert rules from one centralized page. This helps you see patterns in your alerts over time, observe trends, make predictions, and even debug alerts that might be firing too often.

An alert event is displayed each time an alert instance changes its state over a period of time. All alert events are displayed regardless of whether silences or mute timings are set, so you’ll see a complete set of your data history even if you’re not necessarily being notified.

For Grafana Enterprise and OSS users:

To try out the new alert history page, enable the `alertingCentralAlertHistory` feature toggle and [configure Loki annotations](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alert-state-history/).

{{< youtube id="0fNtby8ieEw" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/manage-notifications/view-state-health/)

## Explore

### Logs filtering and pinning in Explore content outline

<!-- Haris Rozajac -->

_Generally available in all editions of Grafana_

Grafana Explore now allows for logs filtering and pinning in content outline.

**Filtering Logs:** All log levels are now automatically available in the content outline. You can filter by log level, currently supported for Elasticsearch and Loki data sources. To select multiple filters, hold down the command key on Mac or the control key on Windows while clicking.

**Pinning Logs:** The new pinning feature allows users to pin logs to the content outline, making them easily accessible for quick reference during investigations. To pin a log, hover over a log in the logs panel and click on the **Pin to content outline** icon in the log row menu. Clicking on a pinned log will open the log context modal, showing the log highlighted in context with other logs. From here, you can also open the log in split mode to preserve the time range in the left pane while having the time range specific to that log in the right pane.

### Forward direction search for Loki

<!-- #observability-logs -->

_Generally available in all editions of Grafana_

Explore now supports forward direction search for Loki logs searches. This allows users to seamlessly browse logs in a time range in forward chronological order (for example, tracing a specific user's actions using logs).

{{< figure src="/static/img/logs/forward_search.png" alt="Explore logs with the Direction option selected" caption-align="left" >}}

To use this feature, select **Forward** for the **Direction** option. Note that in the screenshot above, logs are rendered beginning from the starting time period of the query, not the end.

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/loki/query-editor/)

## Data sources

### Cloudwatch Metric Insights cross account observability support

<!-- #grafana-aws-datasources -->

_Generally available in all editions of Grafana_

We are excited to announce support for cross-account querying in Metric Insights query builder for AWS Cloudwatch Plugin. This enables building SQL queries to monitor across multiple accounts in the same region in AWS Cloudwatch.

This feature introduces an account drop-down for selecting one or all of your source accounts and builds a query that targets them. Furthermore, results can be grouped by account ID by selecting **Account ID** in the **Group By** drop-down.

For more complex queries that are not covered by the options in the builder you can switch to the manual Code editor and edit the query.

To set up cross-account querying for AWS Cloudwatch Plugin, see instructions [here](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/query-editor/#cross-account-observability).

### Zendesk data source for Grafana

<!-- #grafana-oss-big-tent -->

_Available in public preview in Grafana Enterprise and Grafana Cloud_

We are excited to announce the release of a new Zendesk data source for Grafana. This addition extends Grafana's capabilities, enabling seamless integration with Zendesk.

You can find out more about the data source in the [Zendesk data source documentation](https://grafana.com/docs/plugins/grafana-zendesk-datasource/latest/).

{{< figure src="/media/docs/zendesk/zendesk_query_editor.png" alt="Zendesk Query Editor" >}}

### Catchpoint Enterprise data source for Grafana

<!-- Taewoo Kim -->

_Available in public preview in Grafana Enterprise and Grafana Cloud_

**Introducing Catchpoint data source plugin.**

The Catchpoint data source plugin allows you to query and visualize `Tests`, `RUM` and `SLO` data from within Grafana.

{{< video-embed src="/media/docs/plugins/Catchpoint.mp4" >}}

### Yugabyte data source for Grafana

<!-- #grafana-oss-big-tent -->

_Available in public preview in all editions of Grafana_

We are excited to announce the release of a new data source for Grafana: **Yugabyte**. This addition extends Grafana's capabilities, enabling seamless integration with YugabyteDB.

You can find out more about the data source in the [Yugabyte data source documentation](https://grafana.com/docs/plugins/grafana-yugabyte-datasource/latest/).

The datasource has some known limitations: ad-hoc filters and TLS/network customization are not yet supported. Improvements and additional supported features are planned for future updates.

{{< figure src="/media/docs/yugabyte/yugabyte_explore_builder.png" alt="Yugabyte query editor" >}}

## Authentication and authorization

### Map org-specific user roles from your OAuth provider

<!-- #identity-access, @Misi -->

_Generally available in Grafana Open Source and Enterprise_

Assign users to particular organizations with a specific role in Grafana, depending on an attribute value obtained from your identity provider.

This is a longstanding feature request from the community. We collaborated with our community to implement the request and have added this capability in Grafana 11.2.0.

For Generic OAuth and Okta, you can configure the claim (using the `org_attribute_path` setting) that contains the organizations which the user belongs to. Other OAuth providers use the same attribute for organization mapping that is used for group mapping: Entra ID (previously Azure AD), GitLab and Google use the current user’s Groups, and GitHub uses the user’s Teams.

To configure organization mapping for your instance, please check the documentation for the OAuth provider you are using in the [Grafana documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/). You can find an example of how to configure organization mapping on each OAuth provider page under the **Org roles mapping example** section.

### Better SAML integration for Azure AD

_Generally available in all editions of Grafana_

<!-- Lino Urdiales -->

When setting up Grafana with Azure AD using the SAML protocol, the Azure AD Graph API sometimes returns a follow-up Graph API call rather than the information itself. This is the case for users who belong to more than 150 groups when using SAML.

With Grafana 11.2, we offer a mechanism for setting up an application as a Service Account in Azure AD and retrieving information from Graph API.

Please refer to our [documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/saml/#configure-a-graph-api-application-in-azure-ad) on how to set up an Azure AD registered application for this setup.

### API support for LDAP configuration

<!-- #proj-grafana-sso-config -->

_Available in public preview in all editions of Grafana_

[The SSO settings API](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/developers/http_api/sso-settings/) has been updated to include support for LDAP settings. This feature is experimental behind the feature flag `ssoSettingsLDAP`.

You will soon be able to configure LDAP from the UI and Terraform.

### Reduce number of required fields from the SAML form

<!-- #proj-grafana-sso-config -->

_Generally available in Grafana Enterprise and Grafana Cloud Pro and Advanced_

The private key and certificate fields are no longer mandatory in the SAML form. To configure SAML without providing a private key and a certificate you have to opt out from using signed requests.

{{< figure src="/media/docs/grafana/screenshot-grafana-11-2-saml-sign-requests.png" alt="Sign requests in SAML config form" >}}

### Generate SAML certificate and private key

<!-- #proj-grafana-sso-config -->

_Generally available in Grafana Enterprise and Grafana Cloud Pro_

You can generate a new certificate and private key for SAML directly from the UI form. Click on the **Generate key and certificate** button from the **Sign requests** tab in the SAML form and then fill in the information you want to be embedded in your generated certificate.

{{< video-embed src="/media/docs/grafana/screen-recording-11-2-generate-saml-certificate.mp4" >}}

### OpenID Connect Discovery URL for Generic OAuth

<!-- #proj-grafana-sso-config -->

_Generally available in all editions of Grafana_

The OpenID Connect Discovery URL is available in the Generic OAuth form. The info extracted from this URL will be used to populate the Auth URL, Token URL and API URL fields.

{{< video-embed src="/media/docs/grafana/screen-recording-11-2-openid-discovery-url.mp4" >}}

# What's new in Grafana v11.3

<!-- vale GoogleWe = NO -->
<!-- vale We = NO -->

# What’s new in Grafana v11.3

Welcome to Grafana 11.3! Scenes-powered dashboards are now generally available and the Explore Logs plugin is now installed by default. The dashboard experience has also improved in other ways including the ability to trigger API calls from any canvas element with the new **Actions** option and an update to transformations so you can apply calculations to dynamic fields. We've also simplified the alert setup experience, added customizable announcement banners that admins can send to all users, and improved some default permissions.

{{< youtube id="a0rdRbYPCls" >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v11.3, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v11/upgrade-v11.3/).

## Scenes-powered Dashboards are generally available

<!-- #grafana-dashboards -->

_Generally available in all editions of Grafana_

For the past few months we've been working on a major update of our **Dashboards** architecture and migrated it to the Scenes library. This migration provides us with more stable, dynamic, and flexible dashboards as well as setting the foundation for what we envision the future of Grafana dashboards will be. Here are four of the improvements that are being introduced as part of this work:

**View mode and Edit mode**

It can be difficult to efficiently navigate through the visually cluttered options during the dashboard editing process. When in View mode, the dashboard screen is less clutteed. In Edit mode, options like adding a panel and changing settings are easier to access.

**Template variables and the time range picker remain visible as your scroll**

The time picker is now the dashboard canvas rather than the toolbar, and now, together with template variables, it will stick to the top as you scroll through your dashboard. This is a highly [requested feature](https://github.com/grafana/grafana/issues/11166) that we're very happy to be able to roll out!

**Timezone parameter in Grafana URL**

We've added a new time zone URL parameter, `tz`. This allows sharing dashboards with a selected time zone, ensuring that the receiver views it in the intended time zone regardless of their local settings.

**Kiosk mode displays dashboard controls**

When playing a playlist or displaying a dashboard in full screen, you can see controls by default. These controls include the time range picker, refresh button, variables, annotations, and links.

If you prefer to hide these controls during playlist playback, new configuration options are available when starting a playlist. You can choose which controls to display while the playlist is running.

For configuring controls outside of playlist playback, you can use the following URL parameters:

- `_dash.hideTimePicker`: Hides the time and refresh picker
- `_dash.hideVariables`: Hides variables and annotations controls
- `_dash.hideLinks`: Hides dashboard links

### Known limitations

- The [variable usage check](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/variables/inspect-variable/) is not yet available.
- Editing a panel:
  - The **Library panels** tab is not available anymore. You can replace a library panel from the panel menu.
  - The **Overrides** tab is not in panel options (coming in Grafana v11.3.0). Overrides are shown at the bottom of the option list.
  - The drop-down menu to collapse the visualization picker is missing (coming in Grafana v11.3.0).

- The Share button is not visible when edit mode is enabled (coming in Grafana v11.3.0).

If you want to learn more, in detail, about all the improvements we've made, don't miss our blog post.

{{< youtube id="kcKwBhvrsHc" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/)

## Additional dashboard and visualization improvements

### Auto-formatted table cell values in Cell Inspect

<!-- Kyle Cunningham, Isabel Matwawana -->

_Generally available in all editions of Grafana_

We've improved the inspect value experience in table visualizations with the addition of tabs in the **Inspect value** drawer: **Plain text** and **Code editor**.

When the **Cell inspect value** switch is toggled on, clicking the inspect icon in a cell opens the drawer. Grafana attempts to automatically detect the type of data in the cell and opens the drawer with the associated tab showing. However, you can switch back and forth between tabs.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-cell-inspect-11.3.png" max-width="650px" alt="Inspect value drawer opened to Plain text tab" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/table/#cell-value-inspect)

### Canvas actions

<!-- Adela Almasan, #grafana-dataviz -->

_Experimental in all editions of Grafana_

We've updated canvas visualizations so that now you can add actions to canvas elements, like opening an issue in GitHub or calling any API. The **Selected element** configuration now includes a **Data links and actions** section where you can add actions to elements. Each action can be configured to call an API endpoint.

Actions can also be configured to be triggered with a single click. To enable this functionality, select **Action** under the one **One-click** section in the **Selected element** data links and actions option. If there are multiple actions for an element, the first action in the list has the one-click functionality.

Also, we've also added the ability to control the order in which actions are displayed in the tooltip by dragging and dropping them.

{{< video-embed src="/media/docs/grafana/panels-visualizations/canvas-actions-11.3.mp4" >}}

To try out this feature, enable the `vizActions` feature toggle.

### Legend support in bar gauge visualizations

<!-- Adela Almasan, #grafana-dataviz -->

_Generally available in all editions of Grafana_

We've added legend support to bar gauge visualizations. This is part an effort to standardize legends and make them available across more panels. You can customize legends by navigating to the **Legend** section in panel options. By default, the legend is disabled.

You can also hide names in each bar gauge, since they are redundant to the legend. To do that, in the **Name placement** option, choose **Hidden**.

{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-bargauge-legend1-11.3.png" alt="Bar gauge legend" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/bar-gauge/#legend-options)

### Apply the same binary transformation to all the number fields in a given table at once

<!-- Drew Slobodnjak; #grafana-dataviz -->

_Generally available in all editions of Grafana_

When you set up a binary operation using the **Add field from calculation** transformation (for example, dividing each column by a certain value), there's a new **All number fields** option available to select. Use this to apply a mathematical operator to multiple number fields simultaneously. This feature is particularly useful when you're scaling or offsetting data containing multiple, dynamically-named fields, allowing the transformation to be applied when dealing with unknown field names.

{{< figure src="/media/docs/grafana/transformations/screenshot-grafana-11-3-all-number-fields-binary-operation.png" max-width="650px" alt="Binary operation calculation being applied to all number fields" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/#add-field-from-calculation)

### Actions for visualizations

<!-- Adela Almasan, #grafana-dataviz -->

_Experimental in all editions of Grafana_

We've updated several visualizations so that now you can add actions to them. The panel configuration now includes a **Data links and actions** section where you can add actions that can each be configured to call an API endpoint.

Also, we've also added the ability to control the order in which actions are displayed in the tooltip by dragging and dropping them.

{{< video-embed src="/media/docs/grafana/panels-visualizations/visualizations-actions-11.3.mp4" >}}

This functionality has been added for the following visualizations:

- Bar chart
- Candlestick
- Heatmap
- State timeline
- Status history
- Time series
- Trend
- XY chart

To try out this feature, enable the `vizActions` feature toggle.

## Explore Logs

### The Explore Logs plugin is installed by default

<!-- #proj-explorelogs-dev -->

_Generally available in all editions of Grafana_

With **Explore Logs** you can visualize and explore your logs to troubleshoot without having to write queries. It is easy to find spikes in your log volume, filter your logs and pinpoint problematic log lines.

While **Explore Logs** is [GA in cloud](https://grafana.com/blog/2024/09/24/queryless-metrics-logs-traces-profiles/#explore-logs) and installed there by default already, with Grafana v11.3.0 it will be automatically installed on your Open Source or Enterprise instance as well. This will let you use Explore Logs alongside **Explore Metrics** without needing to install it manually.

This is configured by the [`preinstall` configuration parameter](https://github.com/grafana/grafana/blob/9ece88d5852dceb90f83271e66902eece24f908f/conf/defaults.ini#L1748) in your Grafana configuration. For more information about Explore logs, refer to [the documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/simplified-exploration/logs/).

## Correlations

### Add correlations to external URLs in Explore

<!-- #grafana-correlations or @kdurivage -->

_Generally available in all editions of Grafana_

Correlations is a feature that allows Grafana users to set up links between their data sources. Previously, the link generated would only be from one query to another—meaning results from a query could only generate links to open a second Explore pane with other query results.

With this feature, users can now link to third party web-based software based on their search results. The format follows the standard Grafana format for using variables.

{{< youtube id="KuVlzjaVc0Y" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/correlations/)

## Alerting

### Simplified query section for alert rule creation

<!-- Sonia Aguilar -->

_Generally available in all editions of Grafana_

Use a simplified version of the query and alert conditions step to create Grafana-managed alert rules. The default options streamline rule creation with a cleaner header and a single query and condition. For more complex rules, switch to advanced options to add multiple queries and expressions.

This feature is rolling out to Grafana Cloud over the next couple of weeks.

Grafana Enterprise and OSS:

To use this feature, enable the `alertingQueryAndExpressionsStepMode` feature toggle.

{{< figure src="/media/docs/alerting/screenshot-simple-query-form-v11.3.png" max-width="800px"alt="Image shows the alert creation form in simple mode" >}}

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/)

### Role-based access control for notifications in Alerting

<!-- #alerting, #alerting-xl-rbac -->

_Available in public preview in Grafana Enterprise and Grafana Cloud_

Manage contact points, mute timings, and notification templates through Role-Based Access Control (RBAC). Choose who can create, edit, and read contact points, mute timings, and notification templates using permissions or fixed roles.

For mute timings and notification templates, you can grant all users all permissions or no permissions. For contact points, you can extend or limit permissions to individual contact points.

### Recording rules for Grafana-managed alerts

<!-- Alex Weaver -->

_Generally available in all editions of Grafana_

Create recording rules for Grafana-managed alert rules to calculate frequently needed expressions or computationally expensive expressions in advance and save the result as a new set of time series. Querying this new time series is faster, especially for dashboards since they query the same expression every time the dashboards refresh. Previously, this was only available for data-source managed alert rules.

In Grafana OSS and Enterprise, you can create both Grafana-managed and data source-managed recording rules if you enable the `grafanaManagedRecordingRules` feature flag.

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-mimir-loki-managed-recording-rule/)

## Data sources

### GitHub App authentication for the GitHub data source

<!-- #grafana-oss-big-tent -->

_Generally available in all editions of Grafana_

Authenticate the GitHub data source using a GitHub App, an alternative to personal access tokens (PATs). GitHub App authentication offers enhanced security by granting more granular permissions, reducing the risk of over-permissioning.

For more information, refer to the [GitHub data source documentation](https://grafana.com/docs/plugins/grafana-github-datasource/latest/setup/token/#using-github-app-authentication) for detailed instructions on setting up GitHub App authentication.

[Documentation](https://grafana.com/docs/plugins/grafana-github-datasource/latest/)

## Announcement banner

<!-- #grafana-frontend-platform -->

_Available in public preview in Grafana Enterprise and Grafana Cloud_

Grafana admins struggle to effectively communicate important updates and maintenance information to their users through traditional channels like email and Slack. Now you can display customizable banners within the Grafana interface to ensure critical information is visible and timely. This ensures that all users are immediately informed of important updates, maintenance schedules, compliance info, or other crucial messages, reducing the likelihood of missed communications and enhancing overall user awareness and engagement.

By default, only organization administrators can create announcement banners. You can customize who can create announcement banners with [Role-based access control](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/).

To use the Announcement banner in self-managed Grafana, turn on the `notificationBanner` [feature toggle](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/) in Grafana v11.3 or newer and navigate in your Grafana instance to Admin > General > Announcement Banner

{{< figure src="/media/docs/grafana/grafana-announcement-banner.png" max-width="650px" caption="Announcement banner configuration page." alt="Announcement banner configuration page" >}}

## Improved subfolder creation flow

<!-- #identity-access -->

_Generally available in all editions of Grafana_

You can now create subfolders within folders where you have Edit or Admin rights without needing any additional permissions. This enables users and teams to fully manage their folder and dashboard hierarchy, and allows you to keep your instance secure by granting users the minimum necessary set of permissions.

[Documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/manage-dashboards/#create-a-dashboard-folder)

## Plugins

### Plugin details page redesign

<!-- #grafana-plugins-platform -->

_Available in public preview in all editions of Grafana_

To help make it easier for administrators to assess and interact with Grafana plugins, we are reworking the plugin details page to highlight important metadata, such as when the plugin was last updated.

We intend to further extend this new layout with consistent links for all plugins, to complement the custom links which can currently be optionally configured. This improved consistency will enable simpler interaction with a plugin's developer - whether that is Grafana Labs, our commercial partners, or our community. These links will include actions such as raising feature requests or bug reports, as well as allowing our community developers to indicate available support and sponsorship options for those that depend on their work.

## Authentication and authorization

### Configure LDAP through the UI

<!-- #proj-grafana-sso-config -->

_Available in public preview in all editions of Grafana_

{{< figure src="/media/docs/grafana/2024-10-04-sso_ldap_2.png" max-width="700px" alt="Advanced LDAP settings" >}}

Use the new UI to set up your LDAP server as an Identity Provider, to smooth out the setup process and get quick confirmation that it works properly.

The new user interface makes it much clearer what each option does, and setting up the various configurations is now more transparent. Also, you no longer need to restart the Grafana instance for the new settings to take effect.

To use LDAP in the UI, enable the feature toggle `ssoSettingsLDAP`.

### Developers: Support RBAC in Plugins

<!-- Gabriel Mabille -->

_Generally available in all editions of Grafana_

[Documentation](https://grafana.com/developers/plugin-tools/reference-plugin-json#roles)

We're excited to announce that plugins can now leverage [Grafana's role based access control](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/#about-rbac) to define their own roles and permissions in order to control access to their routes.

To define roles and their default assignments, plugin developers need to add a `roles` section to their `plugin.json` file. Grafana will automatically register these roles and assign them to the corresponding basic roles: `Viewer`, `Editor`, `Admin`, and `Grafana Admin`.

Following is an example of defining two RBAC plugin roles and assigning them to Admins and Viewers (and thus Editors and Admins) by default:

```json
"roles": [
  {
    "role": {
      "name": "Patents Reader",
      "description": "Read patents",
      "permissions": [
        {"action": "grafana-appwithrbac-app.patents:read"}
      ]
    },
    "grants": ["Admin"]
  },
  {
    "role": {
      "name": "Research papers Reader",
      "description": "Read research papers",
      "permissions": [
        {"action": "grafana-appwithrbac-app.papers:read"}
      ]
    },
    "grants": ["Viewer"]
  }
]
```

Protecting `includes` and `routes` is also straight forward, and can be done through the new `action` and `reqAction` field of these sections of the `plugin.json` file.

**Plugin example**

If you’d like to test this and explore RBAC for plugins further, refer to this [plugin example](https://github.com/grafana/grafana-plugin-examples/blob/main/examples/app-with-rbac/README.md) for guidance.

**Known limitation**

Plugins permissions are currently restricted to actions without scopes.

# What's new in Grafana v11.4

<!-- vale GoogleWe = NO -->
<!-- vale We = NO -->

# What’s new in Grafana v11.4

Welcome to Grafana 11.4! This is a special release with one new feature, created in partnership with AWS. We’ve made improvements to Cloudwatch so you can query your logs using two additional query languages: Opensearch PPL and Opensearch SQL. We’ve also updated the querying experience by providing a sample of commonly-used queries, syntax highlighting in the query editor, and auto-completion suggestions for complex queries. Read on to learn more about this new feature.

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v11.4, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v11/upgrade-v11.4/).

## Query Cloudwatch Logs Insights with PPL and SQL

<!-- Ida Štambuk, Zoe Chou -->

_Generally available in all editions of Grafana_

The AWS CloudWatch data source plugin now offers two new query languages for searching through logs: OpenSearch PPL and OpenSearch SQL. You now have increased flexibility to choose a more familiar query language and to take advantage of their unique features (like the SQL JOIN command) when querying AWS CloudWatch Logs Insights. In addition to the already supported **Logs Insights QL** option, you can find the added query language options in the new **Query language** drop-down list.

![CloudWatch logs explore query languages drop-down list opened](/media/docs/grafana/data-sources/screenshot-awscw-query-languages-v11.4.png)

We’ve also updated the query editor to support syntax highlighting and to provide suggestions while you type, which simplifies writing complex queries. When you select log groups, the suggestions also include [discovered fields](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html).

Not sure where to start? We’ve also updated our **CloudWatch Logs cheat sheet** with the most commonly used queries so you can paste and customize to fit exactly what you need. Clicking on any of the example queries prefills the query input field.

![CloudWatch Logs cheat sheet with help icon highlighted](/media/docs/grafana/data-sources/screenshot-cwl-cheat-sheet-v11.4.png)

For the complete list of commands supported for OpenSearch PPL and SQL, refer to the [Cloudwatch Logs Insights documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_Languages.html).

{{< youtube id="g1BfokyzF4Q" >}}

# What’s new in Grafana v11.5

Welcome to Grafana 11.5!
Read on to learn about new sharing, reporting, and export options, cross-cluster search for Elasticsearch, PDC support for several new data sources, and more.
The Grafana Cloud Migration Assistant is in public preview and now supports all plugins and Grafana Alerts, in addition to dashboards, folders, and data sources.
We've also made it more secure to run third-party apps and data sources, and improved user session handling for OAuth 2.0 and SAML.

{{< youtube id="RGiktzfhRd0" >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v11.5, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v11/upgrade-v11.5/).

{{< docs/whats-new  >}}

# What’s new in Grafana v11.6

Welcome to Grafana 11.6! This minor release includes a number of dashboarding features that are now generally available including one-click data links and actions, Cron syntax support for annotations, and WebGL-powered geomaps for better performance. We've also fully migrated from API keys to service accounts in Grafana for better security. Read on to learn more about version history for Grafana Managed Alerts, label-based access control (LBAC) for Mimir metrics, and more in Grafana v11.6.

{{< youtube id=iF7yxO4nUXQ >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v11.6, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v11/upgrade-v11.6/).

{{< docs/whats-new  >}}

# What’s new in Grafana v12.0

Welcome to Grafana 12.0! We have a _lot_ to share. This release marks general availability for Grafana Drilldown (previously Explore Metrics, Logs, and Traces), Grafana-managed alerts and recording rules, Cloud migration, and plugin management tooling. You can also try new [preview and experimental](https://grafana.com/docs/release-life-cycle/) tools: Sync your dashboards directly to a GitHub repository with Git Sync, and try our new Terraform provider and CLI. Add tabs, new layouts and conditional logic to your dashboards, and load tables and geomaps far faster. Join and transform data limitlessly from multiple sources with SQL Expressions. In Grafana Cloud and Enterprise, sync your users and teams instantly from your SAML identity provider using SCIM (the System for Cross-Domain Identity Management). Lastly, don't forget to try on one of several new color themes for the user interface.

Read on to learn about these and more improvements to Grafana!

{{< youtube id=mHSzaVYBh38 >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v12.0, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v12/upgrade-v12.0/).

## Breaking changes in Grafana v12.0

For Grafana v12.0, we've also provided a list of [breaking changes](https://grafana.com/docs/grafana/latest/whatsnew/whats-new-in-v12/whats-new-in-v12-0/#breaking-changes) to help you upgrade with greater confidence. For our purposes, a breaking change is any change that requires users or operators to do something. This includes:

- Changes in one part of the system that could cause other components to fail
- Deprecations or removal of a feature
- Changes to an API that could break automation
- Changes that affect some plugins or functions of Grafana
- Migrations that can’t be rolled back

For each change, the provided information:

- Helps you determine if you’re affected
- Describes the change or relevant background information
- Guides you in how to mitigate for the change or migrate
- Provides more learning resources

{{< docs/whats-new  >}}

# What’s new in Grafana v12.1

It’s 12.1 time! We’re really excited about GA for a new alert rule page that makes it easier to find what you need quickly, as well as the regression analysis transformation, helping you predict future data values or estimate missing data points that might not be exactly represented in the original dataset. Also, visualization actions now support custom variables. When triggered, actions prompt you to input whatever custom variable was defined, allowing you to tailor requests in real time without modifying dashboard configuration. This is super helpful for when you’re triggering alerts, filtering API calls, or sending user-defined parameters to external systems. And you can try out Grafana Advisor, which automatically detects plugin, data source, and SSO issues, keeping your Grafana instance healthy and secure.

Speaking of security, Grafana now supports Entra Workload Identity, simplifying OAuth and increasing security for instances using Microsoft Azure. Thanks to community contributor [mehighlow](https://github.com/mehighlow) for this feature!

We have one more community contributor to thank for this release. [Chris Hodges](https://github.com/chodges15) delivered server-configurable quick time ranges for dashboards. Now you can define custom time range presets for the time picker on dashboards, perfect for teams that routinely analyze specific, context-driven time windows.

Keep reading to learn about what else we have in store for 12.1.

{{< youtube id=GOflMq8nSwg >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v12.1, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v12/upgrade-v12.1/).

{{< docs/whats-new  >}}

# What’s new in Grafana v12.2

Welcome to Grafana 12.2! This release focuses on making it easier to gain insights from your data.

We're excited to announce several features are now GA. Enhanced ad hoc filtering transforms your dashboards into true command centers, allowing you to slice and dice datasets on the fly. The redesigned table visualization offers improved performance and visual aids for quick pattern and anomaly identification, helping you make faster decisions. The Logs Drilldown JSON viewer makes intimidating log structures organized and explorable. Metrics Drilldown now integrates with alert creation in Grafana, so you can explore Prometheus data with intuitive point-and-click interactions, find the right visualization, and easily use its query in your alert rule.

We're also collecting feedback on some new public preview features. AI-powered SQL expressions eliminate the barrier between questions and answers by generating SQL queries from natural language and providing instant explanations for existing queries. Our enhanced Canvas Pan and Zoom experience lets you design complex dashboards exactly as you envision them.

Keep reading to learn more about everything 12.2 has in store.

{{< youtube id=-7A_tePidEM >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v12.2, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v12/upgrade-v12.2/).

{{< docs/whats-new  >}}

# What’s new in Grafana v12.3

Welcome to Grafana 12.3!

This release reimagines how you explore logs, delivers more ways to connect your data, and enables more intuitive sharing, so you can spend less time navigating tools and more time turning signals into action. It starts with a completely redesigned logs panel that brings faster pattern recognition, clearer context, and a smoother exploration experience. We’ve also expanded what you can do with new data sources and query improvements, from SolarWinds Enterprise to advanced Honeycomb and OpenSearch capabilities.

Sharing insights is easier too, with streamlined dashboard image exports and consolidated panel time controls that make comparisons and overrides more intuitive. And to help you learn as you go, a new interactive learning experience now brings contextual guidance directly into Grafana, giving you tips, tutorials, and documentation right when you need them. All of this comes together to make open observability easier, more accessible, and more powerful for every team.

{{< youtube id=VidGeu4PRZw >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v12.3, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v12/upgrade-v12.3/).

{{< docs/whats-new  >}}

# What’s new in Grafana v12.4

Welcome to Grafana 12.4!

This release doubles down on dashboard automation and Git-powered workflows while delivering a range of polish and usability improvements.

The headline is improved dashboard productivity: Dynamic dashboards and template-driven workflows make it far easier to create, reuse, and operate dashboards at scale. Create dashboards from templates, map one variable to multiple values, apply regular expression transforms to variable values or display text, and benefit from smarter visualization suggestions and an updated gauge panel that help surface the right view for your data. At the same time, Git Sync is now available in public preview, providing a smooth Git-backed workflow to keep dashboards in source control and enable safer, auditable changes to dashboard content. These two areas together are aimed at speeding up development, improving consistency, and making dashboard CI/CD practical for teams.

Beyond dashboards, this version includes improvements across the product: Logs and Logs Drilldown get better default columns and save/resume log explorations, data sources receive sensible new defaults and expanded support, SCIM provisioning simplifies user and team management, short URLs are now persistent, and a small set of breaking changes and storage migration enhancements help ensure long-term manageability.

{{< youtube id=fWpL1Upc754 >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v12.4, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v12/upgrade-v12.4/).

{{< docs/whats-new  >}}

# What’s new in Grafana v13.0

Welcome to Grafana 13.0! In this Grafana release we help solve the blinking cursor problem, helping teams to onboard and get insights from their data faster than ever. Easily take advantage of dashboards bundled with data sources or provided by the community, use templates to build faster and more consistently, save and share queries between your teams, and take the guess work out of panel choices with recommended suggestions.

Dynamic dashboards, our next generation of dashboarding, reaches general availability. The increased flexibility and adaptability makes building dashboards delightful, and helps teams consolidate and share a single source of truth. Additionally reaching general availability, Git Sync allows powerful bidirectional GitOps-helping you to manage your Grafana resources reliably at scale.

And that’s not all. There is a lot more to discover in Grafana 13, from the new Gauge visualization, new data sources, improved annotations, the list goes on. Read on to find out more and try for yourself!

{{< youtube id=OYd0ahylGGI >}}

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v13.0, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v13/upgrade-v13.0/).

## Breaking changes in Grafana v13.0

For Grafana v13.0, we've also provided a list of [breaking changes](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/whatsnew/whats-new-in-v13/whats-new-in-v13-0/#breaking-changes) to help you upgrade with greater confidence.

{{< docs/whats-new  >}}

# What's new in Grafana v13.1

Welcome to Grafana 13.1!

This release helps you build dashboards with less setup, connect more of your data securely, and get answers across your whole stack without leaving Grafana. Apply quick filters and grouping without configuring template variables, scope variables to the rows and tabs where they belong, and copy panel styling in a couple of clicks. Git Sync matures with verified commits, dashboard import, and root-level sync, bringing auditable GitOps to teams that require signed commits and branch protection. And Grafana Assistant now queries Snowflake, Jira, Dynatrace, and five more sources directly, so you can ask questions across your databases and observability data in one place.

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v13.1, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v13/upgrade-v13.1/).

{{< docs/whats-new  >}}

# What's new in Grafana v13.2

Welcome to Grafana 13.2!

This release helps you share and reuse trusted queries to get answers faster and more easily onboard team members, dig into crowded panels without leaving the dashboard, and securely connect to your data with fewer long-lived secrets to manage.

Saved queries are now generally available in Grafana Enterprise and Grafana Cloud, helping individual's knowledge become a library everyone can draw on: find them from anywhere using the command palette (cmd/ctrl+K) then quickly jump into Explore, and reliably manage them as code with Terraform.

View panel mode gains a controls sidebar in public preview, so you can adjust visualization options without edit permissions and fan a busy time series out into one graph per series or label to see which one is actually moving. Query variables get a redesigned editor, OpenSearch adds an index picker to configuration and query editing, and BigQuery and Google Cloud Monitoring can now authenticate with Google Workload Identity Federation in Grafana Cloud instead of a service account key file.

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v13.2, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v13/upgrade-v13.2/).

{{< docs/whats-new  >}}

# What's new in Grafana v13.3

Welcome to Grafana 13.3! This release brings our biggest visual refresh since Grafana 7, introduces Notebooks as a new artifact type to help you share investigations with your teams, lets you define cross-dashboard variables and brings a whole wealth of other new features and fixes.

For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v13.3, check out our [Upgrade Guide](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/upgrade-guide/upgrade-v13/upgrade-v13.3/).

{{< docs/whats-new  >}}

# What's new in Grafana v9.2

Welcome to Grafana v9.2, a hefty minor release with a swath of improvements that help you create and share dashboards and alerts.
Read on to learn about progress on public dashboards, our new panel help menu, custom branding in Grafana Enterprise, and improvements to access control.
If you'd prefer to dig into the details, check out the complete [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## Panel help menu

_Available in beta in Grafana Open Source._

Dashboard panel issues can occur for all kinds of reasons, from problems processing data to issues with rendering or configuration.
Shorten your communication time when reporting issues and requesting help from Grafana Labs by retrieving a panel's query response data and panel settings.
This will help the support team reproduce, diagnose, and fix the issue as quickly as possible.
See our [documentation](/docs/grafana/latest/troubleshooting/send-panel-to-grafana-support/) for more info.

For details, see [GitHub issue #55005](https://github.com/grafana/grafana/issues/55005) and ["Send a panel to Grafana Labs support"](/docs/grafana/latest/troubleshooting/send-panel-to-grafana-support/) in the documentation.

{{< figure src="/static/img/docs/panels/panel-help-9-2.gif" max-width="750px" caption="Retrieving a panel's query response data and panel settings" >}}

## Canvas panel

_Available in beta in Grafana Open Source._

Introducing the Canvas panel, a new panel that combines the power of Grafana with the flexibility of custom elements.
Canvas visualizations are extensible form-built panels that allow you to explicitly place elements within static and dynamic layouts. This empowers you to design custom visualizations and overlay data in ways that aren't possible with standard Grafana panels, all within Grafana's UI. If you've used popular UI and web design tools, then designing Canvas panels will feel very familiar.

For example, you can place image layers and then overlay text that's updated by Grafana data sources, and display icons that can change color conditionally based on data.

We've planned additional features and design elements for future releases to make Canvas panels even more powerful tools for creating custom, interactive, data-driven visualizations. To learn more about the Canvas panel, see the [documentation](../../../panels-visualizations/visualizations/canvas/).

{{< video-embed src="/static/img/docs/canvas-panel/canvas-beta-overview-9-2-0.mp4" max-width="750px" caption="Canvas panel beta overview" >}}

## Support for Google Analytics 4 properties

_Generally available in Grafana Open Source._

You can now use Google Analytics 4 (GA4) to track usage of Grafana.
To enable tracking with GA4, specify your property's measurement ID in Grafana's configuration file.
Read more [in our documentation](/docs/grafana/latest/setup-grafana/configure-grafana/#google_analytics_4_id).

Google Analytics 4 replaces Universal Analytics, which will stop processing hits in 2023 (July 1 for free accounts, October 1 for Google Analytics 360).
You can continue using Universal Analytics with Grafana and send analytics data to both types of properties.

## Alertmanager updated to be based on Prometheus Alertmanager v0.24

_Generally available in all editions._

The Alertmanager used for Grafana-managed alert rules is now based on the latest release of the Prometheus Alertmanager, v0.24.
We continue to unify the different Alertmanagers that you can use with Grafana Alertmanager and will provide updates on this topic in a future release of Grafana.

For details, see [GitHub pull request #53555](https://github.com/grafana/grafana/pull/53555).

## Grafana Alerting alert rules now return an Error state by default on execution error or timeout

_Generally available in all editions._

Error rules created with Grafana Alerting were previously switching to an Alerting state when the rule was facing an execution error or timeout.
New error rules now switch by default to the `Error` state when failing to execute or timing out.
You can change this default to either `Alerting` or `OK`.

This change does not update existing alert rules.

For details on this change, see [GitHub pull request #55345](https://github.com/grafana/grafana/pull/55345).
For more information about alerting states, see the [alerting documentation](/docs/grafana/latest/alerting/fundamentals/state-and-health/).

## Configure external alertmanagers as data sources

_Generally available in all editions._

Starting with release 9.2, the URL configuration of external alertmanagers from the Admin tab on the Alerting page is deprecated. It will be removed in a future release.

External alertmanagers should now be configured as data sources using Grafana Configuration from the main Grafana navigation menu. This enables you to manage the contact points and notification policies of external alertmanagers from within Grafana and also encrypts HTTP basic authentication credentials that were previously visible when configuring external alertmanagers by URL.

## Public dashboards

_Available in Experimental in Grafana Open Source, Enterprise, and Cloud._
_To enable public dashboards, you must enable a feature flag or request this feature from support._

[Public dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/share-dashboards-panels/shared-dashboards) launched as an experimental feature in Grafana v9.1.
We've received lots of great feedback on this much-anticipated feature and thank everyone who has helped us improve it.
The team has been hard at work polishing public dashboards, and we've closed quite a few bugs related to community reports.
If you're trying out public dashboards, let us know how it is going in the [open discussion in GitHub](https://github.com/grafana/grafana/discussions/49253).
Beyond bug fixes, we're excited to share some new features:

### Expression support

We have added the ability to use [expressions](/docs/grafana/latest/panels/query-a-data-source/use-expressions-to-manipulate-data/about-expressions/) for your public dashboards.

{{< figure src="/static/img/docs/dashboards/public-dashboards-expressions-9-2.png" max-width="750px" caption="Using expressions in a public dashboard" >}}

### Public dashboard usage insights

_Available in Experimental in Grafana Enterprise._

Usage Insights now has a "Public Dashboards" tab, which tracks how many people are viewing your public dashboard and how many queries have ran, and counts errors.

{{< figure src="/static/img/docs/dashboards/public-dashboards-usage-insights-9-2.png" max-width="750px" caption="Usage Insights for a public dashboard" >}}

### Use RBAC to allow any user to share dashboards publicly

_RBAC is available in Grafana Enterprise and Cloud._

We have introduced a new role called "Public Dashboard writer" that grants access to publish new public dashboards to additional roles and users.
By default, only admins can share dashboards publicly.

{{< figure src="/static/img/docs/dashboards/public-dashboards-writer-role-9-2.png" max-width="750px" caption="The Public Dashboard writer role for users" >}}

## Revamped UI for Google Cloud monitoring

_Generally available in Grafana Open Source and Grafana Cloud Free, Pro, and Advanced._

The Google Cloud monitoring data source UI has been brought up to date with the latest Grafana UI design.
This new interface provides a more consistent experience as you switch between different data sources.
It also groups query builder items together more logically, so it's easier to write queries.
In the case of Cloud monitoring, the query builder's groupings should more closely match the groupings in Google's Cloud console.

{{< figure src="/static/img/docs/queries/gcloud-data-source-query-grouping-9-2.png" max-width="750px" caption="Query builder groupings for Google Cloud monitoring" >}}

## App Plugins: better handling for secure fields

Plugin developers can mark certain fields as secure to encrypt those fields in Grafana's database.
This is important for sensitive fields like data source passwords or API keys.
Previously, plugin developers needed to manually track which fields were labeled secure.
We have extended our plugins platform to simplify this, which means one less thing for developers to worry about and better security for all users' data.
For details on using this functionality, see [GitHub pull request #55313](https://github.com/grafana/grafana/pull/55313) and our [plugin examples](https://github.com/grafana/grafana-plugin-examples) repository.

## Transformations: INNER JOINs

[Transformations](../../../panels-visualizations/query-transform-data/transform-data/) allow you to shape raw data from data sources, like metrics series or GitHub issues, into a format that's appropriate for the chosen visualization.
We have extended the [Join transformation](../../../panels-visualizations/query-transform-data/transform-data/#join-by-field) to support INNER JOINs in addition to OUTER JOINs. These work similarly to SQL JOINs.

{{< figure src="/static/img/docs/transformations/transform-outer-join-9-2.png" max-width="750px" caption="Query builder groupings for Google Cloud monitoring" >}}

Also, you can now click on the `x` to clear values in the select fields for the OUTER JOIN and Grouping to Matrix transformations as expected.

## Simplified UI to create template variable queries for Loki data source

_Generally available in Grafana Open Source._

We have significantly simplified and improved the way you can create template variable queries for Loki data sources in dashboards.
Use drop-downs to choose query type, label, and stream selector, without needing to worry about templating query syntax.
For more information, refer to [Loki data source documentation](/docs/grafana/latest/datasources/loki/#query-variable).

{{< figure src="/static/img/docs/queries/loki-template-variable-queries-9-2.png" max-width="750px" caption="Creating a template variable query for Loki" >}}

## Authentication and authorization

### Teams can be empty or without any Admin user

_Generally available in all editions._

You can now leave a team empty without any users, or have only Members in a team.
This helps you sync teams and users from a single sign-on provider like Active Directory or Okta, or if you use teams as collections of permissions.
Previously, teams required at least one Admin user to be in a team.

{{< figure src="/static/img/docs/manage-users/member-only-team-9-2.png" max-width="750px" caption="Creating a team without an Admin user" >}}

### Role-based access control is easier to use

_Generally available in Grafana Enterprise and Grafana Cloud._

#### Role picker when creating and editing teams

You can now choose a team's roles and permissions when creating a new team or editing an existing team, which helps you grant permissions to teams more efficiently.
Previously, you could assign roles only when looking at a list of all teams.

{{< figure src="/static/img/docs/manage-users/team-role-assignment-during-creation-9-2.png" max-width="750px" caption="Assigning roles to a team when creating it" >}}

#### RBAC: Organize custom roles into groups

If you create many custom roles in Grafana – for example, different roles for editing specific dashboards or folders, or specific roles for certain teams in your company – you can now organize them into groups for more efficient navigation, browsing, and custom role assignment.
For more information on custom roles, see the [documentation](/docs/grafana/latest/developers/http_api/access_control/#create-a-new-custom-role).

{{< figure src="/static/img/docs/manage-users/rbac-groups-9-2.png" max-width="750px" caption="Organizing custom roles into RBAC groups" >}}

#### RBAC: Terraform resource for assigning fixed and custom roles to users, teams, and service accounts

You can now assign fixed and custom roles to users, teams, and service accounts with the Grafana role_assignment resource in Terraform.
This allows you to provision user permissions in a version-able, repeatable way if you use Terraform.
Learn more in the [documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/role_assignment) for Grafana's Terraform provider.

## SAML role mapping improvements

_Generally available in Grafana Enterprise, Grafana Cloud Pro, and Advanced._

### Map a user to all organizations in Grafana

You can now use `*` as the Grafana organization in the mapping to add all users from a given SAML Organization to all existing Grafana organizations.
For more information, see ["Configure SAML authentication"](/docs/grafana/next/setup-grafana/configure-access/configure-authentication/saml/#configure-organization-mapping) in the documentation.

### Skip organization role sync

Generally available in Grafana Enterprise, Grafana Cloud Pro, and Advanced.

If you use a SAML identity provider to manage your users but prefer to assign roles and permissions in the Grafana UI or via API, you can now enable a configuration option to skip user organization and roles synchronization with your SAML provider.

Use the `skip_org_role_sync` configuration option when configuring SAML to prevent synchronization with SAML roles and make user roles editable from within Grafana.

For more information, see the [SAML configuration documentation](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/saml/).

## Assign Server Admin permissions from Oauth

You can now map OAuth groups and roles to Server Admin for the GitLab, GitHub, AzureAD, Okta, and Generic OAuth integrations.
To enable this functionality, set the `allow_assign_grafana_admin` configuration option to `true` in the desired OAuth integration section.
For more information, see the [authentication configuration documentation](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/) for each OAuth client.

## Match parameter support in Prometheus labels API

Prometheus users running Prometheus v2.24 and higher can use the [labels endpoint](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values) instead of the [series endpoint](https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers) for the [`label_values` function](../../../datasources/prometheus/#query-variable).
This decreases load times for templated high-cardinality Prometheus instances.

If you want to benefit from this endpoint you must first configure the Prometheus type and version in any Prometheus data sources' [configuration](../../../datasources/prometheus/).

## New Prometheus streaming parser

In Grafana v9.2, you can enable the `prometheusStreamingJSONParser` [feature toggle](../../../setup-grafana/configure-grafana/#feature_toggles) to use a better-performing, memory-efficient streaming JSON client for Prometheus.
We'll make this client the default in Grafana v9.3.

When Prometheus returns `NaN` values, this new client doesn't change them, neither to the value `null` nor to `0` as in recent Grafana versions.
If you use this new Prometheus streaming parser with Grafana Managed Alerts, this change in behavior might trigger alerts.
To avoid this, select the "Drop non-numeric values" option in the Reduce expression to drop `NaN` values.

# What's new in Grafana v9.3

Welcome to Grafana 9.3! Read on to learn about our navigation overhaul, support for four new languages, new panels and transformations, several often-requested auth improvements, usability improvements to Alerting, and more. For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## New navigation

Available in **beta** in all editions of Grafana

Use Grafana’s redesigned navigation to get full visibility into the health of your systems, by quickly jumping between features as part of your incident response workflow.

As Grafana has grown from a data visualization tool to an observability solution, we’ve added many new features along the way. This has resulted in pages that are visually inconsistent or hard to find. These updates to navigation give Grafana a new look and feel and make page layouts and navigation patterns more consistent.

We’ve revamped the navigation menu and grouped related tools together, making it easier to find what you need. Pages in Grafana now leverage new layouts that include breadcrumbs and a sidebar, allowing you to quickly jump between pages. We’ve also introduced a header that appears on all pages in Grafana, making dashboard search accessible from any page.

To try out Grafana’s new navigation, enable the `topnav` feature toggle. If you are a Cloud Advanced customer, open a ticket with our support team and we will enable it for you.

**Note:** The Grafana and Grafana Cloud documentation has not yet been updated to reflect changes to the navigation - these changes will roll out when the new navigation becomes generally available.

{{< figure src="/static/img/docs/navigation/navigation-9-3.png" max-width="750px" caption="New navigation for Grafana" >}}

## View dashboards in Spanish, French, German, and Simplified Chinese

Generally available in all editions of Grafana

We have added four new languages to Grafana: Spanish, French, German, and Simplified Chinese.

With millions of users across the globe, Grafana has a global footprint. In order to make it accessible to a wider audience, we have taken the first steps in localizing key workflows. You can now set Grafana’s language for the navigation, viewing dashboards, and some settings. This will cover the main activities a Viewer performs within Grafana.

Read more about configuring the [default language for your organization](../../../administration/organization-preferences/) and [updating your profile](../../../administration/user-management/user-preferences/) in our documentation.

{{< figure src="/static/img/docs/internationalization/internationalization-9-3.png" max-width="750px" caption="Grafana available in Spanish, French, German, and Simplified Chinese" >}}

## Geomap panel

Generally available in all editions of Grafana

We have added a new alpha layer type in Geomap called photo layer. This layer enables you to render a photo at each data point. To learn more about the photo layer and the geomap panel, refer to [Photos layer](../../../panels-visualizations/visualizations/geomap/#photos-layer-alpha).

{{< figure src="/static/img/docs/geomap-panel/geomap-photos-9-3-0.png" max-width="750px" caption="Geomap panel photos layer" >}}

## Canvas panel

Available in **beta** in all editions of Grafana

Canvas is a new panel that combines the power of Grafana with the flexibility of custom elements. Canvas visualizations are extensible form-built panels that allow you to explicitly place elements within static and dynamic layouts. This empowers you to design custom visualizations and overlay data in ways that aren’t possible with standard Grafana panels, all within Grafana’s UI. If you’ve used popular UI and web design tools, then designing Canvas panels will feel very familiar.

In Grafana v9.3, we have added icon value mapping support to the Canvas panel. This enables you to dynamically set which icon to display based on your data. To learn more about the Canvas panel, refer to [Canvas](../../../panels-visualizations/visualizations/canvas/).

{{< video-embed src="/static/img/docs/canvas-panel/canvas-icon-value-mapping-support-9-3-0.mp4" max-width="750px" caption="Canvas panel icon value mapping support" >}}

## Public dashboards improvements

We've made the following improvements to public dashboards.

### Manage all of your public dashboards in one place

Available in **experimental** in Grafana Open Source, Enterprise, and Cloud Advanced

You can use Public Dashboards to make a given dashboard available to anyone on the internet without needing to sign in. In Grafana v9.3, we have introduced a new screen where you can manage all of your public dashboards. From here, you can view a list of all of the public dashboards in your Grafana instance, navigate to the underlying dashboard, see if it is enabled, link out to the public version of the dashboard, or update the public dashboard's configuration. You can see a public dashboard's configuration if you have view access to the dashboard itself, and you can edit its configuration if you have the Admin or Server Admin role or the "Public Dashboard writer" role if you are using RBAC in Grafana Enterprise or Cloud Advanced.

To check out this new screen and configure your public dashboards, navigate to **Dashboards > Public Dashboards**.

### Choose to display annotations in public dashboards

Available in **experimental** in Grafana Open Source, Enterprise, and Cloud Advanced

Annotations are now supported in public dashboards, with the exception of query annotations. They are turned off by default, but can be turned on in your public dashboard settings.

Note that because Public Dashboards is an experimental feature, you need to enable it in Grafana using the `publicDashboards` [feature toggle](../../../setup-grafana/configure-grafana/#feature_toggles), or open a support ticket requesting public dashboards if you are a Cloud Advanced customer.

To learn more about public dashboards, refer to [Public dashboards](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/dashboards/share-dashboards-panels/shared-dashboards/).

## New transformation: Partition by values

Available in **experimental** in all editions of Grafana

This new transformation can help eliminate the need for multiple queries to the same datasource with different WHERE clauses when graphing multiple series.

Consider a metrics SQL table with the following data:

| Time                | Region | Value |
| ------------------- | ------ | ----- |
| 2022-10-20 12:00:00 | US     | 1520  |
| 2022-10-20 12:00:00 | EU     | 2936  |
| 2022-10-20 01:00:00 | US     | 1327  |
| 2022-10-20 01:00:00 | EU     | 912   |

Prior to v9.3, if you wanted to plot a red trendline for US and a blue one for EU in the same TimeSeries panel, you would likely have to split this into two queries:

```
   SELECT Time, Value FROM metrics WHERE Time > ‘2022-10-20’ AND Region=’US’
   SELECT Time, Value FROM metrics WHERE Time > ‘2022-10-20’ AND Region=’EU’
```

This approach also requires you to know ahead of time which regions exist in the metrics table.

With the partition by values transformer, you can issue a single query and split the results by unique (enum) values from one or more columns (fields) of your choosing. In this case, Region.

```
   SELECT Time, Region, Value FROM metrics WHERE Time > ‘2022-10-20’
```

| Time                | Region | Value |
| ------------------- | ------ | ----- |
| 2022-10-20 12:00:00 | US     | 1520  |
| 2022-10-20 01:00:00 | US     | 1327  |

| Time                | Region | Value |
| ------------------- | ------ | ----- |
| 2022-10-20 12:00:00 | EU     | 2936  |
| 2022-10-20 01:00:00 | EU     | 912   |

## Reporting: Zoom in and out to fit your data better into a PDF

Generally available in Grafana Enterprise, Cloud Pro, and Cloud Advanced.

Because dashboards appear on a screen and reports are PDFs, it can be challenging to render data just the way you want to. Sometimes the report doesn't show enough columns in a table, or the titles appear too small. Now you can adjust the scale of your report to zoom in and make each text field and panel larger or zoom out to show more data.

The zoom feature is located in the **Format Report** section of your reporting configuration. To learn more about reporting, refer to [Create and manage reports](../../../dashboards/create-reports/).

{{< figure src="/static/img/docs/enterprise/reports/report-zoom.png" max-width="750px" caption="Report zoom feature with PDF documents at three different zoom levels" >}}

## Users and access

We've made the following improvements to users and access.

### OAuth: token handling improvements

Generally available in all editions of Grafana

As part of our efforts to improve the security of Grafana, we are introducing a long-awaited feature that enhances Grafana's OAuth 2.0 compatibility. When a user logs in using an OAuth provider, Grafana verifies on each request that the user's access token has not expired. Grafana uses the refresh token provided (if any exists) when an access token expires to obtain a new access token.

Because this feature introduces a breaking change, it is behind the `accessTokenExpirationCheck` feature toggle and is disabled by default. Enabling this functionality without configuring refresh tokens for the specific OAuth provider will sign users out after their access token has expired, and they would need to sign in again every time.

Complete documentation on how to configure obtaining a refresh token can be found on the [authentication configuration page](../../../setup-grafana/configure-access/configure-authentication/), in the instructions for your Oauth identity provider.

### Resolve user conflicts in Grafana's CLI

In the older versions of Grafana, usernames were case-sensitive. This created conflicts, where a user might sign in using two different methods (like SAML and OAuth) and have two accounts created, like `elastigirl@incredibles.com` and `ElastiGirl@incredibles.com`. Users in this situation might think they have lost their preferences and permissions. If this has occurred in your Grafana instance, you can use a new Grafana CLI command to resolve user identity conflicts between users within Grafana.

> Note: If you use Grafana Cloud or you run Grafana with MySQL as your database, you will not experience any user identity conflicts and you do not need to use this tool.

```bash
# lists all the conflicting users
$ grafana-cli user-manager conflicts list

# creates a conflict patch file to edit
$ grafana-cli user-manager conflicts generate-file

# reads edited conflict patch file for validation
$ grafana-cli user-manager conflicts validate-file <filepath>

# ingests the conflict users file. Can be executed once per file and will change the state of the database.
$ grafana-cli user-manager conflicts ingest-file <filepath>
```

### LDAP: Role mapping improvements

Generally available in all editions of Grafana

If you use an LDAP directory to authenticate to Grafana but prefer to assign organizations and roles in the Grafana UI
or via API, you can now skip user organization role synchronization with your LDAP
directory.

Use the `skip_org_role_sync` [LDAP authentication configuration option](../../../setup-grafana/configure-access/configure-authentication/ldap/#disable-org-role-synchronization)
when configuring LDAP authentication to prevent the synchronization between your LDAP groups and organization roles
and make user roles editable manually.

### Azure AD OAuth2: New option to always fetch groups from the Graph API

Generally available in all editions of Grafana

If you use Azure AD OAuth2 authentication and use `SecurityEnabled` groups that you don't want Azure to embed in the
authentication token, you can configure Grafana to use Microsoft's Graph API instead.

Use the [`force_use_graph_api` configuration option](../../../setup-grafana/configure-access/configure-authentication/azuread/#force-fetching-groups-from-microsoft-graph-api)
when configuring Azure AD authentication to force Grafana to fetch groups using Graph API.

### RBAC: List token's permissions

Generally available in Grafana Enterprise and Cloud Advanced

We added a new endpoint to help users diagnose permissions-related issues with user and token authorization.
[This endpoint](../../../developers/http_api/access_control/#list-your-permissions) allows users to get the
full list of RBAC permissions associated with their token.

For more details, refer to [Debug the permissions of a service account token](../../../administration/service-accounts/#debug-the-permissions-of-a-service-account-token).

### RBAC with Terraform: Extended support for provisioning permissions

Generally available in Grafana Enterprise and Cloud Advanced

All Grafana users can now use the latest release of [Terraform's Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) (version 1.31.1+) to provision [user and team access to service accounts](../../../administration/service-accounts/#manage-users-and-teams-permissions-for-a-service-account-in-grafana).

This allows full management of service accounts through Terraform - from creating a service account and allowing users to access it to assigning roles to the service account and generating service account tokens.

Grafana Enterprise and Cloud Pro and Advanced users can now provision [access to data sources](../../../administration/data-source-management/#data-source-permissions) for Grafana's `Viewer`, `Editor`, and `Admin` basic roles, as well as assign `Edit` permission.

We have also added [documentation on provisioning RBAC roles and role assignments](../../../administration/roles-and-permissions/access-control/rbac-terraform-provisioning/) to guide our Grafana Enterprise and Cloud Pro and Advanced users through this process.

Finally, we have fixed several access control related bugs to ensure a smoother provisioning experience.

## Alerting

All of these new alerting features are generally available in all editions of Grafana.

### Support for Webex Teams

You can now use Cisco Webex Teams as a contact point, to send alerts to a Webex Teams space.

### Edit alert rules created using the provisioning API

Edit API-provisioned alert rules from the Grafana UI. To make a provisioned alert editable, add the `x-disable-provenance` header to the following requests when creating or editing your alert rules in the API:

POST /api/v1/provisioning/alert-rules

PUT /api/v1/provisioning/alert-rules/{UID}

### Support values in notification templates

Add alert values to notification templates, so that you can create a single template that prints the annotations, labels, and values for your alerts in a format of your choice.

### View notification errors

When an alert fails to fire, see when something is wrong with your contact point(s) and the reason for the error. The Receivers API contains information on the error, including a time stamp, duration of the attempt, and the error. You can also view the errors for each contact point in the UI.

{{< figure src="/static/img/docs/alerting/alert-view-notification-errors-whats-new-v9.3.png" max-width="750px" caption="Alert notification errors" >}}

### Redesign of the expressions pipeline

We've redesigned the expressions pipeline editor to combine the expressions editor and the preview into a single view.

{{< figure src="/static/img/docs/alerting/alert-expression-pipeline-whats-new-v9.3.png" max-width="750px" caption="Expression pipeline redesign" >}}

# What's new in Grafana v9.4

Welcome to Grafana 9.4! Read on to learn about changes to search and navigation, dashboards and visualizations, and authentication and security. For even more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md).

## Search and navigation

We've made the following changes to search and navigation.

### Command palette enhancements

_Generally available in all editions of Grafana._

The command palette has been updated to provide a more efficient way to navigate Grafana. You can now search and access all pages and recent dashboards, making it easier to perform tasks without taking your hands off the keyboard.

To launch the command palette, use the keyboard shortcut `cmd + K` on Mac or `ctrl + K` on Linux/Windows.

To learn more about the command palette, refer to [Search](../../../search/).

{{< figure src="/media/docs/grafana/screenshot-grafana-94-command-palette.png" max-width="750px" caption="Grafana command palette" >}}

### New navigation

_Generally available on Grafana Cloud, and available to preview using the `topnav` [feature toggle](../../../setup-grafana/configure-grafana/#feature_toggles) in all editions of Grafana._

The navigation in Grafana has been updated with a new design and an improved structure to make it easier for you to access the data you need. With this update, you'll be able to quickly navigate between features, giving you full visibility into the health of your systems.

The new navigation is gradually rolling out to all users on Grafana Cloud. If you’re using Grafana Open Source and Enterprise, you can enable this feature using the `topnav` [feature toggle](../../../setup-grafana/configure-grafana/#feature_toggles).

> **Note:** The Grafana documentation has not yet been updated to reflect changes to the navigation.

> **Note:** Plugin developers should refer to [the migration guide](/developers/plugin-tools/migration-guides/update-from-grafana-versions/migrate-9_3_x-to-9_4_x#new-navigation-layout-is-supported) to upgrade their plugins to work seamlessly with the new navigation layout.

{{< figure src="/media/docs/grafana/navigation-9-4.png" max-width="750px" caption="Grafana new navigation" >}}

## Dashboards and visualizations

We've made the following changes to dashboards and visualizations. Learn more about dashboards in our [dashboards documentation](../../../dashboards/).

### Dashboard panel redesign

_Available to preview using the `newPanelChromeUI` [feature toggle](../../../setup-grafana/configure-grafana/#feature_toggles) in all editions of Grafana_.

Dashboard panels contain a lot of information, some of which is difficult to discover or access from the dashboard. With our redesigned panels, we've improved accessibility and made it easier to understand the status of a panel by adding and moving key elements.

We’ve rethought the panel information architecture, added additional interaction points, and reduced visual clutter. To start, we’ve improved the support of panels without a header, made a distinction between details set by you and data-induced information, and then included all essential components in the header of the panel. All of these are laid out from left to right in a row, so there are no overlapping, unusable components.

Grafana’s new panel is available only for React-based panels; no Angular-based panels are redesigned. For example, Angular-based panel will still have the old Graph and Table visualizations.

{{< video-embed src="/media/docs/grafana/screen-recording-panel-header-redesign-whats-new-9-4.mp4" max-width="750px" caption="Panel header redesign" >}}

However, we have more planned: we’re going to make even more improvements to the accessibility of panels and improvements to panels without a header.

### New data source connection page in Dashboards and Explore

_Available to preview using the `datasourceOnboarding` [feature toggle](../../../setup-grafana/configure-grafana/#feature_toggles)._

When you start your journey to create a dashboard or explore your data, but you don't have a data source connected yet, you’ll be shown a page that guides you to set up a first connection.

Administrators can choose between selecting one of the most popular data sources or viewing the entire list. Editors are guided to contact their administrator to configure data sources. In both cases, there's also an option to continue without setting up a data source and to use sample data instead.

{{< figure src="/media/docs/grafana/screenshot-datasource-connection-onboarding-whats-new-9-4.png" max-width="750px" caption="Admin view of data source connection page on dashboard creation" >}}

### Log details redesign

_Generally available in all editions of Grafana._

We've updated the **Details** section of a log line. Previously some of the interactions, such as filtering, showing statistics, or toggling the visibility were split across **Labels** and **Detected fields**. With the recent changes those two sections are combined and the interactions are available for all fields.

{{< figure src="/static/img/logs/log-details-whats-new-9-4.png" max-width="750px" caption="Log details redesign with interactions" >}}

Learn more about viewing logs in our [Logs panel documentation](../../../panels-visualizations/visualizations/logs/).

### Loki data source query validation

_Generally available in all editions of Grafana._

We added support to validate queries and visually display errors as a query is being written, without having to execute it to receive this feedback. This feature supports single and multi-line queries, with and without variables.

{{< figure src="/media/docs/grafana/logs-loki-query-validation-whats-new-9-4.png" max-width="750px" caption="Loki query validation" >}}

Learn more about viewing logs in our [Logs panel documentation](../../../panels-visualizations/visualizations/logs/).

### Loki logs sample in Explore

_Generally available in all editions of Grafana._

For Loki metric queries in Explore, you can now see the sample of log lines that contributed to the displayed results. To see these logs, click on the collapsed **Logs sample** panel under your graph or table panel. If you want to interact with your log lines or modify the log query, click on the "Open logs in split view" button and the log query will be executed in the split view.

{{< figure src="/media/docs/grafana/logs-sample-whats-new-9-4.png" max-width="750px" caption="Logs sample in Explore" >}}

## Canvas panel

_Available in **beta** in all editions of Grafana_

Canvas is a new panel that combines the power of Grafana with the flexibility of custom elements. Canvas visualizations are extensible form-built panels that allow you to explicitly place elements within static and dynamic layouts. This empowers you to design custom visualizations and overlay data in ways that aren’t possible with standard Grafana panels, all within Grafana’s UI. If you’ve used popular UI and web design tools, then designing Canvas panels will feel very familiar.

In Grafana v9.4, we have added the ability to create connections (arrows). Connections enable you to connect elements together to create more complex visualizations. We also added support for data links and a brand new server element. To learn more about the Canvas panel, refer to [Canvas](../../../panels-visualizations/visualizations/canvas/).

{{< video-embed src="/media/docs/grafana/canvas-connections-9-4-0.mp4" max-width="750px" caption="Canvas panel connections" >}}

## Auth and security

_All auth updates are generally available in all editions of Grafana._

We've made the following changes to authentication and security.

### Service account expiration dates

We have added a configuration option that enables you to require an expiration date limit for all newly created service account tokens.

This change will not affect existing tokens. However, newly created tokens will require an expiration date that doesn't exceed the configuration option `token_expiration_day_limit`. This option is disabled by default.

Learn more about service accounts in our [Service account documentation](../../../administration/service-accounts/).

### OAuth providers setting for skip org role sync

While Grafana integrates with many different auth providers, we have received requests for a feature that enables you to bypass organization role synchronization for individual providers rather than for all configured providers. This option is now available for users who want to be able to use Grafana to manage their org roles.

This option enables you to skip synchronization from your configured OAuth provider specifically in the auth provider section under `skip_org_role_sync`. Previously users could only do this for certain providers using the `oauth_skip_org_role_sync_update` option, but this would include all of the configured providers.

Learn more about Oauth in our [Oauth configuration guide](../../../setup-grafana/configure-access/configure-authentication/generic-oauth/).

### RBAC support for Grafana OnCall plugin

We're rolling out RBAC support to Grafana plugins, with Grafana OnCall being the first plugin to fully support RBAC.
Previously, Grafana OnCall relied on the Grafana basic roles (for example, Viewer, Editor, and Admin) for authorization within the plugin.

Before RBAC support in Grafana OnCall, it was only possible to allow your organization's users to either view everything, edit everything, or be an admin (which allowed edit access plus a few additional behaviors). With this new functionality, organizations will be able to harness fine-grained access control within Grafana OnCall.

For example, you can assign a Viewer basic role to a user in your organization (users must still have a basic role assigned) and also assign them the new Grafana OnCall RBAC role of **Schedules Editor**. This assignment enables the user to view everything in Grafana OnCall, and edit OnCall schedules.

Learn more about role-based access control in our [RBAC docs](../../../administration/roles-and-permissions/access-control/).

### SAML auto login

We've added auto-login support for SAML authentication, which you can turn on with the `auto_login` configuration option. We also
have a unified configuration style among all authentication providers. Instead of using
`oauth_auto_login`, use the new `auto_login` option to enable automatic login for specific OAuth providers.

Learn more about SAML setup in our [SAML configuration guide](../../../setup-grafana/configure-access/configure-authentication/saml/).

## Auditing and Usage Insights: Support for Loki multi-tenancy

_This feature is available for Enterprise customers_

This feature adds support to push analytics events and auditing logs to Loki with multi-tenancy mode, by specifying a tenant id. Learn more about [auditing](../../../setup-grafana/configure-security/audit-grafana/) and [usage insights](../../../setup-grafana/configure-security/export-logs/) in our docs.

## Reporting: Zoom in and out on your dashboard in a report PDF

_This feature is available for Enterprise customers_

Zoom is a new feature for reports that allows you to change the dimension of the panels of the PDF document. It enables you to zoom out to show more columns in a table, or zoom in to enlarge panels.
You can modify the scale factor for each report in the report editor when you share the PDF directly from the dashboard page.

{{< figure src="/media/docs/grafana/FormatReportScheduler9.4.png" max-width="750px" caption="Scale factor feature in Report format page" >}}

{{< figure src="/media/docs/grafana/FormatReportShare9.4.png" max-width="750px" caption="Scale factor feature in Share functionality" >}}

Learn more about reporting in our [documentation](../../../dashboards/create-reports/)

## Alerting

We've made major improvements to Grafana Alerts, from new contact points and search options to improved workflows between Alerting and OnCall. For all the details, refer to our [Alerting documentation](../../../alerting/).

### Alerting: alert rules

We've made the following changes to alert rules.

#### Declare incidents from firing alerts

Declare an incident from a firing alert, streamlining the alert to incident workflow.

{{< figure src="/media/docs/alerting/declare-incident.png" max-width="500px" caption="Declare incidents from firing alerts" >}}

#### Copy alert rules and notification templates

To help you reuse existing alert rules or templates, make copies of alert rules from the Alert rule list view and templates from the Contact points page.

{{< figure src="/media/docs/alerting/copy-alert-rules.png" max-width="750px" caption="Copy alert rules and notification templates" >}}

#### View query definitions for provisioned alerts

View read-only query definitions for provisioned alerts from the Alert rule details page. Check quickly if your alert rule queries are correct, without diving into your "as-code" repository for rule definitions.

{{< figure src="/media/docs/alerting/view-query-definitions.png" max-width="750px" caption="View query definitions for provisioned alerts" >}}

#### Export alert rules to use in the provisioning API or files

Create and tune an alert rule in the UI, then export to YAML or JSON, and use it in the provisioning API or files. You can also export an entire rule group to review or use. This is supported in both the UI and provisioning API.

{{< figure src="/media/docs/alerting/export-alert-rules.png" max-width="750px" caption="Export alert rules" >}}

#### Pause alert rule evaluation

Pause alert rule evaluation to prevent noisy alerting while tuning your alerts. Pausing stops alert rule evaluation and does not create any alert instances. This is different to mute timings, which stop notifications from being delivered, but still allow for alert rule evaluation and the creation of alert instances.

{{< figure src="/media/docs/alerting/pause-alerts.png" max-width="750px" caption="Pause alert rule evaluations" >}}

#### View an alert's evaluation interval in Alert Group view

View the evaluation interval more easily from the grouped view on the Alert list page. The view now also always displays recording and normal alert rules and highlights alert rule status in different colors.

{{< figure src="/media/docs/alerting/view-evaluation-interval.png" max-width="750px" caption="View evaluation interval on the Group view" >}}

#### Improved search for your alert rules

When managing large volumes of alerts, use extended alert rule search capabilities to filter folders, evaluation groups, and rules. Additionally, you can filter alert rules by their properties like labels, state, type, and health.

{{< figure src="/media/docs/alerting/search-improvements.png" max-width="750px" caption="Improved search for your alert rules" >}}

#### Adjust the amount and resolution of data used in your alerting queries

Lower costs and improve performance by adjusting the maximum number of data points returned from your alerting queries.

#### Edit alert rule evaluation interval

Simplifies editing the evaluation interval for an alert rule within a new group. You no longer have to save the alert rule and group before editing the evaluation interval.

### Alerting: contact points

We've made the following changes to alert contact points.

#### View Grafana OnCall contact point

Connecting your OnCall workflows just got easier. OnCall has been added as a contact point to simplify the integration between alert notifications and your OnCall implementation.

{{< figure src="/media/docs/alerting/on-call-contact-point.png" max-width="750px" caption="View Grafana OnCall contact point" >}}

### Alert email templating

We've improved the design and functionality of email templates to make template creation much easier and more customizable. The email template framework utilizes MJML to define and compile the final email HTML output. Sprig functions in the email templates provide more customizable template functions.

{{< figure src="/static/img/docs/alerting/alert-templates-whats-new-v9.3.png" max-width="750px" caption="Email template redesign" >}}

#### Add support for Discord as a contact point receiver

We've added Discord as a contact point receiver for Grafana Cloud alert rules.

{{< figure src="/media/docs/alerting/support-discord.png" max-width="750px" caption="Add support for Discord" >}}

### Alerting: administration

We've made the following changes to alert administration.

#### Alerting landing page

Introduces a new landing page that helps you get started quickly with Alerting. It also provides you with at a glance information on how Alerting works and a video to introduce you to key concepts.

{{< figure src="/media/docs/alerting/landing-page.png" max-width="750px" caption="Alerting landing page" >}}

#### Compatibility with AWS Aurora

Grafana Alerting is now compatible with AWS Aurora, but does not provide technical support for it.

## Enterprise Datasources

We've made improvements to all [Enterprise Datasources](../../../introduction/grafana-enterprise/#enterprise-data-sources), fixing small bugs, and updating libraries. We've also added many new features and support for additional APIs. Refer to each datasource's documentation and the change log for additional information.

### DataDog Datasource

We've added support for many new query types, including: SLO/SLI Values, RUM data, Events, and monitor group status.

### Dynatrace Datasource

We've updated the Metric Selector to be faster and added support for filtering by management zone. We've also added support for Log Queries and querying the Dynatrace audit log.

### GitLab Datasource

We've added support for many new query types, including: Audit events, Users, Merge request approvals, Field tags, Environments, and Pipelines.

### Honeycomb Datasource

We've added support for derived columns and Honeycomb Environments.

### NewRelic Datasource

We've added support for trace search, log search, and support for NRQL histogram queries.

### Salesforce Datasource

We've added support for JWT authentication.

### Snowflake Datasource

We've added support for custom session parameters.

## Postgres, MySQL, and MSSQL data sources

The `database` property is now under the `jsonData` key in the data source configuration. This change is backward compatible, and existing configurations will continue to work.

## Before you upgrade

There are no known breaking changes associated with this version of Grafana.

<!-- TODO: Add content -->

# What's new in Grafana v9.5

Welcome to Grafana 9.5! We're excited to share some major updates to Grafana's navigation, tons of usability improvements to Alerting, and some promising experiments to help you query your Prometheus metrics. Also, read on to learn about our continued migration from API keys to service accounts, as well as deprecation of plugins that use Angular and a field in the InfluxDB data source.

For more detail about all the changes in this release, refer to the [changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md). For the specific steps we recommend when you upgrade to v9.5, check out our [Upgrade Guide](https://archive.grafana.com/docs/grafana/v9.5/upgrade-guide/upgrade-v9.5/).

<!-- Template below

## Feature
[Generally available | Available in experimental/beta] in Grafana [Open Source, Enterprise, Cloud Free, Cloud Pro, Cloud Advanced]

Description. Include an overview of the feature and problem it solves, and where to learn more (like a link to the docs).

> **Note:** You must use relative references when linking to docs within the Grafana repo. Please do not use absolute URLs. For more information about relrefs, refer to [Links and references](/docs/writers-toolkit/writing-guide/references/).-->

## Grafana's new navigation is generally available

_Generally available in all editions of Grafana._

The navigation in Grafana has been updated with a new design and an improved structure to make it easier for you to access the data you need. With this update, you can quickly navigate between pages, giving you full visibility into the health of your systems.

As Grafana evolved from a visualization platform to a comprehensive observability solution, we added numerous tools to support users throughout the software development life cycle. These tools focus on preventing incidents, monitoring applications or infrastructure, and aiding incident response. However, the added functionality must be easily discoverable and navigable to be truly helpful. These key updates to Grafana’s navigation experience help address this:

- A redesigned navigation menu that groups related tools together for easy access.
- A command palette you can use to take actions in Grafana, like creating a dashboard or navigating to an app or page.
- Updated layouts featuring breadcrumbs and a sidebar, allowing you to quickly jump between pages.
- A new header that appears on all pages in Grafana, which includes a search function.

Join the [discussion on GitHub](https://github.com/grafana/grafana/discussions/58910) and share your feedback.

{{< figure src="/media/docs/grafana/navigation-9-4.png" max-width="750px" caption="Grafana new navigation" >}}

## Dashboards and visualizations

### Redesigned empty dashboard state

_Generally available in all editions of Grafana._

Dashboards have been updated so that it’s easier to begin building from scratch. The options displayed when you add a new dashboard—adding a visualization, a row, or importing panels—each include brief explanations of what those steps will do, so you can begin building with confidence.

Also, a text **Add** dropdown with these options has replaced the previous "+" icon at the top of the dashboard. This makes it clearer that this element allows you not only to add new panels, but to take all the actions associated with building a new dashboard.

{{< figure src="/media/docs/grafana/screenshot-empty-dashboard-whats-new-9-5.png" max-width="750px" caption="Dashboard without any visualizations added" >}}

### Redesigned dashboard panel is generally available

_Generally available in all editions of Grafana._

Dashboard panels contain a lot of information, some of which is difficult to discover or access from the dashboard. With our redesigned panels, we've improved accessibility and made it easier to understand the status of a panel by adding and moving key elements.

We’ve improved panels without titles, made panel descriptions and errors more succinct, and linked key actions from the header of the panel. All of these are laid out from left to right in a row, so there are no overlapping, unusable components.

Grafana’s new panel design is available only for React-based panels. No Angular-based panels, like the legacy Graph and Worldmap panels, are redesigned. As a reminder, Angular is deprecated in Grafana and will be removed in a future release. See our [deprecation docs](https://archive.grafana.com/docs/grafana/v12.2/developers/angular_deprecation/) for more information.

{{< figure src="/media/docs/grafana/panel-redesign-whats-new-9.5.png" max-width="750px" caption="Panel header with open menu" >}}

## Prometheus performance and usability improvements

### Prometheus metric encyclopedia

_Experimental in all editions of Grafana._

When you have thousands (or millions) of Prometheus metrics, it can be hard to find the exact one you're looking for. Enable feature toggle `prometheusMetricEncyclopedia` to replace the basic metric select dropdown in the Prometheus query builder with a paginated and searchable metric _encyclopedia_.

Here's what you can do with the metric encyclopedia:

- Fuzzy search for metrics by name, type, and description
- Filter metrics by Prometheus types (gauge, counter, histogram, summary)
- Display metrics in a paginated list, sort the results, and choose a number of results per page, so that you don't wait a long time for search results
- View metric details, like type and description
- [Expert feature] Search metric names by regex using the backend only

### Prometheus browser cache

_Experimental in all editions of Grafana._

New feature toggle `prometheusResourceBrowserCache` provides the ability to cache Prometheus editor API calls in the Prometheus data source configuration.
This improves Prometheus query editor performance, with the biggest performance improvements seen by users with high cardinality Prometheus instances.

## Removal of API key creation from the UI

With this update we are going one step further in deprecating API keys in favor of [service accounts](../../../administration/service-accounts/). We've removed the button for creating new API keys through Grafana's user interface, and now only allow the creation of API keys using our HTTP API. We recommend that you migrate your existing API keys to service accounts, and opt for new service accounts instead of new API keys. This change is part of our long-term strategy for sunsetting API keys.

Learn more about the deprecation strategy for API keys and how to manage them in our [Sunsetting API keys](https://github.com/grafana/grafana/issues/53567) GitHub issue.

## Resolve Grafana issues faster with support bundles

_Generally available in all editions of Grafana._

Support bundles provide a simple way to collect information about your Grafana instance through Grafana's user interface. In a few clicks, you can create a support bundle containing data about migrations, plugins, settings, and more. After you've created a support bundle, you can either examine it yourself, or share it with your colleagues or Grafana engineers to aid in troubleshooting of your Grafana instance.

Learn more about support bundles and how to configure them in our [support bundle documentation](https://archive.grafana.com/docs/grafana/v12.4/troubleshooting/support-bundles/).

{{< figure src="/static/img/docs/troubleshooting/support-bundle.png" max-width="750px" caption="Create a support bundle to resolve issues faster" >}}

## Alerting

_All Alerting improvements are generally available in all editions of Grafana._

### Search for alert rules from multiple data sources

Search for and display alert rules for multiple data sources at the same time.

### Fuzzy search on the Alert rule list view

Search for namespaces or folders, evaluation groups, and alert rule names on the Alert rules list view with immediate results, and regardless of typos.

### Access an alert rule from a dashboard or a panel

Navigate to an alert rule directly from a dashboard or a panel to easily access the alert rule details.

{{< figure src="/media/docs/alerting/alert-rule-dashboard.png" max-width="750px" caption="Access an alert rule from a dashboard or a panel" >}}

### Access a dashboard or panel from an alert rule

Navigate from an alert rule straight to a dashboard or a panel associated with the alert rule to visualize your alerting data.

{{< figure src="/media/docs/alerting/dashboard-alert-rule-2.png" max-width="750px" caption="Access a dashboard or panel from an alert rule" >}}

### Preview queries for recording rules

Visualize queries when creating or editing recording rules, so you can see the results of your query before saving your recording rule.

{{< figure src="/media/docs/alerting/preview-queries-recording-rule.png" max-width="750px" caption="Preview queries for recording rules" >}}

### Updated alert behavior when an evaluation returns no data

Alert rules that are configured to fire when an evaluation returns no data now only fire when the entire duration of the evaluation period has finished. This means that rather than immediately firing when the alert rule condition is breached, the alert rule waits until the time set in the **For** field has finished and then fires, reducing alert noise and allowing for temporary data availability issues.

### Improved Notification policies view

Updates to the Notification policies view make it easier to use and manage in the following ways:

- View default policy and nested policies at a glance
- New tab for mute timings
- View alert instances for each policy
- View contact points and which integrations are configured for each policy
- View inherited properties on nested policies
- Search for label matchers and for contact points to see which notifications are going where

{{< figure src="/media/docs/alerting/notification-policies-view.png" max-width="750px" caption="Improved Notification policies view" >}}

### Guidance for configuring your Alertmanager

Get additional help while configuring your Alertmanager. If you enter an invalid Alertmanager configuration, an error message displays, and you can choose from a previous working configuration to restart it.

## InfluxDB plugin database field deprecation

The `database` field in the provisioning file has been deprecated.
This information will be stored in the `jsonData` field using the `dbName` property.
The `database` field will be removed in the future to make InfluxDB consistent with other data sources.
For more information and examples please refer to the [InfluxDB Provisioning docs](../../../datasources/influxdb/#provision-the-data-source).

## Auth: Lock organization roles synced from auth providers

_Generally available in all editions of Grafana._

Grafana v9.4 provided the ability to configure synchronization of organization roles for each OAuth provider. With synchronization on, the organization role was applied to the user from the OAuth provider upon signing in. However, after the user signed in, you could still change the user’s organization role during the session.

With this release, we are reinforcing organization role syncing behavior by introducing a new feature toggle called `onlyExternalOrgRoleSync`. Once enabled, users signing in to Grafana cannot change organization roles that have been synchronized from an external authentication provider, like Active Directory or Google OAuth. This can help ensure the right users maintain the right level of access at all times.

This feature should be used if you want to enforce strict role synchronization from your auth provider to the organization roles.

To use this feature, enable the `onlyExternalOrgRoleSync` feature toggle. If you’re using Grafana Cloud and would like to enable this feature, please contact customer support. We'll also be automatically enabling this feature for Grafana Cloud instances over the upcoming weeks.

You can also _prevent_ the synchronization of organization roles from a given authentication provider. Learn more in our [skip org role sync](../../../setup-grafana/configure-grafana/#authgrafana_com-skip_org_role_sync/)! documentation.

## Reporting UI adapted to match the new navigation style

_Generally available in Grafana Enterprise, Cloud Pro, and Cloud Advanced._

We updated the reporting UI to better fit the new navigation style, adding a horizontal slider and moving the **Preview** and **Send** buttons to the Action section in the page. We also fixed the alignment of the different sections.

{{< figure src="/media/docs/grafana/Screenshot-newUI-report.png" max-width="750px" caption="New Grafana report UI" >}}

## Experimental support for using JWTs as auth method

_Experimental in Grafana Open Source and Enterprise._

This feature adds support for using JWT tokens to store rendering keys instead of relying on “remote caching”. It covers most rendering use cases, though some still rely on the remote cache as a store. You can enable this by enabling the feature flag `renderAuthJWT` in the `custom.ini` configuration file.

## Note for plugin developers

One of the major changes coming in Grafana 10 will be our upgrade to React 18 and use of the new React client rendering API. There are many significant benefits we gain from this: access to new React features like [transitions](https://react.dev/reference/react/useTransition) and concurrent rendering, as well as other general performance and security improvements. These changes have now been delivered to the core `grafana` repo with [PR 64428](https://github.com/grafana/grafana/pull/64428).

As with any major upgrade, there's a potential for this to impact the way your plugin works. In particular, there could be unintended side effects caused by the changes around improving consistency with `useEffect` timings and automatic batching of state updates.

Recommended actions:

- Review the React 18 [upgrade docs](https://react.dev/blog/2022/03/08/react-18-upgrade-guide)
- Test your plugins against one of the latest [grafana-dev docker images](https://hub.docker.com/r/grafana/grafana-dev/tags?page=1) (for example, [this one](https://hub.docker.com/layers/grafana/grafana-dev/10.0.0-111404pre/images/sha256-ac78acf54b44bd2ce7e68b796b1df47030da7f35e53b02bc3eec3f4de05f780f?context=explore))
- Add a comment to the [forum discussion](https://community.grafana.com/t/grafana-10-is-upgrading-to-react-18/86051) if your plugin is impacted in any way. Either to socialise the changes needed for your plugin or to reach out and ask for help yourself.

# Contribute to 'What's new in Grafana Cloud'

To contribute to [What's new in Grafana Cloud](https://grafana.com/docs/grafana-cloud/whatsnew/), refer to [Contribute to What’s new or release notes](https://grafana.com/docs/writers-toolkit/contribute-documentation/contribute-release-notes/).