# Datasources # Data sources, plugins, and integrations When working with Grafana, you'll encounter three key concepts: data sources, plugins, and integrations. Each one is essential in building effective monitoring solutions, but they serve distinct purposes, and are often confused with one another. This document clarifies the meaning of each concept and what each one does, when to use it, and how they work together to create observability solutions in Grafana. The most important thing to remember: every data source is powered by a plugin. You _install_ a plugin, then _configure_ a data source. Core plugins like Prometheus come bundled with Grafana. ## Data sources A data source is a connection to a specific database, monitoring system, service, or other external location that stores metrics, logs, traces, or other data. Examples include Prometheus, InfluxDB, PostgreSQL, and Amazon CloudWatch. When you configure a data source in Grafana, you provide connection details, credentials, and endpoints that tell Grafana where to fetch data from. Data sources are the foundation for working with Grafana, without them, Grafana has nothing to visualize. Every data source is powered by a data source plugin. Grafana comes bundled with core data source plugins (like Prometheus, Loki, and MySQL), and hundreds more are available in the [plugin catalog](https://grafana.com/grafana/plugins/) for systems ranging from databases and cloud providers to tools like Jira, GitHub, and Datadog. After you configure a data source, you can query it to build dashboards. For example, you can query a Prometheus data source to display CPU metrics or query CloudWatch to visualize AWS infrastructure performance. ## Plugins A plugin extends the core functionality of Grafana. Plugins can add new data source types, visualization panels, or full-featured applications that integrate with Grafana. They make Grafana modular and extensible. Plugins come in three types: - **Data source plugins** enable Grafana to connect to specific types of databases or services. Core data source plugins like Prometheus, Loki, and MySQL come bundled with Grafana. You can install additional data source plugins to connect to other systems. Examples include MSSQL, Databricks, and MongoDB. - **Panel plugins** control how data appears in Grafana dashboards. While Grafana comes with several built-in panel types (like graphs, single stats, and tables), panel plugins extend this by providing specialized ways to display data. Examples include pie chart, candlestick, and traffic light. Note that in some cases, panels don't rely on a data source at all. The **Text** panel can render static or templated content without querying data. Panels can also support user-driven actions. For example, the **Button** panel can trigger workflows or external calls. - **App plugins** package data sources, panels, and custom pages into a single install. When you add an app plugin, you get a purpose-built experience for a specific use case. For example, the Azure Cloud Native Monitoring app gives you pre-configured data sources, dashboards, and a dedicated management page for your Azure environment, all in one install. If you're a developer, you can also [build your own app plugin](https://grafana.com/developers/plugin-tools/how-to-guides/app-plugins/). ## Integrations _Integrations are exclusive to Grafana Cloud._ Integrations are pre-packaged monitoring solutions that bundle data collection instructions, pre-built dashboards, and alert and recording rules. They support metrics, logs, and traces. Unlike standalone data sources, integrations provide tested configuration and ready-to-use dashboards and alerts. For example, a Linux integration provides instructions and configuration for collecting metrics and logs from your hosts, creates dashboards for monitoring, and sets up common alerts—all working together out of the box. ## When to use each Use a data source when: - You want to connect Grafana to a specific system (for example, Prometheus or MySQL). - You're building custom dashboards with hand-picked metrics and visualizations. - Your monitoring needs are unique or not covered by pre-packaged integrations. Use a plugin when: - You need to connect to a system Grafana doesn't support natively. - You want to add new functionality (visualizations, workflows, or app-style extensions). - You have specialized or industry-specific requirements (for example, IoT). Use an integration when: - You're using Grafana Cloud and want a quick, pre-built setup. - You prefer minimal configuration with ready-to-use dashboards and alerts. - You're new to observability and want to learn what good monitoring looks like. ## Relationships and interactions Data sources, plugins, and integrations work together to build observability solutions. The following diagram shows two paths: the custom path where you install plugins and configure data sources yourself, and the integration path where Grafana Cloud provides a pre-packaged setup. {{< figure src="/media/docs/datasources/concepts-diagram-1.png" alt="Diagram showing the custom path and integration path for working with data in Grafana" >}} App plugins can combine both paths by bundling data source plugins, panel plugins, and custom pages into a single package. The following examples show how these concepts work together in practice: - Grafana includes a built-in Prometheus data source plugin. Configure a Prometheus data source by providing your server URL, then build a dashboard with a time series panel to monitor CPU usage across your hosts. - Install the Databricks data source plugin. Configure the Databricks data source and run SQL queries against your Databricks workspace. Use the `Histogram` panel to visualize distributions in your query results, such as latency buckets, job durations, or model output scores. - Install the Redis Application app plugin. This app provides a unified experience for monitoring Redis by working with your existing Redis data source. It adds custom pages for configuration and exploration, along with prebuilt dashboards, commands, and visualizations that help you analyze performance, memory usage, and key activity. - If you're using Grafana Cloud, add the ClickHouse integration. This integration provides pre-built dashboards and alerts to monitor ClickHouse cluster metrics and logs, so you can visualize and analyze ClickHouse performance and health. ## Frequently asked questions **What's the difference between a data source and a data source plugin?** A data source plugin is a **software component that enables Grafana to communicate** with specific types of databases or services, like Prometheus, MySQL, or InfluxDB. A data source is **an actual configured connection** to one of these databases, including the credentials, URL, and settings needed to retrieve data. Think of it this way: You _install_ a plugin but _configure_ a data source. **Do I need a plugin to use a data source?** You must install the plugin before you configure or use the data source. Each data source plugin has its own versioning and lifecycle. Grafana includes built-in core data sources, which can be thought of as pre-installed plugins. **Can I use integrations in self-managed Grafana?** No, integrations are exclusive to Grafana Cloud. In self-managed Grafana, you can replicate similar setups manually using data sources and dashboards. **Aren't integrations just pre-built dashboards?** No, integrations are much more than just dashboards. While dashboards are part of an integration, they're only one piece. Integrations typically include: - Data collection setup (for example, pre-configured agents or exporters). - Predefined metrics and queries tailored to the technology. - Alerting rules and notifications to help detect common issues. - Dashboards to visualize and explore that data. **How do data sources and integrations differ in how they handle data?** Data sources query data where it already lives. They connect Grafana to an external system or database, such as Prometheus, MySQL, or Elasticsearch, and fetch data on demand. You keep full control over your own data stores, schemas, and retention policies. In contrast, integrations focus on getting data into Grafana Cloud's managed backends for metrics, logs, and traces. They ingest telemetry using pre-configured agents and pipelines. Instead of querying an external database, Grafana queries its own managed storage where the integration has placed the data. ## Summary reference The following table compares the three core concepts at a glance. | Concept | What it is | When to use it | Example | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------- | | **Data source** | A configured connection to an external database or service. Available in self-managed and Cloud. | You want to query a specific system and build custom dashboards. | Prometheus, CloudWatch, PostgreSQL | | **Plugin** | An extension that adds new data source types, visualizations, or app experiences. Available in self-managed and Cloud. Comes in three types: data source, panel, and app. | You need functionality that isn't built into Grafana. | MongoDB data source, Geomap panel, Azure Cloud Native Monitoring app | | **Integration** | A pre-packaged monitoring solution with collection configuration, dashboards, and alerts. Grafana Cloud only. | You want a quick, ready-to-use setup with minimal configuration. | Linux, Redis, NGINX | For detailed documentation and how-to guides related to data sources, plugins, and integrations, refer to the following references: **Data sources**: - [Manage data sources](ref:data-source-management) **Plugins**: - [Browse the Grafana plugin catalog](https://grafana.com/grafana/plugins/) - [Manage plugins](ref:plugin-management) **Integrations**: - [Grafana integrations](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/) - [Install and manage integrations](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/install-and-manage-integrations/) **For plugin developers**: - [Plugin types and usage](https://grafana.com/developers/plugin-tools/key-concepts/plugin-types-usage) - [Build a data source plugin](https://grafana.com/developers/plugin-tools/how-to-guides/data-source-plugins/) - [Build a panel plugin](https://grafana.com/developers/plugin-tools/how-to-guides/panel-plugins/) - [Build an app plugin](https://grafana.com/developers/plugin-tools/how-to-guides/app-plugins/) # Troubleshoot general data source issues This page provides solutions to common issues that apply across data sources in Grafana. For troubleshooting specific to a data source, refer to the troubleshooting page within each data source's documentation. ## Connection errors These errors occur when Grafana can't reach the data source backend. ### "Connection refused" or timeout errors **Symptoms:** - **Save & test** fails with a connection error or timeout - Panels show "error" or fail to load data - Intermittent connectivity issues **Possible causes and solutions:** | Cause | Solution | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | Incorrect URL | Verify the data source URL is correct, including the protocol (`http://` or `https://`) and port number. | | Network or firewall rules | Ensure the Grafana server can reach the data source endpoint. Check that firewall rules allow outbound traffic on the required port. | | Data source is down | Verify the data source service is running and accepting connections. | | DNS resolution failure | Confirm the hostname resolves correctly from the Grafana server. | | Private network access | If the data source is on a private network and you're using Grafana Cloud, configure [Private data source connect](ref:private-data-source-connect). | ### TLS/SSL errors **Symptoms:** - Errors mentioning "certificate," "TLS handshake," or "x509" - **Save & test** fails with SSL-related messages **Solutions:** 1. Verify the data source is using a valid TLS certificate. 1. If you're using a self-signed certificate, enable **Skip TLS Verify** in the data source configuration (not recommended for production) or add the CA certificate to the list of trusted certificates in Grafana. 1. Ensure the certificate hasn't expired. 1. Confirm the certificate's Common Name or Subject Alternative Name matches the hostname in the data source URL. ## Authentication errors These errors occur when credentials are invalid, missing, or lack the required permissions. ### "Unauthorized" or "Access denied" **Symptoms:** - **Save & test** fails with `401 Unauthorized` or `403 Forbidden` - Queries return access denied messages - Drop-down menus don't populate **Possible causes and solutions:** | Cause | Solution | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | Invalid credentials | Double-check the username, password, API key, or token. Regenerate credentials if necessary. | | Expired credentials | Create new credentials and update the data source configuration. | | Insufficient permissions | Ensure the account or API key has the permissions required by the data source. Refer to the specific data source's documentation for required permissions. | | Wrong authentication method | Verify you've selected the correct authentication type for your setup. | ## Query errors These errors occur when executing queries against a properly connected data source. ### "No data" or empty results **Symptoms:** - Queries execute without error but return no data - Panels show "No data" - Graphs are empty **Possible causes and solutions:** | Cause | Solution | | ------------------------------- | ------------------------------------------------------------------------------------------------- | | Time range doesn't contain data | Expand the dashboard time range or verify data exists in the data source for the selected period. | | Incorrect query | Review the query syntax. Use the data source's query editor to build or validate the query. | | Wrong data source selected | Verify you've selected the correct data source in the panel or Explore. | | Permissions issue | Ensure the credentials have read access to the specific resource or index being queried. | ### Query timeout **Symptoms:** - Queries run for a long time then fail - Error messages mention timeout or query limits **Solutions:** 1. Narrow the dashboard time range to reduce the volume of data. 1. Add filters to the query to reduce the result set. 1. Break complex queries into smaller parts. 1. Increase the data source timeout setting if your data source legitimately needs more time. ## Data source configuration errors ### "Save & test" fails after provisioning **Symptoms:** - Provisioned data sources fail the connection test - Errors appear after deploying provisioning YAML files **Solutions:** 1. Verify the provisioning YAML syntax is correct. Refer to [Provision data sources](ref:provisioning-data-sources) for the expected format. 1. Ensure `secureJsonData` values (such as passwords, API keys, and tokens) are set correctly. These values can't be read back after being saved. 1. Check that the provisioning file is in the correct directory and that Grafana has read access. 1. Restart Grafana after making changes to provisioning files. ### Data source disappears or resets **Symptoms:** - Data source changes revert after Grafana restarts - Data source configuration can't be saved through the UI **Solutions:** 1. Provisioned data sources can't be edited through the UI. Make changes in the provisioning YAML file instead. 1. Verify no other provisioning file is overwriting your data source configuration. ## Enable debug logging To capture detailed error information for troubleshooting: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Restart Grafana for the change to take effect. 1. Reproduce the issue and review logs at `/var/log/grafana/grafana.log` (or your configured log location). 1. Look for entries related to your data source that include request and response details. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If the solutions on this page don't resolve your issue: 1. Check the troubleshooting page for your specific data source. 1. Search the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs. 1. Contact [Grafana Support](https://grafana.com/support/) if you're a Grafana Enterprise, Cloud Pro, or Cloud Advanced user. When reporting issues, include: - Grafana version and data source plugin version - Exact error messages (redact sensitive information) - Steps to reproduce the issue - Relevant configuration (redact credentials) # Amazon CloudWatch alerting The Amazon CloudWatch data source supports [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). You can create alert rules on CloudWatch metrics and CloudWatch Logs, so Grafana notifies you when a metric crosses a threshold or when a log query returns a value that meets your alert condition. This is Grafana Alerting evaluated by Grafana, not CloudWatch alarms. To visualize the history of existing CloudWatch alarms instead, refer to [Amazon CloudWatch annotations](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/annotations/). ## Before you begin Before you create alert rules with CloudWatch data, ensure you have: - A [configured Amazon CloudWatch data source](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/configure/). - Permissions to create alert rules in Grafana. - Familiarity with [Grafana Alerting concepts](https://grafana.com/docs/grafana//alerting/fundamentals/). ## Supported query types for alerting Alert rules require queries that return numeric data that Grafana can evaluate against a threshold. CloudWatch supports alerting for the following query types: | Query type | Use case | Notes | | ---------- | -------------------------------------------------- | -------------------------------------------------------------- | | Metrics | Threshold-based alerts on CloudWatch metrics | Returns time series data; best suited for alerting. | | Logs | Alert on log patterns, error counts, or thresholds | Use the `stats` command to aggregate logs into numeric values. | {{< admonition type="note" >}} You can't create an alert rule from a metric math expression that references another query, such as `queryA * 2`. Define the metric directly in the alert query instead. {{< /admonition >}} ## Create an alert rule To create an alert rule using CloudWatch data: 1. Go to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for your alert rule. 1. In the **Define query and alert condition** section: - Select your Amazon CloudWatch data source. - Configure your query, for example, a Metrics query for `CPUUtilization` or a Logs query that uses the `stats` command. - Add a **Reduce** expression if your query returns multiple series. - Add a **Threshold** expression to define the alert condition. 1. Configure the **Set evaluation behavior** section: - Select or create a folder and evaluation group. - Set the evaluation interval. - Set the pending period. 1. Add labels and annotations to provide context for notifications. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example: metric threshold alert This example fires when average EC2 CPU utilization exceeds 80%: 1. Create a new alert rule. 1. Configure the query: - **Namespace**: `AWS/EC2` - **Metric name**: `CPUUtilization` - **Statistic**: `Average` - **Dimensions**: `InstanceId` 1. Add expressions: - **Reduce**: Last, to get the most recent data point. - **Threshold**: Is above 80. 1. Set evaluation to run every 1 minute with a 5-minute pending period. 1. Save the rule. ## Example: log error count alert This example alerts when the number of log entries that contain `Exception` exceeds a threshold. CloudWatch Logs alerting requires a query that returns numeric data, which you can produce with the `stats` command: 1. Create a new alert rule. 1. Configure the query: - Select **CloudWatch Logs** as the query mode. - Select the region and log groups to query. - Enter the following query: ``` filter @message like /Exception/ | stats count(*) as exceptionCount by bin(5m) ``` 1. Add expressions: - **Reduce**: Max, to get the highest count in the period. - **Threshold**: Is above 10. 1. Set evaluation to run every 5 minutes. 1. Save the rule. {{< admonition type="note" >}} If you receive an error such as `input data must be a wide series but got ...`, make sure your query returns numeric data that can render in a time series panel. {{< /admonition >}} ## CloudWatch Logs timeouts during alert evaluation For CloudWatch Logs queries, Grafana polls AWS until the query completes or a timeout is reached. During alert evaluation, the timeout defined in the [Grafana configuration file](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/) takes precedence over the **Query Result Timeout** setting on the data source. Set the configuration timeout high enough for your log queries to finish. ## Best practices Follow these recommendations to create reliable alerts with CloudWatch data. ### Test queries before alerting Verify your query returns numeric data before you create an alert: 1. Go to **Explore**. 1. Select your Amazon CloudWatch data source. 1. Run the query you plan to use for alerting. 1. Confirm the result is numeric and suitable for threshold evaluation. ### Reduce multiple series When a query returns multiple time series, use the **Reduce** expression to aggregate them into a single value with **Last**, **Mean**, **Max**, **Min**, or **Sum**. ### Handle no data conditions Configure how the rule behaves when no data is returned under **Configure no data and error handling**. Choose **No Data**, **Alerting**, or **OK** based on whether missing data should be treated as a problem. ## Troubleshoot alerting If your CloudWatch alerts don't work as expected, use the following sections to diagnose common issues. ### Alerts don't fire - Confirm the query returns numeric data in Explore. - Ensure the evaluation interval allows enough time for data to be available. - Review the alert rule's health and any error messages in the Alerting UI. ### Log query alerts time out - Increase the Grafana configuration file timeout, which takes precedence during alert evaluation. - Narrow the time range or add filters to reduce the volume of data the query scans. For more help, refer to [Troubleshoot Amazon CloudWatch](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/troubleshooting/). ## Related resources - [Grafana Alerting](https://grafana.com/docs/grafana//alerting/) - [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/) - [Amazon CloudWatch query editor](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/query-editor/) # Amazon CloudWatch annotations [Annotations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/) overlay rich event information on top of graphs. With the Amazon CloudWatch data source, annotations are based on [CloudWatch alarm history](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html). Each alarm history item within the dashboard time range—such as a state change, configuration update, or action—becomes an annotation, so you can correlate alarm activity with the rest of your dashboard data. ## Before you begin Before you create CloudWatch annotations, ensure you have: - A [configured Amazon CloudWatch data source](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/configure/). - One or more CloudWatch alarms in the AWS account and region you want to query. - IAM permissions to describe alarms and alarm history. Refer to [Required permissions](#required-permissions). ## How CloudWatch annotations work Unlike data sources that build annotations from free-form queries, CloudWatch annotations query existing CloudWatch alarms and return their state-change history. Grafana finds the matching alarms using one of two methods, retrieves each alarm's history with `cloudwatch:DescribeAlarmHistory`, and renders every history item as an annotation: - **Match alarms by metric:** Grafana calls `cloudwatch:DescribeAlarmsForMetric` to find alarms attached to a specific metric. - **Match alarms by prefix:** Grafana calls `cloudwatch:DescribeAlarms` with your alarm name and action prefixes, so AWS returns only alarms whose names and actions start with those prefixes. Grafana then filters the returned alarms by the namespace, metric name, dimensions, statistic, and period you specify. Each annotation includes the alarm name as its title, the alarm history item type as a tag (for example, `StateUpdate`, `ConfigurationUpdate`, or `Action`), and the history summary as its text. ## Create an annotation query To add a CloudWatch annotation to a dashboard: 1. Open the dashboard where you want to add annotations. 1. Click **Edit**, then click **Settings** in the top navigation. 1. Select the **Annotations** tab. 1. Click **Add annotation query**. 1. Enter a **Name** for the annotation, for example, `CloudWatch alarms`. 1. Select your **Amazon CloudWatch** data source. 1. Configure the query fields described in the following sections. 1. Click **Save dashboard**. The annotation query editor provides the following fields: | Field | Description | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | **Region** | The AWS region to query for alarms. | | **Namespace** | The metric namespace, for example, `AWS/EC2`. | | **Metric name** | The name of the metric the alarm watches, for example, `CPUUtilization`. | | **Statistic** | The statistic the alarm uses, for example, `Average`. | | **Dimensions** | The dimensions that identify the resource, for example, `InstanceId`. | | **Period** | _Optional._ The minimum interval between data points, in seconds. Defaults to `300` when prefix matching is disabled. | | **Enable Prefix Matching** | _Optional._ Match alarms by name and action prefix instead of by metric. | | **Action** | Match only alarms whose actions start with this prefix. Available when prefix matching is enabled, and required for the query to run. | | **Alarm Name** | Match only alarms whose names start with this prefix. Available when prefix matching is enabled, and required for the query to run. | ### Match alarms by metric This is the default behavior, used when **Enable Prefix Matching** is off. Grafana returns annotations for any alarm attached to the metric you specify. When you match by metric, **Region**, **Namespace**, **Metric name**, and **Statistic** are all required. If any of them are missing, the query returns an `invalid annotations query` error. Dimensions are optional, but adding them narrows the results to alarms on a specific resource. For example, to annotate the history of the CPU utilization alarm on a single EC2 instance, set: - **Region:** `us-east-1` - **Namespace:** `AWS/EC2` - **Metric name:** `CPUUtilization` - **Statistic:** `Average` - **Dimensions:** `InstanceId = i-0123456789abcdef0` ### Match alarms by prefix Enable **Enable Prefix Matching** to find alarms by name and action prefix rather than by a specific metric. Grafana requests up to 100 alarms from `cloudwatch:DescribeAlarms` using the **Alarm Name** and **Action** prefixes, then filters the returned alarms by the namespace, metric name, dimensions, statistic, and period you specify. You must provide both an **Alarm Name** prefix and an **Action** prefix for the query to run. Use prefix matching when you want to surface a group of related alarms, for example, all production alarms that trigger a specific SNS action. To do this, set **Enable Prefix Matching** on, then set **Alarm Name** to `prod-` and **Action** to the ARN prefix of your SNS topic, such as `arn:aws:sns:us-east-1:123456789012:prod-`. ## Required permissions Annotation queries require the following CloudWatch API actions in the IAM policy attached to the role or user running the data source: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowReadingAlarmsFromCloudWatch", "Effect": "Allow", "Action": ["cloudwatch:DescribeAlarms", "cloudwatch:DescribeAlarmsForMetric", "cloudwatch:DescribeAlarmHistory"], "Resource": "*" } ] } ``` The metrics IAM policy on the [configure page](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/configure/#iam-policy-examples) already includes these actions. ## Troubleshoot annotations If annotations don't appear as expected, try the following solutions. ### Annotations don't appear - Verify the alarm has state changes within the dashboard time range. Annotations come from alarm history, so an alarm that never changed state produces no annotations. - Confirm the **Region**, **Namespace**, **Metric name**, **Statistic**, and **Dimensions** exactly match an existing alarm. - Check that the IAM role or user has the `cloudwatch:DescribeAlarms`, `cloudwatch:DescribeAlarmsForMetric`, and `cloudwatch:DescribeAlarmHistory` permissions. ### `invalid annotations query` error - This error appears when prefix matching is disabled and the query reaches the backend with an empty **Region**, **Namespace**, **Metric name**, or **Statistic**. Provide all four fields, or enable **Enable Prefix Matching**. - In the dashboard annotation editor, an incomplete metric query is often skipped silently and produces no annotations rather than returning this error. If you see no annotations, confirm that all required fields are set. ## Related resources - [Annotate visualizations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/) - [Amazon CloudWatch query editor](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/query-editor/) - [Configure the Amazon CloudWatch data source](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/configure/) # Configure AWS authentication Grafana data source plugins make requests to AWS on behalf of an AWS Identity and Access Management (IAM) [role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) or IAM [user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html). The IAM user or IAM role must have the associated policies to perform certain API actions for querying data source data. Since these policies are specific to each data source, refer to individual data source plugin documentation for details. The Grafana backend performs all AWS API requests server-side using the official [AWS SDK](https://github.com/aws/aws-sdk-go). This document explores the following topics: - [Select an authentication method](#select-an-authentication-method) - [Assume a role](#assume-a-role) - [Use a custom endpoint](#use-a-custom-endpoint) - [Use an AWS credentials file](#use-an-aws-credentials-file) - [Use EKS IAM roles for service accounts](#use-eks-iam-roles-for-service-accounts) ## Select an authentication method Available authentication methods depend on your configuration and the environment where Grafana runs. Open source Grafana enables the `AWS SDK Default`, `Credentials file`, and `Access & secret key` methods by default. Cloud Grafana enables `Access & secret key` and `Grafana Assume Role` by default. Users with server configuration access can enable or disable specific auth providers as needed. For more information, refer to the [`allowed_auth_providers` documentation](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#allowed_auth_providers). - `AWS SDK Default` uses the [default provider](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html) from the [AWS SDK for Go](https://github.com/aws/aws-sdk-go) without custom configuration. This method requires configuring AWS credentials outside Grafana through [the CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html), or by [attaching credentials directly to an EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html), [in an ECS task](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html), or for a [Service Account in a Kubernetes cluster](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). You can attach permissions directly to the data source with AWS SDK Default or combine it with the optional [`Assume Role ARN`](#assume-a-role) field. - `Credentials file` maps to the [SharedCredentialsProvider](https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/#SharedCredentialsProvider) provider in the [AWS SDK for Go](https://github.com/aws/aws-sdk-go). This method reads the AWS shared credentials file for a specified profile. Unlike `AWS SDK Default` which also reads the shared credentials file, this option lets you specify a profile directly without environment variables. This option provides no fallback to other credential providers and fails if the file credentials are invalid. - `Access & secret key` corresponds to the [StaticProvider](https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/#StaticProvider) and authenticates using a specified access key ID and secret key pair. This method doesn't provide fallback authentication and fails if the key pair is invalid. It's available in Grafana Cloud, though `Grafana Assume Role` is recommended there because it avoids managing long-term access keys. - `Grafana Assume Role` - With this authentication method, Grafana Cloud users create an AWS IAM role that has a trust relationship with the Grafana AWS account. Grafana uses [STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) to generate temporary credentials on its behalf. This method eliminates the need to generate secret and access keys for users. Refer to [Use Grafana Assume Role](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/aws-authentication/#use-grafana-assume-role) for more information. - `Workspace IAM Role` corresponds to the [EC2RoleProvider](https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/ec2rolecreds/#EC2RoleProvider). The EC2RoleProvider retrieves credentials from a role attached to the EC2 instance running Grafana. While AWS SDK Default can achieve similar results, this option provides no fallback authentication. Amazon Managed Grafana enables this option by default. ## Assume a role Specify an IAM role to assume in the **Assume Role ARN** field. When you configure **Assume Role ARN**, Grafana uses the provided credentials to perform an [sts:AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) call. The primary authentication method only needs permission to assume the role, while the assumed role requires CloudWatch access permissions. For example, you might use one set of long-term credentials across all AWS data sources but want to limit each data source's AWS access (such as separating staging and production data access). You could create separate credentials for each data source with distinct permissions, but this approach requires managing and rotating multiple secret and access keys across many data source instances. Instead, assume role functionality lets you use one set of AWS credentials across all AWS data sources with a single permission: the ability to assume roles through STS. You then create separate IAM roles for each data source that specify temporary permissions. Since IAM roles are not credentials, they require no rotation and simplify management. If the **Assume Role ARN** field is left empty, Grafana uses the provided credentials from the selected authentication method directly, and permissions to AWS data must be attached directly to those credentials. The **Assume Role ARN** field is optional for all authentication methods except for Grafana Assume Role. To disable this feature in open source Grafana or Grafana Enterprise, refer to [`assume_role_enabled`](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#assume_role_enabled). ### Use an external ID {{< admonition type="note" >}} You cannot use an external ID for the Grafana Assume Role authentication provider. {{< /admonition >}} To assume a role in another account created with an external ID, specify the external ID in the **External ID** field. For more information, refer to the [AWS documentation on external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html). ## Use a custom endpoint Use the **Endpoint** field to specify a custom endpoint URL that overrides the default AWS service API endpoint. Leave this field blank to use the default generated endpoint. For more information about using service endpoints, refer to the [AWS service endpoints documentation](https://docs.aws.amazon.com/general/latest/gr/rande.html). ## Use an AWS credentials file Create a file at `~/.aws/credentials`, the `HOME` path for the user running the `grafana-server` service. {{< admonition type="note" >}} If the credentials file appears to be in the correct location but isn't working, move your `aws` file to `/usr/share/grafana/` and set the credentials file permissions to `0644`. {{< /admonition >}} **Credentials file example:** ```bash [default] aws_access_key_id = asdsadasdasdasd aws_secret_access_key = dasdasdsadasdasdasdsa region = us-west-2 ``` ## Use EKS IAM roles for service accounts EKS IAM roles for service accounts (IRSA) are an AWS EKS feature that allows pods to assume IAM roles without storing long-term credentials. When you configure IRSA in your EKS cluster, AWS injects temporary credentials into your pod as projected volume mounts. In Grafana containers, the process runs as user `472` (`grafana`). By default, Kubernetes mounts the projected credentials with permissions for the root user only. To grant user `472` permission to access the credentials, and prevent fallback to the IAM role attached to the EC2 instance, set a [security context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for your pod. **Security context example:** ```yaml securityContext: fsGroup: 472 runAsUser: 472 runAsGroup: 472 ``` ## Use Grafana Assume Role {{< admonition type="note" >}} Grafana Assume Role is available only in Grafana Cloud. It's currently supported only for the Amazon CloudWatch and Amazon Athena data sources. {{< /admonition >}} The Grafana Assume Role authentication provider lets you access AWS without creating or managing long-term AWS IAM users or rotating access keys. Instead, you can create an IAM role that has permissions to access CloudWatch and trusts a Grafana AWS account. The Grafana AWS account then makes a Security Token Service (STS) request to generate temporary credentials for your AWS data. This request includes an `externalID` unique to your Grafana Cloud account, which ensures users can access only their own AWS resources. {{< admonition type="note" >}} The trust policy names the shared Grafana AWS account as the trusted principal, but this doesn't grant every Grafana customer access to your role. The `sts:ExternalId` condition restricts the trust to your unique external ID, so AWS only allows an assume-role request that presents that exact value. Grafana sends your external ID only for your own data source requests. The external ID isn't a secret, but combined with the account-level trust it ensures one customer can't assume another customer's role. Keep the `sts:ExternalId` condition in the trust policy — removing it would let any account that Grafana trusts assume your role. {{< /admonition >}} For more information, refer to the [AWS documentation on external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html). To use the Grafana Assume Role: 1. Create a new CloudWatch data source (or update an existing one) and select **Grafana Assume Role** as an authentication provider. 2. In the AWS Console, create a new IAM role, and under **Trusted entity type**, select **Another AWS account** as the trusted Entity. 3. Enter the Grafana account id (displayed in the instructions box on the **Settings** tab of the CloudWatch data source configuration) and check the **Require external ID** box. 4. Enter the external ID specified in the instructions box on the **Settings** tab of the CloudWatch data source configuration in Grafana. This external ID will be unique to your Grafana instance. 5. Attach any required permissions you would like Grafana to be able to access on your behalf (for example, CloudWatch Logs and CloudWatch Metrics policies). 6. Give the role a name and description, and click **Create role**. 7. Copy the ARN of the role you just created and paste it into the **Assume Role ARN** field on the **Settings** tab of CloudWatch data source configuration in Grafana. **Sample Trust Relationship for an IAM role:** ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": {Grafana's AWS Account} }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": {External ID unique to your account} } } } ] } ``` # Configure the Amazon CloudWatch data source This document provides instructions for configuring the Amazon CloudWatch data source and explains available configuration options. For general information on adding and managing data sources, refer to [Data source management](https://grafana.com/docs/grafana//administration/data-source-management/). ## Before you begin - You must have the `Organization administrator` role to configure the CloudWatch data source. Organization administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system. - Grafana comes with a built-in CloudWatch data source plugin, so you do not need to install a plugin. - Familiarize yourself with your CloudWatch security configuration and gather any necessary security certificates, client certificates, and client keys. ## Key concepts If you're new to AWS, these terms are used throughout the configuration: | Term | Description | | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **IAM policy** | A JSON document attached to an identity that grants permissions to AWS API actions. | | **IAM role** | An identity with permissions that trusted entities can assume temporarily, instead of using long-lived keys. | | **Assume role** | An AWS mechanism that lets one identity temporarily take on the permissions of an IAM role, often used for cross-account access. | | **External ID** | A shared value in a role's trust policy that prevents another party from assuming the role on your behalf. | | **STS (Security Token Service)** | The AWS service that issues the short-lived temporary credentials used when assuming a role. | | **Cross-account observability (OAM)** | An AWS feature, managed through Observability Access Manager, that links monitoring and source accounts so you can query metrics and logs across accounts. | ## Add the CloudWatch data source Complete the following steps to set up a new CloudWatch data source: 1. Click **Connections** in the left-side menu. 1. Click **Add new connection** 1. Type `CloudWatch` in the search bar. 1. Select the **CloudWatch data source**. 1. Click **Add new data source** in the upper right. Grafana takes you to the **Settings** tab, where you will set up your CloudWatch configuration. ## Configure the data source in the UI The following are configuration options for the CloudWatch data source. | **Setting** | **Description** | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Name** | The data source name. Sets the name you use to refer to the data source in panels and queries. | | **Default** | Toggle to select as the default name in dashboard panels. When you go to a dashboard panel, this will be the default selected data source. | Grafana plugin requests to AWS are made on behalf of an AWS Identity and Access Management (IAM) role or IAM user. The IAM user or IAM role must have the associated policies to perform certain API actions. For authentication options and configuration details, refer to [AWS authentication](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/aws-authentication/). | Setting | Description | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Authentication Provider** | Specify which AWS credentials chain to use. A Grafana plugin's requests to AWS are made on behalf of an IAM role or IAM user. The IAM user or IAM role must have the necessary policies to perform the required API actions. | **Access & secret key:** You must use both an access key ID and a secret access key to authenticate. | Setting | Description | | --------------------- | ---------------------------- | | **Access Key ID** | Enter your key ID. | | **Secret Access Key** | Enter the secret access key. | **Credentials file:** When you select the **Credentials file** authentication provider, you can specify which profile to read from the AWS shared credentials file. | Setting | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | **Credentials Profile Name** | The profile name in `~/.aws/credentials`, as specified in the credentials file. Leave blank to use the `default` profile. | **Assume Role**: | Setting | Description | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | **Assume Role ARN** | _Optional._ Specify the ARN of an IAM role to assume. This ensures the selected authentication method is used to assume the role, not used directly. | | **External ID** | If you're assuming a role in another AWS account that requires an external ID, specify it here. | **Additional Settings:** | Setting | Description | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Endpoint** | _Optional_. Specify a custom endpoint for the AWS service. | | **Default Region** | Specify the AWS region. Example: If the region is US West (Oregon), use `us-west-2`. | | **Namespaces of Custom Metrics** | Add one or more custom metric namespaces, separated by commas (for example,`Namespace1,Namespace2`). Grafana can't automatically load custom namespaces using the [CloudWatch GetMetricData API](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html). To make custom metrics available in the query editor, manually specify the namespaces in the `Namespaces of Custom Metrics` field in the data source configuration. | **Proxy configuration:** The **Proxy Configuration** settings appear only when per-data source HTTP proxy support is enabled with the `per_datasource_http_proxy_enabled` option in the Grafana configuration file. They let each CloudWatch data source use its own outbound HTTP proxy for AWS requests. | Setting | Description | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Proxy Type** | Select how the proxy is configured: `Environment (default)` uses the `HTTP_PROXY` and `HTTPS_PROXY` environment variables, `None` disables the proxy, and `URL` lets you specify a proxy URL. Don't set this when Secure Socks Proxy is enabled. | | **Proxy URL** | _Only when Proxy Type is `URL`._ The proxy URL, for example `https://localhost:3004`. Don't include the username or password in the URL. | | **Proxy Username** | _Only when Proxy Type is `URL`._ _Optional._ The proxy username. | | **Proxy Password** | _Only when Proxy Type is `URL`._ _Optional._ The proxy password. | **CloudWatch Logs**: | Setting | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Query Result Timeout** | Grafana polls CloudWatch Logs every second until AWS returns a `Done` status or the timeout is reached. An error is returned if the timeout is exceeded. For alerting, the timeout defined in the Grafana configuration file takes precedence. Enter a valid duration string, such as `30m`, `30s` or `200ms`. The default is `30m`. | | **Default Log Groups** | _Optional_. Specify the default log groups for CloudWatch Logs queries. | **Application Signals trace link:** | Setting | Description | | --------------- | ------------------------------------------------------------------- | | **Data source** | Select the Application Signals data source from the drop-down menu. | Grafana automatically creates a link to a trace in Application Signals data source if logs contain the `@xrayTraceId` field. To use this feature, you must already have an Application Signals data source configured. For details, refer to the [Application Signals data source docs](https://grafana.com/docs/plugins/grafana-x-ray-datasource/latest/). To view the Application Signals link, select the log row in either the Explore view or dashboard [Logs panel](https://grafana.com/docs/grafana//panels-visualizations/visualizations/logs/) to view the log details section. To log the `@xrayTraceId`, refer to the [AWS Application Signals documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html). To provide the field to Grafana, your log queries must also contain the `@xrayTraceId` field, for example by using the query `fields @message, @xrayTraceId`. **Private data source connect** - _Only for Grafana Cloud users._ | Setting | Description | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Private data source connect** | Establishes a private, secured connection between a Grafana Cloud stack and data sources within a private network. Use the drop-down to locate the PDC URL. For setup instructions, refer to [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) and [Configure PDC](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc). Click **Manage private data source connect** to open your PDC connection page and view your configuration details. | After configuring your Amazon CloudWatch data source options, click **Save & test** at the bottom to test the connection. When the test succeeds, Grafana confirms that it reached both CloudWatch APIs: ``` 1. Successfully queried the CloudWatch metrics API. 2. Successfully queried the CloudWatch logs API. ``` {{< figure src="/media/docs/CloudWatch/CloudWatch-config-success-message.png" >}} {{< admonition type="note" >}} To troubleshoot issues while setting up the CloudWatch data source, check the `/var/log/grafana/grafana.log` file. Common issues include invalid credentials, missing regions and metrics-only credentials. {{< /admonition >}} ### IAM policy examples To read CloudWatch metrics and EC2 tags, instances, regions, and alarms, you must grant Grafana permissions via IAM. You can attach these permissions to the IAM role or IAM user you configured in [AWS authentication](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/aws-authentication/). **Metrics-only permissions:** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowReadingMetricsFromCloudWatch", "Effect": "Allow", "Action": [ "CloudWatch:DescribeAlarmsForMetric", "CloudWatch:DescribeAlarmHistory", "CloudWatch:DescribeAlarms", "CloudWatch:ListMetrics", "CloudWatch:GetMetricData", "CloudWatch:GetInsightRuleReport" ], "Resource": "*" }, { "Sid": "AllowReadingTagsInstancesRegionsFromEC2", "Effect": "Allow", "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"], "Resource": "*" }, { "Sid": "AllowReadingResourcesForTags", "Effect": "Allow", "Action": "tag:GetResources", "Resource": "*" }, { "Sid": "AllowReadingResourceMetricsFromPerformanceInsights", "Effect": "Allow", "Action": "pi:GetResourceMetrics", "Resource": "*" } ] } ``` **Logs-only permissions:** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowReadingLogsFromCloudWatch", "Effect": "Allow", "Action": [ "logs:DescribeLogGroups", "logs:ListAggregateLogGroupSummaries", "logs:GetLogGroupFields", "logs:StartQuery", "logs:StopQuery", "logs:GetQueryResults", "logs:GetLogEvents" ], "Resource": "*" }, { "Sid": "AllowReadingTagsInstancesRegionsFromEC2", "Effect": "Allow", "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"], "Resource": "*" }, { "Sid": "AllowReadingResourcesForTags", "Effect": "Allow", "Action": "tag:GetResources", "Resource": "*" } ] } ``` **Metrics and logs permissions:** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowReadingMetricsFromCloudWatch", "Effect": "Allow", "Action": [ "CloudWatch:DescribeAlarmsForMetric", "CloudWatch:DescribeAlarmHistory", "CloudWatch:DescribeAlarms", "CloudWatch:ListMetrics", "CloudWatch:GetMetricData", "CloudWatch:GetInsightRuleReport" ], "Resource": "*" }, { "Sid": "AllowReadingResourceMetricsFromPerformanceInsights", "Effect": "Allow", "Action": "pi:GetResourceMetrics", "Resource": "*" }, { "Sid": "AllowReadingLogsFromCloudWatch", "Effect": "Allow", "Action": [ "logs:DescribeLogGroups", "logs:ListAggregateLogGroupSummaries", "logs:GetLogGroupFields", "logs:StartQuery", "logs:StopQuery", "logs:GetQueryResults", "logs:GetLogEvents" ], "Resource": "*" }, { "Sid": "AllowReadingTagsInstancesRegionsFromEC2", "Effect": "Allow", "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"], "Resource": "*" }, { "Sid": "AllowReadingResourcesForTags", "Effect": "Allow", "Action": "tag:GetResources", "Resource": "*" } ] } ``` #### Cross-account observability permissions ```json { "Version": "2012-10-17", "Statement": [ { "Action": ["oam:ListSinks", "oam:ListAttachedLinks"], "Effect": "Allow", "Resource": "*" } ] } ``` {{< admonition type="note" >}} Cross-account observability lets you retrieve metrics and logs across different accounts in a single region, but you can't query EC2 Instance Attributes across accounts because those come from the EC2 API and not the CloudWatch API. {{< /admonition >}} For more information on configuring authentication, refer to [Configure AWS authentication](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/aws-authentication/). ### Permissions reference The following table explains why each permission is needed and whether it's required, so you can scope a policy to only the features you use. Grant `cloudwatch:GetMetricData` only for metrics, the `logs:` actions only for logs, and the EC2 and tag actions only if you use the related template variables. Granting a permission that you don't use has no effect on functionality. | Permission | Purpose | Required | | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | | `cloudwatch:ListMetrics` | Lists available metrics and populates the namespace, metric, and dimension fields in the query editor. | Required for metrics queries and metrics template variables. | | `cloudwatch:GetMetricData` | Retrieves metric data points for queries and metric math expressions. | Required for metrics queries. | | `cloudwatch:GetInsightRuleReport` | Retrieves data from CloudWatch Contributor Insights rules. | Optional. Only needed if you query Contributor Insights rules. | | `cloudwatch:DescribeAlarms`, `cloudwatch:DescribeAlarmsForMetric`, `cloudwatch:DescribeAlarmHistory` | Reads CloudWatch alarm configuration and history. | Optional. Only needed if you display alarm data. | | `pi:GetResourceMetrics` | Retrieves Performance Insights metrics for supported resources, such as RDS databases. | Optional. Only needed if you query Performance Insights metrics. | | `logs:DescribeLogGroups` | Lists log groups and populates the log group selector. | Required for logs queries and the Log Groups template variable. | | `logs:StartQuery`, `logs:StopQuery`, `logs:GetQueryResults` | Runs CloudWatch Logs Insights queries and retrieves their results. | Required for logs queries. | | `logs:GetLogGroupFields` | Retrieves the fields available in a log group for autocomplete. | Optional. Improves the logs query editor experience. | | `logs:GetLogEvents` | Retrieves individual log events. | Optional. Only needed for queries that read raw log events. | | `logs:Unmask` | Reveals data masked by a CloudWatch Logs data protection policy. | Optional. Only needed to view masked sensitive data. | | `ec2:DescribeRegions` | Lists available AWS regions. | Optional. Only needed if you rely on dynamic region discovery. | | `ec2:DescribeInstances`, `ec2:DescribeTags` | Resolves EC2 instance attributes and tags for dimension and EC2 template variables. | Optional. Only needed for EC2 Instance Attributes template variables. | | `ec2:DescribeVolumes` | Resolves EBS volume IDs for the EBS Volume IDs template variable. | Optional. Only needed for EBS Volume IDs template variables. | | `tag:GetResources` | Resolves resource ARNs for the Resource ARNs template variable. | Optional. Only needed for Resource ARNs template variables. | | `oam:ListSinks`, `oam:ListAttachedLinks` | Discovers linked accounts for CloudWatch cross-account observability. | Optional. Only needed for cross-account observability. | When you use an **Assume Role ARN**, attach these query permissions to the assumed role. The primary credentials only need permission to perform `sts:AssumeRole`. For details, refer to [Assume a role](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/aws-authentication/#assume-a-role). ### CloudWatch Logs data protection CloudWatch Logs can protect data by applying log group data protection policies. When data protection is enabled for a log group, any sensitive data that matches the identifiers you select is automatically masked. To view masked data, your IAM role or user must have the `logs:Unmask` permission. For more details, refer to [the AWS guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html) on masking sensitive log data. ### Configure the data source with grafana.ini The Grafana [configuration file](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#aws) includes an `AWS` section where you can configure data source options: | Configuration option | Description | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `allowed_auth_providers` | Specifies which authentication providers are allowed for the CloudWatch data source as a comma-separated list. The default is `default,keys,credentials`: `default` (AWS SDK Default), `keys` (Access and secret key), and `credentials` (Credentials file). The `ec2_iam_role` (EC2 IAM role) provider is also available but isn't enabled by default. | | `assume_role_enabled` | Allows you to disable `assume role (ARN)` in the CloudWatch data source. The assume role (ARN) is enabled by default in open source Grafana. | | `per_datasource_http_proxy_enabled` | Allows each CloudWatch data source instance to use its own HTTP proxy configuration for requests to AWS, instead of a shared proxy. Disabled by default. Set to `true` to enable. | | `list_metrics_page_limit` | Sets the limit of List Metrics API pages. When a custom namespace is specified in the query editor, the [List Metrics API](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_ListMetrics.html) populates the _Metrics_ field and _Dimension_ fields. The API is paginated and returns up to 500 results per page, and the data source also limits the number of pages to 500 by default. This setting customizes that limit. | ### Provision the data source You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning and available configuration options, refer to [Provision Grafana](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). **Using AWS SDK (default)**: ```yaml apiVersion: 1 datasources: - name: CloudWatch type: CloudWatch jsonData: authType: default defaultRegion: eu-west-2 ``` **Using credentials' profile name (non-default)**: ```yaml apiVersion: 1 datasources: - name: CloudWatch type: CloudWatch jsonData: authType: credentials defaultRegion: eu-west-2 customMetricsNamespaces: 'CWAgent,CustomNameSpace' profile: secondary ``` **Using `accessKey` and `secretKey`**: ```yaml apiVersion: 1 datasources: - name: CloudWatch type: CloudWatch jsonData: authType: keys defaultRegion: eu-west-2 secureJsonData: accessKey: '' secretKey: '' ``` **Using AWS SDK Default and ARN of IAM Role to Assume:** ```yaml apiVersion: 1 datasources: - name: CloudWatch type: CloudWatch jsonData: authType: default assumeRoleArn: arn:aws:iam::123456789012:root defaultRegion: eu-west-2 ``` ### Configure with Terraform You can configure the CloudWatch data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). For more information about provisioning resources with Terraform, refer to [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/). The following example uses the **Access and secret key** authentication method: ```hcl resource "grafana_data_source" "cloudwatch" { name = "CloudWatch" type = "cloudwatch" json_data_encoded = jsonencode({ authType = "keys" defaultRegion = "eu-west-2" }) secure_json_data_encoded = jsonencode({ accessKey = "" secretKey = "" }) } ``` The following example uses the **AWS SDK Default** authentication method with an assumed role: ```hcl resource "grafana_data_source" "cloudwatch" { name = "CloudWatch" type = "cloudwatch" json_data_encoded = jsonencode({ authType = "default" assumeRoleArn = "arn:aws:iam::123456789012:role/" defaultRegion = "eu-west-2" }) } ``` Replace the placeholders with your own values: - `` and ``: The AWS access key ID and secret access key for the **Access and secret key** method. - ``: The name of the IAM role to assume. For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). # Amazon CloudWatch query editor Grafana provides a query editor for the CloudWatch data source, which allows you to query, visualize, and alert on logs and metrics stored in Amazon CloudWatch. It is located on the [Explore](https://grafana.com/docs/grafana//explore/) page. For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/). ## Key concepts If you're new to CloudWatch, these terms are used throughout this documentation: | Term | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------- | | **Namespace** | A container for CloudWatch metrics, usually one per AWS service, such as `AWS/EC2` or `AWS/Lambda`. | | **Metric** | A time-ordered set of data points published to CloudWatch, such as `CPUUtilization`. | | **Dimension** | A name-value pair that identifies a specific resource for a metric, such as `InstanceId=i-1234567890abcdef0`. | | **Statistic** | An aggregation applied to metric data over a period, such as `Average`, `Sum`, or `Maximum`. | | **Period** | The length of time, in seconds, used to aggregate metric data points. | | **Metric math** | Expressions that combine or transform metrics, such as computing a rate or a sum across metrics. | | **Metrics Insights** | A SQL-like query engine for filtering and aggregating metrics in near real time. | | **Search expression** | A CloudWatch expression that matches multiple metrics at query time, often used with wildcards. | | **Logs Insights QL** | The CloudWatch Logs query language used to search and aggregate log data. | ## Choose a query editing mode The CloudWatch data source can query data from both CloudWatch metrics and CloudWatch Logs APIs, each with its own specialized query editor. - [CloudWatch Metrics](#query-cloudwatch-metrics) - [CloudWatch Logs](#query-cloudwatch-logs) Select the API to query using the drop-down to the right of the **Region** setting. ## CloudWatch Metrics query editor components The following are the components of the CloudWatch query editor. | **Setting** | **Description** | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Region** | Select an AWS region if it differs from the default. | | **Namespace** | The AWS service namespace. Examples: `AWS/EC2`, `AWS/Lambda`. | | **Metric name** | The name of the metric you want to visualize. Example: `CPUUtilization`. | | **Statistic** | Choose how to aggregate your data. Options are `Average`, `Maximum`, `Minimum`, `Sum`, `SampleCount`, and `IQM`. You can also enter a custom value, such as a percentile like `p99`. | | **Dimensions** | Select dimensions from the drop-down. Examples: `InstanceId`, `FunctionName`, `latency`. You can add several dimensions to your query. | | **Match exact** | _Optional_. When enabled, this option restricts query results to metrics that precisely match the specified dimensions and their values. All dimensions of the queried metric must be explicitly defined in the query to ensure an exact schema match. If disabled, the query will also return metrics that match the defined schema but possess additional dimensions. | | **ID** | _Optional_. Unique identifier required by the GetMetricData API for referencing queries in math expressions. Must start with a lowercase letter and can include letters, numbers, and underscores. If not specified, Grafana generates an ID using the pattern `query[refId]` (for example, `queryA`for the first query row). | | **Period** | The minimum time interval, in seconds, between data points. The default is `auto`. CloudWatch accepts periods of 1, 5, 10, 30, or any multiple of 60 seconds. When set to `auto` or left blank, Grafana calculates the period by dividing the time range in seconds by 2000, then rounding up to the next value among 60, 300, 900, 3600, 21600, and 86400. The available periods depend on the age of the data: because CloudWatch stores older metrics at a lower resolution under its [retention policy](https://aws.amazon.com/about-aws/whats-new/2016/11/cloudwatch-extends-metrics-retention-and-new-user-interface/), the shorter periods aren't available for older time ranges. | | **Label** | _Optional_. Add a customized time series legend name. The label field overrides the default metric legend name using CloudWatch dynamic labels. Time-based dynamic labels like ${MIN_MAX_TIME_RANGE} derive legend values from the current timezone in the time range picker. For the full list of label patterns and limitations, refer to [CloudWatch dynamic labels](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/graph-dynamic-labels.html). | ## Use Builder and Code modes Both the Metric Search and Metric Insights query types provide two editing modes. Use the **Builder** and **Code** toggle in the query editor to switch between them. ### Builder mode Builder mode lets you construct a query from drop-downs and form fields without writing any syntax: - For **Metric Search** queries, select the namespace, metric name, statistic, and dimensions. - For **Metric Insights** queries, select the namespace, metric name, filter, group, and order options using the [Metrics Insights keywords](#use-metrics-insights-keywords). Grafana constructs a SQL query based on your selections. ### Code mode Code mode lets you write your query directly in a code editor: - For **Metric Search** queries, write a metric math or `SEARCH` expression. - For **Metric Insights** queries, write a Metrics Insights SQL query. The code editor includes a built-in autocomplete feature that suggests keywords, aggregations, namespaces, metrics, labels, and label values. Suggestions appear after typing a space, comma, or dollar (`$`) character, or by pressing CTRL+Space. {{< admonition type="note" >}} Template variables in the code editor can interfere with autocompletion. {{< /admonition >}} To run the query, click **Run query** above the code editor. ## Query CloudWatch metrics You can create two types of queries in the CloudWatch query editor: - [Metric Search](#metric-search-queries), which helps you retrieve and filter available metrics. - [Metric Insights](#use-metric-insights-syntax), which uses the Metrics Insights feature to fetch time series data. The query type you use depends on how you want to interact with AWS metrics. Use the drop-down in the upper middle of the query editor to select which type you want to create. ### Metric Search queries Metric search queries help you discover and filter available metrics. These queries use wildcards and filters to find metrics without needing to know exact metric names. A valid metric query requires a specified namespace, metric name, and at least one statistic. Dimensions are optional, but if included, you must provide both a `key` and a `value`. The `Match Exact` option controls how dimension filtering is applied to metric queries. When you enable `match exact`, the query returns only metrics whose dimensions precisely match the specified criteria. This requires the following: - All dimensions present on the target metric must be explicitly specified. - Dimensions you don't want to filter by must use a wildcard (\*) filter. - The metric schema must match exactly as defined in the [CloudWatch metric schema](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/search-expression-syntax.html) documentation. When `Match Exact` is disabled, you can specify any subset of dimensions for filtering. The query returns metrics that: - Match the specified namespace and metric name. - Match all defined dimension filters. - May contain additional dimensions beyond those specified. This mode provides more flexible querying but may return metrics with unexpected additional dimensions. Because the data source retrieves metrics through the CloudWatch `GetMetricData` API, a single query returns data for up to 100 metrics matching your filter criteria, which is the per-request limit that AWS enforces. Enhance metric queries using template variables to create dynamic, reusable dashboards. #### Create dynamic queries with dimension wildcards Use the asterisk (`*`) wildcard for dimension values to create dynamic queries that automatically monitor changing sets of AWS resources. {{< figure src="/static/img/docs/cloudwatch/cloudwatch-dimension-wildcard-8.3.0.png" max-width="500px" caption="CloudWatch dimension wildcard" >}} The query returns the average CPU utilization for all EC2 instances in the default region. With `Match Exact` disabled and `InstanceId` using a wildcard, the query retrieves metrics for any EC2 instance regardless of additional dimensions. Auto-scaling events add new instances to the graph without manual instance ID tracking. As with other metric search queries, AWS returns up to 100 metrics per `GetMetricData` request. Click the [**Query inspector**](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/#navigate-the-query-editor) button and select **Meta Data** to see the search expression that's automatically built to support wildcards. To learn more about search expressions, refer to the [CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/search-expression-syntax.html). The search expression is defined by default in such a way that the queried metrics must match the defined dimension names exactly. This means that in the example, the query returns only metrics with exactly one dimension containing the name `InstanceId`. {{< figure src="/static/img/docs/cloudwatch/cloudwatch-meta-inspector-8.3.0.png" max-width="500px" class="docs-image--right" caption="CloudWatch Meta Inspector" >}} Disabling `Match Exact` includes metrics with additional dimensions and creates a search expression even without wildcards. Grafana searches for any metric matching at least the namespace, metric name, and all defined dimensions. #### Use multi-value template variables When defining dimension values based on multi-valued template variables, the data source uses a search expression to query for the matching metrics. This enables the use of multiple template variables in one query, and also lets you use template variables for queries that have the `Match Exact` option disabled. Search expressions are limited to 1,024 characters, so your query might fail if you have a long list of values. We recommend using the asterisk (`*`) wildcard instead of the `All` option to query all metrics that have any value for a certain dimension name. Multi-valued template variables are supported only for dimension values. Using multi-valued template variables for `Region`, `Namespace`, or `Metric Name` is not supported. #### Use metric math expressions Create new time series metrics using mathematical functions on CloudWatch metrics. This supports arithmetic operators, unary subtraction, and other functions. For available functions, refer to [AWS Metric Math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html). To apply arithmetic operations, assign a unique string ID to the raw metric, then reference this ID in the `Expression` field of the new metric. {{< admonition type="note" >}} If you use the expression field to reference another query, such as `queryA * 2`, you can't create an alert rule based on that query. {{< /admonition >}} ### Query metrics across AWS monitoring accounts When you select `Builder` mode within the Metric search editor, a new Account field is displayed. Use the `Account` field to specify which of the linked monitoring accounts to target for the given query. By default, the `All` option is specified, which will target all linked accounts. While in `Code` mode, you can specify any math expression. If the Monitoring account badge displays in the query editor header, all `SEARCH` expressions entered in this field will be cross-account by default and can query metrics from linked accounts. Note that while queries run cross-account, the autocomplete feature currently doesn't fetch cross-account resources, so you'll need to manually specify resource names when writing cross-account queries. You can limit the search to one or a set of accounts, as documented in the [AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html). ### Period macro If you use a CloudWatch [`SEARCH`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/search-expression-syntax.html) expression, consider using the `$__period_auto` macro rather than specifying a period explicitly. The `$__period_auto` macro will resolve to a [CloudWatch period](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html) that is suitable for the chosen time range. ### Deep-link Grafana panels to the CloudWatch console Left-clicking a time series in the panel displays a context menu with a link to `View in CloudWatch console`. Clicking the link opens a new tab that takes you to the CloudWatch console and displays all metrics for that query. If you're not logged in to the CloudWatch console, the link forwards you to the login page. The link provided is valid for any account but displays the expected metrics only if you're logged in to the account that corresponds to the selected data source in Grafana. {{< figure src="/media/docs/cloudwatch/cloudwatch-deep-link-v12.1.png" caption="CloudWatch deep linking" >}} This feature is not available for metrics based on [metric math expressions](#use-metric-math-expressions). ### Use Metric Insights syntax Metric Insights uses a dialect of SQL and this query syntax: ```sql SELECT FUNCTION(MetricName) FROM Namespace | SCHEMA(...) [ WHERE labelKey OPERATOR labelValue [AND|...]] [ GROUP BY labelKey [, ...]] [ ORDER BY FUNCTION() [DESC | ASC] ] [ LIMIT number] ``` For details about the Metrics Insights syntax, refer to the [AWS reference documentation](https://docs.aws.amazon.com/console/cloudwatch/metricsinsights-syntax). For information about Metrics Insights limits, refer to the [AWS feature documentation](https://docs.aws.amazon.com/console/cloudwatch/metricsinsights). You can also augment queries by using [template variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/). ### Use Metrics Insights keywords This table summarizes common Metrics Insights query keywords: | Keyword | Description | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `FUNCTION` | Required. Specifies the aggregate function to use, and also specifies the name of the metric to be queried. Valid values are AVG, COUNT, MAX, MIN, and SUM. | | `MetricName` | Required. For example, `CPUUtilization`. | | `FROM` | Required. Specifies the metric's source. You can specify either the metric namespace that contains the metric to be queried, or a SCHEMA table function. Namespace examples include `AWS/EC2`, `AWS/Lambda`. | | `SCHEMA` | Optional. Narrows the query results to only the metrics that are an exact match, or to metrics that do not match. | | `WHERE` | Optional. Filters the query results to only the metrics that match your specified expression. For example, `WHERE InstanceType != 'c3.4xlarge'`. | | `GROUP BY` | Optional. Groups the query results into multiple time series. For example, `GROUP BY ServiceName`. | | `ORDER BY` | Optional. Specifies the order in which time series are returned. Options are `ASC`, `DESC`. | | `LIMIT` | Optional. Limits the number of time series returned. | ## Query CloudWatch Logs The logs query editor helps you write CloudWatch Logs queries across a selected region and a set of log groups or Amazon CloudWatch Logs data sources. You can query CloudWatch Logs using three supported query language options: - **Logs Insights QL** - The AWS native query language specifically designed for CloudWatch Logs. It uses a SQL-like syntax with commands like `fields`, `filter`, `stats`, and `sort`. It's optimized for the CloudWatch log structure and offers built-in functions for parsing timestamps, extracting fields from JSON logs, and performing aggregations. - **OpenSearch SQL** - OpenSearch SQL is a query language that uses a SQL-like syntax for querying data in OpenSearch. It supports standard SQL queries and is designed for users familiar with SQL. - **OpenSearch PPL** - The OpenSearch query language is based on Elasticsearch's query DSL (Domain Specific Language). It uses a pipe-based syntax similar to Unix command-line tools or the Splunk search language, and supports complex boolean logic, range queries, wildcard matching, and full-text search capabilities. **Create a CloudWatch Logs query:** 1. Select a region. 1. Select **CloudWatch Logs** from the query type drop-down. 1. Use the Logs Mode selector to choose between Logs Insights and Log Anomalies queries. **Log Anomalies** [Anomaly detection](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/LogsAnomalyDetection.html) uses machine-learning and pattern recognition to establish baselines of typical log content. The Log Anomalies query editor fetches the list of anomalies detected in your CloudWatch service. In order to query log anomalies in the editor, a log anomaly detector must be created in the AWS CloudWatch console first. The log trend cell shows the number of occurrences of the pattern over the selected query time range. The table shows 50 log anomalies at a time. If you would like to narrow down the list, you can filter anomalies by their ARN and suppressed state. In addition to this, you can use the Logs Insights QL editor and the `anomaly` command together with the `patterns` command to define and display log anomalies in real time. See the [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/LogsAnomalyDetection-Insights.html) documentation for more info. **Logs Insights** The **Query language** drop-down is available only when **Logs Mode** is set to Logs Insights. 1. Select the query language you want to use in the **Query language** drop-down. 1. Specify the log groups, CloudWatch Logs data sources, or both that you want to query. 1. Use the main input area to write your logs query. Amazon CloudWatch only supports a subset of OpenSearch SQL and PPL commands. To find out more about the syntax supported, consult [Amazon CloudWatch Logs documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_Languages.html) ### Specify log groups to query To target log groups in your query, use one of the following methods: - **Log group name**: Select specific log groups. This is the default behavior. You can select up to 50 log groups. Click **Select log groups** to choose the target log groups for your query. When the **Monitoring account** badge appears in the query editor header, you can search and select log groups across multiple accounts. Use the **Accounts** filter to narrow results by account. For large lists, use prefix search to narrow the selection. The selector loads log groups in pages, so you can scroll to load more results when many log groups are available. - **Name prefix**: Select the **Name prefix** query scope and provide up to five log group name prefixes. The query will run against log groups that have names that start with the specified prefixes. Each prefix must be at least three characters and must not include `*`. This option is only available for **Logs Insights QL**. - **All log groups**: Select the **All log groups** query scope. This queries all log groups in the selected region. This option is only available for **Logs Insights QL**. You can also apply optional filters when specifying log groups with the **Name prefix** or **All log groups** query scopes: - **Class**: Filter by [log group class](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch_Logs_Log_Classes.html). Choose **Standard** or **Infrequent Access**. - **Accounts**: Filter by AWS account. This option is only available when the **Monitoring account** badge appears in the query editor header. You can select up to 20 accounts. ### Specify Amazon CloudWatch Logs data sources to query Amazon CloudWatch Logs data sources categorize logs by the service or application that generates them and by the type of log. They complement log groups and let you query logs without knowing individual log group names. For details about how Amazon CloudWatch categorizes logs, refer to [Data source discovery and management](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/data-source-discovery-management.html) in the AWS documentation. Before you use the data source selector, ensure that the IAM role or IAM user for the Grafana data source has the `logs:ListAggregateLogGroupSummaries` permission. For the complete IAM policy examples, refer to [IAM policy examples](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/configure/#iam-policy-examples). To query logs by CloudWatch Logs data source, follow these steps: 1. Click **Select data sources**. 1. Search for a data source name or type. 1. Select up to 10 data sources. 1. Click **Apply selection**. You can combine CloudWatch Logs data source selections with specific log groups or with the **Name prefix** and **All log groups** query scopes. When the **Monitoring account** badge appears, the selector includes data sources from the monitoring account and its linked source accounts. The CloudWatch data source automatically adds the selected CloudWatch Logs data sources to Logs Insights QL and OpenSearch PPL queries. {{< admonition type="note" >}} When you use **Logs Insights QL**, **OpenSearch PPL**, or **OpenSearch SQL**, you must specify a region and define the logs to query by selecting log groups, selecting CloudWatch Logs data sources, or choosing a supported log group query scope. In **OpenSearch SQL**, you can specify log groups in multiple ways. For details, refer to the [Query log groups and Amazon CloudWatch Logs data sources with OpenSearch SQL](#query-log-groups-with-opensearch-sql) section. {{< /admonition >}} Click **View in CloudWatch console** to interactively view, search, and analyze your log data in the CloudWatch Logs Insights console. If you're not logged in to the CloudWatch console, the link forwards you to the login page. ### Query log groups and Amazon CloudWatch Logs data sources with OpenSearch SQL {#query-log-groups-with-opensearch-sql} When querying log groups and Amazon CloudWatch Logs data sources with OpenSearch SQL, you can use the `$__source` macro to automatically reference the log groups and CloudWatch Logs data sources selected in the query editor. This is the recommended approach as it allows you to manage both types of selection through the UI. ```sql SELECT window.start, COUNT(*) AS exceptionCount FROM `$__source` WHERE `@message` LIKE '%Exception%' ``` The `$__source` macro expands to the proper syntax for the log groups and CloudWatch Logs data sources you've selected in the UI. Alternatively, you can manually specify a single log group directly in the `FROM` clause: ```sql SELECT window.start, COUNT(*) AS exceptionCount FROM `log_group` WHERE `@message` LIKE '%Exception%' ``` When querying multiple log groups you **must** use the `logGroups(logGroupIdentifier: [...])` syntax: ```sql SELECT window.start, COUNT(*) AS exceptionCount FROM `logGroups(logGroupIdentifier: ['LogGroup1', 'LogGroup2'])` WHERE `@message` LIKE '%Exception%' ``` To reference log groups in a monitoring account, use ARNs instead of LogGroup names. You can also write queries returning time series data by using the [`stats` command](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_Insights-Visualizing-Log-Data.html). When making `stats` queries in [Explore](https://grafana.com/docs/grafana//explore/), ensure you are in Metrics Explore mode. ### Create queries for alerting Alerting requires queries that return numeric data, which CloudWatch Logs supports. For example, you can enable alerts through the use of the `stats` command. The following is a valid query for alerting on messages that include the text "Exception": ``` filter @message like /Exception/ | stats count(*) as exceptionCount by bin(1h) | sort exceptionCount desc ``` {{< admonition type="note" >}} If you receive an error like `input data must be a wide series but got ...` when trying to alert on a query, make sure that your query returns valid numeric data that can be output to a Time series panel. {{< /admonition >}} For more information on Grafana alerts, refer to [Alerting](https://grafana.com/docs/grafana//alerting/). ## Common use cases The following scenarios show how to combine the query editor's features to solve real monitoring problems. - **Monitor an EC2 fleet:** Use a Metric Search query for `AWS/EC2` `CPUUtilization` with the `InstanceId` dimension set to a wildcard (`*`) and `Match Exact` disabled. The graph automatically includes new instances as your fleet scales. - **Track Lambda errors and invocations:** Query the `AWS/Lambda` namespace for the `Errors` and `Invocations` metrics, then add a metric math expression such as `errors / invocations * 100` to chart an error rate. - **Rank the busiest resources:** Use a Metrics Insights query with `ORDER BY` and `LIMIT` to return the top N resources by a metric, for example, the 10 EC2 instances with the highest average CPU utilization. - **Investigate application errors:** Use a CloudWatch Logs query with `filter` and `stats` to count errors over time, then correlate the spikes with metric panels on the same dashboard. - **Alert on log error rates:** Build a Logs query that returns numeric data with the `stats` command, then create an alert rule on the result. Refer to [Create queries for alerting](#create-queries-for-alerting). ## Cross-account observability The CloudWatch plugin monitors and troubleshoots applications that span multiple accounts within a region. Cross-account observability enables seamless searching, visualization, and analysis of metrics and logs across account boundaries. To enable cross-account observability, complete the following steps: 1. Go to the [Amazon CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html) and follow the instructions for enabling cross-account observability. 1. Add [two API actions](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/configure/#cross-account-observability-permissions) to the IAM policy attached to the role/user running the plugin. Cross-account querying is available in the plugin through the **Logs**, **Metric search**, and **Metric Insights** modes. After you have configured it, you'll see a **Monitoring account** badge in the query editor header. {{< figure src="/static/img/docs/cloudwatch/cloudwatch-monitoring-badge-9.3.0.png" max-width="1200px" caption="Monitoring account badge" >}} ## Query caching When you enable [query and resource caching](https://grafana.com/docs/grafana//administration/data-source-management/#query-and-resource-caching), Grafana temporarily stores the results of data source queries and resource requests. Query caching is available in CloudWatch Metrics in Grafana Cloud and Grafana Enterprise. It is not available in CloudWatch Logs Insights due to how query results are polled from AWS. # CloudWatch template variables Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana lists these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables. For an introduction to templating and template variables, refer to [Templating](https://grafana.com/docs/grafana//dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/). ## Use query variables You can specify these CloudWatch data source queries in the Variable edit view's **Query type** field. Use them to fill a variable's options list with values like `regions`, `namespaces`, `metric names`, and `dimension keys/values`. | Name | List returned | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | **Regions** | All AWS regions. | | **Namespaces** | All namespaces CloudWatch supports. | | **Metrics** | Metrics in the namespace. (Specify region or use "default" for custom metrics.) | | **Dimension Keys** | Dimension keys in the namespace. | | **Dimension Values** | Dimension values matching the specified `region`, `namespace`, `metric`, and `dimension key`. Use dimension `filters` for more specific results. | | **EBS Volume IDs** | Volume IDs matching the specified `region` and `instance ID`. | | **EC2 Instance Attributes** | Attributes matching the specified `region`, `attribute name`, and `filters`. | | **Resource ARNs** | ARNs matching the specified `region`, `resource type`, and `tags`. | | **Statistics** | All standard statistics. | | **Log Groups** | Log groups matching the specified `region` and optional log group `prefix`. The variable value is the log group ARN. | | **Accounts** | The IDs of linked monitoring accounts for the specified `region`. Available only when cross-account observability is enabled. | When cross-account observability is enabled, the **Metrics**, **Dimension Keys**, **Dimension Values**, and **Log Groups** query types also show an optional **Account** field. Use it to target a specific linked monitoring account, or leave it unset to query all linked accounts. For more information, refer to [Cross-account observability](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/query-editor/#cross-account-observability). For details on the available dimensions, refer to the [CloudWatch Metrics and Dimensions Reference](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CW_Support_For_AWS.html). For details about the metrics CloudWatch provides, refer to the [CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/CW_Support_For_AWS.html). ### Use variables in queries Use the Grafana variable syntax to include variables in queries. A query variable dynamically retrieves values from your data source using a query. For details, refer to the [variable syntax documentation](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/). CloudWatch metric queries support variables in the **Region**, **Namespace**, **Metric name**, dimension key, and dimension value fields. For example, if you create a variable named `region` with the **Regions** query type and a variable named `instance` with the **EC2 Instance Attributes** query type, you can reference them in a metric query by setting the query editor **Region** to `$region` and a dimension value to `$instance`. The dashboard then updates as you change the selected region and instance. ## Query EC2 instance attributes The **EC2 Instance Attributes** query type retrieves instance metadata using the AWS [`ec2:DescribeInstances`](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html) API. Configure it with the following fields in the variable editor: - **Region:** The AWS region to query. - **Attribute name:** The instance attribute to return for each matching instance. - **Filters:** _Optional._ Filters that limit which instances are returned. {{< admonition type="note" >}} The earlier `ec2_instance_attribute(region, attributeName, filters)` query syntax is still supported. Grafana automatically migrates existing variables that use it to the EC2 Instance Attributes query type. {{< /admonition >}} ### Filters Each filter has a key and a comma-separated list of matching values. Use the [filters supported by `ec2:DescribeInstances`](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html), such as instance properties. To filter by tag, use a key in the form `tag:`, for example `tag:Environment`. ### Select attributes A query returns one attribute per instance. You can select any attribute that has a single value and isn't an object or array, also known as a flat attribute. These attributes mirror the fields of the AWS EC2 [`Instance`](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_Instance.html) object, where you can find a definition of each one. Because the available attributes track the AWS API, the following list might not be exhaustive: - `AmiLaunchIndex` - `Architecture` - `ClientToken` - `EbsOptimized` - `EnaSupport` - `Hypervisor` - `ImageId` - `InstanceId` - `InstanceLifecycle` - `InstanceType` - `KernelId` - `KeyName` - `LaunchTime` - `Platform` - `PrivateDnsName` - `PrivateIpAddress` - `PublicDnsName` - `PublicIpAddress` - `RamdiskId` - `RootDeviceName` - `RootDeviceType` - `SourceDestCheck` - `SpotInstanceRequestId` - `SriovNetSupport` - `SubnetId` - `VirtualizationType` - `VpcId` Some attributes are objects rather than flat values. To select a nested value, use dot notation, for example `IamInstanceProfile.Arn`. To select a tag value, prepend the tag name with `Tags.`. For example, select the `Name` tag by using `Tags.Name`. # Troubleshoot Amazon CloudWatch data source issues This document provides solutions to common issues you may encounter when configuring or using the Amazon CloudWatch data source. For configuration instructions, refer to [Configure CloudWatch](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/configure/). {{< admonition type="note" >}} The data source health check validates both metrics and logs permissions. If your IAM policy only grants access to one of these (for example, metrics-only or logs-only), the health check displays a red status. However, the service you have permissions for is still usable—you can query metrics or logs based on whichever permissions are configured. {{< /admonition >}} ## Authentication errors These errors occur when AWS credentials are invalid, missing, or don't have the required permissions. Use the following table to match the error message you see to its most likely cause and the section that resolves it. | Error message | Likely cause | Refer to | | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `AccessDenied: User is not authorized to perform: cloudwatch:...` | The IAM policy is missing required CloudWatch permissions. | [Access Denied or Not authorized to perform this operation](#access-denied-or-not-authorized-to-perform-this-operation) | | `InvalidClientTokenId` or `UnrecognizedClientException: The security token included in the request is invalid` | Credentials are invalid, deactivated, or rotated, or a transient Grafana Cloud incident is in progress. | [InvalidClientTokenId or UnrecognizedClientException](#invalidclienttokenid-or-unrecognizedclientexception) | | `AccessDenied: ... not authorized to perform: sts:AssumeRole` | The role trust relationship, role ARN, or external ID is incorrect. | [Unable to assume role](#unable-to-assume-role) | | `AccessDenied: ... no VPC endpoint policy allows the cloudwatch:ListMetrics action` | A VPC endpoint policy or service control policy (SCP) blocks the action. | [Access blocked by a VPC endpoint policy or service control policy](#access-blocked-by-a-vpc-endpoint-policy-or-service-control-policy) | ### "Access Denied" or "Not authorized to perform this operation" **Symptoms:** - Save & test fails with "Access Denied" - Queries return authorization errors - Namespaces, metrics, or dimensions don't load **Possible causes and solutions:** | Cause | Solution | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | IAM policy missing required permissions | Attach the appropriate IAM policy to your user or role. For metrics, you need `cloudwatch:ListMetrics`, `cloudwatch:GetMetricData`, and related permissions. For logs, you need `logs:DescribeLogGroups`, `logs:StartQuery`, `logs:GetQueryResults`, and related permissions. Refer to [Configure CloudWatch](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/configure/) for complete policy examples. | | Incorrect access key or secret key | Verify the credentials in the AWS Console under **IAM** > **Users** > your user > **Security credentials**. Generate new credentials if necessary. | | Credentials have expired | For temporary credentials, generate new ones. For access keys, verify they haven't been deactivated or deleted. | | Wrong AWS region | Verify the default region in the data source configuration matches where your resources are located. | | IAM role lacks permissions in the queried region | AWS evaluates permissions per region. If your role only has CloudWatch access in some regions, querying another region returns `AccessDenied`, and the error might not name the region. Set the data source **Default Region** to a region where your role has permissions, and only query regions your IAM policy allows. | | Assume Role ARN is incorrect | Verify the role ARN format: `arn:aws:iam:::role/`. Check that the role exists in the AWS Console. | ### "InvalidClientTokenId" or "UnrecognizedClientException" **Symptoms:** - Save & test fails with `InvalidClientTokenId: The security token included in the request is invalid` - Queries fail with `UnrecognizedClientException: The security token included in the request is invalid` - The error appears suddenly without any configuration change **Possible causes and solutions:** | Cause | Solution | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Access key or secret key is invalid or mistyped | Verify the credentials in the AWS Console under **IAM** > **Users** > your user > **Security credentials**. Re-enter them in the data source configuration. | | Credentials were deactivated, deleted, or rotated | Generate new credentials in AWS and update the data source configuration. | | Temporary credentials expired | Generate new temporary credentials, or switch to a long-lived authentication method such as Assume Role. | | Transient Grafana Cloud incident | If multiple data sources fail at the same time with `InvalidClientTokenId` and your credentials are unchanged, check the [Grafana Cloud status page](https://status.grafana.com/) for an ongoing incident before you rotate credentials. | ### "Unable to assume role" **Symptoms:** - Authentication fails when using Assume Role ARN - Error references STS or AssumeRole, for example, `AccessDenied: User is not authorized to perform: sts:AssumeRole` **Solutions:** 1. Verify the trust relationship on the IAM role allows the Grafana credentials to assume it. 1. Check the trust policy includes the correct principal (the user or role running Grafana). 1. If you use the **Grafana Assume Role** authentication method, confirm the trust policy uses the exact Grafana AWS account ID and external ID shown in the instructions box on the data source's **Settings** tab. Entering a different account ID is a common mistake that causes the assume-role request to fail. 1. If using an external ID, ensure it matches exactly in both the role's trust policy and the Grafana data source configuration. 1. Verify the base credentials have the `sts:AssumeRole` permission. 1. Check that the role ARN is correct and the role exists. **Example trust policy:** ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::user/" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } ``` ### "Save & test" passes but queries return no data **Symptoms:** - **Save & test** succeeds, but panels and the query editor return no data - Namespaces, metrics, or dimensions don't load even though the connection test passed - The issue appears only when you use an **Assume Role ARN** **Possible causes and solutions:** | Cause | Solution | | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The connection test only validates the assume-role step | When you configure an **Assume Role ARN**, **Save & test** confirms that the primary credentials can perform `sts:AssumeRole`. It doesn't verify that the assumed role has CloudWatch or Logs permissions. Attach the required query permissions to the assumed role, not only to the primary credentials. | | The assumed role is missing CloudWatch permissions | Add the metrics and logs actions, such as `cloudwatch:ListMetrics`, `cloudwatch:GetMetricData`, `logs:DescribeLogGroups`, and `logs:StartQuery`, to the assumed role's policy. Refer to [Configure CloudWatch](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/configure/) for complete policy examples. | | Permissions are attached to the wrong identity | Confirm that the policy is attached to the role named in **Assume Role ARN**, not to the IAM user or role that provides the primary credentials. | ### Data source stops working after about an hour **Symptoms:** - The data source works initially, then queries fail after roughly an hour - Errors reference an expired or invalid security token - Saving the data source again, or restarting Grafana, temporarily restores access until it fails again **Possible causes and solutions:** | Cause | Solution | | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Temporary credentials expired and weren't refreshed | Methods that use temporary credentials, such as Assume Role or STS, issue short-lived tokens. If the data source uses static temporary credentials, switch to an authentication method that refreshes automatically, such as Assume Role with long-lived base credentials or an EKS/EC2 instance role. Refer to [AWS authentication](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/aws-authentication/). | ### Access blocked by a VPC endpoint policy or service control policy **Symptoms:** - Credentials are valid and the IAM policy grants the action, but requests still fail with `AccessDenied` - The error references a VPC endpoint policy, for example, `AccessDenied: ... no VPC endpoint policy allows the cloudwatch:ListMetrics action` - Access fails only from within a VPC or in a restricted account, such as a FedRAMP environment **Possible causes and solutions:** | Cause | Solution | | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A VPC endpoint policy restricts which actions are allowed | Update the policy on the CloudWatch, Logs, EC2, or STS VPC endpoints to allow the required actions, such as `cloudwatch:ListMetrics` and `cloudwatch:GetMetricData`. | | A service control policy (SCP) explicitly denies a required action | Review the service control policies applied to the account or organizational unit. An explicit `Deny` in an SCP overrides any `Allow` in an IAM policy, so the action must be permitted at both levels. | | Requests route through the wrong endpoint | Confirm that the data source region and any custom endpoint match the VPC endpoints that permit the required actions. | ### AWS SDK Default authentication not working **Symptoms:** - Data source test fails when using AWS SDK Default - Works locally but fails in production **Solutions:** 1. Verify AWS credentials are configured in the environment where Grafana runs. 1. Check for credentials in the default locations: - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) - Shared credentials file (`~/.aws/credentials`) - EC2 instance metadata (if running on EC2) - ECS task role (if running in ECS) - EKS service account (if running in EKS) 1. Ensure the Grafana process has permission to read the credentials file. 1. For EKS with IRSA, set the pod's security context to allow user `472` (`grafana`) to access the projected token. Refer to [AWS authentication](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/aws-authentication/) for details. ### Credentials file not found **Symptoms:** - Error indicates credentials file cannot be read - Authentication fails with "Credentials file" option **Solutions:** 1. Create the credentials file at `~/.aws/credentials` for the user running the `grafana-server` service. 1. Verify the file has correct permissions (`0644`). 1. If the file exists but isn't working, move it to `/usr/share/grafana/` and set permissions to `0644`. 1. Ensure the profile name in the data source configuration matches a profile in the credentials file. ## Connection errors These errors occur when Grafana cannot reach AWS CloudWatch endpoints. ### "Request timed out" or connection failures **Symptoms:** - Data source test times out - Queries fail with timeout errors - Intermittent connection issues **Solutions:** 1. Verify network connectivity from the Grafana server to AWS endpoints. 1. Check firewall rules allow outbound HTTPS (port 443) to AWS services. 1. If using a VPC, ensure proper NAT gateway or VPC endpoint configuration. 1. For Grafana Cloud connecting to private resources, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). 1. Check if the default region is correct—incorrect regions may cause longer timeouts. 1. Increase the timeout settings if queries involve large data volumes. ### Custom endpoint configuration issues **Symptoms:** - Connection fails when using a custom endpoint - Endpoint URL rejected **Solutions:** 1. Verify the endpoint URL format is correct. 1. Ensure the endpoint is accessible from the Grafana server. 1. Check that the endpoint supports the required AWS APIs. 1. For VPC endpoints, verify the endpoint policy allows the required actions. ## CloudWatch Metrics query errors These errors occur when querying CloudWatch Metrics. ### "No data" or empty results **Symptoms:** - Query executes without error but returns no data - Charts show "No data" message **Possible causes and solutions:** | Cause | Solution | | ------------------------------- | ---------------------------------------------------------------------------------------------------------- | | Time range doesn't contain data | Expand the dashboard time range. CloudWatch metrics have different retention periods based on resolution. | | Wrong namespace or metric name | Verify the namespace (for example, `AWS/EC2`) and metric name (for example, `CPUUtilization`) are correct. | | Incorrect dimensions | Ensure dimension names and values match your AWS resources exactly. | | Match Exact enabled incorrectly | When Match Exact is enabled, all dimensions must be specified. Try disabling it to see if metrics appear. | | Period too large | Reduce the period setting or set it to "auto" to ensure data points are returned for your time range. | | Custom metrics not configured | Add custom metric namespaces in the data source configuration under **Namespaces of Custom Metrics**. | ### "Metric not found" or metrics don't appear in drop-down **Symptoms:** - Expected metrics don't appear in the query editor - Metric drop-down is empty for a namespace - A specific metric is missing from the drop-down even though it exists in the CloudWatch console **Solutions:** 1. Verify the metric exists in the selected region. 1. For custom metrics, add the namespace to **Namespaces of Custom Metrics** in the data source configuration. 1. Check that the IAM policy includes `cloudwatch:ListMetrics` permission. 1. CloudWatch limits `ListMetrics` to 500 results per page. To retrieve more metrics, increase the `list_metrics_page_limit` setting in the [Grafana configuration file](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/configure/#configure-the-data-source-with-grafanaini). 1. Type the metric name directly into the **Metric name** field. For some AWS namespaces, the data source uses a predefined metric list, so newer or less common metrics might not appear in the drop-down even though you can still query them. If the metric exists in your CloudWatch console, enter its exact name to query it. 1. Use the Query Inspector to verify the API request and response. ### Dimension values not loading **Symptoms:** - Dimension value drop-down doesn't populate - Wildcard searches return no results **Solutions:** 1. Verify the IAM policy includes `cloudwatch:ListMetrics` permission. 1. Check that the namespace and metric are selected before dimension values can load. 1. For EC2 dimensions, ensure `ec2:DescribeTags` and `ec2:DescribeInstances` permissions are granted. 1. Dimension values require existing metrics—if no metrics match, no values appear. ### "Too many data points" or API throttling **Symptoms:** - Queries fail with throttling errors - Performance degrades with multiple panels **Solutions:** 1. Increase the period setting to reduce the number of data points. 1. Reduce the time range of your queries. 1. Use fewer dimensions or wildcard queries per panel. 1. Request a quota increase for `GetMetricData` requests per second in the [AWS Service Quotas console](https://console.aws.amazon.com/servicequotas/). 1. Enable query caching in Grafana to reduce API calls. This section covers throttling at the query level. For account-wide rate limits and quota increases, refer to [API throttling errors](#api-throttling-errors). ### Metric math expression errors **Symptoms:** - Expression returns errors - Referenced metrics not found **Solutions:** 1. Verify each referenced metric has a unique ID set. 1. Check that metric IDs start with a lowercase letter and contain only letters, numbers, and underscores. 1. Ensure all referenced metrics are in the same query. 1. Verify the expression syntax follows [AWS Metric Math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) documentation. 1. Metric math expressions can't be used with Grafana alerting if they reference other query rows. ## CloudWatch Logs query errors These errors occur when querying CloudWatch Logs. ### "Query failed" or logs don't appear **Symptoms:** - Log queries return errors - No log data is displayed **Solutions:** 1. Verify log group names are correct and exist in the selected region. 1. Check the IAM policy includes `logs:StartQuery`, `logs:GetQueryResults`, and `logs:DescribeLogGroups` permissions. 1. Ensure the time range contains log data. 1. Verify the query syntax is valid. For CloudWatch Logs Insights QL, test the query in the AWS Console. 1. Select the correct query language (Logs Insights QL, OpenSearch PPL, or OpenSearch SQL) based on your query syntax. ### Log query timeout **Symptoms:** - Query runs for a long time then fails - Error mentions timeout **Solutions:** 1. Increase the **Query Result Timeout** setting in the data source configuration (default is 30 minutes). 1. Narrow the time range to reduce the amount of data scanned. 1. Add filters to your query to limit results. 1. Break complex queries into smaller, more focused queries. 1. For alerting, the timeout defined in the [Grafana configuration file](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#unified_alerting) takes precedence. ### Log groups not appearing in selector **Symptoms:** - Log group selector is empty - Can't find expected log groups **Solutions:** 1. Verify the IAM policy includes `logs:DescribeLogGroups` permission. 1. Check that log groups exist in the selected region. 1. For cross-account observability, ensure proper IAM permissions for `oam:ListSinks` and `oam:ListAttachedLinks`. 1. Use prefix search to filter log groups if you have many groups. 1. Verify the selected account (for cross-account) contains the expected log groups. ### OpenSearch SQL query errors **Symptoms:** - OpenSearch SQL queries fail - Syntax errors with SQL queries **Solutions:** 1. Specify the log group identifier or ARN in the `FROM` clause: ```sql SELECT * FROM `log_group_name` WHERE `@message` LIKE '%error%' ``` 1. For multiple log groups, use the `logGroups` function: ```sql SELECT * FROM `logGroups(logGroupIdentifier: ['LogGroup1', 'LogGroup2'])` ``` 1. Amazon CloudWatch supports only a subset of OpenSearch SQL commands. Refer to the [CloudWatch Logs documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_Languages.html) for supported syntax. ## Template variable errors These errors occur when using template variables with the CloudWatch data source. ### Variables return no values **Symptoms:** - Variable drop-down is empty - Dashboard fails to load with variable errors **Solutions:** 1. Verify the data source connection is working. 1. Check that the IAM policy includes permissions for the variable query type: - **Regions:** No additional permissions needed. - **Namespaces:** No additional permissions needed. - **Metrics:** Requires `cloudwatch:ListMetrics`. - **Dimension Values:** Requires `cloudwatch:ListMetrics`. - **EC2 Instance Attributes:** Requires `ec2:DescribeInstances`. - **EBS Volume IDs:** Requires `ec2:DescribeVolumes`. - **Resource ARNs:** Requires `tag:GetResources`. - **Log Groups:** Requires `logs:DescribeLogGroups`. 1. For dependent variables, ensure parent variables have valid selections. 1. Verify the region is set correctly (use "default" for the data source's default region). For more information on template variables, refer to [CloudWatch template variables](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/template-variables/). ### Multi-value template variables cause query failures **Symptoms:** - Queries fail when selecting multiple dimension values - Error about search expression limits **Solutions:** 1. Search expressions are limited to 1,024 characters. Reduce the number of selected values. 1. Use the asterisk (`*`) wildcard instead of selecting "All" to query all metrics for a dimension. 1. Multi-valued template variables are only supported for dimension values—not for Region, Namespace, or Metric Name. ## Cross-account observability errors These errors occur when using CloudWatch cross-account observability features. ### Cross-account queries fail **Symptoms:** - Can't query metrics or logs from linked accounts - Monitoring account badge doesn't appear **Solutions:** 1. Verify cross-account observability is configured in the AWS CloudWatch console. 1. Add the required IAM permissions: ```json { "Version": "2012-10-17", "Statement": [ { "Action": ["oam:ListSinks", "oam:ListAttachedLinks"], "Effect": "Allow", "Resource": "*" } ] } ``` 1. Check that the monitoring account and source accounts are properly linked in AWS. 1. Cross-account observability works within a single region—verify all accounts are in the same region. 1. EC2 Instance Attributes can't be queried across accounts because they use the EC2 API, not the CloudWatch API. ## Quota and pricing issues These issues relate to AWS service quotas and cost management. ### API throttling errors **Symptoms:** - "Rate exceeded" errors - Dashboard panels intermittently fail to load **Solutions:** 1. Reduce the frequency of dashboard refreshes. 1. Increase the period setting to reduce `GetMetricData` requests. 1. Enable query caching in Grafana (available in Grafana Enterprise and Grafana Cloud). 1. Request a quota increase in the [AWS Service Quotas console](https://console.aws.amazon.com/servicequotas/). 1. Consider consolidating similar queries using metric math. If throttling happens because a single query returns too many data points, refer to ["Too many data points" or API throttling](#too-many-data-points-or-api-throttling). ### Unexpectedly high CloudWatch costs **Symptoms:** - AWS CloudWatch costs are higher than expected - Frequent API calls from Grafana **Solutions:** 1. The `GetMetricData` API doesn't qualify for the CloudWatch API free tier. 1. Reduce dashboard auto-refresh frequency. 1. Increase the period setting to reduce data points returned. 1. Use query caching to reduce repeated API calls. 1. Review variable query settings—set variable refresh to "On dashboard load" instead of "On time range change." 1. Avoid using wildcards in dimensions when possible, as they generate search expressions with multiple API calls. ## Other common issues These issues don't produce specific error messages but are commonly encountered. ### Custom metrics don't appear **Symptoms:** - Custom metrics from applications or agents don't show in the namespace drop-down - Only standard AWS namespaces are visible **Solutions:** 1. Add your custom metric namespace to the **Namespaces of Custom Metrics** field in the data source configuration. 1. Separate multiple namespaces with commas (for example, `CWAgent,CustomNamespace`). 1. Verify custom metrics have been published to CloudWatch in the selected region. ### Pre-configured dashboards not working **Symptoms:** - Imported dashboards show no data - Dashboard variables don't load **Solutions:** 1. Verify the data source name in the dashboard matches your CloudWatch data source. 1. Check that the dashboard's AWS region setting matches where your resources are located. 1. Ensure the IAM policy grants access to the required services (EC2, Lambda, RDS, etc.). 1. Verify resources exist and are emitting metrics in the selected region. ### Application Signals trace links not appearing **Symptoms:** - Log entries don't show Application Signals trace links - `@xrayTraceId` field not appearing **Solutions:** 1. Verify an Application Signals data source is configured and linked in the CloudWatch data source settings. 1. Ensure your logs contain the `@xrayTraceId` field. 1. Update log queries to include `@xrayTraceId` in the fields, for example: `fields @message, @xrayTraceId`. 1. Configure your application to log Application Signals trace IDs. Refer to the [AWS Application Signals documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html). ## Enable debug logging To capture detailed error information for troubleshooting: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location). 1. Look for CloudWatch-specific entries that include request and response details. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you've tried the solutions above and still encounter issues: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review the [CloudWatch plugin GitHub issues](https://github.com/grafana/grafana/issues) for known bugs. 1. Consult the [AWS CloudWatch documentation](https://docs.aws.amazon.com/cloudwatch/) for service-specific guidance. 1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user. 1. When reporting issues, include: - Grafana version - AWS region - Error messages (redact sensitive information) - Steps to reproduce - Query configuration (redact credentials and account IDs) # Azure Monitor alerting The Azure Monitor data source supports [Grafana Alerting](https://grafana.com/docs/grafana//alerting/) and [Grafana-managed recording rules](https://grafana.com/docs/grafana//alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/), allowing you to create alert rules based on Azure metrics, logs, traces, and resource data. You can monitor your Azure environment and receive notifications when specific conditions are met. ## Before you begin - Ensure you have the appropriate permissions to create alert rules in Grafana. - Verify your Azure Monitor data source is configured and working correctly. - Familiarize yourself with [Grafana Alerting concepts](https://grafana.com/docs/grafana//alerting/fundamentals/). - **Important**: Verify your data source uses a supported authentication method. Refer to [Authentication requirements](#authentication-requirements). ## Supported query types for alerting All Azure Monitor query types support alerting and recording rules: | Query type | Use case | Notes | | -------------------- | -------------------------------------------------- | -------------------------------------------------------- | | Metrics | Threshold-based alerts on Azure resource metrics | Best suited for alerting; returns time-series data | | Logs | Alert on log patterns, error counts, or thresholds | Use KQL to aggregate data into numeric values | | Azure Resource Graph | Alert on resource state or configuration changes | Use count aggregations to return numeric data | | Traces | Alert on trace data and application performance | Use aggregations to return numeric values for evaluation | {{< admonition type="note" >}} Alert queries must return numeric data that Grafana can evaluate against a threshold. Queries that return only text or non-numeric data cannot be used directly for alerting. {{< /admonition >}} ## Authentication requirements Alerting and recording rules run as background processes without a user context. This means they require service-level authentication and don't work with all authentication methods. | Authentication method | Supported | | ------------------------------------- | ------------------------------------- | | App Registration (client secret) | ✓ | | App Registration (client certificate) | ✓ | | Managed Identity | ✓ | | Workload Identity | ✓ | | Current User | ✓ (with fallback service credentials) | {{< admonition type="note" >}} If you use **Current User** authentication, you must configure **fallback service credentials** for alerting and recording rules to function. User credentials aren't available for background operations, so Grafana uses the fallback credentials instead. Refer to [configure the data source](https://grafana.com/docs/grafana//datasources/azure-monitor/configure/) for details on setting up fallback credentials. {{< /admonition >}} ## Create an alert rule To create an alert rule using Azure Monitor data: 1. Go to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for your alert rule. 1. In the **Define query and alert condition** section: - Select your Azure Monitor data source. - Configure your query (for example, a Metrics query for CPU usage or a Logs query using KQL). - Add a **Reduce** expression if your query returns multiple series. - Add a **Threshold** expression to define the alert condition. 1. Configure the **Set evaluation behavior**: - Select or create a folder and evaluation group. - Set the evaluation interval (how often the alert is checked). - Set the pending period (how long the condition must be true before firing). 1. Add labels and annotations to provide context for notifications. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example: VM CPU usage alert This example creates an alert that fires when virtual machine CPU usage exceeds 80%: 1. Create a new alert rule. 1. Configure the query: - **Service**: Metrics - **Resource**: Select your virtual machine - **Metric namespace**: `Microsoft.Compute/virtualMachines` - **Metric**: `Percentage CPU` - **Aggregation**: `Average` 1. Add expressions: - **Reduce**: Last (to get the most recent data point) - **Threshold**: Is above 80 1. Set evaluation to run every 1 minute with a 5-minute pending period. 1. Save the rule. ## Example: Error log count alert This example alerts when error logs exceed a threshold using a KQL query: 1. Create a new alert rule. 1. Configure the query: - **Service**: Logs - **Resource**: Select your Log Analytics workspace - **Query**: ```kusto AppExceptions | where TimeGenerated > ago(5m) | summarize ErrorCount = count() by bin(TimeGenerated, 1m) ``` 1. Add expressions: - **Reduce**: Max (to get the highest count in the period) - **Threshold**: Is above 10 1. Set evaluation to run every 5 minutes. 1. Save the rule. ## Example: Resource count alert This example alerts when the number of running virtual machines drops below a threshold using Azure Resource Graph: 1. Create a new alert rule. 1. Configure the query: - **Service**: Azure Resource Graph - **Subscriptions**: Select your subscriptions - **Query**: ```kusto resources | where type == "microsoft.compute/virtualmachines" | where properties.extended.instanceView.powerState.displayStatus == "VM running" | summarize RunningVMs = count() ``` 1. Add expressions: - **Reduce**: Last - **Threshold**: Is below 3 1. Set evaluation to run every 5 minutes. 1. Save the rule. ## Best practices Follow these recommendations to create reliable and efficient alerts with Azure Monitor data. ### Use appropriate query intervals - Set the alert evaluation interval to be greater than or equal to the minimum data resolution from Azure Monitor. - Azure Monitor Metrics typically have 1-minute granularity at minimum. - Avoid very short intervals (less than 1 minute) as they may cause evaluation timeouts or miss data points. ### Reduce multiple series When your Azure Monitor query returns multiple time series (for example, CPU usage across multiple VMs), use the **Reduce** expression to aggregate them: - **Last**: Use the most recent value - **Mean**: Average across all series - **Max/Min**: Use the highest or lowest value - **Sum**: Total across all series ### Optimize Log Analytics queries For Logs queries used in alerting: - Use `summarize` to aggregate data into numeric values. - Include appropriate time filters using `ago()` or `TimeGenerated`. - Avoid returning large result sets; aggregate data in the query. - Test queries in Explore before using them in alert rules. ### Handle no data conditions Configure what happens when no data is returned: 1. In the alert rule, find **Configure no data and error handling**. 1. Choose an appropriate action: - **No Data**: Keep the alert in its current state - **Alerting**: Treat no data as an alert condition - **OK**: Treat no data as a healthy state ### Test queries before alerting Always verify your query returns expected data before creating an alert: 1. Go to **Explore**. 1. Select your Azure Monitor data source. 1. Run the query you plan to use for alerting. 1. Confirm the data format and values are correct. 1. Verify the query returns numeric data suitable for threshold evaluation. ## Troubleshooting If your Azure Monitor alerts aren't working as expected, use the following sections to diagnose and resolve common issues. ### Alerts not firing - Verify the data source uses a supported authentication method. If using Current User authentication, ensure fallback service credentials are configured. - Check that the query returns numeric data in Explore. - Ensure the evaluation interval allows enough time for data to be available. - Review the alert rule's health and any error messages in the Alerting UI. ### Authentication errors in alert evaluation If you see authentication errors when alerts evaluate: - Confirm the data source is configured with App Registration, Managed Identity, Workload Identity, or Current User with fallback service credentials. - If using App Registration, verify the client secret hasn't expired. - If using Current User, verify that fallback service credentials are configured and valid. - Check that the service principal has appropriate permissions on Azure resources. ### Query timeout errors - **Increase the data source timeout.** The default timeout is 30 seconds, but Azure Log Analytics queries can take up to 3 minutes. Open the Azure Monitor data source settings, scroll to **Misc**, and set **Timeout** to `300`. Refer to [Query timeout](https://grafana.com/docs/grafana//datasources/azure-monitor/configure/#query-timeout). - Simplify complex KQL queries. - Reduce the time range in Log Analytics queries. - Add more specific filters to narrow result sets. - Ensure the alert evaluation interval is long enough to accommodate the query duration. For additional troubleshooting help, refer to [Troubleshoot Azure Monitor](https://grafana.com/docs/grafana//datasources/azure-monitor/troubleshooting/). ## Additional resources - [Grafana Alerting documentation](https://grafana.com/docs/grafana//alerting/) - [Create alert rules](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/) - [Azure Monitor query editor](https://grafana.com/docs/grafana//datasources/azure-monitor/query-editor/) - [Grafana-managed recording rules](https://grafana.com/docs/grafana//alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/) # Azure Monitor annotations [Annotations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/) overlay rich event information on top of graphs. You can use Azure Monitor Log Analytics queries to create annotations that mark important events, deployments, alerts, or other significant occurrences on your dashboards. ## Before you begin - Ensure you have configured the Azure Monitor data source. - You need access to a Log Analytics workspace containing the data you want to use for annotations. - Annotations use Log Analytics (KQL) queries only. Metrics, Traces, and Azure Resource Graph queries are not supported for annotations. ## Create an annotation query To add an Azure Monitor annotation to a dashboard: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Annotation query**. 1. Enter a name for the annotation query. 1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox. 1. Select a color for the annotation event markers. 1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed. 1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed. 1. Click **Open query editor** to open the **Annotation Query** dialog box. 1. Select your **Azure Monitor** data source from the **Data source** drop-down list. 1. Choose the **Logs** service. 1. Select a **Resource** (Log Analytics workspace or Application Insights resource). 1. Write a KQL query that returns the annotation data. 1. Configure the annotation query and field mappings. 1. (Optional) Click **Test annotation query** to ensure that the query is working properly. 1. Click **Close** when you've completed the query setup. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. ## Query requirements Your KQL query should return columns that Grafana can use to create annotations: | Column | Required | Description | | ------------------ | ----------- | ------------------------------------------------------------------------------------------------ | | `TimeGenerated` | Yes | The timestamp for the annotation. Grafana uses this to position the annotation on the time axis. | | `Text` | Recommended | The annotation text displayed when you hover over or click the annotation. | | Additional columns | Optional | Any other columns returned become annotation tags. | {{< admonition type="note" >}} Always include a time filter in your query to limit results to the dashboard's time range. Use the `$__timeFilter()` macro. {{< /admonition >}} ## Annotation query examples The following examples demonstrate common annotation use cases. ### Azure Activity Log events Display Azure Activity Log events such as resource modifications, deployments, and administrative actions: ```kusto AzureActivity | where $__timeFilter(TimeGenerated) | where Level == "Error" or Level == "Warning" or CategoryValue == "Administrative" | project TimeGenerated, Text=OperationNameValue, Level, ResourceGroup, Caller | order by TimeGenerated desc | take 100 ``` ### Deployment events Show deployment-related activity: ```kusto AzureActivity | where $__timeFilter(TimeGenerated) | where OperationNameValue contains "deployments" | project TimeGenerated, Text=strcat("Deployment: ", OperationNameValue), Status=ActivityStatusValue, ResourceGroup | order by TimeGenerated desc ``` ### Application Insights exceptions Mark application exceptions as annotations: ```kusto AppExceptions | where $__timeFilter(TimeGenerated) | project TimeGenerated, Text=strcat(ProblemId, ": ", OuterMessage), SeverityLevel, AppRoleName | order by TimeGenerated desc | take 50 ``` ### Custom events from Application Insights Display custom events logged by your application: ```kusto AppEvents | where $__timeFilter(TimeGenerated) | where Name == "DeploymentStarted" or Name == "DeploymentCompleted" | project TimeGenerated, Text=Name, AppRoleName | order by TimeGenerated desc ``` ### Security alerts Show security-related alerts: ```kusto SecurityAlert | where $__timeFilter(TimeGenerated) | project TimeGenerated, Text=AlertName, Severity=AlertSeverity, Description | order by TimeGenerated desc | take 50 ``` ### Resource health events Display resource health status changes: ```kusto AzureActivity | where $__timeFilter(TimeGenerated) | where CategoryValue == "ResourceHealth" | project TimeGenerated, Text=OperationNameValue, Status=ActivityStatusValue, ResourceId | order by TimeGenerated desc ``` ### VM start and stop events Mark virtual machine state changes: ```kusto AzureActivity | where $__timeFilter(TimeGenerated) | where OperationNameValue has_any ("start", "deallocate", "restart") | where ResourceProviderValue == "MICROSOFT.COMPUTE" | project TimeGenerated, Text=OperationNameValue, VM=Resource, Status=ActivityStatusValue | order by TimeGenerated desc ``` ### Autoscale events Show autoscale operations: ```kusto AzureActivity | where $__timeFilter(TimeGenerated) | where OperationNameValue contains "autoscale" | project TimeGenerated, Text=strcat("Autoscale: ", OperationNameValue), Status=ActivityStatusValue, ResourceGroup | order by TimeGenerated desc ``` ## Customize annotation appearance After creating an annotation query, you can customize its appearance: | Setting | Description | | ------------- | -------------------------------------------------------------------------------------------------------- | | **Color** | Choose a color for the annotation markers. Use different colors to distinguish between annotation types. | | **Show in** | Select which panels display the annotations. | | **Filter by** | Add filters to limit when annotations appear. | ## Best practices Follow these recommendations when creating annotations: 1. **Limit results**: Always use `take` or `limit` to restrict the number of annotations. Too many annotations can clutter your dashboard and impact performance. 2. **Use time filters**: Include `$__timeFilter()` to ensure queries only return data within the dashboard's time range. 3. **Create meaningful text**: Use `strcat()` or `project` to create descriptive annotation text that provides context at a glance. 4. **Add relevant tags**: Include columns like `ResourceGroup`, `Severity`, or `Status` that become clickable tags for filtering. 5. **Use descriptive names**: Name your annotations clearly (for example, "Production Deployments", "Critical Alerts") so dashboard users understand what they represent. ## Troubleshoot annotations If annotations aren't appearing as expected, try the following solutions. ### Annotations don't appear - Verify the query returns data in the selected time range. - Check that the query includes a `TimeGenerated` column. - Test the query in the Azure Portal Log Analytics query editor. - Ensure the annotation is enabled (toggle is on). ### Too many annotations - Add more specific filters to your query. - Use `take` to limit results. - Narrow the time range. ### Annotations appear at wrong times - Verify the `TimeGenerated` column contains the correct timestamp. - Check your dashboard's timezone settings. # Configure the Azure Monitor data source This document explains how to configure the Azure Monitor data source and the available configuration options. For general information about data sources, refer to [Grafana data sources](https://grafana.com/docs/grafana//datasources/) and [Data source management](https://grafana.com/docs/grafana//administration/data-source-management/). ## Before you begin Before configuring the Azure Monitor data source, ensure you have the following: - **Grafana permissions:** You must have the `Organization administrator` role to configure data sources. Organization administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system or [using Terraform](#configure-with-terraform). - **Azure prerequisites:** Depending on your chosen authentication method, you may need: - A Microsoft Entra ID (formerly Azure AD) app registration with a service principal (for App Registration authentication) - A Managed Identity enabled on your Azure VM or App Service (for Managed Identity authentication) - Workload identity configured in your Kubernetes cluster (for Workload Identity authentication) - Microsoft Entra ID authentication configured for Grafana login (for Current User authentication) {{< admonition type="note" >}} **Grafana Cloud users:** Managed Identity and Workload Identity authentication methods are not available in Grafana Cloud because they require Grafana to run on your Azure infrastructure. Use **App Registration** authentication instead. {{< /admonition >}} - **Azure RBAC permissions:** The identity used to authenticate must have the `Reader` role on the Azure subscription containing the resources you want to monitor. For Log Analytics queries, the identity also needs appropriate permissions on the Log Analytics workspaces to be queried. Refer to the [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). {{< admonition type="note" >}} The Azure Monitor data source plugin is built into Grafana. No additional installation is required. {{< /admonition >}} ## Add the data source To add the Azure Monitor data source: 1. Click **Connections** in the left-side menu. 1. Click **Add new connection**. 1. Type `Azure Monitor` in the search bar. 1. Select **Azure Monitor**. 1. Click **Add new data source** in the upper right. You're taken to the **Settings** tab where you can configure the data source. ## Choose an authentication method The Azure Monitor data source supports five authentication methods. Choose based on where Grafana is hosted, your security requirements, and whether you need alerting: | Authentication method | Best for | Grafana Cloud | Supports alerting | Server configuration required | | ----------------------------------------- | ------------------------------------------ | ------------- | ----------------- | ----------------------------- | | **App Registration (client secret)** | Any Grafana deployment | ✓ | ✓ | No | | **App Registration (client certificate)** | Any Grafana deployment | ✓ | ✓ | No | | **Managed Identity** | Grafana hosted in Azure (VMs, App Service) | ✗ | ✓ | Yes | | **Workload Identity** | Grafana in Kubernetes (AKS) | ✗ | ✓ | Yes | | **Current User** | User-level access control | ✓ | Partial | Yes | {{< admonition type="note" >}} **Current User** authentication doesn't support background operations like alerting, reporting, and recording rules. To use alerting with Current User, you must configure **fallback service credentials**. Alerts then run under the fallback credential's permissions. Refer to [Limitations and fallback credentials](#limitations-and-fallback-credentials) for details. {{< /admonition >}} ## Configure authentication Select one of the following authentication methods and complete the configuration. ### App Registration (client secret) Use a Microsoft Entra ID app registration (service principal) to authenticate. This method works with any Grafana deployment. #### App Registration prerequisites 1. Create an app registration in Microsoft Entra ID. Refer to the [Azure documentation for creating a service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#get-tenant-and-app-id-values-for-signing-in). 1. Create a client secret for the app registration. Refer to the [Azure documentation for creating a client secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret). 1. Assign the `Reader` role to the app registration on the subscription or resources you want to monitor. Refer to the [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). #### App Registration UI configuration | Setting | Description | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Authentication** | Select **App Registration**. | | **Azure Cloud** | The Azure environment to connect to. Select **Azure** for the public cloud, or choose Azure Government or Azure China for national clouds. | | **Directory (tenant) ID** | The GUID that identifies your Microsoft Entra ID tenant. | | **Application (client) ID** | The GUID for the app registration you created. | | **Client secret** | The secret key for the app registration. Keep this secure and rotate periodically. | | **Default Subscription** | Click **Load Subscriptions** to populate available subscriptions, then select your default. | #### Provision App Registration with YAML ```yaml apiVersion: 1 datasources: - name: Azure Monitor type: grafana-azure-monitor-datasource access: proxy jsonData: azureAuthType: clientsecret cloudName: azuremonitor # See supported cloud names below tenantId: clientId: subscriptionId: # Optional, default subscription secureJsonData: clientSecret: version: 1 ``` ### App Registration (client certificate) Use a Microsoft Entra ID app registration with a certificate. #### App Registration (client certificate) prerequisites 1. Create an app registration in Microsoft Entra ID. Refer to the [Azure documentation for creating a service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#get-tenant-and-app-id-values-for-signing-in). 1. Create or retrieve an existing certificate (see examples below for creating self-signed PEM or PFX certificates). 1. Upload a certificate to the app registration in PEM or PFX format. Refer to the [Azure documentation here](https://learn.microsoft.com/en-us/entra/identity-platform/how-to-add-credentials?source=recommendations&tabs=certificate). 1. Assign the `Reader` role to the app registration on the subscription or resources you want to monitor. Refer to the [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). #### Add the certificate to the app registration Add the public certificate to your app registration in the Azure portal: 1. Open the app registration in Azure. 1. Select **Certificates & secrets**. 1. Select **Certificates**. 1. Click **Upload certificate**. 1. Upload the certificate file that contains the public certificate (`.cer`, `.crt`, or `.pem`). 1. Save your changes. The uploaded certificate is used by Microsoft Entra ID to validate the client assertion signature from Grafana. #### Certificate formats The Azure Monitor data source supports the following certificate formats: - **PEM:** Provide both the certificate and private key in PEM text format. - **PFX:** Provide a base64-encoded PFX payload in the **Client Certificate** field, and provide the certificate password. #### Create self-signed certificates The following examples are for test and lab environments. For production, use certificates from your internal or public certificate authority. Create a self-signed PEM certificate and private key with OpenSSL: ```sh openssl genrsa -out key.pem 2048 openssl req -new -sha256 -key key.pem -out csr.csr openssl req -x509 -sha256 -days 365 -key key.pem -in csr.csr -out certificate.pem ``` In this example: - `certificate.pem` is uploaded to **Certificates & secrets** in Azure. - `certificate.pem` is the client certificate added to Grafana. - `key.pem` is the private key added to Grafana. Create an encrypted, base64 encoded PFX certificate: ```sh openssl genrsa -out key.pem 2048 openssl req -new -sha256 -key key.pem -out csr.csr openssl req -x509 -sha256 -days 365 -key key.pem -in csr.csr -out certificate.pem openssl pkcs12 -export -inkey key.pem -in certificate.pem -out certificate.pfx openssl base64 -in certificate.pfx -out certificate.b64 ``` Ensure you make note of the password you use to encrypt the certificate. In this example: - `certificate.pem` is uploaded to **Certificates & secrets** in Azure. - `certificate.b64` is the encrypted, encoded file added to Grafana (in the client certificate field). #### App Registration (client certificate) UI configuration | Setting | Description | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Authentication** | Select **App Registration (Client Certificate)**. | | **Azure Cloud** | The Azure environment to connect to. Select **Azure** for the public cloud, or choose Azure Government or Azure China for national clouds. | | **Directory (tenant) ID** | The GUID that identifies your Microsoft Entra ID tenant. | | **Application (client) ID** | The GUID for the app registration you created. | | **Format** | Select **PEM** or **PFX**. | | **Client Certificate** | Paste the certificate content (PEM text or base64-encoded PFX payload). | | **Private Key** | Required only for PEM. Paste the private key PEM. | | **Certificate Password** | Required for encrypted PFX files. | | **Default Subscription** | Click **Load Subscriptions** to populate available subscriptions, then select your default. | #### Provision App Registration (client certificate) with YAML To provision the Azure Monitor data source with client certificate authentication, use the following configuration: ```yaml apiVersion: 1 datasources: - name: Azure Monitor type: grafana-azure-monitor-datasource access: proxy jsonData: azureAuthType: clientcertificate cloudName: azuremonitor # See supported cloud names below tenantId: clientId: certificateFormat: pem # Use pem or pfx subscriptionId: # Optional, default subscription secureJsonData: clientCertificate: privateKey: # pem only certificatePassword: # pfx only, if set version: 1 ``` ### Managed Identity Use Azure Managed Identity for secure, credential-free authentication when Grafana is hosted in Azure. {{< admonition type="note" >}} Managed Identity is available in [Azure Managed Grafana](https://azure.microsoft.com/en-us/products/managed-grafana) or self-managed Grafana deployed in Azure. It is not available in Grafana Cloud. {{< /admonition >}} #### Managed Identity prerequisites - Grafana must be hosted in Azure (App Service, Azure VMs, or Azure Managed Grafana). - Managed identity must be enabled on the Azure resource hosting Grafana. - The managed identity must have the `Reader` role on the subscription or resources you want to monitor. For details on Azure managed identities, refer to the [Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview). #### Managed Identity Grafana server configuration Enable managed identity in the Grafana server configuration: ```ini [azure] managed_identity_enabled = true ``` To use a user-assigned managed identity instead of the system-assigned identity, also set: ```ini [azure] managed_identity_enabled = true managed_identity_client_id = ``` {{< admonition type="caution" >}} If you've customized the `forward_settings_to_plugins` setting under `[azure]`, verify that `grafana-azure-monitor-datasource` is included. This setting controls which plugins receive the Azure configuration from the Grafana server. By default, all Grafana Labs Azure plugins are included. Missing this setting causes `401 Unauthorized` errors even when Managed Identity is correctly configured on your Azure resource. {{< /admonition >}} Refer to [Grafana Azure configuration](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#azure) for more details. #### Managed Identity UI configuration | Setting | Description | | ------------------------ | --------------------------------------------------------------------------------------------------- | | **Authentication** | Select **Managed Identity**. The directory ID, application ID, and client secret fields are hidden. | | **Default Subscription** | Click **Load Subscriptions** to populate available subscriptions, then select your default. | {{< figure src="/media/docs/grafana/data-sources/screenshot-managed-identity-2.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor data source configured with Managed Identity" >}} #### Provision Managed Identity with YAML ```yaml apiVersion: 1 datasources: - name: Azure Monitor type: grafana-azure-monitor-datasource access: proxy jsonData: azureAuthType: msi subscriptionId: # Optional, default subscription version: 1 ``` ### Workload Identity Use Azure Workload Identity for secure authentication in Kubernetes environments like AKS. #### Workload Identity prerequisites - Grafana must be running in a Kubernetes environment with workload identity federation configured. - The workload identity must have the `Reader` role on the subscription or resources you want to monitor. For details, refer to the [Azure workload identity documentation](https://azure.github.io/azure-workload-identity/docs/). #### Workload Identity Grafana server configuration Enable workload identity in the Grafana server configuration: ```ini [azure] workload_identity_enabled = true ``` Optional configuration variables: ```ini [azure] workload_identity_enabled = true workload_identity_tenant_id = # Microsoft Entra ID tenant containing the managed identity workload_identity_client_id = # Client ID if different from default workload_identity_token_file = # Path to the token file ``` {{< admonition type="caution" >}} If you've customized the `forward_settings_to_plugins` setting under `[azure]`, verify that `grafana-azure-monitor-datasource` is included. Refer to [Managed Identity Grafana server configuration](#managed-identity-grafana-server-configuration) for details. {{< /admonition >}} Refer to [Grafana Azure configuration](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#azure) and the [Azure workload identity documentation](https://azure.github.io/azure-workload-identity/docs/) for more details. #### Workload Identity UI configuration | Setting | Description | | ------------------------ | ---------------------------------------------------------------------------------------------------- | | **Authentication** | Select **Workload Identity**. The directory ID, application ID, and client secret fields are hidden. | | **Default Subscription** | Click **Load Subscriptions** to populate available subscriptions, then select your default. | {{< figure src="/media/docs/grafana/data-sources/screenshot-workload-identity.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor data source configured with Workload Identity" >}} #### Provision Workload Identity with YAML ```yaml apiVersion: 1 datasources: - name: Azure Monitor type: grafana-azure-monitor-datasource access: proxy jsonData: azureAuthType: workloadidentity subscriptionId: # Optional, default subscription version: 1 ``` ### Current User Forward the logged-in Grafana user's Azure credentials to the data source for user-level access control. {{< admonition type="note" >}} Current User authentication is a [generally available feature](/docs/release-life-cycle/). Contact Grafana Support to enable this feature in Grafana Cloud. {{< /admonition >}} #### Current User prerequisites Your Grafana instance must be configured with Microsoft Entra ID authentication. Refer to the [Microsoft Entra ID authentication documentation](https://grafana.com/docs/grafana//setup-grafana/configure-security/configure-authentication/azuread/). #### Configure your Azure App Registration The App Registration used for Grafana login requires additional configuration: **Enable token issuance:** 1. In the Azure Portal, open your App Registration. 1. Select **Authentication** in the side menu. 1. Under **Implicit grant and hybrid flows**, check both **Access tokens** and **ID tokens**. 1. Save your changes. **Add API permissions:** 1. In the Azure Portal, open your App Registration. 1. Select **API Permissions** in the side menu. 1. Ensure these permissions are present under **Microsoft Graph**: `openid`, `profile`, `email`, and `offline_access`. 1. Add the following permissions: - **Azure Service Management** > **Delegated permissions** > `user_impersonation` - **APIs my organization uses** > Search for **Log Analytics API** > **Delegated permissions** > `Data.Read` Refer to the [Azure documentation](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-configure-app-access-web-apis) for more information. **Update Grafana scopes:** Update the `scopes` section in your Grafana Azure authentication configuration to include the `.default` scope: ``` .default openid email profile ``` #### Current User Grafana server configuration Enable current user authentication in the Grafana server configuration: ```ini [azure] user_identity_enabled = true ``` By default, this also enables fallback service credentials. To disable fallback credentials at the instance level: ```ini [azure] user_identity_enabled = true user_identity_fallback_credentials_enabled = false ``` {{< admonition type="caution" >}} If you've customized the `forward_settings_to_plugins` setting under `[azure]`, verify that `grafana-azure-monitor-datasource` is included. Refer to [Managed Identity Grafana server configuration](#managed-identity-grafana-server-configuration) for details. {{< /admonition >}} #### Limitations and fallback credentials Current User authentication doesn't support backend functionality like alerting, reporting, and recorded queries because user credentials aren't available for background operations. To support these features, you must configure **fallback service credentials**, typically an App Registration with client secret. When enabled, Grafana uses the fallback credentials for backend operations. Operations using fallback credentials are limited to the permissions of those credentials, not the user's permissions. {{< admonition type="caution" >}} If you use Current User authentication without configuring fallback service credentials, alerting rules, recording rules, and reports fail with authentication errors. Refer to the [alerting documentation](https://grafana.com/docs/grafana//datasources/azure-monitor/alerting/) for more details. {{< /admonition >}} {{< admonition type="note" >}} Query and resource caching is disabled by default for data sources using Current User authentication. {{< /admonition >}} #### Current User UI configuration | Setting | Description | | -------------------------------- | ------------------------------------------------------------------------------------------- | | **Authentication** | Select **Current User**. | | **Default Subscription** | Click **Load Subscriptions** to populate available subscriptions, then select your default. | | **Fallback Service Credentials** | Enable and configure credentials for backend features like alerting. | {{< figure src="/media/docs/grafana/data-sources/screenshot-current-user.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor data source configured with Current User authentication" >}} #### Provision Current User with YAML {{< admonition type="note" >}} The `oauthPassThru` property is required for Current User authentication. The `disableGrafanaCache` property prevents returning cached responses for resources users don't have access to. {{< /admonition >}} ```yaml apiVersion: 1 datasources: - name: Azure Monitor type: grafana-azure-monitor-datasource access: proxy jsonData: azureAuthType: currentuser oauthPassThru: true disableGrafanaCache: true subscriptionId: # Optional, default subscription version: 1 ``` ## Additional configuration options These settings apply to all authentication methods. ### General settings | Setting | Description | | ----------- | ------------------------------------------------------------------------------- | | **Name** | The data source name used in panels and queries. Example: `azure-monitor-prod`. | | **Default** | Toggle to make this the default data source for new panels. | ### Query timeout The default data source timeout is 30 seconds. Azure Log Analytics queries can take up to 3 minutes on the Azure side, especially for complex KQL queries, large time ranges, or queries across multiple workspaces. If queries fail with `DatasourceError`, `context deadline exceeded`, or timeout errors, increase the timeout. To change the timeout: 1. Open your Azure Monitor data source settings. 1. Scroll to the **Misc** section at the bottom of the configuration page. 1. Set the **Timeout** value. A value of `300` (300 seconds / 5 minutes) accommodates the maximum Azure Log Analytics server-side query duration. 1. Click **Save & test**. {{< admonition type="caution" >}} This timeout also affects alerting evaluations. If your alert rules use Log Analytics queries that exceed 30 seconds, you must increase this setting or alert evaluations fail with timeout errors. For alerting, also ensure your alert evaluation interval is long enough to accommodate the query duration. {{< /admonition >}} To set the timeout in a provisioning file, add `timeout` to the `jsonData` section: ```yaml datasources: - name: Azure Monitor type: grafana-azure-monitor-datasource access: proxy jsonData: timeout: 300 ``` ### Enable Basic Logs Toggle **Enable Basic Logs** to allow queries against [Basic Logs tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1) in supported Log Analytics Workspaces. {{< admonition type="note" >}} Querying Basic Logs tables incurs additional costs on a per-query basis. {{< /admonition >}} ### Enable Batch API Toggle **Enable Batch API** to query metrics for multiple resources in a single request using the [Azure Monitor Metrics Batch API](https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/migrate-to-batch-api). Instead of sending one request per resource, Grafana groups eligible metric queries and sends them together. This reduces the number of calls to Azure, which improves query performance and lowers the chance of throttling on dashboards that query many resources. {{< admonition type="note" >}} This setting is only available when the `datasources.azureMonitorBatchAPI` feature toggle is enabled. {{< /admonition >}} Because requests to the Batch API are sent to the `metrics.monitor.azure.com` data plane endpoint, you need the [Monitoring Reader](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/rest-api-walkthrough#retrieve-metric-values-multi-dimensional-api) role at the subscription scope to use it. {{< admonition type="note" >}} The Metrics Batch API has a free monthly allowance, after which additional calls incur costs. Refer to [Azure Monitor pricing](https://azure.microsoft.com/en-us/pricing/details/monitor/) for current rates. {{< /admonition >}} ### Private data source connect (Grafana Cloud only) If you're using Grafana Cloud and need to connect to Azure resources in a private network, use Private Data Source Connect (PDC). 1. Click the **Private data source connect** dropdown to select your PDC configuration. 1. Click **Manage private data source connect** to view your PDC connection details. For more information, refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) and [Configure PDC](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc). ### Configure exemplars You can link Azure Monitor metrics to traces in a tracing data source by configuring exemplars. This lets you navigate from a metric data point directly to related traces for deeper investigation. To configure exemplars: 1. Open your Azure Monitor data source settings. 1. Scroll to the **Exemplar** configuration section. 1. Click **Add**. 1. Select a **Data source** from the drop-down. The data source must be a tracing backend such as Tempo, Zipkin, Jaeger, or X-Ray. 1. (Optional) Configure a **URL** label to customize the link. 1. Click **Save & test**. After configuration, exemplar links appear on supported Azure Monitor metrics visualizations, allowing you to jump to the corresponding trace in your tracing data source. ## Supported cloud names When provisioning the data source, use the following `cloudName` values: | Azure Cloud | `cloudName` value | | -------------------------------- | ------------------------ | | Microsoft Azure public cloud | `azuremonitor` (default) | | Microsoft Chinese national cloud | `chinaazuremonitor` | | US Government cloud | `govazuremonitor` | {{< admonition type="note" >}} For Current User authentication, the cloud names differ: use `AzureCloud` for public cloud, `AzureChinaCloud` for the Chinese national cloud, and `AzureUSGovernment` for the US Government cloud. {{< /admonition >}} ## Verify the connection After configuring the data source, click **Save & test**. A successful connection displays a message confirming that the credentials are valid and have access to the configured default subscription. If the test fails, verify: - Your credentials are correct (tenant ID, client ID, and client secret or certificate fields) - The identity has the required Azure RBAC permissions - For Managed Identity or Workload Identity, that the Grafana server configuration is correct - Network connectivity to Azure endpoints ## Provision the data source You can define and configure the Azure Monitor data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, refer to [Provisioning Grafana](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ### Provision quick reference | Authentication method | `azureAuthType` value | Required fields | | -------------------------------- | --------------------- | ---------------------------------------------------------------- | | App Registration (client secret) | `clientsecret` | `tenantId`, `clientId`, `clientSecret` | | App Registration (certificate) | `clientcertificate` | `tenantId`, `clientId`, `certificateFormat`, `clientCertificate` | | Managed Identity | `msi` | None (uses VM identity) | | Workload Identity | `workloadidentity` | None (uses Pod identity) | | Current User | `currentuser` | `oauthPassThru: true`, `disableGrafanaCache: true` | All methods support the optional `subscriptionId` field to set a default subscription. For complete YAML examples, see the [authentication method sections](#configure-authentication) above. ## Configure with Terraform You can configure the Azure Monitor data source using the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). This approach enables infrastructure-as-code workflows and version control for your Grafana configuration. ### Terraform prerequisites - [Terraform](https://www.terraform.io/downloads) installed - Grafana Terraform provider configured with appropriate credentials - For Grafana Cloud: A [Cloud Access Policy token](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) with data source permissions ### Provider configuration Configure the Grafana provider to connect to your Grafana instance: ```hcl terraform { required_providers { grafana = { source = "grafana/grafana" version = ">= 2.0.0" } } } # For Grafana Cloud provider "grafana" { url = "" auth = "" } # For self-managed Grafana # provider "grafana" { # url = "http://localhost:3000" # auth = "" # } ``` ### Terraform examples The following examples show how to configure the Azure Monitor data source for each authentication method. **App Registration (client secret):** ```hcl resource "grafana_data_source" "azure_monitor" { type = "grafana-azure-monitor-datasource" name = "Azure Monitor" json_data_encoded = jsonencode({ azureAuthType = "clientsecret" cloudName = "azuremonitor" tenantId = "" clientId = "" subscriptionId = "" }) secure_json_data_encoded = jsonencode({ clientSecret = "" }) } ``` **App Registration (client certificate with PEM):** ```hcl resource "grafana_data_source" "azure_monitor" { type = "grafana-azure-monitor-datasource" name = "Azure Monitor" json_data_encoded = jsonencode({ azureAuthType = "clientcertificate" cloudName = "azuremonitor" tenantId = "" clientId = "" certificateFormat = "pem" subscriptionId = "" }) secure_json_data_encoded = jsonencode({ clientCertificate = "" privateKey = "" }) } ``` **App Registration (client certificate with PFX):** ```hcl resource "grafana_data_source" "azure_monitor" { type = "grafana-azure-monitor-datasource" name = "Azure Monitor" json_data_encoded = jsonencode({ azureAuthType = "clientcertificate" cloudName = "azuremonitor" tenantId = "" clientId = "" certificateFormat = "pfx" subscriptionId = "" }) secure_json_data_encoded = jsonencode({ clientCertificate = "" certificatePassword = "" }) } ``` **Managed Identity:** ```hcl resource "grafana_data_source" "azure_monitor" { type = "grafana-azure-monitor-datasource" name = "Azure Monitor" json_data_encoded = jsonencode({ azureAuthType = "msi" subscriptionId = "" }) } ``` **Workload Identity:** ```hcl resource "grafana_data_source" "azure_monitor" { type = "grafana-azure-monitor-datasource" name = "Azure Monitor" json_data_encoded = jsonencode({ azureAuthType = "workloadidentity" subscriptionId = "" }) } ``` **Current User:** ```hcl resource "grafana_data_source" "azure_monitor" { type = "grafana-azure-monitor-datasource" name = "Azure Monitor" json_data_encoded = jsonencode({ azureAuthType = "currentuser" oauthPassThru = true disableGrafanaCache = true subscriptionId = "" }) } ``` **With Basic Logs enabled:** Add `basicLogsEnabled = true` to any of the above configurations: ```hcl resource "grafana_data_source" "azure_monitor" { type = "grafana-azure-monitor-datasource" name = "Azure Monitor" json_data_encoded = jsonencode({ azureAuthType = "clientsecret" cloudName = "azuremonitor" tenantId = "" clientId = "" subscriptionId = "" basicLogsEnabled = true }) secure_json_data_encoded = jsonencode({ clientSecret = "" }) } ``` For more information about the Grafana Terraform provider, refer to the [provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs) and the [`grafana_data_source` resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). # Azure Monitor query editor Grafana provides a query editor for the Azure Monitor data source, which is located on the [Explore page](https://grafana.com/docs/grafana//explore/). You can also access the Azure Monitor query editor from a dashboard panel. Click the menu in the upper right of the panel and select **Edit**. This document explains querying specific to the Azure Monitor data source. For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/). The Azure Monitor data source can query data from Azure Monitor Metrics and Logs, the Azure Resource Graph, and Application Insights Traces. Each source has its own specialized query editor. ## Before you begin - Ensure you have [configured the Azure Monitor data source](https://grafana.com/docs/grafana//datasources/azure-monitor/configure/). - Verify your credentials have appropriate permissions for the resources you want to query. ## Key concepts If you're new to Azure Monitor, here are some key terms used throughout this documentation: | Term | Description | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **KQL (Kusto Query Language)** | The query language used for Azure Monitor Logs and Azure Resource Graph. KQL uses a pipe-based syntax similar to Unix commands and is optimized for read-only data exploration. If you know SQL, the [SQL to Kusto cheat sheet](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/sqlcheatsheet) can help you get started. | | **Log Analytics workspace** | An Azure resource that collects and stores log data from your Azure resources, applications, and services. You query this data using KQL. | | **Application Insights** | Azure's application performance monitoring (APM) service. It collects telemetry data like requests, exceptions, and traces from your applications. | | **Metrics vs. Logs** | **Metrics** are lightweight numeric values collected at regular intervals (for example, CPU percentage). **Logs** are detailed records of events with varying schemas (for example, request logs, error messages). Metrics use a visual query builder; Logs require KQL. | ## Choose a query editor mode The Azure Monitor data source's query editor has four modes depending on which Azure service you want to query: - **Metrics** for [Azure Monitor Metrics](#query-azure-monitor-metrics) - **Logs** for [Azure Monitor Logs](#query-azure-monitor-logs) - **Traces** for [Application Insights Traces](#query-application-insights-traces) - **Azure Resource Graph** for [Azure Resource Graph](#query-azure-resource-graph) ## Query Azure Monitor Metrics Azure Monitor Metrics collects numeric data from [supported resources](https://docs.microsoft.com/en-us/azure/azure-monitor/monitor-reference), and you can query them to investigate your resources' health and usage and maximize availability and performance. Monitor Metrics use a lightweight format that stores only numeric data in a specific structure and supports near real-time scenarios, making it useful for fast detection of issues. In contrast, Azure Monitor Logs can store a variety of data types, each with their own structure. {{< figure src="/static/img/docs/azure-monitor/query-editor-metrics.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor Metrics sample query visualizing CPU percentage over time" >}} ### Create a Metrics query **To create a Metrics query:** 1. In a Grafana panel, select the **Azure Monitor** data source. 1. Select the **Metrics** service. 1. Select a resource from which to query metrics by using the subscription, resource group, resource type, and resource fields. Multiple resources can also be selected as long as they belong to the same subscription, region and resource type. Note that only a limited amount of resource types support this feature. 1. To select a different namespace than the default—for instance, to select resources like storage accounts that are organized under multiple namespaces—use the **Namespace** option. {{< admonition type="note" >}} Not all metrics returned by the Azure Monitor Metrics API have values. {{< /admonition >}} > The data source retrieves lists of supported metrics for each subscription and ignores metrics that never have values. 1. Select a metric from the **Metric** field. Optionally, you can apply further aggregations or filter by dimensions. 1. Change the aggregation from the default average to show minimum, maximum, or total values. 1. Specify a custom time grain. By default, Grafana selects a time grain interval for you based on your selected time range. 1. For metrics with multiple dimensions, you can split and filter the returned metrics. For example, the Application Insights dependency calls metric supports returning multiple time series for successful and unsuccessful calls. {{< figure src="/static/img/docs/azure-monitor/query-editor-metrics-dimensions.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor Metrics screenshot showing Dimensions" >}} The available options change depending on what is relevant to the selected metric. You can also augment queries by using [template variables](https://grafana.com/docs/grafana//datasources/azure-monitor/template-variables/). ### Format legend aliases You can change the legend label for Metrics by using aliases. In the Legend Format field, you can combine aliases defined below any way you want. For example: - `Blob Type: {{ blobtype }}` becomes `Blob Type: PageBlob`, `Blob Type: BlockBlob` - `{{ resourcegroup }} - {{ resourcename }}` becomes `production - web_server` | Alias pattern | Description | | ------------------------------ | ------------------------------------------------------------------------------------------------------ | | `{{ subscriptionid }}` | Replaced with the subscription ID. | | `{{ subscription }}` | Replaced with the subscription name. | | `{{ resourcegroup }}` | Replaced with the resource group. | | `{{ namespace }}` | Replaced with the resource type or namespace, such as `Microsoft.Compute/virtualMachines`. | | `{{ resourcename }}` | Replaced with the resource name. | | `{{ metric }}` | Replaced with the metric name, such as "Percentage CPU". | | _`{{ arbitraryDimensionID }}`_ | Replaced with the value of the specified dimension. For example, `{{ blobtype }}` becomes `BlockBlob`. | | `{{ dimensionname }}` | _(Legacy for backward compatibility)_ Replaced with the name of the first dimension. | | `{{ dimensionvalue }}` | _(Legacy for backward compatibility)_ Replaced with the value of the first dimension. | ### Filter with dimensions Some metrics also have dimensions, which associate additional metadata. Dimensions are represented as key-value pairs assigned to each value of a metric. Grafana can display and filter metrics based on dimension values. The data source supports the `equals`, `not equals`, and `starts with` operators as detailed in the [Monitor Metrics API documentation](https://docs.microsoft.com/en-us/rest/api/monitor/metrics/list). For more information on multi-dimensional metrics, refer to the [Azure Monitor data platform metrics documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/data-platform-metrics#multi-dimensional-metrics) and [Azure Monitor filtering documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-charts#filters). ## Query Azure Monitor Logs Azure Monitor Logs collects and organizes log and performance data from [supported resources](https://docs.microsoft.com/en-us/azure/azure-monitor/monitor-reference), and makes many sources of data available to query together with the [Kusto Query Language (KQL)](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/). While Azure Monitor Metrics stores only simplified numerical data, Logs can store different data types, each with their own structure. You can also perform complex analysis of Logs data by using KQL. The Azure Monitor data source also supports querying of [Basic Logs](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1) tables (if they exist in your Log Analytics workspace). This feature must be enabled in the data source configuration. {{< figure src="/static/img/docs/azure-monitor/query-editor-logs.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor Logs sample query comparing successful requests to failed requests" >}} ### Logs query builder (public preview) {{< admonition type="note" >}} The Logs query builder is a [public preview feature](/docs/release-life-cycle/). It may not be enabled in all Grafana environments. {{< /admonition >}} The Logs query builder provides a visual interface for building Azure Monitor Logs queries without writing KQL. This is helpful if you're new to KQL or want to quickly build queries. **To enable the Logs query builder:** 1. Enable the `azureMonitorLogsBuilderEditor` [feature toggle](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/feature-toggles/) in your Grafana configuration. 1. Restart Grafana for the change to take effect. **To switch between Builder and Code modes:** When the feature is enabled, a **Builder / Code** toggle appears in the Logs query editor: - **Builder**: Use the visual interface to select tables, columns, filters, and aggregations. The builder generates the KQL query for you. - **Code**: Write KQL queries directly. Use this mode for complex queries that require full KQL capabilities. New queries default to Builder mode. Existing queries that were created with raw KQL remain in Code mode. When you switch to Builder mode, the time-range setting automatically changes to **Dashboard** because the builder manages time filtering for you. {{< admonition type="note" >}} You can switch from Builder to Code mode at any time to view or edit the generated KQL. However, switching from Code to Builder mode may not preserve complex queries that can't be represented in the builder interface. {{< /admonition >}} ### Create a Logs query **To create a Logs query:** 1. In a Grafana panel, select the **Azure Monitor** data source. 1. Select the **Logs** service. 1. Select a resource to query. Multiple resources can be selected as long as they are of the same type. Alternatively, you can dynamically query all resources under a single resource group or subscription. {{< admonition type="note" >}} If a time span is specified in the query, the overlap between the query time span and the dashboard time range is used. Refer to the [API documentation for details.](https://learn.microsoft.com/en-us/rest/api/loganalytics/dataaccess/query/get?tabs=HTTP#uri-parameters) {{< /admonition >}} 1. Enter your KQL query. You can also augment queries by using [template variables](https://grafana.com/docs/grafana//datasources/azure-monitor/template-variables/). **To create a Basic Logs query:** 1. Ensure that the data source has the **Enable Basic Logs** toggle enabled in the [data source configuration](https://grafana.com/docs/grafana//datasources/azure-monitor/configure/#enable-basic-logs). 1. In a Grafana panel, select the **Azure Monitor** data source. 1. Select the **Logs** service. 1. Select a single resource to query. Basic Logs queries only support a single workspace resource. 1. Switch the **Logs** toggle from **Analytics** to **Basic**. A confirmation dialog box displays to notify you of potential additional costs. 1. Enter your KQL query. {{< admonition type="caution" >}} Basic Logs queries are billed on a per-query basis by Azure. Review the [Azure pricing documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1) before using this feature. {{< /admonition >}} Basic Logs queries have the following restrictions: - **Time range:** The dashboard time range is always used. You can't specify a time range within the query itself. - **Single resource:** Only a single Log Analytics workspace can be queried. Multi-resource selection isn't supported. - **Alerting:** Basic Logs queries aren't available when creating alert rules. - **KQL limitations:** Some KQL operators aren't supported. Refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1#limitations) for a full list of limitations. You can also augment queries by using [template variables](https://grafana.com/docs/grafana//datasources/azure-monitor/template-variables/). ### Logs result row limit When a Logs query uses the **Logs** result format, Grafana truncates the results to 30,000 rows to prevent the browser tab from crashing, because the logs visualization renders every row at once. When truncation happens, Grafana returns the first 30,000 rows along with a notice that the limit was reached. To remove this limit, enable the `azureMonitorDisableLogLimit` [feature toggle](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/feature-toggles/) in your Grafana configuration. {{< admonition type="caution" >}} Disabling the row limit can degrade browser performance for very large result sets. Prefer aggregating with `summarize` or narrowing the time range to reduce the number of rows returned. {{< /admonition >}} ### Logs query examples Azure Monitor Logs queries are written using the Kusto Query Language (KQL), a rich language similar to SQL. The Azure documentation includes resources to help you learn KQL: - [Log queries in Azure Monitor](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/log-query-overview) - [Getting started with Kusto](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/concepts/) - [Tutorial: Use Kusto queries in Azure Monitor](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/tutorial?pivots=azuremonitor) - [SQL to Kusto cheat sheet](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/sqlcheatsheet) {{< admonition type="note" >}} **Time-range:** The time-range used for the query can be modified via the time-range switch: - Selecting **Query** uses only time-ranges specified within the query. - Selecting **Dashboard** uses only the Grafana dashboard time-range. - If no time-range is specified in the query, the default Log Analytics time-range applies. For more details, refer to the [Azure Monitor Logs API documentation](https://learn.microsoft.com/en-us/rest/api/loganalytics/dataaccess/query/get?tabs=HTTP#uri-parameters). If you previously used the `Intersection` option, it has been migrated to `Dashboard`. {{< /admonition >}} This example query returns a virtual machine's CPU performance, averaged over 5-minute time grains: ```kusto Perf // $__timeFilter is a special Grafana macro that filters the results to the time span of the dashboard | where $__timeFilter(TimeGenerated) | where CounterName == "% Processor Time" | summarize avg(CounterValue) by bin(TimeGenerated, 5m), Computer | order by TimeGenerated asc ``` Use time series queries for values that change over time, usually for graph visualizations such as the Time series panel. Each query should return at least a `datetime` column and numeric value column. The result must also be sorted in ascending order by the `datetime` column. You can also create a query with at least one non-numeric, non-`datetime` column. Azure Monitor considers those columns to be dimensions, and they become labels in the response. For example, this query returns the aggregated count grouped by hour, Computer, and the CounterName: ```kusto Perf | where $__timeFilter(TimeGenerated) | summarize count() by bin(TimeGenerated, 1h), Computer, CounterName | order by TimeGenerated asc ``` You can also select additional number value columns, with or without multiple dimensions. For example, this query returns a count and average value by hour, Computer, CounterName, and InstanceName: ```kusto Perf | where $__timeFilter(TimeGenerated) | summarize Samples=count(), ["Avg Value"]=avg(CounterValue) by bin(TimeGenerated, $__interval), Computer, CounterName, InstanceName | order by TimeGenerated asc ``` Use table queries with the Table panel to produce a list of columns and rows. This query returns rows with the six specified columns: ```kusto AzureActivity | where $__timeFilter() | project TimeGenerated, ResourceGroup, Category, OperationName, ActivityStatus, Caller | order by TimeGenerated desc ``` ### Use macros in Logs queries To help you write queries, you can use several Grafana macros in the `where` clause: | Macro | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `$__timeFilter()` | Filters the results to the time range of the dashboard. Example: `TimeGenerated >= datetime(2018-06-05T18:09:58.907Z) and TimeGenerated <= datetime(2018-06-05T20:09:58.907Z)`. | | `$__timeFilter(datetimeColumn)` | Like `$__timeFilter()`, but specifies a custom field to filter on. | | `$__timeFrom()` | Returns the start of the dashboard time range. Example: `datetime(2018-06-05T18:09:58.907Z)`. | | `$__timeTo()` | Returns the end of the dashboard time range. Example: `datetime(2018-06-05T20:09:58.907Z)`. | | `$__escapeMulti($myVar)` | Escapes illegal characters in multi-value template variables. If `$myVar` has the values `'\\grafana-vm\Network(eth0)\Total','\\hello!'` as a string, use this to expand it to `@'\\grafana-vm\Network(eth0)\Total', @'\\hello!'`. If using single-value variables, escape the variable inline instead: `@'\$myVar'`. | | `$__contains(colName, $myVar)` | Expands multi-value template variables. If `$myVar` has the value `'value1','value2'`, use this to expand it to `colName in ('value1','value2')`. If using the `All` option, check the `Include All Option` checkbox, and type the value `all` in the `Custom all value` field. If `$myVar` has the value `all`, the macro instead expands to `1 == 1`. For template variables with many options, this avoids building a large "where..in" clause, which improves performance. | Additionally, Grafana has the built-in [`$__interval` macro](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/#query-options), which calculates an interval in seconds. ## Query Azure Resource Graph Azure Resource Graph (ARG) is an Azure service designed to extend Azure Resource Management with efficient resource exploration and the ability to query at scale across a set of subscriptions, so that you can more effectively govern an environment. By querying ARG, you can query resources with complex filtering, iteratively explore resources based on governance requirements, and assess the impact of applying policies in a vast cloud environment. {{< figure src="/static/img/docs/azure-monitor/query-editor-arg.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Resource Graph sample query listing virtual machines on an account" >}} ### Create a Resource Graph query ARG queries are written in a variant of the [Kusto Query Language (KQL)](https://docs.microsoft.com/en-us/azure/governance/resource-graph/concepts/query-language), but not all Kusto language features are available in ARG. An Azure Resource Graph query is formatted as table data. If your Azure credentials grant you access to multiple subscriptions, you can choose multiple subscriptions before entering queries. It is also possible to run queries against the directory by changing the scope of the query. {{< admonition type="note" >}} Some queries that function at a directory level may not work at a subscription level and vice-versa. {{< /admonition >}} ### Resource Graph query examples The Azure documentation also includes [sample queries](https://docs.microsoft.com/en-gb/azure/governance/resource-graph/samples/starter) to help you get started. **Sort results by resource properties:** This query returns all resources in the selected subscriptions, but only the name, type, and location properties: ```kusto Resources | project name, type, location | order by name asc ``` This query uses `order by` to sort the properties by the `name` property in ascending (`asc`) order. You can change which property to sort by and the order (`asc` or `desc`). This query uses `project` to show only the listed properties in the results. You can use this to add or remove properties in your queries. **Query resources with complex filtering:** You can filter for Azure resources with a tag name and value. For example, this query returns a list of resources with an `environment` tag value of `Internal`: ```kusto Resources | where tags.environment=~'internal' | project name ``` This query uses `=~` in the `type` match to make the query case-insensitive. You can also use `project` with other properties, or add or remove more. **Group and aggregate the values by property:** You can use `summarize` and `count` to define how to group and aggregate values by property. For example, this query returns counts of healthy, unhealthy, and not applicable resources per recommendation: ```kusto securityresources | where type == 'microsoft.security/assessments' | extend resourceId=id,     recommendationId=name,     resourceType=type,     recommendationName=properties.displayName,     source=properties.resourceDetails.Source,     recommendationState=properties.status.code,     description=properties.metadata.description,     assessmentType=properties.metadata.assessmentType,     remediationDescription=properties.metadata.remediationDescription,     policyDefinitionId=properties.metadata.policyDefinitionId,     implementationEffort=properties.metadata.implementationEffort,     recommendationSeverity=properties.metadata.severity,     category=properties.metadata.categories,     userImpact=properties.metadata.userImpact,     threats=properties.metadata.threats,     portalLink=properties.links.azurePortal | summarize numberOfResources=count(resourceId) by tostring(recommendationName), tostring(recommendationState) ``` In ARG, many nested properties (`properties.displayName`) are of a `dynamic` type and should be cast to a string with `tostring()` to operate on them. ### Use macros in Resource Graph queries To help you write queries, you can use several Grafana macros in the `where` clause: | Macro | Description | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `$__timeFilter()` | Expands to `timestamp ≥ datetime(2018-06-05T18:09:58.907Z) and timestamp ≤ datetime(2018-06-05T20:09:58.907Z)`, where the from and to date-time values are from the Grafana time picker. | | `$__timeFilter(datetimeColumn)` | Expands to `datetimeColumn ≥ datetime(2018-06-05T18:09:58.907Z) and datetimeColumn ≤ datetime(2018-06-05T20:09:58.907Z)`, where the from and to date-time values are from the Grafana time picker. | | `$__timeFrom()` | Returns the start of the dashboard time range from the Grafana picker. Example: `datetime(2018-06-05T18:09:58.907Z)`. | | `$__timeTo()` | Returns the end of the dashboard time range from the Grafana picker. Example: `datetime(2018-06-05T20:09:58.907Z)`. | | `$__escapeMulti($myVar)` | Escapes illegal characters from multi-value template variables. If `$myVar` has the values `'\\grafana-vm\Network(eth0)\Total','\\hello!'` as a string, this expands it to `@'\\grafana-vm\Network(eth0)\Total', @'\\hello!'`. If you use single-value variables, escape the variable inline instead: `@'\$myVar'`. | | `$__contains(colName, $myVar)` | Expands multi-value template variables. If `$myVar` has the value `'value1','value2'`, this expands it to `colName in ('value1','value2')`. If using the `All` option, then check the `Include All Option` checkbox and in the `Custom all value` field type in the following value: `all`. If `$myVar` has value `all`, this instead expands to `1 == 1`. For template variables with many options, this avoids building a large "where..in" clause, which improves performance. | ## Query Application Insights Traces [Azure Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview?tabs=net) is a service that provides application performance monitoring (APM) features. Application insights can be used to collect metrics, telemetry, and trace logging data. Application Insights stores trace data in an underlying Log Analytics workspace and traces can be [extended](https://learn.microsoft.com/en-us/azure/azure-monitor/app/api-custom-events-metrics) to provide additional supporting information as required by the application developer. ### Create a Traces query **To create a Traces query:** 1. In a Grafana panel, select the **Azure Monitor** data source. 1. Select the **Traces** service. 1. Select a resource to query. Multiple resources can be selected as long as they are of the same type. {{< admonition type="note" >}} This query type only supports Application Insights resources. {{< /admonition >}} 1. (Optional) Specify an **Operation ID** value to filter traces. 1. (Optional) Specify **event types** to filter by. 1. (Optional) Specify **event properties** to filter by. 1. (Optional) Change the **Result format** to switch between tabular format and trace format. {{< admonition type="note" >}} Selecting the trace format filters events to only the `trace` type. Use this format with the Trace visualization. {{< /admonition >}} Running a query returns all trace data within the time span specified by the panel or dashboard time range. You can also augment queries by using [template variables](https://grafana.com/docs/grafana//datasources/azure-monitor/template-variables/). ## Use queries for alerting and recording rules All Azure Monitor query types (Metrics, Logs, Azure Resource Graph, and Traces) can be used with Grafana Alerting and recording rules. For detailed information about creating alert rules, supported query types, authentication requirements, and examples, refer to [Azure Monitor alerting](https://grafana.com/docs/grafana//datasources/azure-monitor/alerting/). ## Work with large Azure resource datasets If a request exceeds the [maximum allowed value of records](https://docs.microsoft.com/en-us/azure/governance/resource-graph/concepts/work-with-data#paging-results), the result is paginated and only the first page of results are returned. You can use filters to reduce the amount of records returned under that value. ## Next steps - [Use template variables](../template-variables/) to create dynamic, reusable dashboards - [Add annotations](https://grafana.com/docs/grafana//datasources/azure-monitor/annotations/) to overlay events on your graphs - [Set up alerting](https://grafana.com/docs/grafana//datasources/azure-monitor/alerting/) to create alert rules based on Azure Monitor data - [Troubleshoot](https://grafana.com/docs/grafana//datasources/azure-monitor/troubleshooting/) common query and configuration issues # Azure Monitor template variables Instead of hard-coding details such as resource group or resource name values in metric queries, you can use variables. This helps you create more interactive, dynamic, and reusable dashboards. Grafana refers to such variables as template variables. For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana//dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/). ## Before you begin - Ensure you have [configured the Azure Monitor data source](https://grafana.com/docs/grafana//datasources/azure-monitor/configure/). - If you want template variables to auto-populate subscriptions, set a **Default Subscription** in the data source configuration. ## Create a template variable To create a template variable for Azure Monitor: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Variable**. 1. Select **Query** as the variable type. 1. Enter a **Name** for your variable (for example, `subscription`, `resourceGroup`, or `resource`). 1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed. 1. Click **Open variable editor** to open the **Query Variable** dialog box. 1. In the **Data source** drop-down list, select your Azure Monitor data source. 1. In the **Query Type** drop-down list, select the appropriate query type (refer to [Available query types](#available-query-types)). 1. Configure any additional fields required by the selected query type. 1. Click **Preview** to preview the variable values. 1. Click **Close** to close the query variable editor dialog box. 1. Configure display options such as **Multi-value**, **Include All value**, or **Allow custom values** as needed. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. ## Available query types The Azure Monitor data source provides the following query types for template variables: | Query type | Description | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | **Subscriptions** | Returns a list of Azure subscriptions accessible to the configured credentials. | | **Resource Groups** | Returns resource groups for a specified subscription. Supports multi-value selection. | | **Namespaces** | Returns metric namespaces for the specified subscription. If a resource group is specified, returns only namespaces within that group. | | **Regions** | Returns Azure regions available for the specified subscription. | | **Resource Names** | Returns resource names for a specified subscription, resource group, and namespace. Supports multi-value selection. | | **Metric Names** | Returns available metric names for a specified resource. | | **Workspaces** | Returns Log Analytics workspaces for the specified subscription. | | **Logs** | Executes a KQL query and returns the results as variable values. Refer to [Create a Logs variable](#create-a-logs-variable). | | **Custom Namespaces** | Returns custom metric namespaces for a specified resource. | | **Custom Metric Names** | Returns custom metric names for a specified resource. | {{< admonition type="note" >}} Custom metrics cannot be emitted against a subscription or resource group. Select specific resources when retrieving custom metric namespaces or custom metric names. {{< /admonition >}} ## Create cascading variables Cascading variables (also called dependent or chained variables) allow you to create dropdown menus that filter based on previous selections. This is useful for drilling down from subscription to resource group to specific resource. ### Example: Subscription → Resource Group → Resource Name **Step 1: Create a Subscription variable** 1. Create a variable named `subscription`. 1. Set **Query Type** to **Subscriptions**. **Step 2: Create a Resource Group variable** 1. Create a variable named `resourceGroup`. 1. Set **Query Type** to **Resource Groups**. 1. In the **Subscription** field, select `$subscription`. **Step 3: Create a Resource Name variable** 1. Create a variable named `resource`. 1. Set **Query Type** to **Resource Names**. 1. In the **Subscription** field, select `$subscription`. 1. In the **Resource Group** field, select `$resourceGroup`. 1. Select the appropriate **Namespace** for your resources (for example, `Microsoft.Compute/virtualMachines`). Now when you change the subscription, the resource group dropdown updates automatically, and when you change the resource group, the resource name dropdown updates. ## Create a Logs variable The **Logs** query type lets you use a KQL query to populate variable values. The query must return a single column of values. **To create a Logs variable:** 1. Create a new variable with **Query Type** set to **Logs**. 1. Select a **Resource** (Log Analytics workspace or Application Insights resource). 1. Enter a KQL query that returns a single column. ### Logs variable query examples | Query | Returns | | ----------------------------------------- | ------------------------------------- | | `Heartbeat \| distinct Computer` | List of virtual machine names | | `Perf \| distinct ObjectName` | List of performance object names | | `AzureActivity \| distinct ResourceGroup` | List of resource groups with activity | | `AppRequests \| distinct Name` | List of application request names | You can reference other variables in your Logs query: ```kusto workspace("$workspace").Heartbeat | distinct Computer ``` ```kusto workspace("$workspace").Perf | where ObjectName == "$object" | distinct CounterName ``` ## Variable refresh options Control when your variables refresh by setting the **Refresh** option: | Option | Behavior | | ------------------------ | ----------------------------------------------------------------------------------------- | | **On dashboard load** | Variables refresh each time the dashboard loads. Best for data that changes infrequently. | | **On time range change** | Variables refresh when the dashboard time range changes. Use for time-sensitive queries. | For dashboards with many variables or complex queries, use **On dashboard load** to improve performance. ## Use variables in queries After you create template variables, you can use them in your Azure Monitor queries by referencing them with the `$` prefix. ### Metrics query example In a Metrics query, select your variables in the resource picker fields: - **Subscription**: `$subscription` - **Resource Group**: `$resourceGroup` - **Resource Name**: `$resource` ### Logs query example Reference variables directly in your KQL queries: ```kusto Perf | where ObjectName == "$object" and CounterName == "$metric" | where TimeGenerated >= $__timeFrom() and TimeGenerated <= $__timeTo() | where $__contains(Computer, $computer) | summarize avg(CounterValue) by bin(TimeGenerated, $__interval), Computer | order by TimeGenerated asc ``` ## Multi-value variables You can enable **Multi-value** selection for **Resource Groups** and **Resource Names** variables. When using multi-value variables in a Metrics query, all selected resources must: - Belong to the same subscription - Be in the same Azure region - Be of the same resource type (namespace) {{< admonition type="note" >}} When a multi-value variable is used as a parameter in another variable query (for example, to retrieve metric names), only the first selected value is used. Ensure the first resource group and resource name combination is valid. {{< /admonition >}} ## Troubleshoot template variables If you encounter issues with template variables, try the following solutions. For additional troubleshooting, refer to the [Azure Monitor troubleshooting guide](https://grafana.com/docs/grafana//datasources/azure-monitor/troubleshooting/#template-variable-errors). ### Variable returns no values - Verify the Azure Monitor data source is configured correctly and can connect to Azure. - Check that the credentials have appropriate permissions to list the requested resources. - For cascading variables, ensure parent variables have valid selections. ### Variable values are outdated - Check the **Refresh** setting and adjust if needed. - Click the refresh icon next to the variable dropdown to manually refresh. ### Variables break after a Grafana upgrade If template variables return errors such as `Properties found in series but missing valueProp and textProp` after upgrading Grafana, this typically indicates a version-specific bug. Update to the latest patch release for your Grafana version. For Grafana Cloud, verify you're on the **stable** release channel. Refer to [Template variable errors](https://grafana.com/docs/grafana//datasources/azure-monitor/troubleshooting/#template-variable-errors) for detailed solutions. ### Multi-value selection not working in queries - Ensure the resources meet the requirements (same subscription, region, and type). - For Logs queries, use the `$__contains()` macro to handle multi-value variables properly. # Troubleshoot Azure Monitor data source issues This document provides solutions to common issues you may encounter when configuring or using the Azure Monitor data source, organized by the stage where the issue occurs. ## Connection and network errors These errors indicate Grafana can't reach Azure endpoints. Verify connectivity before investigating authentication or query issues. ### "Connection refused" or timeout errors **Symptoms:** - Data source test fails with network errors - Queries timeout without returning results **Solutions:** 1. Verify network connectivity from Grafana to Azure endpoints. 1. Check firewall rules allow outbound HTTPS (port 443) to Azure services. 1. For private networks, ensure Private Link or VPN is configured correctly. 1. For Grafana Cloud, configure [Private Data Source Connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if accessing private resources. ### SSL/TLS certificate errors **Symptoms:** - Certificate validation failures - SSL handshake errors **Solutions:** 1. Ensure the system time is correct (certificate validation fails with incorrect time). 1. Verify corporate proxy isn't intercepting HTTPS traffic. 1. Check that required CA certificates are installed on the Grafana server. ## Authentication errors These errors occur when credentials configured in the data source or in Azure are invalid, expired, or missing permissions. ### "Authorization failed" or "Access denied" **Symptoms:** - Save & test fails with "Authorization failed" - Queries return "Access denied" errors - Subscriptions don't load when clicking **Load Subscriptions** **Possible causes and solutions:** | Cause | Solution | | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | App registration doesn't have required permissions | Assign the `Reader` role to the app registration on the subscription or resource group you want to monitor. Refer to the [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). | | Incorrect tenant ID, client ID, or client secret | Verify the credentials in the Azure Portal under **App registrations** > your app > **Overview** (for IDs) and **Certificates & secrets** (for secret). | | Client secret has expired | Create a new client secret in Azure and update the data source configuration. | | Managed Identity not enabled on the Azure resource | For VMs, enable managed identity in the Azure Portal under **Identity**. For App Service, enable it under **Identity** in the app settings. | | Managed Identity not assigned the Reader role | Assign the `Reader` role to the managed identity on the target subscription or resources. | ### "Invalid client secret" or "Client secret not found" **Symptoms:** - Authentication fails immediately after configuration - Error message references invalid credentials **Solutions:** 1. Ensure you copied the client secret **value**, not the secret ID. In Azure Portal under **Certificates & secrets**, the secret value is only shown once when created. The secret ID is a different identifier and won't work for authentication. 1. Verify the client secret was copied correctly (no extra spaces or truncation). 1. Check if the secret has expired in Azure Portal under **App registrations** > your app > **Certificates & secrets**. 1. Create a new secret and update the data source configuration. ### "Tenant not found" or "Invalid tenant ID" **Symptoms:** - Data source test fails with tenant-related errors - Unable to authenticate **Solutions:** 1. Verify the Directory (tenant) ID in Azure Portal under **Microsoft Entra ID** > **Overview**. 1. Ensure you're using the correct Azure cloud setting (Azure, Azure Government, or Azure China). 1. Check that the tenant ID is written using a valid GUID format. ### Client certificate authentication not working **Symptoms:** - Authentication fails when using App Registration with client certificate - Error references invalid certificate or signature validation failure **Possible causes and solutions:** | Cause | Solution | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Certificate has expired | Check the certificate expiration date in Azure Portal under **App registrations** > your app > **Certificates & secrets**. Upload a new certificate if expired. | | Wrong certificate format | Verify you're using the correct format (PEM or PFX) and that the content matches what you selected. PEM requires separate certificate and private key fields. | | PFX password is incorrect | For encrypted PFX certificates, verify the password is correct. | | Private key doesn't match the certificate | Ensure the private key corresponds to the certificate uploaded to Azure. Regenerate the certificate and key pair if needed. | | Certificate not uploaded to Azure | Verify the public certificate is uploaded to the app registration under **Certificates & secrets** > **Certificates** in the Azure Portal. | ## Grafana server configuration errors These errors occur when the Grafana server `.ini` configuration is missing settings required by Managed Identity, Workload Identity, or Current User authentication. If your Azure credentials are correct but authentication still fails, check this section. ### Managed Identity not working **Symptoms:** - Managed Identity option is available but authentication fails - Error: "Managed identity authentication is not available" **Solutions:** 1. Verify `managed_identity_enabled = true` is set in the Grafana server configuration under `[azure]`. 1. Confirm the Azure resource hosting Grafana has managed identity enabled. 1. For user-assigned managed identity, ensure `managed_identity_client_id` is set correctly. 1. Verify the managed identity has the `Reader` role on the target resources. 1. Restart Grafana after changing server configuration. ### Workload Identity not working **Symptoms:** - Workload Identity authentication fails in Kubernetes/AKS environment - Token file errors **Solutions:** 1. Verify `workload_identity_enabled = true` is set in the Grafana server configuration. 1. Check that the service account is correctly annotated for workload identity. 1. Verify the federated credential is configured in Azure. 1. Ensure the token path is accessible to the Grafana Pod. 1. Check the workload identity webhook is running in the cluster. ### "401 Unauthorized" despite correct Azure setup **Symptoms:** - Authentication fails with `401 Unauthorized` even though Managed Identity or Workload Identity is correctly configured in Azure - The managed identity has the correct RBAC roles - `managed_identity_enabled` or `workload_identity_enabled` is set to `true` in the Grafana `.ini` file **Cause:** The `forward_settings_to_plugins` setting under `[azure]` doesn't include the Azure Monitor plugin, so the plugin doesn't receive the Azure authentication settings from the Grafana server. **Solutions:** 1. Check if you've customized the `forward_settings_to_plugins` setting under `[azure]` in your Grafana `.ini` file. 1. If customized, ensure `grafana-azure-monitor-datasource` is listed. For example: ```ini [azure] managed_identity_enabled = true forward_settings_to_plugins = grafana-azure-monitor-datasource ``` 1. By default, Grafana includes all Grafana Labs Azure plugins. If you haven't customized this setting, the default includes the Azure Monitor plugin. 1. Restart Grafana after making changes. ## Query errors These errors occur when executing queries against Azure Monitor services. ### "No data" or empty results **Symptoms:** - Query executes without error but returns no data - Charts show "No data" message **Possible causes and solutions:** | Cause | Solution | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | Time range doesn't contain data | Expand the dashboard time range or verify data exists in Azure Portal. | | Wrong resource selected | Verify you've selected the correct subscription, resource group, and resource. | | Metric not available for resource | Not all metrics are available for all resources. Check available metrics in Azure Portal under the resource's **Metrics** blade. | | Metric has no values | Some metrics only populate under certain conditions (for example, error counts when errors occur). | | Permissions issue | Verify the identity has read access to the specific resource. | ### "Bad request" or "Invalid query" **Symptoms:** - Query fails with 400 error - Error message indicates query syntax issues **Solutions for Logs queries:** 1. Validate your KQL syntax in the Azure Portal Log Analytics query editor. 1. Check for typos in table names or column names. 1. Ensure referenced tables exist in the selected workspace. 1. Verify the time range is valid (not in the future, not too far in the past for data retention). **Solutions for Metrics queries:** 1. Verify the metric name is valid for the selected resource type. 1. Check that dimension filters use valid dimension names and values. 1. Ensure the aggregation type is supported for the selected metric. ### "Resource not found" **Symptoms:** - Query fails with 404 error - Resource picker shows resources that can't be queried **Solutions:** 1. Verify the resource still exists in Azure (it may have been deleted or moved). 1. Check that the subscription is correct. 1. Refresh the resource picker by re-selecting the subscription. 1. Verify the identity has access to the resource's resource group. ### Logs query timeout **Symptoms:** - Query runs for a long time then fails with `DatasourceError` or `context deadline exceeded` - Alert evaluations fail with timeout errors - Error mentions timeout or query limits **Cause:** The default data source timeout in Grafana is 30 seconds, but Azure Log Analytics queries can take up to 3 minutes on the Azure side, especially for complex KQL queries or large time ranges. **Solutions:** 1. **Increase the data source timeout.** Open the Azure Monitor data source settings, scroll to the **Misc** section, and increase the **Timeout** to `300` (seconds). Refer to [Query timeout](https://grafana.com/docs/grafana//datasources/azure-monitor/configure/#query-timeout) for details. 1. Narrow the time range to reduce data volume. 1. Add filters to reduce the result set. 1. Use `summarize` to aggregate data instead of returning raw rows. 1. Consider using Basic Logs for large datasets (if enabled). 1. Break complex queries into smaller parts. 1. For alerting, ensure the alert evaluation interval is long enough to accommodate the query duration plus the configured timeout. ### Logs query results are truncated **Symptoms:** - A Logs query returns exactly 30,000 rows even though more data exists - Grafana displays a notice that the row limit was reached **Cause:** When a Logs query uses the **Logs** result format, Grafana truncates the results to 30,000 rows to prevent the browser tab from crashing, because the logs visualization renders every row at once. **Solutions:** 1. Aggregate the data in your query with `summarize` instead of returning raw rows. 1. Narrow the dashboard time range to reduce the number of rows returned. 1. Add filters to your KQL query to return only the rows you need. 1. To remove the limit, enable the `azureMonitorDisableLogLimit` [feature toggle](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/feature-toggles/). Disabling the limit can degrade browser performance for very large result sets. ### "Metrics not available" for a resource **Symptoms:** - Resource appears in picker but no metrics are listed - Metric dropdown is empty **Solutions:** 1. Verify the resource type supports Azure Monitor metrics. 1. Check if the resource is in a region that supports metrics. 1. Some resources require diagnostic settings to emit metrics—configure these in Azure Portal. 1. Try selecting a different namespace for the resource. ## Azure Resource Graph errors These errors are specific to Azure Resource Graph (ARG) queries. ### "Query execution failed" **Symptoms:** - ARG query fails with execution errors - Results don't match expected resources **Solutions:** 1. Validate query syntax in Azure Portal Resource Graph Explorer. 1. Check that you have access to the subscriptions being queried. 1. Verify table names are correct (for example, `Resources`, `ResourceContainers`). 1. Some ARG features require specific permissions, check [ARG documentation](https://docs.microsoft.com/en-us/azure/governance/resource-graph/). ### Query returns incomplete results **Symptoms:** - Not all expected resources appear in results - Results seem truncated **Solutions:** 1. ARG queries are paginated. The data source handles pagination automatically, but very large result sets may be limited. 1. Add filters to reduce result set size. 1. Verify you have access to all subscriptions containing the resources. ## Application Insights Traces errors These errors are specific to the Traces query type. ### "No traces found" **Symptoms:** - Trace query returns empty results - Operation ID search finds nothing **Solutions:** 1. Verify the Application Insights resource is collecting trace data. 1. Check that the time range includes when the traces were generated. 1. Ensure the Operation ID is correct (copy directly from another trace or log). 1. Verify the identity has access to the Application Insights resource. ## Basic Logs errors These errors are specific to Basic Logs queries. ### Basic Logs toggle is not available **Symptoms:** - The **Analytics / Basic** toggle doesn't appear in the Logs query editor - Unable to switch to Basic Logs mode **Solutions:** 1. Verify **Enable Basic Logs** is toggled on in the data source configuration. Refer to [Enable Basic Logs](https://grafana.com/docs/grafana//datasources/azure-monitor/configure/#enable-basic-logs). 1. Basic Logs isn't available when creating alert rules. If you're in the alerting UI, use Analytics mode instead. ### Basic Logs query returns errors **Symptoms:** - Query fails with "BadArgumentError" or similar errors - Query returns unexpected results **Solutions:** 1. Ensure you've selected only a single workspace resource. Basic Logs doesn't support multi-resource queries. 1. Remove any time-range filters from the query itself. Basic Logs always uses the dashboard time range. 1. Verify the table you're querying is configured with the Basic or Auxiliary log plan in Azure. Not all tables support Basic Logs. 1. Check that your KQL query uses only [supported operators](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1#limitations). ### Unexpected costs from Basic Logs **Solutions:** 1. Basic Logs queries are billed per query by Azure, separate from Grafana costs. Review the [Azure pricing documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1). 1. Check your query frequency. Dashboard auto-refresh and variable changes can trigger repeated queries. 1. Consider switching infrequently queried tables back to Analytics mode to avoid per-query charges. ## Alerting errors These errors occur when using Azure Monitor data in Grafana alert rules or recording rules. For general alerting configuration, refer to the [Azure Monitor alerting documentation](https://grafana.com/docs/grafana//datasources/azure-monitor/alerting/). ### Alert failure with Current User authentication **Symptoms:** - Alert rules and recording rules fail with authentication errors - Dashboard queries work, but alerting evaluations do not - Error messages reference missing or invalid credentials during background evaluation **Cause:** Current User authentication relies on the logged-in user's credentials, which aren't available for background operations like alerting. **Solutions:** 1. Configure **fallback service credentials** in the data source settings. Typically this means adding an App Registration (client secret) as the fallback. 1. Verify the fallback credentials have the required Azure RBAC permissions on the resources used in your alert rules. 1. Refer to [Limitations and fallback credentials](https://grafana.com/docs/grafana//datasources/azure-monitor/configure/#limitations-and-fallback-credentials) for configuration details. ### Alert evaluation timeout **Symptoms:** - Alert rules fail with `DatasourceError` or `context deadline exceeded` - Alerts work intermittently, failing when queries take longer than usual **Solutions:** 1. **Increase the data source timeout.** The default is 30 seconds, but Azure Log Analytics queries can take up to 3 minutes. Open the Azure Monitor data source settings, scroll to **Misc**, and set **Timeout** to `300`. Refer to [Query timeout](https://grafana.com/docs/grafana//datasources/azure-monitor/configure/#query-timeout). 1. Ensure the alert evaluation interval is long enough to accommodate the query duration. 1. Simplify complex KQL queries used in alert rules. 1. Reduce the time range in Log Analytics queries. ## Template variable errors For general template variable configuration, refer to the [template variables documentation](https://grafana.com/docs/grafana//datasources/azure-monitor/template-variables/). ### "Properties found in series but missing `valueProp` and `textProp`" **Symptoms:** - Template variables using query types such as **Resource Groups**, **Resource Names**, or **Subscriptions** fail with this error - Variables that previously worked stop returning values after a Grafana upgrade - Dashboards that rely on cascading variables break **Cause:** This error occurs when the Azure Monitor plugin's variable query response format doesn't match what Grafana expects. This was a known bug that affected specific Grafana versions. **Solutions:** 1. **Check your Grafana version.** This issue was a software bug fixed in later releases. Update to the latest Grafana patch release for your version. 1. **Grafana Cloud users:** If you're on a release channel that doesn't yet include the fix, switch to the **stable** channel in your Cloud stack settings (**Stack Management** > **General** > **Release channel**) to receive the latest stable patch. 1. **Self-hosted users:** Download the latest patch release from the [Grafana download page](https://grafana.com/grafana/download). 1. After upgrading, reload the dashboard and re-run the variable queries. ### Variables return no values **Solutions:** 1. Verify the data source connection is working (test it in the data source settings). 1. Check that parent variables (for cascading variables) have valid selections. 1. Verify the identity has permissions to list the requested resources. 1. For Logs variables, ensure the KQL query returns a single column. ### Variables stop working after a Grafana upgrade **Symptoms:** - Template variables that worked before an upgrade return errors or empty results - Dashboard variable dropdowns show no options or display error indicators **Solutions:** 1. Check the [Grafana release notes](https://grafana.com/docs/grafana//whatsnew/) for breaking changes related to template variables or the Azure Monitor data source. 1. Verify the Azure Monitor plugin version is compatible with your Grafana version. Go to **Administration** > **Plugins and data** > **Plugins**, search for "Azure Monitor", and check the version. 1. Clear the browser cache and reload the dashboard. 1. Re-open the variable configuration and click **Run query** to verify it returns expected results. 1. If the issue persists, check for known bugs in the [Grafana GitHub issues](https://github.com/grafana/grafana/issues?q=is%3Aissue+label%3A%22datasource%2Fazure-monitor%22+template+variable) and consider rolling back to the previous Grafana version. ### Variables are slow to load **Solutions:** 1. Set variable refresh to **On dashboard load** instead of **On time range change**. 1. Reduce the scope of variable queries (for example, filter by resource group instead of entire subscription). 1. For Logs variables, optimize the KQL query to return results faster. ## Enable debug logging To capture detailed error information for troubleshooting Azure Monitor issues: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Restart Grafana for the change to take effect. 1. Reproduce the issue and review the logs at `/var/log/grafana/grafana.log` (or your configured log location). 1. Look for entries containing `azuremonitor` or `azure` for Azure Monitor-specific request and response details. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you've tried the solutions in this document and still encounter issues: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review the [Azure Monitor data source GitHub issues](https://github.com/grafana/grafana/issues) for known bugs. 1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user. 1. When reporting issues, include: - Grafana version - Error messages (redact sensitive information) - Steps to reproduce - Relevant configuration (redact credentials) # Elasticsearch alerting You can use Grafana Alerting with Elasticsearch to create alerts based on your Elasticsearch data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met. For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). ## Before you begin Before creating alerts with Elasticsearch, ensure you have: - An Elasticsearch data source configured in Grafana - Appropriate permissions to create alert rules - Understanding of the metrics you want to monitor ## Supported query types Elasticsearch alerting works best with **metrics queries** that return time series data. To create a valid alert query: - Use a **Date histogram** as the last bucket aggregation (under **Group by**) - Select appropriate metric aggregations (Count, Average, Sum, Min, Max, etc.) Queries that return time series data allow Grafana to evaluate values over time and trigger alerts when thresholds are crossed. ### Query types and alerting compatibility | Query type | Alerting support | Notes | | ------------------------------ | ---------------- | ----------------------------------------------------------- | | Metrics with Date histogram | Supported | Recommended for alerting | | Metrics without Date histogram | Limited | May not evaluate correctly over time | | Logs | Not supported | Use metrics queries instead | | Raw data | Not supported | Use metrics queries instead | | Raw document (deprecated) | Not supported | Deprecated since Grafana v10.1. Use metrics queries instead | ## Create an alert rule To create an alert rule using Elasticsearch: 1. Navigate to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for the alert rule. 1. Select your **Elasticsearch** data source. 1. Build your query using the query editor: - Add metric aggregations (for example, Average, Count, Sum) - Add a Date histogram under **Group by** - Optionally add filters using Lucene query syntax 1. Configure the alert condition (for example, when the average is above a threshold). 1. Set the evaluation interval and pending period. 1. Configure notifications and labels. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example alert queries The following examples show common alerting scenarios with Elasticsearch. ### Alert on high error count Monitor the number of error-level log entries: 1. **Query:** `level:error` 1. **Metric:** Count 1. **Group by:** Date histogram (interval: 1m) 1. **Condition:** When count is above 100 ### Alert on average response time Monitor API response times: 1. **Query:** `type:api_request` 1. **Metric:** Average on field `response_time` 1. **Group by:** Date histogram (interval: 5m) 1. **Condition:** When average is above 500 (milliseconds) ### Alert on unique user count drop Detect drops in active users: 1. **Query:** `*` (all documents) 1. **Metric:** Unique count on field `user_id` 1. **Group by:** Date histogram (interval: 1h) 1. **Condition:** When unique count is below 100 ## Limitations When using Elasticsearch with Grafana Alerting, be aware of the following limitations: ### Template variables not supported Alert queries cannot contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$hostname` or `$environment` won't be resolved. If your dashboard query uses template variables, create a separate query for alerting with hard coded values. ### Logs queries not supported Queries using the **Logs** metric type cannot be used for alerting. Convert your query to use metric aggregations with a Date histogram instead. ### Query complexity Complex queries with many nested aggregations may timeout or fail to evaluate. Simplify queries for alerting by: - Reducing the number of bucket aggregations - Using appropriate time intervals - Adding filters to limit the data scanned ## Best practices Follow these best practices when creating Elasticsearch alerts: - **Use specific filters:** Add Lucene query filters to focus on relevant data and improve query performance. - **Choose appropriate intervals:** Match the Date histogram interval to your evaluation frequency. - **Test queries first:** Verify your query returns expected results in Explore before creating an alert. - **Set realistic thresholds:** Base alert thresholds on historical data patterns. - **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor. # Elasticsearch annotations Annotations overlay event data on your dashboard graphs, helping you correlate log events with metrics. You can use Elasticsearch as a data source for annotations to display events such as deployments, alerts, or other significant occurrences on your visualizations. For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana//visualizations/dashboards/build-dashboards/annotate-visualizations/). ## Before you begin Before creating Elasticsearch annotations, ensure you have: - An Elasticsearch data source configured in Grafana - Documents in Elasticsearch containing event data with timestamp fields - Read access to the Elasticsearch index containing your events ## Create an annotation query To add an Elasticsearch annotation to your dashboard: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Annotation query**. 1. Enter a name for the annotation query. 1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox. 1. Select a color for the annotation event markers. 1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed. 1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed. 1. Click **Open query editor** to open the **Annotation Query** dialog box. 1. Select your **Elasticsearch** data source from the **Data source** drop-down list. 1. Configure the annotation query and field mappings. 1. (Optional) Click **Test annotation query** to ensure that the query is working properly. 1. Click **Close** when you've completed the query setup. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. ## Query Use the query field to filter which Elasticsearch documents appear as annotations. The query uses [Lucene query syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax). **Examples:** | Query | Description | | ---------------------------------------- | ---------------------------------------------------- | | `*` | Matches all documents. | | `type:deployment` | Shows only deployment events. | | `level:error OR level:critical` | Shows error and critical events. | | `service:api AND environment:production` | Shows events for a specific service and environment. | | `tags:release` | Shows events tagged as releases. | You can use template variables in your annotation queries. For example, `service:$service` filters annotations based on the selected service variable. ## Field mappings Field mappings tell Grafana which Elasticsearch fields contain the annotation data. ### Time The **Time** field specifies which field contains the annotation timestamp. - **Default:** `@timestamp` - **Format:** The field must contain a date value that Elasticsearch recognizes. ### Time End The **Time End** field specifies a field containing the end time for range annotations. Range annotations display as a shaded region on the graph instead of a single vertical line. - **Default:** Empty (single-point annotations) - **Use case:** Display maintenance windows, incidents, or any event with a duration. ### Text The **Text** field specifies which field contains the annotation description displayed when you hover over the annotation. - **Default:** `tags` - **Tip:** Use a descriptive field like `message`, `description`, or `summary`. ### Tags The **Tags** field specifies which field contains tags for the annotation. Tags help categorize and filter annotations. - **Default:** Empty - **Format:** The field can contain either a comma-separated string or an array of strings. ## Example: Deployment annotations To display deployment events as annotations: 1. Create an annotation query with the following settings: - **Query:** `type:deployment` - **Time:** `@timestamp` - **Text:** `message` - **Tags:** `environment` This configuration displays deployment events with their messages as the annotation text and environments as tags. ## Example: Range annotations for incidents To display incidents with duration: 1. Create an annotation query with the following settings: - **Query:** `type:incident` - **Time:** `start_time` - **Time End:** `end_time` - **Text:** `description` - **Tags:** `severity` This configuration displays incidents as shaded regions from their start time to end time. # Configure the Elasticsearch data source Grafana ships with the Elasticsearch data source preinstalled. You can create a variety of queries to visualize logs or metrics stored in Elasticsearch, and annotate graphs with log events stored in Elasticsearch. For details about updating the standalone plugin, refer to [Plugin updates](https://grafana.com/docs/grafana//datasources/elasticsearch/#plugin-updates). For instructions on how to add a data source to Grafana, refer to the [administration documentation](https://grafana.com/docs/grafana//administration/data-source-management/). Administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system. ## Before you begin To configure the Elasticsearch data source, you need: - **Grafana administrator permissions:** Only users with the organization `administrator` role can add data sources. - **A supported Elasticsearch version:** v7.17 or later, v8.x, v9.x or Elastic Cloud Serverless. - **Elasticsearch server URL:** The HTTP or HTTPS endpoint for your Elasticsearch instance, including the port (default: `9200`). - **Authentication credentials:** Depending on your Elasticsearch security configuration, you need one of the following: - Username and password for basic authentication - API key - No credentials (if Elasticsearch security is disabled) - **Network access:** Grafana must be able to reach your Elasticsearch server. For Grafana Cloud, consider using [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your Elasticsearch instance is in a private network. ## Elasticsearch permissions When Elasticsearch security features are enabled, you must configure the following cluster privileges for the user or API key that Grafana uses to connect: - **monitor** - Required for the connection health check. Grants access to the `_cluster/health` endpoint and retrieves the version information of the connected Elasticsearch instance. - **view_index_metadata** - Required for accessing mapping definitions of indices. - **read** - Grants the ability to perform search and retrieval operations on indices. This is essential for querying and extracting data from the cluster. ## Add the data source To add the Elasticsearch data source, complete the following steps: 1. Click **Connections** in the left-side menu. 1. Under **Connections**, click **Add new connection**. 1. Enter `Elasticsearch` in the search bar. 1. Click **Elasticsearch** under the **Data source** section. 1. Click **Add new data source** in the upper right. You are taken to the **Settings** tab where you set up your Elasticsearch configuration. ## Configuration options Configure the following basic settings for the Elasticsearch data source: - **Name** - The data source name. This is how you refer to the data source in panels and queries. Examples: `elastic-1`, `elasticsearch_metrics`. - **Default** - Toggle on to make this the default data source. New panels and Explore queries use the default data source. ## Connection Set the HTTP endpoint Grafana uses to reach your Elasticsearch cluster. - **URL** - The URL of your Elasticsearch server, including the port. Examples: `http://localhost:9200`, `http://elasticsearch.example.com:9200`. ## Authentication Choose how Grafana authenticates to Elasticsearch. Select an authentication method from the drop-down menu: - **Basic authentication** - Enter the username and password for your Elasticsearch user. - **Forward OAuth identity** - Forward the OAuth access token (and the OIDC ID token if available) of the user querying the data source. - **No authentication** - Connect without credentials. Only use this option if your Elasticsearch instance doesn't require authentication. - **Serverless API Key** - Enter your API key to connect to an Elastic serverless instance. For instructions on locating or generating your API key, refer to [Elastic Cloud API keys](https://www.elastic.co/docs/deploy-manage/api-keys/elastic-cloud-api-keys). ### API key authentication To authenticate using an Elasticsearch API key, select **No authentication** and configure the API key using HTTP headers: 1. In the **HTTP headers** section, click **+ Add header**. 1. Set **Header** to `Authorization`. 1. Set **Value** to `ApiKey `, replacing `` with your base64-encoded Elasticsearch API key. For information about creating API keys, refer to the [Elasticsearch API keys documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html). ### Amazon OpenSearch Service and SigV4 Amazon OpenSearch Service is the successor to Amazon Elasticsearch Service. For Amazon OpenSearch Service, use the [OpenSearch data source](https://grafana.com/docs/plugins/grafana-opensearch-datasource/latest/) instead of this Elasticsearch data source. If you still connect the Grafana Elasticsearch data source to a domain that requires AWS Signature Version 4 (AWS SigV4), for example a legacy Amazon Elasticsearch Service domain or another SigV4-protected OpenSearch-compatible endpoint, you must sign all requests with SigV4. For details on AWS SigV4, refer to the [AWS documentation](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html). To sign requests, enable SigV4 in the Grafana [configuration](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#sigv4_auth_enabled). After SigV4 is enabled, configure it on the Elasticsearch data source configuration page. For more information about AWS authentication options, refer to [AWS authentication](https://grafana.com/docs/grafana//datasources/aws-cloudwatch/aws-authentication/). {{< figure src="/static/img/docs/v73/elasticsearch-sigv4-config-editor.png" max-width="500px" class="docs-image--no-shadow" caption="SigV4 configuration for AWS-hosted Elasticsearch-compatible domains" >}} ### TLS settings {{< admonition type="note" >}} Use TLS (Transport Layer Security) for an additional layer of security when working with Elasticsearch. For information on setting up TLS encryption with Elasticsearch, refer to [Configure TLS](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/configuring-tls.html#configuring-tls). You must add TLS settings to your Elasticsearch configuration file **prior** to setting these options in Grafana. {{< /admonition >}} - **Add self-signed certificate** - Check the box to authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file. Required for verifying self-signed TLS certificates. - **TLS client authentication** - Check the box to authenticate with the TLS client, where the server authenticates the client. Add the `Server name`, `Client certificate` and `Client key`. The **ServerName** is used to verify the hostname on the returned certificate. The **Client certificate** can be generated from a Certificate Authority (CA) or be self-signed. The **Client key** can also be generated from a Certificate Authority (CA) or be self-signed. The client key encrypts the data between client and server. - **Skip TLS certificate validation** - Check the box to bypass TLS certificate validation. Skipping TLS certificate validation is not recommended unless absolutely necessary or for testing purposes. ### HTTP headers Click **+ Add header** to add one or more HTTP headers. HTTP headers pass additional context and metadata about the request/response. - **Header** - Add a custom header. This allows custom headers to be passed based on the needs of your Elasticsearch instance. - **Value** - The value of the header. ## Additional settings Additional settings are optional settings that can be configured for more control over your data source. ### Advanced HTTP settings - **Allowed cookies** - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default. - **Timeout** - The HTTP request timeout. This must be in seconds. There is no default, so this setting is up to you. ### Elasticsearch details The following settings are specific to the Elasticsearch data source. - **Index name** - The name of your Elasticsearch index. You can use the following formats: - **Wildcard patterns** - Use `*` to match multiple indices. Examples: `logs-*`, `metrics-*`, `filebeat-*`. - **Time patterns** - Use date placeholders for time-based indices. Wrap the fixed portion in square brackets. Examples: `[logstash-]YYYY.MM.DD`, `[metrics-]YYYY.MM`. - **Specific index** - Enter the exact index name. Example: `application-logs`. - **Cross-cluster search** - Use the `cluster:index` pattern to query remote clusters configured in Elasticsearch. Example: `logs-cluster:logs-*`. Cross-cluster search is always available. No feature toggle is required. - **Pattern** - Select the matching pattern if you use a time pattern in your index name. Options include: - no pattern - hourly - daily - weekly - monthly - yearly Only select a pattern option if you have specified a time pattern in the Index name field. - **Time field name** - Name of the time field. The default value is `@timestamp`. You can enter a different name. - **Max concurrent shard requests** - Sets the number of shards being queried at the same time. The default is `5`. For more information on shards, refer to the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/scalability.html#scalability). - **Min time interval** - Defines a lower limit for the auto group-by time interval. This value **must** be formatted as a number followed by a valid time identifier: | Identifier | Description | | ---------- | ----------- | | `y` | year | | `M` | month | | `w` | week | | `d` | day | | `h` | hour | | `m` | minute | | `s` | second | | `ms` | millisecond | We recommend setting this value to match your Elasticsearch write frequency. For example, set this to `1m` if Elasticsearch writes data every minute. You can also override this setting in a dashboard panel under its data source options. The default is `10s`. - **X-Pack enabled** - Toggle to enable `X-Pack`-specific features and options, which provide the [query editor](https://grafana.com/docs/grafana//datasources/elasticsearch/query-editor/) with additional aggregations, such as `Rate` and `Top Metrics`. - **Include frozen indices** - Toggle on when the `X-Pack enabled` setting is active. Includes frozen indices in searches. You can configure Grafana to include [frozen indices](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/frozen-indices.html) when performing search requests. {{< admonition type="note" >}} Frozen indices are [deprecated in Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/frozen-indices.html) since v7.14. {{< /admonition >}} ### Logs Configure which fields the data source uses for log messages and log levels. - **Message field name** - The field that contains the log message content. - **Level field name** - The field that contains log level or severity information. When specified, Grafana uses this field to determine the log level and color-code each log line. If the log doesn't have a level field, Grafana tries to match the content against [supported expressions](https://grafana.com/docs/grafana//explore/logs-integration/#log-level). If Grafana can't determine the log level, it displays as unknown. ### Data links Data links create a link from a specified field that can be accessed in the Explore logs view. You can add multiple data links by clicking **+ Add**. Each data link configuration consists of: - **Field** - Sets the name of the field used by the data link. - **URL/query** - Sets the full link URL if the link is external. If the link is internal, this input serves as a query for the target data source. In both cases, you can interpolate the value from the field with the `${__value.raw }` macro. - **URL Label** (Optional) - Sets a custom display label for the link. The link label defaults to the full external URL or name of the linked internal data source and is overridden by this setting. - **Internal link** - Toggle on to set an internal link. For an internal link, you can select the target data source with a data source selector. This supports only tracing data sources. ## Private data source connect (PDC) and Elasticsearch Use private data source connect (PDC) to connect to and query data within a secure network without opening that network to inbound traffic from Grafana Cloud. Refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) for more information on how PDC works and [Configure Grafana private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc) for steps on setting up a PDC connection. If you use PDC with SigV4 (AWS Signature Version 4 Authentication), the PDC agent must allow internet egress to `sts..amazonaws.com:443`. - **Private data source connect** - Click in the box to set the default PDC connection from the drop-down menu or create a new connection. After you have configured your Elasticsearch data source options, click **Save & test** to test the connection. A successful connection displays the following message: `Elasticsearch data source is healthy.` ## Provision the data source You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana//administration/provisioning/). {{< admonition type="note" >}} The previously used `database` field has now been [deprecated](https://github.com/grafana/grafana/pull/58647). Use the `index` field in `jsonData` to store the index name. Refer to the examples below. {{< /admonition >}} ### Basic provisioning ```yaml apiVersion: 1 datasources: - name: Elastic type: elasticsearch access: proxy url: http://localhost:9200 jsonData: index: '[metrics-]YYYY.MM.DD' interval: Daily timeField: '@timestamp' ``` ### Provision for logs ```yaml apiVersion: 1 datasources: - name: elasticsearch-v7-filebeat type: elasticsearch access: proxy url: http://localhost:9200 jsonData: index: '[filebeat-]YYYY.MM.DD' interval: Daily timeField: '@timestamp' logMessageField: message logLevelField: fields.level dataLinks: - datasourceUid: my_jaeger_uid # Target UID needs to be known field: traceID url: '$${__value.raw}' # Careful about the double "$$" because of env var expansion ``` ## Provision the data source using Terraform You can provision the Elasticsearch data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation. ### Basic Terraform example The following example creates a basic Elasticsearch data source for metrics: ```hcl resource "grafana_data_source" "elasticsearch" { name = "Elasticsearch" type = "elasticsearch" url = "http://localhost:9200" json_data_encoded = jsonencode({ index = "[metrics-]YYYY.MM.DD" interval = "Daily" timeField = "@timestamp" }) } ``` ### Terraform example for logs The following example creates an Elasticsearch data source configured for logs with a data link to Jaeger: ```hcl resource "grafana_data_source" "elasticsearch_logs" { name = "Elasticsearch Logs" type = "elasticsearch" url = "http://localhost:9200" json_data_encoded = jsonencode({ index = "[filebeat-]YYYY.MM.DD" interval = "Daily" timeField = "@timestamp" logMessageField = "message" logLevelField = "fields.level" dataLinks = [ { datasourceUid = grafana_data_source.jaeger.uid field = "traceID" url = "$${__value.raw}" } ] }) } ``` ### Terraform example with basic authentication The following example includes basic authentication: ```hcl resource "grafana_data_source" "elasticsearch_auth" { name = "Elasticsearch" type = "elasticsearch" url = "http://localhost:9200" basic_auth_enabled = true basic_auth_username = "elastic_user" secure_json_data_encoded = jsonencode({ basicAuthPassword = var.elasticsearch_password }) json_data_encoded = jsonencode({ index = "[metrics-]YYYY.MM.DD" interval = "Daily" timeField = "@timestamp" }) } ``` For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). # Elasticsearch query editor Grafana provides a query editor for Elasticsearch. Elasticsearch queries are in Lucene format. For more information about query syntax, refer to [Lucene query syntax](https://www.elastic.co/guide/en/kibana/current/lucene-query.html) and [Query string syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax). {{< admonition type="note" >}} When composing Lucene queries, ensure that you use uppercase boolean operators: `AND`, `OR`, and `NOT`. Lowercase versions of these operators are not supported by the Lucene query syntax. {{< /admonition >}} {{< figure src="/static/img/docs/elasticsearch/elastic-query-editor-10.1.png" max-width="800px" class="docs-image--no-shadow" caption="Elasticsearch query editor" >}} For general documentation on querying data sources in Grafana, including options and functions common to all query editors, refer to [Query and transform data](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/). ## Aggregation types Elasticsearch groups aggregations into three categories: - **Bucket** - Bucket aggregations don't calculate metrics, they create buckets of documents based on field values, ranges and a variety of other criteria. Refer to [Bucket aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket.html) for additional information. Use bucket aggregations under `Group by` when creating a metrics query in the query builder. - **Metrics** - Metrics aggregations perform calculations such as sum, average, min, etc. They can be single-value or multi-value. Refer to [Metrics aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics.html) for additional information. Use metrics aggregations in the metrics query type in the query builder. - **Pipeline** - Pipeline aggregations work on the output of other aggregations rather than on documents or fields. Refer to [Pipeline aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline.html) for additional information. ## Select a query type There are two types of queries you can create with the Elasticsearch query builder. Each type is explained in detail below. Use the **Preserve query** toggle to keep your Lucene query when you switch between **Metrics**, **Logs**, **Raw Data**, and **Raw Document**. When the toggle is off (the default for new queries), switching query types clears the Lucene query. Grafana remembers your last Preserve query choice in the browser for new queries. ### Metrics query type Metrics queries aggregate data and produce calculations such as count, min, max, and more. Click the metric box to view options in the drop-down menu. The default is `count`. - **Alias** - Aliasing only applies to **time series queries**, where the last group is `date histogram`. This is ignored for any other type of query. - **Metric** - Metrics aggregations include: - count - refer to [Value count aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-valuecount-aggregation.html) - average - refer to [Avg aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-avg-aggregation.html) - sum - refer to [Sum aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-sum-aggregation.html) - max - refer to [Max aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-max-aggregation.html) - min - refer to [Min aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-min-aggregation.html) - extended stats - refer to [Extended stats aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-extendedstats-aggregation.html) - percentiles - refer to [Percentiles aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-percentile-aggregation.html) - unique count - refer to [Cardinality aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html) - top metrics - refer to [Top metrics aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-metrics.html) - rate - refer to [Rate aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-rate-aggregation.html) - **Sibling bucket aggregations** - These metrics compute an outer statistic over a per-group inner statistic within each time bucket. For example, **Sum Bucket** with an inner **Max** produces a sum of maxima across hosts. The following options are available: - Sum Bucket - refer to [Sum bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-sum-bucket-aggregation.html) - Max Bucket - refer to [Max bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-max-bucket-aggregation.html) - Min Bucket - refer to [Min bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-min-bucket-aggregation.html) - Average Bucket - refer to [Avg bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-avg-bucket-aggregation.html) - **Pipeline aggregations** - Pipeline aggregations work on the output of other aggregations rather than on documents. The following pipeline aggregations are available: - moving function - Calculates a value based on a sliding window of aggregated values. Refer to [Moving function aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-movfn-aggregation.html). - derivative - Calculates the derivative of a metric. Refer to [Derivative aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-derivative-aggregation.html). - cumulative sum - Calculates the cumulative sum of a metric. Refer to [Cumulative sum aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-cumulative-sum-aggregation.html). - serial difference - Calculates the difference between values in a time series. Refer to [Serial differencing aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-serialdiff-aggregation.html). - bucket script - Executes a script on metric values from other aggregations. Refer to [Bucket script aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-script-aggregation.html). You can select multiple metrics and group by multiple terms or filters when using the Elasticsearch query editor. Use the **+ sign** to the right to add multiple metrics to your query. Click on the **eye icon** next to **Metric** to hide metrics, and the **garbage can icon** to remove metrics. - **Group by options** - Create multiple group by options when constructing your Elasticsearch query. Date histogram is the default option. The following options are available in the drop-down menu: - terms - refer to [Terms aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html). - filter - refer to [Filter aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html). - `geo hash grid` - refer to the [`geohash_grid` aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-geohashgrid-aggregation.html). - date histogram - for time series queries. Refer to [Date histogram aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html). - histogram - Depicts frequency distributions. Refer to [Histogram aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-histogram-aggregation.html). - nested (experimental) - Refer to [Nested aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-nested-aggregation.html). Each group by option has a different subset of options to further narrow your query. The following options are specific to the **date histogram** bucket aggregation option. - **Time field** - The field used for time-based queries. The default can be set when configuring the data source in the **Time field name** setting under [Elasticsearch details](https://grafana.com/docs/grafana//datasources/elasticsearch/configure/#elasticsearch-details). The default is `@timestamp`. - **Interval** - The time interval for grouping data. Select from the drop-down menu or enter a custom interval such as `30d` (30 days). The default is `Auto`. - **Min doc count** - The minimum number of documents required to include a bucket. The default is `0`. - **Trim edges** - Removes partial buckets at the edges of the time range. The default is `0`. - **Offset** - Shifts the start of each bucket by the specified duration. Use positive (`+`) or negative (`-`) values. Examples: `1h`, `5s`, `1d`. - **Timezone** - The timezone for date calculations. The default is `Coordinated Universal Time`. Configure the following options for the **terms** bucket aggregation option: - **Order** - Sets the order of data. Options are `top` or `bottom.` - **Size** - Limits the number of documents, or size of the dataset. You can set a custom number or `no limit`. - **Min doc count** - The minimum amount of data to include in your query. The default is `0`. - **Order by** - Order terms by `term value`, `doc count` or `count`. - **Missing** - Defines how documents missing a value should be treated. Missing values are ignored by default, but they can be treated as if they had a value. Refer to [Missing value](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_missing_value_5) in the Elasticsearch documentation for more information. Configure the following options for the **filters** bucket aggregation option: - **Query** - Specify the query to create a bucket of documents (data). Examples are `hostname:"hostname1"`, `product:"widget5"`. Use the \* wildcard to match any number of characters. - **Label** - Add a label or name to the bucket. Configure the following options for the **`geo hash grid`** bucket aggregation option: - **Precision** - Specifies the number of characters of the `geo hash`. Configure the following options for the **histogram** bucket aggregation option: - **Interval** - The numeric interval for grouping values into buckets. - **Min doc count** - The minimum number of documents required to include a bucket. The default is `0`. The **nested** group by option is currently experimental, you can select a field and then settings specific to that field. Click the **+ sign** to add multiple group by options. The data is grouped in order (first by, then by). {{< figure src="/static/img/docs/elasticsearch/group-by-then-by-10.2.png" max-width="850px" class="docs-image--no-shadow" caption="Group by options" >}} ### Logs query type Logs queries analyze Elasticsearch log data. You can configure the following options: - **Logs Options/Limit** - Limits the number of logs to analyze. The default is `500`. ## Query options Open the **Options** section in the query editor to configure query-level settings: - **Limit** or **Size** - For Logs and Raw Data queries, sets the maximum number of documents to return. The default is `500`. - **Include runtime fields** - When enabled, runtime fields defined in the index mapping are included in the response. This option is available in Builder mode. ## Raw query editor The raw query editor lets you write Elasticsearch queries using the native [Elasticsearch Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html). ### Switch between Builder and Code modes To access the raw query editor, click the **Code** toggle in the top-right corner of the query editor. You can switch between **Builder** and **Code** modes: - **Builder** - Visual query builder with dropdown menus and forms - **Code** - JSON editor for writing raw Elasticsearch DSL queries ### Write raw DSL queries When in Code mode, you can write complete Elasticsearch query DSL in JSON format. The editor provides: - **Syntax highlighting** for JSON - **Auto-formatting** - Click the **Format** button or press `Shift+Alt+F` to format your query - **Keyboard shortcuts** - Press `Ctrl+Enter` (or `Cmd+Enter` on Mac) to run the query - **Real-time validation** - Invalid JSON is highlighted with error messages ### Time range handling If you want to filter by time range in a dashboard, you need to use the `$__from` and `$__to` macros in your raw DSL. An example query applying dashboard time range using the `@timestamp` field: ```json { "query": { "bool": { "must": [ { "range": { "@timestamp": { "gte": "$__from", "lte": "$__to", "format": "epoch_millis" } } } ] } } } ``` ### Supported query types The raw query editor supports all query types: - **Metrics queries** are used to query time series data with aggregations. The query parser automatically extracts bucket and metric aggregations from your DSL and uses them for response processing. - **Logs queries** are used to query log data. ## ES|QL query editor The ES|QL query editor lets you query Elasticsearch using [ES|QL (Elasticsearch Query Language)](https://www.elastic.co/docs/reference/query-languages/esql), a pipe-based query language. Unlike Lucene queries that rely on aggregation configuration in the builder UI, ES|QL lets you express filtering, aggregation, and transformation in a single query string. For an introduction to ES|QL syntax and concepts, refer to [Get started with ES|QL queries](https://www.elastic.co/docs/reference/query-languages/esql/esql-getting-started) in the Elasticsearch documentation. When an ES|QL query does not include a time filter, Grafana automatically adds a time range filter based on the dashboard or Explore time picker. ### Index selection How the editor handles index selection depends on your data source configuration: - **No index name configured:** You specify which index to query using the `FROM` command directly in your ES|QL query. This lets you query any index without creating a separate data source for each one. - **Index name configured:** The editor automatically inserts ` FROM $__index` when the ES|QL field receives focus. You can override this and query a different index if needed. ### Editor features The ES|QL code editor provides: - **Code suggestions** - Auto-complete for ES|QL commands and functions - **Error highlighting** - Syntax errors are highlighted in the editor and error messages from Elasticsearch are displayed directly - **Syntax highlighting** - ES|QL keywords, operators, and values are color-coded for readability ### Example queries The following examples show common ES|QL query patterns. #### Basic aggregation Count documents grouped by a field: ```esql FROM logs-* | STATS count = COUNT(*) BY host.name | SORT count DESC | LIMIT 10 ``` #### Filter and aggregate over time Filter by a field value and compute an average over time intervals: ```esql FROM metrics-* | WHERE service.name == "api-gateway" | STATS avg_duration = AVG(transaction.duration.us) BY @timestamp = BUCKET(@timestamp, 1 minute) | SORT @timestamp ``` #### Search log messages Search for specific patterns in log data: ```esql FROM logs-* | WHERE message LIKE "*error*" AND log.level == "ERROR" | KEEP @timestamp, message, host.name, log.level | SORT @timestamp DESC | LIMIT 100 ``` #### PromQL metrics queries When you select the **Metrics** query type, ES|QL also accepts the `PROMQL` source command. Grafana processes the response as time series frames, the same way it does for ES|QL `STATS` queries. `PROMQL` must be the first command in the query (leading comments are allowed): ```esql PROMQL index=metrics-* step=1m avg(metrics.system.cpu.logical.count) ``` ### Learn more about ES|QL For more information about ES|QL syntax, commands, and functions, refer to the following Elasticsearch documentation: - [ES|QL reference](https://www.elastic.co/docs/reference/query-languages/esql) - Overview of the ES|QL query language - [ES|QL commands](https://www.elastic.co/docs/reference/query-languages/esql/commands) - Source and processing commands (`FROM`, `WHERE`, `STATS`, `EVAL`, `KEEP`, `SORT`, `LIMIT`, and more) - [ES|QL functions and operators](https://www.elastic.co/docs/reference/query-languages/esql/functions-operators) - Aggregation, math, string, date, and other functions - [ES|QL syntax](https://www.elastic.co/docs/reference/query-languages/esql/esql-syntax) - Identifiers, literals, operators, and special characters ## Use template variables You can also augment queries by using [template variables](https://grafana.com/docs/grafana//datasources/elasticsearch/template-variables/). Queries of `terms` have a 500-result limit by default. To set a custom limit, set the `size` property in your query. # Elasticsearch template variables Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana lists these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables. For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana//visualizations/dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//visualizations/dashboards/variables/add-template-variables/) documentation. ## Use filters Elasticsearch supports the use of filters. You can use them to specify any number of key/value filters, and Grafana applies them automatically to all of your Elasticsearch queries. Filters support the following operators: | Operator | Description | | -------- | ------------------------------------------------------------- | | `=` | Equals. Adds `AND field:"value"` to the query. | | `!=` | Not equals. Adds `AND -field:"value"` to the query. | | `=~` | Matches regex. Adds `AND field:/value/` to the query. | | `!~` | Does not match regex. Adds `AND -field:/value/` to the query. | | `>` | Greater than. Adds `AND field:>value` to the query. | | `<` | Less than. Adds `AND field:/visualizations/dashboards/build-dashboards/filter-group-by/). ## Choose a variable syntax The Elasticsearch data source supports two variable syntaxes for use in the **Query** field: - `$varname`, such as `hostname:$hostname`, which is easy to read and write but doesn't let you use a variable in the middle of a word. - `[[varname]]`, such as `hostname:[[hostname]]` When the _Multi-value_ or _Include all value_ options are enabled, Grafana converts the labels from plain text to a Lucene-compatible condition. For details, refer to the [Multi-value variables](https://grafana.com/docs/grafana//visualizations/dashboards/variables/add-template-variables/#multi-value-variables) documentation. ## Use variables in queries You can use variables in the Lucene query field, metric aggregation fields, bucket aggregation fields, and the alias field. ### Variables in Lucene queries Use variables to filter your Elasticsearch queries dynamically: ``` hostname:$hostname AND level:$level ``` ### Chain or nest variables You can create nested variables, where one variable's values depend on another variable's selection. This example defines a variable named `$host` that only shows hosts matching the selected `$environment`: ```json { "find": "terms", "field": "hostname", "query": "environment:$environment" } ``` Whenever you change the value of the `$environment` variable via the drop-down, Grafana triggers an update of the `$host` variable to contain only hostnames filtered by the selected environment. ### Variables in aggregations You can use variables in bucket aggregation fields to dynamically change how data is grouped. For example, use a variable in the **Terms** group by field to let users switch between grouping by `hostname`, `service`, or `datacenter`. ## Template variable examples {{< figure src="/static/img/docs/elasticsearch/elastic-templating-query-7-4.png" max-width="500px" class="docs-image--no-shadow" caption="Query with template variables" >}} In the above example, a Lucene query filters documents based on the `hostname` property using a variable named `$hostname`. The example also uses a variable in the _Terms_ group by field input box, which you can use to quickly change how data is grouped. ## Create a query Write the query using a custom JSON string, with the field mapped as a [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#keyword) in the Elasticsearch index mapping. If the query is [multi-field](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) with both a `text` and `keyword` type, use `"field":"fieldname.keyword"` (sometimes `fieldname.raw`) to specify the keyword field in your query. | Query | Description | | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | | `{"find": "fields", "type": "keyword"}` | Returns a list of field names with the index type `keyword`. | | `{"find": "fields", "type": "number"}` | Returns a list of numeric field names (includes `float`, `double`, `integer`, `long`, `scaled_float`). | | `{"find": "fields", "type": "date"}` | Returns a list of date field names. | | `{"find": "terms", "field": "hostname.keyword", "size": 1000}` | Returns a list of values for a keyword field. Uses the current dashboard time range. | | `{"find": "terms", "field": "hostname", "query": ""}` | Returns a list of values filtered by a Lucene query. Uses the current dashboard time range. | | `{"find": "terms", "field": "status", "orderBy": "doc_count"}` | Returns values sorted by document count (descending by default). | | `{"find": "terms", "field": "status", "orderBy": "doc_count", "order": "asc"}` | Returns values sorted by document count in ascending order. | Queries of `terms` have a 500-result limit by default. To set a custom limit, set the `size` property in your query. ### Sort query results By default, queries return results in term order (which can then be sorted alphabetically or numerically using the variable's Sort setting). To produce a list of terms sorted by document count (a top-N values list), add an `orderBy` property of `doc_count`. This automatically selects a descending sort: ```json { "find": "terms", "field": "status", "orderBy": "doc_count" } ``` You can also use the `order` property to explicitly set ascending or descending sort: ```json { "find": "terms", "field": "hostname", "orderBy": "doc_count", "order": "asc" } ``` {{< admonition type="note" >}} Elasticsearch [discourages](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order) sorting by ascending doc count because it can return inaccurate results. {{< /admonition >}} To keep terms in the document count order, set the variable's Sort drop-down to **Disabled**. You can alternatively use other sorting criteria, such as **Alphabetical**, to re-sort them. # Troubleshoot issues with the Elasticsearch data source This document provides troubleshooting information for common errors you may encounter when using the Elasticsearch data source in Grafana. ## Connection errors The following errors occur when Grafana cannot establish or maintain a connection to Elasticsearch. ### Failed to connect to Elasticsearch **Error message:** "Health check failed: Failed to connect to Elasticsearch" **Cause:** Grafana cannot establish a network connection to the Elasticsearch server. **Solution:** 1. Verify that the Elasticsearch URL is correct in the data source configuration. 1. Check that Elasticsearch is running and accessible from the Grafana server. 1. Ensure there are no firewall rules blocking the connection. 1. If using a proxy, verify the proxy settings are correct. 1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your Elasticsearch instance is not publicly accessible. ### Request timed out **Error message:** "Health check failed: Elasticsearch data source is not healthy. Request timed out" **Cause:** The connection to Elasticsearch timed out before receiving a response. **Solution:** 1. Check the network latency between Grafana and Elasticsearch. 1. Verify that Elasticsearch is not overloaded or experiencing performance issues. 1. Increase the timeout setting in the data source configuration if needed. 1. Check if any network devices (load balancers, proxies) are timing out the connection. ### Failed to parse data source URL **Error message:** "Failed to parse data source URL" **Cause:** The URL entered in the data source configuration is not valid. **Solution:** 1. Verify the URL format is correct (for example, `http://localhost:9200` or `https://elasticsearch.example.com:9200`). 1. Ensure the URL includes the protocol (`http://` or `https://`). 1. Remove any trailing slashes or invalid characters from the URL. ## Authentication errors The following errors occur when there are issues with authentication credentials or permissions. ### Unauthorized (401) **Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 401 Unauthorized" **Cause:** The authentication credentials are invalid or missing. **Solution:** 1. Verify that the username and password are correct. 1. If using an API key, ensure the key is valid and has not expired. 1. Check that the authentication method selected matches your Elasticsearch configuration. 1. Verify the user has the required permissions to access the Elasticsearch cluster. ### Forbidden (403) **Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 403 Forbidden" **Cause:** The authenticated user does not have permission to access the requested resource. **Solution:** 1. Verify the user has read access to the specified index. 1. Check Elasticsearch security settings and role mappings. 1. Ensure the user has permission to access the `_cluster/health` endpoint. 1. If using SigV4 authentication with an AWS-hosted Elasticsearch-compatible domain, verify the IAM policy grants the required permissions. ## Cluster health errors The following errors occur when the Elasticsearch cluster is unhealthy or unavailable. ### Cluster status is red **Error message:** "Health check failed: Elasticsearch data source is not healthy" **Cause:** The Elasticsearch cluster health status is red, indicating one or more primary shards are not allocated. **Solution:** 1. Check the Elasticsearch cluster health using `GET /_cluster/health`. 1. Review Elasticsearch logs for errors. 1. Verify all nodes in the cluster are running and connected. 1. Check for unassigned shards using `GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason`. 1. Consider increasing the cluster's resources or reducing the number of shards. ### Bad Gateway (502) **Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 502 Bad Gateway" **Cause:** A proxy or load balancer between Grafana and Elasticsearch returned an error. **Solution:** 1. Check the health of any proxies or load balancers in the connection path. 1. Verify Elasticsearch is running and accepting connections. 1. Review proxy/load balancer logs for more details. 1. Ensure the proxy timeout is configured appropriately for Elasticsearch requests. ## Index errors The following errors occur when there are issues with the configured index or index pattern. ### Index not found **Error message:** "Error validating index: `index_not_found`" **Cause:** The specified index or index pattern does not match any existing indices. **Solution:** 1. Verify the index name or pattern in the data source configuration. 1. Check that the index exists using `GET /_cat/indices`. 1. If using a time-based index pattern (for example, `[logs-]YYYY.MM.DD`), ensure indices exist for the selected time range. 1. Verify the user has permission to access the index. ### Time field not found **Error message:** "Could not find time field '@timestamp' with type date in index" **Cause:** The specified time field does not exist in the index or is not of type `date`. **Solution:** 1. Verify the time field name in the data source configuration matches the field in your index. 1. Check the field mapping using `GET //_mapping`. 1. Ensure the time field is mapped as a `date` type, not `text` or `keyword`. 1. If the field name is different (for example, `timestamp` instead of `@timestamp`), update the data source configuration. ## Query errors The following errors occur when there are issues with query syntax or configuration. ### Too many buckets **Error message:** "Trying to create too many buckets. Must be less than or equal to: [65536]." **Cause:** The query is generating more aggregation buckets than Elasticsearch allows. **Solution:** 1. Reduce the time range of your query. 1. Increase the date histogram interval (for example, change from `10s` to `1m`). For nested aggregations with **Interval** set to `Auto`, newer plugin versions widen the auto interval when the query would exceed `max_buckets`. 1. Add filters to reduce the number of documents being aggregated. 1. Increase the `search.max_buckets` setting in Elasticsearch (requires cluster administrator access). 1. Update the Elasticsearch plugin if you are on an older version. Refer to [Plugin updates](https://grafana.com/docs/grafana//datasources/elasticsearch/#plugin-updates). ### Required field missing **Error message:** "Required one of fields [field, script], but none were specified." **Cause:** A metric aggregation (such as Average, Sum, or Min) was added without specifying a field. **Solution:** 1. Select a field for the metric aggregation in the query editor. 1. Ensure the selected field exists in your index and contains numeric data. ### Unsupported interval **Error message:** "unsupported interval '<interval>'" **Cause:** The interval specified for the index pattern is not valid. **Solution:** 1. Use a supported interval: `Hourly`, `Daily`, `Weekly`, `Monthly`, or `Yearly`. 1. If you don't need a time-based index pattern, use `No pattern` and specify the exact index name. ## Version errors The following errors occur when there are Elasticsearch version compatibility issues. ### Unsupported Elasticsearch version **Error message:** "Support for Elasticsearch versions after their end-of-life (currently versions < 7.17) was removed. Using unsupported version of Elasticsearch may lead to unexpected and incorrect results." **Cause:** The Elasticsearch version is no longer supported by the Grafana data source. **Solution:** 1. Upgrade Elasticsearch to a supported version (7.17+, 8.x, or 9.x). 1. Refer to [Elastic Product End of Life Dates](https://www.elastic.co/support/eol) for version support information. 1. Note that queries may still work, but Grafana does not guarantee functionality for unsupported versions. ### Elastic Cloud Serverless health check returns 410 Gone **Error message:** Health check fails with status `410 Gone` when connecting to Elastic Cloud Serverless. **Cause:** An older version of the Elasticsearch plugin used a health-check request that Elastic Cloud Serverless no longer accepts. **Solution:** 1. Update the Elasticsearch plugin to **12.8.0** or later. Refer to [Plugin updates](https://grafana.com/docs/grafana//datasources/elasticsearch/#plugin-updates). 1. Confirm `preinstall_auto_update` is enabled so Grafana installs the latest plugin on restart, or update manually from **Plugins and data** > **Plugins**. 1. Click **Save & test** again after the plugin update. ## Other common issues The following issues don't produce specific error messages but are commonly encountered. ### Empty query results **Cause:** The query returns no data. **Solution:** 1. Verify the time range includes data in your index. 1. Check the Lucene query syntax for errors. 1. Test the query directly in Elasticsearch using the `_search` API. 1. Ensure the index contains documents matching your query filters. ### Slow query performance **Cause:** Queries take a long time to execute. **Solution:** 1. Reduce the time range of your query. 1. Add more specific filters to limit the data scanned. 1. Increase the date histogram interval. 1. Check Elasticsearch cluster performance and resource utilization. 1. Consider using index aliases or data streams for better query routing. ### Browser access mode disabled **Error message:** Direct browser access is no longer available, or the data source fails when configured for browser access. **Cause:** Browser access mode was removed in Grafana 9.2.0. The Elasticsearch data source must use Server (proxy) access so the Grafana backend connects to Elasticsearch. **Solution:** 1. Open the data source configuration in Grafana. 1. Ensure access mode is set to **Server** (proxy). Provisioned data sources should use `access: proxy`. 1. Click **Save & test** to verify the connection. 1. If you still see browser CORS errors, upgrade Grafana and the Elasticsearch plugin. Older configurations that attempted direct browser-to-Elasticsearch requests are no longer supported. ## Get additional help If you continue to experience issues after following this troubleshooting guide: 1. Check the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) for API-specific guidance. 1. Review the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Contact Grafana Support if you have an Enterprise license. # Troubleshoot issues with the Elasticsearch data source This document provides troubleshooting information for common errors you may encounter when using the Elasticsearch data source in Grafana. ## Connection errors The following errors occur when Grafana cannot establish or maintain a connection to Elasticsearch. ### Failed to connect to Elasticsearch **Error message:** "Health check failed: Failed to connect to Elasticsearch" **Cause:** Grafana cannot establish a network connection to the Elasticsearch server. **Solution:** 1. Verify that the Elasticsearch URL is correct in the data source configuration. 1. Check that Elasticsearch is running and accessible from the Grafana server. 1. Ensure there are no firewall rules blocking the connection. 1. If using a proxy, verify the proxy settings are correct. 1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your Elasticsearch instance is not publicly accessible. ### Request timed out **Error message:** "Health check failed: Elasticsearch data source is not healthy. Request timed out" **Cause:** The connection to Elasticsearch timed out before receiving a response. **Solution:** 1. Check the network latency between Grafana and Elasticsearch. 1. Verify that Elasticsearch is not overloaded or experiencing performance issues. 1. Increase the timeout setting in the data source configuration if needed. 1. Check if any network devices (load balancers, proxies) are timing out the connection. ### Failed to parse data source URL **Error message:** "Failed to parse data source URL" **Cause:** The URL entered in the data source configuration is not valid. **Solution:** 1. Verify the URL format is correct (for example, `http://localhost:9200` or `https://elasticsearch.example.com:9200`). 1. Ensure the URL includes the protocol (`http://` or `https://`). 1. Remove any trailing slashes or invalid characters from the URL. ## Authentication errors The following errors occur when there are issues with authentication credentials or permissions. ### Unauthorized (401) **Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 401 Unauthorized" **Cause:** The authentication credentials are invalid or missing. **Solution:** 1. Verify that the username and password are correct. 1. If using an API key, ensure the key is valid and has not expired. 1. Check that the authentication method selected matches your Elasticsearch configuration. 1. Verify the user has the required permissions to access the Elasticsearch cluster. ### Forbidden (403) **Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 403 Forbidden" **Cause:** The authenticated user does not have permission to access the requested resource. **Solution:** 1. Verify the user has read access to the specified index. 1. Check Elasticsearch security settings and role mappings. 1. Ensure the user has permission to access the `_cluster/health` endpoint. 1. If using SigV4 authentication with an AWS-hosted Elasticsearch-compatible domain, verify the IAM policy grants the required permissions. ## Cluster health errors The following errors occur when the Elasticsearch cluster is unhealthy or unavailable. ### Cluster status is red **Error message:** "Health check failed: Elasticsearch data source is not healthy" **Cause:** The Elasticsearch cluster health status is red, indicating one or more primary shards are not allocated. **Solution:** 1. Check the Elasticsearch cluster health using `GET /_cluster/health`. 1. Review Elasticsearch logs for errors. 1. Verify all nodes in the cluster are running and connected. 1. Check for unassigned shards using `GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason`. 1. Consider increasing the cluster's resources or reducing the number of shards. ### Bad Gateway (502) **Error message:** "Health check failed: Elasticsearch data source is not healthy. Status: 502 Bad Gateway" **Cause:** A proxy or load balancer between Grafana and Elasticsearch returned an error. **Solution:** 1. Check the health of any proxies or load balancers in the connection path. 1. Verify Elasticsearch is running and accepting connections. 1. Review proxy/load balancer logs for more details. 1. Ensure the proxy timeout is configured appropriately for Elasticsearch requests. ## Index errors The following errors occur when there are issues with the configured index or index pattern. ### Index not found **Error message:** "Error validating index: `index_not_found`" **Cause:** The specified index or index pattern does not match any existing indices. **Solution:** 1. Verify the index name or pattern in the data source configuration. 1. Check that the index exists using `GET /_cat/indices`. 1. If using a time-based index pattern (for example, `[logs-]YYYY.MM.DD`), ensure indices exist for the selected time range. 1. Verify the user has permission to access the index. ### Time field not found **Error message:** "Could not find time field '@timestamp' with type date in index" **Cause:** The specified time field does not exist in the index or is not of type `date`. **Solution:** 1. Verify the time field name in the data source configuration matches the field in your index. 1. Check the field mapping using `GET //_mapping`. 1. Ensure the time field is mapped as a `date` type, not `text` or `keyword`. 1. If the field name is different (for example, `timestamp` instead of `@timestamp`), update the data source configuration. ## Query errors The following errors occur when there are issues with query syntax or configuration. ### Too many buckets **Error message:** "Trying to create too many buckets. Must be less than or equal to: [65536]." **Cause:** The query is generating more aggregation buckets than Elasticsearch allows. **Solution:** 1. Reduce the time range of your query. 1. Increase the date histogram interval (for example, change from `10s` to `1m`). For nested aggregations with **Interval** set to `Auto`, newer plugin versions widen the auto interval when the query would exceed `max_buckets`. 1. Add filters to reduce the number of documents being aggregated. 1. Increase the `search.max_buckets` setting in Elasticsearch (requires cluster administrator access). 1. Update the Elasticsearch plugin if you are on an older version. Refer to [Plugin updates](https://grafana.com/docs/grafana//datasources/elasticsearch/#plugin-updates). ### Required field missing **Error message:** "Required one of fields [field, script], but none were specified." **Cause:** A metric aggregation (such as Average, Sum, or Min) was added without specifying a field. **Solution:** 1. Select a field for the metric aggregation in the query editor. 1. Ensure the selected field exists in your index and contains numeric data. ### Unsupported interval **Error message:** "unsupported interval '<interval>'" **Cause:** The interval specified for the index pattern is not valid. **Solution:** 1. Use a supported interval: `Hourly`, `Daily`, `Weekly`, `Monthly`, or `Yearly`. 1. If you don't need a time-based index pattern, use `No pattern` and specify the exact index name. ## Version errors The following errors occur when there are Elasticsearch version compatibility issues. ### Unsupported Elasticsearch version **Error message:** "Support for Elasticsearch versions after their end-of-life (currently versions < 7.17) was removed. Using unsupported version of Elasticsearch may lead to unexpected and incorrect results." **Cause:** The Elasticsearch version is no longer supported by the Grafana data source. **Solution:** 1. Upgrade Elasticsearch to a supported version (7.17+, 8.x, or 9.x). 1. Refer to [Elastic Product End of Life Dates](https://www.elastic.co/support/eol) for version support information. 1. Note that queries may still work, but Grafana does not guarantee functionality for unsupported versions. ### Elastic Cloud Serverless health check returns 410 Gone **Error message:** Health check fails with status `410 Gone` when connecting to Elastic Cloud Serverless. **Cause:** An older version of the Elasticsearch plugin used a health-check request that Elastic Cloud Serverless no longer accepts. **Solution:** 1. Update the Elasticsearch plugin to **12.8.0** or later. Refer to [Plugin updates](https://grafana.com/docs/grafana//datasources/elasticsearch/#plugin-updates). 1. Confirm `preinstall_auto_update` is enabled so Grafana installs the latest plugin on restart, or update manually from **Plugins and data** > **Plugins**. 1. Click **Save & test** again after the plugin update. ## Other common issues The following issues don't produce specific error messages but are commonly encountered. ### Empty query results **Cause:** The query returns no data. **Solution:** 1. Verify the time range includes data in your index. 1. Check the Lucene query syntax for errors. 1. Test the query directly in Elasticsearch using the `_search` API. 1. Ensure the index contains documents matching your query filters. ### Slow query performance **Cause:** Queries take a long time to execute. **Solution:** 1. Reduce the time range of your query. 1. Add more specific filters to limit the data scanned. 1. Increase the date histogram interval. 1. Check Elasticsearch cluster performance and resource utilization. 1. Consider using index aliases or data streams for better query routing. ### Browser access mode disabled **Error message:** Direct browser access is no longer available, or the data source fails when configured for browser access. **Cause:** Browser access mode was removed in Grafana 9.2.0. The Elasticsearch data source must use Server (proxy) access so the Grafana backend connects to Elasticsearch. **Solution:** 1. Open the data source configuration in Grafana. 1. Ensure access mode is set to **Server** (proxy). Provisioned data sources should use `access: proxy`. 1. Click **Save & test** to verify the connection. 1. If you still see browser CORS errors, upgrade Grafana and the Elasticsearch plugin. Older configurations that attempted direct browser-to-Elasticsearch requests are no longer supported. ## Get additional help If you continue to experience issues after following this troubleshooting guide: 1. Check the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) for API-specific guidance. 1. Review the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Contact Grafana Support if you have an Enterprise license. # Google Cloud Monitoring alerting The Google Cloud Monitoring data source supports [Grafana Alerting](https://grafana.com/docs/grafana//alerting/), allowing you to create alert rules based on GCP metrics and Service Level Objectives (SLOs). You can monitor your Google Cloud environment and receive notifications when specific conditions are met. ## Before you begin Before you create alert rules, ensure the following: - You have appropriate permissions to create alert rules in Grafana. - Your Google Cloud Monitoring data source is configured and working correctly. Refer to [Configure the data source](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/configure/). - You're familiar with [Grafana Alerting concepts](https://grafana.com/docs/grafana//alerting/fundamentals/). ## Supported query types for alerting The following query types support alerting: | Query type | Use case | Notes | | ---------------------------------- | ------------------------------------------------------ | -------------------------------------------------- | | **Builder** | Threshold-based alerts on GCP resource metrics | Best suited for alerting; returns time-series data | | **MQL** | Complex metric queries using Monitoring Query Language | Use for advanced filtering and aggregations | | **Service Level Objectives (SLO)** | Alert on SLO compliance, error budgets, or burn rate | Monitor service reliability | | **PromQL** | Prometheus-style queries on GCP metrics | Familiar syntax for Prometheus users | {{< admonition type="note" >}} Alert queries must return numeric data that Grafana can evaluate against a threshold. Queries that return only text or non-numeric data can't be used directly for alerting. {{< /admonition >}} ## Authentication requirements Alerting rules run as background processes without a user context. Both supported authentication methods work with alerting: | Authentication method | Supported | | --------------------------- | --------- | | Google JWT File | ✓ | | GCE Default Service Account | ✓ | ## Create an alert rule To create an alert rule using Google Cloud Monitoring data: 1. Go to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for your alert rule. 1. In the **Define query and alert condition** section: - Select your Google Cloud Monitoring data source. - Configure your query (for example, a Builder query for CPU usage or an SLO query for error budget). - Add a **Reduce** expression if your query returns multiple series. - Add a **Threshold** expression to define the alert condition. 1. Configure the **Set evaluation behavior**: - Select or create a folder and evaluation group. - Set the evaluation interval (how often the alert is checked). - Set the pending period (how long the condition must be true before firing). 1. Add labels and annotations to provide context for notifications. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example: VM CPU usage alert This example creates an alert that fires when Compute Engine VM CPU utilization exceeds 80%: 1. Create a new alert rule. 1. Configure the query: - **Query type**: Builder - **Project**: Select your GCP project - **Service**: Compute Engine - **Metric**: `instance/cpu/utilization` - **Group by function**: mean 1. Add expressions: - **Reduce**: Last (to get the most recent data point) - **Threshold**: Is above 0.8 (CPU utilization is returned as a decimal) 1. Set evaluation to run every 1 minute with a 5-minute pending period. 1. Save the rule. ## Example: SLO error budget alert This example alerts when an SLO's error budget remaining drops below 20%: 1. Create a new alert rule. 1. Configure the query: - **Query type**: Service Level Objectives (SLO) - **Project**: Select your GCP project - **Service**: Select your SLO service - **SLO**: Select your SLO - **Selector**: SLO Error Budget Remaining 1. Add expressions: - **Reduce**: Last - **Threshold**: Is below 0.2 (20% remaining) 1. Set evaluation to run every 5 minutes. 1. Save the rule. ## Example: Cloud SQL memory alert This example alerts when Cloud SQL instance memory usage exceeds 90%: 1. Create a new alert rule. 1. Configure the query: - **Query type**: Builder - **Project**: Select your GCP project - **Service**: Cloud SQL - **Metric**: `database/memory/utilization` - **Filter**: Add a filter for specific database instances if needed 1. Add expressions: - **Reduce**: Last - **Threshold**: Is above 0.9 1. Set evaluation to run every 1 minute. 1. Save the rule. ## Best practices Follow these recommendations to create reliable and efficient alerts with Google Cloud Monitoring data. ### Use appropriate query intervals - Set the alert evaluation interval to be greater than or equal to the minimum data resolution from Google Cloud Monitoring. - Most GCP metrics have 1-minute granularity at minimum. - Avoid very short intervals (less than 1 minute) as they may cause evaluation timeouts or miss data points. ### Reduce multiple series When your query returns multiple time series (for example, CPU usage across multiple VMs), use the **Reduce** expression to aggregate them: - **Last**: Use the most recent value - **Mean**: Average across all series - **Max/Min**: Use the highest or lowest value - **Sum**: Total across all series ### Use appropriate alignment periods For alerting queries, ensure the alignment period provides enough data points: - Use "cloud monitoring auto" or "grafana auto" for most cases. - For more precise control, set a fixed alignment period that matches your evaluation interval. ### Handle no data conditions Configure what happens when no data is returned: 1. In the alert rule, find **Configure no data and error handling**. 1. Choose an appropriate action: - **No Data**: Keep the alert in its current state - **Alerting**: Treat no data as an alert condition - **OK**: Treat no data as a healthy state ### Test queries before alerting Always verify your query returns expected data before creating an alert: 1. Go to **Explore**. 1. Select your Google Cloud Monitoring data source. 1. Run the query you plan to use for alerting. 1. Confirm the data format and values are correct. 1. Verify the query returns numeric data suitable for threshold evaluation. ## Recording rules The Google Cloud Monitoring data source supports [Grafana-managed recording rules](https://grafana.com/docs/grafana//alerting/alerting-rules/create-recording-rules/). Recording rules periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric. Use recording rules to: - Reduce query load on Google Cloud Monitoring by pre-computing complex aggregations. - Create derived metrics from GCP data for use in alerts and dashboards. - Import Google Cloud Monitoring data into a Prometheus-compatible database. {{< admonition type="note" >}} Grafana-managed recording rules write results to a Prometheus-compatible database (such as Grafana Mimir or the Grafana Cloud managed Prometheus). You must configure a target data source for storing the recorded metrics. {{< /admonition >}} For instructions on creating recording rules, refer to [Create recording rules](https://grafana.com/docs/grafana//alerting/alerting-rules/create-recording-rules/). ## Troubleshooting If your Google Cloud Monitoring alerts aren't working as expected, use the following sections to diagnose and resolve common issues. ### Alerts not firing - Check that the query returns numeric data in Explore. - Ensure the evaluation interval allows enough time for data to be available. - Verify the threshold is set correctly (remember that many GCP metrics return decimals, not percentages). - Review the alert rule's health and any error messages in the Alerting UI. ### Authentication errors in alert evaluation If you see authentication errors when alerts evaluate: - Verify the service account has the **Monitoring Viewer** role. - If using a JWT key file, ensure it hasn't been deleted or revoked. - Check that the required APIs (Monitoring API, Cloud Resource Manager API) are enabled. ### Query timeout errors - Increase the alignment period to reduce the number of data points. - Reduce the time range in the query. - Simplify complex MQL queries. - Add filters to narrow the result set. For additional troubleshooting help, refer to [Troubleshoot Google Cloud Monitoring](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/troubleshooting/). ## Additional resources - [Grafana Alerting documentation](https://grafana.com/docs/grafana//alerting/) - [Create alert rules](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/) - [Create recording rules](https://grafana.com/docs/grafana//alerting/alerting-rules/create-recording-rules/) - [Google Cloud Monitoring query editor](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/query-editor/) # Google Cloud Monitoring annotations [Annotations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/) overlay rich event information on top of graphs. You can use annotations to mark important events, deployments, or incidents on your dashboards. ## Before you begin Before you configure annotations, ensure you have the following: - A configured Google Cloud Monitoring data source. - A dashboard where you want to add annotations. ## Annotation limitations Keep the following limitations in mind when using annotations: - **Performance:** Rendering annotations is expensive. Limit the number of rows returned to maintain dashboard performance. - **Native events:** There's no support for displaying Google Cloud Monitoring's native annotations and events. However, annotations work well with [custom metrics](https://cloud.google.com/monitoring/custom-metrics/) in Google Cloud Monitoring. ## Add an annotation query To add an annotation query to a dashboard: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Annotation query**. 1. Enter a name for the annotation query. 1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox. 1. Select a color for the annotation event markers. 1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed. 1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed. 1. Click **Open query editor** to open the **Annotation Query** dialog box. 1. Select your Google Cloud Monitoring data source from the **Data source** drop-down list. 1. Configure the annotation query and field mappings. 1. (Optional) Click **Test annotation query** to ensure that the query is working properly. 1. Click **Close** when you've completed the query setup. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. ## Configure the annotation query With the query editor for annotations, you can select a metric and filters to define which data points create annotations. The **Title** and **Text** fields support templating and can use data returned from the query. For example, the Title field could have the following text: `{{metric.type}} has value: {{metric.value}}` Example result: `monitoring.googleapis.com/uptime_check/http_status has this value: 502` ## Annotation patterns Use the following patterns in the **Title** and **Text** fields to display metric data in your annotations: | Pattern format | Description | Example | Result | | ------------------------ | --------------------------------- | -------------------------------- | ------------------------------------------------- | | `{{metric.value}}` | Value of the metric/point. | `{{metric.value}}` | `555` | | `{{metric.type}}` | Returns the full Metric Type. | `{{metric.type}}` | `compute.googleapis.com/instance/cpu/utilization` | | `{{metric.name}}` | Returns the metric name part. | `{{metric.name}}` | `instance/cpu/utilization` | | `{{metric.service}}` | Returns the service part. | `{{metric.service}}` | `compute` | | `{{metric.label.xxx}}` | Returns the metric label value. | `{{metric.label.instance_name}}` | `grafana-1-prod` | | `{{resource.label.xxx}}` | Returns the resource label value. | `{{resource.label.zone}}` | `us-east1-b` | ## Example: Annotate uptime check failures To create annotations for uptime check failures: 1. Add an annotation query using the Google Cloud Monitoring data source. 1. Select the `monitoring.googleapis.com/uptime_check/check_passed` metric. 1. Add a filter for `check_passed = false`. 1. Set the **Title** to: `Uptime check failed: {{metric.label.check_id}}` 1. Set the **Text** to: `Region: {{resource.label.zone}}` This creates an annotation marker on your graph each time an uptime check fails. # Configure the Google Cloud Monitoring data source This document provides instructions for configuring the Google Cloud Monitoring data source in Grafana. ## Before you begin Before you begin, ensure you have the following: - **Grafana permissions:** You must have the `Organization administrator` role to configure data sources. - **GCP project:** A Google Cloud Platform project. - **GCP permissions:** Permissions to create a service account or configure GCE default service account settings in your GCP project. Grafana ships with the Google Cloud Monitoring data source preinstalled in both Grafana OSS and Grafana Enterprise, so there's nothing for you to install. For details on how the standalone plugin updates, refer to [Plugin updates](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/#plugin-updates). ## Set up GCP authentication Before you can request data from Google Cloud Monitoring, you must configure authentication. All requests to Google APIs are performed on the server-side by the Grafana backend. For authentication options and configuration details, refer to [Google authentication](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/google-authentication/). When you configure Google authentication, note the following requirements specific to Google Cloud Monitoring. ### Configure a GCP service account When you [create a Google Cloud Platform (GCP) service account and key file](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/google-authentication/#create-a-gcp-service-account-and-key-file), the service account must have the **Monitoring Viewer** role (**Role > Select a role > Monitoring > Monitoring Viewer**): {{< figure src="/static/img/docs/v71/cloudmonitoring_service_account_choose_role.png" max-width="600px" class="docs-image--no-shadow" caption="Choose role" >}} ### Grant the GCE Default Service Account scope If Grafana is running on a Google Compute Engine (GCE) virtual machine, when you [configure a GCE Default Service Account](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/google-authentication/#use-gce-default-service-account), you must also grant that service account access to the "Cloud Monitoring API" scope. ## Enable Google Cloud Platform APIs Before you can request data from Google Cloud Monitoring, you must enable the necessary APIs in your GCP project. 1. Open the Monitoring and Cloud Resource Manager API pages: - [Monitoring API](https://console.cloud.google.com/apis/library/monitoring.googleapis.com) - [Cloud Resource Manager API](https://console.cloud.google.com/apis/library/cloudresourcemanager.googleapis.com) 1. On each page, click **Enable**. {{< figure src="/static/img/docs/v71/cloudmonitoring_enable_api.png" max-width="450px" class="docs-image--no-shadow" caption="Enable GCP APIs" >}} ## Add the data source To add the Google Cloud Monitoring data source: 1. Click **Connections** in the left-side menu. 1. Click **Add new connection**. 1. Enter `Google Cloud Monitoring` in the search bar. 1. Select **Google Cloud Monitoring**. 1. Click **Add new data source** in the upper right. You're taken to the **Settings** tab where you configure the data source. ## Configure the data source in the UI The following are configuration options for the Google Cloud Monitoring data source. | Setting | Description | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | Sets the name you use to refer to the data source in panels and queries. | | **Default** | Sets whether the data source is pre-selected for new panels. | | **Universe Domain** | The universe domain to connect to. For more information, refer to the [Google Cloud universe domains documentation](https://cloud.google.com/docs/overview#universe_domains). Defaults to `googleapis.com`. | ### Authentication Configure how Grafana authenticates with Google Cloud. | Setting | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Authentication type** | Select the authentication method: **Google JWT File** to use a service account key file, **GCE Default Service Account** if Grafana runs on a GCE virtual machine, **Forward OAuth Identity** to authenticate as the Google-signed-in Grafana user, or **Workload Identity Federation** (Grafana Cloud only) to authenticate through an external OIDC identity provider. For a comparison of the methods, refer to [Supported authentication methods](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/google-authentication/#supported-authentication-methods). | ### Configure Google JWT File authentication These settings appear when you select **Google JWT File** as the authentication type. | Setting | Description | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **JWT token** | Upload or paste your Google JWT token. You can drag and drop a `.json` key file, click **Click to browse files** to upload, or use **Paste JWT Token** or **Fill In JWT Token manually**. | ### Configure Forward OAuth Identity These settings appear when you select **Forward OAuth Identity** as the authentication type. | Setting | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Default project** | Enter the GCP project ID to query. This field is required because the user's OAuth token doesn't carry a project context. The signed-in user must have the **Monitoring Viewer** role on this project. | The Grafana Google authentication must request the following scopes so the forwarded token can read Cloud Monitoring data. Set the full list under `[auth.google]` in `grafana.ini` or `custom.ini`, or under **Scopes** in the SSO Settings UI: ```ini [auth.google] scopes = openid email profile https://www.googleapis.com/auth/monitoring.read ``` - `openid`, `email`, `profile`: the default Grafana sign-in scopes. Required for the user to log in. - `https://www.googleapis.com/auth/monitoring.read`: required for the forwarded token to call the Cloud Monitoring API. After you change the scopes, each user must sign out, revoke the existing grant at [https://myaccount.google.com/permissions](https://myaccount.google.com/permissions), and sign in again. Google reuses the previous consent until the grant is revoked, so existing sessions continue to hold tokens issued under the old scope set. ### Configure Workload Identity Federation _Only available for Grafana Cloud._ These settings appear when you select **Workload Identity Federation** as the authentication type. This method lets each query run as the signed-in user by authenticating through an external OIDC identity provider, such as Okta, instead of a service account key. Setting it up spans Google Cloud, your Grafana Cloud stack, and the data source. For the complete procedure, refer to [Use Workload Identity Federation](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/google-authentication/#use-workload-identity-federation). | Setting | Description | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Workload Identity Pool Provider** | Enter the full resource path of your provider, in the form `projects//locations/global/workloadIdentityPools//providers/`. Use the project number, not the project ID. | | **Service account email** | _(Optional)_ Enter the service account to impersonate. Leave this blank if you granted the **Monitoring Viewer** role directly to the Workload Identity pool principal. | | **Default project** | Enter the GCP project ID where your Cloud Monitoring queries run. | {{< admonition type="note" >}} Credentials from Workload Identity Federation are tied to the signed-in user's session, so features that run without a user present don't work. This includes alerting, scheduled reports, and public dashboards. If you rely on these features, use a service account key (Google JWT File) instead. {{< /admonition >}} ### Configure service account impersonation Use service account impersonation to have Grafana authenticate as a different service account than the one provided in the JWT token. | Setting | Description | | ---------------------------------- | --------------------------------------------------------------------------------------------------- | | **Enable** | Toggle to enable service account impersonation. | | **Service account to impersonate** | Enter the email address of the service account to impersonate when making requests to Google Cloud. | ### Private data source connect _Only available for Grafana Cloud._ Use private data source connect (PDC) to connect to and query data within a secure network without opening that network to inbound traffic from Grafana Cloud. For more information on how PDC works, refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). For steps on setting up a PDC connection, refer to [Configure Grafana private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc). | Setting | Description | | ------------------------------- | --------------------------------------------------------------------------- | | **Private data source connect** | Select a PDC connection from the drop-down menu or create a new connection. | ### Save and test Click **Save & test** to test the connection. A successful connection displays the following message: `Successfully queried the Google Cloud Monitoring API.` ## Provision the data source You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ### Provisioning examples **Using the JWT (service account key file) authentication type:** ```yaml apiVersion: 1 datasources: - name: Google Cloud Monitoring type: stackdriver access: proxy jsonData: tokenUri: https://oauth2.googleapis.com/token clientEmail: stackdriver@myproject.iam.gserviceaccount.com authenticationType: jwt defaultProject: my-project-name universeDomain: googleapis.com secureJsonData: privateKey: | -----BEGIN PRIVATE KEY----- POSEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCb1u1Srw8ICYHS ... yA+23427282348234= -----END PRIVATE KEY----- ``` **Using the JWT (service account private key path) authentication type:** ```yaml apiVersion: 1 datasources: - name: Google Cloud Monitoring type: stackdriver access: proxy jsonData: tokenUri: https://oauth2.googleapis.com/token clientEmail: stackdriver@myproject.iam.gserviceaccount.com authenticationType: jwt defaultProject: my-project-name universeDomain: googleapis.com privateKeyPath: /etc/secrets/gce.pem ``` **Using GCE Default Service Account authentication:** ```yaml apiVersion: 1 datasources: - name: Google Cloud Monitoring type: stackdriver access: proxy jsonData: authenticationType: gce universeDomain: googleapis.com ``` ## Provision the data source using Terraform You can provision the Google Cloud Monitoring data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation. ### Terraform prerequisites Before you begin, ensure you have the following: - [Terraform](https://www.terraform.io/downloads) installed. - Grafana Terraform provider configured with appropriate credentials. - For Grafana Cloud: A [Cloud Access Policy token](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) with data source permissions. ### Provider configuration Configure the Grafana provider to connect to your Grafana instance: ```hcl terraform { required_providers { grafana = { source = "grafana/grafana" version = ">= 2.0.0" } } } # For Grafana Cloud provider "grafana" { url = "" auth = "" } # For self-managed Grafana # provider "grafana" { # url = "http://localhost:3000" # auth = "" # } ``` ### Terraform examples The following examples show how to configure the Google Cloud Monitoring data source for each authentication method. **Using the JWT (service account key file) authentication type:** ```hcl resource "grafana_data_source" "google_cloud_monitoring" { type = "stackdriver" name = "Google Cloud Monitoring" json_data_encoded = jsonencode({ tokenUri = "https://oauth2.googleapis.com/token" clientEmail = "" authenticationType = "jwt" defaultProject = "" universeDomain = "googleapis.com" }) secure_json_data_encoded = jsonencode({ privateKey = "" }) } ``` **Using the JWT (service account private key path) authentication type:** ```hcl resource "grafana_data_source" "google_cloud_monitoring" { type = "stackdriver" name = "Google Cloud Monitoring" json_data_encoded = jsonencode({ tokenUri = "https://oauth2.googleapis.com/token" clientEmail = "" authenticationType = "jwt" defaultProject = "" universeDomain = "googleapis.com" privateKeyPath = "/etc/secrets/gce.pem" }) } ``` **Using GCE Default Service Account authentication:** ```hcl resource "grafana_data_source" "google_cloud_monitoring" { type = "stackdriver" name = "Google Cloud Monitoring" json_data_encoded = jsonencode({ authenticationType = "gce" universeDomain = "googleapis.com" }) } ``` For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). ## Next steps After you configure the Google Cloud Monitoring data source, you can: - [Query GCP metrics](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/query-editor/) using the visual Builder, MQL, SLO, or PromQL query types. - [Create template variables](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/template-variables/) for dynamic, reusable dashboards. - [Add annotations](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/annotations/) to overlay GCP events on your graphs. - [Set up alerting](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/alerting/) to receive notifications based on GCP metrics and SLOs. - [Explore your data](https://grafana.com/docs/grafana//explore/) to investigate metrics without building a dashboard. - [Build dashboards for GCP services](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/#import-dashboards-for-gcp-services) using community and Grafana-authored dashboards. # Google authentication This document explains how to configure authentication between Grafana and Google Cloud Platform (GCP). You must configure authentication before you can use the Google Cloud Monitoring data source to query metrics and SLOs. All requests to Google APIs are performed on the server-side by the Grafana backend. ## Before you begin Before you configure authentication, ensure you have the following: - A Google Cloud Platform project with the Monitoring API enabled. - Permissions to create service accounts or configure GCE instance settings in your GCP project. - Access to the Grafana data source configuration page. ## Supported authentication methods The Google Cloud Monitoring data source supports the following authentication methods: | Method | Use case | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | **Google JWT File** | Use when Grafana runs outside of GCP, or when you need explicit control over credentials. | | **GCE Default Service Account** | Use when Grafana runs on a Google Compute Engine VM with a configured service account. | | **Forward OAuth Identity** | Use when you sign in to Grafana with Google and want each query to run as the signed-in user. | | **Workload Identity Federation** | Use on Grafana Cloud to let users authenticate with an external OIDC identity provider instead of a service account key. | | **Service Account Impersonation** | Use when you need Grafana to act as a different service account than the one it authenticates with. | ## Use a Google service account key file Use this method when Grafana runs outside of Google Cloud Platform, or when you need explicit control over which credentials are used. Each Grafana data source connects to one GCP project by default. To visualize data from multiple GCP projects, create one data source per project, or use service account impersonation. ### Create a GCP service account and key file To create a service account and download its key file: 1. Navigate to the [APIs and Services Credentials page](https://console.cloud.google.com/apis/credentials) in the GCP Console. 1. Click the **Create credentials** dropdown and select **Service account**. 1. In **Service account name**, enter a name for the account. 1. Click **Create and continue**. 1. In the **Grant this service account access to project** section, select the **Monitoring Viewer** role from the **Role** dropdown. 1. Click **Continue**, then click **Done**. 1. In the service accounts list, click the service account you created. 1. Go to the **Keys** tab and click **Add key** > **Create new key**. 1. Select **JSON** and click **Create**. A JSON key file downloads to your computer. 1. Store the key file securely. It grants access to your Google Cloud data. ### Upload the key file to Grafana 1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page. 1. Under **Authentication type**, select **Google JWT File**. 1. Upload the JSON key file using one of the available methods (drag and drop, browse, or paste). 1. Click **Save & test** to verify the connection. ### Grant access to multiple projects You can configure a single service account to access multiple GCP projects: 1. Create a service account and key file following the steps above. 1. Note the service account email address (for example, `grafana-monitoring@my-project.iam.gserviceaccount.com`). 1. In each additional project you want to access: 1. Navigate to **IAM & Admin** > **IAM**. 1. Click **Grant access**. 1. Enter the service account email address. 1. Assign the **Monitoring Viewer** role. 1. Click **Save**. 1. Use the original key file in Grafana. The service account can now access all configured projects. ## Use GCE Default Service Account When Grafana runs on a Google Compute Engine (GCE) virtual machine, it can automatically retrieve credentials from the GCE metadata server. This method doesn't require you to create or manage key files. ### Prerequisites for GCE authentication Before using this method, ensure the following: - Grafana is running on a GCE virtual machine. - The VM has a service account assigned with the **Monitoring Viewer** role. - The VM has the **Cloud Monitoring API** scope enabled. ### Configure the GCE instance 1. In the GCP Console, navigate to **Compute Engine** > **VM instances**. 1. Stop the VM if it's running (you can't change scopes on a running VM). 1. Click the VM name, then click **Edit**. 1. Under **Service account**, select a service account with the **Monitoring Viewer** role. 1. Under **Access scopes**, select **Set access for each API** and enable the **Cloud Monitoring API** (read-only). 1. Click **Save** and restart the VM. ### Configure the data source 1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page. 1. Under **Authentication type**, select **GCE Default Service Account**. 1. Click **Save & test** to verify the connection. For more information about GCE service accounts, refer to the [Google documentation on service accounts for instances](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances). ## Use Forward OAuth Identity Use this method when your Grafana instance authenticates users with Google OAuth and you want each query to run as the signed-in user instead of as a shared service account. This enables per-user access control: a viewer who lacks Cloud Monitoring permissions on a project sees no data from that project, even when the dashboard's data source is shared. Grafana forwards the user's existing Google OAuth access token as the bearer token on outgoing Cloud Monitoring API requests. No service account key is stored on the server. ### Prerequisites for Forward OAuth Identity Before using this method, ensure the following: - Your Grafana instance is configured to sign users in with [Google authentication](https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/google/). - The Google OAuth client requests the `https://www.googleapis.com/auth/monitoring.read` scope on top of the default `openid email profile` scopes. Without this scope, the user's token is rejected by the Cloud Monitoring API with a `403` response. - Each user who needs to query the data source has the **Monitoring Viewer** role (`roles/monitoring.viewer`) on the target GCP project. ### Configure the Google OAuth scope Add the Cloud Monitoring read scope to your Google authentication configuration in `grafana.ini` or `custom.ini`: ```ini [auth.google] scopes = openid email profile https://www.googleapis.com/auth/monitoring.read ``` If you configure Google authentication through the Grafana SSO settings UI, add the same scope value to the **Scopes** field. After you change the scopes, existing user sessions still hold tokens issued under the old scope set. Each affected user must sign out, revoke the existing grant at [https://myaccount.google.com/permissions](https://myaccount.google.com/permissions), and sign in again to consent to the new scope. Otherwise, queries continue to fail with `403`. ### Configure the data source 1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page. 1. Under **Authentication type**, select **Forward OAuth Identity**. 1. In **Default project**, enter the GCP project ID to query. This field is required because the user's OAuth token doesn't carry a project context. 1. Click **Save & test** to verify the connection. {{< admonition type="note" >}} Service account impersonation isn't compatible with Forward OAuth Identity. The data source authenticates as the signed-in user, so there's no service account to impersonate. {{< /admonition >}} ## Use Workload Identity Federation Use [Google Cloud Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) (WIF) to let Grafana users authenticate to Cloud Monitoring with an external identity provider (such as Okta or another OIDC provider) instead of a service account key. {{< admonition type="note" >}} This authentication method is available on **Grafana Cloud** only. Grafana Cloud exchanges the signed-in user's external OIDC token for a short-lived Google Cloud access token before the request reaches the plugin. {{< /admonition >}} Configuring Workload Identity Federation involves three systems: Google Cloud, your Grafana Cloud stack, and the data source itself. ### Configure Google Cloud 1. Create a [Workload Identity Pool and Provider](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-providers) that trusts your OIDC identity provider. When configuring the provider, set up attribute mappings so that `google.subject` maps to the relevant claim from your identity provider, such as `assertion.sub`. The exact mapping depends on your provider's claim format. 1. Grant the Cloud Monitoring permissions needed to run queries. How you grant them depends on whether you use service account impersonation: - **Without impersonation:** Grant the WIF pool principal the **Monitoring Viewer** role (`roles/monitoring.viewer`) directly. - **With impersonation:** Create a service account, grant it the **Monitoring Viewer** role, then grant the WIF pool principal the **Service Account Token Creator** role (`roles/iam.serviceAccountTokenCreator`) on that service account. ### Configure Grafana Cloud 1. Configure your Grafana Cloud stack's SSO integration against the same OIDC provider, so the signed-in user's identity is available for Grafana Cloud to exchange for a Google Cloud access token before the request reaches the plugin. Refer to [Configure OAuth2 authentication](https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/generic-oauth/) for setup details. ### Configure the data source 1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page. 1. Under **Authentication type**, select **Workload Identity Federation**. 1. In the **Workload Identity Pool Provider** field, enter the full resource path of your provider: `projects//locations/global/workloadIdentityPools//providers/` {{< admonition type="note" >}} Use the **project number** (a numeric ID such as `123456789`), not the project ID (such as `my-project`). You can find the project number on the Google Cloud Console home page. {{< /admonition >}} 1. If you set up service account impersonation, enter the service account email in the **Service account email** field. If you granted permissions directly to the WIF pool, leave this blank. 1. In **Default project**, enter the GCP project ID where your Cloud Monitoring queries run. 1. Click **Save & test** to verify the connection. {{< admonition type="note" >}} Credentials from Workload Identity Federation are tied to the signed-in user's active session, so no long-lived credential is available to the Grafana backend. Any feature that runs without a user present won't work, including alerting, scheduled reports, and public dashboards. If you rely on these features, use a service account key (Google JWT File) instead. {{< /admonition >}} ## Configure service account impersonation Service account impersonation allows Grafana to authenticate as one service account but act as a different service account when making API requests. This is useful for: - Accessing resources across multiple projects with a single configuration. - Following the principle of least privilege by separating authentication from authorization. - Auditing and tracking which service account accessed specific resources. ### Prerequisites for impersonation The service account used by Grafana (the "caller") must have the following: - The `iam.serviceAccounts.getAccessToken` permission on the target service account. - This permission is included in the **Service Account Token Creator** role (`roles/iam.serviceAccountTokenCreator`). The service account being impersonated (the "target") must have: - The **Monitoring Viewer** role on the projects you want to access. ### Configure impersonation 1. In the GCP Console, grant the caller service account the **Service Account Token Creator** role on the target service account. 1. Grant the target service account the **Monitoring Viewer** role on the relevant projects. 1. In Grafana, navigate to the Google Cloud Monitoring data source configuration page. 1. Configure authentication using either **Google JWT File** or **GCE Default Service Account**. 1. Enable **Service Account Impersonation**. 1. Enter the email address of the target service account. 1. Click **Save & test** to verify the connection. For more information, refer to the [Google documentation on service account impersonation](https://cloud.google.com/iam/docs/service-account-impersonation). # Google Cloud Monitoring query editor This topic explains querying specific to the Google Cloud Monitoring data source. For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/). ## Query types The Google Cloud Monitoring query editor supports the following query types: | Query type | Description | | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | [**Builder**](#query-metrics) | Build metrics queries visually by selecting a service, metric, filters, and aggregation options. | | [**MQL**](#use-the-monitoring-query-language) | Write queries using the Monitoring Query Language for advanced use cases. | | [**Service Level Objectives (SLO)**](#query-service-level-objectives) | Query SLO data to track service reliability and error budgets. | | [**PromQL**](#query-with-promql) | Write Prometheus-style queries against Google Cloud Monitoring metrics. | ## Query metrics {{< figure src="/static/img/docs/google-cloud-monitoring/metric-query-builder-8-0.png" max-width="400px" class="docs-image--no-shadow" caption="Google Cloud Monitoring metrics query builder" >}} The metrics query editor helps you select metrics, group and aggregate by labels and time, and use filters to specify which time series you want to query. ### Create a metrics query 1. Select **Builder** in the **Query type** drop-down. 1. Select a project from the **Project** drop-down. 1. Select a Google Cloud Platform service from the **Service** drop-down. 1. Select a metric from the **Metric** drop-down. 1. _(Optional)_ Use the plus and minus icons in the filter and group-by sections to add and remove filters or group-by clauses. Google Cloud Monitoring supports several metrics types, such as `GAUGE`, `DELTA,` and `CUMULATIVE`. Each supports different aggregation options, such as reducers and aligners. Additionally, metrics have specific value types that can be either scalar or a distribution. The metrics query editor lists available aggregation methods for a selected metric, and sets a default aggregation, reducer and aligner when you select a metric. In the case that the metric value type is a distribution, the aggregation will be set by default to the mean. For scalar value types, there is no aggregation by default. For a full list of metrics, refer to the [Google Cloud metrics list](https://cloud.google.com/monitoring/api/metrics_gcp). For more details on metric kinds and value types, refer to [Metric kinds and value types](https://cloud.google.com/monitoring/api/v3/kinds-and-types). {{< admonition type="note" >}} Distribution metrics are typically best visualized as either a heatmap or histogram. When visualizing in this way, aggregation is not necessary. However, for other visualization types, performance degradation may be observed when attempting to query distribution metrics that are not aggregated due to the number of potential buckets that can be returned. For more information on how to visualize distribution metrics refer to [this page](https://cloud.google.com/monitoring/charts/charting-distribution-metrics). {{< /admonition >}} ### Apply a filter **To add and apply a filter:** 1. Click the plus icon in the filter section. 1. Select a field to filter by. 1. Enter a filter value, such as `instance_name = grafana-1`. To remove the filter, click the trash icon. #### Use simple wildcards When you set the operator to `=` or `!=`, you can add wildcards to the filter value field. For example, entering `us-*` captures all values that start with "us-", and entering `*central-a` captures all values that end with "central-a". `*-central-*` captures all values with the substring of "-central-". Simple wildcards perform better than regular expressions. #### Use regular expressions When you set the operator to `=~` or `!=~`, you can add regular expressions to the filter value field. For example, entering `us-central[1-3]-[af]` matches all values that start with "us-central", then are followed by a number in the range of 1 to 3, a dash, and then either an "a" or an "f". You don't need leading and trailing slashes when you create these regular expressions. ### Configure pre-processing options The query editor displays pre-processing options when the selected metric has a metric type of `DELTA` or `CUMULATIVE`. - The **Rate** option aligns and converts data points to a rate per time series. - The **Delta** option aligns data points by their delta (difference) per time series. ### Group time series To combine multiple time series and reduce the amount of data returned for a metric, specify a grouping and a function. #### Group by Use the **Group By** segment to group resource or metric labels, which reduces the number of time series and aggregates the results by group. For example, if you group by `instance_name`, Grafana displays an aggregated metric for the specified Compute instance. ##### Metadata labels Resource metadata labels contain information that can uniquely identify a resource in Google Cloud. A time series response returns metadata labels only if they're part of the Group By segment in the time series request. There's no API for retrieving metadata labels, so you can't populate the Group By list with the metadata labels that are available for the selected service and metric. However, the **Group By** field list comes with a pre-defined set of common system labels. You can't pre-define user labels, but you can enter them manually in the **Group By** field. If you include a metadata label, user label, or system label in the **Group By** segment, you can create filters based on it and expand its value on the **Alias** field. #### Group by function To combine the time series in the group into a single time series, select a grouping function. ### Align data You can align all data points received in a fixed length of time by applying an alignment function to combine those data points, and then assigning a timestamp to the result. #### Select an alignment function During alignment, all data points are received in a fixed interval. Within each interval, as defined by the [alignment period](#define-the-alignment-period), and for each time series, the data is aggregated into a single point. The value of that point is determined by the type of alignment function you use. For more information on alignment functions, refer to [alignment metric selector](https://cloud.google.com/monitoring/charts/metrics-selector#alignment). #### Define the alignment period You can set the **Alignment Period** field to group a metric by time if you've chosen an aggregation. The default, called "cloud monitoring auto", uses GCP Google Cloud Monitoring's default groupings. These enable you to compare graphs in Grafana with graphs in the Google Cloud Monitoring UI. The default values for "cloud monitoring auto" are: - 1m for time ranges < 23 hours - 5m for time ranges >= 23 hours and < 6 days - 1h for time ranges >= 6 days The other automatic option is "grafana auto", which automatically sets the Group By time depending on the time range chosen and width of the time series panel. For more information about "grafana auto", refer to [Interval variable](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#add-an-interval-variable). You can also choose fixed time intervals to group by, like `1h` or `1d`. ### Set alias patterns The **Alias By** helps you control the format of legend keys. By default, Grafana shows the metric name and labels, which can be long and difficult to read. You can use patterns in the alias field to customize the legend key's format: | Alias pattern | Description | Alias pattern example | Example result | | -------------------------------- | ------------------------------------------------ | --------------------------------- | ------------------------------------------------- | | `{{metric.label.xxx}}` | Returns the metric label value. | `{{metric.label.instance_name}}` | `grafana-1-prod` | | `{{metric.type}}` | Returns the full Metric Type. | `{{metric.type}}` | `compute.googleapis.com/instance/cpu/utilization` | | `{{metric.name}}` | Returns the metric name part. | `{{metric.name}}` | `instance/cpu/utilization` | | `{{metric.service}}` | Returns the service part. | `{{metric.service}}` | `compute` | | `{{resource.label.xxx}}` | Returns the resource label value. | `{{resource.label.zone}}` | `us-east1-b` | | `{{resource.type}}` | Returns the name of the monitored resource type. | `{{resource.type}}` | `gce_instance` | | `{{metadata.system_labels.xxx}}` | Returns the metadata system label value. | `{{metadata.system_labels.name}}` | `grafana` | | `{{metadata.user_labels.xxx}}` | Returns the metadata user label value. | `{{metadata.user_labels.tag}}` | `production` | #### Alias pattern examples **Using metric labels:** You can select from a list of metric and resource labels for a metric in the Group By drop-down. You can then include them in the legend key using alias patterns. For example, given this value of Alias By: `{{metric.type}} - {{metric.label.instance_name}}` An expected result would look like: `compute.googleapis.com/instance/cpu/usage_time - server1-prod` **Using the Monitored Resource Type:** You can also resolve the name of the Monitored Resource Type with the `resource.type` alias. For example, given this value of Alias By: `{{resource.type}} - {{metric.type}}` An expected result would look like: `gce_instance - compute.googleapis.com/instance/cpu/usage_time` ### Deep-link from Grafana panels to the Google Cloud Console Metrics Explorer {{< figure src="/static/img/docs/v71/cloudmonitoring_deep_linking.png" max-width="500px" class="docs-image--no-shadow" caption="Google Cloud Monitoring deep linking" >}} You can click on a time series in the panel to access a context menu, which contains a link to **View in Metrics Explorer in Google Cloud Console**. The link points to the Google Cloud Console's Metrics Explorer and runs the Grafana panel's query there. If you select the link, you first select an account in the Google Account Chooser. Google then redirects you to the Metrics Explorer. The provided link is valid for any account, but displays the query only if your account has access to the query's specified GCP project. ### Understand automatic unit detection Grafana issues one query to the Cloud Monitoring API per query editor row, and each API response includes a unit. Grafana attempts to convert the returned unit into a unit compatible with its time series panel. If successful, Grafana displays the unit on the panel's Y-axis. If the query editor rows return different units, Grafana uses the unit from the last query editor row in the time series panel. ### Use the Monitoring Query Language The Monitoring Query Language (MQL) query builder helps you query and display MQL results in time series format. To understand basic MQL concepts, refer to [Introduction to Monitoring Query Language](https://cloud.google.com/monitoring/mql). #### Create an MQL query **To create an MQL query:** 1. Select **MQL** in the **Query type** drop-down. 1. Select a project from the **Project** drop-down. 1. Enter your MQL query in the text area. 1. _(Optional)_ Configure the **Graph period** setting. Press `Shift+Enter` to run the query. #### Configure MQL options The following options are available for MQL queries: | Setting | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Alias by** | Control the format of legend keys. Refer to [Set alias patterns for MQL queries](#set-alias-patterns-for-mql-queries) for available patterns. | | **Graph period** | Enable the toggle to override the default time period. Select a period from the drop-down to control the granularity of the returned time series data. | ### Set alias patterns for MQL queries MQL queries use the same alias patterns as [metric queries](#set-alias-patterns). However, `{{metric.service}}` is not supported, and `{{metric.type}}` and `{{metric.name}}` show the time series key in the response. ## Query Service Level Objectives {{< figure src="/static/img/docs/google-cloud-monitoring/slo-query-builder-8-0.png" max-width="400px" class="docs-image--no-shadow" caption="Service Level Objectives (SLO) query editor" >}} The SLO query builder helps you visualize SLO data in time series format. To understand basic concepts in service monitoring, refer to the [Google Cloud Monitoring documentation](https://cloud.google.com/monitoring/service-monitoring). ### Create an SLO query **To create an SLO query:** 1. Select **Service Level Objectives (SLO)** in the **Query type** drop-down. 1. Select a project from the **Project** drop-down. 1. Select an [SLO service](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/services) from the **Service** drop-down. 1. Select an [SLO](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/services.serviceLevelObjectives) from the **SLO** drop-down. 1. Select a [time series selector](https://cloud.google.com/stackdriver/docs/solutions/slo-monitoring/api/timeseries-selectors#ts-selector-list) from the **Selector** drop-down. Time series selectors in Grafana use descriptive names that map to system names that Google uses in the Service Monitoring documentation: | Selector drop-down value | Corresponding time series selector | | ------------------------------ | ---------------------------------- | | **SLI Value** | `select_slo_health` | | **SLO Compliance** | `select_slo_compliance` | | **SLO Error Budget Remaining** | `select_slo_budget_fraction` | | **SLO Burn Rate** | `select_slo_burn_rate` | ### Alias patterns for SLO queries The **Alias By** field helps you control the format of legend keys for SLO queries. | Alias pattern | Description | Example result | | -------------- | ----------------------------- | ------------------- | | `{{project}}` | Returns the GCP project name. | `myProject` | | `{{service}}` | Returns the service name. | `myService` | | `{{slo}}` | Returns the SLO. | `latency-slo` | | `{{selector}}` | Returns the selector. | `select_slo_health` | ### Alignment period and group-by time for SLO queries SLO queries use the same alignment period functionality as [metric queries](#define-the-alignment-period). ## Query with PromQL The PromQL query type allows you to query Google Cloud Monitoring metrics using Prometheus Query Language (PromQL) syntax. This is useful if you're familiar with PromQL from Prometheus or Grafana Mimir and want to use the same query syntax with Google Cloud Monitoring data. For more information about PromQL support in Google Cloud Monitoring, refer to the [Google Cloud documentation on PromQL](https://cloud.google.com/monitoring/promql). ### Create a PromQL query To create a PromQL query: 1. Select **PromQL** in the **Query type** drop-down. 1. Select a project from the **Project** drop-down. 1. Enter your PromQL query in the text area. ### Configure PromQL options The following options are available for PromQL queries: | Setting | Description | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Min step** | Defines the lower bounds on the interval between data points. For example, set this to `1h` to hint that measurements are taken hourly. Supports the `$__interval` and `$__rate_interval` macros. | ### PromQL query examples The following examples show common PromQL query patterns for Google Cloud Monitoring: **Query CPU utilization for Compute Engine instances:** ```promql compute_googleapis_com:instance_cpu_utilization ``` **Filter by label:** ```promql compute_googleapis_com:instance_cpu_utilization{instance_name="my-instance"} ``` **Calculate the rate of a counter metric:** ```promql rate(logging_googleapis_com:log_entry_count[5m]) ``` # Google Cloud Monitoring template variables Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables. For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana//dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/) documentation. ## Use query variables Variables of the type _Query_ help you query Google Cloud Monitoring for various types of data. The Google Cloud Monitoring data source provides the following **Query Types**: | Name | List returned | | ---------------------------------- | --------------------------------------------------------------------- | | **Metric Types** | Metric type names available for the specified service. | | **Labels Keys** | Keys for `metric label` and `resource label` in the specified metric. | | **Labels Values** | Values for the label in the specified metric. | | **Resource Types** | Resource types for the specified metric. | | **Aggregations** | Aggregations (cross-series reducers) for the specified metric. | | **Aligners** | Aligners (per-series aligners) for the specified metric. | | **Alignment periods** | All alignment periods available in the query editor in Grafana. | | **Selectors** | Selectors for SLO (Service Level Objectives) queries. | | **SLO Services** | Service Monitoring services for SLO queries. | | **Service Level Objectives (SLO)** | SLOs for the specified SLO service. | ## Use variables in queries Use Grafana's variable syntax to include variables in queries. For details, refer to the [variable syntax documentation](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/). # Troubleshoot Google Cloud Monitoring data source issues This document provides solutions to common issues you may encounter when configuring or using the Google Cloud Monitoring data source. For configuration instructions, refer to [Configure Google Cloud Monitoring](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/configure/). ## Authentication errors These errors occur when GCP credentials are invalid, missing, or don't have the required permissions. ### "Permission denied" or "Access denied" **Symptoms:** - Save & test fails with permission errors - Queries return authorization errors - Projects, metrics, or labels don't load **Possible causes and solutions:** | Cause | Solution | | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Service account missing required permissions | Assign the **Monitoring Viewer** role to the service account in the GCP Console under **IAM & Admin** > **IAM**. Refer to [Configure Google Cloud Monitoring](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/configure/) for details. | | Incorrect service account key file | Verify the JSON key file was downloaded correctly and contains valid credentials. Generate a new key if necessary. | | Service account key has been deleted | Check the service account in GCP Console under **IAM & Admin** > **Service Accounts**. If the key was deleted, create a new one. | | Wrong project selected | Verify the default project in the data source configuration matches a project the service account has access to. | | APIs not enabled | Enable the Monitoring API and Cloud Resource Manager API in the GCP Console. Refer to [Configure Google Cloud Monitoring](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/configure/) for links. | ### "Invalid JWT" or "JWT token error" **Symptoms:** - Authentication fails when using Google JWT File - Error message references invalid or malformed JWT **Solutions:** 1. Verify you uploaded the complete JSON key file, not just the private key portion. 1. Check that the JSON file is properly formatted and not corrupted. 1. Ensure the key file contains all required fields: `type`, `project_id`, `private_key_id`, `private_key`, `client_email`, `client_id`, `auth_uri`, `token_uri`. 1. Generate a new service account key and re-upload it. ### GCE Default Service Account not working **Symptoms:** - Data source test fails when using GCE Default Service Account - Works with JWT but fails with GCE authentication **Solutions:** 1. Verify Grafana is running on a Google Compute Engine (GCE) virtual machine. 1. Check that the GCE instance has the **Cloud Monitoring API** scope enabled. 1. Verify the GCE default service account has the **Monitoring Viewer** role. 1. If the VM was created without the required scope, you may need to stop the instance, edit it to add the scope, and restart. ### Service account impersonation errors **Symptoms:** - Authentication fails when service account impersonation is enabled - Error: "Unable to impersonate service account" **Solutions:** 1. Verify the primary service account has the `roles/iam.serviceAccountTokenCreator` role on the target service account. 1. Check that the target service account email is entered correctly. 1. Ensure the target service account has the **Monitoring Viewer** role. 1. Verify both service accounts are in projects that have the required APIs enabled. ### Forward OAuth Identity returns 403 **Symptoms:** - Save & Test fails with `403 Forbidden` when using Forward OAuth Identity. - Queries succeed for some users but not others. **Solutions:** 1. Confirm you're signed in to Grafana with Google, not as the local `admin` user. The Forward OAuth Identity method only forwards a token if the current session has one. 1. Verify the Grafana Google authentication includes the `https://www.googleapis.com/auth/monitoring.read` scope. Refer to [Configure the Google OAuth scope](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/google-authentication/#configure-the-google-oauth-scope). 1. After you add the scope, sign out of Grafana, revoke the existing grant at [https://myaccount.google.com/permissions](https://myaccount.google.com/permissions), and sign back in. Google reuses the previous consent and doesn't reissue tokens with new scopes until the grant is revoked. 1. Verify the signed-in user has the **Monitoring Viewer** role (`roles/monitoring.viewer`) on the project set in **Default project**. 1. Check the **Default project** field is populated. The user's OAuth token doesn't carry a project context, so the data source can't fall back to a project from the credentials. ## Connection errors These errors occur when Grafana cannot reach Google Cloud Monitoring endpoints. ### "Request timed out" or connection failures **Symptoms:** - Data source test times out - Queries fail with timeout errors - Intermittent connection issues **Solutions:** 1. Verify network connectivity from the Grafana server to Google Cloud endpoints (`monitoring.googleapis.com`). 1. Check firewall rules allow outbound HTTPS (port 443) to Google Cloud services. 1. For Grafana Cloud connecting to private resources, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). 1. Check if a corporate proxy is blocking connections to Google Cloud. ### "SSL certificate problem" **Symptoms:** - SSL/TLS handshake errors - Certificate verification failures **Solutions:** 1. Ensure the system time is correct on the Grafana server. 1. Verify the Grafana server has up-to-date CA certificates installed. 1. Check if a corporate proxy is intercepting HTTPS traffic. ## Metrics query errors These errors occur when querying Google Cloud Monitoring metrics. ### "No data" or empty results **Symptoms:** - Query executes without error but returns no data - Charts show "No data" message **Possible causes and solutions:** | Cause | Solution | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | Time range doesn't contain data | Expand the dashboard time range. GCP metrics have different retention periods based on the metric type. | | Wrong project selected | Verify you've selected the correct project in the query editor. | | Incorrect metric type | Verify the service, metric type, and metric are correct. Check available metrics in the GCP Console Metrics Explorer. | | Missing labels/filters | Some metrics require specific label filters to return data. Try removing filters to see if data appears. | | Resource not emitting metrics | Verify the resource exists and is actively emitting metrics. Some metrics only populate under certain conditions. | ### Metrics don't appear in drop-down **Symptoms:** - Expected metrics don't appear in the query editor - Metric drop-down is empty for a service **Solutions:** 1. Verify the metric exists in the selected project and region. 1. Check that the service account has the **Monitoring Viewer** role. 1. Some metrics are only available for specific resource types. Check the [Google Cloud metrics list](https://cloud.google.com/monitoring/api/metrics_gcp). 1. Use the Query Inspector to verify the API request and response. ### Label values not loading **Symptoms:** - Label value drop-down doesn't populate - Filters can't be applied **Solutions:** 1. Verify the service account has the **Monitoring Viewer** role. 1. Ensure a project, service, and metric are selected before label values can load. 1. Label values are populated from existing metric data. If no metrics match the current selection, no values appear. ### MQL query errors **Symptoms:** - Monitoring Query Language (MQL) queries fail with syntax errors - MQL query returns unexpected results **Solutions:** 1. Validate your MQL syntax using the [MQL reference documentation](https://cloud.google.com/monitoring/mql/reference). 1. Check for typos in metric types, label names, or function names. 1. Ensure time range syntax is valid in your query. 1. Test the query in the GCP Console Metrics Explorer before using it in Grafana. ### "Too many data points" or API throttling **Symptoms:** - Queries fail with quota errors - Performance degrades with multiple panels **Solutions:** 1. Increase the alignment period to reduce the number of data points. 1. Reduce the time range of your queries. 1. Use fewer metric queries per panel. 1. Request a quota increase in the GCP Console under **APIs & Services** > **Quotas**. 1. Enable query caching in Grafana to reduce API calls. ## SLO query errors These errors are specific to Service Level Objective (SLO) queries. ### SLO services don't appear **Symptoms:** - SLO service selector is empty - Can't find expected SLO services **Solutions:** 1. Verify SLOs are defined in Google Cloud Monitoring for the selected project. 1. Check that the service account has access to view SLOs. 1. Ensure the project has services configured in the Service Monitoring section of the GCP Console. ### SLO query returns no data **Symptoms:** - SLO query executes but returns no data - SLO values show as empty **Solutions:** 1. Verify the SLO exists and is active in the GCP Console. 1. Check that the time range includes periods when the SLO had data. 1. Ensure the selected SLO selector (SLI value, compliance, error budget, etc.) is appropriate for the SLO type. 1. Some SLOs may not have data if the underlying service hasn't received traffic. ## Template variable errors These errors occur when using template variables with the Google Cloud Monitoring data source. ### Variables return no values **Symptoms:** - Variable drop-down is empty - Dashboard fails to load with variable errors **Solutions:** 1. Verify the data source connection is working. 1. Check that the service account has permissions to list the requested resources. 1. For dependent variables, ensure parent variables have valid selections. 1. Verify the project is selected correctly in the variable query. ### Variables are slow to load **Symptoms:** - Dashboard takes a long time to load - Variable selectors are slow to populate **Solutions:** 1. Set variable refresh to **On dashboard load** instead of **On time range change**. 1. Reduce the scope of variable queries (filter by specific project or service). 1. Limit the number of dependent variables in a chain. For more information on template variables, refer to the [template variables documentation](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/template-variables/). ## Dashboard issues These issues occur with dashboards that use the Google Cloud Monitoring data source. As of plugin version 12.6.1, the plugin no longer bundles curated dashboards, so the data source **Dashboards** tab doesn't list them. To build dashboards, use community and Grafana-authored dashboards from the [Grafana dashboards catalog](https://grafana.com/grafana/dashboards/?dataSource=stackdriver) or recreate panels with the [query editor](https://grafana.com/docs/grafana//datasources/google-cloud-monitoring/query-editor/). ### Imported dashboards show no data **Symptoms:** - Imported dashboards show empty panels - Template variables don't load **Solutions:** 1. Verify the data source name in the dashboard matches your Google Cloud Monitoring data source. 1. Check that the service account has access to the projects shown in the project variable. 1. Ensure the resources (Compute Engine instances, Cloud SQL, and similar) exist and are emitting metrics. 1. Verify the required GCP services are enabled in your project. ## Enable debug logging To capture detailed error information for troubleshooting: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location). 1. Look for Google Cloud Monitoring-specific entries that include request and response details. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you've tried the solutions above and still encounter issues: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review [Google Cloud Monitoring issues on GitHub](https://github.com/grafana/grafana/issues) for known bugs. 1. Consult the [Google Cloud Monitoring documentation](https://cloud.google.com/monitoring/docs) for service-specific guidance. 1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user. 1. When reporting issues, include: - Grafana version - GCP project (redact if sensitive) - Error messages (redact sensitive information) - Steps to reproduce - Query configuration (redact credentials) # Graphite alerting You can use Grafana Alerting with Graphite to create alerts based on your time series data. This lets you monitor metrics, detect anomalies, and receive notifications when specific conditions are met. For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). ## Before you begin Before creating alerts with Graphite, ensure you have: - A Graphite data source configured in Grafana. Refer to [Configure the Graphite data source](https://grafana.com/docs/grafana//datasources/graphite/configure/). - Permission to create alert rules. - An understanding of the metrics you want to monitor. ## Supported queries Grafana Alerting evaluates any Graphite query that returns time series data. Grafana runs the query on the backend, reduces each series to a single value, and compares that value against the condition you define. | Query pattern | Alerting support | Notes | | ------------------------------ | ---------------- | ---------------------------------------------------------------------------- | | Single metric path | Yes | The most predictable option for alerting. | | Aggregated series | Yes | Use `sumSeries()`, `averageSeries()`, or similar to combine matching series. | | Wildcard metric path | Yes | Returns one series per match. Reduce or aggregate before the condition. | | Tag query with `seriesByTag()` | Yes | Requires Graphite 1.1 or later. | ## Create an alert rule To create an alert rule using Graphite: 1. Navigate to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for the alert rule. 1. In the query section, select your **Graphite** data source. 1. Build your query: - Click **Select metric** to choose the metric path, or click the pencil icon to write the query in code. - Add Graphite functions to aggregate or shape the series, for example `sumSeries()` or `summarize()`. 1. Add a **Reduce** expression to convert the series to a single value, for example `Last` or `Mean`. 1. Add a **Threshold** expression that defines when the alert fires, for example when the value is above a limit. 1. Set the evaluation interval and pending period. 1. Configure notifications and labels. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example alert queries The following examples show common alerting scenarios with Graphite. Each query returns a single series that the alert condition reduces to one value. ### Alert on high CPU usage Average CPU usage across every core on a host and alert when it exceeds 90%: ```text averageSeries(servers.web01.cpu.*.percent.user) ``` **Condition:** Reduce with `Mean`, then alert when the value is above `90`. ### Alert on low free disk space Monitor available disk space and alert when it drops below 10 GB: ```text servers.web01.disk.root.free_bytes ``` **Condition:** Reduce with `Last`, then alert when the value is below `10737418240`. ### Alert on high error rate Combine error counts across a service and alert on spikes: ```text summarize(sumSeries(stats.myservice.errors.count), '5min', 'sum', false) ``` **Condition:** Reduce with `Last`, then alert when the value is above `100`. ### Alert on request latency Monitor 95th percentile latency and alert when it exceeds 500 ms: ```text stats.timers.myservice.request.latency.p95 ``` **Condition:** Reduce with `Mean`, then alert when the value is above `500`. ### Alert across many hosts Use a wildcard to evaluate every matching host and alert when any host crosses the threshold: ```text aliasByNode(servers.*.cpu.percent.user, 1) ``` Because the wildcard returns one series per host, configure the alert rule to evaluate each series separately so you get a distinct alert instance for each host that breaches the threshold. ## Limitations When using Graphite with Grafana Alerting, be aware of the following limitations. ### Template variables not supported Alert queries can't contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$server` or `$environment` aren't resolved. If your dashboard query uses template variables, create a separate query for alerting with hard-coded values. ### Nested queries not supported Alert queries can't reference another query by letter, such as `#A`. Build a self-contained query for each alert rule instead of chaining query references. ### Consolidation affects results Grafana consolidates data points so that a query doesn't return more points than needed. Consolidation can change the value the alert evaluates. Use `consolidateBy()` or `summarize()` to control how Graphite aggregates points, and match the interval to your evaluation frequency. ### Metrics that stop reporting When a metric stops reporting, the query returns no data rather than a zero value, so a threshold alone can't detect the gap. To alert when a service stops sending metrics, set the alert rule's **No Data** state to **Alerting**. Configure this behavior in the alert rule under the evaluation options. For more information, refer to [Configure no data and error handling](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/#configure-no-data-and-error-handling). ## Best practices Follow these best practices when creating Graphite alerts: - **Return a single series:** Aggregate wildcard queries with `sumSeries()` or `averageSeries()`, or evaluate each series separately when you want per-target alerts. - **Control consolidation:** Use `consolidateBy()` or `summarize()` so the evaluated value matches what you expect. - **Test queries first:** Verify your query returns the expected values in [Explore](https://grafana.com/docs/grafana//explore/) before creating an alert. - **Set realistic thresholds:** Base thresholds on historical data patterns to avoid false positives. - **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor. ## Next steps - [Build queries](https://grafana.com/docs/grafana//datasources/graphite/query-editor/) to explore your metrics before creating alerts. - [Troubleshoot Graphite data source issues](https://grafana.com/docs/grafana//datasources/graphite/troubleshooting/) if alerts aren't firing as expected. # Graphite annotations Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics. You can use Graphite as a data source for annotations to display events such as deployments, incidents, or maintenance windows on your visualizations. For general information about annotations in Grafana, refer to [Annotate visualizations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/). ## Common use cases Graphite annotations help you explain changes in your metrics by marking when related events happened. Common use cases include: - **Deployments:** Mark each release so you can see whether a deployment changed latency, error rate, or throughput. - **Incidents and outages:** Mark when incidents start and resolve to correlate them with metric anomalies. - **Maintenance windows:** Mark planned maintenance so expected dips aren't mistaken for problems. - **Configuration changes:** Mark when you change a feature flag or setting to track its effect over time. - **Scaling events:** Mark when you add or remove capacity to explain shifts in load or performance. Each use case maps to one of the two query modes described in this page: a metric query when an existing metric already marks the event, or an events query when you record discrete events in Graphite. ## Before you begin Before creating Graphite annotations, ensure you have: - A configured Graphite data source in Grafana. - Metrics or events in Graphite that represent the events you want to annotate. - Read access to the Graphite instance. ## Annotation query modes Graphite supports two ways to query annotations. You configure both in the annotation query editor: | Mode | Field | Description | | ---------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | | **Metric query** | **Graphite Query** | Runs a standard Graphite metric query. Each returned data point with a non-zero value becomes an annotation, and the series name becomes the title. | | **Events query** | **Graphite events tags** | Queries the Graphite events API and filters by one or more event tags. Each matching event becomes an annotation. | Use the metric query mode when a metric is non-zero only at the moments you want to mark. A counter that stays above zero after the first event marks every interval, because Grafana creates an annotation for every non-zero point. Use the events query mode when you record discrete events in Graphite and tag them. ## Create an annotation query To add a Graphite annotation to your dashboard: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Annotation query**. 1. Enter a name for the annotation query. 1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox. 1. Select a color for the annotation event markers. 1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed. 1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed. 1. Click **Open query editor** to open the **Annotation Query** dialog box. 1. Select the **Graphite** data source from the **Data source** drop-down list. 1. Configure one of the annotation query modes: - In the **Graphite Query** field, enter a metric query, for example `statsd.application.counters.*.count`. - In the **Graphite events tags** field, enter one or more event tags, for example `deploy`. 1. (Optional) Click **Test annotation query** to ensure that the query is working properly. 1. Click **Close** when you've completed the query setup. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. {{< admonition type="note" >}} Enter a value in either the **Graphite Query** field or the **Graphite events tags** field, not both. If you provide a metric query, Grafana uses the metric query mode and ignores the tags. {{< /admonition >}} ## Metric query annotations In metric query mode, Grafana runs the value in the **Graphite Query** field as a standard Graphite query over the dashboard time range. Grafana converts the results into annotations using these rules: - Every data point with a non-zero value creates an annotation. Data points with a value of zero or `null` are skipped. - The annotation timestamp is the timestamp of the data point. - The annotation title is the series name returned by Graphite. Because every non-zero data point creates an annotation, use a metric that only produces values at the moments you want annotated. Metrics that report continuous values produce an annotation at every step interval and flood your dashboard. ### Example metric queries The following examples show common ways to turn a metric into annotations. #### Deployment markers Record deployments as Graphite events and query them by tag. Refer to [Events query annotations](#events-query-annotations). If you only have a counter, query the increase between points. `nonNegativeDerivative()` is zero while the counter is flat and non-zero only when it goes up, so annotations appear at each deploy: ```text nonNegativeDerivative(deploys.myservice.count) ``` #### Service restarts Use the same pattern for restarts. Query a metric that is non-zero only when a restart happens, or record restarts as events: ```text nonNegativeDerivative(stats.myservice.restart.count) ``` #### Error spikes above a threshold Because Grafana skips zero and `null` data points, use `removeBelowValue()` to annotate only the moments a metric crosses a threshold. The following query annotates every interval where the error count is 100 or greater: ```text removeBelowValue(stats.myservice.errors.count, 100) ``` Values below 100 become `null` and are skipped, so annotations appear only during error spikes. #### Host down Annotate while a host is down by inverting an up metric so that a down state produces a non-zero value: ```text offset(scale(hosts.web01.up, -1), 1) ``` This query returns `1` when `up` is `0`. Grafana skips the zero points from when the host is up, and creates an annotation for each interval the host stays down. ## Events query annotations In events query mode, Grafana queries the Graphite events API and filters events by the tags you provide in the **Graphite events tags** field. Grafana converts each matching event into an annotation using these rules: - The annotation timestamp is the event's `when` value. - The annotation title is the event's `what` value. - The tags you query with are attached to the annotation. Enter multiple tags separated by a space. Grafana sends those tags to the Graphite events API without `set=union`, so an event must include every tag. You can filter events in the following ways: | Value | Result | | ------------- | ------------------------------------------------------------------------------------------ | | Empty | Returns all events in the dashboard time range. | | A single tag | Returns events that include that tag, for example `deploy`. | | Multiple tags | Returns events that include every tag. `deploy myservice` matches events tagged with both. | | A wildcard | Returns events whose tags match the pattern, for example `web*`. | ### Create events in Graphite Before you can query events, record them in Graphite. Graphite exposes an HTTP endpoint for creating events: ```sh curl -X POST http:///events/ \ -H "Content-Type: application/json" \ -d '{ "what": "Deployed myservice v2.5.0", "tags": ["deploy", "myservice"], "data": "Deployed new feature flags and performance improvements" }' ``` Replace _``_ with your Graphite server address. The `what` field becomes the annotation title, and the `tags` field is what you filter on in the **Graphite events tags** field. ### Example events queries The following examples show what to enter in the **Graphite events tags** field for common scenarios. | Goal | Value | Result | | --------------------------- | ------------------ | ---------------------------------------------------------- | | Show all events | Empty | Displays every event in the dashboard time range. | | Show all deployments | `deploy` | Displays events tagged `deploy`. | | Deployments for one service | `deploy myservice` | Displays events tagged with both `deploy` and `myservice`. | | All production events | `prod*` | Displays events whose tags match the `prod*` pattern. | | Incidents | `incident` | Displays events tagged `incident`. | ### Correlate multiple event types To compare different kinds of events on the same dashboard, add more than one annotation query and give each a distinct color. For example: 1. Add a **Deployments** annotation query with the tag `deploy`. 1. Add an **Incidents** annotation query with the tag `incident`. 1. Add a **Maintenance** annotation query with the tag `maintenance`. Each query renders in its own color, so you can see how deployments, incidents, and maintenance windows line up with your metrics. ## Use template variables in annotations You can use template variables in both annotation query modes to filter annotations based on dashboard variable selections. Grafana resolves the variables at query time using the current dashboard variable values. Metric query with a variable: ```text nonNegativeDerivative(deploys.$service.count) ``` Events query with a variable: ```text deploy $environment ``` For more information about template variables, refer to [Graphite template variables](https://grafana.com/docs/grafana//datasources/graphite/template-variables/). ## End-to-end example: correlate deployments with latency This walkthrough shows how to combine a Graphite events annotation query with a metric panel so you can see whether deployments affect request latency. The result is a time series panel that plots latency with a vertical marker at each deployment. ### Before you start Ensure you have the following: - A Graphite data source configured in Grafana. - A latency metric in Graphite, for example `stats.timers.myservice.request.latency.p95`. - Permission to create events in Graphite. ### Step 1: Record a deployment event When your pipeline deploys the service, post an event to Graphite tagged `deploy` and `myservice`: ```sh curl -X POST http:///events/ \ -H "Content-Type: application/json" \ -d '{ "what": "Deployed myservice v2.5.0", "tags": ["deploy", "myservice"], "data": "Rolled out connection pool changes" }' ``` Replace _``_ with your Graphite server address. ### Step 2: Build the latency panel 1. Create a dashboard and add a **Time series** panel. 1. Select the **Graphite** data source. 1. In the query editor, build the latency query, for example `stats.timers.myservice.request.latency.p95`. 1. Confirm the panel plots your latency data over the selected time range. ### Step 3: Add the deployment annotation query 1. In the dashboard toolbar, click the **Add new element** icon (blue plus sign), then click **Annotation query**. 1. Enter `Deployments` as the name and select a color that stands out against the latency line. 1. Click **Open query editor** and select the **Graphite** data source. 1. In the **Graphite events tags** field, enter `deploy myservice`. 1. (Optional) Click **Test annotation query** to confirm the query returns events. 1. Click **Close**, then **Save**. ### Step 4: Interpret the result The panel now shows a vertical marker at each deployment. Hover over a marker to see the deployment title from the event's `what` value. If latency rises immediately after a marker, the deployment is a likely cause. If latency is steady across markers, deployments aren't degrading performance. To extend the dashboard, add a second annotation query tagged `incident` in a different color, so you can tell whether latency changes follow deployments or unrelated incidents. ## Troubleshoot annotation issues The following section addresses common issues you might encounter when using Graphite annotations. ### Annotations don't appear **Possible causes and solutions:** | Cause | Solution | | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | Time range doesn't include the events | Expand the dashboard time range to include the event timestamps. | | Metric query returns only zero values | Annotations are created only for non-zero data points. Verify the metric produces non-zero values at event times. | | No matching events | Confirm events exist in Graphite for the supplied tags and time range. | | Both fields are populated | Clear one of the fields. Grafana uses the metric query mode whenever the **Graphite Query** field has a value. | ### Annotation title is empty In events query mode, Grafana uses the event's `what` value as the title. Ensure the `what` field is populated when you create events in Graphite. For more troubleshooting guidance, refer to [Troubleshoot Graphite data source issues](https://grafana.com/docs/grafana//datasources/graphite/troubleshooting/). # Configure the Graphite data source This document provides instructions for configuring the Graphite data source and explains available configuration options. For general information on managing data sources, refer to [Data source management](https://grafana.com/docs/grafana//administration/data-source-management/). ## Before you begin - You must have the `Organization administrator` role to configure the Graphite data source. Organization administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system. - Grafana comes with a built-in Graphite data source plugin, eliminating the need to install a plugin. - Familiarize yourself with your Graphite security configuration and gather any necessary security certificates and client keys. ## Add the Graphite data source To configure basic settings for the data source, complete the following steps: 1. Click **Connections** in the left-side menu. 1. Click **Add new connection** 1. Type `Graphite` in the search bar. 1. Select the **Graphite data source**. 1. Click **Add new data source** in the upper right. Grafana takes you to the **Settings** tab, where you set up your Graphite configuration. ## Configuration options in the UI Following is a list of configuration options for Graphite. | Setting | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Name** | The display name for the data source. This is how you reference it in panels and queries. For example, `graphite-1` or `graphite-metrics`. | | **Default** | When enabled, sets this data source as the default for dashboard panels. Grafana automatically selects it when you create new panels. | **HTTP:** | Setting | Description | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **URL** | Sets the HTTP protocol, IP, and port of your `graphite-web` or `graphite-api` installation. Because the access method is set to _Server_, the URL must be accessible from the Grafana backend. | | **Allowed cookies** | By default, Grafana removes forwarded cookies. Specify cookie names here to allow them to be forwarded to the data source. | | **Timeout** | Sets the HTTP request timeout in seconds. | **Auth:** | **Setting** | **Description** | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | **Basic Auth** | Toggle on to enable basic authentication to the data source. | |   **User** | Sets the username used for basic authentication. | |   **Password** | Enter the password used for basic authentication. | | **With Credentials** | Toggle on to include cookies and authentication headers in cross-origin requests. | | **TLS Client Auth** | Toggle on to enable TLS client authentication (both server and client are verified). | |   **ServerName** | The server name used to verify the hostname on the certificate returned by the server. | |   **Client Cert** | Client certificate generated by a Certificate Authority (CA) or self-signed. | |   **Client Key** | Private key used to encrypt communication between the client and server. Also generated by a CA or self-signed. | | **With CA Cert** | Toggle on to authenticate with a CA certificate. | |   **CA Cert** | CA certificate used to validate the server certificate. | | **Skip TLS Verify** | Toggle on to bypass TLS certificate validation. Not recommended unless necessary or for testing purposes. | | **Forward OAuth Identity** | Toggle on to forward the user's upstream OAuth identity to the data source. Grafana includes the access token in the request. | **Custom HTTP Headers:** Pass along additional information and metadata about the request or response. | **Setting** | **Description** | | ----------- | ---------------------------------------------------------------------------------------------------------- | | **Header** | Add a custom header. This allows custom headers to be passed based on the needs of your Graphite instance. | | **Value** | The value of the header. | **Graphite details:** | **Setting** | **Description** | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Version** | Select your Graphite version from the drop-down. This controls which functions are available in the Graphite query editor. Use `1.1.x` for Grafana Cloud Graphite. | | **Graphite backend type** | Select the Graphite backend type. Choosing `Metrictank` enables additional features like query processing metadata. (`Metrictank` is a multi-tenant time series engine compatible with Graphite.) Use `Default` for Grafana Cloud Graphite. | | **Rollup indicator** | Toggle on to display an info icon in panel headers when data aggregation occurs. Only available when `Metrictank` is selected. | **Label mappings:** Label mappings are the rules you define to tell Grafana how to pull pieces of the Graphite metric path into Loki labels when switching data sources. They are currently only supported between Graphite and Loki queries. When you change your data source from Graphite to Loki, your queries are automatically mapped based on the rules you define. To create a mapping, specify the full path of the metric and replace the nodes you want to map with label names, using parentheses. The corresponding label values are extracted from your Graphite query during the data source switch. Grafana automatically maps all Graphite tags to labels, even if you haven't defined explicit mappings. When using matching patterns with `{}` (for example, `metric.{a,b}.value`), Grafana converts them to the Loki regular expression matching syntax. If your queries include functions, Graphite extracts the relevant metrics and tags, then matches them against your mappings. For example, given the mapping `servers.(cluster).(server).*`: | **Graphite Query** | **Mapped to Loki Query** | | -------------------------------------------------------- | -------------------------------- | | `alias(servers.west.001.cpu,1,2)` | `{cluster="west", server="001"}` | | `alias(servers.*.{001,002}.*,1,2)` | `{server=~"(001\|002)"}` | | `interpolate(seriesByTag('foo=bar', 'server=002'), inf)` | `{foo="bar", server="002"}` | | **Setting** | **Description** | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Private data source connect** | _Only for Grafana Cloud users._ Establishes a private, secured connection between a Grafana Cloud stack and data sources within a private network. Use the drop-down to locate the PDC URL. For setup instructions, refer to [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) and [Configure PDC](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc). Click **Manage private data source connect** to open your PDC connection page and view your configuration details. | After configuring your Graphite data source options, click **Save & test** at the bottom to test the connection. You should see a confirmation dialog box that says: **Data source is working** ## Provision the data source You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for lists of common configuration options and JSON data options, refer to [Provisioning data sources](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ### YAML example The following example provisions a Graphite data source: ```yaml apiVersion: 1 datasources: - name: Graphite type: graphite access: proxy url: http://localhost:8080 jsonData: graphiteVersion: '1.1' graphiteType: default ``` The following `jsonData` options are supported: | Option | Type | Description | | ------------------------ | ------- | --------------------------------------------------------------------------------------------- | | `graphiteVersion` | string | The Graphite version. Supported values are `0.9`, `1.0`, and `1.1`. Defaults to `1.1`. | | `graphiteType` | string | The Graphite backend type. Supported values are `default` and `metrictank`. | | `rollupIndicatorEnabled` | boolean | Displays an info icon in panel headers when data is aggregated. Applies only to `metrictank`. | ## Provision with Terraform You can provision the Graphite data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation. ### Terraform example The following example provisions a Graphite data source: ```hcl terraform { required_providers { grafana = { source = "grafana/grafana" version = ">= 2.0.0" } } } provider "grafana" { url = "" auth = "" } resource "grafana_data_source" "graphite" { type = "graphite" name = "Graphite" url = "http://localhost:8080" json_data_encoded = jsonencode({ # Graphite version: 0.9, 1.0, or 1.1 graphiteVersion = "1.1" # Backend type: default or metrictank graphiteType = "default" }) } ``` ### Terraform example with basic authentication The following example provisions a Graphite data source with basic authentication: ```hcl resource "grafana_data_source" "graphite_auth" { type = "graphite" name = "Graphite" url = "http://localhost:8080" basic_auth_enabled = true basic_auth_username = "" json_data_encoded = jsonencode({ graphiteVersion = "1.1" graphiteType = "default" }) secure_json_data_encoded = jsonencode({ basicAuthPassword = "" }) } ``` Replace the following placeholders: - _``_: Your Grafana instance URL (for example, `https://your-org.grafana.net` for Grafana Cloud) - _``_: A service account token with data source permissions - _``_: The username for basic authentication - _``_: The password for basic authentication # Graphite query editor Grafana includes a Graphite-specific query editor to help you build queries. The query editor helps you quickly navigate the metric space, add functions, and change function parameters. It supports a variety of Graphite queries, including complex nested queries, through the use of query references. For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/). ## Query editor elements The query editor consists of the following elements: - **Series**: A series in Graphite is a unique time-series dataset, represented by a specific metric name and timestamped values. Click **select metric** to choose a metric from the drop-down. - **Functions**: Graphite uses functions to manipulate data. Click the **+ sign** to view a list of functions in the drop-down. You can add multiple functions to a query. To view the raw query, click the **Pencil icon** in the upper right. Click the **Pencil icon** again to continue adding series and functions. ## Choose metrics to query Click **select metric** to browse the available metrics. You can navigate using your mouse or arrow keys. You can also select a wildcard. {{< figure src="/static/img/docs/graphite/graphite-query-editor-still.png" animated-gif="/static/img/docs/graphite/graphite-query-editor.gif" >}} ## Functions Click the **+ sign** next to **Function** to add a function from the drop-down. You can also search by typing the first few letters of the function name. After selecting a function, Grafana adds it to your query and automatically places your cursor in the first parameter field. To edit a parameter, click it to open an editable text box. To remove a function, click it, then click the **X icon** that appears above it. {{< figure src="/static/img/docs/graphite/graphite-functions-still.png" animated-gif="/static/img/docs/graphite/graphite-functions-demo.gif" >}} Some functions like `aliasByNode` support an optional second argument. To add this argument, hover your mouse over the argument and a dialog box opens. To remove the second optional parameter, click it to delete it. Refer to [Functions](https://graphite.readthedocs.io/en/latest/functions.html) in the Graphite documentation for more information. {{< admonition type="warning" >}} Some functions accept a second argument, which can itself be another function that returns a series. If you need to add a second argument that is a function, Grafana recommends using a series reference from a second query instead of embedding the function directly. Currently, the query editor does not support parsing a second function argument when switching between the query builder and the code editor. {{< /admonition >}} ### Sort labels If the same labels appear on multiple graphs, they may be sorted differently and assigned different colors. To ensure consistent sorting and coloring, use the `sortByName()` function to order labels alphabetically. ### Modify metric names in tables or charts Use `alias` functions, such as `aliasByNode()` or `aliasSub()`, to change metric names on Grafana tables or graphs. ### Consolidate data points Grafana consolidates all Graphite metrics so that Graphite doesn't return more data points than there are pixels in the graph. By default, Grafana consolidates data points using the `avg` function. To control how Graphite consolidates metrics, use the Graphite `consolidateBy()` function. {{< admonition type="note" >}} Grafana calculates legend summary values like `max`, `min`, and `total` on the client side, after data has been calculated. Depending on the consolidation function used, only one or two of these values may be accurate at the same time. {{< /admonition >}} ### Combine time series To combine time series, click **Combine** in the **Functions** list. ### Select and explore data with tags Graphite supports tagged series, so you can filter data by tag key-value pairs instead of only by metric path. When exploring data, previously selected tags filter the remaining result set. To select data, use the `seriesByTag` function, which takes tag expressions (`=`, `!=`, `=~`, `!=~`) to filter the time series. The Grafana query builder does this for you automatically when you select a tag. {{< admonition type="note" >}} Regular expression searches can be slow on high-cardinality tags, so try to use other tags to reduce the scope first. To help reduce the results, start by filtering on a particular name or namespace. {{< /admonition >}} ## Nested queries Grafana lets you reference one query from another using its query letter, similar to how cell references work in a spreadsheet. For example, if you add a second query and want to build on the results of query A, you can reference it using `#A`. This approach lets you build compound or nested queries so your panels are more flexible and easier to manage. ## Use wildcards to make fewer queries To display multiple time series on the same graph, use wildcards in your query to return all matching series at once. For example, to monitor CPU utilization across a variety of metrics, you can use a single query like `cpu.percent.*.g` to retrieve all matching time series. This approach is more efficient than writing separate queries for each series, such as `cpu.percent.user.g`, `cpu.percent.system.g`, and others, which would result in multiple queries to the data source. ## Example queries The following examples show common query patterns. Each one combines a metric path with one or more Graphite functions. ### Rename series for readable legends Wildcard queries return series whose names are full metric paths. Use `aliasByNode()` to display a specific path segment instead. The following query labels each series with the second path node, for example the host name: ```text aliasByNode(servers.*.cpu.percent.user, 1) ``` ### Aggregate across many series Combine matching series into a single line with an aggregation function. The following query sums request counts across all hosts: ```text sumSeries(servers.*.requests.count) ``` Use `averageSeries()`, `maxSeries()`, or `minSeries()` when you want the average, maximum, or minimum instead of the sum. ### Convert a counter to a per-second rate For ever-increasing counters, use `perSecond()` to plot the rate of change: ```text perSecond(servers.web01.nginx.requests.count) ``` ### Summarize into time buckets Use `summarize()` to roll data into fixed intervals, which is useful for bar charts and reducing noise. The following query totals errors in five-minute buckets: ```text summarize(sumSeries(servers.*.errors.count), '5min', 'sum', false) ``` ### Control consolidation When a panel shows a long time range, Grafana consolidates points. Use `consolidateBy()` to choose how points combine so peaks aren't averaged away: ```text consolidateBy(servers.web01.response.max, 'max') ``` ### Show the top series Use a filter function to reduce a large wildcard result to the most significant series. The following query keeps the five hosts with the highest peak CPU: ```text highestMax(servers.*.cpu.percent.user, 5) ``` ### Select series by tag Use `seriesByTag()` to select data by tag expressions instead of a metric path: ```text seriesByTag('name=requests.count', 'env=production') ``` ### Build on another query Reference a query by its letter to reuse its result. If query A returns per-host request counts, query B can sum them: ```text sumSeries(#A) ``` ## Apply annotations [Annotations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/) overlay rich event information on top of graphs. Graphite supports two ways to query annotations: a regular metric query and a Graphite events query. For details on both modes, example queries, and how to create events in Graphite, refer to [Graphite annotations](https://grafana.com/docs/grafana//datasources/graphite/annotations/). ## Integration with Loki When you change the data source to Loki in Explore, your Graphite queries are automatically converted to Loki queries. Loki label names and values are extracted based on the mapping information defined in your Graphite data source configuration. Grafana automatically transforms queries that use tags with `seriesByTag()` without requiring additional setup. # Graphite template variables Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana lists these variables in drop-down selection boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables. For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana//dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/) documentation. ## Use query variables Graphite populates **Query** variables. You can still use other variable types, such as **Custom**, **Interval**, and **Text box**, on a dashboard that queries Graphite. Those types don't send a query to Graphite. To create a Graphite query variable, choose the query type from the **Select query type** drop-down and enter your query in the **Query** field. Grafana supports three query types for Graphite variables: | Query type | Description | Example | | --------------------- | ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------- | | **Default Query** | Runs a metric-find query and returns metric nodes, or tag keys and values when you use `tags()` or `tag_values()`. | `tag_values(apps.*.requests.count, app)` | | **Value Query** | Runs the query through the render endpoint and returns the numeric values of the resulting series. | `groupByNode(apps.*.requests.count, 1, 'sum')` | | **Metric Name Query** | Runs the query through the render endpoint and returns the series names of the result. | `highestAverage(apps.*.requests.count, 5)` | ### Choose a variable syntax The Graphite data source supports two variable syntaxes for use in the **Query** field. ![Variable syntax example](/static/img/docs/v2/templated_variable_parameter.png) Grafana allows two ways to reference variables in a query: | **Syntax** | **Example** | | ------------ | ---------------------------------------- | | `$varname` | `apps.frontend.$server.requests.count` | | `${varname}` | `apps.frontend.${server}.requests.count` | - **Shorthand syntax (`$varname`)** is convenient for standalone path segments but doesn't work when the variable is adjacent to other characters (for example, `cpu$coreLoad`). - **Full syntax (`${varname}`)** is more flexible and works in any part of the string, including embedded within words. Choose the format that best fits the structure of your Graphite metric path. ## Use tag variables Grafana supports tag-based variables for Graphite, allowing you to dynamically populate drop-downs based on tag keys and values in your metric series. To do this, use the Graphite functions `tags()` and `tag_values()` in your variable queries. | Query | Description | | -------------------------------------- | ------------------------------------------------------------------------------------------ | | `tags()` | Returns a list of all tag keys in the Graphite database. | | `tags(server=~backend*)` | Returns tag keys only from series that match the provided filter expression. | | `tag_values(server)` | Returns all values for the specified tag key. | | `tag_values(server, server=~backend*)` | Returns tag values for a given key, filtered to only those that appear in matching series. | You can use multiple filter expressions, and those expressions can include other Grafana variables. For example: ```text tag_values(server, server=~backend*, app=~${apps:regex}) ``` This query returns all server tag values from series where the `server` tag matches `backend*` and the `app` tag matches the regular expression-filtered values from another variable, `${apps}`. For details, refer to the [Graphite docs on the autocomplete API for tags](http://graphite.readthedocs.io/en/latest/tags.html#auto-complete-support). **Using regular expression formatting and the equal tilde operator `=~`:** ```text server=~${servers:regex} ``` This query tells Grafana to format the selected values in the `servers` variable as a regular expression (for example, `(server1|server2)` if two servers are selected). For more information, refer to [Advanced variable format options](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/#advanced-variable-format-options). ### Filter with multiple expressions When using multi-value variables in tag queries, append `${var:regex}` to the variable name to apply regular expression formatting. ```text tag_values(server, app=~${apps:regex}) ``` This query returns only series where the app tag matches the selected values in `${apps}`, formatted as a regular expression. `=~` is the regular expression operator. Non-tag queries use the default `glob` formatting for multi-value variables. ## Use other query variables When writing queries, use the **Default Query** type to retrieve dynamic metric values. For example, the query `prod.servers.*` populates the variable with all values that exist at the wildcard position (`*`). Note that the results include only the values found at the last level of the query path. To return full metric paths that match your query, use the `expand()` function: ```text expand(*.servers.*) ``` ### Compare expanded and non-expanded metric search results When querying Graphite metrics in Grafana, you can choose between using an **expanded** or **non-expanded** query: - **Expanded queries** (using the `expand()` function) return the **full metric paths** that match your query. - **Non-expanded queries** return only the **last segment** of each matching metric path, which limits your ability to extract or filter based on deeper parts of the metric name. Expanded queries are especially useful when working with regular expressions to match or extract specific parts of the metric path. Suppose your Graphite database contains the following metrics: - `prod.servers.001.cpu` - `prod.servers.002.cpu` - `test.servers.001.cpu` The following table illustrates the difference between expanded and non-expanded queries: | **Non-expanded query** | **Results** | **Expanded query** | **Expanded results** | | ---------------------- | -------------- | ------------------------- | ---------------------------------------------------------------------- | | `*` | `prod`, `test` | `expand(*)` | `prod`, `test` | | `*.servers` | `servers` | `expand(*.servers)` | `prod.servers`, `test.servers` | | `test.servers` | `servers` | `expand(test.servers)` | `test.servers` | | `*.servers.*` | `001`, `002` | `expand(*.servers.*)` | `prod.servers.001`, `prod.servers.002`, `test.servers.001` | | `test.servers.*` | `001` | `expand(test.servers.*)` | `test.servers.001` | | `*.servers.*.cpu` | `cpu` | `expand(*.servers.*.cpu)` | `prod.servers.001.cpu`, `prod.servers.002.cpu`, `test.servers.001.cpu` | {{< admonition type="note" >}} A non-expanded query works like an expanded query but returns only the final segment of each matched metric. {{< /admonition >}} Grafana also supports **nested variables**, which allow you to reference other variables in a query. For example: ```text apps.$app.servers.* ``` This query uses the selected value of the `$app` variable to dynamically filter the metric path. The variable `$app` contains one or more application names and `servers.*` matches all servers for the given application. ### Filter query variable results with `__searchFilter` Grafana provides the variable `__searchFilter`, which you can use to dynamically filter query results based on what the user types into the variable drop-down. When the drop-down is empty or blank, `__searchFilter` defaults to `*`, which means it returns all possible values. If you type a string, Grafana replaces `__searchFilter` with that input. To use `__searchFilter` as part of the query field to enable searching for `server` while the user types in the drop-down select box: Query: ```text apps.$app.servers.$__searchFilter ``` TagValues: ```text tag_values(server, server=~${__searchFilter:regex}) ``` # Troubleshoot Graphite data source issues This document provides solutions for common issues you might encounter when using the Graphite data source. ## Connection issues Use the following troubleshooting steps to resolve connection problems between Grafana and your Graphite server. **Data source test fails with "Unable to connect":** If the data source test fails, verify the following: - The URL in your data source configuration is correct and accessible from the Grafana server. - The Graphite server is running and accepting connections. - Any firewall rules or network policies allow traffic between Grafana and the Graphite server. - If using TLS, ensure your certificates are valid and properly configured. To test connectivity, run the following command from the Grafana server: ```sh curl -v /render ``` Replace _``_ with your Graphite server URL. A successful connection returns a response from the Graphite server. **Authentication errors:** If you receive 401 or 403 errors: - Verify your Basic Auth username and password are correct. - Ensure the **With Credentials** toggle is enabled if your Graphite server requires cookies for authentication. - Check that your TLS client certificates are valid and match what the server expects. For detailed authentication configuration, refer to [Configure the Graphite data source](https://grafana.com/docs/grafana//datasources/graphite/configure/). **Deprecation notice about browser access mode:** If the data source configuration page shows a deprecation notice about browser access mode: - The data source is configured to use browser (direct) access, which is deprecated and slated for removal. - Edit the data source so it uses _Server_ (proxy) access instead. Server access routes requests through the Grafana backend, which avoids browser cross-origin restrictions. - Click **Save & test** to re-test the connection after switching. ## Query issues Use the following troubleshooting steps to resolve problems with Graphite queries. **No data returned:** If your query returns no data: - Verify the metric path exists in your Graphite server by testing directly in the Graphite web interface. - Check that the time range in Grafana matches when data was collected. - Ensure wildcards in your query match existing metrics. - Confirm your query syntax is correct for your Graphite version. **HTTP 500 errors with HTML content:** Graphite-web versions before 1.6 return HTTP 500 errors with full HTML stack traces when a query fails. If you see error messages containing HTML tags: - Check the Graphite server logs for the full error details. - Verify your query syntax is valid. - Ensure the requested time range doesn't exceed your Graphite server's capabilities. - Check that all functions used in your query are supported by your Graphite version. **Parser errors in the query editor:** If the query editor displays parser errors: - Check for unbalanced parentheses in function calls. - Verify that function arguments are in the correct format. - Ensure metric paths don't contain unsupported characters. **Queries don't map correctly when switching from Graphite to Loki:** Label mappings apply only when you switch between Graphite and Loki queries. If mapped queries produce unexpected labels or no results: - Verify your label mappings are defined in the Graphite data source configuration. - Remember that Grafana automatically maps all Graphite tags to labels, so you only need explicit mappings for non-tag metric nodes. - Check that multi-value patterns such as `{001,002}` map to the expected regular expression, for example `{server=~"(001|002)"}`. For details, refer to the label mappings section in [Configure the Graphite data source](https://grafana.com/docs/grafana//datasources/graphite/configure/). For query syntax help, refer to [Graphite query editor](https://grafana.com/docs/grafana//datasources/graphite/query-editor/). ## Version and feature issues Use the following troubleshooting steps to resolve problems related to Graphite versions and features. **Functions missing from the query editor:** If expected functions don't appear in the query editor: - Verify the correct Graphite version is selected in the data source configuration. - The available functions depend on the configured version. For example, tag-based functions require Graphite 1.1 or later. - If using a custom Graphite installation with additional functions, ensure the version setting matches your server. **Tag-based queries not working:** If `seriesByTag()` or other tag functions fail: - Confirm your Graphite server is version 1.1 or later. - Verify the Graphite version setting in your data source configuration matches your actual server version. - Check that tags are properly configured in your Graphite server. ## Template variable issues Use the following troubleshooting steps when dashboard variables don't behave as expected. **Variable not interpolated in a query:** - When a variable is adjacent to other characters, use the full syntax `${varname}` instead of `$varname`. For example, use `cpu.${core}.load` rather than `cpu.$coreLoad`. - Confirm the variable name matches exactly, including case. **Multi-value variable returns no data in tag queries:** - Tag queries expect regular expression formatting for multi-value variables. Append `:regex` to the variable, for example `tag_values(server, app=~${apps:regex})`. - Non-tag queries use the default glob formatting and don't require `:regex`. **Dependent variable is empty:** - Define the parent variable above the child variable in the dashboard variable list so it resolves first. - Verify the parent variable has a selected value. An empty parent produces an empty child query. For details, refer to [Graphite template variables](https://grafana.com/docs/grafana//datasources/graphite/template-variables/). ## Performance issues Use the following troubleshooting steps to address slow queries or timeouts. **Queries timing out:** If queries consistently time out: - Increase the **Timeout** setting in the data source configuration. - Reduce the time range of your query. - Use more specific metric paths instead of broad wildcards. - Consider using `summarize()` or `consolidateBy()` functions to reduce the amount of data returned. - Check your Graphite server's performance and resource utilization. **Slow autocomplete in the query editor:** If metric path autocomplete is slow: - This often indicates a large number of metrics in your Graphite server. - Use more specific path prefixes to narrow the search scope. - Check your Graphite server's index performance. ## MetricTank backend issues If you're using MetricTank as your Graphite backend, use the following troubleshooting steps. **Rollup indicator not appearing:** If the **Rollup indicator** doesn't display when expected: - Verify **Metrictank** is selected as the Graphite backend type in the data source configuration. - Ensure the **Rollup indicator** toggle is enabled. - The indicator only appears when data aggregation actually occurs. **Unexpected data aggregation:** If you see unexpected aggregation in your data: - Check the data aggregation (`rollup`) configuration in your MetricTank instance. - Adjust the time range or use `consolidateBy()` to control aggregation behavior. - Review the query processing metadata in the panel inspector for details on how data was processed. ## Enable debug logging To capture detailed request and response information when troubleshooting Graphite issues: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Restart Grafana to apply the change. 1. Reproduce the issue, then review the logs in `/var/log/grafana/grafana.log` (or your configured log location) for Graphite-related entries that include the request URL and response details. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you continue to experience issues: - Check the [Grafana community forums](https://community.grafana.com/) for similar issues and solutions. - Review the [Graphite documentation](https://graphite.readthedocs.io/) for additional configuration options. - Contact [Grafana Support](https://grafana.com/support/) if you're an Enterprise, Cloud Pro, or Cloud Advanced customer. When reporting issues, include the following information: - Grafana version - Graphite version (for example, 1.1.x) and backend type (Default or MetricTank) - Authentication method (Basic Auth, TLS, or none) - Error messages (redact sensitive information) - Steps to reproduce the issue - Relevant configuration such as data source settings, timeout values, and Graphite version setting (redact passwords and other credentials) - Sample query (if applicable, with sensitive data redacted) # InfluxDB alerting You can use Grafana Alerting with InfluxDB to create alerts based on your time-series data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met. For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). ## Before you begin Before creating alerts with InfluxDB, ensure you have: - An InfluxDB data source configured in Grafana - Appropriate permissions to create alert rules - Understanding of the metrics you want to monitor ## Supported query types InfluxDB alerting supports all three query languages: InfluxQL, SQL, and Flux. Queries must return time-series data for Grafana to evaluate values over time and trigger alerts when thresholds are crossed. ### Query language compatibility | Query language | Alerting support | Notes | | -------------- | ---------------- | --------------------------------------------------------------- | | InfluxQL | Yes | Use aggregation functions with GROUP BY time. | | Flux | Yes | Use `aggregateWindow()` for time-based aggregation. | | SQL | Yes | Use `$__timeFilter` and `$__dateBin` macros for time filtering. | ## Create an alert rule To create an alert rule using InfluxDB: 1. Navigate to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for the alert rule. 1. Select your **InfluxDB** data source. 1. Build your query using the query editor. 1. Configure the alert condition (for example, when the average is above a threshold). 1. Set the evaluation interval and pending period. 1. Configure notifications and labels. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example alert queries The following examples show common alerting scenarios with InfluxDB. ### InfluxQL: Alert on high CPU usage Monitor CPU usage and alert when it exceeds a threshold: 1. **Query:** Select the `cpu` measurement with `mean("usage_system")` as the aggregation. 1. **GROUP BY:** `time($__interval)` 1. **Condition:** When mean is above 90. In raw query mode: ```sql SELECT mean("usage_system") FROM "cpu" WHERE $timeFilter GROUP BY time($__interval) fill(null) ``` ### Flux: Alert on memory pressure Monitor memory usage with a Flux query: ```flux from(bucket: "telegraf") |> range(start: v.timeRangeStart, stop: v.timeRangeStop) |> filter(fn: (r) => r["_measurement"] == "mem") |> filter(fn: (r) => r["_field"] == "used_percent") |> aggregateWindow(every: v.windowPeriod, fn: mean) |> yield(name: "mean") ``` Set the condition to alert when the mean value exceeds 85. ### SQL: Alert on disk usage Monitor disk usage with an SQL query: ```sql SELECT $__dateBin(time), mean(used_percent) FROM disk WHERE $__timeFilter(time) GROUP BY $__dateBin(time) ``` Set the condition to alert when the mean value exceeds 80. ## Limitations When using InfluxDB with Grafana Alerting, be aware of the following limitations: ### Template variables not supported Alert queries can't contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$hostname` or `$region` aren't resolved. If your dashboard query uses template variables, create a separate query for alerting with hard-coded values. ### Query complexity Complex queries with many nested functions or large result sets may timeout or fail to evaluate. Simplify queries for alerting by: - Reducing the time range - Using appropriate aggregation intervals - Adding filters to limit the data scanned ## Handle transient errors and timeouts Brief InfluxDB latency spikes or network interruptions can cause alert rule evaluations to fail. By default, a failed evaluation moves the rule into an error state, which can trigger a large number of spurious notifications at once even though your data is healthy. To make alert rules resilient to transient errors: 1. In the alert rule settings, expand **Configure no data and error handling** and set **Alert state if execution error or timeout** to **Keep Last State**. The rule keeps its previous state through brief evaluation failures instead of firing an error alert. 1. Set a **Pending period** so the alert condition must be met for a sustained period across multiple evaluations before the alert fires. 1. If evaluations fail because queries run out of time, increase the data source timeout. Refer to [Advanced HTTP settings](https://grafana.com/docs/grafana//datasources/influxdb/configure/#advanced-http-settings) for details. 1. Simplify or narrow alert queries so they complete well within the timeout. Refer to [Query complexity](#query-complexity) for guidance. For more information about error state handling, refer to [No data and error handling](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/#configure-no-data-and-error-handling). ## Best practices Follow these best practices when creating InfluxDB alerts: - **Use specific filters:** Add WHERE clauses or Flux filters to focus on relevant data and improve query performance. - **Choose appropriate intervals:** Match the GROUP BY time interval to your evaluation frequency. - **Test queries first:** Verify your query returns expected results in Explore before creating an alert. - **Set realistic thresholds:** Base alert thresholds on historical data patterns. - **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor. If you encounter errors when creating or evaluating alert rules, refer to [Troubleshoot InfluxDB data source issues](https://grafana.com/docs/grafana//datasources/influxdb/troubleshooting/#alerting-errors). # InfluxDB annotations Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics. You can use InfluxDB as a data source for annotations to display events such as deployments, alerts, or other significant occurrences on your visualizations. For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/). ## Before you begin Before creating InfluxDB annotations, ensure you have: - An InfluxDB data source configured in Grafana - Data in InfluxDB containing event information with timestamp fields - Read access to the InfluxDB database or bucket containing your events ## Create an annotation query To add an InfluxDB annotation to your dashboard: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Annotation query**. 1. Enter a name for the annotation query. 1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox. 1. Select a color for the annotation event markers. 1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed. 1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed. 1. Click **Open query editor** to open the **Annotation Query** dialog box. 1. Select your **InfluxDB** data source from the **Data source** drop-down list. 1. Configure the annotation query and field mappings. 1. (Optional) Click **Test annotation query** to ensure that the query is working properly. 1. Click **Close** when you've completed the query setup. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. ## InfluxQL annotations For InfluxQL-configured data sources, write an InfluxQL query in the **InfluxQL Query** field. Your query **must** include `WHERE $timeFilter` to scope the results to the dashboard's time range. ### Field mappings Field mappings tell Grafana which InfluxDB columns contain the annotation data. If your query returns only one column, you don't need to enter anything in the field mapping fields. | Field | Description | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | **Text** | The column containing the annotation description displayed when you hover over the annotation. | | **Tags** | The column containing tags for the annotation. The value can be a comma-separated string. Tags help categorize and filter annotations. | | **TimeEnd** | The column containing an end time for range annotations. Range annotations display as a shaded region on the graph instead of a single vertical line. | ### InfluxQL annotation query example The following query retrieves deployment events and displays them as annotations: ```sql SELECT title, description FROM events WHERE $timeFilter ORDER BY time ASC ``` ### Range annotation example To display events with duration as shaded regions: ```sql SELECT description, tags, end_time FROM maintenance_windows WHERE $timeFilter ORDER BY time ASC ``` Configure the field mappings as follows: - **Text:** `description` - **Tags:** `tags` - **TimeEnd:** `end_time` ## Flux annotations For Flux-configured data sources, annotations use the standard Flux query editor. Write a Flux query that returns data frames with time, text, and optional tag fields. ### Flux annotation query example ```flux from(bucket: "events") |> range(start: v.timeRangeStart, stop: v.timeRangeStop) |> filter(fn: (r) => r["_measurement"] == "deployments") |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value") ``` Grafana processes the resulting data frames as annotation events. Ensure the query returns a `_time` column and at least one text column. ## SQL annotations SQL-configured data sources (InfluxDB 3.x) use the standard Grafana annotation query editor. Write an SQL query that returns a time column and at least one text column. Use the `$__timeFilter(time)` macro to scope results to the dashboard time range. ### SQL annotation query example ```sql SELECT time, title, description FROM events WHERE $__timeFilter(time) ORDER BY time ASC ``` ### SQL range annotation example To display events with duration as shaded regions, return both a start time and an end time: ```sql SELECT time, end_time, description, tags FROM maintenance_windows WHERE $__timeFilter(time) ORDER BY time ASC ``` Configure the field mappings: - **Text:** `description` - **Tags:** `tags` - **TimeEnd:** `end_time` ## Use template variables in annotations You can use template variables in your annotation queries to filter annotations based on dashboard variable selections. For example: ```sql SELECT title, description FROM events WHERE $timeFilter AND environment = '$environment' ORDER BY time ASC ``` If your annotations aren't appearing or you encounter errors, refer to [Troubleshoot InfluxDB data source issues](https://grafana.com/docs/grafana//datasources/influxdb/troubleshooting/#annotation-errors). # InfluxDB query editor The InfluxDB query editor helps you build and run queries against your InfluxDB data source. You can access it from the [Explore page](https://grafana.com/docs/grafana//explore/) or from any dashboard panel by clicking the ellipsis in the upper right of the panel and selecting **Edit**. The editor supports three query languages: SQL, InfluxQL, and Flux. Each has its own editing experience and macro system. SQL and InfluxQL provide both a visual builder mode and a code editing mode, while Flux provides a code editor only. You can also use the query editor to retrieve [log data](#query-logs). For general information about Grafana query editors, refer to [Query and transform data](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/). To compare InfluxQL and SQL queries for the same data, explore this demo dashboard on Grafana Play: {{< docs/play title="Influx: Bar Charts and Time Series using Mixed queries" url="https://play.grafana.org/d/dea43745-7fac-4bf4-85d0-d06b168de92b/" >}} ## Before you begin - Ensure you have [configured the InfluxDB data source](https://grafana.com/docs/grafana//datasources/influxdb/configure/). - Verify your credentials have appropriate permissions to query the target database or bucket. ## Key concepts If you're new to InfluxDB, these terms are used throughout the query editor: | Term | Description | | -------------------- | ------------------------------------------------------------------------------------------------ | | **Measurement** | A logical grouping of fields, tags, and timestamps, similar to a table in a relational database. | | **Field** | A column in a measurement that stores the actual data values (numbers, strings, Boolean values). | | **Tag** | A column used for metadata and indexing. Tags are indexed and optimized for filtering. | | **Retention policy** | An InfluxDB 1.x setting that controls how long data is stored before automatic deletion. | | **Bucket** | An InfluxDB 2.x and 3.x storage location that combines a database and retention policy. | ## Choose a query editor The InfluxDB data source has three query editors, each corresponding to the query language selected in the [data source configuration](https://grafana.com/docs/grafana//datasources/influxdb/configure/): - [InfluxQL](#influxql-query-editor) - [SQL](#sql-query-editor) - [Flux](#flux-query-editor) Editor options vary based on query language. ## InfluxQL query editor The InfluxQL query editor helps you select metrics and tags to create InfluxQL queries. There are two modes: `visual editor mode` and `raw query mode`. To switch between the two modes click the **pencil icon** in the upper right. Visual query editor mode contains the following components: - **FROM** - Select a measurement to query. - **WHERE** - Select filters by clicking the **+ sign**. You can enter regular expressions or use the text input for regular expression tag values. - **SELECT** - Select fields and functions from the drop-down. You can add multiple fields and functions by clicking the **+ sign**. - **GROUP BY** - Select a tag from the drop-down menu. - **TIMEZONE** - _Optional_ Group data by a specific timezone. - **ORDER BY TIME** - Sort data by time in either ascending or descending order. - **LIMIT** - _Optional_ Limits the number of rows returned by the query. - **SLIMIT** - _Optional_ Limits the number of series returned by the query. Refer to [SLIMIT clause](https://docs.influxdata.com/influxdb/cloud/query-data/influxql/explore-data/limit-and-slimit/#slimit-clause) for more information. - **FORMAT AS** - Select a format option from the drop-down menu: time series, table, or logs. - **ALIAS** - Add an alias. Refer to [Alias patterns](#alias-patterns) for more information. ### Raw query editor mode You can write raw InfluxQL queries by switching to raw query mode. Click the pencil in the upper right of the query editor to switch modes. When you switch to visual editor mode, any changes made in raw query mode are lost. If you use raw query mode, your query must include `WHERE $timeFilter`. You should also provide a group by time and an aggregation function. Otherwise, InfluxDB may return hundreds of thousands of data points, potentially causing your browser to hang. ### Match by regular expressions You can enter regular expressions for metric names or tag filter values. Wrap the regular expression pattern in forward slashes (`/`), as shown in this example: `/measurement/`. Grafana automatically adjusts the filter tag condition to use the InfluxDB regular expression match condition operator (`=~`). ### Field and aggregation functions In the `SELECT` row, you can specify which fields and functions to use. If you **group by time** you must use an aggregation function. Certain functions such as `derivative` also require an aggregation function. The query editor input generates an InfluxDB `SELECT` clause. For example: ```sql SELECT derivative(mean("value"), 10s) / 10 AS "REQ/s" FROM.... ``` You can also use a \* in a SELECT statement to select all fields. ```sql SELECT * FROM ``` ### GROUP BY results To group results by a tag, specify the tag in the **GROUP BY** row: 1. Click the **+ sign** in the GROUP BY row. 1. Select a tag from the drop-down. You can GROUP BY multiple options. To remove a GROUP BY option click the **X icon** next to the option. ### Alias patterns | Alias pattern | Replaced with | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `$m` | Measurement name. | | `$measurement` | Measurement name. | | `$1` - `$9` | Part of measurement name (if you separate your measurement name with dots). | | `$col` | Column name. | | `$tag_exampletag` | The value of the `exampletag` tag. The syntax is `$tag_yourTagName` and must start with `$tag_`. To use your tag as an alias in the ALIAS BY field, you must use the tag to group by in the query. | You can also use the `[[tag_hostname]]` pattern replacement syntax. For example, entering the value `Host: [[tag_hostname]]` in the ALIAS BY field replaces it with the `hostname` tag value for each legend value. An example legend value is `Host: server1`. ## SQL query editor Grafana supports the [SQL query language](https://docs.influxdata.com/influxdb/cloud-serverless/query-data/sql/) for InfluxDB 3.x and newer cloud products (Cloud Serverless, Cloud Dedicated, Clustered). SQL is InfluxData's recommended query language for new deployments. The SQL query editor provides two modes: - **Builder mode** - A visual query builder that helps you construct SQL queries by selecting tables, columns, filters, and aggregations from drop-down menus. This mode is useful for building queries without writing SQL directly. - **Code mode** - A text editor for writing raw SQL queries with autocomplete and syntax highlighting. Switch between modes using the toggle at the top of the query editor. SQL queries connect to InfluxDB using the FlightSQL (gRPC) protocol, which provides high-performance data transfer for large result sets. ### Format output The SQL query editor supports two output formats: - **Time series** (default) - Returns data as time-series frames. The query must include a timestamp column. - **Table** - Returns data as a table frame, useful for non-time-series queries or when you want to display raw results. Select the format from the **Format** drop-down in the query editor. ### Macros You can use macros in your query to automatically substitute them with values from the Grafana context. | Macro example | Replaced with | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `$__timeFrom` | The start of the currently active time selection, such as `cast('2020-06-11T13:31:00Z' as timestamp)`. | | `$__timeTo` | The end of the currently active time selection, such as `cast('2020-06-11T14:31:00Z' as timestamp)`. | | `$__timeFilter()` | A time range filter applied to the specified column. Expands to ` >= $__timeFrom AND <= $__timeTo`. | | `$__interval` | An interval string that corresponds to the Grafana calculated interval based on the time range of the active time selection, such as `interval '5 second'`. | | `$__dateBin()` | Applies the [`date_bin`](https://docs.influxdata.com/influxdb/cloud-serverless/reference/sql/functions/time-and-date/#date_bin) function using `$__interval`. Column must be a timestamp. | | `$__dateBinAlias()` | Applies the [`date_bin`](https://docs.influxdata.com/influxdb/cloud-serverless/reference/sql/functions/time-and-date/#date_bin) function with a `_binned` suffix alias. Column must be a timestamp. | | `$__timeGroup()` | Groups results by a time interval using `date_bin()` with `$__interval`. Column must be a timestamp. | | `$__timeGroupAlias()` | Groups results by a time interval using `date_bin()` with `$__interval` and adds a `_binned` suffix alias. Column must be a timestamp. | ### SQL query examples **Basic time-series query:** ```sql SELECT time, usage_system, usage_user FROM cpu WHERE $__timeFilter(time) ORDER BY time ``` **Aggregated time-series query with date_bin:** ```sql SELECT $__dateBin(time) AS time, mean(usage_system) AS avg_system, mean(usage_user) AS avg_user FROM cpu WHERE $__timeFilter(time) GROUP BY $__dateBin(time) ORDER BY time ``` **Using JOINs across tables:** SQL supports JOINs, which aren't available in InfluxQL or Flux: ```sql SELECT c.time, c.usage_system, m.used_percent AS mem_used FROM cpu c INNER JOIN mem m ON c.time = m.time AND c.host = m.host WHERE $__timeFilter(c.time) ORDER BY c.time ``` **Filtering with WHERE clauses:** ```sql SELECT $__dateBin(time) AS time, mean(usage_system) AS avg_system FROM cpu WHERE $__timeFilter(time) AND host = 'server01' AND cpu = 'cpu-total' GROUP BY $__dateBin(time) ORDER BY time ``` For the full list of supported SQL statements, operators, and functions, refer to InfluxData's [SQL reference documentation](https://docs.influxdata.com/influxdb/cloud-serverless/reference/sql/). ## Flux query editor Grafana supports Flux when running InfluxDB v1.8 and higher. If your data source is [configured for Flux](https://grafana.com/docs/grafana//datasources/influxdb/configure/), you can use the [Flux](https://docs.influxdata.com/flux/v0/) in the query editor, which serves as a text editor for raw Flux queries with macro support. For more information and connection details, refer to [InfluxDB 1.8 API compatibility](https://github.com/influxdata/influxdb-client-go/#influxdb-18-api-compatibility). ### Use macros You can enter macros in the query to replace them with values from the Grafana context. Macros support copying and pasting from [InfluxData Chronograf](https://www.influxdata.com/time-series-platform/chronograf/). | Macro example | Replaced with | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `v.timeRangeStart` | The start of the currently active time selection, such as `2020-06-11T13:31:00Z`. | | `v.timeRangeStop` | The end of the currently active time selection, such as `2020-06-11T14:31:00Z`. | | `v.windowPeriod` | An interval string compatible with Flux that corresponds to the Grafana calculated interval based on the time range of the active time selection, such as `5s`. | | `v.defaultBucket` | The **Default Bucket** value from the data source configuration. | | `v.organization` | The data source configuration's **Organization** setting. | For example, consider the following Flux query: ```flux from(bucket: v.defaultBucket) |> range(start: v.timeRangeStart, stop: v.timeRangeStop) |> filter(fn: (r) => r["_measurement"] == "cpu" or r["_measurement"] == "swap") |> filter(fn: (r) => r["_field"] == "usage_system" or r["_field"] == "free") |> aggregateWindow(every: v.windowPeriod, fn: mean) |> yield(name: "mean") ``` Grafana interpolates this Flux query into the following and sends it to InfluxDB, with the interval and time period values changing according to the active time selection: ```flux from(bucket: "grafana") |> range(start: 2020-06-11T13:59:07Z, stop: 2020-06-11T14:59:07Z) |> filter(fn: (r) => r["_measurement"] == "cpu" or r["_measurement"] == "swap") |> filter(fn: (r) => r["_field"] == "usage_system" or r["_field"] == "free") |> aggregateWindow(every: 2s, fn: mean) |> yield(name: "mean") ``` To view the interpolated version of a query with the Query inspector, refer to [Panel Inspector](https://grafana.com/docs/grafana//panels-visualizations/panel-inspector/). ## Query logs You can query and display log data from InfluxDB in [Explore](https://grafana.com/docs/grafana//explore/) and in the dashboard [Logs panel](https://grafana.com/docs/grafana//panels-visualizations/visualizations/logs/). **InfluxQL:** Select an InfluxDB data source in the query editor. Under the **Select measurement field** next to the **FROM** section, choose a measurement containing your log data, then choose the appropriate fields that display the log message. Add any additional filters by clicking the **+ sign** next to the **WHERE** field. Set **FORMAT AS** to **Logs**. **SQL:** Write a SQL query that returns a timestamp column and a text column containing the log message. Set the **Format** to **Table**. For example: ```sql SELECT time, message, level, host FROM syslog WHERE $__timeFilter(time) ORDER BY time DESC ``` **Flux:** Use a Flux query with the `_value` field containing the log message. After InfluxDB returns the results, the log panel displays log rows along with a bar chart. The x-axis represents time, while the y-axis shows the frequency or count. ## Next steps - [Use template variables](https://grafana.com/docs/grafana//datasources/influxdb/template-variables/) for dynamic dashboards. - [Set up annotations](https://grafana.com/docs/grafana//datasources/influxdb/annotations/) to overlay events on your graphs. - [Create alert rules](https://grafana.com/docs/grafana//datasources/influxdb/alerting/) based on your InfluxDB queries. # InfluxDB template variables Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana displays these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables. For general information about variables, refer to [Variables](https://grafana.com/docs/grafana//dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/). ## Before you begin - [Configure the InfluxDB data source](https://grafana.com/docs/grafana//datasources/influxdb/configure/). - Understand [Grafana template variables](https://grafana.com/docs/grafana//dashboards/variables/). ## Supported variable types | Variable type | Supported | | ------------- | ------------------- | | Query | Yes | | Custom | Yes | | Data source | Yes | | Ad-hoc filter | Yes (InfluxQL only) | To switch a dashboard between InfluxDB instances, use a data source variable. Don't use a query variable that returns data source names as a workaround. Query-based workarounds can break after Grafana upgrades because they depend on internal identifiers. Refer to [Add a data source variable](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#add-a-data-source-variable) for details. ## Where you can use variables Variable support differs by query editor and mode: | Editor | Variable support | | ---------------------- | ------------------------------------------------------------------------------------------------- | | InfluxQL visual editor | Drop-downs include your template variables alongside values from your database. | | InfluxQL raw mode | Full support. | | SQL builder mode | Drop-downs list only tables and columns from your database. Switch to code mode to use variables. | | SQL code mode | Full support. Press Ctrl+Space to show variable suggestions. | | Flux code editor | Full support. | ## Create a query variable By adding a query template variable, you can write an InfluxDB metadata exploration query. These queries return results such as measurement names, key names, and key values. For more information, refer to [Add a query variable](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#add-a-query-variable). ### InfluxQL query variable examples To create a variable containing all values for the `hostname` tag, use the following query in the **Query** field: ```sql SHOW TAG VALUES WITH KEY = "hostname" ``` You can fetch key names for a given measurement: ```sql SHOW TAG KEYS [FROM ] ``` You can list available measurements: ```sql SHOW MEASUREMENTS ``` ### Flux query variable examples For Flux-configured data sources, write a Flux query that returns a single column of values: ```flux import "influxdata/influxdb/schema" schema.tagValues(bucket: v.defaultBucket, tag: "hostname") ``` ### SQL query variable examples For SQL-configured data sources (InfluxDB 3.x), write an SQL query that returns a single column of values: ```sql SELECT DISTINCT hostname FROM cpu ``` You can list available tables: ```sql SHOW TABLES ``` You can list columns in a specific table: ```sql SHOW COLUMNS FROM cpu ``` ## Scope variables to the dashboard time range By default, metadata queries such as `SHOW TAG VALUES` return values from the entire retention period, not just the dashboard time range. If a host or sensor stopped reporting long ago, its values still appear in the variable drop-down as long as they exist within retention. To limit variable results to the dashboard time range: 1. Include a time condition in the variable query. Variable queries interpolate the same time macros as regular queries: `$timeFilter` for InfluxQL, `$__timeFilter()` for SQL, and `v.timeRangeStart` and `v.timeRangeStop` for Flux. 1. Set the variable's **Refresh** option to **On time range change** so the values update when the dashboard time range changes. **InfluxQL example:** ```sql SHOW TAG VALUES WITH KEY = "hostname" WHERE $timeFilter ``` **SQL example:** ```sql SELECT DISTINCT hostname FROM cpu WHERE $__timeFilter(time) ``` ## Chain or nest variables You can create nested variables, sometimes called [chained variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#chained-variables). For example, if you have a variable named `region`, you can configure the `hosts` variable to display only hosts from the selected region. **InfluxQL:** ```sql SHOW TAG VALUES WITH KEY = "hostname" WHERE region = '$region' ``` **SQL:** ```sql SELECT DISTINCT hostname FROM cpu WHERE region = '$region' ``` If you have a variable containing key names, you can use it in a **GROUP BY** clause. This allows you to adjust the grouping by selecting from the variable list at the top of the dashboard. ## Use filters InfluxDB supports the use of filters for InfluxQL. They allow you to define multiple key/value filters, which Grafana automatically applies to all your InfluxDB queries. Filters also support expressions. To add filters: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Filter and Group by**. 1. Enter a **Name** for the filter. 1. Select an option in the **Display** drop-down list to control where on the dashboard the filter is displayed. 1. Select your InfluxDB data source in the **Data source** drop-down list. For more information, refer to the [Filter and Group by documentation](https://grafana.com/docs/grafana//visualizations/dashboards/build-dashboards/filter-group-by/). ## Choose a variable syntax The InfluxDB data source supports two variable syntaxes for use in the **Query** field: - **`$`** - Use this syntax for standalone variable references. It doesn't allow you to use a variable in the middle of a word or expression. - **`${varname}`** - Use this syntax when you want to interpolate a variable in the middle of an expression. **InfluxQL examples:** ```sql SELECT mean("value") FROM "logins" WHERE "hostname" =~ /^$host$/ AND $timeFilter GROUP BY time($__interval), "hostname" ``` ```sql SELECT mean("value") FROM "logins" WHERE "hostname" =~ /^${host}$/ AND $timeFilter GROUP BY time($__interval), "hostname" ``` When you enable the **Multi-value** or **Include all value** options with InfluxQL, Grafana converts the labels from plain text to a regular expression-compatible string, so you must use `=~` instead of `=` and wrap the variable in a regular expression such as `/^$host$/`. For example, if `$host` is a multi-value variable and you select `server1` and `server2`, Grafana joins the values with `|`, wraps them in parentheses, and escapes any special characters. The interpolated query looks like this: ```sql SELECT mean("value") FROM "logins" WHERE "hostname" =~ /^(server1|server2)$/ AND $timeFilter GROUP BY time($__interval), "hostname" ``` **SQL examples:** ```sql SELECT $__dateBin(time), mean(usage_system) FROM cpu WHERE $__timeFilter(time) AND host = '$host' GROUP BY $__dateBin(time) ``` ```sql SELECT $__dateBin(time), mean(usage_system) FROM cpu WHERE $__timeFilter(time) AND host IN ($host) GROUP BY $__dateBin(time) ``` When you enable the **Multi-value** option with SQL, use the `IN` operator instead of `=` to match multiple values. ### Prevent unwanted value escaping When you use a variable inside a regular expression, or when the variable is multi-value, Grafana escapes special characters in the values so the query remains valid. If you need the literal, unmodified value instead, use the `raw` format option: ```sql SELECT mean("value") FROM "requests" WHERE "path" = '${path:raw}' AND $timeFilter GROUP BY time($__interval) ``` This is useful for custom variables whose values intentionally contain characters that Grafana would otherwise escape. For all available format options, refer to [Advanced variable format options](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/#advanced-variable-format-options). ### Templated dashboard example To view an example of a templated dashboard, refer to this [InfluxDB example dashboard](https://play.grafana.org/d/f62a0410-5abb-4dd8-9dfc-caddfc3e2ffd/eccb2445-b0a2-5e83-8e0f-6d5ea53ad575). # Troubleshoot InfluxDB data source issues This document provides solutions to common issues you may encounter when configuring or using the InfluxDB data source. Issues are organized to follow the typical setup and usage workflow. For configuration instructions, refer to [Configure the InfluxDB data source](https://grafana.com/docs/grafana//datasources/influxdb/configure/). ## Connection errors The following errors occur when Grafana can't establish or maintain a connection to InfluxDB. ### "Plugin health check failed" or "An error occurred within the plugin" **Symptoms:** - All panels using InfluxDB return "An error occurred within the plugin" - Adding a new InfluxDB data source fails with "Plugin health check failed" - Connection settings appear blank in the UI **Possible causes and solutions:** | Cause | Solution | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Platform outage | If all InfluxDB panels fail at the same time, check the [Grafana Cloud status page](https://status.grafana.com/) for active incidents before investigating your own configuration. The issue may be a transient platform problem that resolves without customer-side action. | | Authentication failure | Check the Grafana server logs for auth-related errors. Verify your credentials haven't expired or been rotated. | | Transient network issue | The error may self-resolve. Wait a few minutes and retry **Save & test**. If the issue persists, check network connectivity between Grafana and InfluxDB. | ### Failed to connect to InfluxDB **Error message:** `error performing influxQL query` or `error performing flux query` or `error performing sql query` **Cause:** Grafana can't establish a network connection to the InfluxDB server. **Solution:** 1. Verify that the InfluxDB URL is correct in the data source configuration. 1. Check that InfluxDB is running and accessible from the Grafana server. 1. Ensure the URL includes the protocol (`http://` or `https://`). 1. Verify the port is correct (the InfluxDB default API port is `8086`). 1. Ensure there are no firewall rules blocking the connection. 1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your InfluxDB instance is not publicly accessible. ### PDC connection fails with "no such host" **Error message:** `socks connect tcp ... -> influxdb.host:8086: dial tcp: lookup ... no such host` **Cause:** When using [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/), the InfluxDB URL can't be resolved through the SOCKS proxy tunnel. **Solution:** 1. Don't use `127.0.0.1` or `localhost` as the InfluxDB URL. PDC tunnels traffic over a SOCKS proxy, which can't resolve loopback addresses. 1. Use the machine's LAN IP address or a resolvable hostname instead. 1. Verify the hostname is resolvable from the network where the PDC agent is running. 1. If the error appeared suddenly without configuration changes, check the [Grafana Cloud status page](https://status.grafana.com/) for active incidents. ### PDC connection stops working after certificate renewal fails **Symptoms:** - A previously working PDC-connected data source stops working - The PDC agent logs show errors renewing or signing the SSH certificate - Queries fail with SOCKS proxy connection errors **Cause:** The PDC agent uses a Grafana Cloud API token to periodically renew its SSH certificate. If the token is incorrect, has expired, or has been deleted, certificate renewal fails and the connection stops working when the current certificate expires. **Solution:** 1. Check the PDC agent logs for authentication or certificate signing errors. 1. Verify the API token in the PDC agent configuration is valid. If the token has expired or been deleted, generate a new one from your PDC connection page in Grafana Cloud and update the agent configuration. 1. Restart the PDC agent after updating the token. 1. Refer to [Troubleshoot PDC issues](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/troubleshooting/) for agent error codes, log interpretation, and token management guidance. ### TLS certificate errors **Error message:** `x509: certificate signed by unknown authority` or similar TLS verification errors **Cause:** InfluxDB presents a TLS certificate that isn't signed by a certificate authority (CA) that the Grafana server trusts. This is common with self-signed certificates and internal corporate certificate authorities. TLS verification applies end to end, so this error also occurs when you connect through PDC. **Solution:** 1. The InfluxDB data source supports TLS configuration for each data source instance. In the data source settings, expand **Auth and TLS/SSL Settings**, enable **CA cert**, and paste your CA certificate. Refer to [Auth and TLS/SSL settings](https://grafana.com/docs/grafana//datasources/influxdb/configure/#auth-and-tlsssl-settings) for details. 1. If your InfluxDB server requires client certificates, enable **TLS client auth** and provide the server name, client certificate, and client key. 1. As a last resort, you can enable **Skip TLS verify** to bypass certificate validation. Grafana doesn't recommend this for production use because it disables protection against man-in-the-middle attacks. ### Request timed out **Error message:** `context deadline exceeded` or `request timeout` or `dial tcp :: i/o timeout` **Cause:** The connection to InfluxDB timed out before receiving a response. This is common after Grafana upgrades when infrastructure changes (such as database host migrations) happen at the same time. **Solution:** 1. Verify network connectivity from the Grafana server to your InfluxDB endpoint. Check DNS resolution, firewall rules, and port access. 1. Confirm the InfluxDB host IP address or hostname hasn't changed. This is especially important after infrastructure migrations or Grafana upgrades. 1. Check the network latency between Grafana and InfluxDB. 1. Verify that InfluxDB is not overloaded or experiencing performance issues. 1. Increase the timeout setting in the data source configuration under **Advanced HTTP Settings**. 1. Reduce the time range or complexity of your query. ## Authentication errors The following errors occur when there are issues with authentication credentials or permissions. ### Unauthorized (401) **Error message:** "401 Unauthorized" or "authorization failed" **Cause:** The authentication credentials are invalid or missing. Transient network issues can also surface as authentication errors even when your credentials are valid. **Solution:** 1. If the error is intermittent or appeared without a configuration change, wait a few minutes and retry **Save & test** to rule out a transient network issue before rotating credentials. 1. Verify that the token or password is correct in the data source configuration. 1. For Flux and SQL, ensure the token has not expired. 1. For InfluxQL with InfluxDB 2.x, verify the token is set as an `Authorization` header with the value `Token `. 1. For InfluxDB 1.x, verify the username and password are correct. 1. Check that the token has the required permissions to access the specified bucket or database. ### Forbidden (403) **Error message:** "403 Forbidden" or "access denied" **Cause:** The authenticated user or token doesn't have permission to access the requested resource. **Solution:** 1. Verify the token has read access to the specified bucket or database. 1. Check the token's permissions in the InfluxDB UI under **API Tokens**. 1. Ensure the organization ID is correct for Flux queries. 1. For InfluxQL with InfluxDB 2.x, verify the DBRP mapping is configured correctly. ## Configuration errors The following errors occur when the data source is not configured correctly. ### URL not configured **Error message:** `missing URL from datasource configuration` **Cause:** The data source URL field is empty. **Solution:** 1. Open the data source configuration in Grafana. 1. Enter the full URL of your InfluxDB instance in the **URL** field, including the protocol and port (for example, `http://localhost:8086`). 1. Click **Save & test** to verify the connection. ### Unknown influx version **Error message:** "unknown influx version" **Cause:** The query language is not properly configured in the data source settings. **Solution:** 1. Open the data source configuration in Grafana. 1. Verify that a valid query language is selected: **Flux**, **InfluxQL**, or **SQL**. 1. Match the query language to your InfluxDB version: | InfluxDB version | Recommended query language | Notes | | ---------------------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1.x | InfluxQL | Flux is available from 1.8+ but InfluxQL is the primary language. | | 2.x (OSS/Cloud) | Flux | InfluxQL is also available via the v1 compatibility API, but requires [DBRP mapping](https://docs.influxdata.com/influxdb/cloud/query-data/influxql/dbrp/). | | 3.x / Cloud Dedicated / Cloud Serverless | SQL or InfluxQL | Flux is not supported on InfluxDB 3.x. | Each query language uses a different API endpoint. If you select the wrong language for your InfluxDB version, health checks and queries fail. ### Invalid data source info received **Error message:** `invalid data source info received` **Cause:** The data source configuration is incomplete or corrupted. **Solution:** 1. Delete and recreate the data source. 1. Ensure all required fields are populated based on your query language: - **Flux:** URL, Organization, Token, Default Bucket - **InfluxQL:** URL, Database, User, Password - **SQL:** URL, Database, Token ### DBRP mapping required **Error message:** "database not found" or queries return no data with InfluxQL on InfluxDB 2.x **Cause:** InfluxQL queries on InfluxDB 2.x require a Database and Retention Policy (DBRP) mapping. **Solution:** 1. Create a DBRP mapping in InfluxDB using the CLI or API. 1. Refer to [Manage DBRP Mappings](https://docs.influxdata.com/influxdb/cloud/query-data/influxql/dbrp/) for guidance. 1. Verify the database name in Grafana matches the DBRP mapping. ### Browser access mode disabled **Error message:** `Direct browser access in the InfluxDB datasource is no longer available. Switch to server access mode.` **Cause:** The data source is configured for direct browser access, which is no longer supported. **Solution:** 1. Open the data source configuration in Grafana. 1. Change the access mode to **Server (default)**. 1. Click **Save & test** to verify the connection. ### Content Security Policy (CSP) violation **Symptoms:** - CSP violation errors in the browser console referencing the InfluxDB plugin - `net::ERR_ABORTED` on proxy requests - The InfluxDB plugin attempts direct browser-to-InfluxDB connections **Cause:** You're running an outdated version of Grafana. Browser access mode was removed in Grafana 9.2.0, and older versions may attempt direct browser connections that violate CSP policies. Current versions of the data source support only server (proxy) access, where the Grafana server sends all queries to InfluxDB and the browser never connects to InfluxDB directly. **Solution:** 1. Upgrade to the latest stable Grafana release. The InfluxDB data source requires Grafana 12.3.0 or later. 1. After upgrading, verify the data source access mode is set to **Server (default)**. ## Health check errors The following errors occur when clicking **Save & test** to validate the data source connection. Each query language uses a different health check query. ### Flux health check errors **"error performing flux query"** **Cause:** The health check query `buckets()` failed to execute. **Solution:** 1. Verify the InfluxDB URL is correct and reachable. 1. Check that the token is valid and has not expired. 1. Ensure the organization ID is correct. **"error reading buckets"** **Cause:** The `buckets()` query executed but returned an error. **Solution:** 1. Verify the token has permission to list buckets. 1. Check that the organization ID matches the token's organization. **"error getting flux query buckets"** **Cause:** The `buckets()` query executed without error but returned no data. **Solution:** 1. Verify the token has permission to list buckets. 1. Check that the organization ID is correct. 1. Ensure InfluxDB is running and accessible. ### InfluxQL health check errors **"error performing influxQL query"** **Cause:** The health check query `SHOW MEASUREMENTS` failed to execute. **Solution:** 1. Verify the InfluxDB URL is correct and reachable. 1. Check the username and password (or token for InfluxDB 2.x). 1. Verify the database name exists. **"error reading influxDB"** **Cause:** The `SHOW MEASUREMENTS` query executed but returned an error. **Solution:** 1. Verify the database name is correct. 1. Check that the user has permission to run `SHOW MEASUREMENTS`. 1. For InfluxDB 2.x, verify DBRP mapping is configured. **"error connecting InfluxDB influxQL"** **Cause:** The health check completed but the response couldn't be processed. **Solution:** 1. Verify the database name is correct. 1. Check that the user has permission to run `SHOW MEASUREMENTS`. 1. Ensure the database exists and contains measurements. 1. For InfluxDB 2.x, verify DBRP mapping is configured. ### SQL health check errors **"error performing sql query"** **Cause:** The health check query `select 1` failed to execute against the FlightSQL endpoint. **Solution:** 1. Verify the InfluxDB URL is correct. The SQL health check connects via gRPC (FlightSQL). 1. Check the token is valid and has the required permissions. 1. If using TLS, verify the certificate configuration. Toggle **Insecure Connection** if connecting without TLS. 1. Ensure the InfluxDB 3.x instance is running and the FlightSQL endpoint is accessible. ### 0 measurements found **Error message:** `datasource is working. 0 measurements found` **Cause:** The connection is successful, but the database contains no measurements. **Solution:** 1. Verify you are connecting to the correct database. 1. Check that data has been written to the database. 1. If the database is new, add some test data to verify the connection. ## Query errors The following errors occur when there are issues with query syntax or execution. ### Query syntax error **Error message:** "error parsing query: found THING" or "failed to parse query: found WERE, expected ; at line 1, char 38" **Cause:** The query contains invalid syntax. **Solution:** 1. Check your query syntax for typos or invalid keywords. 1. For InfluxQL, verify the query follows the correct syntax: ```sql SELECT FROM WHERE ``` 1. For SQL, verify your query uses standard SQL syntax supported by InfluxDB 3.x. Common issues include using InfluxQL-specific syntax (such as `GROUP BY time()`) in SQL mode. Refer to the [InfluxDB SQL reference](https://docs.influxdata.com/influxdb/cloud-serverless/reference/sql/) for supported functions. 1. For Flux, ensure proper pipe-forward syntax and function calls. 1. Use the InfluxDB UI or CLI to test your query directly. ### Query timeout limit exceeded **Error message:** "query-timeout limit exceeded" **Cause:** The query took longer than the configured timeout limit in InfluxDB. **Solution:** 1. Reduce the time range of your query. 1. Add more specific filters to limit the data scanned. 1. Increase the query timeout setting in InfluxDB if you have administrator access. 1. Optimize your query to reduce complexity. ### Too many series or data points **Error message:** "max-series-per-database limit exceeded" or "A query returned too many data points and the results have been truncated" **Cause:** The query is returning more data than the configured limits allow. **Solution:** 1. Reduce the time range of your query. 1. Add filters to limit the number of series returned. 1. Increase the **Max series** setting in the data source configuration under **Advanced Database Settings**. 1. Use aggregation functions to reduce the number of data points. 1. For SQL, use `$__dateBin(time)` to aggregate data into time buckets and reduce cardinality. 1. For Flux, use `aggregateWindow()` to downsample data. ### FlightSQL errors (SQL query language) **Error message:** Messages prefixed with `"flightsql: "` followed by a gRPC error description. **Cause:** The SQL (FlightSQL) backend encountered an error communicating with InfluxDB 3.x. **Possible causes and solutions:** | Error code | Cause | Solution | | ------------------ | -------------------------------------------------------- | ----------------------------------------------------- | | `InvalidArgument` | The SQL query syntax is invalid. | Check your SQL query for syntax errors. | | `PermissionDenied` | The token doesn't have access to the requested resource. | Verify the token has read access to the database. | | `NotFound` | The requested table or database doesn't exist. | Check the database name and table name in your query. | | `Unavailable` | The InfluxDB server is unreachable. | Verify InfluxDB is running and the URL is correct. | | `Unauthenticated` | The token is missing, invalid, or expired. | Update the token in the data source configuration. | ### No time column found **Error message:** "no time column found" **Cause:** The query result doesn't include a time column, which is required for time-series visualization. **Solution:** 1. Ensure your query includes a time field. 1. For Flux, verify the query includes `_time` in the output. 1. For SQL, ensure the query returns a timestamp column. 1. Check that the time field is not being filtered out or excluded. ## Annotation errors The following errors occur when using InfluxDB annotations on dashboards. ### "Query missing in annotation definition" **Cause:** The annotation query field is empty. **Solution:** 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Dashboard options** icon to open the sidebar. 1. Expand the **Annotations** section. 1. Select the InfluxDB annotation. 1. Click **Open query editor** to open the **Annotation Query** dialog box. 1. Enter a valid query in the **InfluxQL Query** field. The query must include `WHERE $timeFilter`. For example: ```sql SELECT title, description FROM events WHERE $timeFilter ORDER BY time ASC ``` ### "Flux requires the standard annotation query" **Cause:** A Flux data source is using the legacy InfluxQL annotation editor instead of the standard Flux query editor. **Solution:** 1. Delete the existing annotation query. 1. Create a new annotation query and select your Flux-configured InfluxDB data source. 1. Write a Flux query that returns data frames with time and text fields. For example: ```flux from(bucket: "events") |> range(start: v.timeRangeStart, stop: v.timeRangeStop) |> filter(fn: (r) => r["_measurement"] == "deployments") ``` ### Annotations don't appear on the graph **Cause:** Annotations are configured but aren't visible on the dashboard. **Solution:** 1. Verify the annotation query returns data by testing it in Explore. 1. Check the dashboard time range covers the time period of your annotation events. 1. Ensure the annotation toggle is enabled in the dashboard (check the annotation icon in the top menu bar). 1. For InfluxQL, confirm the query includes `WHERE $timeFilter`. 1. If your query returns multiple columns, verify the field mappings (**Text**, **Tags**, **TimeEnd**) are set correctly. ## Alerting errors The following errors occur when using InfluxDB queries with Grafana Alerting. ### Alert rule fails with template variable errors **Cause:** The alert query contains template variables such as `$hostname` or `$region`. **Solution:** Alert queries can't use template variables because Grafana evaluates alert rules on the backend without dashboard context. Replace template variables with hard-coded values: 1. Open the alert rule. 1. Replace any `$variable` references with literal values. 1. Save the alert rule. If you need the same query in both a dashboard panel and an alert rule, maintain two separate queries: one with variables for the dashboard and one with hard-coded values for alerting. ### Alert evaluation returns "no data" **Cause:** The alert query doesn't return time-series data that Grafana can evaluate. **Solution:** 1. Test the query in Explore first to verify it returns data. 1. For InfluxQL, ensure the query uses an aggregation function (such as `mean`, `sum`, `count`) with `GROUP BY time($__interval)`. 1. For Flux, use `aggregateWindow()` to produce time-bucketed results. 1. For SQL, use `$__dateBin(time)` or `$__timeGroup(time)` to aggregate by time. 1. Check that the alert evaluation time range contains data. Alerts use a fixed time range, not the dashboard's time picker. 1. Verify the data source connection is working by clicking **Save & test** in the data source settings. ## Template variable errors The following issues occur when using template variables with InfluxDB queries. ### Variable drop-down shows stale or historical values **Cause:** Metadata queries such as `SHOW TAG VALUES` return values from the entire retention period, not just the dashboard time range. Hosts or sensors that stopped reporting long ago still appear in the drop-down. **Solution:** 1. Add a time condition to the variable query, such as `WHERE $timeFilter` for InfluxQL or `WHERE $__timeFilter(time)` for SQL. 1. Set the variable's **Refresh** option to **On time range change**. 1. Refer to [Scope variables to the dashboard time range](https://grafana.com/docs/grafana//datasources/influxdb/template-variables/#scope-variables-to-the-dashboard-time-range) for examples. ### Multi-select variable breaks the query **Cause:** When a variable has the **Multi-value** or **Include all value** option enabled, Grafana interpolates the selected values as a regular expression group, such as `(server1|server2)`. Queries that compare with `=` or that don't wrap the variable in a regular expression fail or return no data. **Solution:** 1. For InfluxQL, use the `=~` operator and wrap the variable in a regular expression: `"hostname" =~ /^$host$/`. 1. For SQL, use the `IN` operator: `host IN ($host)`. 1. Refer to [Choose a variable syntax](https://grafana.com/docs/grafana//datasources/influxdb/template-variables/#choose-a-variable-syntax) for working examples. ### Variable values are escaped unexpectedly **Cause:** Grafana escapes special characters in variable values when the variable is multi-value or used inside a regular expression. Custom variable values that intentionally contain special characters, such as paths or expressions, arrive at InfluxDB modified. **Solution:** Use the `raw` format option, such as `${path:raw}`, to interpolate the literal value without escaping. Refer to [Prevent unwanted value escaping](https://grafana.com/docs/grafana//datasources/influxdb/template-variables/#prevent-unwanted-value-escaping) for details. ### Variable drop-down contains a blank or duplicate entry **Cause:** The variable query returns duplicate values. Grafana deduplicates results, but depending on the plugin version, duplicates can surface as a blank entry in the drop-down. **Solution:** 1. Update the variable query to return unique values. For SQL, use `SELECT DISTINCT`. For InfluxQL, `SHOW` metadata queries already return unique values, so check for duplicates across retention policies. 1. Alternatively, use the variable's **Regex** option to filter the results. ## Other common issues The following issues don't produce specific error messages but are commonly encountered during day-to-day use. ### "Data source was not found" **Symptoms:** - Dashboard panels display "data source \ was not found" - Manually re-running queries in the panel editor works after you select the data source again **Cause:** Dashboard panels reference an old or deleted data source UID. This happens when a data source is deleted and recreated, since the new data source gets a different UID. Dashboard schema migrations can also leave panels with invalid data source references, so verify your panels after a dashboard is migrated to a new schema version. **Solution:** 1. Find the current UID of your InfluxDB data source. Navigate to **Connections** > **Data sources**, open the data source, and copy the UID from the page URL. 1. To find every stale reference at once, open the dashboard's JSON model from the dashboard settings and search for the UID shown in the panel error message. 1. Fix the references by editing each affected panel and selecting the correct InfluxDB data source from the drop-down again, or by replacing the stale UID in the JSON model and saving the dashboard. 1. To avoid this issue, update existing data sources instead of deleting and recreating them. ### 404 Not Found when sending Telegraf metrics to Grafana Cloud **Error message:** "404 Not Found" when Telegraf writes to the Grafana Cloud InfluxDB-compatible endpoint. **Cause:** The Telegraf `influxdb_v2` output plugin isn't compatible with the Grafana Cloud metrics endpoint. This commonly occurs when using PrivateLink or the standard InfluxDB-compatible write endpoint. **Solution:** 1. Switch the Telegraf output plugin from `influxdb_v2` to `influxdb` (v1) in your Telegraf configuration. 1. Ensure the endpoint URL and credentials match those shown in your Grafana Cloud InfluxDB configuration page. 1. Restart Telegraf after making the change. ### Empty query results **Cause:** The query returns no data. **Solution:** 1. Verify the time range includes data in your database. 1. Check that the measurement and field names are correct. For SQL, table names in InfluxDB 3.x are case-sensitive. 1. Test the query directly in the InfluxDB UI or CLI. 1. Ensure filters are not excluding all data. 1. For SQL, verify the `$__timeFilter(time)` macro is included so the query uses the dashboard time range. 1. For InfluxQL, verify the retention policy contains data for the selected time range. ### Numeric values stored as strings display a blank panel **Symptoms:** - Time series panels show no data even though the query returns results - Values appear in the table view but can't be graphed **Cause:** InfluxDB sets a field's data type when the field is first written. If numeric values were written as strings, for example quoted in line protocol, InfluxDB returns them as strings and Grafana can't graph them. This is a data issue in InfluxDB, not in Grafana. **Solution:** 1. As a workaround, add the [Convert field type](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/transform-data/#convert-field-type) transformation to the panel and convert the field to **Number**. 1. To fix the root cause, correct the data type at write time. InfluxDB can't change the type of an existing field, so you may need to write the data to a new field or measurement. 1. In older Grafana versions, the **Convert field type** transformation converted null values to zeros. Current versions preserve null values. If you see nulls displayed as zeros after conversion, upgrade Grafana. ### Panel values differ between view and edit mode **Cause:** Grafana calculates the query interval (`$__interval`) from the time range and the maximum number of data points, which is based on the panel's width. A panel rendered in edit mode has a different width than in the dashboard view, so queries that group by `time($__interval)` can aggregate into different bucket sizes and display different values. Both results are correct. They're aggregated at different resolutions. **Solution:** 1. To make values consistent, set a fixed **Min interval** or **Max data points** value in the panel's query options. 1. Alternatively, use an explicit interval in the query, such as `GROUP BY time(1m)` instead of `GROUP BY time($__interval)`. ### Legend or tooltip colors don't match the series **Cause:** The query returns multiple series with the same name. This commonly happens when a query doesn't group by the tags that differentiate series, or when an alias hides the distinguishing information. Grafana assigns colors and tooltip values by series name, so duplicate names cause the legend and hover tooltip to display mismatched colors or values. **Solution:** 1. Group by the tags that make each series unique, such as `GROUP BY "hostname"`. 1. Use alias patterns such as `$tag_hostname` in the **ALIAS** field so each series has a unique display name. Refer to [Alias patterns](https://grafana.com/docs/grafana//datasources/influxdb/query-editor/#alias-patterns) for details. 1. Use the panel inspector to confirm each returned series has a distinct name. ### Slow query performance **Cause:** Queries take a long time to execute. **Solution:** 1. Reduce the time range of your query. 1. Add more specific filters to limit the data scanned. 1. Increase the **Min time interval** setting to reduce the number of data points. 1. Check InfluxDB server performance and resource utilization. 1. For SQL, use `$__dateBin(time)` with aggregation functions to downsample data. Add `WHERE` clauses to narrow the query scope. 1. For Flux, use `aggregateWindow()` to downsample data before visualization. 1. Consider using continuous queries or tasks to pre-aggregate data. ### InfluxDB receives a high volume of queries from Grafana **Symptoms:** - Unexpected query load on your InfluxDB server - You're unsure whether the traffic originates from Grafana **Cause:** Every panel on a dashboard sends its queries to InfluxDB each time the dashboard refreshes. Query volume scales with the number of panels, the number of queries per panel, the refresh interval, and the number of people viewing the dashboard at the same time. For example, a dashboard with 20 panels refreshing every 10 seconds generates at least 120 queries per minute for each viewer. **Solution:** 1. Increase the dashboard refresh interval or turn off auto-refresh where it isn't needed. 1. Reduce the number of panels and queries per dashboard. 1. Increase the **Min time interval** setting to reduce the resolution of each query. 1. In Grafana Enterprise and Grafana Cloud, enable [query caching](https://grafana.com/docs/grafana//administration/data-source-management/#query-and-resource-caching) so identical queries within the cache window are served from cache instead of hitting InfluxDB. 1. To confirm whether traffic originates from your Grafana Cloud instance, compare the source IP addresses against the [Grafana Cloud allowlist](https://grafana.com/docs/grafana-cloud/platform/security-and-account-management/security-and-access/allow-list/). ### Data appears delayed or missing recent points **Cause:** The visualization doesn't show the most recent data. **Solution:** 1. Check the dashboard time range and refresh settings. 1. Verify the **Min time interval** is not set too high. 1. Ensure InfluxDB has finished writing the data. 1. Check for clock synchronization issues between Grafana and InfluxDB. ## Enable debug logging {{< admonition type="caution" >}} InfluxDB plugin versions earlier than 13.1.0 could write API tokens to Grafana server logs in plain text at default log levels. Before troubleshooting with logs, upgrade to plugin version 13.1.0 or later. Treat log files as sensitive, and rotate any tokens that may have been exposed. {{< /admonition >}} To capture detailed error information for troubleshooting: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location). 1. Look for InfluxDB-specific entries that include request and response details. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you've tried the solutions in this guide and still encounter issues: 1. Check the [InfluxDB documentation](https://docs.influxdata.com/) for API-specific guidance. 1. Review the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review [InfluxDB data source issues on GitHub](https://github.com/grafana/grafana-influxdb-datasource/issues) for known bugs. 1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user. 1. When reporting issues, include: - Grafana version - InfluxDB data source plugin version, found on the **Administration > Plugins** page - InfluxDB version and product (OSS, Cloud, Enterprise) - Query language (Flux, InfluxQL, or SQL) - Error messages (redact sensitive information) - Steps to reproduce - Relevant configuration such as data source settings, HTTP method, and TLS settings (redact tokens, passwords, and other credentials) # Configure the Jaeger data source This document explains how to configure the Jaeger data source in Grafana, including connection settings, authentication, trace correlation, and provisioning. ## Before you begin Before configuring the data source, ensure you have: - **Grafana permissions:** Organization administrator role - **Jaeger instance:** A running Jaeger instance with an accessible query endpoint (default: `http://localhost:16686`) ## Add the data source To add the Jaeger data source: 1. Click **Connections** in the left-side menu. 1. Click **Add new connection**. 1. Type `Jaeger` in the search bar. 1. Select **Jaeger**. 1. Click **Add new data source**. ## Connection settings Configure the basic connection options for the data source. | Setting | Description | | ----------- | ------------------------------------------------------------------ | | **Name** | The name used to refer to the data source in panels and queries. | | **Default** | Toggle to make this the default data source for new panels. | | **URL** | The URL of your Jaeger instance, such as `http://localhost:16686`. | ## Authentication The Jaeger data source supports the following authentication methods. ### Basic authentication To use basic authentication: 1. Select **Basic auth** from the drop-down. 1. Enter the **User** name. 1. Enter the **Password**. ### Forward OAuth identity Select **Forward OAuth identity** from the drop-down to forward the user's upstream OAuth identity to the data source. This is useful when Jaeger is behind an authenticating proxy. ### Custom headers You can add custom HTTP headers to requests sent to the Jaeger instance. Use this to pass authorization tokens or other required headers. To add custom headers, expand the **Additional settings** section and configure them under **Custom HTTP headers**. ### TLS configuration To configure TLS/mTLS for the connection, expand the **Additional settings** section and configure the following options: | Setting | Description | | ------------------- | ----------------------------------------------------------- | | **TLS client auth** | Toggle to enable client certificate authentication. | | **With CA cert** | Toggle to provide a custom CA certificate. | | **Skip TLS verify** | Toggle to skip TLS certificate verification (testing only). | | **Server name** | Server name for TLS certificate validation. | ### Private data source connect **Private data source connect** - _Only for Grafana Cloud users._ {{< admonition type="note" >}} This section is only visible when the Grafana instance has the secure SOCKS proxy feature enabled. {{< /admonition >}} Private data source connect, or PDC, allows you to establish a private, secured connection between a Grafana Cloud instance, or stack, and data sources secured within a private network. Click the drop-down to locate the URL for PDC. For more information regarding Grafana PDC refer to [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) and [Configure Grafana private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc) for instructions on setting up a PDC connection. Click **Manage private data source connect** to open your PDC connection page and view your configuration details. ## Trace to logs {{< figure src="/media/docs/tempo/tempo-trace-to-logs-9-4.png" max-width="800px" class="docs-image--no-shadow" caption="Trace to logs configuration settings" >}} The **Trace to logs** setting configures the [trace to logs feature](https://grafana.com/docs/grafana//explore/trace-integration/) that links trace spans to related log entries. {{< admonition type="note" >}} If you use Grafana Cloud, open a [support ticket in the Cloud Portal](/profile/org#support) to access this feature. {{< /admonition >}} There are two ways to configure trace to logs: - **Simple configuration:** Use a default query with tag-based filtering. - **Custom query:** Write a query using [template variable syntax](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/) to interpolate values from the trace or span. ### Use a simple configuration 1. Select the target data source from the drop-down list. 1. Set **Span start time shift** and **Span end time shift** to widen or shift the time range if log timestamps don't exactly match span timestamps. 1. Configure **Tags** to use in the logs query. Tags must be present in span attributes or resources for the link to appear. You can remap tag names if the target data source doesn't allow dots in labels (for example, remap `http.status` to `http_status`). 1. Optionally, toggle on **Filter by trace ID** or **Filter by span ID** to further filter logs. ### Configure a custom query 1. Select the target data source from the drop-down list. 1. Set **Span start time shift** and **Span end time shift**. 1. Optionally, configure **Tags** to map. Use the `${__tags}` variable in your custom query to interpolate mapped tags. If you don't map tags, you can still reference any span tag directly, such as `method="${__span.tags.method}"`. 1. Toggle on **Use custom query**. 1. Write a custom query using the variables listed in this table. The link only appears when all variables resolve to non-empty values. ### Variables for custom queries To use a variable, wrap it in `${}`. For example: `${__span.name}`. | Variable | Description | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | **\_\_tags** | Interpolates mapped tags as a label matcher string in the target data source syntax. Only includes tags present in the span. | | **\_\_span.spanId** | The ID of the span. | | **\_\_span.traceId** | The ID of the trace. | | **\_\_span.duration** | The duration of the span. | | **\_\_span.name** | The name of the span. | | **\_\_span.tags** | Namespace for span tags. Access a specific tag with `${__span.tags.version}`. For tags with dots, use `${__span.tags["http.status"]}`. | | **\_\_trace.traceId** | The ID of the trace. | | **\_\_trace.duration** | The duration of the trace. | | **\_\_trace.name** | The name of the trace. | ### Trace to logs settings | Setting | Description | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Data source** | The target logs data source. You can select Loki or Splunk logs data sources. | | **Span start time shift** | Shifts the start time for the logs query based on the span's start time. Use time units such as `5s`, `1m`, `3h`. Use negative values to extend to the past. Default: `0`. | | **Span end time shift** | Shifts the end time for the logs query based on the span's end time. Default: `0`. | | **Tags** | Tags to use in the logs query. Default: `cluster`, `hostname`, `namespace`, `pod`, `service.name`, `service.namespace`. | | **Filter by trace ID** | Toggles whether to append the trace ID to the logs query. | | **Filter by span ID** | Toggles whether to append the span ID to the logs query. | | **Use custom query** | Toggles the use of a custom query with variable interpolation. | | **Query** | The custom query. Use variable interpolation to customize with values from the span. | ## Trace to metrics The **Trace to metrics** setting lets you link trace spans to related metrics queries. To configure trace to metrics: 1. Select the target metrics data source from the drop-down list. 1. Configure **Tags** to map span attributes to metric label names. For example, map `k8s.pod` to `pod`. 1. Create linked queries using the `$__tags` keyword to interpolate the mapped tags. | Setting | Description | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | **Data source** | The target metrics data source. | | **Tags** | Maps span attribute names to metric label names. For example, map `k8s.pod` to `pod`. Use the `$__tags` keyword to interpolate tags in queries. | Each linked query consists of: - **Link Label:** _(Optional)_ A descriptive label for the linked query. - **Query:** The query to run when navigating from a trace to the metrics data source. Use `$__tags` to interpolate tags. For example, the query `requests_total{$__tags}` with tags `k8s.pod=pod` and `cluster` produces `requests_total{pod="nginx-554b9", cluster="us-east-1"}`. ## Node graph The **Node graph** setting is located under **Additional settings**, which is collapsed by default. Expand the **Additional settings** section on the data source configuration page to access it. The **Node graph** setting enables the [Node Graph visualization](https://grafana.com/docs/grafana//panels-visualizations/visualizations/node-graph/), which is disabled by default. When enabled, Grafana displays the Node Graph above the trace view, showing the relationships between services in the trace. ## Span bar The **Span bar** setting is located under **Additional settings**, which is collapsed by default. Expand the **Additional settings** section on the data source configuration page to access it. The **Span bar** setting controls what additional information appears in the span bar row of the trace view. | Option | Description | | ------------ | ----------------------------------------------------------------------------------------------- | | **None** | Adds nothing to the span bar row. | | **Duration** | _(Default)_ Displays the span duration on the span bar row. | | **Tag** | Displays a span tag value on the span bar row. Specify the tag key to use, such as `component`. | ## Trace ID time parameters The **Query Trace by ID with Time Params** setting is located under **Additional settings**, which is collapsed by default. Expand the **Additional settings** section on the data source configuration page to access it. The **Enable Time Parameters** toggle controls whether Grafana sends `start` and `end` time parameters when querying a trace by ID. Enable this if your Jaeger instance benefits from time-bounded trace lookups. ## Verify the connection Click **Save & test** to verify the data source connection. A successful test displays the message **Data source is working**. If the test fails, refer to [Troubleshoot Jaeger data source issues](https://grafana.com/docs/grafana//datasources/jaeger/troubleshooting/) for help resolving common errors. ## Provision the data source You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, refer to [Provisioning Grafana](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ### Provisioning example ```yaml apiVersion: 1 datasources: - name: Jaeger type: jaeger uid: jaeger-ds url: http://localhost:16686 access: proxy basicAuth: true basicAuthUser: isDefault: false jsonData: tracesToLogsV2: datasourceUid: 'loki' spanStartTimeShift: '1h' spanEndTimeShift: '-1h' tags: - key: 'job' - key: 'instance' - key: 'pod' - key: 'namespace' filterByTraceID: false filterBySpanID: false customQuery: true query: 'method="$${__span.tags.method}"' tracesToMetrics: datasourceUid: 'prom' spanStartTimeShift: '-2m' spanEndTimeShift: '2m' tags: - key: 'service.name' value: 'service' - key: 'job' queries: - name: 'Sample query' query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))' nodeGraph: enabled: true traceIdTimeParams: enabled: true spanBar: type: 'Duration' secureJsonData: basicAuthPassword: ``` Replace `` and `` with your Jaeger credentials. ## Provision with Terraform You can provision the Jaeger data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation. ### Terraform example The following example provisions a Jaeger data source: ```hcl terraform { required_providers { grafana = { source = "grafana/grafana" version = ">= 2.0.0" } } } provider "grafana" { url = "" auth = "" } resource "grafana_data_source" "jaeger" { type = "jaeger" name = "Jaeger" url = "http://localhost:16686" json_data_encoded = jsonencode({ nodeGraph = { enabled = true } traceIdTimeParams = { enabled = true } tracesToLogsV2 = { datasourceUid = "loki" filterByTraceID = true filterBySpanID = false tags = [ { key = "service.name", value = "service" }, { key = "job" } ] } }) } ``` ### Terraform example with basic authentication The following example provisions a Jaeger data source with basic authentication: ```hcl resource "grafana_data_source" "jaeger_auth" { type = "jaeger" name = "Jaeger" url = "http://localhost:16686" basic_auth_enabled = true basic_auth_username = "" json_data_encoded = jsonencode({ nodeGraph = { enabled = true } traceIdTimeParams = { enabled = true } }) secure_json_data_encoded = jsonencode({ basicAuthPassword = "" }) } ``` Replace the following placeholders: - _``_: Your Grafana instance URL (for example, `https://your-org.grafana.net` for Grafana Cloud) - _``_: A service account token with data source permissions - _``_: The username for basic authentication - _``_: The password for basic authentication For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). # Jaeger query editor This document explains how to use the Jaeger query editor to search for traces, query by trace ID, visualize service dependencies, and import trace files. For general information about querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/). ## Before you begin - Ensure you have [configured the Jaeger data source](https://grafana.com/docs/grafana//datasources/jaeger/configure/). - Verify the connection is working by clicking **Save & test** in the data source settings. ## Query types The Jaeger query editor supports the following query types: - **Search:** Find traces by service, operation, tags, and duration. - **TraceID:** Query a specific trace by its ID. - **Dependency graph:** Visualize service dependencies within a time range. - **Import trace:** Upload a JSON trace file for visualization. ## Search for traces To search for traces: 1. Select **Search** from the **Query type** selector. 1. Fill out the search form: | Field | Description | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | | **Service Name** | Select a service from the drop-down list, or type to filter. Supports template variables. | | **Operation Name** | Select an operation for the chosen service. Select **All** to query all operations. This field is disabled until you select a service. | | **Tags** | Enter tags in [`logfmt`](https://brandur.org/logfmt) format, such as `error=true db.statement="select * from User"`. | | **Min Duration** | Filter traces with a duration greater than this value. Use formats like `1.2s`, `100ms`, or `500us`. | | **Max Duration** | Filter traces with a duration less than this value. Use the same format as **Min Duration**. | | **Limit** | Maximum number of traces to return. | {{< figure src="/static/img/docs/explore/jaeger-search-form.png" class="docs-image--no-shadow" caption="Jaeger query editor showing a search query" >}} ## Query by trace ID To query a specific trace: 1. Select **TraceID** from the **Query type** selector. 1. Enter the trace ID into the **Trace ID** field. 1. Press **Shift+Enter** to run the query. {{< figure src="/static/img/docs/explore/jaeger-trace-id.png" class="docs-image--no-shadow" caption="Jaeger query editor with TraceID selected" >}} ## Visualize the dependency graph The dependency graph query type displays service dependencies as a [Node Graph](https://grafana.com/docs/grafana//panels-visualizations/visualizations/node-graph/). It shows how services communicate and the volume of calls between them. To visualize the dependency graph: 1. Select **Dependency graph** from the **Query type** selector. 1. Set the dashboard time range to the period you want to analyze. 1. Run the query. The dependency graph uses the dashboard time range to query the Jaeger `/api/dependencies` endpoint. Grafana displays the result as a Node Graph with: - **Nodes:** Each node represents a service. - **Edges:** Each edge represents calls between services, with the call count shown as the edge label. {{< admonition type="note" >}} Your Jaeger instance must have dependency data available. If the graph is empty, verify that Jaeger is collecting and processing dependency information for the selected time range. {{< /admonition >}} ## Import a trace You can upload a JSON file that contains a single trace and visualize it in Grafana. If the file contains multiple traces, Grafana visualizes the first trace. To import a trace: 1. Click **Import trace** in the query editor. 1. Select a JSON file in the Jaeger trace format. {{< figure src="/static/img/docs/explore/jaeger-upload-json.png" class="docs-image--no-shadow" caption="Jaeger data source with import trace selected" >}} ### Trace JSON example The JSON file must follow the Jaeger trace format with a `data` array containing trace objects: ```json { "data": [ { "traceID": "2ee9739529395e31", "spans": [ { "traceID": "2ee9739529395e31", "spanID": "2ee9739529395e31", "flags": 1, "operationName": "CAS", "references": [], "startTime": 1616095319593196, "duration": 1004, "tags": [ { "key": "sampler.type", "type": "string", "value": "const" } ], "logs": [], "processID": "p1", "warnings": null } ], "processes": { "p1": { "serviceName": "loki-all", "tags": [ { "key": "jaeger.version", "type": "string", "value": "Go-2.25.0" } ] } }, "warnings": null } ], "total": 0, "limit": 0, "offset": 0, "errors": null } ``` ## Query data via gRPC endpoint (experimental) Jaeger offers an alternative method for querying data that uses their gRPC service over HTTP. For detailed information about the API and setup requirements, refer to the [Jaeger API documentation](https://www.jaegertracing.io/docs/2.12/architecture/apis/#query-json-over-http). The following queries are supported through the gRPC endpoint: - Service search - Operation search - Trace ID search To enable gRPC querying for Jaeger within Grafana, enable the `jaegerEnableGrpcEndpoint` feature flag. This feature is experimental. Grafana Cloud customers should contact support to request access and provide feedback on this feature. ## Use template variables The Jaeger query editor supports [Grafana template variables](https://grafana.com/docs/grafana//dashboards/variables/) for creating dynamic, reusable dashboards. You can use template variables in the following query fields: - **Service Name** - **Operation Name** - **Trace ID** - **Tags** - **Min Duration** - **Max Duration** Use standard Grafana variable syntax such as `$variable` or `${variable}` in these fields. For example, set **Service Name** to `$service` to let dashboard users select the service from a drop-down list. # Troubleshoot Jaeger data source issues This document provides solutions to common issues you may encounter when configuring or using the Jaeger data source. For configuration instructions, refer to [Configure the Jaeger data source](https://grafana.com/docs/grafana//datasources/jaeger/configure/). ## Connection errors These errors occur when Grafana can't reach the Jaeger instance. ### "Connection refused" or timeout errors **Symptoms:** - **Save & test** fails with a connection error - Queries fail with network errors - Service and operation drop-downs don't load **Possible causes and solutions:** | Cause | Solution | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Jaeger isn't running | Verify Jaeger is running and accessible at the configured URL. | | Incorrect URL | Check the **URL** setting in the data source configuration. The default Jaeger query endpoint is `http://localhost:16686`. | | Firewall blocking access | Ensure firewall rules allow traffic from the Grafana server to the Jaeger endpoint. | | Network segmentation | For Grafana Cloud accessing private Jaeger instances, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). | ### "request failed: 404 Not Found" **Symptoms:** - **Save & test** returns a 404 error - Queries fail with "request failed" messages **Solutions:** 1. Verify the Jaeger URL doesn't include a trailing path such as `/api`. The correct URL is the base endpoint, for example `http://localhost:16686`. 1. Confirm the Jaeger query service is running and listening on the expected port. 1. If using a reverse proxy, verify the proxy forwards requests to the correct Jaeger endpoint. ### TLS errors **Symptoms:** - Certificate validation errors during **Save & test** - Errors mentioning `x509` or certificate verification **Solutions:** 1. Verify your TLS certificates are valid and not expired. 1. Ensure the CA certificate is configured in the data source **Additional settings** if using a private CA. 1. For testing, you can temporarily toggle on **Skip TLS verify** in the data source settings. Don't use this in production. 1. Verify the **Server name** setting matches the certificate's Common Name or Subject Alternative Name. ### Timeout errors **Symptoms:** - Queries take a long time then fail - Errors mention timeout **Solutions:** 1. Increase the timeout value in **Additional settings** > **Timeout**. 1. Verify network latency between Grafana and Jaeger is acceptable. 1. Check that the Jaeger query service isn't overloaded. ## Authentication errors These errors occur when credentials are invalid or missing. ### "401 Unauthorized" or "403 Forbidden" **Symptoms:** - **Save & test** fails with authorization errors - Queries return access denied messages - Services and operations don't load in drop-downs **Possible causes and solutions:** | Cause | Solution | | ------------------------ | ------------------------------------------------------------------------------------------------ | | Invalid credentials | Verify the **User** and **Password** values in the data source configuration. | | Missing authentication | If Jaeger requires authentication, toggle on the appropriate auth method. | | Incorrect custom headers | Verify custom header names and values in **Additional settings**. | | OAuth forwarding issues | If using **Forward OAuth identity**, verify the upstream OAuth provider is configured correctly. | ## Query errors These errors occur when executing queries against Jaeger. ### "traceID is empty" **Symptoms:** - The query fails with "traceID is empty" when using the TraceID query type **Solutions:** 1. Enter a valid trace ID in the **Trace ID** field. 1. Verify the trace ID is a valid hexadecimal string. 1. If using a template variable, verify the variable resolves to a non-empty value. ### No data in search results **Symptoms:** - Search query returns no traces - The trace list is empty **Possible causes and solutions:** | Cause | Solution | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------- | | Time range doesn't contain data | Expand the dashboard or Explore time range. Verify traces exist in Jaeger for the selected period. | | Service or operation doesn't exist | Verify the selected service and operation names match what's in Jaeger. | | Tags syntax error | Ensure tags use [`logfmt`](https://brandur.org/logfmt) format: `error=true db.statement="select * from User"`. | | Duration format error | Use supported duration formats: `1.2s`, `100ms`, `500us`. | | Limit set too low | Increase the **Limit** value or leave it empty for the default. | ### "The JSON file uploaded is not in a valid Jaeger format" **Symptoms:** - Importing a trace file fails with a format error **Solutions:** 1. Verify the JSON file follows the Jaeger trace format with a `data` array at the root level. 1. Ensure the file contains at least one trace object in the `data` array. 1. Validate the JSON syntax using a JSON linter. Expected format: ```json { "data": [ { "traceID": "", "spans": [...], "processes": {...} } ] } ``` ## Dependency graph errors These errors relate to the dependency graph query type. ### Empty dependency graph **Symptoms:** - The dependency graph query returns no nodes or edges - The Node Graph panel is empty **Possible causes and solutions:** | Cause | Solution | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | No dependency data | Verify Jaeger is collecting and processing service dependency information. | | Time range too narrow | Expand the dashboard time range. Dependencies are calculated over the selected period. | | Jaeger storage backend | Some Jaeger storage backends may not support dependency queries. Refer to the [Jaeger documentation](https://www.jaegertracing.io/docs/) for storage-specific details. | ## gRPC endpoint errors These errors occur when using the gRPC query endpoint. ### gRPC queries fail **Symptoms:** - Queries don't use the gRPC endpoint - Search, service, and operation queries still use the REST API **Solutions:** 1. Verify the `jaegerEnableGrpcEndpoint` feature flag is enabled in Grafana. This feature is experimental. 1. Grafana Cloud customers should contact support to request access. 1. Note that search and dependency graph queries currently use the REST endpoint even when the feature flag is enabled. Only service search, operation search, and trace ID queries use gRPC. ## Enable debug logging To capture detailed error information for troubleshooting: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Reproduce the issue. 1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location). 1. Look for Jaeger-specific entries that include request URLs, response status codes, and error details. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you've tried the solutions in this document and still encounter issues: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs related to the Jaeger data source. 1. Refer to the [Jaeger documentation](https://www.jaegertracing.io/docs/) for service-specific guidance. 1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Advanced user. 1. When reporting issues, include: - Grafana version - Jaeger version - Error messages (redact sensitive information) - Steps to reproduce - Relevant data source configuration (redact credentials) # Loki alerting The Loki data source supports [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). You can alert on your logs so Grafana notifies you when a LogQL metric query crosses a threshold, for example when the rate of error logs spikes. Loki works with two kinds of alert rules: - **Grafana-managed alert rules:** Grafana evaluates the rule against a LogQL metric query. This is the most common approach and works with any Loki data source. - **Data source-managed alert rules:** The Loki ruler stores and evaluates the rule. Use these when you want alert rules to live alongside your Loki deployment. Enable **Manage alert rules in Alerting UI** on the data source to create and edit them from Grafana. ## Before you begin Before you create alert rules with Loki data, ensure you have: - A [configured Loki data source](https://grafana.com/docs/grafana//datasources/loki/configure/). - Permissions to create alert rules in Grafana. - Familiarity with [Grafana Alerting concepts](https://grafana.com/docs/grafana//alerting/fundamentals/). - For data source-managed rules, a Loki deployment with the ruler enabled, and the **Manage alert rules in Alerting UI** setting turned on in the data source configuration. To route the resulting alerts, add a separate [Alertmanager data source](https://grafana.com/docs/grafana//datasources/alertmanager/). ## Supported queries for alerting Alert rules require queries that return numeric data that Grafana can evaluate against a threshold. Wrap a Loki log query in a LogQL metric function, such as `rate` or `count_over_time`, to produce numeric time series. {{< admonition type="note" >}} Use a metric query for alerting, not a plain log query. A log query returns log lines, which Grafana can't evaluate against a threshold. {{< /admonition >}} The following queries are common starting points for alerting. Replace the label matchers with values from your own logs. Rate of error lines per second: ```logql sum(rate({app="my-app"} |= `error` [5m])) ``` Error rate broken down by status code, to alert per endpoint or service: ```logql sum by (status) (rate({app="nginx"} | logfmt | status=~`5..` [5m])) ``` Count of slow requests, for a latency alert: ```logql sum(count_over_time({app="my-app"} | logfmt | duration > 1s [5m])) ``` Detect a service that stopped logging by alerting when the count drops to zero: ```logql sum(count_over_time({app="my-app"} [10m])) ``` ## Create a Grafana-managed alert rule To create a Grafana-managed alert rule using Loki data: 1. Go to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for your alert rule. 1. In the **Define query and alert condition** section: - Select your Loki data source. - Enter a LogQL metric query. - Add a **Reduce** expression if your query returns multiple series. - Add a **Threshold** expression to define the alert condition. 1. Configure the **Set evaluation behavior** section: - Select or create a folder and evaluation group. - Set the evaluation interval and pending period. 1. Add labels and annotations to provide context for notifications. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example: alert on error log rate This example fires when the rate of log lines that contain `error` exceeds a threshold: 1. Create a new alert rule. 1. Configure the query, replacing the label matcher with your own: ```logql sum(rate({app="my-app"} |= `error` [5m])) ``` 1. Add expressions: - **Reduce**: Last, to get the most recent value. - **Threshold**: Is above `10`. 1. Set evaluation to run every 1 minute with a 5-minute pending period. 1. Save the rule. ## Manage data source-managed alert rules When you turn on **Manage alert rules in Alerting UI** for the Loki data source, you can create and edit alert and recording rules that the Loki ruler stores and evaluates. These rules appear in the Grafana Alerting UI alongside your Grafana-managed rules, grouped under the Loki data source. The Loki data source configuration doesn't include Alertmanager routing. To route the resulting alerts, add a separate [Alertmanager data source](https://grafana.com/docs/grafana//datasources/alertmanager/). For more information, refer to [Configure the Loki data source](https://grafana.com/docs/grafana//datasources/loki/configure/#alerting) and [Data source-managed alert rules](https://grafana.com/docs/grafana//alerting/alerting-rules/create-data-source-managed-rule/). ## Best practices Follow these recommendations to create reliable alerts with Loki data. ### Test queries before alerting Verify your query returns numeric data before you create an alert: 1. Go to **Explore**. 1. Select your Loki data source. 1. Run the LogQL metric query you plan to use for alerting. 1. Confirm the result is numeric and suitable for threshold evaluation. ### Reduce a time series to a single value A Grafana-managed rule evaluates a single number against the threshold. A `range` query returns a time series with many points, which the rule can't evaluate directly. Either set the query **Type** to `instant`, or add a **Reduce** expression, such as **Last** or **Mean**, between the query and the **Threshold** expression. Without one of these, the rule can evaluate incorrectly. ### Set a pending period to avoid false positives A transient condition, such as a brief network blip or a momentary spike in errors, can cross the threshold for a single evaluation. Set a pending period so the rule fires only after the condition holds for a sustained duration. A pending period of a few evaluation intervals reduces false positives from short-lived conditions. ### Handle no data versus zero results A LogQL metric query returns _no series_ when no log lines match, rather than returning a value of `0`. Grafana evaluates a rule that receives no series as **No Data**, not **Normal**, which can cause an alert to enter the No Data state instead of resolving. To avoid unexpected No Data states: - Apply the threshold in the alert condition rather than writing a query so restrictive that it returns nothing. For example, query the error rate and compare it in a **Threshold** expression, instead of writing a query that only returns lines above the threshold. - Set the No Data behavior under **Configure no data and error handling** to the state you want. The options are **No Data**, **Alerting**, **Normal**, and **Keep Last State**. Choose **Normal** if the absence of matching logs isn't a problem. {{< admonition type="note" >}} Editing and saving an alert rule resets its state to **Normal**. The rule re-enters **Pending** or **Alerting** on the next evaluation if the condition is still met. This is standard [Grafana Alerting](https://grafana.com/docs/grafana//alerting/) behavior and isn't specific to the Loki data source. {{< /admonition >}} ## Troubleshoot alerting If your Loki alerts don't work as expected, use the following sections to diagnose common issues. ### Alerts don't fire - Confirm the query is a LogQL metric query that returns numeric data in Explore. - Ensure the evaluation interval allows enough time for data to be available. - Review the alert rule's health and any error messages in the Alerting UI. ### Alert enters No Data unexpectedly - The query returned no series because no log lines matched. A LogQL metric query returns no series rather than `0` when nothing matches. Set the No Data behavior under **Configure no data and error handling**, or restructure the query so it returns a value that you compare with a threshold. - Confirm logs exist for the query and time range in Explore. ### Alert fires on brief spikes - Add or increase the pending period so the rule fires only when the condition holds over several evaluations. - Widen the range in the metric query, for example from `[1m]` to `[5m]`, so short-lived spikes have less effect. ### Data source-managed rules don't appear - Confirm **Manage alert rules in Alerting UI** is enabled in the Loki data source configuration. - Verify the Loki ruler is enabled and reachable from Grafana. For more help, refer to [Troubleshoot Loki issues](https://grafana.com/docs/grafana//datasources/loki/troubleshooting/). ## Related resources - [Grafana Alerting](https://grafana.com/docs/grafana//alerting/) - [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/) - [Loki query editor](https://grafana.com/docs/grafana//datasources/loki/query-editor/) # Loki annotations [Annotations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/) overlay rich event information on top of graphs. With the Loki data source, annotations are built from log queries: each log line that a LogQL query returns within the dashboard time range becomes an annotation, so you can correlate log events with the rest of your dashboard data. ## Before you begin Before you create Loki annotations, ensure you have: - A [configured Loki data source](https://grafana.com/docs/grafana//datasources/loki/configure/). - A [LogQL](https://grafana.com/docs/loki/latest/logql/) log query that returns the events you want to annotate. ## How Loki annotations work Unlike data sources that build annotations from a separate query language, Loki annotations use the same LogQL log queries you use elsewhere. Grafana runs your log query for the dashboard time range and turns each returned log line into an annotation. By default, Grafana uses the log content as the annotation text and the log stream labels as tags, so you don't need to create any additional mapping. You can only use log queries as a source for annotations. Metric queries aren't supported. ## Create an annotation query To add a Loki annotation to a dashboard: 1. Open the dashboard where you want to add annotations. 1. Click **Edit**, then click **Settings** in the top navigation. 1. Select the **Annotations** tab. 1. Click **Add annotation query**. 1. Enter a **Name** for the annotation, for example, `Deploy events`. 1. Select your **Loki** data source. 1. Enter a LogQL log query and configure the optional fields described in the following table. 1. Click **Save dashboard**. The annotation query editor provides the following fields: | Field | Description | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Query** | The LogQL log query that selects the log lines to annotate. | | **Title** | _Optional._ A literal name or a template for the annotation title. Reference labels with `{{label}}`, for example `{{level}}`. | | **Tags** | _Optional._ A comma-separated list of label keys to use as annotation tags, for example `app,level`. When empty, Grafana uses the log stream labels as tags. | | **Text** | _Optional._ A literal string or a template for the annotation text. Reference labels with `{{label}}`. When empty, Grafana uses the log line content as the text. | Template fields resolve against the labels on each returned log line, including parsed fields. To reference a value that isn't an indexed label, add a parser such as `logfmt` or `json` to your query so the field is available. ### Example To annotate deployment events, configure the annotation query with the following values: | Field | Value | | --------- | ----------------------------------- | | **Query** | `{app="deploy-bot"} \|= "deployed"` | | **Title** | `{{app}} deployed` | | **Tags** | `app,env` | | **Text** | `{{message}}` | Each matching log line becomes an annotation titled with the deploying app, tagged with the `app` and `env` labels, and described with the parsed `message` field. Because the **Text** template references a parsed field, the query needs a parser, for example: ```logql {app="deploy-bot"} |= `deployed` | logfmt ``` ## Troubleshoot annotations If annotations don't appear as expected, try the following solutions. ### Annotations don't appear - Verify the log query returns results in [Explore](https://grafana.com/docs/grafana//explore/) for the same time range. Annotations come from returned log lines, so a query that returns no logs produces no annotations. - Confirm the query is a log query, not a metric query. - Widen the dashboard time range to include the log events you expect. ### Too many annotations appear - Add label filters or a line filter to your LogQL query to narrow the returned log lines. - Reduce the maximum number of lines the query returns. ## Related resources - [Annotate visualizations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/) - [Loki query editor](https://grafana.com/docs/grafana//datasources/loki/query-editor/) - [Configure the Loki data source](https://grafana.com/docs/grafana//datasources/loki/configure/) # Configure the Loki data source This document explains how to configure the Loki data source and describes the available configuration options. For general information on adding and managing data sources, refer to [Data source management](https://grafana.com/docs/grafana//administration/data-source-management/). Grafana includes built-in support for Loki, so you don't need to install a plugin. ## Before you begin Before you configure the data source, ensure you have: - **Grafana permissions:** The `Organization administrator` role to add and configure data sources. Administrators can also configure the data source with [YAML provisioning](#provision-the-data-source). - **Loki server URL:** The address of your Loki server, including the port. Loki listens on port `3100` by default. - **Authentication details:** Any credentials or certificates your Loki server requires, such as a basic auth user and password or TLS certificates. {{< admonition type="note" >}} Use TLS (Transport Layer Security) for an additional layer of security when working with Loki. For information on setting up TLS encryption with Loki, refer to [Grafana Loki configuration parameters](https://grafana.com/docs/loki/latest/configure/). {{< /admonition >}} ## Add the Loki data source To add the Loki data source: 1. Click **Connections** in the left-side menu. 1. Click **Add new connection**. 1. Type `Loki` in the search bar. 1. Select the **Loki** data source. 1. Click **Add new data source** in the upper right. Grafana takes you to the **Settings** tab, where you set up your Loki configuration. ## Configure the data source in the UI The following sections describe the configuration options available for the Loki data source. The first options set the name of your connection: | Setting | Description | | ----------- | ---------------------------------------------------------------------------------------------------------------------- | | **Name** | The data source name. This is how you refer to the data source in panels and queries. Examples: `loki-1`, `loki_logs`. | | **Default** | Toggle to make this the default data source. Grafana selects it by default in new panels and queries. | ### Connection Configure how Grafana connects to your Loki server. | Setting | Description | | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **URL** | The base URL of your Loki server. If Loki runs locally, use `http://localhost:3100`. If Loki runs on a networked server, use its URL and port, for example `http://loki.example.com:3100`. | {{< admonition type="note" >}} Enter only the base URL. Don't append API paths such as `/loki/api/v1/push`. That endpoint sends logs to Loki with an agent like Grafana Alloy; it isn't used to query Loki from the data source. If your Grafana instance runs on Grafana Cloud, a `localhost` or private network address refers to Grafana servers rather than your network, so it can't reach a self-hosted Loki. Use [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) to query Loki on a private network. {{< /admonition >}} ### Authentication Configure how Grafana authenticates with your Loki server. Select an authentication method and provide any required credentials. | Setting | Description | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | **Basic authentication** | The most common authentication method. Enter the user name and password for your Loki server. | | **Forward OAuth Identity** | Toggle on to forward the OAuth access token, and the OIDC ID token if available, of the user querying the data source. | | **With credentials** | Toggle on to send credentials such as cookies or authentication headers with cross-site requests. | | **TLS Client Auth** | Toggle on to authenticate with a client certificate. Provide the **Server name**, **Client cert**, and **Client key**. | | **With CA cert** | Toggle on to verify a self-signed TLS certificate. Follow the instructions from your certificate authority (CA) to obtain the certificate file. | | **Skip TLS verify** | Toggle on to bypass TLS certificate verification. Skipping verification isn't recommended unless absolutely necessary or for testing. | {{< admonition type="note" >}} For Grafana Cloud-hosted Loki, use **Basic authentication** with your Grafana Cloud user ID as the user name and a Cloud Access Policy token as the password. The token's [access policy](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) must include the `logs:read` scope. Create tokens in the Grafana Cloud Portal. A token's value is shown only once, so copy it when you create it. {{< /admonition >}} #### Custom HTTP headers Add custom HTTP headers to pass values that your Loki instance requires. | Setting | Description | | ---------- | ------------------------------ | | **Header** | The name of the custom header. | | **Value** | The value of the header. | For a multi-tenant Loki, one configured with `auth_enabled: true`, add the `X-Scope-OrgID` header with your tenant ID so Loki knows which tenant to query. Without it, queries against a multi-tenant Loki fail with an authentication error or return no data. ### Additional settings The **Additional settings** section is collapsible and contains optional settings that give you more control over the data source. It's open by default and includes advanced HTTP settings, alerting, query, and derived field options. #### Advanced HTTP settings Configure additional HTTP behavior for requests to Loki. | Setting | Description | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | **Allowed cookies** | Specify cookies by name that should be forwarded to the data source. By default, the Grafana proxy deletes forwarded cookies. | | **Timeout** | The HTTP request timeout, in seconds. There's no default, so set a value that suits your queries. | #### Secure Socks Proxy The **Secure Socks Proxy** settings appear only when the secure SOCKS data source proxy is enabled in the Grafana configuration file. When enabled, you can route data source requests through a secure SOCKS proxy. For more information, refer to [Configure a data source connection proxy](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/proxy/). #### Alerting Configure how the data source works with [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). | Setting | Description | | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Manage alert rules in Alerting UI** | Toggle on to manage data source-managed alert rules for the Loki data source in the Grafana Alerting UI. These rules are stored and evaluated by the Loki ruler. The default follows your Grafana configuration. | To manage other alerting resources, such as the alerts these rules generate, add an [Alertmanager data source](https://grafana.com/docs/grafana//datasources/alertmanager/). For more information on alerting with Loki, refer to [Loki alerting](https://grafana.com/docs/grafana//datasources/loki/alerting/). #### Queries Configure query behavior for the data source. | Setting | Description | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Maximum lines** | Sets the maximum number of log lines Loki returns. Increase the limit for a larger result set for ad hoc analysis. Decrease the limit if your browser is slow when displaying log results. The default is `1000`. | #### Derived fields Use derived fields to extract new fields from your logs and create a link from the value of the field. For example, you can link to your tracing backend directly from your logs, or link to a user profile page when a log line contains a corresponding `userId`. These links appear in the [log details](https://grafana.com/docs/grafana//explore/logs-integration/#labels-and-detected-fields). You can add multiple derived fields. {{< admonition type="note" >}} If you use Grafana Cloud, you can request modifications to this feature by opening a support ticket from the Grafana Cloud Portal. {{< /admonition >}} Each derived field has the following options: | Setting | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Name** | The field name. Displayed as a label in the log details. | | **Type** | The type of the derived field. Choose **Regex in log line** to parse a value from the log content, or **Label** to match a label key. | | **Regex** | When the type is **Regex in log line**, a regular expression that parses part of the log message. Include a capture group; Grafana uses the first captured group as the field value. | | **Label** | When the type is **Label**, the input matches as a regular expression against label keys, so a pattern like `trace[_]?id` matches variations such as `traceid` and `trace_id`. Matches any label: indexed, parsed, or structured metadata. | | **URL/query** | The full link URL for external links, or a query for the target data source for internal links. Interpolate the field value with the `${__value.raw}` macro. | | **URL Label** | _Optional._ A custom display label for the link. Overrides the default label, which is the full external URL or the name of the linked internal data source. | | **Internal link** | Toggle on to define an internal link to a tracing data source. Select the target data source from the selector. | | **Open in new tab** | Toggle on to open the link in a new tab or window. | | **Show example log message** | Paste an example log line to test the regular expression of your derived fields. | {{< admonition type="caution" >}} Using complex regular expressions in either type can affect browser performance when processing large volumes of logs. Use simpler patterns when possible. {{< /admonition >}} {{< admonition type="note" >}} A derived field produces a single value per log line, so you can't combine multiple labels or capture groups into one field or link. For an internal link, the derived field only builds a link to the target data source using the extracted value. The trace must already be ingested into the target data source, such as Tempo, for the link to resolve. If a matching trace doesn't exist, the link opens the target data source but returns no trace. {{< /admonition >}} ##### Troubleshoot interpolation Use the debug section to see what your fields extract and how the URL interpolates. Click **Show example log message** to display a text area where you can enter a log message. {{< figure src="/static/img/docs/v75/loki_derived_fields_settings.png" class="docs-image--no-shadow" max-width="800px" caption="Screenshot of the derived fields debug section" >}} The new field with the link appears in the log details: {{< figure src="/static/img/docs/explore/data-link-9-4.png" max-width="800px" caption="Data link in Explore" >}} ## Verify the connection Click **Save & test** at the bottom of the settings. Grafana attempts to connect to your Loki server and query its labels. When the test succeeds, Grafana displays a success message confirming the data source is working. If the test fails, review the error message and refer to [Troubleshoot Loki issues](https://grafana.com/docs/grafana//datasources/loki/troubleshooting/). {{< admonition type="note" >}} To troubleshoot configuration and other issues, check the log file located at `/var/log/grafana/grafana.log` on Unix systems, or in `/data/log` on other platforms and manual installations. {{< /admonition >}} ## Provision the data source You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ### Provisioning examples The following example provisions a basic Loki data source: ```yaml apiVersion: 1 datasources: - name: Loki type: loki access: proxy url: http://localhost:3100 jsonData: timeout: 60 maxLines: 1000 ``` **Using basic authorization and a derived field:** You must escape the dollar (`$`) character in YAML values because it can be used to interpolate environment variables: ```yaml apiVersion: 1 datasources: - name: Loki type: loki access: proxy url: http://localhost:3100 basicAuth: true basicAuthUser: my_user jsonData: maxLines: 1000 derivedFields: # Field with internal link pointing to data source in Grafana. # datasourceUid value can be anything, but it should be unique across all defined data source uids. - datasourceUid: my_jaeger_uid matcherRegex: "traceID=(\\w+)" name: TraceID # url will be interpreted as query for the datasource url: '$${__value.raw}' # optional for URL Label to set a custom display label for the link. urlDisplayLabel: 'View Trace' # Field with external link. - matcherRegex: "traceID=(\\w+)" name: TraceID url: 'http://localhost:16686/trace/$${__value.raw}' secureJsonData: basicAuthPassword: test_password ``` **Using a Jaeger data source:** In this example, the Jaeger data source's `uid` value should match the Loki data source's `datasourceUid` value. ```yaml datasources: - name: Jaeger type: jaeger url: http://jaeger-tracing-query:16686/ access: proxy # UID should match the datasourceUid in derivedFields. uid: my_jaeger_uid ``` ## Configure with Terraform You can configure the Loki data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). For more information about provisioning resources with Terraform, refer to [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/). The following example provisions a Loki data source with a maximum line limit: ```hcl resource "grafana_data_source" "loki" { name = "Loki" type = "loki" url = "http://localhost:3100" json_data_encoded = jsonencode({ maxLines = 1000 }) } ``` The following example uses basic authentication and defines a derived field that links to a Jaeger data source: ```hcl resource "grafana_data_source" "loki" { name = "Loki" type = "loki" url = "http://localhost:3100" basic_auth_enabled = true basic_auth_username = "my_user" json_data_encoded = jsonencode({ maxLines = 1000 derivedFields = [ { datasourceUid = "my_jaeger_uid" matcherRegex = "traceID=(\\w+)" name = "TraceID" url = "$${__value.raw}" urlDisplayLabel = "View Trace" } ] }) secure_json_data_encoded = jsonencode({ basicAuthPassword = "" }) } ``` Replace the placeholders with your own values: - ``: The password for the basic authentication user. For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). # Loki query editor The Loki data source's query editor helps you create [log](#create-a-log-query) and [metric](#create-a-metric-query) queries that use the Loki query language, [LogQL](https://grafana.com/docs/loki/latest/logql/). For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/). ## Choose a query editing mode The Loki query editor has two modes: - [Builder mode](#builder-mode), which provides a visual query designer. - [Code mode](#code-mode), which provides a feature-rich editor for writing queries. To switch between the editor modes, select the corresponding **Builder** and **Code** tabs. To run a query, select **Run queries** located at the top of the editor. {{< admonition type="note" >}} To run Loki queries in [Explore](https://grafana.com/docs/grafana//explore/), select **Run query**. {{< /admonition >}} Each mode is synchronized, so you can switch between them without losing your work, although there are some limitations. Builder mode doesn't support some complex queries. When you switch from Code mode to Builder mode with such a query, the editor displays a warning message that explains how you might lose parts of the query if you continue. You can then decide whether you still want to switch to Builder mode. You can also augment queries by using [template variables](https://grafana.com/docs/grafana//datasources/loki/template-variables/). To see example LogQL log and metric queries, explore this demo dashboard on Grafana Play. Open any panel's editor to view its query: {{< docs/play title="Loki NGINX Service Mesh - Pattern version" url="https://play.grafana.org/d/-lPcmrp7z/" >}} ## Toolbar elements The query editor toolbar contains the following elements: - **Kick start your query** - Click to see a list of sample queries that help you quickly start creating LogQL queries. These include log query starters and metric query starters. Click the arrow next to each to see the available query options, then continue to complete your query. - **Label browser** - Use the Loki label browser to navigate through your labels and values, and build queries. To navigate Loki and build a query: 1. Choose labels to locate. 1. Search for the values of your selected labels. The search field supports fuzzy search, and the label browser also supports faceting to list only possible label combinations. 1. Select the **Show logs** button to display log lines based on the selected labels, or select the **Show logs rate** button to show the rate based on metrics such as requests per second. Additionally, you can validate the selector by clicking the **Validate selector** button. Click **Clear** to start from the beginning. {{< figure src="/static/img/docs/explore/Loki_label_browser.png" class="docs-image--no-shadow" max-width="800px" caption="The Loki label browser" >}} - **Explain query** - Toggle to display a step-by-step explanation of all query components and operations. {{< figure src="/static/img/docs/prometheus/explain-results.png" max-width="500px" class="docs-image--no-shadow" caption="Explain results" >}} - **Builder/Code** - Click the corresponding **Builder** or **Code** tab on the toolbar to select an editor mode. ## Builder mode Builder mode helps you build queries using a visual interface without needing to manually enter LogQL. This option is best for users who have limited or no previous experience working with Loki and LogQL. ### Label filters Select labels and their values from the dropdown list. When you select a label, Grafana retrieves available values from the server. Use the `+` button to add a label and the `x` button to remove a label. You can add multiple labels. Select comparison operators from the following options: - `=` - equal to - `!=` - is not equal - `=~` - matches a regular expression - `!~` - doesn't match a regular expression Select values by using the dropdown, which displays all possible values based on the label selected. ### Operations Select the `+ Operations` button to add operations to your query. The query editor groups operations into related sections, and you can type while the operations dropdown is open to search and filter the list. The query editor displays a query's operations as boxes in the operations section. Each operation's header displays its name, and additional action buttons appear when you hover your cursor over the header: | Button | Action | | ----------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | {{< figure src="/static/img/docs/v95/loki_operation_replace.png" class="docs-image--no-shadow" max-width="30px" >}} | Replaces the operation with different operation of the same type. | | {{< figure src="/static/img/docs/v95/loki_operation_description.png" class="docs-image--no-shadow" max-width="30px" >}} | Opens the operation's description tooltip. | | {{< figure src="/static/img/docs/v95/loki_operation_remove.png" class="docs-image--no-shadow" max-width="30px" >}} | Removes the operation. | The query editor groups operations into the following sections: - Aggregations - see [Built-in aggregation operators](https://grafana.com/docs/loki/latest/logql/metric_queries/#built-in-aggregation-operators) - Range functions - see [Range Vector aggregation](https://grafana.com/docs/loki/latest/logql/metric_queries/#range-vector-aggregation) - Formats - see [Log queries](https://grafana.com/docs/loki/latest/logql/log_queries/#log-queries) - Binary operations - see [Binary operators](https://grafana.com/docs/loki/latest/logql/#binary-operators) - Label filters - see [Label filter expression](https://grafana.com/docs/loki/latest/logql/log_queries/#label-filter-expression) - Line filters - see [Line filter expression](https://grafana.com/docs/loki/latest/logql/log_queries/#label-filter-expression) Some operations make sense only when used in a specific order. If adding an operation would result in a nonsensical query, the query editor adds the operation to the correct place. To reorder operations manually, drag the operation box by its name and drop it into the desired place. For more information, refer to [Order of operations](https://grafana.com/docs/loki/latest/logql/#order-of-operations). ### Hints In some cases, the query editor can detect which operations are most appropriate for a selected log stream. When it does, it shows a hint next to the **+ Operations** button. Click the hint to add the operations to your query. ### Builder mode example The following steps build a metric query that counts the rate of error lines for the `payments` app, without writing any LogQL: 1. Under **Label filters**, select the `app` label and the value `payments`. 1. Select **+ Operations**, then from **Line filters** add a **Line contains** operation and enter `error`. 1. Select **+ Operations** again, then from **Range functions** add **Rate** with a range of `[5m]`. 1. From **Aggregations**, add **Sum** to combine the results into a single series. The builder produces the equivalent LogQL query: ```logql sum(rate({app="payments"} |= `error` [5m])) ``` Switch to [Code mode](#code-mode) at any time to view or refine the generated query. ## Code mode In **Code mode**, you can write complex queries using a text editor with autocompletion, syntax highlighting, and query validation. It also provides the [label browser](#toolbar-elements) to further help you write queries. For more information about the Loki query language, refer to the [Loki documentation](https://grafana.com/docs/loki/latest/logql/). ### Use autocompletion Code mode's autocompletion feature works automatically while typing. The query editor can autocomplete static functions, aggregations, and keywords, and also dynamic items like labels. The autocompletion dropdown includes documentation for the suggested items where available. ## Options The following options are the same for both **Builder** and **Code** mode: - **Legend** - Controls the time series name, using a name or pattern. For example, `{{hostname}}` is replaced with the label value for the label `hostname`. - **Type** - Selects the query type to run. The `instant` type queries against a single point in time, using the **To** time from the time range. The `range` type queries over the selected range of time. - **Line limit** - Defines the upper limit for the number of log lines returned by a query. The default is `1000`. - **Direction** - Determines the search order. **Backward** is a backward search starting at the end of the time range. **Forward** is a forward search starting at the beginning of the time range. The default is **Backward**. - **Step** - Sets the step parameter of Loki metric queries. The default value equals the value of the `$__auto` variable, which Grafana calculates using the time range and the width of the graph in pixels. ## Create a log query Loki log queries return the contents of the log lines. You can query and display log data from Loki via [Explore](https://grafana.com/docs/grafana//explore/), and with the [Logs panel](https://grafana.com/docs/grafana//panels-visualizations/visualizations/logs/) in dashboards. To display the results of a log query, select the Loki data source, then enter a LogQL query. For more information about log queries and LogQL, refer to the [Loki log queries documentation](https://grafana.com/docs/loki/latest/logql/log_queries/). ### Log query examples A log query has two parts: a stream selector that chooses log streams by label, and an optional pipeline that filters and parses the matching lines. Replace the label matchers with values from your own logs. Return every log line from the `payments` app: ```logql {app="payments"} ``` Keep only lines that contain the word `error`, and drop health-check noise: ```logql {app="payments"} |= `error` != `healthcheck` ``` Match lines with a regular expression, for example any 4xx or 5xx status: ```logql {app="nginx"} |~ `status=(4|5)..` ``` Parse structured logs, then filter on an extracted field. This example parses `logfmt` logs and keeps requests slower than 500 ms: ```logql {app="payments"} | logfmt | duration > 500ms ``` Parse JSON logs and keep only error-level entries: ```logql {namespace="prod", app="checkout"} | json | level=`error` ``` **Use case: debug a failing service.** Combine a namespace and app selector with a parser and label filter to isolate the errors for one service during an incident: ```logql {namespace="prod", app="checkout"} |= `error` | logfmt | status >= 500 ``` ### Show log context In Explore, you can retrieve the context surrounding your log results by clicking the **Show context** button. You can then investigate the logs from the same log stream that came before and after the log message you're interested in. The initial log context query is created from all labels defining the stream for the selected log line. You can use the log context query editor to widen the search by removing one or more of the label filters from the log stream. If you used a parser in your original query, you can refine your search by using extracted label filters. You can also adjust the size of the time window that the log context loads, so you can expand or narrow the range of surrounding logs without leaving the context view. To reduce the repetition of selecting and removing the same labels when examining multiple log context windows, Grafana stores your selected labels and applies them to each open context window. This lets you navigate through various log context windows without having to reapply your filters. To reset filters and use the initial log context query, click the **Revert to initial query** button next to the query preview. ### Tail live logs Loki supports live tailing of logs in real-time in [Explore](https://grafana.com/docs/grafana//explore/). Live tailing relies on two WebSocket connections: one between the browser and Grafana server, and another between the Grafana server and Loki server. To start tailing logs click the **Live** button in the top right corner of the Explore view. {{< figure src="/static/img/docs/v95/loki_tailing.png" class="docs-image--no-shadow" max-width="80px" >}} #### Proxying examples If you use reverse proxies, configure them accordingly to use live tailing: **Using Apache2 for proxying between the browser and the Grafana server:** ``` ProxyPassMatch "^/(api/datasources/proxy/\d+/loki/api/v1/tail)" "ws://127.0.0.1:3000/$1" ``` **Using NGINX:** This example provides a basic NGINX proxy configuration. It assumes that the Grafana server is available at `http://localhost:3000/`, the Loki server is running locally without proxy, and your external site uses HTTPS. If you also host Loki behind an NGINX proxy, repeat the following configuration for Loki. In the `http` section of NGINX configuration, add the following map definition: ``` map $http_upgrade $connection_upgrade { default upgrade; '' close; } ``` In your `server` section, add the following configuration: ``` location ~ /(api/datasources/proxy/\d+/loki/api/v1/tail) { proxy_pass http://localhost:3000$request_uri; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto "https"; proxy_set_header Connection $connection_upgrade; proxy_set_header Upgrade $http_upgrade; } location / { proxy_pass http://localhost:3000/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto "https"; } ``` ## Create a metric query You can use LogQL to wrap a log query with functions that create metrics from your logs. For more information about metric queries, refer to the [Loki metric queries documentation](https://grafana.com/docs/loki/latest/logql/metric_queries/). ### Metric query examples Metric queries apply a range aggregation, such as `rate` or `count_over_time`, over a log query and an optional range like `[5m]`. You can then wrap the result in an aggregation operator, such as `sum` or `topk`, to group or rank the series. Count the rate of error lines per second across the `payments` app: ```logql sum(rate({app="payments"} |= `error` [5m])) ``` Break the request rate down by status code: ```logql sum by (status) (rate({app="nginx"} | logfmt [5m])) ``` Count how many timeouts occurred in the last hour: ```logql count_over_time({app="payments"} |= `timeout` [1h]) ``` Calculate the 95th percentile request duration per route. This example parses `logfmt` logs and unwraps the numeric `duration` field: ```logql quantile_over_time(0.95, {app="payments"} | logfmt | unwrap duration [5m]) by (route) ``` **Use case: find the noisiest endpoints.** Rank the top 10 request paths by log volume over the last hour: ```logql topk(10, sum by (path) (count_over_time({app="nginx"} | logfmt [1h]))) ``` ## Query splitting To keep large queries responsive, Grafana automatically splits Loki queries that cover a long time range into a series of smaller sub-queries. Grafana runs the sub-queries and merges the results, and the query progresses incrementally so you see partial results as they arrive. Query splitting applies to both log and metric queries and requires no configuration. ## Structured metadata Loki stores three types of labels: indexed labels, parsed fields, and structured metadata. The query editor and log details treat all three types as labels you can filter on and use in [derived fields](https://grafana.com/docs/grafana//datasources/loki/configure/#derived-fields). For more information, refer to the [Loki structured metadata documentation](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/). ## Apply annotations You can use Loki log queries as a source for dashboard annotations. For details, refer to [Loki annotations](https://grafana.com/docs/grafana//datasources/loki/annotations/). # Loki template variables Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables. For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana//dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/) documentation. ## Use query variables Variables of the type _Query_ help you query Loki for lists of labels or label values. The Loki data source provides a form to select the type of values expected for a given variable. The form has these options: | Query type | Example label | Example stream selector | List returned | | ------------ | ------------- | ----------------------- | ---------------------------------------------------------------- | | Label names | Not required | Not required | Label names. | | Label values | `label` | | Label values for `label`. | | Label values | `label` | `log stream selector` | Label values for `label` in the specified `log stream selector`. | ### Query variable example To create a variable that lists the values of the `app` label: 1. Add a new variable of type _Query_ and select the Loki data source. 1. Set **Query type** to **Label values** and **Label** to `app`. 1. Optionally, enter a **Stream selector** such as `{namespace="$namespace"}` to return only the values that appear within another variable's selection. This creates a chained variable that updates when the parent variable changes. Reference the variable in a query with its name, prefixed by `$`. For a single-value variable, use an exact match: ```logql {app="$app"} ``` When a variable has **Multi-value** or **Include All** enabled, Grafana joins the selected values into a regular expression, such as `payments|checkout`. Use the regular expression match operator `=~` so the query matches any of the selected values: ```logql {app=~"$app"} ``` ## Format variable values By default, the Loki data source formats variable values based on the variable's settings: - A single-value variable is inserted as-is, without escaping. - A **Multi-value** or **Include All** variable has each value escaped for use in a regular expression and joined with `|`, for example `payments|checkout`. This default suits the regular expression match operator `=~`, but it can produce unexpected results when a custom or static value contains characters that Grafana escapes, or when you insert a value into a context that isn't a regular expression. To control the formatting, use a [format option](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/#advanced-variable-format-options) in the variable syntax: | Syntax | Result | | -------------- | ---------------------------------------------------------------------------------------------------------- | | `${app:raw}` | Inserts the value with no escaping. Use this for custom or static values that must pass through unchanged. | | `${app:pipe}` | Joins multiple values with `\|` without regular expression escaping. | | `${app:regex}` | Escapes the values for a regular expression and joins them with `\|`. | For example, to use a raw custom value in an exact-match selector: ```logql {app="${app:raw}"} ``` For the complete list of format options, refer to [Advanced variable format options](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/#advanced-variable-format-options). ## Use ad hoc filters Loki supports ad hoc filters. Use them to specify any number of key/value filters that Grafana applies automatically to all of your Loki queries, without editing each query. For example, if you add an ad hoc filter for `level = error`, Grafana appends the matcher to the stream selector of every Loki query on the dashboard, so a query like `{app="payments"}` runs as `{app="payments", level="error"}`. For more information, refer to [Add ad hoc filters](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#add-ad-hoc-filters). ## Use the `$__auto` variable for Loki metric queries Consider using the `$__auto` variable in your Loki metric queries, which will automatically be substituted with the [step value](https://grafana.com/docs/grafana//datasources/loki/query-editor/#options) for range queries, and with the selected time range's value (computed from the starting and ending times) for instant queries. Use it as the range in a range aggregation: ```logql sum(rate({app="payments"} |= `error` [$__auto])) ``` {{< admonition type="note" >}} Prefer `$__auto` over `$__range` for the range in a metric query. The `$__range` variable resolves to the entire selected time range, so using it as a range vector, such as `[$__range]`, makes each data point aggregate over the whole window. This can scan far more data and look back further than you intend. {{< /admonition >}} For more information about variables, refer to [Global built-in variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#global-variables). ## Known limitations Keep the following limitations in mind when you use template variables with the Loki data source: - **Only indexed labels are available for label variable queries.** A _Query_ variable of type **Label values** returns the values of indexed stream labels. Values that exist only as parsed fields or [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) aren't available, because the variable query doesn't run a log pipeline to extract them. - **Long time ranges can return incomplete values.** Label and label-value variable queries run over the dashboard's selected time range. Over a long range with high log volume, the query can time out or hit the limits configured in Loki and return an incomplete list of values. Narrow the time range, or add a stream selector to the variable query, so it scans less data. ## Label extraction and indexing in Loki Labels are key to how Loki aggregates and queries logs. Labels are `key-value` pairs that add contextual information to log entries, and Loki uses them to organize, filter, and search log data efficiently. ### Label extraction When it ingests logs, Loki extracts labels from log lines using regular expressions, so you can define custom patterns that match your log formats. For example, given a log line like the following: ```text 2023-07-25 12:34:56 INFO: Request from IP A.B.C.D to endpoint /api/data ``` You can define a regular expression that extracts the log level (`INFO`), IP address (`A.B.C.D`), and endpoint (`/api/data`) as labels. You can then use those labels to filter and aggregate log entries. ### Index labels Loki indexes the extracted labels. The index maps labels to their log entries, so Loki can retrieve logs by label without scanning the entire dataset. For example, if a `job` label represents the services in your application, Loki indexes each job's logs separately, so you can query a single job quickly. Combine Loki indexed labels with Grafana template variables to build dynamic queries. Use template variables to select and filter logs by labels such as job names, instance IDs, or severity levels, so you can explore and visualize your log data interactively. # Troubleshoot Loki data source issues This document provides solutions to common issues you might encounter when configuring or using the Loki data source. For configuration instructions, refer to [Configure the Loki data source](https://grafana.com/docs/grafana//datasources/loki/configure/). {{< admonition type="note" >}} This guide covers the Loki data source in Grafana, such as connecting to Loki, running queries, and configuring features. It doesn't cover troubleshooting the Loki service itself. For help with a Loki deployment, including ingestion and query performance, refer to [Manage and debug errors](https://grafana.com/docs/loki/latest/operations/troubleshooting/) in the Loki documentation. {{< /admonition >}} ## Connection errors These errors occur when Grafana can't reach your Loki server. ### Save & test fails **Symptoms:** - The data source test fails or times out. - Queries fail with network errors. **Possible causes and solutions:** | Cause | Solution | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Incorrect URL or port | Verify the **URL** setting. Loki listens on port `3100` by default, for example `http://localhost:3100`. | | API path in the URL | Enter only the base Loki URL. Don't append API paths such as `/loki/api/v1/push`, which sends logs to Loki rather than querying it. | | Localhost URL on Grafana Cloud | On Grafana Cloud, a `localhost` or private address points to Grafana servers, not your network. Use [PDC](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) to reach a self-hosted Loki. | | Missing tenant header | For a multi-tenant Loki, one with `auth_enabled: true`, add the `X-Scope-OrgID` custom HTTP header with your tenant ID. | | Network or firewall block | Confirm the Grafana server can reach the Loki server, and that firewall rules allow the connection. | | Authentication failure | Verify the credentials, tokens, or TLS certificates in the data source configuration. For Grafana Cloud Loki, confirm the access policy token includes the `logs:read` scope. | | TLS verification failure | If Loki uses a self-signed certificate, add the CA certificate with **With CA cert**, or enable **Skip TLS verify** for testing. | ## Query errors These errors occur when running queries against Loki. ### No data or empty results **Symptoms:** - A query runs without error but returns no data. - Panels show a "No data" message. **Possible causes and solutions:** | Cause | Solution | | --------------------------------- | ----------------------------------------------------------------------------------------- | | Time range doesn't contain logs | Expand the dashboard time range or confirm logs exist in Loki for that period. | | Label selector matches no streams | Verify your stream selector labels and values with the label browser in the query editor. | | Logs exceed retention | Confirm the logs you expect are within the retention period configured in Loki. | ### Query timeout or "too many outstanding requests" **Symptoms:** - A query runs for a long time and then fails. - Errors mention timeouts, query limits, or too many outstanding requests. **Solutions:** 1. Narrow the time range to reduce the volume of data scanned. 1. Add label filters or a line filter to reduce the result set. 1. Lower the **Maximum lines** setting on the data source or the line limit on the query. 1. Review the query and rate limits configured on your Loki deployment. ### "maximum of series reached for a single query" **Symptoms:** - A metric query fails with an error about the maximum number of series. **Solutions:** 1. Add label matchers to reduce the number of streams the query touches. 1. Aggregate with `sum by (...)` to reduce cardinality. 1. Avoid high-cardinality labels in aggregations. ## Performance issues These issues relate to slow queries or rate limits. ### Queries are slow or get rate limited **Solutions:** 1. Narrow the time range and add label filters so queries scan less data. 1. Enable query caching, available in Grafana Enterprise and Grafana Cloud. 1. Review the query, ingestion, and rate limits on your Loki deployment. ## Template variable errors These errors occur when using template variables with the data source. ### Variables return no values **Solutions:** 1. Confirm the data source connection works by running **Save & test**. 1. Verify the label or stream selector in the variable query returns values in the query editor. 1. For cascading variables, confirm parent variables have valid selections. ## Derived field and log-to-trace link issues These issues affect [derived fields](https://grafana.com/docs/grafana//datasources/loki/configure/#derived-fields), including the links from logs to traces. ### The derived field or trace link doesn't appear - Confirm the regular expression matches your log line format. A pattern that doesn't match produces no value and no link. Use **Show example log message** in the data source configuration to test the pattern against a real log line. - For a **Label** type derived field, confirm a label key matches the pattern. The match runs against label keys, not values. - Check that the field value is captured. For a **Regex in log line** type, Grafana uses the first capture group, so the expression must include one. ### The trace link opens but shows no trace - The derived field only builds a link to the target data source using the extracted value; it doesn't create the trace. Confirm the trace is ingested into the target data source, such as Tempo, and that the extracted ID matches the trace ID stored there. - Verify the extracted value is the correct ID. An overly permissive regular expression can capture the wrong part of the log line. Anchor the pattern and make it specific, for example match `traceID=(\w+)` rather than a broad `(\w+)`. ### Can't edit derived fields on a provisioned data source Data sources created through provisioning are read-only in the UI, so you can't change their derived fields from the data source configuration page. - For self-managed Grafana, edit the derived field configuration in the provisioning YAML file and restart Grafana, or reload provisioning. - For Grafana Cloud, open a support ticket from the Grafana Cloud Portal to request changes to a provisioned data source. ### A single link needs values from more than one field A derived field produces a single value per log line, so you can't combine multiple labels or capture groups into one field or link. Create a separate derived field for each value you need to surface. ## Live tailing issues These issues affect real-time log tailing in Explore. ### Live tailing doesn't stream logs Live tailing relies on WebSocket connections between the browser and Grafana, and between Grafana and Loki. Reverse proxies often block these connections. **Solutions:** 1. Confirm your reverse proxy forwards WebSocket connections for the tail endpoint. Refer to the proxy examples in the [Loki query editor](https://grafana.com/docs/grafana//datasources/loki/query-editor/#tail-live-logs). 1. Verify that intermediate load balancers and firewall rules allow WebSocket upgrades. 1. Confirm Loki itself is reachable and that the tail endpoint isn't blocked. ## Label-based access control (LBAC) issues Label-based access control (LBAC) filters the logs a team can query based on labels. It's available on Grafana Cloud and Grafana Enterprise, where it requires Grafana Enterprise Logs (GEL). If you can't enable LBAC, or teams see more logs than expected, check the following. ### Can't enable LBAC on a provisioned data source {{< admonition type="note" >}} You can't enable LBAC on a provisioned or auto-provisioned Loki data source. Provisioning of LBAC data sources isn't supported. {{< /admonition >}} To use LBAC, create a new Loki data source manually and configure it with basic authentication, using your Cloud Access Policy (CAP) token as the password. Point it at the same Loki endpoint as the provisioned data source, then define LBAC rules on the new data source's **Permissions** tab. For the full procedure, refer to [Configure LBAC for data sources for Loki](https://grafana.com/docs/grafana//administration/data-source-management/teamlbac/configure-teamlbac-for-loki/). ### Teams can still query all logs - LBAC applies only when a team has at least one rule. If a team has no LBAC rules, its members can query all logs. Confirm each team has the rules you intend. - Grafana permissions are additive. A user's basic role, such as Viewer, can grant access to the data source independently of team-scoped rules. Remove role and team permissions that aren't required so that only the intended teams have **Query** permission on the LBAC data source. - Remove any label selectors from the Cloud Access Policy used by the data source. CAP label selectors override LBAC rules. For details and limitations, refer to [LBAC for data sources](https://grafana.com/docs/grafana//administration/data-source-management/teamlbac/). ## Enable debug logging To capture detailed error information for troubleshooting: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Review logs in `/var/log/grafana/grafana.log`, or in your configured log location. 1. Look for Loki-specific entries that include request and response details. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you've tried these solutions and still encounter issues: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review the [Loki data source GitHub issues](https://github.com/grafana/grafana-loki-datasource/issues) for known bugs. 1. Consult the [Loki documentation](https://grafana.com/docs/loki/latest/) for service-specific guidance. 1. When reporting issues, include: - Grafana and Loki versions - Error messages, with sensitive information redacted - Steps to reproduce - Relevant configuration, with credentials redacted # Microsoft SQL Server alerting You can use Grafana Alerting with Microsoft SQL Server to create alerts based on your SQL Server data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met. For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). ## Before you begin Before creating alerts with Microsoft SQL Server, ensure you have: - A [configured Microsoft SQL Server data source](https://grafana.com/docs/grafana//datasources/mssql/configure/) - Appropriate permissions to create alert rules - Understanding of the metrics you want to monitor ## Supported query types Microsoft SQL Server alerting works with **time series queries** that return numeric data over time. Table-formatted queries are not supported in alert rule conditions. To create a valid alert query: - Include a `time` column that returns an SQL `datetime`/`datetime2` or a Unix epoch timestamp - Return numeric values for the metrics you want to alert on - Sort results by the time column For more information on writing time series queries, refer to [Microsoft SQL Server query editor](https://grafana.com/docs/grafana//datasources/mssql/query-editor/). ### Query format requirements | Query format | Alerting support | Notes | | ------------ | ---------------- | ---------------------------------------- | | Time series | Yes | Required for alerting | | Table | No | Convert to time series format for alerts | ## Create an alert rule To create an alert rule using Microsoft SQL Server: 1. Navigate to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for the alert rule. 1. Select your **Microsoft SQL Server** data source. 1. Build your query using the query editor: - Set the **Format** to **Time series**. - Include a time column using the `$__timeGroup()` or `$__timeGroupAlias()` macro. - Add numeric columns for the values to monitor. - Use `$__timeFilter()` to filter data by the evaluation time range. 1. Configure the alert condition (for example, when the average is above a threshold). 1. Set the evaluation interval and pending period. 1. Configure notifications and labels. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example alert queries The following examples show common alerting scenarios with Microsoft SQL Server. ### Alert on high error count Monitor the number of errors over time: ```sql SELECT $__timeGroupAlias(created_at, '1m'), COUNT(*) AS error_count FROM error_logs WHERE $__timeFilter(created_at) AND level = 'error' GROUP BY $__timeGroup(created_at, '1m') ORDER BY 1 ``` **Condition:** When `error_count` is above 100. ### Alert on average response time Monitor API response times: ```sql SELECT $__timeGroupAlias(request_time, '5m'), AVG(response_time_ms) AS avg_response_time FROM api_requests WHERE $__timeFilter(request_time) GROUP BY $__timeGroup(request_time, '5m') ORDER BY 1 ``` **Condition:** When `avg_response_time` is above 500 (milliseconds). ### Alert on low order volume Detect drops in order activity: ```sql SELECT $__timeGroupAlias(order_date, '1h'), COUNT(*) AS order_count FROM orders WHERE $__timeFilter(order_date) GROUP BY $__timeGroup(order_date, '1h') ORDER BY 1 ``` **Condition:** When `order_count` is below 10. ### Alert on high CPU utilization Monitor server resource metrics: ```sql SELECT $__timeGroupAlias(recorded_at, '5m'), AVG(cpu_percent) AS avg_cpu FROM sys_metrics WHERE $__timeFilter(recorded_at) GROUP BY $__timeGroup(recorded_at, '5m') ORDER BY 1 ``` **Condition:** When `avg_cpu` is above 85. ## Limitations When using Microsoft SQL Server with Grafana Alerting, be aware of the following limitations. ### Template variables not supported Alert queries cannot contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$hostname` or `$environment` aren't resolved. If your dashboard query uses template variables, create a separate query for alerting with hard-coded values. ### Table format not supported Queries using the **Table** format cannot be used for alerting. Set the query format to **Time series** and ensure your query returns a time column. ### Query timeout Complex queries with large datasets may time out during alert evaluation. Optimize queries for alerting by: - Adding appropriate `WHERE` clauses to limit data - Using indexes on time and filter columns - Reducing the time range evaluated ### Current User authentication not supported If your data source uses Azure Entra ID **Current User** authentication, alerting, reporting, and recorded queries are not supported. These features require backend-level credentials that don't rely on a specific user's session. ## Best practices Follow these best practices when creating Microsoft SQL Server alerts: - **Use time series format:** Always set the query format to time series for alert queries. - **Include time filters:** Use the `$__timeFilter()` macro to limit data to the evaluation window. - **Use MSSQL macros:** Prefer `$__timeGroupAlias()` and `$__timeGroup()` over manual time-bucketing expressions. - **Optimize queries:** Add indexes on columns used in `WHERE` clauses and `GROUP BY`. - **Test queries first:** Verify your query returns expected results in Explore before creating an alert. - **Set realistic thresholds:** Base alert thresholds on historical data patterns. - **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor. # Microsoft SQL Server annotations Annotations overlay event markers on your dashboard visualizations, helping you correlate metric behavior with events like deployments, incidents, or configuration changes. You can use the Microsoft SQL Server data source to create annotation queries that display SQL query results as annotation events. For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/). ## Before you begin Before you create MSSQL annotations, ensure you have: - A [configured Microsoft SQL Server data source](https://grafana.com/docs/grafana//datasources/mssql/configure/) - A table containing event data with at least a time column and a description column - Familiarity with [T-SQL](https://learn.microsoft.com/en-us/sql/t-sql/language-reference) query syntax ## Create an annotation query To add a Microsoft SQL Server annotation to your dashboard: 1. Navigate to your dashboard and click **Dashboard settings** (gear icon). 1. Select **Annotations** in the left menu. 1. Click **Add annotation query**. 1. Enter a **Name** for the annotation. 1. Select your **Microsoft SQL Server** data source from the **Data source** drop-down. 1. Write an SQL query that returns the required columns. 1. Click **Save dashboard**. ## Annotation query columns Your annotation query must return columns with specific names. Grafana uses these names to map query results to annotation fields. | Column | Required | Description | | --------- | -------- | ------------------------------------------------------------------------------------------------------ | | `time` | Yes | The date/time of the event. Can be a native SQL date/time type or a Unix epoch value in seconds. | | `timeend` | No | The end time for region annotations. Same format as `time`. Creates a shaded region instead of a line. | | `text` | Yes | The event description displayed in the annotation tooltip. | | `tags` | No | Comma-separated string used for event tags. Tags help categorize and filter annotations. | ## Example: Annotation with epoch time Given a table that stores events with Unix epoch timestamps: ```sql CREATE TABLE [events] ( time_sec bigint, description nvarchar(100), tags nvarchar(100), ) ``` Query to display events as annotations: ```sql SELECT time_sec as time, description as [text], tags FROM [events] WHERE $__unixEpochFilter(time_sec) ORDER BY 1 ``` The `$__unixEpochFilter` macro automatically filters events to the dashboard's selected time range. ## Example: Region annotation with start and end times To display annotations as shaded regions (spanning a duration), include both `time` and `timeend` columns: ```sql SELECT time_sec as time, time_end_sec as timeend, description as [text], tags FROM [events] WHERE $__unixEpochFilter(time_sec) ORDER BY 1 ``` ## Example: Annotation with native `datetime` column If your table uses native SQL `datetime` or `datetime2` columns instead of epoch values: ```sql SELECT time, measurement as text, convert(varchar, valueOne) + ',' + convert(varchar, valueTwo) as tags FROM metric_values WHERE $__timeFilter(time) ORDER BY 1 ``` The `$__timeFilter` macro works with native SQL date/time types and filters to the dashboard time range. ## Example: Deployment annotations Display deployment events on your graphs: ```sql SELECT deployed_at as time, 'Deployed ' + version + ' to ' + environment as [text], environment as tags FROM [deployments] WHERE $__timeFilter(deployed_at) ORDER BY 1 ``` ## Example: Maintenance window annotations Display maintenance windows as shaded regions: ```sql SELECT start_time as time, end_time as timeend, 'Maintenance: ' + description as [text], 'maintenance' as tags FROM [maintenance_windows] WHERE $__timeFilter(start_time) ORDER BY 1 ``` ## Use template variables You can use [template variables](https://grafana.com/docs/grafana//datasources/mssql/template-variables/) in your annotation queries to make them dynamic. For example, filter events by a selected server: ```sql SELECT time_sec as time, description as [text], tags FROM [events] WHERE $__unixEpochFilter(time_sec) AND server IN ($server) ORDER BY 1 ``` ## Macros Use these macros in your annotation queries to filter by the dashboard time range: | Macro | Description | | ---------------------------- | ----------------------------------------------------------------------- | | `$__timeFilter(column)` | Filters by time range using a native SQL `datetime`/`datetime2` column. | | `$__unixEpochFilter(column)` | Filters by time range using a column with Unix epoch timestamps. | For the full list of available macros, refer to the [Microsoft SQL Server query editor](https://grafana.com/docs/grafana//datasources/mssql/query-editor/#macros). ## Best practices Follow these best practices when creating Microsoft SQL Server annotations: - **Use time filters:** Always include `$__timeFilter()` or `$__unixEpochFilter()` to limit results to the dashboard time range. - **Keep queries efficient:** Add indexes on time columns and filter columns to improve query performance. - **Use meaningful text:** Include descriptive information in the `text` column to make annotations useful at a glance. - **Organize with tags:** Use consistent tag values to categorize annotations and enable filtering. - **Test queries first:** Verify your query returns expected results in Explore before adding it as an annotation. ## Get help If your annotations aren't appearing or behaving as expected, refer to [Troubleshoot Microsoft SQL Server data source issues](https://grafana.com/docs/grafana//datasources/mssql/troubleshooting/) for common solutions. # Configure the Microsoft SQL Server data source This document provides instructions for configuring the Microsoft SQL Server data source and explains available configuration options. For general information on adding and managing data sources, refer to [Grafana data sources](https://grafana.com/docs/grafana//datasources/) and [Data source management](https://grafana.com/docs/grafana//administration/data-source-management/). {{< admonition type="note" >}} This page documents the **Microsoft SQL Server data source**, which lets you query an existing SQL Server database and visualize the results in Grafana. This is different from the [Microsoft SQL Server integration](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/integration-reference/integration-mssql/) (available in Grafana Cloud), which uses Grafana Alloy to collect performance metrics _about_ your SQL Server instance (CPU, memory, connections, and similar). If you want to monitor the health of SQL Server itself, refer to the integration documentation instead. {{< /admonition >}} ## Before you begin Before configuring the Microsoft SQL Server data source, ensure you have the following: - **Grafana permissions:** You must have the `Organization administrator` role to add and configure data sources. Users with the `Editor` or `Viewer` role cannot access the data source configuration form unless granted additional permissions through [RBAC](https://grafana.com/docs/grafana//administration/roles-and-permissions/access-control/). Organization administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system. - **A running SQL Server instance:** Microsoft SQL Server 2012 or newer, Azure SQL Database, or Azure SQL Managed Instance. - **Network access:** Grafana must be able to reach your SQL Server. The default port is `1433`. - **Authentication credentials:** Depending on your authentication method, you need one of: - SQL Server login credentials (username and password). - Windows/Kerberos credentials and configuration (not supported in Grafana Cloud). - Azure Entra ID app registration or managed identity. - **Security certificates:** If using encrypted connections, gather any necessary TLS/SSL certificates. {{< admonition type="note" >}} The Microsoft SQL Server data source plugin is preinstalled in Grafana, so no additional installation is required. As of Grafana 13.2, it's packaged as a standalone plugin that updates independently of Grafana releases. Refer to [Plugin updates](https://grafana.com/docs/grafana//datasources/mssql/#plugin-updates) for details. {{< /admonition >}} {{< admonition type="tip" >}} **Grafana Cloud users:** If your SQL Server is in a private network, you can configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) to establish connectivity. {{< /admonition >}} ## Add the MSSQL data source To add the MSSQL data source, complete the following steps: 1. Click **Connections** in the left-side menu. 1. Click **Add new connection** 1. Type `Microsoft SQL Server` in the search bar. 1. Select **Microsoft SQL Server** under data source. 1. Click **Add new data source** in the upper right. Grafana takes you to the **Settings** tab, where you set up your Microsoft SQL Server configuration. ## Configure the data source in the UI Following are configuration options for the Microsoft SQL Server data source. {{< admonition type="warning" >}} Kerberos is not supported in Grafana Cloud. {{< /admonition >}} | **Setting** | **Description** | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | The data source name. Sets the name you use to refer to the data source in panels and queries. Examples: `MSSQL-1`, `MSSQL_Sales1`. | | **Default** | Toggle to select as the default name in dashboard panels. When you go to a dashboard panel, this is the default selected data source. | **Connection:** | Setting | Description | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Host** | Sets the IP address or hostname (and optional port) of your MSSQL instance. The default port is `0`, which uses the driver's default. You can include additional connection properties (for example, `ApplicationIntent`) by separating them with semicolons (`;`). | | **Database** | Sets the name of the MSSQL database to connect to. | {{< admonition type="caution" >}} If you add `ApplicationIntent=ReadOnly` to the Host field to route queries to a read-only replica, be aware that read-only replicas may be significantly slower than the primary. This can cause dashboard load times of several minutes. Only use `ApplicationIntent=ReadOnly` if your replica is sized for the query workload and you've tested performance. {{< /admonition >}} **TLS/SSL encryption:** The **Encrypt** setting determines whether and to what extent a secure TLS/SSL connection is negotiated with the server. | Encrypt setting | Description | When to use | | --------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------- | | **Disable** | Data sent between the client and server is **not encrypted**. | Legacy environments where encryption isn't supported or required. Not recommended for production. | | **False** | The default setting. Only the login packet is encrypted; **all other data is sent unencrypted**. | Development environments or when SQL Server doesn't have a valid certificate. | | **True** | **All data** sent between the client and server is **encrypted**. | Production environments. Required for Azure SQL Database. | When **Encrypt** is set to **True**, the following additional TLS options are available: | Setting | Description | Default | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- | | **Skip TLS Verify** | If enabled, the server's certificate chain and hostname aren't validated. Equivalent to `trustServerCertificate=true` in the connection string. | Disabled | | **TLS/SSL Root Certificate** | Path to a PEM-encoded CA certificate file used to verify the server's certificate. Required when the SQL Server uses a certificate signed by a private CA. | System CA bundle | | **Hostname in server certificate** | The expected hostname in the server's TLS certificate. Use this when the certificate's Common Name (CN) or Subject Alternative Name (SAN) doesn't match the configured hostname (for example, when connecting through a load balancer or PDC). | Value from **Host** field | **Choosing the right encryption configuration:** | Scenario | Encrypt | Skip TLS Verify | Root Certificate | Notes | | ----------------------------------------------------- | ---------------- | --------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------- | | Azure SQL Database | True | No | Not needed | Azure SQL always requires encryption and uses a publicly trusted certificate. | | SQL Server with public CA certificate | True | No | Not needed | Certificate is already in the system trust store. | | SQL Server with private/self-signed certificate | True | No | Path to CA cert | Upload or reference the CA certificate that signed the SQL Server certificate. | | SQL Server with unverified certificate (testing only) | True | Yes | Not needed | Skips certificate validation entirely. Don't use in production. | | SQL Server 2008/2008R2 without valid certificate | Disable or False | N/A | N/A | Older versions may not support TLS negotiation. | | Grafana Cloud with PDC | True | No | Not needed (usually) | PDC tunnels are encrypted at the SSH layer; the TLS certificate must still be valid from the PDC agent's perspective. | {{< admonition type="note" >}} If you're using an older version of Microsoft SQL Server like 2008 or 2008R2, you may need to set Encrypt to **Disable** or **False** to connect. {{< /admonition >}} {{< admonition type="caution" >}} Grafana builds using BoringCrypto (FIPS-compliant builds) enforce FIPS 140-3 cipher requirements. If your SQL Server only supports older cipher suites (for example, TLS 1.0 or RC4-based ciphers), connections from a FIPS-enabled Grafana instance fail with a TLS handshake error. Upgrade your SQL Server TLS configuration to support TLS 1.2 with FIPS-approved cipher suites. {{< /admonition >}} **Authentication:** | Authentication Type | Description | Credentials / Fields | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **SQL Server Authentication** | Default method to connect to MSSQL. Use a SQL Server or Windows login in `DOMAIN\User` format. | **Username**: SQL Server username. **Password**: SQL Server password. | | **Windows Authentication** (Integrated Security) | Uses the logged-in Windows user's credentials via single sign-on. Available only when SQL Server allows Windows Authentication. | No input required; uses the logged-in Windows user's credentials. | | **Windows AD** (Username/Password) | Authenticates a domain user with their Active Directory username and password. | **Username**: `user@example.com`. **Password**: Active Directory password. | | **Windows AD** (Keytab) | Authenticates a domain user using a keytab file. | **Username**: `user@example.com`. **Keytab file path**: Path to your keytab file. | | **Windows AD** (Credential Cache) | Uses a Kerberos credential cache already loaded in memory (for example, from a prior `kinit` command). No file needed. | **Credential cache path**: Path to in-memory credential (for example, `/tmp/krb5cc_1000`). | | **Windows AD** (Credential Cache File) | Authenticates a domain user using a credential cache file (`.ccache`). | **Username**: `user@example.com`. **Credential cache file path**: for example, `/home/grot/cache.json`. | | **Azure Entra ID (formerly Azure AD) Authentication** | Authenticates the data source using Azure authentication methods. The UI label is **Azure AD Authentication**. | For supported methods and configuration steps, refer to [Azure Entra ID authentication](https://grafana.com/docs/grafana//datasources/mssql/configure/#azure-entra-id-formerly-azure-ad-authentication). | {{< admonition type="note" >}} As of Grafana v13.0, passwords and usernames that contain semicolons (`;`) or closing braces (`}`) are handled correctly. If you previously encountered connection failures due to special characters in credentials, upgrade to Grafana v13.0 or later. {{< /admonition >}} **Additional settings:** Additional settings are optional settings you configure for more control over your data source. This includes connection limits, connection timeout, group-by time interval, and Secure Socks Proxy. **Connection limits:** Grafana maintains a connection pool for each configured MSSQL data source. These settings control pool behavior and directly affect performance, especially under concurrent load from dashboards and alerting. | Setting | Description | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Max open** | The maximum number of open connections to the database. If set to `0`, there is no limit. If `max open` is greater than `0` and less than `max idle`, `max idle` is adjusted to match. | | **Auto max idle** | When enabled, automatically sets `max idle` to match `max open`. If `max open` isn't set, it defaults to `100`. | | **Max idle** | The maximum number of idle connections in the pool. If `max open` is set and is lower than `max idle`, then `max idle` is reduced to match. If set to `0`, no idle connections are retained. | | **Max lifetime** | The maximum time (in seconds) a connection can be reused before being closed and replaced. If set to `0`, connections are reused indefinitely. | {{< admonition type="caution" >}} If **Max open** is set too low, concurrent dashboard panels and alert rule evaluations compete for connections. Alerting queries may fail with "failed to connect to server" while dashboards continue to work, because alert evaluations use a separate execution path that can't wait indefinitely for a free connection. Increase **Max open** if you see intermittent alert failures. {{< /admonition >}} **Connection details:** | **Setting** | **Description** | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Min time interval** | Specifies the lower bound for the auto-generated `GROUP BY` time interval. Grafana recommends matching this value to the data write frequency. For example, use `1m` if data is written every minute. Refer to [Min time interval](#min-time-interval) for details. | | **Connection timeout** | Specifies the maximum number of seconds to wait when attempting to connect to the database before timing out. A value of `0` (the default) disables the timeout. | **Windows AD Advanced settings:** | Setting | Description | Default | | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | **UDP Preference Limit** | Defines the maximum packet size (in bytes) that Kerberos libraries attempt to send over UDP before retrying with TCP. A value of `1` forces all communication to use TCP. | `1` (always use TCP) | | **DNS Lookup KDC** | Controls whether DNS `SRV` records are used to locate [Key Distribution Centers (KDCs)](https://web.mit.edu/kerberos/krb5-latest/doc/admin/realm_config.html#key-distribution-centers) and other servers for the realm. | `true` | | **krb5 config file path** | Specifies the path to the Kerberos configuration file used by the [MIT krb5 package](https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html). | `/etc/krb5.conf` | {{< admonition type="note" >}} Windows AD (Kerberos) authentication is only available for self-managed Grafana installations. It is not supported in Grafana Cloud. For Grafana Cloud deployments connecting to SQL Server, use SQL Server Authentication or Azure Entra ID. {{< /admonition >}} **Kerberos SPN requirements:** For Kerberos authentication to succeed, a Service Principal Name (SPN) must be registered for the SQL Server instance that matches the hostname Grafana uses to connect. This is especially important when connecting through Availability Group Listeners or DNS aliases. Register SPNs using the `setspn` command on the SQL Server host: ```text setspn -S MSSQLSvc/:1433 \ setspn -S MSSQLSvc/ \ ``` If Grafana connects through a Listener or alias, register an additional SPN for that name. Verify registered SPNs with `setspn -L \`. ### Connect to on-premises SQL Server from Grafana Cloud If your SQL Server instance is in a private network (on-premises, VPN, or VPC), Grafana Cloud cannot reach it directly over the internet. You must use **Private data source connect (PDC)** to establish a secure tunnel between your Grafana Cloud stack and your private network. {{< admonition type="note" >}} PDC is required for any SQL Server that isn't publicly accessible from the internet. Firewall allowlisting alone is not sufficient because Grafana Cloud doesn't use static IP addresses for data source connections. {{< /admonition >}} **Network prerequisites:** - **Port 22 (SSH):** The PDC agent on your network must be able to reach the Grafana Cloud PDC endpoint over port 22 for the SSH tunnel. - **Port 1433 (SQL Server):** The PDC agent must be able to reach your SQL Server instance on port 1433 (or your custom SQL Server port). - **DNS resolution:** The PDC agent must be able to resolve the SQL Server hostname from within your private network. **Set up PDC for Microsoft SQL Server:** 1. In your Grafana Cloud stack, navigate to **Connections** > **Private data source connect**. 1. Create a new PDC connection or use an existing one. 1. Install and configure the PDC agent on a host within your private network that has access to the SQL Server instance. For detailed instructions, refer to [Configure Grafana private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc). 1. Return to the Microsoft SQL Server data source configuration. 1. Under **Private data source connect**, select your PDC connection from the drop-down. 1. Enter the SQL Server hostname as it's resolvable from within your private network (not a public DNS name). 1. Click **Save & test** to verify connectivity. Click **Manage private data source connect** to open your PDC connection page and view your configuration details. For more information about PDC, refer to [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). {{< admonition type="caution" >}} If you previously connected using a self-managed Grafana instance and are migrating to Grafana Cloud, you must configure PDC before your private SQL Server is accessible. Direct network access methods that worked with self-managed Grafana (VPN peering, private endpoints) don't apply to Grafana Cloud. {{< /admonition >}} After configuring your MSSQL data source options, click **Save & test** at the bottom to test the connection. You should see a confirmation dialog box that says: **Database Connection OK** ### Azure Entra ID (formerly Azure AD) Authentication The following Azure authentication methods are supported: - App Registration (service principal) - Managed Identity - Current User authentication - Azure Entra Password The Azure SQL Server that you are connecting to must support Azure Entra authentication. For configuration details, refer to the [Azure SQL documentation](https://learn.microsoft.com/en-us/azure/azure-sql/database/authentication-aad-configure?view=azuresql&tabs=azure-portal). #### Required server configuration Before you can use any Azure Entra ID authentication method, you must enable Azure authentication in the Grafana server configuration. Add the following to your `grafana.ini` or `custom.ini` file: ```ini [azure] azure_auth_enabled = true ``` Depending on the method you choose, additional flags are required (documented in each section below). After changing the `.ini` file, restart the Grafana server for the changes to take effect. #### Current User authentication This is the recommended authentication mechanism when working with SQL Server instances that are hosted in Azure. It allows users to be authenticated to and query the database using their own credentials rather than long-lived credentials. This authentication method requires your Grafana instance to be configured with Azure Entra ID (formerly Active Directory) authentication for login. With Azure Entra ID login, this method can be used to forward the currently logged in user's credentials to the data source. The user's credentials are then used when requesting data from the data source. For details on how to configure your Grafana instance using Azure Entra, refer to the [documentation](https://grafana.com/docs/grafana//setup-grafana/configure-access/configure-authentication/azuread/). {{< admonition type="note" >}} Additional configuration is required to ensure that the App Registration used to login a user via Azure provides an access token with the permissions required by the data source. The App Registration must be configured to issue both **Access Tokens** and **ID Tokens**. 1. In the Azure Portal, open the App Registration that requires configuration. 2. Select **Authentication** in the side menu. 3. Under **Implicit grant and hybrid flows** check both the **Access tokens** and **ID tokens** boxes. 4. Save the changes to ensure the App Registration is updated. The App Registration must also be configured with additional **API Permissions** to provide authenticated users with access to the APIs utilised by the data source. 1. In the Azure Portal, open the App Registration that requires configuration. 1. Select **API Permissions** in the side menu. 1. Ensure the `openid`, `profile`, `email`, and `offline_access` permissions are present under the **Microsoft Graph** section. If not, they must be added. 1. Select **Add a permission** and choose the following permissions. They must be added individually. Refer to the [Azure documentation](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-configure-app-access-web-apis) for more information. - Select **APIs my organization uses** > Search for **Azure SQL** and select it > **Delegated permissions** > `user_impersonation` > **Add permissions** After all permissions have been added, the Azure authentication section in Grafana must be updated. The `scopes` section must be updated to include the `.default` scope to ensure that a token with access to all APIs declared on the App Registration is requested by Grafana. After updated the scopes value should equal: `.default openid email profile`. {{< /admonition >}} This method of authentication doesn't inherently support all backend functionality as a user's credentials won't be in scope. Affected functionality includes alerting, reporting, and recorded queries. Also, note that query and resource caching is disabled by default for data sources using current user authentication. **To enable current user authentication for Grafana:** 1. Set the `user_identity_enabled` flag in the `[azure]` section of the [Grafana server configuration](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#azure). ```ini [azure] user_identity_enabled = true ``` 2. In the SQL Server data source configuration, set **Authentication** to **Azure AD Authentication** and the Azure Authentication type to **Current User**. #### App Registration Use an app registration (service principal) to authenticate the data source with a client ID and secret. This is the recommended method for automated, non-interactive access to Azure SQL. **Prerequisites:** - An app registration created in Azure Entra. For instructions, refer to the [Azure documentation for service principals](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#get-tenant-and-app-id-values-for-signing-in). - The tenant ID, client (application) ID, and a client secret from the app registration. **Step 1: Add the app registration as a database user** 1. Connect to your Azure SQL database as a user with administrative permissions (the user must have the ability to read your Azure Entra directory, for example, by possessing the `Directory Readers` role). 1. Run the following SQL to create a user for the app registration: ```sql CREATE USER [] FROM EXTERNAL PROVIDER; ``` 1. Grant the created user appropriate permissions. Grafana recommends read-only access: ```sql ALTER ROLE db_datareader ADD MEMBER []; ``` **Step 2: Configure the Grafana server** Ensure `azure_auth_enabled` is set in your `grafana.ini`: ```ini [azure] azure_auth_enabled = true ``` **Step 3: Configure the data source** 1. In the SQL Server data source configuration, set **Authentication** to **Azure AD Authentication** and the Azure Authentication type to **App Registration**. 1. Set **Azure Cloud** to **Azure** (for the Azure public cloud). 1. Enter the **Directory (tenant) ID**, **Application (client) ID**, and **Client Secret** from your app registration. #### Managed Identity {{< admonition type="note" >}} Managed Identity is available only in [Azure Managed Grafana](https://azure.microsoft.com/en-us/products/managed-grafana) or self-managed Grafana deployed in Azure. It is not available in Grafana Cloud. {{< /admonition >}} You can use managed identity to configure SQL Server in Grafana if you host Grafana in Azure (such as an App Service or with Azure Virtual Machines) and have managed identity enabled on your VM. This lets you securely authenticate data sources without manually configuring credentials. For details on Azure managed identities, refer to the [Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview). **To enable managed identity for Grafana:** 1. Set both `azure_auth_enabled` and `managed_identity_enabled` in the `[azure]` section of the [Grafana server configuration](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#azure): ```ini [azure] azure_auth_enabled = true managed_identity_enabled = true ``` 1. In the SQL Server data source configuration, set **Authentication** to **Azure AD Authentication** and the Azure Authentication type to **Managed Identity**. This hides the directory ID, application ID, and client secret fields, and the data source uses managed identity to authenticate to SQL Server. 1. Optionally, set `managed_identity_client_id` in the `[azure]` section to use a user-assigned managed identity instead of the default system-assigned identity. Ensure that the managed identity is added to your Azure SQL instance as a user. ### Azure Entra Password {{< admonition type="warning" >}} Azure Entra Password is not a recommended authentication mechanism as it requires configuration using a single users password. Consider an alternative authentication method such as current user authentication or app registration. {{< /admonition >}} You can connect to an Azure SQL database using the username and password of a user that has permissions in the desired database. This also requires an app registration to be configured with access to the database. **To enable Azure Entra password for Grafana:** 1. Set the `azure_entra_password_credentials_enabled` flag in the `[azure]` section of the [Grafana server configuration](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#azure). ```ini [azure] azure_entra_password_credentials_enabled = true ``` 2. In the SQL Server data source configuration, set **Authentication** to **Azure AD Authentication** and the Azure Authentication type to **Azure Entra Password**. 3. Set the **User ID** value to the username of the user in the Azure SQL database. 4. Set the **Application Client ID** to the client ID of the app registration that has been added to the Azure SQL database 5. Set the **Password** value to the password of the user in the Azure SQL database. ### Min time interval The **Min time interval** setting defines a lower limit for the [`$__interval`](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#__interval) and [`$__interval_ms`](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#__interval_ms) variables. This value _must_ be formatted as a number followed by a valid time identifier: | Identifier | Description | | ---------- | ----------- | | `y` | year | | `M` | month | | `w` | week | | `d` | day | | `h` | hour | | `m` | minute | | `s` | second | | `ms` | millisecond | Grafana recommends setting this value to match your Microsoft SQL Server write frequency. For example, use `1m` if Microsoft SQL Server writes data every minute. You can also override this setting in a dashboard panel under its data source options. ### Database user permissions When adding a data source, ensure the database user you specify has only SELECT permissions on the relevant database and tables. Grafana does not validate the safety of queries, which means they can include potentially harmful SQL statements, such as `USE otherdb`; or `DROP TABLE user;`, which could get executed. To minimize this risk, Grafana strongly recommends creating a dedicated MSSQL user with restricted permissions. ```sql CREATE USER grafanareader WITH PASSWORD 'password' GRANT SELECT ON dbo.YourTable3 TO grafanareader ``` Also, ensure that the user doesn't have any unwanted privileges from the public role. ### Diagnose connection issues If you use older versions of Microsoft SQL Server, such as 2008 and 2008R2, you might need to disable encryption before you can connect the data source. Grafana recommends that you use the latest available service pack for optimal compatibility. ### Provision the data source You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to [Provision Grafana](https://grafana.com/docs/grafana//administration/provisioning/). #### Example ```yaml apiVersion: 1 datasources: - name: MSSQL type: mssql url: localhost:1433 user: grafana jsonData: database: grafana maxOpenConns: 100 maxIdleConns: 100 maxIdleConnsAuto: true connMaxLifetime: 14400 connectionTimeout: 0 encrypt: 'false' secureJsonData: password: 'Password!' ``` ### Configure with Terraform You can configure the Microsoft SQL Server data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation. #### Terraform example The following example creates a basic Microsoft SQL Server data source: ```hcl resource "grafana_data_source" "mssql" { name = "MSSQL" type = "mssql" url = "localhost:1433" user = "grafana" json_data_encoded = jsonencode({ database = "grafana" maxOpenConns = 100 maxIdleConns = 100 maxIdleConnsAuto = true connMaxLifetime = 14400 connectionTimeout = 0 encrypt = "false" }) secure_json_data_encoded = jsonencode({ password = "Password!" }) } ``` For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). ## Next steps After configuring your Microsoft SQL Server data source, you can: - [Write queries](https://grafana.com/docs/grafana//datasources/mssql/query-editor/) using the query editor to explore and visualize your data - [Create template variables](https://grafana.com/docs/grafana//datasources/mssql/template-variables/) to build dynamic, reusable dashboards - [Add annotations](https://grafana.com/docs/grafana//datasources/mssql/annotations/) to overlay SQL Server events on your graphs - [Set up alerting](https://grafana.com/docs/grafana//datasources/mssql/alerting/) to create alert rules based on your SQL Server data - [Troubleshoot issues](https://grafana.com/docs/grafana//datasources/mssql/troubleshooting/) if you encounter problems with your data source # Microsoft SQL Server query editor The MSSQL query editor lets you write Transact-SQL queries directly in Grafana, with built-in macros and formatting options designed for time series and table visualizations. You can open it from [Explore](https://grafana.com/docs/grafana//explore/) or from any dashboard panel in edit mode. This page covers MSSQL-specific query features. For general querying concepts shared across all data sources, refer to [Query and transform data](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/). For Transact-SQL syntax, refer to [Write Transact-SQL statements](https://learn.microsoft.com/en-us/sql/t-sql/tutorial-writing-transact-sql-statements?view=sql-server-ver17) and [Transact-SQL reference](https://learn.microsoft.com/en-us/sql/t-sql/language-reference?view=sql-server-ver17) in the Microsoft documentation. The Microsoft SQL Server query editor has two modes: - [Builder mode](#builder-mode) - [Code mode](#code-mode) To switch between the editor modes, select the corresponding **Builder** and **Code** tabs in the upper right. ![MSSQL query builder](/media/mssql/mssql-query-editor-v12.png) {{< admonition type="warning" >}} When switching from **Code** mode to **Builder** mode, any changes made to your SQL query aren't saved and aren't shown in the builder interface. You can choose to copy your code to the clipboard or discard the changes. {{< /admonition >}} To run a query, select **Run query** in the upper right of the editor. In addition to writing queries, the query editor also allows you to create and use: - [Macros](#macros) - [Stored procedures](#use-stored-procedures) ## Builder mode **Builder mode** allows you to build queries using a visual interface. This mode is great for users who prefer a guided query experience or are just getting started with SQL. {{< figure alt="MSSQL builder mode>" src="/media/docs/mssql/mssql-builder-mode-v12.png" class="docs-image--no-shadow" >}} The following components help you build a T-SQL query: - **Format** - Select a format response from the drop-down for the MSSQL query. The default is **Table**. Refer to [Table queries](#table-queries) and [Time series queries](#time-series-queries) for more information and examples. If you select the **Time series** format option, you must include a `time` column. - **Dataset** - Select a database to query from the drop-down. Grafana automatically populates the drop-down with all databases the user has access to. If a database is set in the **Database** field on the data source configuration page or via a provisioning file, users are limited to querying only that database. Note that `tempdb`, `model`, `msdb`, and `master` system databases are not included in the query editor drop-down. - **Table** - Select a table from the drop-down. After selecting a database, the next drop-down displays all available tables in that database. - **Data operations** - _Optional_. Select an aggregation or a macro from the drop-down. You can add multiple data operations by clicking the **+ sign**. Click the **garbage can icon** to remove data operations. - **Column** - Select a column on which to run the aggregation. - **Interval** - Select an interval from the drop-down. You'll see this option when you choose a `time group` macro from the drop-down. - **Fill** - _Optional_. Add a `FILL` method to populate missing time intervals with default values (such as NULL, 0, or a specified value) when no data exists for those intervals. This ensures continuity in the time series, avoiding gaps in visualizations. - **Alias** - _Optional_. Add an alias from the drop-down. You can also add your own alias by typing it in the box and clicking **Enter**. Remove an alias by clicking the **X**. - **Filter** - Toggle to add filters. - **Filter by column value** - _Optional_. If you toggle **Filter** you can add a column to filter by from the drop-down. To filter by additional columns, click the **+ sign** to the right of the condition drop-down. You can choose a variety of operators from the drop-down next to the condition. When multiple filters are added, use the `AND` or `OR` operators to define how conditions are evaluated. `AND` requires all conditions to be true, while `OR` requires any condition to be true. Use the second drop-down to select the filter value. To remove a filter, click the **X icon** next to it. If you select a `date-type` column, you can use macros from the operator list and choose `timeFilter` to insert the `$\_\_timeFilter` macro into your query with the selected date column. After selecting a date type column, you can choose Macros from the operators list and select `timeFilter`, which adds the `$\_\_timeFilter` macro to the query with the selected date column. Refer to [Macros](#macros) for more information. - **Group** - Toggle to add a `GROUP BY` column. - **Group by column** - Select a column to filter by from the drop-down. Click the **+sign** to filter by multiple columns. Click the **X** to remove a filter. - **Order** - Toggle to add an `ORDER BY` statement. - **Order by** - Select a column to order by from the drop-down. Select ascending (`ASC`) or descending (`DESC`) order. - **Limit** - You can add an optional limit on the number of retrieved results. Default is 50. - **Preview** - Toggle for a preview of the SQL query generated by the query builder. Preview is toggled on by default. For additional detail about using formats, refer to [Table queries](#table-queries) and [Time series queries](#time-series-queries). ## Code mode {{< figure src="/static/img/docs/v92/sql_code_editor.png" class="docs-image--no-shadow" >}} **Code mode** lets you build complex queries using a text editor with helpful features like autocompletion and syntax highlighting. This mode is ideal for advanced users who need full control over the SQL query or want to use features not available in visual query mode. It's especially useful for writing subqueries, using macros, or applying advanced filtering and formatting. You can switch back to visual mode, but note that some custom queries may not be fully compatible. ### Code mode toolbar features Code mode has several features in a toolbar located in the editor's lower-right corner. - To reformat the query, click the brackets button (`{}`). - To expand the code editor, click the chevron button pointing downward. - To run the query, click the **Run query** button or use the keyboard shortcut **Ctrl/Cmd + Enter/Return**. ### Use autocompletion Code mode's autocompletion feature works automatically while typing. To manually trigger autocompletion, use the keyboard shortcut Ctrl/Cmd + Space. Code mode supports autocompletion of tables, columns, SQL keywords, standard SQL functions, Grafana template variables, and Grafana macros. {{< admonition type="note" >}} You can't autocomplete columns until you've specified a table. {{< /admonition >}} ## Macros To simplify syntax and to allow for dynamic components, such as date range filters, you can add macros to your query. Use macros in the `SELECT` clause to simplify the creation of time series queries. From the **Data operations** drop-down, choose a macro such as `$\_\_timeGroup` or `$\_\_timeGroupAlias`. Then, select a time column from the **Column** drop-down and a time interval from the **Interval** drop-down. This generates a time-series query based on your selected time grouping. {{< admonition type="warning" >}} Time macros (`$__time`, `$__timeFilter`, etc.) don't support time zone parameters in Microsoft SQL Server and always expand to UTC values. If your timestamps aren't stored in UTC (common with `datetime`/`datetime2` types), convert them to UTC in your SQL query using `AT TIME ZONE … AT TIME ZONE 'UTC'` rather than passing a time zone argument to a macro. {{< /admonition >}} | **Macro** | **Description** | | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `$__time(dateColumn)` | Renames the specified column to `_time`. Example: `dateColumn AS time` | | `$__timeEpoch(dateColumn)` | Converts a `DATETIME` column to a Unix timestamp and renames it to `_time`. Example: `DATEDIFF(second, '1970-01-01', dateColumn) AS time` | | `$__timeFilter(dateColumn)` | Adds a time range filter for the specified column. Example: `dateColumn BETWEEN '2017-04-21T05:01:17Z' AND '2017-04-21T05:06:17Z'` | | `$__timeFrom()` | Returns the start of the current time range. Example: `'2017-04-21T05:01:17Z'` | | `$__timeTo()` | Returns the end of the current time range. Example: `'2017-04-21T05:06:17Z'` | | `$__timeGroup(dateColumn, '5m'[, fillValue])` | Groups the specified time column into intervals (for example, 5 minutes). Optionally fills gaps with a value like `0`, `NULL`, or `previous`. Example: `CAST(ROUND(DATEDIFF(second, '1970-01-01', time_column)/300.0, 0) AS bigint) * 300` | | `$__timeGroup(dateColumn, '5m', 0)` | Same as above, with `0` used to fill missing data points. | | `$__timeGroup(dateColumn, '5m', NULL)` | Same as above, with `NULL` used for missing data points. | | `$__timeGroup(dateColumn, '5m', previous)` | Same as above, using the previous value to fill gaps. If no previous value exists, `NULL` is used. | | `$__timeGroupAlias(dateColumn, '5m')` | Same as `$__timeGroup`, but also adds an alias to the resulting column. | | `$__unixEpochFilter(dateColumn)` | Adds a time range filter using Unix timestamps. Example: `dateColumn > 1494410783 AND dateColumn < 1494497183` | | `$__unixEpochFrom()` | Returns the start of the current time range as a Unix timestamp. Example: `1494410783` | | `$__unixEpochTo()` | Returns the end of the current time range as a Unix timestamp. Example: `1494497183` | | `$__unixEpochNanoFilter(dateColumn)` | Adds a time range filter using nanosecond-precision Unix timestamps. Example: `dateColumn > 1494410783152415214 AND dateColumn < 1494497183142514872` | | `$__unixEpochNanoFrom()` | Returns the start of the current time range as a nanosecond Unix timestamp. Example: `1494410783152415214` | | `$__unixEpochNanoTo()` | Returns the end of the current time range as a nanosecond Unix timestamp. Example: `1494497183142514872` | | `$__unixEpochGroup(dateColumn, '5m', [fillMode])` | Same as `$__timeGroup`, but for Unix timestamps. Optional `fillMode` controls how to handle missing points. | | `$__unixEpochGroupAlias(dateColumn, '5m', [fillMode])` | Same as above, but adds an alias to the grouped column. | ### View the interpolated query The query editor includes a **Generated SQL** link that appears after you run a query while editing a panel. Click this link to view the raw interpolated SQL that Grafana executed, including any macros that were expanded during query processing. ## Table queries To create a Table query, set the **Format** option in the query editor to [**Table**](https://grafana.com/docs/grafana//panels-visualizations/visualizations/table/). This allows you to write any valid SQL query, and the Table panel displays the results using the returned columns and rows. **Example:** ```sql CREATE TABLE [event] ( time_sec bigint, description nvarchar(100), tags nvarchar(100), ) ``` ```sql CREATE TABLE [mssql_types] ( c_bit bit, c_tinyint tinyint, c_smallint smallint, c_int int, c_bigint bigint, c_money money, c_smallmoney smallmoney, c_numeric numeric(10,5), c_real real, c_decimal decimal(10,2), c_float float, c_char char(10), c_varchar varchar(10), c_text text, c_nchar nchar(12), c_nvarchar nvarchar(12), c_ntext ntext, c_datetime datetime, c_datetime2 datetime2, c_smalldatetime smalldatetime, c_date date, c_time time, c_datetimeoffset datetimeoffset ) INSERT INTO [mssql_types] SELECT 1, 5, 20020, 980300, 1420070400, '$20000.15', '£2.15', 12345.12, 1.11, 2.22, 3.33, 'char10', 'varchar10', 'text', N'☺nchar12☺', N'☺nvarchar12☺', N'☺text☺', GETDATE(), CAST(GETDATE() AS DATETIME2), CAST(GETDATE() AS SMALLDATETIME), CAST(GETDATE() AS DATE), CAST(GETDATE() AS TIME), SWITCHOFFSET(CAST(GETDATE() AS DATETIMEOFFSET), '-07:00') ``` **Example query with output:** ```sql SELECT * FROM [mssql_types] ``` {{< figure src="/static/img/docs/v51/mssql_table_query.png" max-width="500px" class="docs-image--no-shadow" >}} Use the keyword `AS` to define an alias in your query to rename a column or table. **Example query with output:** ```sql SELECT c_bit AS [column1], c_tinyint AS [column2] FROM [mssql_types] ``` {{< figure src="/static/img/docs/v51/mssql_table_result.png" max-width="1489px" class="docs-image--no-shadow" >}} ## Time series queries {{< admonition type="note" >}} Store timestamps in UTC to avoid issues with time shifts in Grafana when using non-UTC timezones. {{< /admonition >}} To create a time series query, set the **Format** option in the query editor to **Time series**. The query must include a column named `time`, which should contain either an SQL `datetime` value or a numeric value representing Unix epoch time in seconds. The result set must be sorted by the `time` column for panels to visualize the data correctly. A time series query returns results in [wide data frame format](https://grafana.com/developers/plugin-tools/key-concepts/data-frames#wide-format). - Any column except `time` or of the type `string` transforms into value fields in the data frame query result. - Any string column transforms into field labels in the data frame query result. You can enable macro support in the `SELECT` clause to simplify time series query creation. Use the **Data operations** drop-down to choose a macro such as `$\_\_timeGroup` or `$\_\_timeGroupAlias`, then select a time column from the Column drop-down and a time interval from the Interval drop-down. This generates a time-series query based on your selected time grouping. {{< docs/shared source="grafana" lookup="datasources/sql-query-builder-macros.md" version="" >}} ### Create a metric query For backward compatibility, there's an exception to the above rule for queries that return three columns and include a string column named `metric`. Instead of transforming the `metric` column into field labels, it becomes the field name, and then the series name is formatted as the value of the `metric` column. Refer to the example with the `metric` column below. To optionally customize the default series name formatting, refer to [Standard options definitions](https://grafana.com/docs/grafana//panels-visualizations/configure-standard-options/#display-name). **Example with `metric` column:** ```sql SELECT $__timeGroupAlias(time_date_time, '5m'), min("value_double"), 'min' as metric FROM test_data WHERE $__timeFilter(time_date_time) GROUP BY time ORDER BY 1 ``` Data frame result: ```text +---------------------+-----------------+ | Name: time | Name: min | | Labels: | Labels: | | Type: []time.Time | Type: []float64 | +---------------------+-----------------+ | 2020-01-02 03:05:00 | 3 | | 2020-01-02 03:10:00 | 6 | +---------------------+-----------------+ ``` ### Time series query examples **Use the fill parameter in the $\_\_timeGroupAlias macro to convert null values to be zero instead:** ```sql SELECT $__timeGroupAlias(createdAt, '5m', 0), sum(value) as value, hostname FROM test_data WHERE $__timeFilter(createdAt) GROUP BY time, hostname ORDER BY 1 ``` Given the data frame result in the following example and using the graph panel, you get two series named _value 10.0.1.1_ and _value 10.0.1.2_. To render the series with a name of _10.0.1.1_ and _10.0.1.2_, use a [Standard options definitions](https://grafana.com/docs/grafana//panels-visualizations/configure-standard-options/#display-name) display name value of `${__field.labels.hostname}`. Data frame result: ```text +---------------------+---------------------------+---------------------------+ | Name: time | Name: value | Name: value | | Labels: | Labels: hostname=10.0.1.1 | Labels: hostname=10.0.1.2 | | Type: []time.Time | Type: []float64 | Type: []float64 | +---------------------+---------------------------+---------------------------+ | 2020-01-02 03:05:00 | 3 | 4 | | 2020-01-02 03:10:00 | 6 | 7 | +---------------------+---------------------------+---------------------------+ ``` **Use multiple columns:** ```sql SELECT $__timeGroupAlias(time_date_time, '5m'), min(value_double) as min_value, max(value_double) as max_value FROM test_data WHERE $__timeFilter(time_date_time) GROUP BY time ORDER BY 1 ``` Data frame result: ```text +---------------------+-----------------+-----------------+ | Name: time | Name: min_value | Name: max_value | | Labels: | Labels: | Labels: | | Type: []time.Time | Type: []float64 | Type: []float64 | +---------------------+-----------------+-----------------+ | 2020-01-02 03:04:00 | 3 | 4 | | 2020-01-02 03:05:00 | 6 | 7 | +---------------------+-----------------+-----------------+ ``` ## Annotations You can use Microsoft SQL Server queries as annotation sources to overlay events on your dashboard graphs. For detailed instructions, query examples, and best practices, refer to [Microsoft SQL Server annotations](https://grafana.com/docs/grafana//datasources/mssql/annotations/). ## Use stored procedures Stored procedures have been verified to work with Grafana queries. However, note that there is no special handling or extended support for stored procedures, so some edge cases may not behave as expected. Stored procedures can be used in table, time series, and annotation queries, provided that the returned data matches the expected column names and formats described in the relevant previous sections in this document. {{< admonition type="note" >}} Grafana macro functions do not work inside stored procedures. {{< /admonition >}} {{< figure src="/static/img/docs/v51/mssql_metrics_graph.png" class="docs-image--no-shadow docs-image--right" >}} For the following examples, the database table is defined in [Time series queries](#time-series-queries). Let's say that we want to visualize four series in a graph panel, such as all combinations of columns `valueOne`, `valueTwo` and `measurement`. Graph panel to the right visualizes what we want to achieve. To solve this, you need to use two queries: **First query:** ```sql SELECT $__timeGroup(time, '5m') as time, measurement + ' - value one' as metric, avg(valueOne) as valueOne FROM metric_values WHERE $__timeFilter(time) GROUP BY $__timeGroup(time, '5m'), measurement ORDER BY 1 ``` **Second query:** ```sql SELECT $__timeGroup(time, '5m') as time, measurement + ' - value two' as metric, avg(valueTwo) as valueTwo FROM metric_values GROUP BY $__timeGroup(time, '5m'), measurement ORDER BY 1 ``` ### Stored procedure with epoch time format You can define a stored procedure to return all the data needed to render multiple series (for example, 4) in a graph panel. In the following example, the stored procedure accepts two parameters, `@from` and `@to`, both of type `int`. These parameters represent a time range (from/to) in epoch time format and are used to filter the results returned by the procedure. The query inside the procedure simulates the behavior of `$__timeGroup(time, '5m')` by grouping timestamps into 5-minute intervals. While the expressions for time grouping are somewhat verbose, they can be extracted into reusable SQL Server functions to simplify the procedure. ```sql CREATE PROCEDURE sp_test_epoch( @from int, @to int ) AS BEGIN SELECT cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int) as time, measurement + ' - value one' as metric, avg(valueOne) as value FROM metric_values WHERE time >= DATEADD(s, @from, '1970-01-01') AND time <= DATEADD(s, @to, '1970-01-01') GROUP BY cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int), measurement UNION ALL SELECT cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int) as time, measurement + ' - value two' as metric, avg(valueTwo) as value FROM metric_values WHERE time >= DATEADD(s, @from, '1970-01-01') AND time <= DATEADD(s, @to, '1970-01-01') GROUP BY cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int), measurement ORDER BY 1 END ``` Then, in your graph panel, you can use the following query to call the stored procedure with the time range dynamically populated by Grafana: ```sql DECLARE @from int = $__unixEpochFrom(), @to int = $__unixEpochTo() EXEC dbo.sp_test_epoch @from, @to ``` This uses Grafana built-in macros to convert the selected time range into epoch time (`$__unixEpochFrom()` and `$__unixEpochTo()`), which are passed to the stored procedure as input parameters. ### Stored procedure with `datetime` format You can define a stored procedure to return all the data needed to render four series in a graph panel. In the following example, the stored procedure accepts two parameters, `@from` and `@to`, of the type `datetime`. These parameters represent the selected time range and are used to filter the returned data. The query within the procedure mimics the behavior of `$__timeGroup(time, '5m')` by grouping data into 5-minute intervals. These expressions can be verbose, but you may extract them into reusable SQL Server functions for improved readability and maintainability. ```sql CREATE PROCEDURE sp_test_datetime( @from datetime, @to datetime ) AS BEGIN SELECT cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int) as time, measurement + ' - value one' as metric, avg(valueOne) as value FROM metric_values WHERE time >= @from AND time <= @to GROUP BY cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int), measurement UNION ALL SELECT cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int) as time, measurement + ' - value two' as metric, avg(valueTwo) as value FROM metric_values WHERE time >= @from AND time <= @to GROUP BY cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int), measurement ORDER BY 1 END ``` To call this stored procedure from a graph panel, use the following query with Grafana built-in macros to populate the time range dynamically: ```sql DECLARE @from datetime = $__timeFrom(), @to datetime = $__timeTo() EXEC dbo.sp_test_datetime @from, @to ``` ## Next steps After building your queries, you can: - [Use template variables](https://grafana.com/docs/grafana//datasources/mssql/template-variables/) to create dynamic, reusable dashboards - [Add annotations](https://grafana.com/docs/grafana//datasources/mssql/annotations/) to overlay SQL Server events on your graphs - [Set up alerting](https://grafana.com/docs/grafana//datasources/mssql/alerting/) to create alert rules based on your query results - [Troubleshoot issues](https://grafana.com/docs/grafana//datasources/mssql/troubleshooting/) if queries return unexpected results # Microsoft SQL Server template variables Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana displays these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as **template variables**. For general information on using variables in Grafana, refer to [Add variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/). For an introduction to templating and template variables, refer to [Templating](https://grafana.com/docs/grafana//dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/). ## Query variable A query variable in Grafana dynamically retrieves values from your data source using a query. With a query variable, you can write a SQL query that returns values such as measurement names, key names, or key values that are shown in a drop-down select box. For example, the following query returns all values from the `hostname` column: ```sql SELECT hostname FROM host ``` A query can return multiple columns, and Grafana automatically generates a list using the values from those columns. For example, the following query returns values from both the `hostname` and `hostname2` columns, which are included in the variable's drop-down list. ```sql SELECT [host].[hostname], [other_host].[hostname2] FROM host JOIN other_host ON [host].[city] = [other_host].[city] ``` You can also create a key/value variable using a query that returns two columns named `__text` and `__value`. - The `__text` column defines the label shown in the drop-down. - The `__value` column defines the value passed to panel queries. This is useful when you want to display a user-friendly label (like a hostname) but use a different underlying value (like an ID). Note that the values in the `_text` column should be unique. If there are duplicates, Grafana uses only the first matching entry. ```sql SELECT hostname __text, id __value FROM host ``` You can also create nested variables, where one variable depends on the value of another. For example, if you have a variable named `region`, you can configure a `hosts` variable to only show hosts from the selected region. If `region` is a multi-value variable, use the `IN` operator instead of `=` to match against multiple selected values. ```sql SELECT hostname FROM host WHERE region IN ($region) ``` ## Use variables in queries Grafana automatically quotes template variable values only when the template variable is a `multi-value`. When using a multi-value variable, use the `IN` comparison operator instead of `=` to match against multiple values. Grafana supports two syntaxes for using variables in queries: - **`$` syntax** Example with a template variable named `hostname`: ```sql SELECT atimestamp time, aint value FROM table WHERE $__timeFilter(atimestamp) and hostname in($hostname) ORDER BY atimestamp ``` - **`[[varname]]` syntax** Example with a template variable named `hostname`: ```sql SELECT atimestamp as time, aint as value FROM table WHERE $__timeFilter(atimestamp) and hostname in([[hostname]]) ORDER BY atimestamp ``` ## Format variables for SQL queries When using template variables inside SQL queries, the formatting option you choose determines how Grafana renders the variable value. Choosing the correct format prevents SQL injection risks and avoids broken queries from incorrect quoting. ### Use `sqlstring` for safe single-value interpolation The `${var:sqlstring}` format wraps the variable value in single quotes and escapes any internal single quotes. This is the recommended approach for single-value variables used in `WHERE` clauses: ```sql SELECT * FROM events WHERE hostname = ${hostname:sqlstring} ``` If `hostname` is set to `server01`, this expands to: ```sql SELECT * FROM events WHERE hostname = 'server01' ``` {{< admonition type="caution" >}} Don't manually wrap variables in quotes (for example, `'${var}'`). Since Grafana v11.3, multi-value variables are automatically quoted when used with the `IN` operator. Manually adding quotes results in double-quoting (for example, `''value''`), which breaks queries. {{< /admonition >}} ### Multi-value variable formatting When a variable allows multiple selections, Grafana provides several formatting options: | Format | Syntax | Output for selections `a`, `b` | Use case | | --------------------- | -------------------- | ------------------------------ | ------------------------------ | | Default (auto-quoted) | `$var` or `${var}` | `'a','b'` | `IN` clauses (recommended) | | `sqlstring` | `${var:sqlstring}` | `'a'` (single value only) | Equality comparisons | | `csv` | `${var:csv}` | `a,b` | Unquoted lists | | `singlequote` | `${var:singlequote}` | `'a','b'` | Explicit single-quote wrapping | | `doublequote` | `${var:doublequote}` | `"a","b"` | MSSQL identifier quoting | | `pipe` | `${var:pipe}` | `a\|b` | Regex patterns | | `raw` | `${var:raw}` | `a,b` | No escaping (use with caution) | For multi-value variables in SQL `WHERE` clauses, the default formatting with `IN` is typically correct: ```sql SELECT * FROM events WHERE hostname IN ($hostname) ``` If `hostname` has values `server01` and `server02` selected, this expands to: ```sql SELECT * FROM events WHERE hostname IN ('server01','server02') ``` ### Disable quoting for multi-value variables To output multi-value variables without automatic quoting, use the `csv` format: ```text ${servers:csv} ``` This outputs the values as an unquoted comma-separated list. Use this only when you handle quoting yourself or when the values are numeric. Refer to [Advanced variable format options](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/#advanced-variable-format-options) for the full list of formatting options. ## Handle the "All" option in SQL queries When a variable includes the **All** option (enabled in variable settings), you need to handle the case where the user selects "All" differently from individual selections. ### Pattern: Use a custom "All" value with `LIKE` Set the **Custom all value** field in the variable settings to `%`, then use `LIKE` in your query: ```sql SELECT * FROM events WHERE hostname LIKE ${hostname:sqlstring} ``` When the user selects "All", `hostname` resolves to `%`, and the `LIKE '%'` matches all rows. When a specific value is selected, `LIKE 'server01'` matches only that value. ### Pattern: Conditional `WHERE` clause with "All" For `IN` clauses where `%` doesn't work, use a conditional approach. Set the **Custom all value** to `ALL`, then write: ```sql SELECT * FROM events WHERE ( ${hostname:raw} = 'ALL' OR hostname IN ($hostname) ) ``` When the user selects "All", the first condition is true and all rows match. When specific values are selected, the `IN` clause filters normally. ### Pattern: Chained variables with "All" When you have chained (dependent) variables and the parent uses "All", apply the same pattern. For example, if `region` feeds into `hostname`: Variable query for `hostname`: ```sql SELECT hostname FROM host WHERE (${region:raw} = 'ALL' OR region IN ($region)) ``` Panel query: ```sql SELECT $__timeGroupAlias(time, '5m'), COUNT(*) as count FROM events WHERE $__timeFilter(time) AND (${region:raw} = 'ALL' OR region IN ($region)) AND (${hostname:raw} = 'ALL' OR hostname IN ($hostname)) GROUP BY $__timeGroup(time, '5m') ORDER BY 1 ``` ## Conditional SQL clauses Grafana doesn't have built-in support for conditionally omitting entire `WHERE` clauses based on variable selection. Use one of these patterns to achieve conditional filtering. ### Pattern: Always-true condition Use a condition that evaluates to true when the variable should be ignored: ```sql SELECT * FROM events WHERE $__timeFilter(time) AND (${env:raw} = 'ALL' OR environment IN ($env)) AND (${sev:raw} = 'ALL' OR severity IN ($sev)) ``` ### Pattern: CASE expression for optional filters For more complex scenarios, use a `CASE` expression: ```sql SELECT * FROM events WHERE $__timeFilter(time) AND 1 = CASE WHEN ${hostname:raw} = 'ALL' THEN 1 WHEN hostname IN ($hostname) THEN 1 ELSE 0 END ``` ## Known behaviors and limitations Be aware of the following behaviors when using template variables with Microsoft SQL Server. ### Automatic quoting in multi-value variables Since Grafana v11.3, multi-value variables used with the `IN` operator are automatically quoted. If you previously wrote `WHERE col IN ('${var}')` with manual quotes, remove the manual quotes and use `WHERE col IN ($var)` instead. ### Comments inside string literals The Grafana SQL query parser strips SQL comments (`--` and `/* */`) before macro interpolation. In some cases, `--` inside a quoted string literal could be incorrectly interpreted as a comment. If your query contains strings with `--`, verify the query expands correctly by clicking **Generated SQL** after running the query. ### Variables in alerting queries Template variables are not supported in alerting queries. Grafana evaluates alert rules on the backend without dashboard context. If your dashboard query uses variables, create a separate alerting query with hard-coded values. For more information, refer to [Microsoft SQL Server alerting](https://grafana.com/docs/grafana//datasources/mssql/alerting/#template-variables-not-supported). # Troubleshoot Microsoft SQL Server data source issues This document provides solutions to common issues you may encounter when configuring or using the Microsoft SQL Server (MSSQL) data source in Grafana. ## Configuration errors These errors occur when setting up or accessing the data source configuration in Grafana. ### Configuration form not displaying **Symptoms:** The data source settings page shows only **Delete** and **Back** buttons with no configuration fields. **Cause:** The logged-in user has insufficient permissions to configure data sources. Only users with the `Organization administrator` role (or a custom RBAC role with `datasources:write` permissions) can access the data source configuration form. **Solution:** 1. Ask an organization administrator to grant you the `Organization administrator` role, or assign you a custom RBAC role with data source write permissions. 1. Alternatively, ask an administrator to configure the data source on your behalf or use [provisioning](https://grafana.com/docs/grafana//administration/provisioning/) to define the data source in YAML. ## Connection errors These errors occur when Grafana cannot establish or maintain a connection to the Microsoft SQL Server. ### Unable to connect to the server **Error message:** `Unable to open tcp connection` or `dial tcp: connection refused` **Cause:** Grafana cannot establish a network connection to the SQL Server. **Solution:** 1. Verify that the SQL Server is running and accessible. 1. Check that the host and port are correct in the data source configuration. The default SQL Server port is `1433`. 1. Ensure there are no firewall rules blocking the connection between Grafana and SQL Server. 1. Verify that SQL Server is configured to allow remote connections. 1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your SQL Server instance is not publicly accessible. ### Connection timeout **Error message:** "Connection timed out" or "I/O timeout" **Cause:** The connection to SQL Server timed out before receiving a response. **Solution:** 1. Check the network latency between Grafana and SQL Server. 1. Verify that SQL Server is not overloaded or experiencing performance issues. 1. Increase the **Connection timeout** setting in the data source configuration under **Additional settings**. 1. Check if any network devices (load balancers, proxies) are timing out the connection. ### Encryption-related connection failures **Error message:** "TLS handshake failed" or "certificate verify failed" **Cause:** There is a mismatch between the encryption settings in Grafana and what the SQL Server supports or requires. **Solution:** 1. For older versions of SQL Server (2008, 2008R2), set the **Encrypt** option to **Disable** or **False** in the data source configuration. 1. If Encrypt is **True**, verify that the SQL Server has a valid TLS certificate. 1. Check that the certificate is trusted by the Grafana server (either in the system CA bundle or specified in **TLS/SSL Root Certificate**). 1. If the certificate's hostname doesn't match the configured **Host** value (for example, when using a load balancer or PDC), set **Hostname in server certificate** to match the certificate's CN or SAN. 1. As a temporary workaround for testing, enable **Skip TLS Verify** (not recommended for production). 1. Ensure you're using the latest available service pack for your SQL Server version for optimal compatibility. ### TLS handshake failures with FIPS-enabled Grafana **Error message:** "TLS handshake failed" or "no cipher suite supported by both client and server" **Cause:** Grafana builds using BoringCrypto (FIPS-compliant builds) enforce FIPS 140-3 cipher requirements. SQL Server instances that only support older cipher suites (TLS 1.0, RC4, or 3DES) are incompatible with these builds. **Solution:** 1. Upgrade the SQL Server instance to support TLS 1.2 with FIPS-approved cipher suites (AES-based). 1. On the SQL Server host, verify TLS 1.2 is enabled in the Windows registry or SQL Server Network Configuration. 1. If you cannot upgrade SQL Server, use a non-FIPS Grafana build. 1. Verify which TLS versions your SQL Server supports by checking the SQL Server error log for "TLS handshake" entries. ### TLS certificate errors through PDC **Error message:** "certificate verify failed" or "x509: certificate signed by unknown authority" when connecting through Private data source connect **Cause:** The SQL Server's TLS certificate must be verifiable from the PDC agent's perspective, not from Grafana Cloud. If the SQL Server uses a private CA certificate, the PDC agent host must trust that CA. **Solution:** 1. Install the private CA certificate on the PDC agent host's system trust store. 1. Alternatively, specify the CA certificate path in the data source's **TLS/SSL Root Certificate** field (the path must be accessible from the PDC agent, not from Grafana Cloud). 1. If the certificate's hostname doesn't match what the PDC agent sees, set **Hostname in server certificate** to the correct value. 1. Verify the certificate is valid (not expired) from the PDC agent host: `openssl s_client -connect :1433 -starttls mssql`. ### Named instance connection issues **Error message:** "Cannot connect to named instance" or connection fails when using instance name **Cause:** Grafana cannot resolve the SQL Server named instance. **Solution:** 1. Use the format `hostname\instancename` or `hostname\instancename,port` in the **Host** field. 1. Verify that the SQL Server Browser service is running on the SQL Server machine. 1. If the Browser service is unavailable, specify the port number directly: `hostname,port`. 1. Check that UDP port 1434 is open if using the SQL Server Browser service. ## Private data source connect (PDC) errors These errors occur when using Grafana Cloud with Private data source connect to reach an on-premises or private-network SQL Server. ### "No such host" when connecting through PDC **Error message:** `dial tcp: lookup sqlserver.internal.example.com: no such host` **Cause:** The PDC agent cannot resolve the SQL Server hostname from its network. This commonly occurs when: - The hostname in the data source configuration is a public DNS name that doesn't resolve inside the private network. - The PDC agent host doesn't have access to the internal DNS server. - A cluster or Availability Group Listener DNS entry is misconfigured. **Solution:** 1. In the data source **Host** field, use the hostname as resolvable from the PDC agent's network (for example, the internal FQDN or IP address, not a public DNS name). 1. Verify the PDC agent host can resolve the hostname: run `nslookup ` from the agent machine. 1. If using a cluster or Availability Group Listener, confirm the listener DNS record exists and resolves to the correct IP from the agent's network. 1. Check that the PDC agent host has the correct DNS servers configured. ### PDC connection breaks after maintenance or IP changes **Cause:** The PDC SSH tunnel endpoint IP changed after infrastructure maintenance, breaking the established connection. **Solution:** 1. Restart the PDC agent service to re-establish the SSH tunnel with the updated endpoint. 1. Verify the PDC agent status shows a healthy connection in **Connections** > **Private data source connect** in your Grafana Cloud stack. 1. If your network uses IP-based firewall rules for outbound traffic, update the allowlist. PDC endpoint IP addresses can change during maintenance windows. 1. Consider using DNS-based allowlisting instead of IP-based rules where possible to avoid future disruptions. ### Credentials not persisting with PDC enabled **Cause:** A known issue can cause the data source username to not persist when PDC is enabled. **Solution:** 1. After enabling PDC, save the data source configuration first, then re-enter the username and password and save again. 1. Click **Save & test** to verify the credentials are stored and the connection succeeds. 1. If the issue persists, try disabling PDC, saving credentials, then re-enabling PDC. ### Cannot connect to private SQL Server from Grafana Cloud **Cause:** Attempting to connect to an on-premises or private-network SQL Server from Grafana Cloud without PDC configured. **Solution:** 1. Grafana Cloud requires [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) for any SQL Server that isn't publicly accessible. 1. Firewall allowlisting alone doesn't work because Grafana Cloud data source connections don't use static IP addresses. 1. Install and configure a PDC agent on a host in your private network that can reach the SQL Server on port 1433. 1. Ensure the PDC agent host allows outbound connections on port 22 to the Grafana Cloud PDC endpoint. For setup instructions, refer to [Connect to on-premises SQL Server from Grafana Cloud](https://grafana.com/docs/grafana//datasources/mssql/configure/#connect-to-on-premises-sql-server-from-grafana-cloud). ## Authentication errors These errors occur when there are issues with authentication credentials or permissions. ### Login failed for user **Error message:** "Login failed for user 'username'" or "Authentication failed" **Cause:** The authentication credentials are invalid or the user doesn't have permission to access the database. **Solution:** 1. Verify that the username and password are correct. 1. Check that the user exists in SQL Server and is enabled. 1. Ensure the user has access to the specified database. 1. For Windows Authentication, verify that the credentials are in the correct format (`DOMAIN\User`). 1. Check that the SQL Server authentication mode allows the type of login you're using (SQL Server Authentication, Windows Authentication, or Mixed Mode). ### Login failed for user '' (empty username) **Error message:** "Login failed for user ''". The saved username appears blank when Grafana connects to SQL Server. **Cause:** A known issue on certain Grafana release channels caused the SQL Server Authentication username to appear saved in the UI but send an empty string to the server during connection. This was identified on the "fast" release channel. **Solution:** 1. Switch to the "steady" release channel if you're on "fast" and encounter this issue. 1. After upgrading or switching channels, open the data source configuration, re-enter the username and password, and click **Save & test** to confirm the credentials are stored correctly. 1. If the issue persists after re-saving, clear your browser cache and try again. ### Connection fails when password contains special characters **Error message:** "Login failed for user" or "Connection string parse error" when credentials contain semicolons (`;`) or closing braces (`}`) **Cause:** In Grafana versions prior to v13.0, semicolons and closing braces in usernames or passwords were not properly escaped in the MSSQL connection string, causing authentication failures. **Solution:** 1. Upgrade to Grafana v13.0 or later, which correctly handles these special characters. 1. If you cannot upgrade, change the SQL Server password to avoid `;` and `}` characters. ### Access denied to database **Error message:** `Cannot open database 'dbname' requested by the login` **Cause:** The authenticated user doesn't have permission to access the specified database. **Solution:** 1. Verify that the database name is correct in the data source configuration. 1. Ensure the user is mapped to the database with appropriate permissions. 1. Grant at least `SELECT` permission on the required tables: ```sql USE [your_database] GRANT SELECT ON dbo.YourTable TO [your_user] ``` 1. Check that the user doesn't have any conflicting permissions from the public role. ### Windows Authentication (Kerberos) issues **Error message:** "Kerberos authentication failed" or "Cannot initialize Kerberos" **Cause:** Kerberos configuration is incorrect or incomplete. **Solution:** 1. Verify that the Kerberos configuration file (`krb5.conf`) path is correct in the data source settings. The default path is `/etc/krb5.conf`. 1. For keytab authentication, ensure the keytab file exists and is readable by the Grafana service account. 1. Check that the realm and KDC settings are correct in the Kerberos configuration. 1. Verify DNS is correctly resolving the KDC servers. 1. Ensure the service principal name (SPN) is registered for the SQL Server instance. {{< admonition type="note" >}} Kerberos authentication is not supported in Grafana Cloud. Use SQL Server Authentication or Azure Entra ID instead. {{< /admonition >}} ### `KDC_ERR_C_PRINCIPAL_UNKNOWN` with Availability Group listeners **Error message:** "KDC_ERR_C_PRINCIPAL_UNKNOWN" when connecting through a SQL Server Availability Group Listener **Cause:** The SPN registered for the SQL Server instance doesn't match the Availability Group Listener's DNS name. Kerberos authentication requires an SPN that matches the hostname the client connects to. **Solution:** 1. Register an SPN for the Availability Group Listener DNS name: ```text setspn -S MSSQLSvc/:1433 \ setspn -S MSSQLSvc/ \ ``` 1. Verify the SPN is correctly registered: `setspn -L \`. 1. Ensure `krb5.conf` includes the correct realm mapping for the listener's domain. 1. If using multiple realms, add cross-realm trust entries to `krb5.conf`. ### Untrusted domain errors with Windows Authentication **Error message:** "The login is from an untrusted domain and cannot be used with Windows authentication" **Cause:** The Grafana server's domain isn't trusted by the SQL Server's domain, or the Kerberos realm configuration doesn't map correctly between domains. **Solution:** 1. Verify that a trust relationship exists between the Grafana server's domain and the SQL Server's domain. 1. Check that `krb5.conf` includes realm mappings for both domains: ```ini [realms] DOMAIN_A.COM = { kdc = kdc1.domain_a.com } DOMAIN_B.COM = { kdc = kdc1.domain_b.com } [domain_realm] .domain_a.com = DOMAIN_A.COM .domain_b.com = DOMAIN_B.COM ``` 1. Ensure the SQL Server service account has delegation permissions if cross-domain access is needed. 1. As an alternative, use SQL Server Authentication or Azure Entra ID (App Registration), which don't require domain trust. ### Azure Entra ID authentication errors **Error message:** "AADSTS error codes" or "Azure AD authentication failed" **Cause:** Azure Entra ID (formerly Azure AD) authentication is misconfigured. **Solution:** 1. For **App Registration** authentication: - Verify the tenant ID, client ID, and client secret are correct. - Ensure the app registration has been added as a user in the Azure SQL database. - Check that the client secret hasn't expired. 1. For **Managed Identity** authentication: - Verify `managed_identity_enabled = true` is set in the Grafana server configuration. - Ensure the managed identity has been added to the Azure SQL database. - Confirm the Azure resource hosting Grafana has managed identity enabled. 1. For **Current User** authentication: - Ensure `user_identity_enabled = true` is set in the Grafana server configuration. - Verify the app registration is configured to issue both Access Tokens and ID Tokens. - Check that the required API permissions are configured (`user_impersonation` for Azure SQL). For detailed Azure authentication configuration, refer to [Configure the Microsoft SQL Server data source](https://grafana.com/docs/grafana//datasources/mssql/configure/). ## Query errors These errors occur when there are issues with query syntax or configuration. ### Time column not found or invalid **Error message:** "Could not find time column" or time series visualization shows no data **Cause:** The query doesn't return a properly formatted `time` column for time series visualization. **Solution:** 1. Ensure your query includes a column named `time` when using the **Time series** format. 1. Use the `$__time()` macro to rename your date column: `$__time(your_date_column)`. 1. Verify the time column is of a valid SQL date/time type (`datetime`, `datetime2`, `date`) or contains Unix epoch values. 1. Ensure the result set is sorted by the time column using `ORDER BY`. ### Macro expansion errors **Error message:** "Error parsing query" or macros appear unexpanded in the query **Cause:** Grafana macros are being used incorrectly. **Solution:** 1. Verify macro syntax: use `$__timeFilter(column)` not `$_timeFilter(column)`. 1. Macros don't work inside stored procedures. Use explicit date parameters instead. 1. Check that the column name passed to macros exists in your table. 1. View the expanded query by clicking **Generated SQL** after running the query to debug macro expansion. ### Timezone and time shift issues **Cause:** Time series data appears shifted or doesn't align with expected times. **Solution:** 1. Store timestamps in UTC in your database to avoid timezone issues. 1. Time macros (`$__time`, `$__timeFilter`, etc.) always expand to UTC values. 1. If your timestamps are stored in local time, convert them to UTC in your query: ```sql SELECT your_datetime_column AT TIME ZONE 'Your Local Timezone' AT TIME ZONE 'UTC' AS time, value FROM your_table ``` 1. Don't pass timezone parameters to time macros. They're not supported. ### Query returns too many rows **Error message:** "Result set too large" or browser becomes unresponsive **Cause:** The query returns more data than can be efficiently processed. **Solution:** 1. Add time filters using `$__timeFilter(column)` to limit data to the dashboard time range. 1. Use aggregations (`AVG`, `SUM`, `COUNT`) with `GROUP BY` instead of returning raw rows. 1. Add a `TOP` clause to limit results: `SELECT TOP 1000 ...`. 1. Use the `$__timeGroup()` macro to aggregate data into time intervals. ### Stored procedure returns no data **Cause:** Stored procedure output isn't being captured correctly. **Solution:** 1. Ensure the stored procedure uses `SELECT` statements, not just variable assignments. 1. Remove `SET NOCOUNT ON` if present, or ensure it's followed by a `SELECT` statement. 1. Verify the stored procedure parameters are being passed correctly. 1. Test the stored procedure directly in SQL Server Management Studio with the same parameters. For more information on using stored procedures, refer to the [query editor documentation](https://grafana.com/docs/grafana//datasources/mssql/query-editor/). ## Performance issues These issues relate to slow queries or high resource usage. ### Slow query execution **Cause:** Queries take a long time to execute. **Solution:** 1. Reduce the dashboard time range to limit data volume. 1. Add indexes to columns used in `WHERE` clauses and time filters. 1. Use aggregations instead of returning individual rows. 1. Increase the **Min time interval** setting to reduce the number of data points. 1. Review the query execution plan in SQL Server Management Studio to identify bottlenecks. ### Connection pool exhaustion **Error message:** "Too many connections", "Connection pool exhausted", or "failed to connect to server" **Cause:** Too many concurrent connections to the database. This commonly manifests as intermittent alert failures while dashboards continue to work, because alert evaluations and dashboard queries compete for connections from the same pool. **Solution:** 1. Increase the **Max open** connection limit in the data source configuration. A value of `50` to `100` handles most deployments with concurrent dashboards and alerts. 1. Enable **Auto max idle** to automatically manage idle connections. 1. Set **Max lifetime** to `14400` (4 hours) to ensure stale connections are recycled. 1. Reduce the number of panels querying the same data source simultaneously, or split heavy dashboards into multiple pages. 1. Check for long-running queries that might be holding connections open (review SQL Server's `sys.dm_exec_sessions`). 1. If alerts fail intermittently but dashboards work, this is a strong indicator of pool exhaustion under concurrent load. ### Slow dashboard load with read-only replicas **Symptoms:** Dashboards take 5 to 6 minutes to load initially, or queries are significantly slower than expected. **Cause:** The data source is configured with `ApplicationIntent=ReadOnly` in the **Host** field, which routes all queries to a read-only secondary replica. If the replica is undersized, under-indexed, or experiencing replication lag, queries run much slower than on the primary. **Solution:** 1. Remove `ApplicationIntent=ReadOnly` from the Host field to route queries back to the primary instance. 1. If you need to use a read-only replica, ensure it has the same indexes and resources as the primary. 1. Check the replica's replication lag and resource utilization (CPU, I/O) during slow periods. 1. Consider creating a separate data source for replica queries and using it only for specific dashboards that don't require low latency. ## Alert errors These errors occur when using Microsoft SQL Server queries in Grafana Alerting. ### Alerts fail intermittently while dashboards work **Cause:** Connection pool exhaustion. Alert evaluations and dashboard queries share the same connection pool. Under concurrent load, alert evaluations may time out waiting for a free connection. **Solution:** Refer to [Connection pool exhaustion](#connection-pool-exhaustion) in the Performance issues section above. ### Alert queries return no data **Cause:** Alert queries have different requirements than dashboard queries. **Solution:** 1. Ensure the query format is set to **Time series** (Table format isn't supported for alerting). 1. Verify the query doesn't use template variables (alert queries can't resolve dashboard variables). 1. Check that the query returns data within the alert evaluation time range. 1. Test the query in Explore with a fixed time range to confirm it returns results. For more information, refer to [Microsoft SQL Server alerting](https://grafana.com/docs/grafana//datasources/mssql/alerting/). ## Template variable errors These errors occur when using template variables in queries. ### Template variable queries fail **Cause:** Variable queries return unexpected results or errors. **Solution:** 1. Verify the variable query syntax is valid SQL that returns a single column. 1. Check that the data source connection is working. 1. Ensure the user has permission to access the tables referenced in the variable query. 1. Test the query in the query editor before using it as a variable query. ### Double-quoting breaks queries with multi-value variables **Cause:** Since Grafana v11.3, multi-value variables used with `IN` are automatically quoted. If you manually wrapped the variable in quotes (for example, `WHERE col IN ('${var}')`), values are now double-quoted (for example, `''value''`), causing query failures. **Solution:** 1. Remove manual quotes around the variable: use `WHERE col IN ($var)` instead of `WHERE col IN ('${var}')`. 1. For single-value comparisons, use the `sqlstring` format: `WHERE col = ${var:sqlstring}`. 1. Refer to [Template variables](https://grafana.com/docs/grafana//datasources/mssql/template-variables/) for formatting options. ## Other common issues The following issues don't produce specific error messages but are commonly encountered. ### System databases appear in queries **Cause:** Queries accidentally access system databases. **Solution:** 1. The query editor automatically excludes `tempdb`, `model`, `msdb`, and `master` from the database dropdown. 1. Always specify the database in your data source configuration to restrict access. 1. Ensure the database user only has permissions on the intended database. ### Data appears incorrect or misaligned **Cause:** Data formatting or type conversion issues. **Solution:** 1. Use explicit column aliases to ensure consistent naming: `SELECT value AS metric`. 1. Verify numeric columns are actually numeric types, not strings. 1. Check for `NULL` values that might affect aggregations. 1. Use the `FILL` option in `$__timeGroup()` macro to handle missing data points. ## Get additional help If you continue to experience issues after following this troubleshooting guide: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs. 1. Enable debug logging in Grafana to capture detailed error information. 1. Check SQL Server logs for additional error details. 1. Contact [Grafana Support](https://grafana.com/contact/) if you're an Enterprise or Cloud customer. When reporting issues, include: - Grafana version - Microsoft SQL Server data source plugin version, found on the **Administration > Plugins** page. The plugin updates independently of Grafana releases. - SQL Server version - Error messages (redact sensitive information) - Steps to reproduce - Relevant query examples (redact sensitive data) # MySQL alerting You can use Grafana Alerting with MySQL to create alerts based on your MySQL data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met. For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). ## Before you begin Before creating alerts with MySQL, ensure you have: - A MySQL data source configured in Grafana. - Appropriate permissions to create alert rules. - Understanding of the metrics you want to monitor. ## Supported query types MySQL alerting works with **time series queries** that return numeric data over time. Table formatted queries are not supported in alert rule conditions. To create a valid alert query: - Include a `time` column that returns a SQL datetime or UNIX epoch timestamp - Return numeric values for the metrics you want to alert on - Sort results by the time column For more information on writing time series queries, refer to [MySQL query editor](https://grafana.com/docs/grafana//datasources/mysql/query-editor/). ### Query format requirements | Query format | Alerting support | Notes | | ------------ | ---------------- | ---------------------------------------- | | Time series | Yes | Required for alerting | | Table | No | Convert to time series format for alerts | ## Create an alert rule To create an alert rule using MySQL: 1. Navigate to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for the alert rule. 1. Select your **MySQL** data source. 1. Build your query using the query editor: - Set the **Format** to **Time series** - Include a time column using the `$__time()` or `$__timeGroup()` macro - Add numeric columns for the values to monitor - Use `$__timeFilter()` to filter data by the dashboard time range 1. Configure the alert condition (for example, when the average is above a threshold). 1. Set the evaluation interval and pending period. 1. Configure notifications and labels. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example alert queries The following examples show common alerting scenarios with MySQL. ### Alert on high error count Monitor the number of errors over time: ```sql SELECT $__timeGroup(created_at, '1m') AS time, COUNT(*) AS error_count FROM error_logs WHERE $__timeFilter(created_at) AND level = 'error' GROUP BY time ORDER BY time ``` **Condition:** When error_count is above 100. ### Alert on average response time Monitor API response times: ```sql SELECT $__timeGroup(request_time, '5m') AS time, AVG(response_time_ms) AS avg_response_time FROM api_requests WHERE $__timeFilter(request_time) GROUP BY time ORDER BY time ``` **Condition:** When avg_response_time is above 500 (milliseconds). ### Alert on low order volume Detect drops in order activity: ```sql SELECT $__timeGroup(order_date, '1h') AS time, COUNT(*) AS order_count FROM orders WHERE $__timeFilter(order_date) GROUP BY time ORDER BY time ``` **Condition:** When order_count is below 10. ### Alert on disk usage percentage Monitor database storage metrics: ```sql SELECT $__timeGroup(recorded_at, '5m') AS time, AVG(disk_used_percent) AS disk_usage FROM system_metrics WHERE $__timeFilter(recorded_at) AND metric_type = 'disk' GROUP BY time ORDER BY time ``` **Condition:** When disk_usage is above 85. ## Limitations When using MySQL with Grafana Alerting, be aware of the following limitations: ### Template variables not supported Alert queries cannot contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$hostname` or `$environment` won't be resolved. If your dashboard query uses template variables, create a separate query for alerting with hard coded values. ### Table format not supported Queries using the **Table** format cannot be used for alerting. Set the query format to **Time series** and ensure your query returns a time column. ### Query timeout Complex queries with large datasets may timeout during alert evaluation. Optimize queries for alerting by: - Adding appropriate `WHERE` clauses to limit data - Using indexes on time and filter columns - Reducing the time range evaluated ## Best practices Follow these best practices when creating MySQL alerts: - **Use time series format:** Always set the query format to Time series for alert queries. - **Include time filters:** Use the `$__timeFilter()` macro to limit data to the evaluation window. - **Optimize queries:** Add indexes on columns used in `WHERE` clauses and `GROUP BY`. - **Test queries first:** Verify your query returns expected results in Explore before creating an alert. - **Set realistic thresholds:** Base alert thresholds on historical data patterns. - **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor. # MySQL annotations Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics. You can use MySQL as a data source for annotations to display events such as deployments, alerts, or other significant occurrences on your visualizations. For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/). ## Before you begin Before creating MySQL annotations, ensure you have: - A MySQL data source configured in Grafana. - Tables containing event data with timestamp fields. - Read access to the tables containing your events. ## Create an annotation query To add a MySQL annotation to your dashboard: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Annotation query**. 1. Enter a name for the annotation query. 1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox. 1. Select a color for the annotation event markers. 1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed. 1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed. 1. Click **Open query editor** to open the **Annotation Query** dialog box. 1. Select your **MySQL** data source from the **Data source** drop-down list. 1. Configure the annotation query and field mappings. 1. (Optional) Click **Test annotation query** to ensure that the query is working properly. 1. Click **Close** when you've completed the query setup. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. ## Query columns Your annotation query must return a `time` column and can optionally include `timeend`, `text`, and `tags` columns. | Column | Required | Description | | --------- | -------- | --------------------------------------------------------------------------------------------- | | `time` | Yes | The timestamp for the annotation. Can be a SQL datetime or UNIX epoch value. | | `timeend` | No | The end timestamp for range annotations. Creates a shaded region instead of a vertical line. | | `text` | No | The annotation description displayed when you hover over the annotation. | | `tags` | No | Tags for the annotation as a comma-separated string. Helps categorize and filter annotations. | ## Example queries The following examples show common annotation query patterns. ### Basic annotation with epoch time Display events using UNIX epoch timestamps: ```sql SELECT epoch_time as time, description as text, CONCAT(tag1, ',', tag2) as tags FROM events WHERE $__unixEpochFilter(epoch_time) ``` ### Annotation with a single tag Display events with a single tag value: ```sql SELECT epoch_time as time, message as text, category as tags FROM event_log WHERE $__unixEpochFilter(epoch_time) ``` ### Range annotation with start and end time Display events with duration as shaded regions: ```sql SELECT start_time as time, end_time as timeend, description as text, CONCAT(type, ',', severity) as tags FROM incidents WHERE $__unixEpochFilter(start_time) ``` ### Annotation with native SQL datetime Display events using native MySQL datetime columns: ```sql SELECT event_date as time, message as text, CONCAT(category, ',', priority) as tags FROM system_events WHERE $__timeFilter(event_date) ``` ### Deployment annotations Display deployment events: ```sql SELECT deployed_at as time, CONCAT('Deployed ', version, ' to ', environment) as text, environment as tags FROM deployments WHERE $__timeFilter(deployed_at) ``` ### Maintenance window annotations Display maintenance windows as range annotations: ```sql SELECT start_time as time, end_time as timeend, CONCAT('Maintenance: ', description) as text, 'maintenance' as tags FROM maintenance_windows WHERE $__timeFilter(start_time) ``` ## Macros Use these macros in your annotation queries to filter by the dashboard time range: | Macro | Description | | ---------------------------- | ---------------------------------------------------------------- | | `$__timeFilter(column)` | Filters by time range using a native SQL datetime column. | | `$__unixEpochFilter(column)` | Filters by time range using a column with UNIX epoch timestamps. | ## Best practices Follow these best practices when creating MySQL annotations: - **Use time filters:** Always include `$__timeFilter()` or `$__unixEpochFilter()` to limit results to the dashboard time range. - **Keep queries efficient:** Add indexes on time columns and filter columns to improve query performance. - **Use meaningful text:** Include descriptive information in the `text` column to make annotations useful. - **Organize with tags:** Use consistent tag values to categorize annotations and enable filtering. - **Test queries first:** Verify your query returns expected results in Explore before adding it as an annotation. # MySQL template variables Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana displays these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as **template variables**. For an introduction to templating and template variables, refer to [Templating](https://grafana.com/docs/grafana//dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/). ## Query variable A query variable in Grafana dynamically retrieves values from your data source using a query. With a query variable, you can write a SQL query that returns values such as measurement names, key names, or key values that are shown in a drop-down select box. The variable query editor supports the same **Builder** and **Code** modes as the main query editor. Use Builder mode to construct queries visually by selecting a dataset, table, columns, and filters. Use Code mode to write SQL directly. For example, the following query returns all values from the `hostname` column: ```sql SELECT hostname FROM my_host ``` A query can return multiple columns, and Grafana automatically generates a list using the values from those columns. For example, the following query returns values from both the `hostname` and `hostname2` columns, which are included in the variable's drop-down list. ```sql SELECT my_host.hostname, my_other_host.hostname2 FROM my_host JOIN my_other_host ON my_host.city = my_other_host.city ``` To use time range dependent macros like `$__timeFilter(column)` in your query, you must set the template variable's refresh mode to **On Time Range Change**. ```sql SELECT event_name FROM event_log WHERE $__timeFilter(time_column) ``` ### Key/value variables You can create a key/value variable so the drop-down shows a user-friendly label (for example, hostname) while panel queries use a different value (for example, ID). Use the variable editor's **Value field** and **Text field** at the bottom of the query section to specify which query columns supply the value and the label. Your query can use any column names; you don't need `__value` or `__text` in the SQL. Example: run a query that returns `hostname` and `id`, then set **Text field** to `hostname` and **Value field** to `id`. ```sql SELECT hostname, id FROM my_host ``` Note that the values in the text column should be unique. If there are duplicates, Grafana uses only the first matching entry. Alternatively, you can use the legacy approach: return columns named `__text` and `__value` in your query (for example, `SELECT hostname AS __text, id AS __value FROM my_host`). ### Nested variables You can create nested variables, where one variable depends on the value of another. For example, if you have a variable named `region`, you can configure a `hosts` variable to only show hosts from the selected region. If `region` is a multi-value variable, use the `IN` operator instead of `=` to match against multiple selected values. ```sql SELECT hostname FROM my_host WHERE region IN($region) ``` ### Filter results with `__searchFilter` Using `__searchFilter` in the query field allows the query results to be filtered based on the user's input in the drop-down selection box. If you don't enter anything, the default value for `__searchFilter` is `%`. Note that you must enclose the `__searchFilter` expression in quotes as Grafana doesn't add them automatically. The following example demonstrates how to use `__searchFilter` in the query field to enable real-time searching for `hostname` as the user types in the drop-down selection box. ```sql SELECT hostname FROM my_host WHERE hostname LIKE '$__searchFilter' ``` ## Use variables in queries Grafana automatically quotes template variable values only when the template variable is a `multi-value`. When using a multi-value variable, use the `IN` comparison operator instead of `=` to match against multiple values. Grafana supports two syntaxes for using variables in queries: - **`$` syntax** Example with a template variable named `hostname`: ```sql SELECT UNIX_TIMESTAMP(atimestamp) as time, aint as value, avarchar as metric FROM my_table WHERE $__timeFilter(atimestamp) and hostname in($hostname) ORDER BY atimestamp ASC ``` - **`[[varname]]` syntax** Example with a template variable named `hostname`: ```sql SELECT UNIX_TIMESTAMP(atimestamp) as time, aint as value, avarchar as metric FROM my_table WHERE $__timeFilter(atimestamp) and hostname in([[hostname]]) ORDER BY atimestamp ASC ``` ### Disable quoting for multi-value variables By default, Grafana formats multi-value variables as a quoted, comma-separated string. For example, if `server01` and `server02` are selected, the result will be `'server01'`, `'server02'`. To disable quoting, use the `csv` formatting option for variables: ```text ${servers:csv} ``` This outputs the values as an unquoted comma-separated list. ### Use explicit quoting for string variables Don't rely on implicit quoting behavior for string variables in SQL queries. Quoting behavior can vary depending on panel type and context: - **Repeat panels** may not quote single-value variables at all, causing bare values to appear in the SQL and breaking the query. - If you manually wrap a variable in quotes (for example, `WHERE name = '$myvar'`) and Grafana also applies its own quoting, the value gets double-quoted (for example, `''30''` instead of `'30'`). This intentional quoting behavior was restored in Grafana 11.3. To avoid both problems, use the `sqlstring` format option. It handles escaping and quoting in a single step, so you don't add your own quotes around the variable: ```sql SELECT * FROM my_table WHERE hostname = ${hostname:sqlstring} ``` This produces a safely quoted string (for example, `'server01'`) regardless of panel type or context. For multi-value variables, use `IN` with `${var:sqlstring}`: ```sql SELECT * FROM my_table WHERE hostname IN (${hostname:sqlstring}) ``` {{< admonition type="caution" >}} Don't wrap `${var:sqlstring}` in additional quotes. The `sqlstring` formatter already produces a quoted value. Writing `'${var:sqlstring}'` results in double quoting. {{< /admonition >}} Refer to [Advanced variable format options](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/#advanced-variable-format-options) for additional information. # Troubleshoot MySQL data source issues This document provides solutions to common issues you may encounter when configuring or using the MySQL data source in Grafana. ## Connection errors These errors occur when Grafana cannot establish or maintain a connection to the MySQL server. ### Unable to connect to the server **Error message:** "dial tcp: connection refused" or "Could not connect to MySQL" **Cause:** Grafana cannot establish a network connection to the MySQL server. **Solution:** 1. Verify that the MySQL server is running and accessible. 1. Check that the host and port are correct in the data source configuration. The default MySQL port is `3306`. 1. Ensure there are no firewall rules blocking the connection between Grafana and MySQL. 1. Verify that MySQL is configured to allow remote connections by checking the `bind-address` setting in your MySQL configuration. 1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your MySQL instance isn't publicly accessible. ### Connection timeout **Error message:** "Connection timed out" or "I/O timeout" **Cause:** The connection to MySQL timed out before receiving a response. **Solution:** 1. Check the network latency between Grafana and MySQL. 1. Verify that MySQL isn't overloaded or experiencing performance issues. 1. Check if any network devices (load balancers, proxies) are timing out the connection. 1. Increase the `wait_timeout` setting in MySQL if connections are timing out during idle periods. ### TLS/SSL connection failures TLS errors occur when the MySQL server requires or expects encrypted connections but the Grafana data source isn't configured to match. Common error messages include: - "TLS handshake failed" or "x509: certificate verify failed" - "Connection refused" or connection drops immediately - "Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON" **Possible causes and solutions:** | Cause | Solution | | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | MySQL has `require_secure_transport` enabled or AWS RDS Proxy enforces TLS | Enable **With CA Cert** or **Skip TLS Verification** in the data source configuration. Either option establishes a TLS connection. | | Self-signed or private CA certificate isn't trusted | Enable **With CA Cert** and provide the root certificate under **TLS/SSL Root Certificate**. | | Certificate has expired or doesn't match the hostname | Renew the certificate, or enable **Skip TLS Verification** for development environments only. | | TLS handshake fails against AWS RDS or RDS Proxy | Provide the [Amazon RDS root CA certificate](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html) under **TLS/SSL Root Certificate** with **With CA Cert** enabled. | | Organization requires mutual TLS (mTLS) | Enable **Use TLS Client Auth** and provide both a client certificate and key in addition to the CA certificate. | {{< admonition type="note" >}} **Skip TLS Verification** and **With CA Cert** both establish encrypted connections. You don't need **Use TLS Client Auth** unless your setup requires mutual TLS. For most AWS RDS and RDS Proxy connections, **With CA Cert** with the Amazon root CA is sufficient. {{< /admonition >}} For provisioned data sources, enable TLS in `jsonData`: ```yaml jsonData: tlsAuthWithCACert: true secureJsonData: tlsCACert: ${CA_CERT} ``` To skip certificate validation instead, use: ```yaml jsonData: tlsSkipVerify: true ``` For all available TLS provisioning options, refer to the [TLS provisioning examples](https://grafana.com/docs/grafana//datasources/mysql/configure/#mysql-provisioning-examples). ### Connection reset by peer **Error message:** "Connection reset by peer" or "EOF" **Cause:** The MySQL server closed the connection unexpectedly. **Solution:** 1. Check the `max_connections` setting on the MySQL server to ensure it isn't being exceeded. 1. Verify the `wait_timeout` and `interactive_timeout` settings in MySQL aren't set too low. 1. Increase the **Max lifetime** setting in Grafana's data source configuration to be lower than the MySQL `wait_timeout`. 1. Check MySQL server logs for any errors or connection-related messages. ## Authentication errors These errors occur when there are issues with authentication credentials or permissions. ### Access denied for user **Error message:** "Access denied for user 'username'@'host'" or "Authentication failed" **Cause:** The authentication credentials are invalid or the user doesn't have permission to connect from the Grafana server's host. **Solution:** 1. Verify that the username and password are correct. 1. Check that the user exists in MySQL and is enabled. 1. Ensure the user has permission to connect from the Grafana server's IP address. MySQL restricts access based on the connecting host: ```sql SELECT user, host FROM mysql.user WHERE user = 'your_user'; ``` 1. If necessary, create a user that can connect from the Grafana server: ```sql CREATE USER 'grafana'@'grafana_server_ip' IDENTIFIED BY 'password'; ``` 1. If using the `mysql_native_password` authentication plugin, ensure it's enabled on the server. ### Cannot access database **Error message:** "Access denied for user 'username'@'host' to database 'dbname'" **Cause:** The authenticated user doesn't have permission to access the specified database. **Solution:** 1. Verify that the database name is correct in the data source configuration. 1. Ensure the user has the required permissions on the database: ```sql GRANT SELECT ON your_database.* TO 'grafana'@'grafana_server_ip'; FLUSH PRIVILEGES; ``` 1. For production environments, grant permissions only on specific tables: ```sql GRANT SELECT ON your_database.your_table TO 'grafana'@'grafana_server_ip'; ``` ### PAM authentication issues **Error message:** "Authentication plugin 'auth_pam' cannot be loaded" or cleartext password errors **Cause:** PAM (Pluggable Authentication Modules) authentication requires cleartext password transmission. **Solution:** 1. Enable **Allow Cleartext Passwords** in the data source configuration if using PAM authentication. 1. Ensure TLS is enabled to protect password transmission when using cleartext passwords. 1. Verify that the PAM plugin is correctly installed and configured on the MySQL server. ## Query errors These errors occur when there are issues with query syntax or configuration. ### Time column not found or invalid **Error message:** "Could not find time column" or time series visualization shows no data **Cause:** The query doesn't return a properly formatted `time` column for time series visualization. **Solution:** 1. Ensure your query includes a column named `time` when using the **Time series** format. 1. Use the `$__time()` macro to convert your date column: `$__time(your_date_column)`. 1. Verify the time column is of a valid MySQL date/time type (`DATETIME`, `TIMESTAMP`, `DATE`) or contains Unix epoch values. 1. Ensure the result set is sorted by the time column using `ORDER BY`. ### Macro expansion errors **Error message:** "Error parsing query" or macros appear unexpanded in the query **Cause:** Grafana macros are being used incorrectly. **Solution:** 1. Verify macro syntax: use `$__timeFilter(column)` not `$_timeFilter(column)`. 1. Check that the column name passed to macros exists in your table. 1. View the expanded query by clicking **Generated SQL** after running the query to debug macro expansion. 1. Ensure backticks are used for reserved words or special characters in column names: `$__timeFilter(\`time-column\`)`. ### Hash character in quoted strings causes query errors **Error message:** Query returns unexpected results or errors when string values contain `#` **Cause:** In versions before Grafana 13.0, the SQL comment stripping logic incorrectly treated `#` inside quoted strings (for example, `WHERE color = '#FF0000'`) as a comment delimiter, which stripped the rest of the line. **Solution:** 1. Upgrade to Grafana 13.0 or later, which preserves `#` inside quoted strings. 1. As a workaround on older versions, use the `CONCAT` function or a hexadecimal literal to avoid the `#` character in string literals. ### Timezone and time shift issues **Cause:** Time series data appears shifted or doesn't align with expected times. **Solution:** 1. Store timestamps in UTC in your database to avoid timezone issues. 1. Time macros (`$__time`, `$__timeFilter`, etc.) always expand to UTC values. 1. Set the **Session Timezone** in the data source configuration to match your data's timezone, or use `+00:00` for UTC. 1. If your timestamps are stored in local time, convert them to UTC in your query: ```sql SELECT CONVERT_TZ(your_datetime_column, 'Your/Timezone', 'UTC') AS time, value FROM your_table ``` ### Query returns too many rows **Error message:** "Result set too large" or browser becomes unresponsive **Cause:** The query returns more data than can be efficiently processed. **Solution:** 1. Add time filters using `$__timeFilter(column)` to limit data to the dashboard time range. 1. Use aggregations (`AVG`, `SUM`, `COUNT`) with `GROUP BY` instead of returning raw rows. 1. Add a `LIMIT` clause to restrict results: `SELECT ... LIMIT 1000`. 1. Use the `$__timeGroup()` macro to aggregate data into time intervals. ### Syntax error in SQL statement **Error message:** "You have an error in your SQL syntax" followed by specific error details **Cause:** The SQL query contains invalid syntax. **Solution:** 1. Check for missing or extra commas, parentheses, or quotes. 1. Ensure reserved words used as identifiers are enclosed in backticks: `` `table` ``, `` `select` ``. 1. Verify that template variable syntax is correct: `$variable` or `${variable}`. 1. Test the query directly in a MySQL client to isolate Grafana-specific issues. ### Unknown column in field list **Error message:** "Unknown column 'column_name' in 'field list'" **Cause:** The specified column doesn't exist in the table or is misspelled. **Solution:** 1. Verify the column name is spelled correctly. 1. Check that the column exists in the specified table. 1. If the column name contains special characters or spaces, enclose it in backticks: `` `column-name` ``. 1. Ensure the correct database is selected if you're referencing columns without the full table path. ### `CASE` expression fails on non-standard string column types **Error message:** Query with `CASE WHEN` returns an error or unexpected results in Grafana, but works in other MySQL clients **Cause:** The Grafana MySQL driver doesn't handle all MySQL string subtypes (such as `ENUM`, `SET`, `TINYTEXT`, or `MEDIUMTEXT`) in `CASE` expressions. The driver may fail to infer the result type correctly. **Solution:** 1. Use `CAST()` to normalize the column to `CHAR` before using it in a `CASE` expression: ```sql SELECT CASE WHEN CAST(status_column AS CHAR) = 'active' THEN 1 ELSE 0 END AS is_active FROM my_table ``` 1. This is a known limitation of the Go MySQL driver used by Grafana. Explicitly casting to `CHAR` ensures consistent type handling. ## Performance issues These issues relate to slow queries or high resource usage. ### Slow query execution **Cause:** Queries take a long time to execute. **Solution:** 1. Reduce the dashboard time range to limit data volume. 1. Add indexes to columns used in `WHERE` clauses and time filters: ```sql CREATE INDEX idx_time ON your_table(time_column); ``` 1. Use aggregations instead of returning individual rows. 1. Increase the **Min time interval** setting to reduce the number of data points. 1. Review the query execution plan using `EXPLAIN` to identify bottlenecks: ```sql EXPLAIN SELECT * FROM your_table WHERE time_column > NOW() - INTERVAL 1 HOUR; ``` ### Connection pool exhaustion **Error message:** "Too many connections" or "Connection pool exhausted" **Cause:** Too many concurrent connections to the database. **Solution:** 1. Increase the **Max open** connection limit in the data source configuration. 1. Enable **Auto (max idle)** to automatically manage idle connections. 1. Reduce the number of panels querying the same data source simultaneously. 1. Check for long-running queries that might be holding connections. 1. Increase the `max_connections` setting in MySQL if necessary: ```sql SHOW VARIABLES LIKE 'max_connections'; SET GLOBAL max_connections = 200; ``` ### Query timeout **Error message:** "Query execution was interrupted" or "Lock wait timeout exceeded" **Cause:** The query takes too long and exceeds the configured timeout. **Solution:** 1. Optimize the query by adding appropriate indexes. 1. Reduce the amount of data being queried by narrowing the time range. 1. Use aggregations to reduce the result set size. 1. Check for table locks that might be blocking the query. ## Other common issues The following issues don't produce specific error messages but are commonly encountered. ### Template variable queries fail **Cause:** Variable queries return unexpected results or errors. **Solution:** 1. Verify the variable query syntax is valid SQL that returns a single column. 1. Check that the data source connection is working. 1. Ensure the user has permission to access the tables referenced in the variable query. 1. Test the query in the query editor before using it as a variable query. ### Data appears incorrect or misaligned **Cause:** Data formatting or type conversion issues. **Solution:** 1. Use explicit column aliases to ensure consistent naming: `SELECT value AS metric`. 1. Verify numeric columns are actually numeric types, not strings. 1. Check for `NULL` values that might affect aggregations. 1. Use the `FILL` option in `$__timeGroup()` macro to handle missing data points. ### Special characters in database or table names **Cause:** Queries fail when tables or databases contain reserved words or special characters. **Solution:** 1. Enclose identifiers with special characters in backticks: `` `my-database`.`my-table` ``. 1. The query editor automatically handles this for selections, but manual queries require backticks. 1. Avoid using reserved words as identifiers when possible. ### Query caching can't be enabled on a provisioned data source **Cause:** When a MySQL data source is provisioned via YAML, the query caching toggle in the UI is greyed out. YAML provisioning doesn't currently support setting caching configuration. **Solution:** 1. Use the Grafana HTTP API to enable query caching on the provisioned data source. Send a `PUT` request to update the data source with caching enabled: ```sh curl -X PUT -H "Content-Type: application/json" \ -H "Authorization: Bearer " \ "https:///api/datasources/" \ -d '{ "jsonData": { "queryCachingTTL": 300 } }' ``` Replace `` with a valid Grafana API token, `` with your Grafana instance URL, and `` with the numeric ID of the data source. Set `queryCachingTTL` to the desired cache duration in seconds. 1. Retrieve the data source ID by calling `GET /api/datasources/name/`. 1. YAML-based caching configuration is an open feature request. Check [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for current status. {{< admonition type="note" >}} Query caching is available in Grafana Enterprise and Grafana Cloud. For more information, refer to [Query and resource caching](https://grafana.com/docs/grafana//administration/data-source-management/#query-and-resource-caching). {{< /admonition >}} ### An unexpected error happened **Error message:** "An unexpected error happened" **Cause:** A general error occurred that doesn't have a specific error message. **Solution:** 1. Check the Grafana server logs for more details about the error. 1. Verify all data source configuration settings are correct. 1. Test the connection using the **Save & test** button. 1. Ensure the MySQL server is accessible and responding to queries. 1. For Grafana Cloud customers, contact support for assistance. ## Get additional help If you continue to experience issues after following this troubleshooting guide: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs. 1. Enable debug logging in Grafana to capture detailed error information. 1. Check MySQL error logs for additional details. 1. Contact Grafana Support if you're an Enterprise or Cloud customer. When reporting issues, include: - Grafana version - MySQL data source plugin version, found on the **Administration > Plugins** page. The plugin updates independently of Grafana releases. - MySQL version - Error messages (redact sensitive information) - Steps to reproduce - Relevant query examples (redact sensitive data) # OpenTSDB alerting You can use Grafana Alerting with OpenTSDB to create alerts based on your time series data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met. For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). ## Before you begin Before creating alerts with OpenTSDB, ensure you have: - An OpenTSDB data source configured in Grafana. Refer to [Configure the OpenTSDB data source](https://grafana.com/docs/grafana//datasources/opentsdb/configure/). - Appropriate permissions to create alert rules. - Understanding of the metrics you want to monitor. ## Supported features OpenTSDB alerting works with standard metric queries that return time series data. The following table summarizes alerting compatibility: | Query type | Alerting support | Notes | | ----------------------------- | ---------------- | ------------------------------------ | | Metrics with aggregation | Yes | Recommended for alerting | | Metrics with downsampling | Yes | Use appropriate intervals | | Metrics with rate calculation | Yes | Useful for counter metrics | | Metrics with filters/tags | Yes | Filter to specific hosts or services | ## Create an alert rule To create an alert rule using OpenTSDB: 1. Navigate to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for the alert rule. 1. Select your **OpenTSDB** data source. 1. Build your query: - Select the metric to monitor. - Choose an appropriate aggregator (for example, `avg`, `sum`, `max`). - Add tag filters to target specific resources. - Enable downsampling with an interval matching your evaluation frequency. 1. Configure the alert condition (for example, when the value is above a threshold). 1. Set the evaluation interval and pending period. 1. Configure notifications and labels. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example alert queries The following examples show common alerting scenarios with OpenTSDB. ### Alert on high CPU usage Monitor CPU usage and alert when it exceeds 90%: | Field | Value | | --------------------- | -------------- | | Metric | `sys.cpu.user` | | Aggregator | `avg` | | Tags | `host=*` | | Downsample Interval | `1m` | | Downsample Aggregator | `avg` | **Condition:** When average is above `90` ### Alert on low disk space Monitor available disk space and alert when it drops below a threshold: | Field | Value | | --------------------- | ------------------- | | Metric | `sys.disk.free` | | Aggregator | `min` | | Tags | `host=*`, `mount=/` | | Downsample Interval | `5m` | | Downsample Aggregator | `min` | **Condition:** When minimum is below `10737418240` (10 GB in bytes) ### Alert on high network traffic rate Monitor network bytes received and alert on high traffic: | Field | Value | | --------------------- | -------------------- | | Metric | `net.bytes.received` | | Aggregator | `sum` | | Tags | `host=webserver01` | | Rate | enabled | | Counter | enabled | | Downsample Interval | `1m` | | Downsample Aggregator | `avg` | **Condition:** When sum is above `104857600` (100 MB/s in bytes) ### Alert on error count spike Monitor application error counts: | Field | Value | | --------------------- | ------------------------------- | | Metric | `app.errors.count` | | Aggregator | `sum` | | Tags | `service=api`, `env=production` | | Downsample Interval | `5m` | | Downsample Aggregator | `sum` | **Condition:** When sum is above `100` ## Limitations When using OpenTSDB with Grafana Alerting, be aware of the following limitations. ### Template variables not supported Alert queries can't contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$hostname` or `$environment` aren't resolved. If your dashboard query uses template variables, create a separate query for alerting with hard-coded values. ### Query complexity Complex queries with many tags or long time ranges may timeout or fail to evaluate. Simplify queries for alerting by: - Using specific tag filters instead of wildcards where possible. - Enabling downsampling with appropriate intervals. - Reducing the evaluation time range. ### OpenTSDB 2.4 behavior When using OpenTSDB 2.4 with alerting, Grafana executes queries with the parameter `arrays=true`. This causes OpenTSDB to return data points as an array of arrays instead of a map of key-value pairs. Grafana automatically converts this data to the appropriate format. ## Best practices Follow these best practices when creating OpenTSDB alerts: - **Use specific tag filters:** Add tag filters to focus on relevant resources and improve query performance. - **Match downsample interval to evaluation:** Set the downsample interval to match or be slightly smaller than your alert evaluation interval. - **Test queries first:** Verify your query returns expected results in [Explore](https://grafana.com/docs/grafana//explore/) before creating an alert. - **Set realistic thresholds:** Base alert thresholds on historical data patterns to avoid false positives. - **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor. - **Enable downsampling:** Always enable downsampling for alerting queries to reduce data volume and improve reliability. - **Consider counter resets:** For counter metrics, enable the Counter option and set appropriate max values to handle resets correctly. ## Next steps - [Build queries](https://grafana.com/docs/grafana//datasources/opentsdb/query-editor/) to explore your metrics before creating alerts. - [Troubleshoot issues](https://grafana.com/docs/grafana//datasources/opentsdb/troubleshooting/) if alerts aren't firing as expected. # OpenTSDB annotations Annotations allow you to overlay event information on graphs, providing context for metric changes. The OpenTSDB data source supports both metric-specific annotations and global annotations stored in OpenTSDB. For general information about annotations in Grafana, refer to [Annotate visualizations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/). ## Annotation types OpenTSDB supports two types of annotations: | Type | Description | Use case | | ---------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | **Metric annotations** | Annotations attached to a specific time series (TSUID). Retrieved by querying the associated metric. | Track events affecting a specific host or service. | | **Global annotations** | Annotations not tied to any time series. Apply system-wide. | Track deployments, maintenance windows, or infrastructure-wide events. | ## How Grafana retrieves annotations When you configure an annotation query, Grafana queries OpenTSDB for the specified metric and retrieves any annotations associated with that metric's time series. The query includes the `globalAnnotations=true` parameter, which allows Grafana to also retrieve global annotations when enabled. Grafana displays the `description` field from each annotation as the annotation text. ## Configure an annotation query To add OpenTSDB annotations to a dashboard: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Annotation query**. 1. Enter a name for the annotation query. 1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox. 1. Select a color for the annotation event markers. 1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed. 1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed. 1. Click **Open query editor** to open the **Annotation Query** dialog box. 1. Select the **OpenTSDB** data source from the **Data source** drop-down list. 1. Configure the annotation query and field mappings. 1. (Optional) Click **Test annotation query** to ensure that the query is working properly. 1. Click **Close** when you've completed the query setup. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. ## Annotation query fields | Field | Description | | --------------------------- | -------------------------------------------------------------------------------- | | **Name** | A descriptive name for this annotation query. Appears in the annotation legend. | | **Data source** | Select the OpenTSDB data source. | | **Enabled** | Toggle to enable or disable this annotation query. | | **OpenTSDB metrics query** | The metric name to query for annotations (for example, `events.deployment`). | | **Show Global Annotations** | Toggle to include global annotations that aren't tied to a specific time series. | ## Example annotation queries The following examples demonstrate common annotation use cases. ### Track application deployments Monitor when deployments occur for a specific application: | Field | Value | | ----------------------- | --------------- | | Name | App Deployments | | OpenTSDB metrics query | `deploy.myapp` | | Show Global Annotations | disabled | This query retrieves annotations attached to the `deploy.myapp` metric, showing deployment events for that specific application. ### Monitor infrastructure-wide events Capture system-wide events such as network changes or data center maintenance: | Field | Value | | ----------------------- | ----------------------- | | Name | Infrastructure Events | | OpenTSDB metrics query | `events.infrastructure` | | Show Global Annotations | enabled | This query retrieves both metric-specific and global annotations, providing a complete picture of infrastructure events. ### Track incidents and outages Mark incident start and resolution times: | Field | Value | | ----------------------- | ----------------- | | Name | Incidents | | OpenTSDB metrics query | `events.incident` | | Show Global Annotations | enabled | ### Monitor configuration changes Track when configuration changes are applied: | Field | Value | | ----------------------- | --------------------- | | Name | Configuration changes | | OpenTSDB metrics query | `events.config` | | Show Global Annotations | disabled | ### Correlate multiple event types You can add multiple annotation queries to a single dashboard to correlate different event types. For example: 1. Add a "Deployments" annotation query for `deploy.*` metrics. 1. Add an "Incidents" annotation query for `events.incident`. 1. Add a "Maintenance" annotation query with global annotations enabled. This allows you to see how deployments, incidents, and maintenance windows relate to your metric data. ## How annotations appear Annotations appear as vertical lines on time series panels at the timestamps where events occurred. Hover over an annotation marker to view: - The annotation name (from your query configuration) - The event description (from the OpenTSDB annotation's `description` field) - The timestamp Different annotation queries can be assigned different colors in the annotation settings to distinguish between event types. ## Create annotations in OpenTSDB To display annotations in Grafana, you must first create them in OpenTSDB. OpenTSDB provides an HTTP API for managing annotations. ### Annotation data structure OpenTSDB annotations have the following fields: | Field | Required | Description | | ------------- | -------- | ------------------------------------------------------------------------------------------ | | `startTime` | Yes | Unix epoch timestamp in seconds when the event started. | | `endTime` | No | Unix epoch timestamp in seconds when the event ended. Useful for duration-based events. | | `tsuid` | No | The time series UID to associate this annotation with. If empty, the annotation is global. | | `description` | No | Brief description of the event. This text displays in Grafana. | | `notes` | No | Detailed notes about the event. | | `custom` | No | A map of custom key-value pairs for additional metadata. | ### Create a global annotation Use the OpenTSDB API to create a global annotation: ```sh curl -X POST http://:4242/api/annotation \ -H "Content-Type: application/json" \ -d '{ "startTime": 1609459200, "description": "Production deployment v2.5.0", "notes": "Deployed new feature flags and performance improvements", "custom": { "version": "2.5.0", "environment": "production", "deployer": "jenkins" } }' ``` ### Create a metric-specific annotation To attach an annotation to a specific time series, include the `tsuid`: ```sh curl -X POST http://:4242/api/annotation \ -H "Content-Type: application/json" \ -d '{ "startTime": 1609459200, "endTime": 1609462800, "tsuid": "000001000001000001", "description": "Server maintenance", "notes": "Scheduled maintenance window for hardware upgrade" }' ``` To find the TSUID for a metric, use the OpenTSDB `/api/uid/tsmeta` endpoint. ### Create annotations programmatically Integrate annotation creation into your deployment pipelines or monitoring systems: **Deployment script example:** ```sh #!/bin/bash VERSION=$1 TIMESTAMP=$(date +%s) curl -X POST http://opentsdb.example.com:4242/api/annotation \ -H "Content-Type: application/json" \ -d "{ \"startTime\": $TIMESTAMP, \"description\": \"Deployed version $VERSION\", \"custom\": { \"version\": \"$VERSION\", \"environment\": \"production\" } }" ``` For more details on the annotation API, refer to the [OpenTSDB annotation API documentation](http://opentsdb.net/docs/build/html/api_http/annotation/index.html). ## Troubleshoot annotation issues The following section addresses common issues you may encounter when using OpenTSDB annotations. ### Annotations don't appear **Possible causes and solutions:** | Cause | Solution | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | Time range doesn't include annotations | Expand the dashboard time range to include the annotation timestamps. | | Wrong metric name | Verify the metric name in your annotation query matches the metric associated with the annotations in OpenTSDB. | | Annotations are global but toggle is off | Enable **Show Global Annotations** if your annotations don't have a TSUID. | | No annotations exist | Verify annotations exist in OpenTSDB using the API: `curl http://:4242/api/annotation?startTime=&endTime=` | ### Annotation text is empty The annotation displays but has no description text. **Solution:** Ensure the `description` field is populated when creating annotations in OpenTSDB. Grafana displays the `description` field as the annotation text. ## Next steps - [Build queries](https://grafana.com/docs/grafana//datasources/opentsdb/query-editor/) to visualize metrics alongside annotations. - [Use template variables](https://grafana.com/docs/grafana//datasources/opentsdb/template-variables/) to create dynamic dashboards. - [Set up alerting](https://grafana.com/docs/grafana//datasources/opentsdb/alerting/) to get notified when metrics cross thresholds. # Configure the OpenTSDB data source This document explains how to configure the OpenTSDB data source in Grafana. ## Before you begin Before configuring the OpenTSDB data source, ensure you have: - **Grafana permissions:** Organization administrator role to add data sources. - **OpenTSDB instance:** A running OpenTSDB server (version 2.1 or later recommended). - **Network access:** The Grafana server can reach the OpenTSDB HTTP API endpoint (default port 4242). - **Metrics in OpenTSDB:** For autocomplete to work, metrics must exist in your OpenTSDB database. ## Add the data source To add and configure the OpenTSDB data source: 1. Click **Connections** in the left-side menu. 1. Click **Add new connection**. 1. Type `OpenTSDB` in the search bar. 1. Select **OpenTSDB**. 1. Click **Add new data source** in the upper right. 1. Configure the data source settings as described in the following sections. ## Configuration options The following table describes the available configuration options: | Setting | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | **Name** | The data source name. This is how you refer to the data source in panels and queries. | | **Default** | Toggle to make this the default data source for new panels. | | **URL** | The HTTP protocol, IP address, and port of your OpenTSDB server. The default port is `4242`. Example: `http://localhost:4242`. | | **Allowed cookies** | Cookies to forward to the data source. Use this when your OpenTSDB server requires specific cookies for authentication. | | **Timeout** | HTTP request timeout in seconds. Increase this value for slow networks or complex queries. | ## Auth settings Configure authentication if your OpenTSDB server requires it: | Setting | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Basic auth** | Enable to authenticate with a username and password. When enabled, enter the username and password in the fields that appear. | | **With Credentials** | Enable to send cookies or auth headers with cross-site requests. Use this when OpenTSDB is on a different domain and requires credentials. | | **TLS Client Authentication** | Enable to use client certificates for authentication. Requires configuring client certificate and key. | | **Skip TLS Verify** | Enable to skip verification of the OpenTSDB server's TLS certificate. Only use this in development environments. | | **Forward OAuth Identity** | Enable to forward the user's OAuth token to the data source. Useful when OpenTSDB is behind an OAuth-protected proxy. | | **Custom HTTP Headers** | Add custom headers to all requests sent to OpenTSDB. Useful for API keys or custom authentication schemes. | ## OpenTSDB settings Configure these settings based on your OpenTSDB server version and configuration: | Setting | Description | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | **Version** | Select your OpenTSDB version. This affects available query features. Refer to the following section for version-specific features. | | **Resolution** | The resolution of your metric data. Select `second` for second-precision timestamps or `millisecond` for millisecond-precision timestamps. | | **Lookup limit** | Maximum number of results returned by suggest and lookup API calls. Default is `1000`. Increase this if you have many metrics or tag values. | ### Version-specific features The version setting enables different query features in Grafana: | Version | Available features | | --------- | ---------------------------------------------------------------------------------------------------------------------------- | | **<=2.1** | Basic queries with tags. Uses legacy tag-based filtering. | | **==2.2** | Adds filter support (`literal_or`, `wildcard`, `regexp`, and more). Filters replace tags for more flexible queries. | | **==2.3** | Adds explicit tags support for rate calculations and additional filter types. | | **==2.4** | Adds fill policy support for downsampling (`none`, `null`, `zero`, `nan`). Enables `arrays=true` for alerting compatibility. | Select the version that matches your OpenTSDB server. If you're unsure, check your OpenTSDB version with the `/api/version` endpoint. ## Secure SOCKS proxy The OpenTSDB data source supports connecting through a secure SOCKS5 proxy. Use this to reach an OpenTSDB server hosted in a different network than Grafana without exposing it to the public internet. You configure the proxy at the Grafana instance level. For setup instructions, refer to [Configure a data source SOCKS5 connection proxy](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/proxy/). ## Verify the connection Click **Save & test** to verify that Grafana can connect to your OpenTSDB server. A successful test confirms that the URL is correct and the server is responding. If the test fails, refer to [Troubleshooting](https://grafana.com/docs/grafana//datasources/opentsdb/troubleshooting/) for common issues and solutions. ## Provision the data source You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ### YAML example The following example provisions an OpenTSDB data source: ```yaml apiVersion: 1 datasources: - name: OpenTSDB type: opentsdb access: proxy url: http://localhost:4242 jsonData: # OpenTSDB version: 1 = <=2.1, 2 = 2.2, 3 = 2.3, 4 = 2.4 tsdbVersion: 3 # Resolution: 1 = second, 2 = millisecond tsdbResolution: 1 # Maximum results for suggest/lookup API calls lookupLimit: 1000 ``` ### YAML example with basic authentication The following example provisions an OpenTSDB data source with basic authentication: ```yaml apiVersion: 1 datasources: - name: OpenTSDB type: opentsdb access: proxy url: http://localhost:4242 basicAuth: true basicAuthUser: jsonData: tsdbVersion: 3 tsdbResolution: 1 lookupLimit: 1000 secureJsonData: basicAuthPassword: ``` ### YAML example with custom headers The following example provisions an OpenTSDB data source with custom HTTP headers: ```yaml apiVersion: 1 datasources: - name: OpenTSDB type: opentsdb access: proxy url: http://localhost:4242 jsonData: tsdbVersion: 3 tsdbResolution: 1 lookupLimit: 1000 httpHeaderName1: X-Custom-Header secureJsonData: httpHeaderValue1: ``` The following table describes the available fields: | Field | Type | Description | | ---------------------------------- | ------- | ---------------------------------------------------------------------------- | | `basicAuth` | boolean | Enable basic authentication. | | `basicAuthUser` | string | Username for basic authentication. | | `jsonData.tsdbVersion` | number | OpenTSDB version: `1` (<=2.1), `2` (2.2), `3` (2.3), `4` (2.4). | | `jsonData.tsdbResolution` | number | Timestamp resolution: `1` (second), `2` (millisecond). | | `jsonData.lookupLimit` | number | Maximum results for suggest and lookup API calls. Default: `1000`. | | `jsonData.httpHeaderName1` | string | Name of a custom HTTP header. Use incrementing numbers for multiple headers. | | `secureJsonData.basicAuthPassword` | string | Password for basic authentication. | | `secureJsonData.httpHeaderValue1` | string | Value for the custom HTTP header. | ## Provision with Terraform You can provision the OpenTSDB data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation. ### Terraform example The following example provisions an OpenTSDB data source: ```hcl terraform { required_providers { grafana = { source = "grafana/grafana" version = ">= 2.0.0" } } } provider "grafana" { url = "" auth = "" } resource "grafana_data_source" "opentsdb" { type = "opentsdb" name = "OpenTSDB" url = "http://localhost:4242" json_data_encoded = jsonencode({ # OpenTSDB version: 1 = <=2.1, 2 = 2.2, 3 = 2.3, 4 = 2.4 tsdbVersion = 3 # Resolution: 1 = second, 2 = millisecond tsdbResolution = 1 # Maximum results for suggest/lookup API calls lookupLimit = 1000 }) } ``` ### Terraform example with basic authentication The following example provisions an OpenTSDB data source with basic authentication: ```hcl resource "grafana_data_source" "opentsdb_auth" { type = "opentsdb" name = "OpenTSDB" url = "http://localhost:4242" basic_auth_enabled = true basic_auth_username = "" json_data_encoded = jsonencode({ tsdbVersion = 3 tsdbResolution = 1 lookupLimit = 1000 }) secure_json_data_encoded = jsonencode({ basicAuthPassword = "" }) } ``` Replace the following placeholders: - _``_: Your Grafana instance URL (for example, `https://your-org.grafana.net` for Grafana Cloud) - _``_: A service account token with data source permissions - _``_: The username for basic authentication - _``_: The password for basic authentication ## Next steps Now that you've configured OpenTSDB, you can: - [Query OpenTSDB data](https://grafana.com/docs/grafana//datasources/opentsdb/query-editor/) to build dashboards and visualizations - [Use template variables](https://grafana.com/docs/grafana//datasources/opentsdb/template-variables/) to create dynamic, reusable dashboards - [Add annotations](https://grafana.com/docs/grafana//datasources/opentsdb/annotations/) to overlay events on your graphs - [Set up alerting](https://grafana.com/docs/grafana//datasources/opentsdb/alerting/) to get notified when metrics cross thresholds - [Troubleshoot issues](https://grafana.com/docs/grafana//datasources/opentsdb/troubleshooting/) if you encounter problems # OpenTSDB query editor The query editor allows you to build OpenTSDB queries visually. The available options depend on the OpenTSDB version you configured for the data source. ## Access the query editor The OpenTSDB query editor is located on the [Explore](https://grafana.com/docs/grafana//explore/) page. You can also access the OpenTSDB query editor from a dashboard panel. Click the ellipsis in the upper right of the panel and select **Edit**. ## Create a query To create a query: 1. Select the **OpenTSDB** data source in a panel. 1. Configure the query using the sections described in the following documentation. ## Metric section The Metric section contains the core query configuration: | Field | Description | | -------------- | ------------------------------------------------------------------------------------------------- | | **Metric** | The metric name to query. Start typing to see autocomplete suggestions from your OpenTSDB server. | | **Aggregator** | The aggregation function to combine multiple time series. Default: `sum`. | | **Alias** | Custom display name for the series. Use `$tag_` to include tag values in the alias. | ### Alias patterns The alias field supports dynamic substitution using tag values. Use the pattern `$tag_` where `` is the name of a tag on your metric. | Pattern | Description | Example output | | ---------------------- | ----------------------------------- | -------------------------- | | `$tag_host` | Inserts the value of the `host` tag | `webserver01` | | `$tag_env` | Inserts the value of the `env` tag | `production` | | `$tag_host - CPU` | Combines tag value with static text | `webserver01 - CPU` | | `$tag_host ($tag_env)` | Multiple tag substitutions | `webserver01 (production)` | ## Downsample section Downsampling reduces the number of data points returned by aggregating values over time intervals. This improves query performance and reduces the amount of data transferred. | Field | Description | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Down sample** | The time interval for downsampling. Enter a value in the input next to the **Down sample** label, or leave it blank to use the automatic interval based on the panel's time range and width. | | **Aggregator** | The aggregation function for downsampling. Default: `avg`. | | **Fill** | (Version 2.2+) The fill policy for missing data points. Default: `none`. | | **Disable downsampling** | Toggle to disable downsampling entirely. Use this when you need raw data points. | ### Interval format The interval field accepts time duration strings: | Format | Description | Example | | ------ | ----------- | ------- | | `s` | Seconds | `30s` | | `m` | Minutes | `5m` | | `h` | Hours | `1h` | | `d` | Days | `1d` | | `w` | Weeks | `1w` | When the interval is left blank, Grafana automatically calculates an appropriate interval based on the panel's time range and pixel width. This ensures optimal data density for visualization. ## Filters section Filters (available in OpenTSDB 2.2+) provide advanced filtering capabilities that replace the legacy tag-based filtering. | Field | Description | | ------------ | --------------------------------------------------------------------------------------------------------------- | | **Key** | The tag key to filter on. Select from autocomplete suggestions or type a custom value. | | **Type** | The filter type. Determines how the filter value is matched. Default: `iliteral_or`. | | **Filter** | The filter value or pattern. Supports autocomplete for tag values. | | **Group by** | Toggle to group results by this tag key. When enabled, separate time series are returned for each unique value. | ### Add, edit, and remove filters To manage filters: 1. Click the **+** button next to "Filters" to add a new filter. 1. Configure the filter fields (Key, Type, Filter, Group by). 1. Click **add filter** to apply the filter. 1. To edit an existing filter, click the **pencil** icon next to it. 1. To remove a filter, click the **x** icon next to it. You can add multiple filters to a single query. All filters are combined with AND logic. ### Filter types | Type | Description | Example | | ----------------- | ---------------------------------------------------------- | --------------------- | | `literal_or` | Matches exact values. Use `\|` to specify multiple values. | `web01\|web02\|web03` | | `iliteral_or` | Case-insensitive literal match. | `WEB01\|web02` | | `wildcard` | Matches using `*` as a wildcard character. | `web-*-prod` | | `iwildcard` | Case-insensitive wildcard match. | `WEB-*` | | `regexp` | Matches using regular expressions. | `web-[0-9]+` | | `not_literal_or` | Excludes exact values. | `web01\|web02` | | `not_iliteral_or` | Case-insensitive exclusion. | `TEST\|DEV` | ### Group by behavior When **Group by** is enabled for a filter: - Results are split into separate time series for each unique value of the filtered tag. - Each time series is labeled with its tag value. - This is useful for comparing values across hosts, environments, or other dimensions. When **Group by** is disabled: - All matching time series are combined using the selected aggregator. - A single aggregated time series is returned. ## Tags section Tags filter metrics by key-value pairs. This is the legacy filtering method for OpenTSDB versions prior to 2.2. | Field | Description | | --------- | ----------------------------------------------------------------- | | **Key** | The tag key to filter on. Select from autocomplete suggestions. | | **Value** | The tag value to match. Use `*` to match all values for this key. | ### Add, edit, and remove tags To manage tags: 1. Click the **+** button next to "Tags" to add a new tag. 1. Select or type a tag key. 1. Select or type a tag value (use `*` for wildcard). 1. Click **add tag** to apply the tag filter. 1. To edit an existing tag, click the **pencil** icon next to it. 1. To remove a tag, click the **x** icon next to it. {{< admonition type="note" >}} Tags are deprecated in OpenTSDB 2.2 and later. Use Filters instead for more powerful filtering options including wildcards, regular expressions, and exclusion patterns. {{< /admonition >}} {{< admonition type="caution" >}} Tags and Filters are mutually exclusive. If you have filters defined, you cannot add tags, and vice versa. The query editor displays a warning if you attempt to use both. {{< /admonition >}} ## Rate section The Rate section computes the rate of change, which is essential for counter metrics that continuously increment. | Field | Description | | ----------------- | ---------------------------------------------------------------------------------------------------------- | | **Rate** | Toggle to enable rate calculation. Computes the per-second rate of change between consecutive values. | | **Counter** | (When Rate is enabled) Toggle to indicate the metric is a monotonically increasing counter that may reset. | | **Counter max** | (When Counter is enabled) The maximum value before the counter wraps around. | | **Reset value** | (When Counter is enabled) The value the counter resets to after wrapping. Default: `0`. | | **Explicit tags** | (Version 2.3+) Toggle to require all specified tags to exist on matching time series. | ### When to use rate calculation Enable **Rate** when your metric is a continuously increasing counter, such as: - Network bytes sent/received - Request counts - Error counts - Disk I/O operations The rate calculation converts cumulative values into per-second rates, making the data more meaningful for visualization. ### Counter settings Enable **Counter** when your metric can reset to zero (for example, after a service restart). The counter settings help OpenTSDB calculate correct rates across resets: - **Counter max**: Set this to the maximum value your counter can reach before wrapping. For 64-bit counters, use `18446744073709551615`. For 32-bit counters, use `4294967295`. - **Reset value**: The value the counter resets to, typically `0`. ### Explicit tags When **Explicit tags** is enabled (version 2.3+), OpenTSDB only returns time series that have all the tags specified in your query. This prevents unexpected results when some time series are missing tags that others have. ## Aggregators The aggregator function combines multiple time series into one. The data source includes a built-in default list (`avg`, `sum`, `min`, `max`, `dev`, `zimsum`, `mimmin`, and `mimmax`), then replaces it with the list fetched from your OpenTSDB server. As a result, you may see additional aggregators, such as `count`, beyond the built-in defaults. ### Common aggregators | Aggregator | Description | Use case | | ---------- | ----------------------------------------- | -------------------------------------- | | `sum` | Sum all values at each timestamp. | Total requests across all servers. | | `avg` | Average all values at each timestamp. | Average CPU usage across hosts. | | `min` | Take the minimum value at each timestamp. | Lowest response time. | | `max` | Take the maximum value at each timestamp. | Peak memory usage. | | `dev` | Calculate the standard deviation. | Measure variability in response times. | | `count` | Count the number of data points. | Number of reporting hosts. | ### Interpolation aggregators These aggregators handle missing data points differently: | Aggregator | Description | | ---------- | ---------------------------------------------------- | | `zimsum` | Sum values, treating missing data as zero. | | `mimmin` | Minimum value, ignoring missing (interpolated) data. | | `mimmax` | Maximum value, ignoring missing (interpolated) data. | {{< admonition type="note" >}} The available aggregators depend on your OpenTSDB server version and configuration. The aggregator dropdown is populated dynamically from the `/api/aggregators` endpoint on your OpenTSDB server. {{< /admonition >}} ## Fill policies Fill policies (available in OpenTSDB 2.2+) determine how to handle missing data points during downsampling. This is important when your data has gaps or irregular collection intervals. | Policy | Description | Use case | | ------ | --------------------------------------------------- | ----------------------------------------------------------- | | `none` | Don't fill missing values. Gaps remain in the data. | Default behavior; preserves data fidelity. | | `nan` | Fill missing values with NaN (Not a Number). | Useful for calculations that should propagate missing data. | | `null` | Fill missing values with null. | Visualizations show gaps at null points. | | `zero` | Fill missing values with zero. | Treat missing data as zero values; useful for counters. | ### Choose the right fill policy - Use `none` (default) when you want to see actual data gaps in your visualizations. - Use `null` when you want graphs to show breaks at missing data points. - Use `zero` when missing data should be interpreted as zero (for example, no requests during a period). - Use `nan` when you need missing values to propagate through calculations. ## Autocomplete suggestions The query editor provides autocomplete suggestions to help you build queries quickly and accurately. ### What autocomplete provides | Field | Source | Description | | --------------- | --------------------------- | ----------------------------------------------- | | **Metric** | `/api/suggest?type=metrics` | Suggests metric names as you type. | | **Tag keys** | Previous query results | Suggests tag keys based on the selected metric. | | **Tag values** | `/api/suggest?type=tagv` | Suggests tag values as you type. | | **Filter keys** | Previous query results | Suggests tag keys for filter configuration. | ### Autocomplete requirements For autocomplete to work: - The OpenTSDB suggest API must be enabled on your server. - Metrics must exist in your OpenTSDB database. - The **Lookup limit** setting in your data source configuration controls the maximum number of suggestions returned. If autocomplete isn't working, refer to [Troubleshooting](https://grafana.com/docs/grafana//datasources/opentsdb/troubleshooting/). ## Use template variables You can use template variables in any text field in the query editor. Template variables are replaced with their current values when the query executes. Common uses include: - **Metric field**: `$metric` to dynamically select metrics. - **Filter values**: `$host` to filter by a variable-selected host. - **Tag values**: `$environment` to filter by environment. For more information about creating and using template variables, refer to [Template variables](https://grafana.com/docs/grafana//datasources/opentsdb/template-variables/). ## Query examples The following examples demonstrate common query patterns. ### Basic metric query with tag filtering | Field | Value | | ---------- | ------------------ | | Metric | `sys.cpu.user` | | Aggregator | `avg` | | Tags | `host=webserver01` | This query returns the average CPU usage for the host `webserver01`. ### Query with wildcard filter (OpenTSDB 2.2+) | Field | Value | | ------------ | --------------------- | | Metric | `http.requests.count` | | Aggregator | `sum` | | Filter Key | `host` | | Filter Type | `wildcard` | | Filter Value | `web-*` | | Group by | enabled | This query sums HTTP request counts across all hosts matching `web-*` and groups results by host. ### Rate calculation for network counters | Field | Value | | ----------- | ---------------------- | | Metric | `net.bytes.received` | | Aggregator | `sum` | | Rate | enabled | | Counter | enabled | | Counter max | `18446744073709551615` | This query calculates the rate of bytes received per second. The counter max is set to the 64-bit unsigned integer maximum to handle counter wraps correctly. ### Use alias patterns | Field | Value | | ---------- | --------------------------- | | Metric | `app.response.time` | | Aggregator | `avg` | | Tags | `host=*`, `env=production` | | Alias | `$tag_host - Response Time` | This query uses the alias pattern to create readable legend labels like `webserver01 - Response Time`. ### Downsample with a custom interval | Field | Value | | --------------------- | ------------------- | | Metric | `sys.disk.io.bytes` | | Aggregator | `sum` | | Downsample Interval | `5m` | | Downsample Aggregator | `avg` | | Fill | `zero` | This query reduces disk I/O data to 5-minute averages, filling gaps with zero values. ### Compare environments with filters | Field | Value | | ------------ | --------------------- | | Metric | `app.errors.count` | | Aggregator | `sum` | | Filter Key | `env` | | Filter Type | `literal_or` | | Filter Value | `staging\|production` | | Group by | enabled | This query shows error counts for both staging and production environments as separate time series for comparison. ### Exclude specific hosts | Field | Value | | ------------ | ------------------------- | | Metric | `sys.cpu.user` | | Aggregator | `avg` | | Filter Key | `host` | | Filter Type | `not_literal_or` | | Filter Value | `test-server\|dev-server` | | Group by | enabled | This query shows CPU usage for all hosts except test-server and dev-server. ### Query with explicit tags (version 2.3+) | Field | Value | | ------------- | --------------------- | | Metric | `app.request.latency` | | Aggregator | `avg` | | Filter Key | `host` | | Filter Type | `wildcard` | | Filter Value | `*` | | Group by | enabled | | Explicit tags | enabled | This query only returns time series that have the `host` tag defined, excluding any time series that are missing this tag. ## Next steps - [Use template variables](https://grafana.com/docs/grafana//datasources/opentsdb/template-variables/) to create dynamic, reusable dashboards. - [Set up alerting](https://grafana.com/docs/grafana//datasources/opentsdb/alerting/) to get notified when metrics cross thresholds. - [Troubleshoot issues](https://grafana.com/docs/grafana//datasources/opentsdb/troubleshooting/) if you encounter problems with queries. # OpenTSDB template variables Instead of hard-coding server, application, and sensor names in your metric queries, you can use template variables. Variables appear as drop-down menus at the top of the dashboard, so you can change the data being displayed without editing queries. For an introduction to template variables, refer to the [Variables](https://grafana.com/docs/grafana//dashboards/variables/) documentation. ## Query variable The OpenTSDB data source supports query-type template variables that fetch values directly from OpenTSDB. These variables dynamically populate based on data in your OpenTSDB database. ### Supported query functions | Query | Description | API used | | ---------------------------- | -------------------------------------------------------------------------------- | --------------------------- | | `metrics(prefix)` | Returns metric names matching the prefix. Use empty parentheses for all metrics. | `/api/suggest?type=metrics` | | `tag_names(metric)` | Returns tag keys (names) that exist for a specific metric. | `/api/search/lookup` | | `tag_values(metric, tagkey)` | Returns tag values for a specific metric and tag key. | `/api/search/lookup` | | `suggest_tagk(prefix)` | Returns tag keys matching the prefix across all metrics. | `/api/suggest?type=tagk` | | `suggest_tagv(prefix)` | Returns tag values matching the prefix across all metrics. | `/api/suggest?type=tagv` | {{< admonition type="note" >}} The `tag_names` and `tag_values` functions use the OpenTSDB lookup API, which requires metrics to exist in your database. The `suggest_tagk` and `suggest_tagv` functions use the suggest API, which searches across all metrics. {{< /admonition >}} ### Create a query variable To create a query variable: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Variable**. 1. Select **Query** as the variable type. 1. Enter a **Name** for your variable (for example, `host`). 1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed. 1. Click **Open variable editor** to open the **Query Variable** dialog box. 1. Select the **OpenTSDB** data source. 1. Enter your query using one of the supported query functions. 1. Click **Close** when you've completed the query setup. 1. (Optional) Configure **Multi-value** to allow selecting multiple values. 1. (Optional) Configure **Include All value** to add an "All" option. 1. (Optional) Turn on the **Allow custom values** switch to allow users to enter custom variable values. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. ### Query variable examples **List all metrics:** ``` metrics() ``` Returns all metric names in your OpenTSDB database. Useful for creating a metric selector. **List metrics with a prefix:** ``` metrics(sys.cpu) ``` Returns metrics starting with `sys.cpu`, such as `sys.cpu.user`, `sys.cpu.system`, `sys.cpu.idle`. **List tag keys for a metric:** ``` tag_names(sys.cpu.user) ``` Returns tag keys like `host`, `env`, `datacenter` that exist on the `sys.cpu.user` metric. **List tag values for a metric and tag key:** ``` tag_values(sys.cpu.user, host) ``` Returns all host values for the `sys.cpu.user` metric, such as `webserver01`, `webserver02`, `dbserver01`. **Search for tag keys by prefix:** ``` suggest_tagk(host) ``` Returns tag keys matching `host` across all metrics, such as `host`, `hostname`, `host_id`. **Search for tag values by prefix:** ``` suggest_tagv(web) ``` Returns tag values matching `web` across all metrics, such as `webserver01`, `webserver02`, `web-prod-01`. If template variables aren't populating in the **Preview of values** section, refer to [Troubleshooting](https://grafana.com/docs/grafana//datasources/opentsdb/troubleshooting/). ## Nested template variables You can use one template variable to filter values for another. This creates cascading filters, such as selecting a data center first, then showing only hosts in that data center. ### Filter syntax The `tag_values` function accepts additional tag filters after the tag key: ``` tag_values(metric, tagkey, tag1=value1, tag2=value2, ...) ``` Use template variables as filter values to create dynamic dependencies: ``` tag_values(metric, tagkey, tag1=$variable1, tag2=$variable2) ``` ### Nested variable examples | Query | Description | | ---------------------------------------------------------- | ------------------------------------------------------------ | | `tag_values(sys.cpu.user, host, env=$env)` | Returns host values filtered by the selected `env` value. | | `tag_values(sys.cpu.user, host, env=$env, datacenter=$dc)` | Returns host values filtered by both `env` and `datacenter`. | | `tag_values(app.requests, endpoint, service=$service)` | Returns endpoint values for the selected service. | ### Create cascading filters To create a hierarchy of dependent variables: 1. **Create the parent variable:** - Name: `datacenter` - Query: `tag_values(sys.cpu.user, datacenter)` 2. **Create the child variable:** - Name: `host` - Query: `tag_values(sys.cpu.user, host, datacenter=$datacenter)` 3. **Create additional levels as needed:** - Name: `cpu` - Query: `tag_values(sys.cpu.user, cpu, datacenter=$datacenter, host=$host)` When users change the data center selection, the host variable automatically refreshes to show only hosts in that data center. ## Use variables in queries Reference variables in your queries using the `$variablename` or `${variablename}` syntax. Grafana replaces the variable with its current value when the query executes. ### Where to use variables Variables can be used in these query editor fields: | Field | Example | Description | | ----------------------- | ------------------- | ------------------------------------------- | | **Metric** | `$metric` | Dynamically select which metric to query. | | **Tag value** | `host=$host` | Filter by a variable-selected tag value. | | **Filter value** | `$host` | Use in filter value field for filtering. | | **Alias** | `$tag_host - $host` | Include variable values in legend labels. | | **Downsample interval** | `$interval` | Use a variable for the downsample interval. | ### Variable syntax options | Syntax | Description | | ------------------------ | -------------------------------------------------------------------------------- | | `$variablename` | Simple syntax for most cases. | | `${variablename}` | Use when the variable is adjacent to other text (for example, `${host}_suffix`). | | `${variablename:format}` | Apply a specific format to the variable value. | ## Multi-value variables When you enable **Multi-value** for a variable, users can select multiple values simultaneously. The OpenTSDB data source handles multi-value variables using pipe (`|`) separation, which is compatible with OpenTSDB's `literal_or` filter type. ### Configure multi-value variables 1. When creating the variable, enable **Multi-value**. 1. Optionally enable **Include All option** to add an "All" selection. 1. Use the variable in a filter with the `literal_or` filter type. ### Multi-value example With a `host` variable configured as multi-value: | Field | Value | | ------------ | ------------ | | Filter Key | `host` | | Filter Type | `literal_or` | | Filter Value | `$host` | If the user selects `webserver01`, `webserver02`, and `webserver03`, the filter value becomes `webserver01|webserver02|webserver03`. ### All value behavior When the user selects "All", Grafana sends all available values pipe-separated. For large value sets, consider using a wildcard filter instead: | Field | Value | | ------------ | ---------- | | Filter Key | `host` | | Filter Type | `wildcard` | | Filter Value | `*` | ## Interval and auto-interval variables Grafana provides built-in interval variables that are useful with OpenTSDB downsampling: | Variable | Description | | ---------------- | ---------------------------------------------------------------------- | | `$__interval` | Automatically calculated interval based on time range and panel width. | | `$__interval_ms` | Same as `$__interval` but in milliseconds. | Use these in the downsample interval field for automatic interval adjustment: | Field | Value | | ------------------- | ------------- | | Downsample Interval | `$__interval` | ## Next steps - [Build queries](https://grafana.com/docs/grafana//datasources/opentsdb/query-editor/) using your template variables. - [Set up alerting](https://grafana.com/docs/grafana//datasources/opentsdb/alerting/) with templated queries. - [Troubleshoot issues](https://grafana.com/docs/grafana//datasources/opentsdb/troubleshooting/) if variables aren't populating. # Troubleshoot OpenTSDB data source issues This document provides solutions to common issues you may encounter when configuring or using the OpenTSDB data source. For configuration instructions, refer to [Configure the OpenTSDB data source](https://grafana.com/docs/grafana//datasources/opentsdb/configure/). ## Connection errors These errors occur when Grafana can't connect to the OpenTSDB server. ### "Connection refused" or timeout errors **Symptoms:** - Save & test fails - Queries return connection errors - Intermittent timeouts **Possible causes and solutions:** | Cause | Solution | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------- | | Wrong URL or port | Verify the URL includes the correct protocol, IP address, and port. The default port is `4242`. | | OpenTSDB not running | Check that the OpenTSDB server is running and accessible. | | Firewall blocking connection | Ensure firewall rules allow outbound connections from Grafana to the OpenTSDB server on the configured port. | | Network issues | Verify network connectivity between Grafana and OpenTSDB. Try pinging the server or using `curl` to test the API. | To test connectivity manually, run: ```sh curl http://:4242/api/version ``` ## Authentication errors These errors occur when credentials are invalid or misconfigured. ### "401 Unauthorized" or "403 Forbidden" **Symptoms:** - Save & test fails with authentication error - Queries return authorization errors **Solutions:** 1. Verify that basic authentication credentials are correct in the data source configuration. 1. Check that the OpenTSDB server is configured to accept the provided credentials. 1. If using cookies for authentication, ensure the required cookies are listed in **Allowed cookies**. ## Query errors These errors occur when executing queries against OpenTSDB. ### No data returned **Symptoms:** - Query executes without error but returns no data - Panels show "No data" message **Possible causes and solutions:** | Cause | Solution | | ------------------------------- | ------------------------------------------------------------------------------------------------------- | | Time range doesn't contain data | Expand the dashboard time range. Verify data exists in OpenTSDB for the selected period. | | Wrong metric name | Verify the metric name is correct. Use autocomplete to discover available metrics. | | Incorrect tag filters | Remove or adjust tag filters. Use `*` as a wildcard to match all values. | | Version mismatch | Ensure the configured OpenTSDB version matches your server. Filters are only available in version 2.2+. | | Using both Filters and Tags | Use either Filters or Tags, not both. They're mutually exclusive in OpenTSDB 2.2+. | ### Query timeout **Symptoms:** - Queries take a long time and then fail - Error message mentions timeout **Solutions:** 1. Reduce the time range of your query. 1. Add more specific tag filters to reduce the data volume. 1. Increase the **Timeout** setting in the data source configuration. 1. Enable downsampling to reduce the number of data points returned. 1. Check OpenTSDB server performance and HBase health. ## Autocomplete doesn't work **Symptoms:** - No suggestions appear when typing metric names, tag names, or tag values - Drop-down menus are empty **Solutions:** 1. Verify that the OpenTSDB `/api/suggest` endpoint is accessible. Test it manually with `curl http://:4242/api/suggest?type=metrics`. 1. Increase the **Lookup limit** setting if you have many metrics or tags. 1. Verify that the data source connection is working by clicking **Save & test**. 1. Check that metrics exist in OpenTSDB. The suggest API only returns metrics that have been written to the database. ## Template variables don't populate **Symptoms:** - Template variable drop-down menus are empty - **Preview of values** shows no results **Solutions:** 1. Enable real-time metadata tracking in OpenTSDB by setting `tsd.core.meta.enable_realtime_ts` to `true` in your OpenTSDB configuration. 1. Sync existing metadata by running `tsdb uid metasync` on the OpenTSDB server. 1. Verify the variable query syntax is correct. Refer to [Template variables](https://grafana.com/docs/grafana//datasources/opentsdb/template-variables/) for the correct syntax. 1. Check that the data source connection is working. ## Performance issues These issues relate to slow queries or high resource usage. ### Slow queries **Symptoms:** - Dashboards take a long time to load - Queries are slow even for small time ranges **Solutions:** 1. Enable downsampling in the query editor to reduce data volume. 1. Use more specific tag filters to limit the time series returned. 1. Reduce the time range. 1. Check OpenTSDB and HBase performance metrics. 1. Consider increasing OpenTSDB heap size if memory is constrained. ### HBase performance issues OpenTSDB relies on HBase for data storage. Performance problems in HBase directly affect OpenTSDB query performance. **Solutions:** 1. Monitor HBase region server health and compaction status. 1. Ensure sufficient heap memory is allocated to HBase region servers. 1. Check for region hot spots and redistribute regions if necessary. 1. Refer to the [OpenTSDB troubleshooting guide](http://opentsdb.net/docs/build/html/user_guide/troubleshooting.html) for HBase-specific issues. ## Enable debug logging To capture detailed error information for troubleshooting: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location). 1. Look for OpenTSDB-specific entries that include request and response details. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you've tried the solutions in this document and still encounter issues: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review [OpenTSDB issues on GitHub](https://github.com/grafana/grafana/issues?q=opentsdb) for known bugs. 1. Consult the [OpenTSDB documentation](http://opentsdb.net/docs/build/html/index.html) for server-specific guidance. 1. Contact Grafana Support if you're a Grafana Enterprise, Cloud Pro, or Cloud Contracted user. When reporting issues, include: - Grafana version - OpenTSDB version - Error messages (redact sensitive information) - Steps to reproduce - Data source configuration (redact credentials) # Configure the Parca data source {{< admonition type="warning" >}} This plugin is deprecated and will only receive critical security updates. Support will end on January 2, 2027. {{< /admonition >}} This document explains how to configure the Parca data source in Grafana. You can configure the data source using the Grafana UI, a YAML provisioning file, or Terraform. If you make any changes in the UI, select **Save & test** to preserve those changes. ## Before you begin Before configuring the data source, ensure you have: - **Grafana permissions:** `Organization administrator` role. - **Parca instance:** A running Parca instance (v0.19 or later) accessible from your Grafana server. If you're provisioning the data source, you also need administrative rights on the server hosting your Grafana instance. ## Add the data source To add the Parca data source: 1. Click **Connections** in the left-side menu. 1. Click **Add new connection**. 1. Type `Parca` in the search bar. 1. Select **Parca**. 1. Click **Add new data source**. ## Configure settings The **Settings** tab contains the following configuration sections. ### Basic settings | Setting | Description | | ----------- | -------------------------------------------------------------------- | | **Name** | A name to identify this data source in panels, queries, and Explore. | | **Default** | Toggle to make this the default data source for new panels. | ### Connection | Setting | Description | | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **URL** | The URL of your Parca instance. For example, `http://localhost:7070`. Grafana connects to Parca using gRPC-Web, so the URL should point to the Parca HTTP endpoint. | ### Authentication Use this section to select an authentication method to access the data source. The available methods are: | Method | Description | | -------------------------- | ------------------------------------------------------------------------------------------------ | | **Basic authentication** | Authenticate with a username and password. Enter values in the **User** and **Password** fields. | | **Forward OAuth Identity** | Forward the logged-in user's OAuth token to the data source. | | **No Authentication** | Connect without credentials. Use this when the Parca instance doesn't require authentication. | ### TLS settings Configure TLS for secure communication with your Parca instance. | Setting | Description | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | **Add self-signed certificate** | Toggle to provide a CA certificate. When enabled, paste the CA certificate in the **CA Certificate** field. | | **TLS Client Authentication** | Toggle to enable mutual TLS. When enabled, provide the **Server Name**, **Client Certificate**, and **Client Key**. | | **Skip TLS certificate validation** | Toggle to skip server certificate verification. Use only for testing. | {{< admonition type="note" >}} Use Transport Layer Security (TLS) for an additional layer of security when working with Parca. {{< /admonition >}} ### Custom HTTP headers Add custom HTTP headers to requests sent to the Parca instance. Click **Add header** to add a new row with **Header** (name) and **Value** (secret) fields. Custom headers are useful for passing authentication tokens or routing information required by proxies between Grafana and Parca. ## Additional settings Click the down arrow to expand the **Additional settings** section. These settings are optional. ### Advanced HTTP settings | Setting | Description | | ------------------- | ----------------------------------------------------------------------------------------------------- | | **Allowed cookies** | Cookies that should be forwarded to the data source. The Grafana proxy strips all cookies by default. | | **Timeout** | The HTTP request timeout in seconds. | ### Secure SOCKS proxy {{< admonition type="note" >}} This section is only visible when the Grafana server has the secure SOCKS proxy feature enabled. {{< /admonition >}} | Setting | Description | | ----------- | ---------------------------------------------------------------------------- | | **Enabled** | Toggle to route requests to the Parca instance through a secure SOCKS proxy. | ## Verify the connection Click **Save & test** to verify that Grafana can connect to your Parca instance. The health check queries the available profile types from the Parca server. A successful connection displays the message **Data source is working**. If the test fails, verify that the URL is correct and that your Parca instance is running and accessible from the Grafana server. For more help, refer to [Troubleshoot Parca data source issues](https://grafana.com/docs/grafana//datasources/parca/troubleshooting/). ## Provision the data source You can define the data source in YAML files as part of the Grafana provisioning system. For more information, refer to [Provisioning Grafana](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ### YAML provisioning example ```yaml apiVersion: 1 datasources: - name: Parca type: parca url: http://localhost:7070 ``` To provision with basic authentication: ```yaml apiVersion: 1 datasources: - name: Parca type: parca url: http://localhost:7070 basicAuth: true basicAuthUser: secureJsonData: basicAuthPassword: ``` Replace the following: - ``: Your Parca username. - ``: Your Parca password. ### Terraform example To provision the data source with Terraform, use the [`grafana_data_source` resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source): ```hcl resource "grafana_data_source" "parca" { type = "parca" name = "Parca" url = "http://localhost:7070" } ``` To provision with basic authentication: ```hcl resource "grafana_data_source" "parca" { type = "parca" name = "Parca" url = "http://localhost:7070" basic_auth_enabled = true basic_auth_username = "" secure_json_data_encoded = jsonencode({ basicAuthPassword = "" }) } ``` Replace the following: - ``: Your Parca username. - ``: Your Parca password. ## Next steps - [Query profiling data with the Parca query editor](https://grafana.com/docs/grafana//datasources/parca/query-editor/) # Parca query editor {{< admonition type="warning" >}} This plugin is deprecated and will only receive critical security updates. Support will end on January 2, 2027. {{< /admonition >}} This document explains how to use the Parca query editor to query and visualize continuous profiling data. The query editor gives you access to a profile type selector, a label selector with autocomplete, and collapsible options for controlling query behavior. ## Before you begin Before using the query editor, ensure you have: - [Configured the Parca data source](https://grafana.com/docs/grafana//datasources/parca/configure/). - Verified the connection using **Save & test**. ## Access the query editor To access the query editor: 1. Sign in to Grafana. 1. Select **Explore** from the left-side menu. 1. Select your Parca data source from the data source drop-down. You can also access the query editor from a dashboard panel by selecting the Parca data source when editing a panel. ## Profile type selector Select a profile type from the cascading drop-down menu. Profile types are grouped by name at the first level, with a second level showing the sample type. For example, a `process_cpu` profile might show sample types like `cpu` or `samples`. You must select a profile type for the query to be valid. Grafana doesn't return any data if the profile type isn't selected when a query runs. ### Profile type ID format Each profile type has an ID in the format `name:sampleType:sampleUnit:periodType:periodUnit`. Delta profile types include an additional `:delta` suffix. You don't need to construct these IDs manually -- the cascading drop-down handles selection for you. Examples of profile type IDs: | Profile type ID | Description | | --------------------------------------------------- | ------------------------------------------- | | `process_cpu:samples:count:cpu:nanoseconds` | CPU profile measuring sample counts. | | `memory:alloc_objects:count:space:bytes` | Memory profile measuring allocated objects. | | `process_cpu:cpu:nanoseconds:cpu:nanoseconds:delta` | Delta CPU profile measuring nanoseconds. | ## Label selector Use the label selector to filter profiles by labels. The editor provides autocomplete for both label names and label values. ### Syntax Parca uses a syntax similar to Prometheus for label filtering. Wrap label matchers in curly braces using the `{key="value"}` format. The following operators are supported: | Operator | Description | Example | | -------- | --------------------------------- | ----------------------- | | `=` | Equals | `{job="my-service"}` | | `!=` | Not equals | `{job!="test-service"}` | | `=~` | Regular expression match | `{job=~"prod-.*"}` | | `!~` | Negative regular expression match | `{job!~"test-.*"}` | You can combine multiple label matchers separated by commas: ```text {job="my-service", instance=~"10.0.0.*"} ``` The label selector can be left empty to query all profiles of the selected type without filtering. Refer to the [Parca documentation](https://www.parca.dev/docs) for additional syntax details. ### Autocomplete The label selector provides autocomplete suggestions as you type. Autocomplete is triggered by the following characters: `{`, `,`, `[`, `(`, `=`, `~`, space, and `"`. - **Label names:** Suggested when you start typing inside curly braces or after a comma. - **Label values:** Suggested after typing a label name followed by `=` or `="`. ### Keyboard shortcuts | Shortcut | Action | | ----------------- | --------------------------------- | | **Shift + Enter** | Run the query with current input. | ## Query types Expand the **Options** section to select a query type. The query type controls what data is returned and how it's visualized. | Query type | Description | | ----------- | ---------------------------------------------------------------- | | **Metric** | Returns aggregated metric data visualized as a time series. | | **Profile** | Returns a merged profile visualized as a flame graph. | | **Both** | Returns both metric and profile data. Only available in Explore. | {{< admonition type="note" >}} The **Both** option is only available in [Explore](https://grafana.com/docs/grafana//explore/). In dashboard panels, you can select either **Metric** or **Profile** because panels support only one visualization type. If a query set to **Both** is moved from Explore to a dashboard, Grafana automatically changes it to **Profile**. {{< /admonition >}} ## Use template variables in queries The label selector field supports Grafana template variables. You can use variables like `$job` or `$instance` in the label selector to create dynamic, reusable dashboards. For more information about supported variable types, examples, and limitations, refer to [Parca template variables](https://grafana.com/docs/grafana//datasources/parca/template-variables/). ## Query examples The following examples show common profiling queries. ### Query CPU profiles for a specific service 1. Select **process_cpu** > **cpu** from the profile type drop-down. 1. Enter a label selector to filter by service: ```text {job="my-service"} ``` 1. Set query type to **Profile** to view a flame graph, or **Metric** to view CPU usage over time. ### Compare profiles across instances 1. Select a profile type such as **process_cpu** > **samples**. 1. Filter to a specific service and instance: ```text {job="my-service", instance="10.0.0.1:7070"} ``` 1. Add a second query (click **+ Add query**) with a different instance: ```text {job="my-service", instance="10.0.0.2:7070"} ``` 1. Set both queries to **Metric** to compare time-series data across instances. ### Find memory allocation hot spots 1. Select **memory** > **alloc_objects** from the profile type drop-down. 1. Optionally filter by service: ```text {job="api-gateway"} ``` 1. Set query type to **Profile** to view a flame graph showing which functions allocate the most objects. 1. Adjust the time range to narrow down to a specific incident window. ## Profile query results Profiles are visualized as a flame graph. Refer to the [flame graph documentation](https://grafana.com/docs/grafana//panels-visualizations/visualizations/flame-graph/) to learn about the visualization and its features. Parca returns profiles aggregated over the selected time range. The absolute values in the flame graph grow as the time range gets bigger while keeping the relative values meaningful. You can zoom in on the time range to get a higher granularity profile, down to the point of a single Parca scrape interval. The flame graph displays the following columns: | Column | Description | | --------- | ----------------------------------------------- | | **level** | The depth of the function in the call stack. | | **value** | The cumulative value including child functions. | | **self** | The value attributable to this function only. | | **label** | The function name or symbol. | Units are automatically mapped from Parca to Grafana formats: `nanoseconds` displays as `ns` and `count` displays as a short number. ## Metrics query results Metrics results represent the aggregated value over time of the selected profile type. Parca returns a separate time series for each label combination, allowing you to quickly identify spikes in the value of scraped profiles and zoom in to a particular time range for further investigation. The time-series value field is named after the sample type from the profile type ID. For example, a query using `process_cpu:samples:count:cpu:nanoseconds` produces a value field named `samples`. # Parca template variables {{< admonition type="warning" >}} This plugin is deprecated and will only receive critical security updates. Support will end on January 2, 2027. {{< /admonition >}} Instead of hard-coding label values in your profiling queries, you can use template variables to create dynamic, reusable dashboards. Variables appear as drop-down menus at the top of the dashboard, making it easy to switch between services, instances, or environments without editing queries. For an introduction to template variables, refer to the [Variables](https://grafana.com/docs/grafana//dashboards/variables/) documentation. ## Supported variable types | Variable type | Supported | Notes | | ------------- | --------- | ---------------------------------------------------------------------------- | | Custom | Yes | Define a static list of values manually. | | Text box | Yes | Enter a free-form value at the top of the dashboard. | | Constant | Yes | Define a hidden, fixed value. | | Data source | Yes | Select a data source instance by name. | | Query | No | Parca doesn't implement variable queries to dynamically populate drop-downs. | {{< admonition type="note" >}} Parca doesn't support query-type variables. You can't use the Parca data source to dynamically populate variable drop-downs. Use custom variables to define values manually, or use another data source to populate query variables. {{< /admonition >}} ## Use variables in the label selector The Parca query editor's label selector field supports template variable interpolation. Use the standard `$variablename` or `${variablename}` syntax. Variables are interpolated in the label selector only -- the profile type drop-down doesn't support them. ### Variable syntax options | Syntax | Description | | ------------------------ | -------------------------------------------------------------------------------- | | `$variablename` | Simple syntax for most cases. | | `${variablename}` | Use when the variable is adjacent to other text (for example, `${host}_suffix`). | | `${variablename:format}` | Apply a specific format to the variable value. | ## Example: Filter profiles by service Create a custom variable to switch between services in your profiling dashboard. 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Variable**. 1. Select **Custom** as the variable type. 1. Enter a **Name**, for example `service`. 1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed. 1. Click **Open variable editor** to open the **Custom options** dialog box. 1. Enter comma-separated values, for example `frontend, backend, api-gateway`. 1. Click **Apply**. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. In your Parca query's label selector, reference the variable: ```text {job="$service"} ``` When you change the drop-down selection, the panel refreshes to show profiles for the selected service. ## Example: Filter by service and instance Create two custom variables to filter by multiple dimensions. 1. Create a variable named `service` with custom values `frontend, backend, api-gateway`. 1. Create a second variable named `instance` with custom values for your instance addresses, for example `10.0.0.1:7070, 10.0.0.2:7070, 10.0.0.3:7070`. In your Parca query's label selector, reference both variables: ```text {job="$service", instance="$instance"} ``` ## Example: Use a text box variable for free-form filtering Create a text box variable for free-form label filtering. 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Variable**. 1. Select **Textbox** as the variable type. 1. Enter a **Name**, for example `label_filter`. 1. Optionally set a default **Value**, for example `job="my-service"`. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. In your Parca query's label selector, reference the variable: ```text {$label_filter} ``` This lets you type any label matcher directly into the variable input at the top of the dashboard without editing the panel query. ## Limitations Parca template variable support has the following limitations: - **No query variable support:** You can't use Parca as a data source for populating variable options. Define values manually with custom variables or use another data source. - **Label selector only:** Variables are interpolated in the label selector field. The profile type drop-down doesn't support variables. - **No free-form filters:** Parca doesn't support free-form [filters](https://grafana.com/docs/grafana//visualizations/dashboards/build-dashboards/filter-group-by/). ## Next steps - [Build queries](https://grafana.com/docs/grafana//datasources/parca/query-editor/) using your template variables. - [Troubleshoot issues](https://grafana.com/docs/grafana//datasources/parca/troubleshooting/) if queries aren't returning expected data. # Troubleshoot Parca data source issues {{< admonition type="warning" >}} This plugin is deprecated and will only receive critical security updates. Support will end on January 2, 2027. {{< /admonition >}} This page provides solutions to common issues you might encounter when configuring or using the Parca data source. For configuration instructions, refer to [Configure the Parca data source](https://grafana.com/docs/grafana//datasources/parca/configure/). ## Connection errors These errors occur when Grafana can't reach the Parca instance. ### "Save & test" fails **Symptoms:** - Data source test times out or returns an error. - Unable to connect to the Parca server. **Possible causes and solutions:** | Cause | Solution | | -------------------- | -------------------------------------------------------------------------------------------------------------------------- | | Incorrect URL | Verify the URL points to your Parca HTTP endpoint (for example, `http://localhost:7070`). Grafana connects using gRPC-Web. | | Parca isn't running | Verify that your Parca instance is running and accessible from the Grafana server. | | Firewall blocking | Check that firewall rules allow outbound traffic from Grafana to the Parca server on the configured port. | | TLS misconfiguration | If using HTTPS, verify that TLS certificates are correctly configured on both Grafana and Parca. | When the connection succeeds, the health check displays **"Data source is working"**. The health check queries the available profile types from the Parca server, so any connectivity or authentication issue causes it to fail. ### Connection refused or timeout errors **Symptoms:** - Queries fail with network errors. - Intermittent connection issues. **Solutions:** 1. Verify network connectivity from the Grafana server to the Parca endpoint. 1. Check that the Parca server is healthy and responding to requests. 1. For Grafana Cloud, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if accessing a private Parca instance. ## Query errors These errors occur when running queries against the Parca data source. ### "Invalid report type" or "try updating Parca to v0.19+" **Symptoms:** - Queries fail with an error containing "invalid report type." - The error message suggests updating Parca to v0.19+. - Flame graph data doesn't load. **Solutions:** 1. Upgrade your Parca server to v0.19 or later. This error occurs because older versions of Parca don't support the flame graph Arrow format that Grafana requires. ### `Unknown report type returned from query. update parca` **Symptoms:** - Profile queries fail with this exact error message. - Metric queries may still succeed. **Solutions:** 1. Upgrade your Parca server. This error occurs when the Parca server returns a response format that Grafana doesn't recognize. Updating to the latest Parca version resolves the issue. ### "No data" or empty results **Symptoms:** - Query runs without error but returns no data. - Flame graph or metrics panel shows "No data." **Possible causes and solutions:** | Cause | Solution | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | No profile type selected | Select a profile type from the drop-down menu. This field is required -- queries silently return empty data without it. | | Time range doesn't contain data | Expand the dashboard time range or verify that profiling data exists for the selected period. | | Label selector too restrictive | Remove or broaden label filters to verify that matching profiles exist. | | Label selector syntax error | Verify the label selector uses valid syntax (for example, `{job="my-service"}`). Refer to the [query editor documentation](https://grafana.com/docs/grafana//datasources/parca/query-editor/) for supported operators. | | Parca isn't scraping targets | Check your Parca configuration to verify that targets are being scraped and profiles collected. | ## Query editor issues These issues relate to the query editor interface. ### Profile type drop-down is empty **Symptoms:** - The profile type selector shows no options. - The button text reads **Select a profile type** with nothing to choose. **Possible causes and solutions:** | Cause | Solution | | -------------------------- | ------------------------------------------------------------------------------------------------ | | Connection to Parca failed | Verify the data source connection using **Save & test** in the data source settings. | | Parca has no scraped data | Verify that Parca is scraping targets and has collected profile data. | | Network or auth error | Check the browser developer console for failed requests to the `profileTypes` resource endpoint. | ### "Both" query type is missing **Symptoms:** - Only **Metric** and **Profile** appear in the query type options. - The **Both** option isn't available. **Solution:** This is expected behavior. The **Both** query type is only available in [Explore](https://grafana.com/docs/grafana//explore/). Dashboard panels support only one visualization type, so Grafana limits the options to **Metric** or **Profile**. If a query set to **Both** in Explore is used in a dashboard, Grafana automatically changes it to **Profile**. ### Autocomplete suggestions don't appear **Symptoms:** - No label name or value suggestions appear when typing in the label selector. - Autocomplete works for some labels but not others. **Possible causes and solutions:** | Cause | Solution | | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Label names haven't loaded | Wait a moment after opening the query editor. Autocomplete loads label names from Parca on initialization. | | Connection to Parca failed | Verify the data source connection. Autocomplete queries the `labelNames` and `labelValues` resource endpoints. | | Cursor position not recognized | Place your cursor inside curly braces `{}` and after an `=` sign for value suggestions. Autocomplete triggers on specific characters: `{`, `,`, `=`, `~`, `"`, space. | ## Template variable issues These issues relate to using template variables with the Parca data source. ### Variables don't resolve in queries **Symptoms:** - Variable syntax like `$variable` appears literally in query results instead of being replaced. - Queries return no data when using variables. **Possible causes and solutions:** | Cause | Solution | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | Variable used in profile type | Template variables are only supported in the **label selector** field. The profile type drop-down doesn't support variable interpolation. | | Variable not defined | Verify the variable exists in **Dashboard options** > **Variables** and has valid values. | | Wrong variable syntax | Use `$variablename` or `${variablename}`. Verify there are no typos in the variable name. | ### Can't use Parca to populate variable options **Symptoms:** - When creating a query-type variable with the Parca data source, no values are returned. **Solution:** Parca doesn't support query-type variables. You can't use the Parca data source to dynamically populate variable drop-downs. Use **Custom** variables with manually defined values, **Text box** variables for free-form input, or a different data source for query-type variables. Refer to [Parca template variables](https://grafana.com/docs/grafana//datasources/parca/template-variables/) for supported variable types and examples. ## Enable debug logging To capture detailed error information for troubleshooting: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location). 1. Look for Parca-specific entries that include request and response details. Common log messages include: - `"Failed to get profile types"` -- profile type loading failed. - `"Failed to get label names"` -- label autocomplete data failed to load. - `"Failed to process query"` -- a query to Parca returned an error. - `"Failed to unmarshall query"` -- the query JSON couldn't be parsed. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you've tried the solutions on this page and still encounter issues: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs related to the Parca data source. 1. Consult the [Parca documentation](https://www.parca.dev/docs) for service-specific guidance. 1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user. 1. When reporting issues, include: - Grafana version - Parca server version - Error messages (redact sensitive information) - Steps to reproduce - Relevant configuration (redact credentials) # PostgreSQL alerting The PostgreSQL data source supports [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). You can create alert rules that evaluate time series queries against your PostgreSQL database and send notifications when conditions are met. ## Before you begin - Ensure your [PostgreSQL data source is configured](https://grafana.com/docs/grafana//datasources/postgres/configure/). - Familiarize yourself with [Alert rules](https://grafana.com/docs/grafana//alerting/fundamentals/alert-rules/) and [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Supported query format Only **time series** queries can be used in alert rule conditions. - Your query must return a column named `time` (native SQL date/time or UNIX epoch) and one or more numeric value columns. - **Table** formatted queries aren't supported in alert rule conditions. For details on writing time series queries, refer to [Time series queries](https://grafana.com/docs/grafana//datasources/postgres/query-editor/#time-series-queries) in the PostgreSQL query editor. ## Limitations Keep the following limitations in mind when using PostgreSQL with Grafana Alerting: - **Time series format only:** Alert rules require the query **Format** set to **Time series**. Table format queries can't be used as alert conditions. - **No fill mode in alerts:** The fill parameter in `$__timeGroup` (for example, `$__timeGroup(col, '5m', 0)`) isn't applied during alert evaluation. Missing data points remain as gaps. - **Single query per condition:** Each alert condition evaluates a single query. If you need to combine results from multiple queries, use [math expressions](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/#add-expressions) in the alert rule. - **Template variables aren't supported:** Alert queries don't support dashboard template variables because alerts evaluate independently of any dashboard context. Hard-code values or use [alert rule labels](https://grafana.com/docs/grafana//alerting/fundamentals/alert-rules/annotation-label/) instead. ## Create an alert rule To create an alert rule that uses PostgreSQL: 1. Go to **Alerting** in the left menu and select **Alert rules**. 1. Click **New alert rule**. 1. In the query section, select your **PostgreSQL** data source. 1. Set the query **Format** to **Time series**. 1. Write a SQL query that returns a `time` column and one or more numeric values. 1. Configure the condition, evaluation group, and notification settings. 1. Save the rule. For step-by-step guidance, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example queries for alerting The following examples show common PostgreSQL alerting patterns. ### Threshold alert The following query returns a time series suitable for a threshold alert (for example, alert when average value exceeds a limit): ```sql SELECT $__timeGroupAlias("time_date_time", '5m'), avg("value_double") AS value FROM test_data WHERE $__timeFilter("time_date_time") GROUP BY time ORDER BY time ``` Use condition types such as **Is above** or **Is below** in the alert rule to evaluate the series. ### Multi-dimensional alert To create separate alert instances for each value of a label column, include a string column in your query. Grafana creates one alert instance per unique label value, so you can monitor each host, service, or region independently. ```sql SELECT $__timeGroupAlias("time_date_time", '5m'), avg("value_double") AS value, hostname FROM test_data WHERE $__timeFilter("time_date_time") GROUP BY time, hostname ORDER BY time ``` This query produces a separate time series for each `hostname`. The alert rule evaluates the condition against each series independently, so you receive individual notifications per host. ### Alert on row count To alert when the number of rows in a time window exceeds a threshold (for example, error count): ```sql SELECT $__timeGroupAlias("created_at", '5m'), count(*) AS error_count FROM error_logs WHERE $__timeFilter("created_at") AND severity = 'ERROR' GROUP BY time ORDER BY time ``` ## Template annotations and labels You can use [template annotations and labels](https://grafana.com/docs/grafana//alerting/alerting-rules/templates/) to include query results or metadata in alert notifications and labels. ## Best practices Follow these best practices when using PostgreSQL for alerting: - **Keep queries fast:** Alert rules evaluate on a schedule (for example, every minute). Avoid expensive joins, subqueries, or scanning large tables without indexes. Add indexes on time columns and any columns used in `WHERE` clauses. - **Use `$__timeFilter`:** Always include a `$__timeFilter` or `$__unixEpochFilter` macro to limit data to the evaluation window. Without it, the query scans the entire table on every evaluation. - **Handle `NULL` values:** If your data contains `NULL` values, Grafana treats them as "no data." Configure the alert rule's **No data** behavior to match your expectations (for example, **Alerting**, **No data**, or **OK**). - **Test in Explore first:** Verify your query returns the expected time series in [Explore](https://grafana.com/docs/grafana//explore/) before using it in an alert rule. Check the **Format** is set to **Time series** and results are sorted by time. - **Set realistic evaluation intervals:** Choose an evaluation interval that gives your query enough time to complete. If your query takes 10 seconds, don't evaluate every 10 seconds—allow for headroom. # PostgreSQL annotations Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics. You can use PostgreSQL as a data source for annotations to display events such as deployments, alerts, or other significant occurrences on your visualizations. For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/). ## Before you begin Before creating PostgreSQL annotations, ensure you have: - [A configured PostgreSQL data source](https://grafana.com/docs/grafana//datasources/postgres/configure/). - Tables containing event data with timestamp fields. - Read access to the tables containing your events. ## Create an annotation query To add a PostgreSQL annotation to your dashboard: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Annotation query**. 1. Enter a name for the annotation query. 1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox. 1. Select a color for the annotation event markers. 1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed. 1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed. 1. Click **Open query editor** to open the **Annotation Query** dialog box. 1. Select your **PostgreSQL** data source from the **Data source** drop-down list. 1. Configure the annotation query and field mappings. 1. (Optional) Click **Test annotation query** to ensure that the query is working properly. 1. Click **Close** when you've completed the query setup. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. ## Query columns Your annotation query must return a `time` column and can optionally include `timeend`, `text`, and `tags` columns. | Column | Required | Description | | --------- | -------- | --------------------------------------------------------------------------------------------- | | `time` | Yes | The timestamp for the annotation. Can be a native SQL date/time type or UNIX epoch value. | | `timeend` | No | The end timestamp for range annotations. Creates a shaded region instead of a vertical line. | | `text` | No | The annotation description displayed when you hover over the annotation. | | `tags` | No | Tags for the annotation as a comma-separated string. Helps categorize and filter annotations. | ## Example queries The following examples show common annotation query patterns. ### Basic annotation with epoch time Display events using UNIX epoch timestamps: ```sql SELECT epoch_time as time, metric1 as text, concat_ws(', ', metric1::text, metric2::text) as tags FROM public.test_data WHERE $__unixEpochFilter(epoch_time) ``` ### Range annotation with start and end time Display events with duration as shaded regions: ```sql SELECT epoch_time as time, epoch_time_end as timeend, metric1 as text, concat_ws(', ', metric1::text, metric2::text) as tags FROM public.test_data WHERE $__unixEpochFilter(epoch_time) ``` ### Annotation with native SQL date/time Display events using native PostgreSQL date/time columns: ```sql SELECT native_date_time as time, metric1 as text, concat_ws(', ', metric1::text, metric2::text) as tags FROM public.test_data WHERE $__timeFilter(native_date_time) ``` ### Deployment annotations Display deployment events: ```sql SELECT deployed_at as time, concat('Deployed ', version, ' to ', environment) as text, environment as tags FROM deployments WHERE $__timeFilter(deployed_at) ``` ### Range annotation for maintenance windows Display maintenance windows as shaded regions: ```sql SELECT start_time as time, end_time as timeend, concat('Maintenance: ', description) as text, 'maintenance' as tags FROM maintenance_windows WHERE $__timeFilter(start_time) ``` ## Macros Use these macros in your annotation queries to filter by the dashboard time range: | Macro | Description | | ---------------------------- | ---------------------------------------------------------------- | | `$__timeFilter(column)` | Filters by time range using a native SQL date/time column. | | `$__unixEpochFilter(column)` | Filters by time range using a column with UNIX epoch timestamps. | ## Best practices Follow these best practices when creating PostgreSQL annotations: - **Use time filters:** Always include `$__timeFilter()` or `$__unixEpochFilter()` to limit results to the dashboard time range. - **Keep queries efficient:** Add indexes on time columns and filter columns to improve query performance. - **Use meaningful text:** Include descriptive information in the `text` column to make annotations useful. - **Organize with tags:** Use consistent tag values to categorize annotations and enable filtering. - **Test queries first:** Verify your query returns expected results in Explore before adding it as an annotation. # PostgreSQL template variables Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana displays these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as **template variables**. For an introduction to templating and template variables, refer to [Templating](https://grafana.com/docs/grafana//dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/). ## Query variable A query variable in Grafana dynamically retrieves values from your data source using a query. With a query variable, you can write a PostgreSQL query that returns values such as measurement names, key names, or key values that are shown in a drop-down select box. The PostgreSQL data source includes a dedicated variable query editor with the full SQL editor (Code mode with auto-completion and syntax highlighting) and **Value Field** / **Text Field** drop-downs. After you write and run a query, the editor automatically detects the returned columns and populates the field drop-downs so you can map which column provides the variable value and which provides the display label. To create a query variable: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Variable**. 1. Select **Query** as the variable type. 1. Enter a **Name** for your variable (for example, `host`). 1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed. 1. Click **Open variable editor** to open the **Query Variable** dialog box. 1. Select your PostgreSQL data source. 1. Write a SQL query in the editor. Macros like `$__timeFilter` are supported. 1. Optionally set **Value Field** and **Text Field** to control which columns map to the variable value and display label. For example, the following query returns all values from the `hostname` column: ```sql SELECT hostname FROM host ``` A query can return multiple columns, and Grafana automatically generates a list using the values from those columns. For example, the following query returns values from both the `hostname` and `hostname2` columns, which are included in the variable's drop-down list. ```sql SELECT host.hostname, other_host.hostname2 FROM host JOIN other_host ON host.city = other_host.city ``` To use time range dependent macros like `$__timeFilter(column)` in your query, you must set the template variable's refresh mode to **On Time Range Change**. ```sql SELECT event_name FROM event_log WHERE $__timeFilter(time_column) ``` ### Key/value variables You can create a key/value variable so the drop-down shows a user-friendly label (for example, hostname) while panel queries use a different value (for example, ID). Use the variable editor’s **Value field** and **Text field** at the bottom of the query section to specify which query columns supply the value and the label. Your query can use any column names; you do not need `__value` or `__text` in the SQL. Example: run a query that returns `hostname` and `id`, then set **Text field** to `hostname` and **Value field** to `id`. ```sql SELECT hostname, id FROM host ``` Note that the values in the text column should be unique. If there are duplicates, Grafana uses only the first matching entry. Alternatively, you can use the legacy approach: return columns named `__text` and `__value` in your query (for example, `SELECT hostname AS __text, id AS __value FROM host`). ### Nested variables You can create nested variables, where one variable depends on the value of another. For example, if you have a variable named `region`, you can configure a `hosts` variable to only show hosts from the selected region. If `region` is a multi-value variable, use the `IN` operator instead of `=` to match against multiple selected values. ```sql SELECT hostname FROM host WHERE region IN($region) ``` {{< admonition type="note" >}} When using a multi-value PostgreSQL query variable with **Include All option** enabled and **Display** set to **Hidden**, the variable defaults to the first returned value, not **All**. This is because Grafana persists the active selection at dashboard save time, and a hidden variable has no UI for users to change it. To default a hidden variable to **All**, temporarily show the variable, select **All** in the drop-down, save the dashboard, then hide the variable and save again. {{< /admonition >}} ### Filter results with `__searchFilter` Using `__searchFilter` in the query field filters the query results based on user input in the drop-down selection box. If nothing is entered, the default value for `__searchFilter` is `%`. You must enclose the `__searchFilter` expression in quotes because Grafana doesn't add them automatically. The following example enables real-time searching for `hostname` as the user types in the drop-down selection box: ```sql SELECT hostname FROM host WHERE hostname LIKE '$__searchFilter' ``` For tables with a large number of rows, combine `__searchFilter` with `LIMIT` to keep the drop-down responsive: ```sql SELECT DISTINCT hostname FROM host WHERE hostname LIKE '$__searchFilter' ORDER BY hostname LIMIT 100 ``` ### Regex variable You can use a **Regex** type variable to filter or transform values from another variable. For example, if you have a variable named `server` that returns values like `web-prod-01`, `web-staging-01`, and `db-prod-01`, you can create a regular expression variable to extract only the environment: 1. Create a new variable with **Type: Query**. 1. Set the query to return the raw values. 1. In the **Regex** field, enter a pattern to extract or filter. For example, `/^web-(.+)-\d+$/` extracts `prod` and `staging` from web server names. You can also use regular expression to filter a variable's options. For example, setting the **Regex** field to `/prod/` on a variable that returns all server names limits the drop-down to only production servers. ## Multi-property variables The PostgreSQL data source supports **multi-property variables**. Use them when the same logical concept has different identifiers in different contexts (for example, an environment called `dev` in one system and `development` in another). Instead of maintaining several variables in sync, you can map all of those values to one variable and reference the property you need in each panel or query. {{< figure alt="PostgreSQL multi-property variable example" src="/media/docs/postgres/postgreSQL-multi-prop-variable-v12.3.png" >}} You can create a multi-property variable with either **Type: Custom** or **Type: Query**: - **Type: Custom** – In **Custom options** > **JSON**, paste your own JSON array with the mapping. Each object in the array can have any number of properties; use `text` and `value` for the label and value shown in the drop-down, and add additional properties as needed. For the JSON format and examples, refer to [Multi-property custom variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#multi-property-custom-variables) in Add and manage variables. - **Type: Query** – Write a SQL query that returns multiple columns. In the variable editor, set **Value field** and **Text field** to the columns that supply the value and the label for the drop-down. Add one column per property you want to reference; each column name becomes a property name. In panels and queries, reference a property with `${varName.columnName}`. **Example (Type: Query):** A variable named `env` that lists environments with different identifiers per cloud. In the variable editor, set **Text field** to `name` and **Value field** to `id`. ```sql SELECT name, id, aws_identifier AS env_aws, azure_identifier AS env_azure FROM environments ``` In a panel query you might use `$env.env_aws` for an AWS-related query and `$env.env_azure` for an Azure-related query. For more on the concept, refer to [Configure multi-property variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#configure-multi-property-variables) in [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/). ## Use variables in queries Grafana automatically quotes template variable values only when the template variable is a **multi-value** variable. When using a multi-value variable, use the `IN` comparison operator instead of `=` to match against multiple values. Grafana supports two syntaxes for using variables in queries: - **`$` syntax** Example with a template variable named `hostname`: ```sql SELECT atimestamp AS time, aint AS value FROM table WHERE $__timeFilter(atimestamp) AND hostname IN($hostname) ORDER BY atimestamp ASC ``` - **`[[varname]]` syntax** Example with a template variable named `hostname`: ```sql SELECT atimestamp AS time, aint AS value FROM table WHERE $__timeFilter(atimestamp) AND hostname IN([[hostname]]) ORDER BY atimestamp ASC ``` ### Multi-value variable with IN clause When a variable has **Multi-value** enabled, Grafana automatically adds quotes around each selected value. For example, if the user selects `web01` and `web02`, `$hostname` expands to `'web01','web02'`. Use the `IN` operator to match: ```sql SELECT $__timeGroupAlias("created_at", '5m'), count(*) AS requests FROM access_log WHERE $__timeFilter("created_at") AND hostname IN($hostname) GROUP BY time ORDER BY time ``` If only a single value is selected, the `IN` clause still works correctly—`IN('web01')` is equivalent to `= 'web01'`. ### Multi-value variable with numeric columns For numeric columns (such as IDs), disable quoting with the `csv` format option so values aren't wrapped in quotes: ```sql SELECT name FROM host WHERE id IN(${host_id:csv}) ``` If `host_id` has values `1`, `2`, and `3` selected, this expands to `IN(1,2,3)` instead of `IN('1','2','3')`. ### Disable quoting for multi-value variables By default, Grafana formats multi-value variables as a quoted, comma-separated string. For example, if `server01` and `server02` are selected, the result is `'server01','server02'`. To disable quoting, use the `csv` formatting option: ```text ${servers:csv} ``` This outputs the values as an unquoted comma-separated list. Refer to [Advanced variable format options](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/#advanced-variable-format-options) for additional information. # Troubleshoot PostgreSQL data source issues This document provides troubleshooting information for common errors you may encounter when using the PostgreSQL data source in Grafana. ## Connection errors The following errors occur when Grafana cannot establish or maintain a connection to PostgreSQL. ### Failed to connect to PostgreSQL **Error message:** `failed to connect to ... : connect: connection refused` or `dial tcp: connect: connection refused` **Cause:** Grafana cannot establish a network connection to the PostgreSQL server. **Solution:** 1. Verify that the Host URL is correct in the data source configuration. 1. Check that PostgreSQL is running and accessible from the Grafana server. 1. Verify the port is correct (the PostgreSQL default port is `5432`). 1. Ensure there are no firewall rules blocking the connection. 1. Check that PostgreSQL is configured to accept connections from the Grafana server in `pg_hba.conf`. 1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your PostgreSQL instance is not publicly accessible. ### Grafana Cloud can't reach a private database **Error message:** `dial tcp: connect: connection refused`, `i/o timeout`, or `context deadline exceeded` when using Grafana Cloud with a database on a private network. **Cause:** Grafana Cloud runs in a hosted environment and can't directly reach databases on `localhost`, `127.0.0.1`, or private IP ranges (`10.x`, `172.16.x`, `192.168.x`). This is the most common issue when migrating from self-managed Grafana to Grafana Cloud. **Solution:** 1. Set up [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) to create a secure tunnel between Grafana Cloud and your private network. 1. Install the PDC agent on a machine that has network access to your PostgreSQL instance. 1. If you experience intermittent connection drops with the Docker-based PDC agent, try the binary-based agent instead. This has resolved stability issues in some environments. 1. Update the **Host URL** in the data source settings to use the hostname as seen from the PDC agent's network (not `localhost`). ### Request timed out **Error message:** "context deadline exceeded" or "i/o timeout" **Cause:** The connection to PostgreSQL timed out before receiving a response. **Solution:** 1. Check the network latency between Grafana and PostgreSQL. 1. Verify that PostgreSQL is not overloaded or experiencing performance issues. 1. Increase the **Max lifetime** setting in the data source configuration under **Connection limits**. 1. Reduce the time range or complexity of your query. 1. Check if any network devices (load balancers, proxies) are timing out the connection. ### Host not found **Error message:** `failed to connect to ... : hostname resolving error` or `lookup hostname: no such host` **Cause:** The hostname specified in the data source configuration can't be resolved. **Solution:** 1. Verify the hostname is spelled correctly. 1. Check that DNS resolution is working on the Grafana server. 1. Try using the database's public IP address instead of a hostname. This is a useful diagnostic step and can serve as a workaround if DNS resolution is the issue. 1. Ensure the PostgreSQL server is accessible from the Grafana server's network. 1. For Grafana Cloud, DNS resolution behavior can differ between stack regions and cloud providers. If a hostname resolves from one stack but not another, try the direct IP address and contact [Grafana Support](https://grafana.com/docs/grafana-cloud/account-management/support/). ## Authentication errors The following errors occur when there are issues with authentication credentials or permissions. ### No PostgreSQL user name specified **Error message:** `FATAL: no PostgreSQL user name specified in startup packet (SQLSTATE 28000)` **Cause:** The connection to PostgreSQL was attempted without a username. This typically means the **Username** field in the data source configuration is empty or was cleared. **Solution:** 1. Open the data source settings and verify the **Username** field contains a valid PostgreSQL user. 1. Click **Save & test** and confirm the connection succeeds. 1. If the username disappears after saving, check your Grafana version. A bug in Grafana v13.1 on the fast release channel caused the username field to be cleared on save. Upgrading to a patched release or switching to the steady release channel resolves this issue. ### Password authentication failed **Error message:** `failed to connect to ... : server error: FATAL: password authentication failed for user "username" (SQLSTATE 28P01)` **Cause:** The username or password is incorrect. **Solution:** 1. Verify that the username and password are correct in the data source configuration. 1. Check that the user exists in PostgreSQL. 1. Verify the password has not expired. 1. If no password is specified, ensure a [PostgreSQL password file](https://www.postgresql.org/docs/current/static/libpq-pgpass.html) is configured. ### Permission denied **Error message:** `ERROR: permission denied for table table_name (SQLSTATE 42501)` or `ERROR: permission denied for schema schema_name (SQLSTATE 42501)` **Cause:** The database user does not have permission to access the requested table or schema. **Solution:** 1. Verify the user has `SELECT` permissions on the required tables. 1. Grant the necessary permissions: ```sql GRANT USAGE ON SCHEMA schema_name TO grafanareader; GRANT SELECT ON schema_name.table_name TO grafanareader; ``` 1. Check that the user has access to the correct database. 1. Verify the search path includes the schema containing your tables. ### No pg_hba.conf entry **Error message:** `failed to connect to ... : server error: FATAL: no pg_hba.conf entry for host "ip_address", user "username", database "database_name" (SQLSTATE 28000)` **Cause:** PostgreSQL is not configured to accept connections from the Grafana server. **Solution:** 1. Edit the `pg_hba.conf` file on the PostgreSQL server. 1. Add an entry to allow connections from the Grafana server: ```text host database_name username grafana_ip/32 md5 ``` 1. Reload PostgreSQL configuration: `SELECT pg_reload_conf();` 1. If using SSL, ensure the correct authentication method is specified (for example, `hostssl` instead of `host`). ## TLS and certificate errors The following errors occur when there are issues with TLS configuration. ### Certificate verification failed **Error message:** "x509: certificate signed by unknown authority" or "certificate verify failed" **Cause:** Grafana cannot verify the TLS certificate presented by PostgreSQL. **Solution:** 1. Set the **TLS/SSL Mode** to the appropriate level (`require`, `verify-ca`, or `verify-full`). 1. If using a self-signed certificate, add the CA certificate in **TLS/SSL Auth Details**. 1. Verify the certificate chain is complete and valid. 1. Ensure the certificate has not expired. 1. For testing only, set **TLS/SSL Mode** to `disable` (not recommended for production). ### SSL not supported **Error message:** `failed to connect to ... : server refused TLS connection` or `server does not support SSL` **Cause:** The PostgreSQL server is not configured for SSL connections, but the data source requires SSL. **Solution:** 1. Set **TLS/SSL Mode** to `disable` if SSL is not required. 1. Alternatively, enable SSL on the PostgreSQL server by configuring `ssl = on` in `postgresql.conf`. 1. Ensure the server has valid SSL certificates configured. ### Client certificate error **Error message:** "TLS: failed to find any PEM data in certificate input" or "could not load client certificate" **Cause:** The client certificate or key is invalid or incorrectly formatted. **Solution:** 1. Verify the certificate and key are in PEM format. 1. Ensure the certificate file path is correct and readable by the Grafana process. 1. Check that the certificate and key match (belong to the same key pair). 1. If using certificate content, ensure you've pasted the complete certificate including headers. ## Database errors The following errors occur when there are issues with the database configuration. ### Database does not exist **Error message:** `failed to connect to ... : server error: FATAL: database "database_name" does not exist (SQLSTATE 3D000)` **Cause:** The specified database name is incorrect or the database doesn't exist. **Solution:** 1. Verify the database name in the data source configuration. 1. Check that the database exists: `\l` in `psql` or `SELECT datname FROM pg_database;` 1. Ensure the database name is case-sensitive and matches exactly. 1. Verify the user has permission to connect to the database. ### Relation does not exist **Error message:** `ERROR: relation "table_name" does not exist (SQLSTATE 42P01)` **Cause:** The specified table or view does not exist, or the user cannot access it. **Solution:** 1. Verify the table name is correct and exists in the database. 1. Check the schema name if the table is not in the public schema. 1. Use fully qualified names: `schema_name.table_name`. 1. Verify the user has `SELECT` permission on the table. 1. Check the search path: `SHOW search_path;` ## Query errors The following errors occur when there are issues with SQL syntax or query execution. ### Query truncated by double-dash in string literals **Error message:** Unexpected syntax errors or truncated results when string values contain `--` (double dash). **Cause:** In older Grafana versions, the SQL comment-stripping parser didn't correctly handle `--` inside single-quoted strings. A query like `WHERE name = 'value--suffix'` would be truncated at the `--`, causing the rest of the query to be silently dropped. This also affected strings with consecutive hyphens (for example, `'10YDE-VE-------2'` would be truncated to `'10YDE-VE`). This was fixed with a quote-aware comment-stripping parser (PR #121772). The fix also handles PostgreSQL dollar-quoted strings (`$$...$$`). It is available in Grafana 13.1 and later, Grafana 13.0.2 and later, and these 12.x patch releases: 12.1.11, 12.2.9, 12.3.7, and 12.4.3. **Solution:** 1. Upgrade to a Grafana release that includes the fix (13.1+, 13.0.2+, or one of the 12.x patch releases listed above). 1. If you can't upgrade immediately, work around the issue by using PostgreSQL string concatenation to avoid literal `--` in your queries: ```sql WHERE name = 'value' || '--' || 'suffix' ``` 1. Alternatively, use a parameterized approach with a template variable for the value. ### Query syntax error **Error message:** `ERROR: syntax error at or near "keyword" (SQLSTATE 42601)` **Cause:** The SQL query contains invalid syntax. **Solution:** 1. Check your query syntax for typos or invalid keywords. 1. Verify column and table names are correctly quoted if they contain special characters or are reserved words. 1. Use double quotes for identifiers: `"column_name"`. 1. Test the query directly in a PostgreSQL client such as `psql`. ### Column does not exist **Error message:** `ERROR: column "column_name" does not exist (SQLSTATE 42703)` **Cause:** The specified column name is incorrect or doesn't exist in the table. **Solution:** 1. Verify the column name is spelled correctly. 1. Check that column names are case-sensitive in PostgreSQL when quoted. 1. Use the correct quoting for column names: `"Column_Name"` for case-sensitive names. 1. Verify the column exists in the table: `\d table_name` in `psql`. ### No time column found **Error message:** "no time column found" or time series visualization shows no data **Cause:** The query result does not include a properly formatted time column. **Solution:** 1. Ensure your query includes a column named `time` that returns a timestamp or epoch value. 1. Use an alias to rename your time column: `SELECT created_at AS time`. 1. Ensure the time column is of type `timestamp`, `timestamptz`, or a numeric epoch value. 1. Order results by the time column: `ORDER BY time ASC`. ### Macro expansion error **Error message:** "macro '$\_\_timeFilter' not found" or incorrect query results with macros **Cause:** Grafana macros aren't being properly expanded. **Solution:** 1. Verify the macro syntax is correct, for example `$\_\_timeFilter(time_column)`. 1. Ensure the column name passed to the macro exists in your table. 1. Check that the macro isn't inside a SQL comment (`--` or `/* */`). Grafana strips comments before expanding macros, so macros inside comments are silently ignored. 1. Use the **Preview** toggle in Builder mode to see the expanded query. 1. Open the [Query inspector](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/#query-inspector) to view the exact SQL sent to PostgreSQL after macro expansion. 1. For time-based macros, ensure the column contains timestamp data. ## Performance issues The following issues relate to slow query execution or resource constraints. ### Query timeout **Error message:** "canceling statement due to statement timeout" or "query timeout" **Cause:** The query took longer than the configured timeout. **Solution:** 1. Reduce the time range of your query. 1. Add indexes to columns used in WHERE clauses and joins. 1. Use the `$\_\_timeFilter` macro to limit data to the dashboard time range. 1. Increase the statement timeout in PostgreSQL if you have admin access. 1. Optimize your query to reduce complexity. ### Too many connections **Error message:** `failed to connect to ... : server error: FATAL: too many connections for role "username" (SQLSTATE 53300)` or `connection pool exhausted` **Cause:** The maximum number of connections to PostgreSQL has been reached. **Solution:** 1. Reduce the **Max open** connections setting in the data source configuration. 1. Increase `max_connections` in PostgreSQL's `postgresql.conf` if you have admin access. 1. Check for connection leaks in other applications connecting to the same database. 1. Enable **Auto max idle** to automatically manage idle connections. ### Slow query performance **Cause:** Queries take a long time to execute. **Solution:** 1. Reduce the time range of your query. 1. Add appropriate indexes to your tables. 1. Use the `$\_\_timeFilter` macro to limit the data scanned. 1. Increase the **Min time interval** setting to reduce the number of data points. 1. Use `EXPLAIN ANALYZE` in PostgreSQL to identify query bottlenecks. 1. Consider using materialized views for complex aggregations. ## Provisioning errors The following errors occur when provisioning the data source via YAML. ### Invalid provisioning configuration **Error message:** "metric request error" or data source test fails after provisioning **Cause:** The provisioning YAML file contains incorrect configuration. **Solution:** 1. Ensure parameter names match the expected format exactly. 1. Verify the database name is **not** included in the URL. 1. Use the correct format for the URL: `hostname:port`. 1. Check that string values are properly quoted in the YAML file. 1. Refer to the [provisioning example](https://grafana.com/docs/grafana//datasources/postgres/configure/#provision-the-data-source) for the correct format. Example correct configuration: ```yaml datasources: - name: Postgres type: postgres url: localhost:5432 user: grafana secureJsonData: password: 'Password!' jsonData: database: grafana sslmode: 'disable' ``` ## Other common issues The following issues don't produce specific error messages but are commonly encountered. ### Empty query results **Cause:** The query returns no data. **Solution:** 1. Verify the time range includes data in your database. 1. Check that table and column names are correct. 1. Test the query directly in PostgreSQL. 1. Ensure filters are not excluding all data. 1. Verify the `$\_\_timeFilter` macro is using the correct time column. ### TimescaleDB functions not available **Cause:** TimescaleDB-specific functions like `time_bucket` are not available in the query builder. **Solution:** 1. Enable the **TimescaleDB** toggle in the data source configuration under **PostgreSQL Options**. 1. Verify TimescaleDB is installed and enabled in your PostgreSQL database. 1. Check that the `timescaledb` extension is created: `CREATE EXTENSION IF NOT EXISTS timescaledb;` ### Data appears delayed or missing recent points **Cause:** The visualization doesn't show the most recent data. **Solution:** 1. Check the dashboard time range and refresh settings. 1. Verify the **Min time interval** is not set too high. 1. Ensure data has been committed to the database (not in an uncommitted transaction). 1. Check for clock synchronization issues between Grafana and PostgreSQL. ## Get additional help If you continue to experience issues after following this troubleshooting guide: 1. Check the [PostgreSQL documentation](https://www.postgresql.org/docs/) for database-specific guidance. 1. Review the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Contact Grafana Support if you are a Cloud Pro, Cloud Contracted, or Enterprise user. 1. When reporting issues, include: - Grafana version - PostgreSQL version - Error messages (redact sensitive information) - Steps to reproduce - Relevant configuration such as data source settings, TLS mode, and connection limits (redact passwords and other credentials) # Prometheus alerting You can use Grafana Alerting with Prometheus to create alerts based on your time-series data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met. For general information about Grafana Alerting, refer to [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). ## Before you begin Before creating alerts with Prometheus, ensure you have: - A Prometheus data source configured in Grafana - Appropriate permissions to create alert rules - Understanding of the PromQL metrics you want to monitor ## Alert rule types Prometheus supports two alerting workflows in Grafana: | Type | Description | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Grafana-managed alert rules** | Alert rules defined and evaluated within Grafana, using Prometheus as the query data source. You create and manage these entirely in the Grafana Alerting UI. | | **Data source-managed rules** | Alerting rules defined in Prometheus itself (in `prometheus.yml` or rule files). When **Manage alerts via Alerting UI** is enabled in the data source configuration, Grafana displays these existing rules in the Alerting UI. For Prometheus (unlike Mimir), this is read-only. | ## Create a Grafana-managed alert rule To create a Grafana-managed alert rule using Prometheus: 1. Navigate to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for the alert rule. 1. Select your **Prometheus** data source. 1. Write a PromQL query in the query editor. 1. Configure the alert condition (for example, when the last value is above a threshold). 1. Set the evaluation interval and pending period. 1. Configure notifications and labels. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## View data source-managed rules When **Manage alerts via Alerting UI** is enabled in the Prometheus data source configuration, Grafana fetches and displays alerting rules defined in Prometheus. These appear in the Alerting UI alongside Grafana-managed rules but are marked as data source-managed. For Prometheus data sources, this view is read-only. To modify these rules, update your Prometheus rule files directly. {{< admonition type="note" >}} For Mimir and Cortex data sources, the Alerting UI supports both viewing and creating data source-managed rules. Prometheus only supports viewing. {{< /admonition >}} ## Evaluation groups and intervals Alert rules are organized into evaluation groups. Each group has an evaluation interval that determines how frequently the rules in that group are evaluated. For example, an evaluation interval of `1m` means the alert query runs every 60 seconds. The **pending period** determines how long a condition must be continuously true before the alert fires. For example, with a 1-minute evaluation interval and a 5-minute pending period, the condition must be true for 5 consecutive evaluations before firing. Choose evaluation intervals based on your use case: - **15s–30s:** Critical infrastructure alerts where fast detection matters. - **1m:** Standard monitoring alerts (recommended default). - **5m:** Non-urgent or noisy metrics where you want to reduce evaluation load. ## Example alert queries The following examples show common alerting scenarios with Prometheus. Each example shows the PromQL query and how to configure the alert condition. ### Alert on high CPU usage Monitor CPU usage and alert when it exceeds 90%: **Query A:** ```promql 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[$__rate_interval])) * 100) ``` **Condition:** Set the threshold to alert when the last value of **A** is above `90`. This approach separates the metric query from the threshold, making it easier to adjust the threshold later without editing the PromQL. ### Alert on high memory usage Monitor memory usage across nodes: **Query A:** ```promql (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 ``` **Condition:** Alert when the last value of **A** exceeds `85`. ### Alert on high error rate Monitor HTTP error rates per service: **Query A:** ```promql sum(rate(http_requests_total{status=~"5.."}[$__rate_interval])) by (job) / sum(rate(http_requests_total[$__rate_interval])) by (job) * 100 ``` **Condition:** Alert when the last value of **A** exceeds `5` (meaning error rate above 5%). ### Alert on target down Monitor whether Prometheus scrape targets are reachable: **Query A:** ```promql up{job="myservice"} ``` **Condition:** Alert when the last value of **A** is below `1`. ### Alert when a metric disappears Use `absent()` to detect when a metric stops being scraped entirely, for example, when a service crashes and no longer reports metrics: **Query A:** ```promql absent(up{job="myservice"}) ``` **Condition:** Alert when the last value of **A** equals `1` (the `absent()` function returns 1 when the metric is missing, and nothing when the metric exists). For detecting staleness over a time window (metric exists but hasn't reported recently): ```promql absent_over_time(up{job="myservice"}[5m]) ``` ### Multi-condition alert (high latency AND high traffic) Use multiple queries and expressions to alert only when multiple conditions are true simultaneously. This reduces noise by avoiding alerts during low-traffic periods. **Query A** (P95 latency): ```promql histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="api"}[$__rate_interval])) by (le)) ``` **Query B** (request rate): ```promql sum(rate(http_requests_total{job="api"}[$__rate_interval])) ``` **Expression C** (Math, combining both conditions): ``` $A > 2 && $B > 100 ``` **Condition:** Alert when **C** has a value (it only returns data when both latency exceeds 2 seconds AND request rate exceeds 100 req/s). ## Recording rules [Grafana-managed recording rules](https://grafana.com/docs/grafana//alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/) pre-compute expensive PromQL expressions on a schedule and write the results back to a Prometheus-compatible data source as a new metric. This lets you query the pre-aggregated metric instead of re-evaluating the expensive expression on every dashboard load or alert evaluation. ### When to use recording rules - **Dashboard panels that query the same expensive expression repeatedly:** Pre-compute it once and query the result. - **Alert rules on complex expressions:** Simplify the alert query by alerting on the pre-aggregated metric. - **High-cardinality aggregations:** Reduce thousands of series to a handful of pre-computed series. ### Set up Grafana-managed recording rules for Prometheus 1. **Enable the data source as a recording rules target:** In the Prometheus data source configuration, verify that **Allow as recording rules target** is toggled on (it's on by default). This allows Grafana to write recording rule results back to this instance. 1. **Verify write access:** The Prometheus-compatible backend must support remote write. Grafana Cloud Metrics (Mimir) and self-managed Mimir support this natively. Standard Prometheus requires the `--web.enable-remote-write-receiver` flag (Prometheus 2.33+). 1. **Create a recording rule:** 1. Navigate to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Select **Recording rule** as the rule type (under the Grafana-managed section). 1. Enter the PromQL expression you want to pre-compute: ```promql sum(rate(http_requests_total[$__rate_interval])) by (service) ``` 1. Enter a metric name for the result (for example, `service:http_requests:rate5m`). Follow the Prometheus [recording rule naming convention](https://prometheus.io/docs/practices/rules/#naming-and-aggregation): `level:metric:operations`. 1. Select the **Target data source**, which is the Prometheus instance where results are written. 1. Set the evaluation interval (for example, every 1 minute). 1. Click **Save rule**. 1. **Query the recorded metric:** After the first evaluation, the new metric is available for dashboards and alerts: ```promql service:http_requests:rate5m{service="api"} ``` ### Recording rules with PDC If your Prometheus instance is behind [Private data source connect (PDC)](https://grafana.com/docs/grafana//datasources/prometheus/configure/#private-data-source-connect), Grafana can write recording rule results through the PDC tunnel. No additional configuration is needed, because PDC supports both reads and writes. ### Limitations - Recording rules require the target data source to support remote write. Standard Prometheus needs the `--web.enable-remote-write-receiver` flag. - Thanos does not support recording rules as a write target (refer to the [Prometheus type comparison](https://grafana.com/docs/grafana//datasources/prometheus/configure/#performance)). - Recording rule evaluation uses the configured evaluation interval, not dashboard time ranges. Use a fixed range vector (for example, `[5m]`) rather than `$__rate_interval`. For more information, refer to [Create Grafana-managed recording rules](https://grafana.com/docs/grafana//alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/). ## Limitations When using Prometheus with Grafana Alerting, be aware of the following limitations. ### Template variables not supported Alert queries don't support template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$instance` or `$job` aren't resolved. If your dashboard query uses template variables, create a separate query for alerting with hard-coded values or use label matchers directly. ### Query complexity Complex queries with many nested functions or large result sets may timeout or fail to evaluate. Simplify queries for alerting by: - Reducing the time range used in range vectors - Using appropriate aggregation to limit the number of returned series - Adding label filters to narrow the data scanned - Using recording rules to pre-compute expensive expressions ### OAuth token handling differs between Explore and Alerting When using OAuth-authenticated Prometheus endpoints (Google Managed Prometheus, Azure Managed Prometheus), queries may succeed in Explore and dashboards but fail intermittently during alert evaluation. This happens because the alerting backend handles token refresh differently from the interactive query path. If you're using GCP, consider the **datasource-syncer** pattern, a sidecar process that refreshes OAuth tokens and updates the data source credentials on a schedule shorter than the token lifetime. For detailed troubleshooting steps, refer to [OAuth token expiration errors](https://grafana.com/docs/grafana//datasources/prometheus/troubleshooting/#oauth-token-expiration-errors-gcp-and-azure). ### Data source-managed rules are read-only Grafana can display Prometheus alerting rules but can't create or modify them through the UI. To manage Prometheus-native alerting rules, edit your Prometheus rule files directly and reload the configuration. ## Configure alert state for execution errors By default, when a Grafana-managed alert rule encounters an execution error or timeout (such as a network blip, i/o timeout, or a transient 502 from Prometheus), the rule enters an **Error** state, which fires the alert. This can cause false alarms and spam on-call teams when the underlying issue is a brief connectivity interruption rather than a genuine threshold breach. To prevent false positives from transient errors, configure the **Alert state if execution error or timeout** setting on each alert rule: 1. Open the alert rule for editing. 1. In the alert conditions section, locate **Alert state if execution error or timeout**. 1. Change the value from **Alerting** (default) to one of: - **Keep Last State:** The alert retains its previous state (firing or normal) until a successful evaluation occurs. This is the recommended setting for most Prometheus alert rules. - **OK:** The alert is set to normal during the error, preventing it from firing. 1. Click **Save rule**. {{< admonition type="note" >}} If your alert rules frequently enter an error state, investigate the root cause (network stability, Prometheus resource limits, query timeout settings) rather than relying solely on this setting to suppress notifications. {{< /admonition >}} Common transient errors that trigger this behavior include: - `sse.dependencyError` or `sse.dataQueryError` in alert state history - "context deadline exceeded" or "i/o timeout" messages - HTTP 502 or 500 responses from the Prometheus server For more details on troubleshooting these errors, refer to [Troubleshoot Prometheus data source issues](https://grafana.com/docs/grafana//datasources/prometheus/troubleshooting/#transient-alert-errors-triggering-false-alarms). ## Best practices Follow these best practices when creating Prometheus alerts: - **Configure error state handling:** Set **Alert state if execution error or timeout** to **Keep Last State** to prevent transient backend errors from triggering false alarms. - **Use `$__rate_interval`:** When using `rate()` or `increase()` in alert queries, use `$__rate_interval` to ensure the range window is always large enough relative to the scrape interval. Grafana resolves this variable based on the evaluation interval and scrape interval configuration. - **Add label filters:** Include specific label matchers to focus on relevant data and improve query performance. - **Set realistic pending periods:** Use the pending period to avoid alerting on brief spikes. For example, set a 5-minute pending period so the condition must persist before firing. - **Test queries first:** Verify your query returns expected results in Explore before creating an alert. - **Use meaningful names:** Give alert rules descriptive names that indicate what they monitor and the severity. - **Pre-aggregate with recording rules:** For complex or frequently evaluated expressions, create recording rules and alert on the pre-aggregated metric. - **Use `absent()` for availability monitoring:** Detect when metrics stop being reported, which often indicates a crashed or unresponsive service. If you encounter errors when creating or evaluating alert rules, refer to [Troubleshoot Prometheus data source issues](https://grafana.com/docs/grafana//datasources/prometheus/troubleshooting/#alerting-errors). # Prometheus annotations Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics. You can use Prometheus as a data source for annotations to display events such as deployments, alerts, or threshold crossings on your visualizations. For general information about annotations, refer to [Annotate visualizations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/). ## Before you begin Before creating Prometheus annotations, ensure you have: - A configured Prometheus data source in Grafana - Metrics in Prometheus that represent the events you want to annotate - Read access to the Prometheus instance ## Create an annotation query To add a Prometheus annotation to your dashboard: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Annotation query**. 1. Enter a name for the annotation query. 1. If you don't want to use the annotation query right away, clear the **Enabled** checkbox. 1. Select a color for the annotation event markers. 1. Select an option in the **Show annotation controls in** drop-down list to control where on the dashboard the annotation is displayed. 1. Select an option in the **Show in** drop-down list to control the panels in which the annotation is displayed. 1. Click **Open query editor** to open the **Annotation Query** dialog box. 1. Select your **Prometheus** data source from the **Data source** drop-down list. 1. Enter a PromQL expression in the query field. 1. Set the **Min step** to control annotation density (a larger step means fewer annotations). 1. Configure the field mappings to control what appears in the annotation tooltip. 1. (Optional) Click **Test annotation query** to ensure that the query is working properly. 1. Click **Close** when you've completed the query setup. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. ## How Prometheus annotations work Prometheus annotations work differently from SQL-based annotations. Instead of querying a table of events, you write a PromQL expression that returns time-series data. Grafana converts the query results into annotation events using these rules: - Grafana executes the PromQL query as a range query over the dashboard's time window. - **Every data point returned creates an annotation.** There is no automatic filtering of zero values. If you only want annotations at specific moments, your PromQL expression must filter the results (for example, using `> 0` or the `ALERTS` metric). - Grafana uses the field mapping configuration to determine what text, title, and tags to display for each annotation. - If the query returns multiple time series, each series produces its own set of annotations. {{< admonition type="note" >}} Because every returned data point creates an annotation, queries that return continuous data (like `node_cpu_seconds_total`) produce an annotation at every step interval, flooding your dashboard. Always use expressions that return data only at the moments you want annotated. {{< /admonition >}} ## Field mappings After entering your PromQL expression, use the field mapping drop-downs to control how query results are displayed as annotations. Grafana shows mapping options for: | Field | Description | Default behavior | | ----------- | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | **Time** | The timestamp for the annotation. | Uses the first time-type field (always present). | | **TimeEnd** | An end timestamp for range annotations, which display as a shaded region instead of a vertical line. | Not set (produces point annotations). | | **Title** | Short label displayed on the annotation marker. | Not set. | | **Text** | The annotation description displayed when you hover over it. | Uses the first string-type field, or the metric/label display name if configured. | | **Tags** | Comma-separated tags for the annotation. Helps categorize and filter annotations. | Not set. | To configure field mappings, select the appropriate field name from each drop-down, or enter a fixed text value. ## Example annotation queries The following examples show common annotation patterns with Prometheus. ### Alert-based annotations using `ALERTS` The most common and reliable way to create Prometheus annotations. Prometheus automatically generates an `ALERTS` metric for all configured alerting rules. ```promql ALERTS{alertstate="firing"} ``` This creates an annotation at every step interval where an alert is firing. The `ALERTS` metric includes labels such as: - `alertname`: The name of the alerting rule - `alertstate`: Either `firing` or `pending` - Any labels defined on the alerting rule To limit to specific alerts or severity levels: ```promql ALERTS{alertname="HighCPUUsage", severity="critical"} ``` Configure the field mappings: - **Text:** `alertname` (displays the alert name on hover) - **Tags:** `severity` (allows filtering by severity) ### Service restart annotations Display annotations when a process restarts. The `changes()` function detects when a value changes, and `> 0` ensures annotations only appear at the moment of change: ```promql changes(process_start_time_seconds{job="myservice"}[5m]) > 0 ``` ### Deployment annotations If you track deployments by pushing a timestamp metric via `Pushgateway` or a recording rule: ```promql changes(deployment_timestamp_seconds{environment="production"}[10m]) > 0 ``` Configure the field mappings: - **Text:** `environment` (shows which environment was deployed) - **Tags:** `environment` ### Threshold crossing annotations Display annotations when available memory drops below 10%: ```promql node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1 ``` {{< admonition type="note" >}} Comparison operators in PromQL act as filters. They only return data points where the condition is true. This means the expression above only returns data when memory is below 10%, and annotations only appear at those times. There's no need to add an outer `> 0` wrapper. {{< /admonition >}} ### Scaling event annotations Display annotations when the number of running pods changes: ```promql changes(kube_deployment_status_replicas{deployment="my-app"}[5m]) > 0 ``` ### Error spike annotations Display annotations when the error rate exceeds a threshold: ```promql sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job) > 0.05 ``` This creates annotations when the error rate exceeds 5%. ### Version or build change annotations If you expose build metadata using a metric like `build_info` (common in Go services), annotate when the version changes: ```promql changes(build_info{job="myservice"}[10m]) > 0 ``` Configure the field mappings: - **Text:** `version` (shows the new version in the annotation tooltip) - **Tags:** `job` If your build info metric uses a `version` label (for example, `build_info{version="1.2.3"}`), the annotation tooltip displays the version that was deployed. ## Control annotation density The **Min step** setting controls how many data points the query returns, which directly affects how many annotations appear. A larger step means fewer annotations: - **Min step `1m`:** Up to one annotation per minute (good for short time ranges). - **Min step `5m`:** Up to one annotation per 5 minutes (good for day-range dashboards). - **Min step `1h`:** Up to one annotation per hour (good for week-range dashboards). If your dashboard shows too many annotation markers, increase the Min step or add more specific filters to your query. ## Use template variables in annotations You can use template variables in your annotation queries to filter annotations based on dashboard variable selections: ```promql ALERTS{alertstate="firing", instance=~"$instance"} ``` ```promql changes(process_start_time_seconds{job="$job"}[5m]) > 0 ``` Template variables in annotations are resolved at query time using the current dashboard variable values. ## Range annotations Range annotations display as shaded regions rather than vertical lines, representing events with duration. To create range annotations, the query result must include a **TimeEnd** field mapping. Because PromQL doesn't natively return start/end time pairs, range annotations with Prometheus are limited to scenarios where you have separate metrics for start and end times (for example, via recording rules or custom metrics pushed to `Pushgateway`). For most use cases, point annotations (using the examples in this page) are the practical approach. If your annotations aren't appearing or you encounter errors, refer to [Troubleshoot Prometheus data source issues](https://grafana.com/docs/grafana//datasources/prometheus/troubleshooting/#annotation-errors). # Migrate from Prometheus SigV4 to Amazon Managed Service for Prometheus {{< admonition type="warning" >}} Using SigV4 authentication with the core Prometheus data source for Amazon Managed Service for Prometheus is **deprecated**. In Grafana 13, the migration to the dedicated [Amazon Managed Service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/) is automatic. Existing data sources using SigV4 are migrated on startup. {{< /admonition >}} For background on this change, refer to [Prometheus data source update: Redefining our big tent philosophy](https://grafana.com/blog/2025/06/16/prometheus-data-source-update-redefining-our-big-tent-philosophy/). ## What changed in Grafana 13 In Grafana 13, the `prometheusTypeMigration` feature toggle is enabled by default and deprecated. This means: - Prometheus data sources configured with SigV4 authentication are **automatically migrated** to the dedicated Amazon Managed Service for Prometheus plugin on Grafana startup. - You no longer need to manually enable the feature toggle. - Grafana Cloud users are migrated automatically with no action required. - Dashboards, alerts, and queries continue to work after migration without changes. ## Check migration status To determine if your Prometheus data sources have been migrated: 1. Navigate to **Connections** > **Data sources**. 1. Select your Prometheus data source. 1. Look for a migration banner at the top of the configuration page. The banner displays one of the following messages: - **"Migration Notice":** The data source has been migrated to the Amazon Managed Service for Prometheus plugin. - **"Deprecation Notice":** The data source hasn't been migrated yet. - **No banner:** No migration is needed (the data source doesn't use SigV4). ## Configure the Amazon Managed Service for Prometheus data source After migration (or for new setups), configure the dedicated plugin: 1. Navigate to **Connections** > **Data sources**. 1. Select your Amazon Managed Service for Prometheus data source. 1. In the **Auth** section, configure SigV4 authentication: | Setting | Description | Example | | --------------------------- | ---------------------------------------------- | --------------------------------------------------------------- | | **Authentication Provider** | Choose your auth method | `AWS SDK Default`, `Access & secret key`, or `Credentials file` | | **Default Region** | AWS region for your workspace | `us-west-2` | | **Access Key ID** | Your AWS access key (if using access key auth) | `AKIA...` | | **Secret Access Key** | Your AWS secret key (if using access key auth) | `wJalrXUtn...` | | **Assume Role ARN** | IAM role ARN (optional) | `arn:aws:iam::123456789:role/GrafanaRole` | 4. Set the **HTTP URL** to your Amazon Managed Service for Prometheus workspace endpoint: `https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-12345678-1234-1234-1234-123456789012/` 5. Click **Save & test** to verify the connection. ## Provision the data source ```yaml apiVersion: 1 datasources: - name: Amazon Managed Prometheus type: grafana-amazonprometheus-datasource url: https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-12345678-1234-1234-1234-123456789012/ jsonData: httpMethod: POST sigV4Auth: true sigV4AuthType: keys sigV4Region: us-east-2 secureJsonData: sigV4AccessKey: sigV4SecretKey: ``` Replace `` and `` with your AWS credentials. ## Troubleshoot migration issues The following sections cover common issues you may encounter during or after the migration and how to resolve them. ### Amazon Managed Service for Prometheus plugin not installed **Symptom:** Migration doesn't occur or the data source type is missing. **Solution:** 1. Navigate to **Connections** > **Add new connection** and search for "Amazon Managed Service for Prometheus". 1. Install the plugin if it isn't already installed. 1. For air-gapped environments, download the plugin from [the Grafana plugin catalog](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/) and install it manually. ### "401 Unauthorized" after migration **Symptom:** The migrated data source returns authentication errors. **Solution:** 1. **Self-managed Grafana:** Verify that `grafana-amazonprometheus-datasource` is included in `forward_settings_to_plugins` under the `[aws]` heading in your `.ini` configuration file. 1. **Grafana Cloud:** Contact [Grafana Support](https://grafana.com/profile/org#support). ### Rollback the migration If you need to revert migrated data sources back to the core Prometheus type: 1. Set `prometheusTypeMigration` to `false` in your Grafana configuration feature toggles. For more information, refer to [Manage feature toggles](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/feature-toggles/#manage-feature-toggles). 1. Restart Grafana. 1. Obtain a bearer token with `read` and `write` permissions for the data source API. For more information, refer to [Data source API](https://grafana.com/docs/grafana//developers/http_api/data_source/). 1. Run the following rollback script, providing your Grafana URL and bearer token: ```sh #!/bin/bash GRAFANA_URL="" BEARER_TOKEN="" LOG_FILE="grafana_migration_rollback_$(date +%Y%m%d_%H%M%S).log" log_message() { local message="$1" local timestamp=$(date '+%Y-%m-%d %H:%M:%S') echo "[$timestamp] $message" | tee -a "$LOG_FILE" } update_data_source() { local uid="$1" local data="$2" response=$(curl -s -w "\n%{http_code}" -X PUT \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BEARER_TOKEN" \ -d "$data" \ "$GRAFANA_URL/api/datasources/uid/$uid") http_code=$(echo "$response" | tail -n1) response_body=$(echo "$response" | sed '$d') if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then log_message "$uid reverted successfully" else log_message "$uid error: HTTP $http_code - $response_body" fi } if ! command -v jq &> /dev/null; then echo "Error: jq is required but not installed." exit 1 fi if [[ -z "$GRAFANA_URL" || -z "$BEARER_TOKEN" ]]; then echo "Error: Set GRAFANA_URL and BEARER_TOKEN variables at the top of the script." exit 1 fi log_message "Starting AMP to Prometheus rollback" response=$(curl -s -w "\n%{http_code}" -X GET \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BEARER_TOKEN" \ "$GRAFANA_URL/api/datasources/") http_code=$(echo "$response" | tail -n1) response_body=$(echo "$response" | sed '$d') if [[ "$http_code" -lt 200 || "$http_code" -ge 300 ]]; then log_message "Error fetching data sources: HTTP $http_code" exit 1 fi total=$(echo "$response_body" | jq '[.[] | select(.jsonData["prometheus-type-migration"] == true and .type == "grafana-amazonprometheus-datasource")] | length') log_message "Found $total data sources to revert" echo "$response_body" | jq -c '.[] | select(.jsonData["prometheus-type-migration"] == true and .type == "grafana-amazonprometheus-datasource")' | while read -r data; do uid=$(echo "$data" | jq -r '.uid') read_only=$(echo "$data" | jq -r '.readOnly // false') if [[ "$read_only" == "true" ]]; then log_message "$uid is readOnly; edit the type to 'prometheus' in the provisioning file instead." continue fi updated_data=$(echo "$data" | jq '.type = "prometheus" | .jsonData["prometheus-type-migration"] = false') update_data_source "$uid" "$updated_data" done log_message "Rollback complete. Log: $LOG_FILE" ``` {{< admonition type="note" >}} Provisioned data sources (`readOnly`) can't be reverted via the API. Update the `type` field to `prometheus` in your provisioning YAML file instead. {{< /admonition >}} If you continue to experience issues, check the Grafana server logs for detailed error messages and contact [Grafana Support](https://grafana.com/help/). # Migrate from Prometheus Azure AD to Azure Monitor Managed Service for Prometheus {{< admonition type="warning" >}} Using Azure AD authentication with the core Prometheus data source for Azure Monitor Managed Service for Prometheus is **deprecated**. In Grafana 13, the migration to the dedicated [Azure Monitor Managed Service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-azureprometheus-datasource/) is automatic. Existing data sources using Azure AD authentication are migrated on startup. {{< /admonition >}} For background on this change, refer to [Prometheus data source update: Redefining our big tent philosophy](https://grafana.com/blog/2025/06/16/prometheus-data-source-update-redefining-our-big-tent-philosophy/). ## What changed in Grafana 13 In Grafana 13, the `prometheusTypeMigration` feature toggle is enabled by default and deprecated. This means: - Prometheus data sources configured with Azure AD authentication are **automatically migrated** to the dedicated Azure Monitor Managed Service for Prometheus plugin on Grafana startup. - You no longer need to manually enable the feature toggle. - Grafana Cloud users are migrated automatically with no action required. - Dashboards, alerts, and queries continue to work after migration without changes. ## Check migration status To determine if your Prometheus data sources have been migrated: 1. Navigate to **Connections** > **Data sources**. 1. Select your Prometheus data source. 1. Look for a migration banner at the top of the configuration page. The banner displays one of the following messages: - **"Migration Notice":** The data source has been migrated to the Azure Monitor Managed Service for Prometheus plugin. - **"Deprecation Notice":** The data source hasn't been migrated yet. - **No banner:** No migration is needed (the data source doesn't use Azure AD authentication). ## Configure the Azure Monitor Managed Service for Prometheus data source After migration (or for new setups), configure the dedicated plugin: 1. Navigate to **Connections** > **Data sources**. 1. Select your Azure Monitor Managed Service for Prometheus data source. 1. In the **Authentication** section, select your authentication method: | Method | Use case | Additional configuration required | | -------------------- | ----------------------------------- | --------------------------------------------------- | | **Managed Identity** | Azure-hosted Grafana instances | None (system-assigned) or Client ID (user-assigned) | | **App Registration** | Service principal authentication | Directory ID, Application ID, Client secret | | **Current User** | Current user's Entra ID credentials | None | For Managed Identity authentication: - No additional configuration is required if using a system-assigned identity. - For a user-assigned identity, provide the **Client ID**. For App Registration authentication: | Setting | Description | Example | | --------------------------- | ------------------------------- | -------------------------------------- | | **Directory (tenant) ID** | Your Entra ID tenant ID | `12345678-1234-1234-1234-123456789012` | | **Application (client) ID** | Your app registration client ID | `87654321-4321-4321-4321-210987654321` | | **Client secret** | Your app registration secret | `your-client-secret` | To learn more about Entra ID authentication for Grafana, refer to [Configure Entra ID OAuth authentication](https://grafana.com/docs/grafana//setup-grafana/configure-access/configure-authentication/azuread/#configure-azure-adentra-id-oauth-authentication). 4. Set the **Prometheus server URL** to your Azure Monitor workspace endpoint: ``` https://your-workspace.eastus2.prometheus.monitor.azure.com ``` 5. Click **Save & test** to verify the connection. ## Provision the data source ```yaml apiVersion: 1 datasources: - name: Azure Monitor Prometheus type: grafana-azureprometheus-datasource url: https://your-workspace.eastus2.prometheus.monitor.azure.com jsonData: azureCredentials: authType: clientsecret azureCloud: AzureCloud clientId: tenantId: httpMethod: POST secureJsonData: azureClientSecret: ``` Replace ``, ``, and `` with your Azure credentials. ## Troubleshoot migration issues The following sections cover common issues you may encounter during or after the migration and how to resolve them. ### Azure Monitor Managed Service for Prometheus plugin not installed **Symptom:** Migration doesn't occur or the data source type is missing. **Solution:** 1. Navigate to **Connections** > **Add new connection** and search for "Azure Monitor Managed Service for Prometheus". 1. Install the plugin if it isn't already installed. 1. For air-gapped environments, download the plugin from [the Grafana plugin catalog](https://grafana.com/grafana/plugins/grafana-azureprometheus-datasource/) and install it manually. ### "401 Unauthorized" after migration **Symptoms:** - The migrated data source returns `401 Unauthorized` errors on **Save & test** or when running queries. - The data source authenticates with **Managed Identity**, **Workload Identity**, or **Current User**. **Cause:** The Azure Monitor Managed Service for Prometheus plugin doesn't receive the Azure authentication settings from the Grafana server. Either the `[azure]` section of your `.ini` configuration file is missing the options required by your authentication method, or `forward_settings_to_plugins` doesn't include the `grafana-azureprometheus-datasource` plugin. **Solution for self-managed Grafana:** 1. Set the options required by your authentication method in the `[azure]` section of your `.ini` configuration file. For a full description of each option, refer to the [`[azure]` configuration reference](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#azure). - **Managed Identity:** Set `managed_identity_enabled = true`. For a user-assigned identity, also set `managed_identity_client_id` to the identity's client ID. Leave it empty for a system-assigned identity. - **Workload Identity:** Set `workload_identity_enabled = true`. - **Current User:** Set `user_identity_enabled = true`. 1. If you've customized `forward_settings_to_plugins`, make sure `grafana-azureprometheus-datasource` is listed. By default, `forward_settings_to_plugins` includes all Grafana Labs Azure plugins, so the Prometheus plugin already receives these settings. For example: ```ini [azure] managed_identity_enabled = true forward_settings_to_plugins = grafana-azureprometheus-datasource ``` 1. Verify that the identity has the required Azure role, such as **Monitoring Data Reader**, on the target Azure Monitor workspace. 1. Restart Grafana after you change the server configuration. **Solution for Grafana Cloud:** Contact [Grafana Support](https://grafana.com/profile/org#support). ### Rollback the migration If you need to revert migrated data sources back to the core Prometheus type: 1. Set `prometheusTypeMigration` to `false` in your Grafana configuration feature toggles. For more information, refer to [Manage feature toggles](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/feature-toggles/#manage-feature-toggles). 1. Restart Grafana. 1. Obtain a bearer token with `read` and `write` permissions for the data source API. For more information, refer to [Data source API](https://grafana.com/docs/grafana//developers/http_api/data_source/). 1. Run the following rollback script, providing your Grafana URL and bearer token: ```sh #!/bin/bash GRAFANA_URL="" BEARER_TOKEN="" LOG_FILE="grafana_azure_migration_rollback_$(date +%Y%m%d_%H%M%S).log" log_message() { local message="$1" local timestamp=$(date '+%Y-%m-%d %H:%M:%S') echo "[$timestamp] $message" | tee -a "$LOG_FILE" } update_data_source() { local uid="$1" local data="$2" response=$(curl -s -w "\n%{http_code}" -X PUT \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BEARER_TOKEN" \ -d "$data" \ "$GRAFANA_URL/api/datasources/uid/$uid") http_code=$(echo "$response" | tail -n1) response_body=$(echo "$response" | sed '$d') if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then log_message "$uid reverted successfully" else log_message "$uid error: HTTP $http_code - $response_body" fi } if ! command -v jq &> /dev/null; then echo "Error: jq is required but not installed." exit 1 fi if [[ -z "$GRAFANA_URL" || -z "$BEARER_TOKEN" ]]; then echo "Error: Set GRAFANA_URL and BEARER_TOKEN variables at the top of the script." exit 1 fi log_message "Starting Azure Prometheus to core Prometheus rollback" response=$(curl -s -w "\n%{http_code}" -X GET \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BEARER_TOKEN" \ "$GRAFANA_URL/api/datasources/") http_code=$(echo "$response" | tail -n1) response_body=$(echo "$response" | sed '$d') if [[ "$http_code" -lt 200 || "$http_code" -ge 300 ]]; then log_message "Error fetching data sources: HTTP $http_code" exit 1 fi total=$(echo "$response_body" | jq '[.[] | select(.jsonData["prometheus-type-migration"] == true and .type == "grafana-azureprometheus-datasource")] | length') log_message "Found $total data sources to revert" echo "$response_body" | jq -c '.[] | select(.jsonData["prometheus-type-migration"] == true and .type == "grafana-azureprometheus-datasource")' | while read -r data; do uid=$(echo "$data" | jq -r '.uid') read_only=$(echo "$data" | jq -r '.readOnly // false') if [[ "$read_only" == "true" ]]; then log_message "$uid is readOnly; edit the type to 'prometheus' in the provisioning file instead." continue fi updated_data=$(echo "$data" | jq '.type = "prometheus" | .jsonData["prometheus-type-migration"] = false') update_data_source "$uid" "$updated_data" done log_message "Rollback complete. Log: $LOG_FILE" ``` {{< admonition type="note" >}} Provisioned data sources (`readOnly`) can't be reverted via the API. Update the `type` field to `prometheus` in your provisioning YAML file instead. {{< /admonition >}} If you continue to experience issues, check the Grafana server logs for detailed error messages and contact [Grafana Support](https://grafana.com/help/). # Troubleshoot Prometheus data source issues This document provides troubleshooting information for common errors you may encounter when using the Prometheus data source in Grafana. ## Connection errors The following errors occur when Grafana cannot establish or maintain a connection to Prometheus. ### Failed to connect to Prometheus **Error message:** "There was an error returned querying the Prometheus API" **Cause:** Grafana cannot establish a network connection to the Prometheus server. **Solution:** 1. Verify that the Prometheus server URL is correct in the data source configuration. 1. Check that Prometheus is running and accessible from the Grafana server. 1. Ensure the URL includes the protocol (`http://` or `https://`). 1. Verify the port is correct (the Prometheus default port is `9090`). 1. Ensure there are no firewall rules blocking the connection. 1. If Grafana and Prometheus are running in separate containers, use the container IP address or hostname instead of `localhost`. 1. For Grafana Cloud, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if your Prometheus instance is not publicly accessible. ### Request timed out **Error message:** "context deadline exceeded" or "request timeout" **Cause:** The connection to Prometheus timed out before receiving a response. **Solution:** 1. Check the network latency between Grafana and Prometheus. 1. Verify that Prometheus is not overloaded or experiencing performance issues. 1. Increase the **Query timeout** setting in the data source configuration under **Interval behavior**. 1. Check the [Grafana server timeout configuration](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana#timeout) for server-level timeout settings. 1. Reduce the time range or complexity of your query. 1. Check if any network devices (load balancers, proxies) are timing out the connection. ### Failed to parse data source URL **Error message:** "Failed to parse data source URL" **Cause:** The URL entered in the data source configuration is not valid. **Solution:** 1. Verify the URL format is correct (for example, `http://localhost:9090` or `https://prometheus.example.com:9090`). 1. Ensure the URL includes the protocol (`http://` or `https://`). 1. Remove any trailing slashes or invalid characters from the URL. ### Data doesn't appear in Metrics Drilldown or Explore **Symptom:** You've successfully tested the data source connection, but no metric data appears in Explore or Metrics Drilldown. **Cause:** The wrong data source is selected, or the data source name doesn't match expectations. **Solution:** 1. Verify you've selected the correct data source from the data source drop-down menu. 1. When using `remote_write` to send metrics to Grafana Cloud, the data source name follows the convention `grafanacloud--prom`. 1. Check that metrics are being actively scraped by querying `up` in the Explore view. 1. If using Grafana Cloud, verify the `remote_write` endpoint URL and credentials are correct. ### PDC connectivity errors **Error messages:** `host unreachable`, `EOF`, `network unreachable`, `connection reset by peer`, `dial tcp: lookup ... no such host` **Symptom:** Prometheus queries fail intermittently or consistently when using Private data source connect (PDC) to reach a Prometheus instance behind a private network. The data source test may pass occasionally but queries fail under load. **Cause:** PDC tunnels traffic through an SSH connection between Grafana Cloud and your PDC agent. Connectivity failures are most commonly caused by DNS resolution issues, network configuration on the customer side, or the PDC agent's default connection limits being too low for the query volume. **Solutions:** 1. **Verify DNS resolution from the PDC agent host.** The PDC agent must be able to resolve the Prometheus hostname from the machine it runs on. Run `nslookup` or `dig` for the Prometheus URL from the agent host to confirm. 1. **Check network connectivity from the agent.** Ensure the PDC agent can reach the Prometheus endpoint directly (for example, `curl http://prometheus-host:9090/-/healthy` from the agent machine). 1. **Increase parallel SSH connections.** The PDC agent defaults to 1 parallel SSH connection, which can bottleneck under load from multiple alert evaluations or dashboard queries. Increase this by setting the `--ssh-connections` flag (or `PDC_SSH_CONNECTIONS` environment variable) to a higher value (for example, 4 or 8): ```sh pdc-agent --ssh-connections=4 ``` 1. **Check firewall rules.** Ensure the PDC agent's outbound SSH connection to Grafana Cloud isn't being interrupted by firewall rules, NAT gateways, or idle connection timeouts. 1. **Verify the PDC agent is running and healthy.** Check agent logs for connection errors or restarts. The agent must maintain a persistent connection to Grafana Cloud. 1. **Check for idle timeout issues.** If the connection drops after periods of inactivity, configure TCP keepalive on the host or add a `keepalive` setting to the PDC agent configuration. {{< admonition type="note" >}} PDC connectivity issues are almost always caused by networking on the customer side (DNS, firewall rules, routing), not by Grafana Cloud. The data source test passing doesn't guarantee sustained connectivity under load. It only verifies a single query succeeds. {{< /admonition >}} For general PDC setup and configuration, refer to [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) and [Configure PDC](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/). ### Certificate verification failed **Error message:** "x509: certificate signed by unknown authority" or "certificate verify failed" **Cause:** Grafana cannot verify the TLS certificate presented by Prometheus. **Solution:** 1. If using a self-signed certificate, enable **Add self-signed certificate** in the TLS settings and add your CA certificate. 1. Verify the certificate chain is complete and valid. 1. Ensure the certificate has not expired. 1. As a temporary workaround for testing, enable **Skip TLS verify** (not recommended for production). ### TLS handshake error **Error message:** "TLS: handshake failure" or "connection reset" **Cause:** The TLS handshake between Grafana and Prometheus failed. **Solution:** 1. Verify that Prometheus is configured to use TLS. 1. Check that the TLS version and cipher suites are compatible. 1. If using client certificates, ensure they are correctly configured in the **TLS client authentication** section. 1. Verify the server name matches the certificate's Common Name or Subject Alternative Name. ## Authentication errors The following errors occur when there are issues with authentication credentials or permissions. ### Unauthorized (401) **Error message:** "401 Unauthorized" or "Authorization failed" **Cause:** The authentication credentials are invalid or missing. **Solution:** 1. Verify that the username and password are correct if using basic authentication. 1. Check that the authentication method selected matches your Prometheus configuration. 1. If using a reverse proxy with authentication, verify the credentials are correct. 1. For AWS SigV4 authentication, verify the IAM credentials and permissions. Alternatively, consider using the [Amazon Managed Service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/) for simplified AWS authentication. ### OAuth token expiration errors (GCP and Azure) **Error messages:** "ACCESS_TOKEN_EXPIRED", "401 Unauthorized" in alerting but not in Explore **Symptom:** Queries in Explore and dashboards work correctly, but alert rule evaluations fail intermittently with 401 errors. This is most common with Google Managed Prometheus (GMP) and Azure-managed Prometheus endpoints using OAuth/OIDC authentication. **Cause:** The Grafana alerting backend and the interactive query path (Explore, dashboards) handle credential refreshes differently. The alerting evaluator can use a cached OAuth token beyond its expiry window due to a token staleness check issue in the Prometheus data source. This causes alerting to fail with expired credentials while interactive queries succeed because they trigger a fresh token exchange. **Solutions:** For Google Managed Prometheus (GMP): 1. Use the **GCP datasource-syncer** pattern: run a sidecar process that refreshes OAuth tokens and updates the Grafana data source credentials on a schedule shorter than the token lifetime (typically every 45 minutes for a 60-minute token). This ensures the stored token is always valid when the alerting backend uses it. 1. Alternatively, use the [Google Cloud Monitor data source](https://grafana.com/grafana/plugins/stackdriver/) with its built-in GCP credential management rather than pointing the Prometheus data source at a GMP endpoint. 1. If running on GKE with Workload Identity, ensure the Kubernetes service account token refresh is functioning and the projected token volume is mounted correctly. For Azure Managed Prometheus: 1. Verify the Azure AD app registration's client secret hasn't expired. 1. If using Managed Identity, ensure the Grafana instance's identity has the **Monitoring Data Reader** role on the Azure Monitor workspace. 1. Check that **Forward OAuth identity** is not enabled alongside Azure AD authentication. Both use the same HTTP authorization headers and conflict with each other. General steps: 1. Check the Grafana server logs for token refresh errors around the time alerts fail. 1. Verify the data source test (**Save & test**) passes. This confirms current credentials are valid but doesn't guarantee the alerting backend has a fresh token. 1. If the issue persists, set **Alert state if execution error or timeout** to **Keep Last State** to prevent false alarms while investigating. Refer to [Configure alert state for execution errors](https://grafana.com/docs/grafana//datasources/prometheus/alerting/#configure-alert-state-for-execution-errors). {{< admonition type="note" >}} This token caching behavior is a known issue that has received code fixes in recent Grafana releases. If you're experiencing this on an older Grafana version, upgrading may resolve it. Check the [Grafana changelog](https://grafana.com/docs/grafana//whatsnew/) for relevant fixes. {{< /admonition >}} ### LBAC not restricting data on non-Mimir backends **Symptom:** You've configured Team LBAC rules, but users can still see all metrics regardless of their team assignments. **Cause:** Label-Based Access Control (LBAC) for the Prometheus data source only works when the backend is **Grafana Cloud Metrics (Mimir)** or **Grafana Enterprise Metrics (GEM)**. It doesn't work with Google Managed Prometheus, self-managed Prometheus, Thanos, or other Prometheus-compatible endpoints. The LBAC enforcement relies on Mimir-specific HTTP headers (`X-Scope-OrgID` and team-scoped label matchers) that other backends ignore. **Solution:** 1. Verify your backend is Grafana Cloud Metrics or GEM. If you're using a different Prometheus-compatible backend, LBAC isn't supported. 1. For Google Managed Prometheus or other external endpoints, use [data source permissions](https://grafana.com/docs/grafana//administration/data-source-management/#data-source-permissions) to control which teams can access the data source entirely, rather than per-label access control. 1. If you need per-label restrictions on a non-Mimir backend, consider proxying through a Mimir instance or using a separate data source per team with different credentials scoped to the appropriate data. ### Azure AD or SigV4 authentication options not available **Symptom:** The Azure AD or SigV4 authentication options don't appear in the authentication drop-down when configuring the Prometheus data source. **Cause:** These authentication methods require server-side feature flags that aren't enabled by default, particularly on Grafana Cloud. **Solution:** 1. **Grafana Cloud:** Contact [Grafana Support](https://grafana.com/profile/org#support) to request the feature flag be enabled for your stack. This isn't a self-service setting. 1. **Self-managed Grafana:** Add the required setting to your Grafana configuration file: - For Azure AD: set `azure_auth_enabled = true` under `[auth]` - For SigV4: set `sigv4_auth_enabled = true` under `[auth]` 1. **For Amazon Managed Prometheus:** Consider using the dedicated [Amazon Managed Service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/) instead, which handles AWS authentication natively without needing the SigV4 flag. ### Forbidden (403) **Error message:** "403 Forbidden" or "Access denied" **Cause:** The authenticated user does not have permission to access the requested resource. **Solution:** 1. Verify the user has read access to the Prometheus API. 1. Check Prometheus security settings and access control configuration. 1. If using a reverse proxy, verify the proxy is not blocking the request. 1. For AWS Managed Prometheus, verify the IAM policy grants the required permissions. Alternatively, consider using the [Amazon Managed Service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/) for simplified AWS authentication. ## Query errors The following errors occur when there are issues with PromQL syntax or query execution. ### Query syntax error **Error message:** `parse error: unexpected character` or `bad_data: 1:X: parse error` **Cause:** The PromQL query contains invalid syntax. **Alternative cause:** A proxy between Grafana and Prometheus requires authentication. When proxy authentication fails, the proxy redirects the request to an HTML authentication page. Grafana cannot parse the HTML response, which results in a parse error. This appears to be a query issue but is actually a proxy authentication issue. **Solution:** 1. Check your query syntax for typos or invalid characters. 1. Verify that metric names and label names are valid identifiers. 1. Ensure string values in label matchers are enclosed in quotes. 1. Use the Prometheus expression browser to test your query directly. 1. Refer to the [Prometheus querying documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) for syntax guidance. 1. If you have a proxy between Grafana and Prometheus, verify that proxy authentication is correctly configured. Check your proxy logs for authentication failures or redirects. ### Query returns no data for a metric **Symptom:** The query returns no data and the visualization is empty. **Cause:** The specified metric does not exist in Prometheus, or there is no data for the selected time range. **Solution:** 1. Verify the metric name is spelled correctly. 1. Check that the metric is being scraped by Prometheus. 1. Use the Prometheus API to browse available metrics at `/api/v1/label/__name__/values`. 1. Use the [target metadata API](https://prometheus.io/docs/prometheus/latest/querying/api#querying-target-metadata) to verify which metrics a target exposes. 1. Verify the time range includes data for the metric. ### Query timeout limit exceeded **Error message:** "query timed out in expression evaluation" or "query processing would load too many samples" **Cause:** The query took longer than the configured timeout limit or would return too many samples. **Solution:** 1. Reduce the time range of your query. 1. Add more specific label filters to limit the data scanned. 1. Increase the **Query timeout** setting in the data source configuration. 1. Use aggregation functions like `sum()`, `avg()`, or `rate()` to reduce the number of time series. 1. Increase the `query.timeout` or `query.max-samples` settings in Prometheus if you have admin access. ### Too many time series **Error message:** `exceeded maximum resolution of 11,000 points per timeseries` or `maximum number of series limit exceeded` **Cause:** The query is returning more time series or data points than the configured limits allow. **Solution:** 1. Reduce the time range of your query. 1. Add label filters to limit the number of time series returned. 1. Increase the **Min interval** or **Resolution** in the query options to reduce the number of data points. 1. Use aggregation functions to combine time series. 1. Adjust the **Series limit** setting in the data source configuration under **Other settings**. ### Memory limit exceeded for high-cardinality queries **Error messages:** "max-estimated-memory-consumption-per-query limit exceeded", "query requires too much memory", "max samples limit reached" **Symptom:** Queries against high-cardinality metrics (thousands of unique label combinations) over long time ranges (days or weeks) fail with memory or sample limit errors. Short time ranges may work but expanding the range causes the failure. **Cause:** Prometheus and Mimir enforce per-query memory and sample limits to protect the system from resource exhaustion. High-cardinality metrics (for example, metrics with a `pod`, `request_id`, or `user_id` label) multiplied by long time ranges produce result sets that exceed these limits. **Solutions:** 1. **Reduce the query scope:** - Shorten the time range. - Add label matchers to select fewer series (for example, filter to a specific `namespace` or `job`). - Increase the **Min step** or use a larger step interval to reduce the number of data points per series. 1. **Use recording rules to pre-aggregate:** Create [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) that pre-compute the aggregation you need. For example, if you commonly query `sum(rate(http_requests_total[5m])) by (service)`, create a recording rule for it and query the pre-aggregated metric instead. ```yaml groups: - name: aggregations rules: - record: service:http_requests_total:rate5m expr: sum(rate(http_requests_total[5m])) by (service) ``` 1. **Use Adaptive Metrics (Grafana Cloud):** If you're on Grafana Cloud, [Adaptive Metrics](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/reduce-costs/metrics-costs/control-metrics-usage-via-adaptive-metrics/) automatically identifies and aggregates high-cardinality metrics that aren't being queried at full resolution, reducing storage and query costs. 1. **Restructure dashboards for high-cardinality data:** - Use template variables to scope queries to a subset of labels rather than querying all series at once. - Split long-range overviews (weekly/monthly) into separate panels that use pre-aggregated recording rules. - Use the **Max data points** setting in the query options to cap the resolution for overview panels. 1. **Increase limits (self-managed only):** If you have admin access to Prometheus or Mimir, you can increase limits in the server configuration: - Prometheus: `--query.max-samples` flag - Mimir: `-querier.max-samples`, `-querier.max-estimated-memory-consumption-per-query` {{< admonition type="note" >}} Increasing limits allows larger queries to succeed but also increases the risk of resource exhaustion. Prefer reducing query scope or using recording rules over raising limits. {{< /admonition >}} ### Invalid function or aggregation **Error message:** "unknown function" or "parse error: unexpected aggregation" **Cause:** The query uses an invalid or unsupported PromQL function. **Solution:** 1. Verify the function name is spelled correctly and is a valid PromQL function. 1. Check that you are using the correct syntax for the function. 1. Ensure your Prometheus version supports the function you are using. 1. Refer to the [PromQL functions documentation](https://prometheus.io/docs/prometheus/latest/querying/functions/) for available functions. ### `rate()` or `increase()` returning unexpected values **Symptom:** `increase()` returns fractional values on integer counters, `rate()` shows an ever-increasing value instead of a steady per-second rate, or counter resets cause large spikes in visualizations. **Possible causes and solutions:** | Cause | Solution | | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `increase()` fractional values | Expected behavior. Prometheus uses linear interpolation. Use `ceil()` or `floor()` if you need integers. | | `rate()` grows over time | Multiple instances write to the same series without unique labels. Ensure each target has unique `instance`/`pod` labels and aggregate with `sum by`. | | Counter reset spikes after pod restarts | Use `$__rate_interval` or a longer range vector to smooth spikes. Investigate frequent restarts as the root cause. | | Values differ between edit mode and dashboard | Panel width affects `$__interval` which affects `rate()` window calculations. Set a **Min step** on the query. | For detailed explanations of these behaviors, refer to [Expected PromQL behaviors](https://grafana.com/docs/grafana//datasources/prometheus/query-editor/#expected-promql-behaviors). ### Aggregation by labels with dots **Symptom:** Queries that aggregate by label names containing dots (for example, `container.name`) return incorrect or incomplete results. **Cause:** Prior to Grafana 13, there was a bug where labels with dots in their names were not handled correctly during aggregation operations like `sum by` or `avg by`. **Solution:** 1. Upgrade to Grafana 13 or later, which correctly handles labels with dots in aggregation queries. 1. Verify your query uses the correct label name with dots (for example, `sum by (container.name) (metric_name)`). ## Configuration errors The following errors occur when the data source is not configured correctly. ### Invalid Prometheus type **Error message:** Unexpected behavior when querying metrics or labels **Cause:** The **Prometheus type** setting does not match your actual Prometheus-compatible database. **Solution:** 1. Open the data source configuration in Grafana. 1. Under **Performance**, select the correct **Prometheus type** (Prometheus, Cortex, Mimir, or Thanos). 1. Different database types support different APIs, so setting this incorrectly may cause unexpected behavior. ### Scrape interval mismatch **Symptom:** Data appears sparse, or `rate()` queries return no data or incomplete results. **Cause:** The **Scrape interval** setting in Grafana does not match the actual scrape interval in Prometheus. This especially affects `rate()` queries, which require at least two data points within the specified time window. For example, if your actual scrape interval is 5 minutes but Grafana uses the default (15 seconds for OSS, 1 minute for Grafana Cloud), a query like `rate(http_requests_total[1m])` returns no data because there are no data points within that 1-minute window. **Solution:** 1. Check your Prometheus configuration file for the `scrape_interval` setting. 1. Update the **Scrape interval** in the Grafana data source configuration under **Interval behavior** to match. 1. Use `$__rate_interval` instead of hard-coded time windows in `rate()` queries. This variable automatically adjusts based on your scrape interval. 1. For more information, refer to [$\_\_rate_interval for Prometheus rate queries that just work](https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/). ### `$__rate_interval` returns no data or incorrect values **Symptom:** Queries using `$__rate_interval` return no data, return different values in edit mode versus the dashboard, or produce unexpected gaps. **Cause:** `$__rate_interval` is calculated as `max($__interval + scrape_interval, 4 * scrape_interval)`. If any input to this formula is incorrect, the resulting window is wrong: either too small (no data) or inconsistent across contexts. **Common causes and solutions:** | Cause | Solution | | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | Data source scrape interval left at default `15s` while actual Prometheus scrape interval is longer (for example, `60s`) | Set the **Scrape interval** under **Interval behavior** in the data source configuration to match your Prometheus `scrape_interval`. | | Query works in edit mode but shows gaps on the dashboard | Panel size affects `$__interval`. Smaller panels produce larger intervals. Set a **Min step** on the query to enforce a consistent floor. | | LBAC-enabled data source doesn't inherit scrape interval | Set the **Min step** explicitly on each query panel rather than relying on data source inheritance. | | Using `$__rate_interval` in recording rules or alerting | Use a fixed interval (for example, `[5m]`) instead of `$__rate_interval` in contexts without a panel/dashboard. | **To debug the current value:** 1. Open the query inspector in a panel (click the panel title, then **Inspect** > **Query**). 1. Look at the expanded query sent to Prometheus. The actual interval value replacing `$__rate_interval` is visible in the request. 1. Compare this value against your actual scrape interval. If it's smaller than your scrape interval, you need to configure the data source scrape interval or set a Min step. For detailed documentation on how `$__rate_interval` works and how to configure it, refer to [Use `$__rate_interval`](https://grafana.com/docs/grafana//datasources/prometheus/template-variables/#use-__rate_interval). ## Performance issues The following issues affect query speed and data freshness. ### Slow query performance **Symptom:** Queries take a long time to execute, dashboards are slow to load, or the loading spinner persists. **Cause:** Queries scan too much data, the Prometheus server is overloaded, or the network connection is slow. **Solution:** 1. Reduce the time range of your query. 1. Add more specific label filters to limit the data scanned. 1. Increase the **Min step** in the query options to reduce the number of data points. 1. Check Prometheus server performance and resource utilization. 1. Enable **Disable metrics lookup** in the data source configuration for large Prometheus instances. 1. Enable **Incremental querying (beta)** to cache query results. 1. Use recording rules to pre-aggregate frequently queried data. 1. For high-cardinality metrics, refer to [Memory limit exceeded for high-cardinality queries](#memory-limit-exceeded-for-high-cardinality-queries). ### Data appears delayed or missing recent points **Symptom:** The visualization doesn't show the most recent data, even after refreshing. **Cause:** Scrape timing, clock drift, or dashboard refresh settings. **Solution:** 1. Check the dashboard time range and refresh settings. 1. Verify the **Scrape interval** is configured correctly in the data source settings. 1. Ensure Prometheus has finished scraping the target (there's a delay between the scrape interval and data availability). 1. Check for clock synchronization issues between Grafana and Prometheus (use NTP). 1. For `rate()` and similar functions, the most recent partial scrape interval won't have enough data points. This is expected. ### Exemplars not showing **Symptom:** Exemplar data doesn't appear on graphs even though you expect it. **Cause:** Exemplars require specific configuration in both the data source and the query editor. **Solution:** 1. Verify that exemplars are configured in the data source settings under **Exemplars** (at least one exemplar link must be defined). 1. Check that your Prometheus version supports exemplars (2.26+). 1. Ensure your instrumented application is actually sending exemplar data with metrics. 1. Verify the tracing data source is correctly configured for the exemplar link. 1. Enable the **Exemplars** toggle in the query editor for the specific query. 1. Exemplars only appear with **Range** query type, not **Instant**. ## Annotation errors The following issues occur when using Prometheus as a data source for annotations. ### Annotations not appearing **Symptom:** You've configured a Prometheus annotation query, but no annotations appear on your dashboard. **Possible causes and solutions:** | Cause | Solution | | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Query returns no data in the current time range | Verify the query returns results in Explore for the dashboard's time range. | | Query returns continuous data (too many annotations) | Every returned data point creates an annotation. If the query returns hundreds of points, annotations may render but are too dense to see. Increase the **Min step** or refine your query to only return data at event moments. | | Wrong data source selected | Verify the correct Prometheus data source is selected in the annotation configuration. | | Annotation is disabled | Check that the annotation toggle is enabled (eye icon) in the dashboard's annotation settings. | | Time range mismatch | Expand the dashboard time range to include the events you expect to see. | {{< admonition type="note" >}} Prometheus annotations create a marker for **every data point** returned by the query. There's no automatic filtering of zero values. If you only want annotations at specific moments, your PromQL expression must return data only at those times (for example, using `> 0`, `changes() > 0`, or the `ALERTS` metric). {{< /admonition >}} For more information on configuring annotations, refer to [Prometheus annotations](https://grafana.com/docs/grafana//datasources/prometheus/annotations/). ## Alerting errors The following issues occur when using Prometheus with Grafana Alerting. ### Transient alert errors triggering false alarms **Error messages:** `sse.dependencyError`, `sse.dataQueryError`, "context deadline exceeded", "i/o timeout" **Symptom:** Alert rules intermittently fire due to execution errors rather than genuine threshold breaches. On-call teams receive false positive notifications. Alert state history shows error states caused by transient backend issues (network blips, HTTP 502/500 responses, timeouts) rather than actual metric conditions being met. **Cause:** By default, when an alert rule encounters an execution error or timeout, Grafana sets the alert state to **Alerting**, which fires the alert. Transient connectivity issues between Grafana and Prometheus (i/o timeouts, deadline exceeded, brief outages) trigger this behavior even though the underlying metric hasn't crossed its threshold. **Solution:** 1. Open each affected alert rule for editing. 1. In the alert conditions section, change **Alert state if execution error or timeout** from **Alerting** to **Keep Last State**. 1. Save the rule. This ensures the alert retains its previous state during transient errors and only fires when a successful evaluation confirms the threshold is breached. **If errors are frequent**, also investigate: 1. Network stability between Grafana and Prometheus. 1. Prometheus resource utilization (CPU, memory, disk I/O). 1. The **Query timeout** setting in the data source configuration. Increase it if complex queries regularly exceed the limit. 1. Query complexity. Simplify queries or use recording rules to pre-compute expensive expressions. For configuration details, refer to [Configure alert state for execution errors](https://grafana.com/docs/grafana//datasources/prometheus/alerting/#configure-alert-state-for-execution-errors). ### Alert rule fails to evaluate **Symptom:** An alert rule using a Prometheus query shows evaluation errors or remains in a "No Data" state. **Possible causes and solutions:** | Cause | Solution | | --------------------------- | --------------------------------------------------------------------------------------------------- | | Template variables in query | Alert queries don't support template variables. Replace variables with hard-coded values. | | Query timeout | Simplify the query or increase the evaluation timeout. Use recording rules for complex expressions. | | Data source unreachable | Verify the Prometheus data source connection is working (test it in the data source settings). | | No data in range | Ensure the metric has recent data. Check that Prometheus is actively scraping the target. | ### Data source-managed rules not visible **Symptom:** Prometheus alerting rules don't appear in the Grafana Alerting UI. **Solution:** 1. Verify that **Manage alerts via Alerting UI** is enabled in the data source configuration. 1. Check that Prometheus has alerting rules configured in its rule files. 1. Ensure Grafana can access the Prometheus rules API endpoint (`/api/v1/rules`). 1. For Prometheus (unlike Mimir), the Alerting UI only supports viewing existing rules, not creating new ones. For more information on alerting with Prometheus, refer to [Prometheus alerting](https://grafana.com/docs/grafana//datasources/prometheus/alerting/). ## Get additional help If you continue to experience issues after following this troubleshooting guide: 1. Check the [Prometheus documentation](https://prometheus.io/docs/) for API and PromQL guidance. 1. Review the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Contact Grafana Support if you are a Cloud Pro, Cloud Contracted, or Enterprise user. 1. When reporting issues, include: - Grafana version - Prometheus version and type (Prometheus, Mimir, Cortex, Thanos) - Error messages (redact sensitive information) - Steps to reproduce - Relevant configuration such as data source settings, query timeout, and TLS settings (redact tokens, passwords, and other credentials) # Configure the Pyroscope data source The Pyroscope data source sets how Grafana connects to your Pyroscope backend. You can configure the data source using either the data source interface in Grafana or using a configuration file. This page explains how to set up and enable the data source capabilities using Grafana. If you make any changes, select **Save & test** to preserve those changes. If you're using your own installation of Grafana, you can provision the Pyroscope data source using a YAML configuration file. For more information about provisioning and available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ## Before you begin To configure a Pyroscope data source, you need administrator rights to your Grafana instance and a running Pyroscope backend that's receiving profiling data. If you're provisioning a Pyroscope data source, then you also need administrative rights on the server hosting your Grafana instance. ## Add or modify a data source You can use these procedures to configure a new Pyroscope data source or to edit an existing one. ### Create a new data source To configure basic settings for the data source, complete the following steps: 1. Select **Connections** in the main menu. 1. Enter `Grafana Pyroscope` in the search bar. 1. Select **Grafana Pyroscope**. 1. Select **Add new data source** in the top-right corner of the page. 1. On the **Settings** tab, complete the **Name**, **Connection**, and **Authentication** sections. - Use the **Name** field to specify the name used for the data source in panels, queries, and Explore. Toggle the **Default** switch for the data source to be pre-selected for new panels. - Under **Connection**, enter the **URL** of the Pyroscope instance. For example, `https://example.com:4040`. Refer to [Connection URL](#connection-url) for more information. - Complete the [**Authentication** section](#authentication). 1. Optional: Use **Additional settings** to configure other options. 1. Select **Save & test**. ### Update an existing data source To modify an existing Pyroscope data source: 1. Select **Connections** in the main menu. 1. Select **Data sources** to view a list of configured data sources. 1. Select the Pyroscope data source you want to modify. 1. Optional: Use **Additional settings** to configure or modify other options. 1. After completing your updates, select **Save & test**. ### Connection URL The data source connection URL should point to a location of a running Pyroscope backend. **Grafana Cloud Profiles** Your Grafana Cloud instance automatically includes a fully provisioned data source. If you are running a self-managed Grafana instance or need to configure an additional Pyroscope data source pointing to Grafana Cloud Profiles, you can find the Pyroscope URL under the **Manage your stack** section for your organization. To authenticate against Grafana Cloud Profiles, enable **Basic auth** and enter your numeric instance ID as the username and an access policy token as the password. The token must belong to an access policy that includes the `profiles:read` scope and targets the correct stack. For more information, refer to [Grafana Cloud access policies](https://grafana.com/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-policies/). **Self-managed Pyroscope backend** The connection URL for a self-managed Pyroscope backend depends on how Pyroscope is deployed. Refer to the steps under [Query profiles in Grafana](https://grafana.com/docs/pyroscope//deploy-kubernetes/helm/#query-profiles-in-grafana) for more information on how to configure the data source. If you plan to use the [Profiles Drilldown](https://grafana.com/docs/grafana//explore/simplified-exploration/profiles/) application and you are running a self-managed Pyroscope backend in microservices mode, the data source connection URL should point to a gateway or proxy that routes requests to the corresponding Pyroscope service. Refer to the [Helm ingress configuration](https://github.com/grafana/pyroscope/blob/main/operations/pyroscope/helm/pyroscope/templates/ingress.yaml) for specific routing requirements. ## Authentication Use this section to select an authentication method to access the data source. {{< admonition type="note" >}} Use Transport Layer Security (TLS) for an additional layer of security when working with Pyroscope. For additional information on setting up TLS encryption with Pyroscope, refer to [Pyroscope configuration](https://grafana.com/docs/pyroscope//configure-server/reference-configuration-parameters/). {{< /admonition >}} [//]: # 'Shared content for authentication section procedure in data sources' {{< docs/shared source="grafana" lookup="datasources/datasouce-authentication.md" leveloffset="+2" version="" >}} ## Additional settings **Additional settings** are optional settings that give you more control over your data source. Select the **Additional settings** section to expand it and configure these options. ### Advanced HTTP settings The Grafana Proxy deletes forwarded cookies. Use the **Allowed cookies** field to specify cookies that should be forwarded to the data source by name. The **Timeout** field sets the HTTP request timeout in seconds. ### Private data source connect [//]: # 'Shared content for authentication section procedure in data sources' {{< docs/shared source="grafana" lookup="datasources/datasouce-private-ds-connect.md" leveloffset="+2" version="" >}} ### Querying Use the **Minimal step** field to set the minimum step used for metric queries that return time-series data. The default value is 15 seconds. Enter a value in Grafana duration format using a number and a time unit specifier, such as `15s`, `1m`, or `1h`. Set it to the same value as, or higher than, the scrape interval configured in your Pyroscope database. Adjusting this option can help prevent gaps when you zoom in to profiling data. ## Verify the connection After you configure the data source, select **Save & test** to verify the connection. When the connection is successful, Grafana displays a **Data source is working** message. If the test fails, Grafana displays the error returned by the Pyroscope backend. For help resolving connection errors, refer to [Troubleshoot the Pyroscope data source](https://grafana.com/docs/grafana//datasources/pyroscope/troubleshooting/). ## Provision the data source You can provision the Pyroscope data source using YAML configuration files or Terraform, as part of the Grafana provisioning system. For more information about provisioning and available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ### YAML provisioning example The following example provisions a Pyroscope data source: ```yaml apiVersion: 1 datasources: - name: Grafana Pyroscope type: grafana-pyroscope-datasource url: http://localhost:4040 jsonData: minStep: '15s' ``` To provision with basic authentication, for example when connecting to Grafana Cloud Profiles: ```yaml apiVersion: 1 datasources: - name: Grafana Pyroscope type: grafana-pyroscope-datasource url: https://example.grafana.net basicAuth: true basicAuthUser: '' jsonData: minStep: '15s' secureJsonData: basicAuthPassword: ``` Replace the following: - ``: Your numeric instance ID, found under **Manage your stack**. - ``: An access policy token with the `profiles:read` scope. ### Terraform example To provision the data source with Terraform, use the [`grafana_data_source` resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source): ```hcl resource "grafana_data_source" "pyroscope" { type = "grafana-pyroscope-datasource" name = "Grafana Pyroscope" url = "http://localhost:4040" json_data_encoded = jsonencode({ minStep = "15s" }) } ``` To provision with basic authentication: ```hcl resource "grafana_data_source" "pyroscope" { type = "grafana-pyroscope-datasource" name = "Grafana Pyroscope" url = "http://localhost:4040" basic_auth_enabled = true basic_auth_username = "" secure_json_data_encoded = jsonencode({ basicAuthPassword = "" }) } ``` Replace the following: - ``: Your Pyroscope username. - ``: Your Pyroscope password. # Configure Trace to profiles Trace to profiles lets you navigate from a trace span directly to the profiling data for that span. You configure it in the Tempo data source, which connects your tracing data in Tempo with your profiling data in Pyroscope. {{< admonition type="note" >}} Your application must be instrumented for profiles and traces. For more information, refer to [Link traces to profiles](https://grafana.com/docs/pyroscope//configure-client/trace-span-profiles/). {{< /admonition >}} [//]: # 'Shared content for Trace to profiles in the Tempo data source' {{< docs/shared source="grafana" lookup="datasources/tempo-traces-to-profiles.md" version="" >}} ## Verify the connection After you configure the Tempo data source and instrument your application, verify that Trace to profiles works. 1. Open a trace in **Explore** using your Tempo data source. 1. Expand a span to view its details. 1. Confirm that the **Profiles for this span** button appears in the span details. 1. Confirm that an embedded flame graph appears in the span details section. If the span attribute `pyroscope.profile.id` isn't present on the span, the OpenTelemetry bridge package isn't configured correctly. Refer to [Link tracing and profiling with span profiles](https://grafana.com/docs/pyroscope//configure-client/trace-span-profiles/) for per-language setup instructions. ## Troubleshoot trace to profiles Use the following checklist to diagnose common issues with Trace to profiles. ### No "Profiles for this span" button appears 1. Verify that you installed and configured the OTel bridge package for your language. This is a separate package from the Pyroscope SDK and OpenTelemetry SDK. Refer to [Link tracing and profiling with span profiles](https://grafana.com/docs/pyroscope//configure-client/trace-span-profiles/) for per-language instructions. 1. Check that the `pyroscope.profile.id` attribute exists on the span in Tempo. If it's missing, the bridge package isn't tagging spans correctly. 1. Check that the `span_name` label exists on the profiling data in Pyroscope. If it's missing, the bridge package isn't labeling profiling samples correctly. 1. Verify that the tags you configured in the Tempo data source are present in the span's attributes or resources. If the tags don't match, the span link doesn't appear. ### Button appears but no profile data is shown 1. The span might be shorter than the sampling interval of the profiler. Span profiling is most effective on spans longer than 20ms. 1. CPU profiling only captures time spent actively executing on the CPU. If the span is mostly waiting on I/O, network calls, or database queries, the CPU profile for that span may be sparse or empty. 1. Verify that the profile type selected in the Tempo data source configuration matches the profile type your application sends. ### Tags don't match Tags you configure in the Tempo data source must be present in the span's attributes or resources. If a tag has dots in its name, for example, `service.name`, and the Pyroscope data source doesn't allow dots in labels, remap it to a name without dots, for example, `service_name`. # Query profile data You can query your profile data using the open source Grafana Profiles Drilldown app or the built-in Pyroscope data source query editor. - Profiles Drilldown provides a queryless experience for inspecting your profiling data with opinionated workflows to assist your investigation. - Data source query editor provides complete control over your data exploration and is recommended for experienced users. In addition, you can embed flame graph panels in Grafana dashboards. Refer to the [Flame graph panel](https://grafana.com/docs/grafana//panels-visualizations/visualizations/flame-graph/) documentation for details. ## Profiles Drilldown [Grafana Profiles Drilldown](https://grafana.com/docs/grafana//explore/simplified-exploration/profiles/) is a native Grafana application designed to integrate seamlessly with Pyroscope, the open source continuous profiling platform, providing a smooth, queryless experience for browsing and analyzing profiling data. You can use Profiles Drilldown in Grafana Cloud or in your own Grafana instance. For more information, refer to [Access or install Profiles Drilldown](https://grafana.com/docs/grafana//explore/simplified-exploration/profiles/access/). ![Profiles Drilldown home screen](/media/docs/explore-profiles/explore-profiles-homescreen-v0.1.17.png) ### Use cases There are several different modes for viewing, analyzing, and comparing profiling data. The main use cases are the following: - Proactive: Cutting costs, addressing latency issues, or optimizing memory usage for applications - Reactive: Resolving incidents with line-level accuracy or debugging active latency/memory issues Profiles Drilldown provides an intuitive interface to specifically support proactive and reactive use cases. You get a holistic view of all of your services and how they're functioning, but also the ability to drill down for more targeted root cause analysis. Profiles Drilldown offers a convenient platform to analyze profiles and get insights that are impossible to get from using other traditional signals like logs, metrics, or tracing. {{< youtube id="x9aPw_CbIQc" >}} {{< docs/play title="the Grafana Play site" url="https://play.grafana.org/a/grafana-pyroscope-app/profiles-explorer" >}} ## Pyroscope query editor The Pyroscope data source query editor gives you access to a profile type selector, a label selector, and collapsible options. Like Profiles Drilldown, the query editor also provides a flame graph to visualize data. ![Query editor](/media/docs/pyroscope/query-editor/query-editor.png 'Query editor') To access the query editor: 1. Sign into Grafana or Grafana Cloud. 1. Select your Pyroscope data source. 1. From the menu, choose **Explore**. 1. Select a profile type from the drop-down menu. {{< figure src="/media/docs/pyroscope/query-editor/select-profile.png" class="docs-image--no-shadow" max-width="450px" caption="Profile selector" >}} 1. Use the labels selector input to filter by labels. Pyroscope uses similar syntax to Prometheus to filter labels. Refer to [Label selector syntax](#label-selector-syntax) for available operators and syntax. While the label selector can be left empty to query all profiles without filtering by labels, you must select a profile type or app for the query to be valid. Grafana doesn't show any data if the profile type or app isn't selected when a query runs. ![Labels selector](/media/docs/pyroscope/query-editor/labels-selector.png 'Labels selector') 1. Expand the **Options** section to view the available query options. ![Options section](/media/docs/pyroscope/query-editor/options-section.png 'Options section') ### Label selector syntax The label selector uses a syntax similar to Prometheus. Enclose one or more label matchers in curly braces, separated by commas. Each matcher combines a label name, an operator, and a value in double quotes. The label selector supports the following operators. | Operator | Description | | -------- | ------------------------------------------------------- | | `=` | Selects labels that exactly match the value. | | `!=` | Selects labels that don't match the value. | | `=~` | Selects labels that match the regular expression. | | `!~` | Selects labels that don't match the regular expression. | The following examples show common label selectors: ``` {service_name="my-service"} {service_name="my-service", namespace="production"} {service_name=~"cart.*"} {service_name="checkout", namespace!="staging"} ``` Leave the label selector empty, or set it to `{}`, to query all profiles for the selected profile type without filtering by labels. ### Query options The **Options** section provides the following settings. | Option | Description | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Query Type** | Sets the type of data the query returns: **Profile** for flame graph data, **Metric** for time-series data, or **Both**. You can only select **Both** in Explore. Panels on dashboards allow only one visualization. | | **Group by** | Groups metric data by one or more labels. Without a **Group by** label, metric data aggregates over all labels into a single time series. **Group by** applies only to metric data and doesn't change the profile results. | | **Limit** | Sets the maximum number of time series returned when **Group by** is set. The series returned are always ordered by descending value for the total aggregated data over the time period. Doesn't apply to profile queries. | | **Span ID** | Sets the span ID from which to search for profiles, so you can view the profile associated with a specific trace span. This option supports Trace to profiles, for example, `64f170a95f537095`. | | **Max Nodes** | Sets the maximum number of nodes rendered in the flame graph. Lower values improve performance for large profiles by aggregating smaller nodes. The field placeholder shows the default value, `16384`. | | **Annotations** | Includes profiling annotations, such as sampling and throttling events, on time-series results. Use these annotations to identify when Pyroscope reduced or limited ingestion for a service. | Select a query type to return the profile data. Data is shown in the [Flame Graph](https://grafana.com/docs/grafana//panels-visualizations/visualizations/flame-graph/), metric data visualized in a graph, or both. Using **Group by**, you can group metric data by a specified label. You can use multiple labels to group by. In conjunction with **Group by**, you can set a positive number in the **Limit** input to limit the maximum number of time series returned by the data source. #### Exemplars The **Exemplars** toggle in the **Options** section links individual profile samples to points on a metrics graph, so you can jump from a spike in a time series to the specific profile that produced it. #### Heatmap When the `profilesHeatmap` feature is enabled, a **Heatmap** toggle appears in the **Options** section. The heatmap visualizes the distribution of profile samples over time, which helps you spot patterns and outliers across many profiles. When you enable **Heatmap**, use the **Heatmap Type** option to choose how samples are aggregated: - **Individual**: Shows individual profile samples. - **Span**: Aggregates samples by span duration, for use with Trace to profiles. {{< admonition type="note" >}} The `profilesHeatmap` feature is experimental. Experimental features are subject to change and may be removed in a future release. Enable it using the `profilesHeatmap` feature toggle. {{< /admonition >}} #### UTF-8 label names When the `pyroscopeUTF8LabelNames` feature is enabled, the label selector supports UTF-8 label names. Enclose label names that contain characters such as dots in double quotes, for example, `{"k8s.namespace"="production"}`. The `pyroscopeUTF8LabelNames` feature is in public preview and enabled by default. ### Query examples The following examples show common ways to build queries in the query editor. Each example lists the profile type to select and the label selector to enter. **Investigate high CPU usage for a service:** Select the `process_cpu:cpu:nanoseconds:cpu:nanoseconds` profile type and filter to the affected service: ``` {service_name="checkout"} ``` Set **Query Type** to **Both** in Explore to view the CPU flame graph alongside a time series of CPU usage. **Compare memory allocations across environments:** Select the `memory:alloc_space:bytes:space:bytes` profile type, set **Query Type** to **Metric**, and set **Group by** to `namespace`: ``` {service_name="cart"} ``` The metrics graph shows a separate time series for each namespace, which lets you compare allocation behavior between environments such as `production` and `staging`. **Analyze goroutine growth:** Select the `goroutine:goroutine:count:goroutine:count` profile type to inspect the number of goroutines and where they're created: ``` {service_name=~"api.*"} ``` **View the profile for a specific trace span:** Select a profile type, set **Query Type** to **Profile**, and enter the span ID in the **Span ID** field, for example, `64f170a95f537095`. The flame graph then shows only the profile data associated with that span. This example supports [Trace to profiles](https://grafana.com/docs/grafana//datasources/pyroscope/configure-traces-to-profiles/). ### Profiles query results Profiles can be visualized in a flame graph. Refer to the [Flame Graph documentation](https://grafana.com/docs/grafana//panels-visualizations/visualizations/flame-graph/) to learn about the visualization and its features. ![Flame graph](/media/docs/pyroscope/query-editor/flame-graph.png 'Flame graph') Pyroscope returns profiles aggregated over a selected time range. The absolute values in the flame graph grow as the time range gets bigger while keeping the relative values meaningful. You can zoom in on the time range to get a higher granularity profile up to the point of a single scrape interval. ### Metrics query results Metrics results represent the aggregated sum value over time of the selected profile type. ![Metrics graph](/media/docs/pyroscope/query-editor/metric-graph.png 'Metrics graph') This allows you to quickly see any spikes in the value of the scraped profiles and zoom in to a particular time range. ## Next steps - [Use template variables](https://grafana.com/docs/grafana//datasources/pyroscope/template-variables/) - [Troubleshoot the Pyroscope data source](https://grafana.com/docs/grafana//datasources/pyroscope/troubleshooting/) # Pyroscope template variables Template variables let you create dynamic, reusable dashboards by replacing hard-coded values, such as service names or profile types, with selectable variables. Grafana displays these variables as drop-down menus at the top of the dashboard, letting viewers change the displayed data without editing queries. For an introduction to templating and template variables, refer to [Templating](https://grafana.com/docs/grafana//dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/). ## Query variable types Query variables query the Pyroscope data source to populate drop-down values. When you create a query variable, select a Pyroscope data source and choose a query type. | Query type | Required inputs | Description | | ---------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Profile type** | None | Returns the profile types available from the data source. The drop-down displays a readable label, while the variable value is the full profile type ID, such as `process_cpu:cpu:nanoseconds:cpu:nanoseconds`. | | **Label** | Profile type | Returns the label names available for the selected profile type, such as `service_name`, `namespace`, and `region`. | | **Label value** | Profile type, label | Returns the values for the selected label, scoped to the selected profile type. | Because the **Label** and **Label value** query types require a profile type, and **Label value** also requires a label, you can chain these variables so that each one depends on the previous selection. ## Create a query variable To create a query variable: 1. Navigate to **Dashboard settings** > **Variables**. 1. Select **Add variable**. 1. Set **Select variable type** to **Query**. 1. Select the Pyroscope data source. 1. Select a **Query type** of **Profile type**, **Label**, or **Label value**. 1. For a **Label** or **Label value** query, select a **Profile type**. 1. For a **Label value** query, select a **Label**. 1. Select **Apply** to save the variable. ## Query variable examples The following examples show common ways to use Pyroscope query variables. **Select a profile type from a drop-down:** Create a variable named `profile_type` to let viewers switch the profile type without editing the query: 1. Set **Query type** to **Profile type**. 1. Save the variable. The drop-down displays readable labels, while each value is the full profile type ID, such as `process_cpu:cpu:nanoseconds:cpu:nanoseconds`. You can then reference `$profile_type` in the query editor's profile type field. **Populate a drop-down with services:** Create a variable named `service` that lists the services sending profiles: 1. Set **Query type** to **Label value**. 1. Set **Profile type** to a profile type, such as `process_cpu:cpu:nanoseconds:cpu:nanoseconds`. 1. Set **Label** to `service_name`. 1. Save the variable. The drop-down now lists the values of the `service_name` label. **Chain dependent variables:** Combine variables so each one narrows the next. This example builds a profile type selector, a label selector, and a label value selector that all update together: 1. Create a `profile_type` variable with **Query type** set to **Profile type**. 1. Create a `label` variable with **Query type** set to **Label** and **Profile type** set to `$profile_type`. 1. Create a `label_value` variable with **Query type** set to **Label value**, **Profile type** set to `$profile_type`, and **Label** set to `$label`. When a viewer changes `profile_type`, the `label` drop-down refreshes with the labels for that profile type, and `label_value` refreshes with the values for the selected label. ## Use variables in queries After you create a variable, reference it in the query editor using the `$variable` or `${variable}` syntax. You can reference variables in both the profile type field and the label selector. For example, if you have a variable named `service` created with the **Label value** query type on the `service_name` label, reference it in the label selector: ``` {service_name="$service"} ``` To let viewers switch the profile type, enter `$profile_type` in the query editor's profile type field. When **Multi-value** or **Include All** is enabled for the variable, the value becomes a regular expression pattern, such as `value1|value2`. Use the `=~` operator instead of `=` in your label selector: ``` {service_name=~"$service"} ``` ## Where you can use template variables The Pyroscope data source interpolates template variables in only some parts of the query editor. The following table lists which fields support variables. | Query editor field | Supports template variables | | ------------------ | --------------------------- | | Profile type | Yes | | Label selector | Yes | | Group by | No | | Span ID | No | | Max Nodes | No | | Limit | No | Query variable definitions also support variables. You can reference one variable in another to chain **Label** and **Label value** queries, as shown in [Chain dependent variables](#query-variable-examples). {{< admonition type="note" >}} Fields other than the profile type and label selector don't interpolate template variables. If you enter a variable such as `$group` in the **Group by**, **Span ID**, **Max Nodes**, or **Limit** fields, Grafana passes the literal text to the backend instead of the variable's value. {{< /admonition >}} ## Filters variable The Pyroscope data source supports the [Filters](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#add-ad-hoc-filters) variable type, which lets dashboard viewers add label filters without editing queries. To set up a Filters variable: 1. Navigate to **Dashboard settings** > **Variables**. 1. Select **Add variable**. 1. Set **Select variable type** to **Filters**. 1. Select the Pyroscope data source. 1. Select **Apply** to save the variable. After you add the variable, a filter bar appears at the top of the dashboard. Viewers can add filters by selecting a label, an operator (`=`, `!=`, `=~`, `!~`), and a value. Grafana applies these filters to all Pyroscope queries on the dashboard. ## Related resources - [Query profile data](https://grafana.com/docs/grafana//datasources/pyroscope/query-editor/): Use variables in the label selector. - [Troubleshoot the Pyroscope data source](https://grafana.com/docs/grafana//datasources/pyroscope/troubleshooting/): Resolve variable-related query issues. # Troubleshoot Pyroscope data source issues This document provides solutions to common issues you may encounter when configuring or using the Pyroscope data source. For configuration instructions, refer to [Configure the Pyroscope data source](https://grafana.com/docs/grafana//datasources/pyroscope/configure/). ## Connection and configuration errors These errors occur when setting up the data source or when connecting to the Pyroscope backend. ### "Data source is not working" or connection fails **Symptoms:** - Save & test fails with a connection error - Error message indicates the Pyroscope backend is unreachable - Timeout errors when testing the data source **Possible causes and solutions:** | Cause | Solution | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Incorrect URL | Verify the URL points to your Pyroscope instance. For self-managed deployments, the default port is `4040` (for example, `http://localhost:4040`). | | Pyroscope service not running | Check that the Pyroscope backend is running and accessible. Use `curl` or a browser to test connectivity to the Pyroscope URL. | | Network connectivity issues | Verify network connectivity from the Grafana server to the Pyroscope endpoint. Check firewall rules allow outbound connections on the required port. | | Microservices mode routing | If running Pyroscope in microservices mode, ensure the URL points to a gateway or proxy that routes requests correctly. Refer to the Helm ingress configuration for routing requirements. | | TLS/SSL certificate issues | If using HTTPS, verify the certificate is valid and trusted by the Grafana server. Configure TLS settings in the data source if using self-signed certificates. | ### URL format issues **Symptoms:** - Data source test fails immediately - Error indicates invalid URL **Solutions:** 1. Ensure the URL includes the protocol (`http://` or `https://`). 1. Remove any trailing slashes from the URL. 1. For Grafana Cloud Profiles, find the correct URL under **Manage your stack** in your organization settings. 1. Verify the port number is correct for your deployment. ### "Connection refused" or timeout errors **Symptoms:** - Data source test fails with network errors - Queries fail with connection errors - Intermittent connectivity issues **Solutions:** 1. Verify network connectivity from the Grafana server to the Pyroscope endpoint. 1. Check firewall rules allow outbound connections on the required port (default: 4040). 1. For Kubernetes deployments, verify the service is exposed correctly. 1. Check if a proxy is required and configure it in the data source settings. ### SSL/TLS certificate errors **Symptoms:** - Certificate validation failures - SSL handshake errors - "Certificate not trusted" messages **Solutions:** 1. Ensure the system time is correct on the Grafana server. 1. Verify the certificate is valid and not expired. 1. For self-signed certificates, enable **Skip TLS Verify** in the data source settings (not recommended for production). 1. Add the CA certificate to the Grafana server's trusted certificates. 1. Check that intermediate certificates are included in the certificate chain. ### Private connectivity or reverse proxy issues **Symptoms:** - The data source works from some networks but not others. - Connecting through a reverse proxy, load balancer, or private link fails. **Solutions:** 1. Include the scheme in the **Connection URL**. When connecting through a private endpoint or reverse proxy, use `https://` explicitly rather than a bare host name. 1. Ensure the reverse proxy or gateway forwards the Pyroscope API paths unchanged. Rewriting or stripping the request path prevents the data source from reaching the backend. 1. For Grafana Cloud connecting to a Pyroscope backend on a private network, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). 1. Verify that any firewall or private link allows outbound HTTPS from Grafana to the Pyroscope endpoint. ## Authentication errors These errors occur when the data source cannot authenticate with the Pyroscope backend. ### "Authentication failed" or "Unauthorized" **Symptoms:** - Save & test fails with authentication errors - Queries return 401 or 403 status codes - Profile types don't load **Possible causes and solutions:** | Cause | Solution | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | Invalid credentials | Verify the username and password or API key are correct. | | Expired credentials | Generate new credentials and update the data source configuration. | | Incorrect authentication type | Ensure you've selected the correct authentication method for your Pyroscope deployment (Basic auth, API key, or no authentication for local setups). | | Missing authentication headers | For custom authentication, verify the required headers are configured correctly in the data source settings. | ### Basic authentication not working **Symptoms:** - Basic auth credentials are rejected - Works in browser but fails in Grafana **Solutions:** 1. Verify Basic authentication is enabled in the data source settings. 1. Ensure the username and password are entered correctly (no extra spaces). 1. Check that the Pyroscope backend is configured to accept Basic authentication. 1. For Grafana Cloud Profiles, use Basic authentication with your access policy token as the password. Refer to [Grafana Cloud access policy and token issues](#grafana-cloud-access-policy-and-token-issues). ### API key authentication issues **Symptoms:** - API key is rejected - Error indicates invalid token **Solutions:** 1. Verify you're using the correct API key format for your Pyroscope deployment. 1. For Grafana Cloud, generate a new access policy token with the `profiles:read` scope. 1. Ensure the credentials have the required permissions to read profiling data. 1. Check if the token has expired and generate a new one if necessary. ### Grafana Cloud access policy and token issues **Symptoms:** - Authentication fails when connecting a data source to Grafana Cloud Profiles. - Queries return 401 or 403 after a backend migration or stack change. - The data source reads from a different stack or tenant than expected. **Cause:** Grafana Cloud Profiles authenticates with an access policy token rather than a user password. The token must belong to an access policy that includes the `profiles:read` scope and targets the correct stack. A token with the wrong scope, or one created for a different stack, fails authentication or returns data from the wrong tenant. Tokens and connection URLs are stack-specific and don't carry over during a migration. **Solutions:** 1. Create or select an access policy that includes the `profiles:read` scope and whose realm targets the correct stack, then generate a token for that policy. 1. In the data source, enable **Basic auth**. Enter the numeric instance ID as the username and the access policy token as the password. Find the instance ID and endpoint under **Manage your stack**. 1. Confirm the **Connection URL** points to the Pyroscope endpoint for the same stack as the token. 1. After a backend migration or stack change, regenerate the token, then update both the URL and credentials. ## Query errors These errors occur when executing queries against the Pyroscope data source. ### "No data" or empty results **Symptoms:** - Query executes without error but returns no data - Flame graph shows "No data" message - Metrics graph is empty **Possible causes and solutions:** | Cause | Solution | | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | Time range doesn't contain data | Expand the dashboard time range. Profiling data may not exist for the selected period. | | Application not sending profiles | Verify your application is configured to send profiles to Pyroscope. Check the Pyroscope agent or SDK configuration. | | Incorrect profile type selected | Select a different profile type from the drop-down menu. Not all profile types are available for all applications. | | Label selector too restrictive | Remove or modify label filters to broaden the query. Start without filters to verify data exists. | | Wrong service or application selected | Verify the `service_name` label matches your application's configuration. | | Profile type or app not selected | Ensure you've selected a profile type or app in the query editor. The query returns no data if neither is selected. | {{< admonition type="note" >}} If the data source connection tests successfully but no profiles exist for any query, the problem is usually on the ingestion side rather than in the data source. The data source only reads profiles from Pyroscope; it doesn't collect or send them. Verify that your application, SDK, or collector, such as Grafana Alloy or an OpenTelemetry Collector, is sending profiles to your Pyroscope backend. For ingestion setup, refer to [Configure the client to send profiles](https://grafana.com/docs/pyroscope//configure-client/). {{< /admonition >}} ### "Profile type not found" or empty profile type selector **Symptoms:** - Profile type drop-down is empty - Error indicates profile type doesn't exist **Solutions:** 1. Verify the Pyroscope backend is receiving profiles from your applications. 1. Check that the time range includes periods when profiles were collected. 1. Test the data source connection to ensure it's working correctly. 1. For new deployments, wait a few minutes for initial profile data to be ingested. ### Label names or values not loading **Symptoms:** - Label selector doesn't populate - No labels appear in the drop-down menus - Autocomplete doesn't work **Solutions:** 1. Verify the data source connection is working by testing it in the settings. 1. Select a profile type first, as labels are loaded based on the selected profile type. 1. Expand the time range to ensure label data is available. 1. Check the browser console for JavaScript errors that might indicate issues. ### Query syntax errors **Symptoms:** - Error message indicates invalid query syntax - Label selector is rejected **Solutions:** 1. Use the correct label selector syntax. Pyroscope uses a syntax similar to Prometheus. 1. Enclose label values in double quotes: `{service_name="my-service"}`. 1. Use valid operators: `=`, `!=`, `=~`, `!~`. 1. Escape special characters in regular expression patterns when using `=~` or `!~`. Example valid queries: ``` {service_name="my-app"} {service_name="my-app", env="production"} {service_name=~"my-app.*"} ``` ### Queries return only recent data even though older data is retained If a query over a long time range returns data only for the most recent days, the query time range likely exceeds a server-side query limit. This is most commonly reported in Grafana Cloud Profiles. Refer to [Query time range and retention limits](#query-time-range-and-retention-limits). ## Flame graph issues These errors are specific to the flame graph visualization. ### Flame graph not rendering **Symptoms:** - Query returns data but flame graph doesn't display - Flame graph area is blank - Loading indicator never completes **Solutions:** 1. Check that the Query Type is set to include profile data (not metrics only). 1. Verify the response contains valid profile data using the Query Inspector. 1. Try refreshing the page or re-running the query. 1. For large profiles, increase the browser memory limits or reduce the time range. ### Flame graph times out or fails to load on large time ranges **Symptoms:** - A flame graph renders for a short time range, such as five minutes, but times out for a longer range, such as 30 minutes or more. - Queries over large time ranges are slow or fail intermittently. **Cause:** Rendering a flame graph over a large time range requires the Pyroscope backend to merge many profiles, which increases query time and memory use. Large or high-cardinality ranges can exceed request timeouts before a result returns. **Solutions:** 1. Narrow the time range. Start with a short window, such as five minutes, then widen it gradually to find the largest range that renders reliably. 1. Add label filters to reduce the volume of profiles the backend must merge. 1. Confirm the range is within the server-side query limits described in [Query time range and retention limits](#query-time-range-and-retention-limits). 1. For self-managed deployments, review backend resources and query timeout settings if large ranges consistently fail. ### Flame graph shows aggregated data without detail **Symptoms:** - Flame graph lacks expected granularity - Function names are missing or truncated - Stack traces appear incomplete **Solutions:** 1. Zoom in to a smaller time range to get more detailed profiles. 1. Verify your application is instrumented to capture full stack traces. 1. Check if the profiling agent is configured with sufficient sampling rate. 1. For compiled languages, ensure debug symbols are available. ### Unable to interact with flame graph **Symptoms:** - Clicking on flame graph elements doesn't work - Tooltip doesn't appear - Zoom and filter controls are unresponsive **Solutions:** 1. Verify JavaScript is enabled in your browser. 1. Check the browser console for JavaScript errors. 1. Try using a different browser or clearing the browser cache. 1. Update Grafana to the latest version. ## Template variable errors These errors occur when using template variables with the Pyroscope data source. ### Variables return no values **Symptoms:** - The variable drop-down is empty - Dependent variables don't populate **Solutions:** 1. Verify the data source connection is working by testing it in the settings. 1. For **Label** and **Label value** variables, verify that a profile type is selected. Labels and label values are scoped to the selected profile type. 1. Expand the dashboard time range to ensure the variable query covers a period when profiles were collected. 1. Confirm the Pyroscope backend is receiving profiles from your applications. ### Variables don't filter queries as expected **Symptoms:** - Query results don't change when the variable value changes - Queries return no data after selecting a variable value **Solutions:** 1. Verify the variable is referenced correctly in the label selector, for example, `{service_name="$service"}`. 1. When **Multi-value** or **Include All** is enabled, the variable value becomes a regular expression pattern. Use the `=~` operator instead of `=`, for example, `{service_name=~"$service"}`. 1. Confirm the label used in the variable exists on the profiling data for the selected profile type. For more information, refer to [Pyroscope template variables](https://grafana.com/docs/grafana//datasources/pyroscope/template-variables/). ## Profiles Drilldown issues These errors are specific to the [Profiles Drilldown](https://grafana.com/docs/grafana//explore/simplified-exploration/profiles/) application. ### Profiles Drilldown not available **Symptoms:** - Profiles Drilldown option doesn't appear in the menu - Error when trying to access Profiles Drilldown **Solutions:** 1. Verify the Profiles Drilldown plugin is installed and enabled. 1. For self-managed Grafana, install the plugin from the Grafana plugin catalog. 1. Restart Grafana after installing the plugin. 1. Check that you have the required permissions to access the application. ### Services or profiles not appearing in Profiles Drilldown **Symptoms:** - Profiles Drilldown shows no services - Expected applications are missing **Solutions:** 1. Verify the Pyroscope data source is configured and working. 1. Check that profiles are being ingested with the correct `service_name` label. 1. Ensure the time range includes periods when profiles were collected. 1. For microservices mode, verify the data source URL points to the correct gateway. ## Trace to profiles issues These errors occur when linking tracing and profiling data. Trace to profiles is the most configuration-sensitive feature of the data source. Most issues come from an unmet prerequisite rather than a data source misconfiguration. **Prerequisites checklist:** Confirm all of the following before troubleshooting further: 1. Your application is instrumented with a Pyroscope language SDK for profiling and with OpenTelemetry for tracing. 1. You installed the OpenTelemetry span-profiling bridge package for your language. This is a separate package from the Pyroscope SDK and the OpenTelemetry SDK. 1. Profiling uses SDK-based instrumentation. Span profiles aren't available with eBPF-based collection. 1. The Tempo data source has Trace to profiles configured and points to your Pyroscope data source. 1. The tags configured in the Tempo data source are present on the span's attributes or resources. For per-language setup instructions, refer to [Link traces to profiles](https://grafana.com/docs/pyroscope//configure-client/trace-span-profiles/). ### Span profiles not appearing **Symptoms:** - Trace spans don't show profile links - "Profiles for this span" option is missing **Solutions:** 1. Verify the Tempo data source is configured with Trace to profiles enabled and that it points to your Pyroscope data source. 1. Confirm the `pyroscope.profile.id` attribute exists on the span in Tempo. If it's missing, the bridge package isn't tagging spans correctly. 1. Confirm the `span_name` label exists on the profiling data in Pyroscope. If it's missing, the bridge package isn't labeling samples correctly. 1. Verify that the tags you configured in the Tempo data source match the span's attributes or resources. If they don't match, the link doesn't appear. 1. Verify profile data exists for the time range of the trace. For more information, refer to [Configure Trace to profiles](https://grafana.com/docs/grafana//datasources/pyroscope/configure-traces-to-profiles/). ### Profile doesn't match the trace span **Symptoms:** - Profile data doesn't correlate with the trace - Wrong time range in profile **Solutions:** 1. Verify the trace and profile timestamps are synchronized. 1. Check that the application's clock is accurate. 1. Ensure the `span_id` and `trace_id` labels are correctly propagated to profiles. ### Span profiles aren't available with eBPF-based collection **Symptoms:** - Traces and profiles both exist, but no span links appear. - Profiling is collected with eBPF, for example through Grafana Alloy or Beyla. **Cause:** Span profiling correlates profiling samples with the active trace span, which requires a language SDK and the OpenTelemetry span-profiling bridge. eBPF-based collection can't associate samples with individual spans, so it doesn't produce span profiles. **Solutions:** 1. Instrument the application with a supported Pyroscope language SDK and the matching bridge package to enable span profiles. 1. If you must use eBPF profiling, view profiles by service instead. Per-span correlation isn't currently supported for eBPF-collected data. ### Filter span profiles by span name or trace ID, not span ID **Symptoms:** - Queries filtered by `span_id` return no data, while `span_name` works. - Correlating profiles to a specific span by `span_id` is unreliable. **Cause:** The span-profiling bridge labels profiling samples with `span_name` and `trace_id`. Depending on the SDK, profiling context may be attached only to the root span of a trace, with child spans inheriting it. `span_id` is high cardinality and isn't a dependable correlation label. **Solutions:** 1. Filter profiling data by `span_name` or `trace_id` rather than `span_id`. 1. Use the **Profiles for this span** link, which correlates using the `pyroscope.profile.id` span attribute rather than a label query. ### .NET profiler conflict with OpenTelemetry auto-instrumentation **Symptoms:** - On .NET, span profiles don't work when the Pyroscope .NET profiler and OpenTelemetry auto-instrumentation are both enabled. - One profiler fails to attach. **Cause:** The .NET CLR allows only one profiler to attach to a process at a time. The Pyroscope .NET profiler and OpenTelemetry auto-instrumentation each rely on a separate CLR profiler, so they conflict. **Solutions:** 1. Use OpenTelemetry manual instrumentation instead of auto-instrumentation. 1. Add the `Pyroscope.OpenTelemetry` package and register the `PyroscopeSpanProcessor` in your tracing pipeline. 1. For setup instructions, refer to [Span profiles with Traces to profiles for .NET](https://grafana.com/docs/pyroscope//configure-client/trace-span-profiles/dotnet-span-profiles/). ### Correlate logs with profiles **Symptoms:** - You want to navigate from a log line directly to profiling data, but no link exists. **Cause:** Grafana supports Trace to profiles, but there's no direct logs-to-profiles correlation. **Solutions:** 1. Correlate through shared labels, such as `service_name`, to move between logs and profiles for the same service and time range. 1. If your logs contain a `trace_id`, use a Loki derived field to open the trace, then use **Profiles for this span** from the trace to reach the profiling data. ## Performance issues These issues relate to slow queries or high resource usage. ### Slow queries or timeouts **Symptoms:** - Queries take a long time to complete - Requests time out before returning results - Dashboard panels fail to load **Solutions:** 1. Reduce the time range to decrease the amount of data processed. 1. Add label filters to narrow the query scope. 1. Increase the **Timeout** setting in the data source's **Additional settings**. 1. Adjust the **Minimal step** setting to reduce the number of data points. 1. For large deployments, ensure the Pyroscope backend has sufficient resources. ### High memory usage in browser **Symptoms:** - Browser becomes unresponsive when viewing profiles - Memory warnings appear - Page crashes when loading flame graphs **Solutions:** 1. Reduce the time range to load smaller profiles. 1. Close unnecessary browser tabs to free up memory. 1. Use a smaller aggregation window for very large profiles. 1. Consider using Profiles Drilldown for large-scale analysis. ## Grafana Cloud-specific issues These issues are most commonly reported in Grafana Cloud Profiles. Some limits also apply to self-managed Pyroscope deployments, where you can adjust them directly in the Pyroscope configuration. ### Query time range and retention limits **Symptoms:** - A query over a long time range, such as 30 days, returns data only for the most recent days. - Older profiles appear to be missing even though the retention period hasn't elapsed. - Users interpret the missing data as data loss or a bug. **Cause:** Pyroscope enforces query limits on the server side that are separate from, and often shorter than, the data retention period. These limits are enforced by the Pyroscope query frontend, not by the Grafana data source, so the data source can't change or override them. | Limit | Default | Description | | ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `querier.max-query-lookback` | `7d` | How far back from the current time a query can read. Requests for data older than this window don't fail. Instead, the range is silently truncated to the allowed window, which looks like missing data. | | `querier.max-query-length` | `24h` | The maximum time span of a single query. Requests that exceed this span are rejected. | Because retention is typically longer than the query window, profiles can be retained but not queryable. For example, Grafana Cloud Profiles retains data for 14 days on free plans and 30 days on paid plans, while the default query window is 7 days. {{< admonition type="note" >}} The default values shown are the Pyroscope OSS defaults. Grafana Cloud and self-managed deployments may configure different values. {{< /admonition >}} **Solutions:** 1. Confirm this is a limit rather than a gap in ingestion by narrowing the time range to the most recent period and verifying that data appears. 1. To query further back than the query window, raise the limit rather than splitting the query into smaller ranges. Because the window is measured from the current time, splitting a long range into 7-day increments doesn't return older data. 1. For self-managed deployments, increase `querier.max-query-lookback` and, if needed, `querier.max-query-length` in the Pyroscope limits configuration. Set a value to `0` to disable that limit. 1. For Grafana Cloud Profiles, contact Grafana Support to adjust these limits for your tenant. 1. If a single query exceeds `querier.max-query-length`, reduce the query's time span so it stays within the configured limit. ## Enable debug logging To capture detailed error information for troubleshooting: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location). 1. Look for Pyroscope-specific entries that include request and response details. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you've tried the solutions in this guide and still encounter issues: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review the [Pyroscope GitHub issues](https://github.com/grafana/pyroscope/issues) for known bugs. 1. Refer to [Pyroscope documentation](https://grafana.com/docs/pyroscope/latest/) for additional guidance. 1. Contact [Grafana Support](https://grafana.com/contact/) if you're an Enterprise, Cloud Pro, or Cloud Contracted user. 1. When reporting issues, include: - Grafana version - Pyroscope version (if self-managed) - Error messages (redact sensitive information) - Steps to reproduce - Data source configuration (redact credentials) # Service Graph and Service Graph view The Service Graph is a visual representation of the relationships between services in a distributed system. Each node represents a service such as an API or database, and each edge represents requests between services. Use the Service Graph to detect performance issues, track error and latency trends across services, and navigate to the traces that caused them. The Service Graph is built on RED metrics, a monitoring methodology that focuses on three signals: - **Rate:** requests per second - **Errors:** failed requests per second - **Duration:** latency distribution (for example, p50, p95, p99) Service graph data flows from traces through a metrics pipeline into Grafana: 1. Your application emits traces to Tempo. 1. The [Tempo metrics generator](https://grafana.com/docs/tempo//metrics-from-traces/service_graphs/) or [Grafana Alloy](https://grafana.com/docs/tempo//metrics-from-traces/service_graphs/enable-service-graphs/) processes the traces and writes service graph metrics to a Prometheus-compatible backend. 1. Grafana reads those metrics from your linked Prometheus data source and renders the graph. {{< figure src="/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-node-graph.png" class="docs-image--no-shadow" max-width="500px" alt="Node graph showing service dependencies with color-coded circles representing request success and failure rates" >}} ## Before you begin To use the Service Graph, you need: - **Service graph metrics generation** configured in [Grafana Alloy](https://grafana.com/docs/tempo//metrics-from-traces/service_graphs/enable-service-graphs/) or [the Tempo metrics-generator](https://grafana.com/docs/tempo//metrics-from-traces/service_graphs/). Without this, no service graph data exists. - **A Prometheus data source** in Grafana that receives the generated metrics. - **The Prometheus data source linked** in the Tempo data source settings under [Service graph](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/additional-settings/#service-graph). Refer to [Additional settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/additional-settings/#service-graph) for details, or [Provision the data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/) for YAML configuration. If you run a Service Graph query and see an empty result, refer to [Troubleshoot](#troubleshoot) on this page. ## How it works The metrics generator or Alloy inspects trace spans for parent-child relationships that represent a request, using [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/general/trace/). It writes the following metrics to your Prometheus backend: - `traces_service_graph_request_total`: Total requests between two services (counter). - `traces_service_graph_request_failed_total`: Failed requests between two services (counter). - `traces_service_graph_request_server_seconds`: Response time as observed by the server (histogram). - `traces_service_graph_request_client_seconds`: Response time as observed by the client (histogram). Grafana reads these metrics from the linked Prometheus data source to render the node graph and calculate request rates, error rates, and response times. For the full metrics reference, including labels, virtual nodes, and additional metrics, refer to [Service graphs](https://grafana.com/docs/tempo//metrics-from-traces/service_graphs/) in the Tempo documentation. ## Display the Service Graph To display the Service Graph: 1. Navigate to [Explore](https://grafana.com/docs/grafana//explore/). 1. Select the Tempo data source. 1. Select the **Service Graph** query type. 1. Run a query. 1. _(Optional)_ Filter by service name. Each circle in the graph represents a service. Numbers inside the circles indicate the response time and requests per second. Each circle's color shows the proportion of successful and failed requests: | Color | Meaning | | ---------- | ------------------- | | **Green** | Success | | **Red** | Fault | | **Yellow** | Errors | | **Purple** | Throttled responses | The color mix of each circle shows the proportion of each state. A mostly green circle indicates healthy traffic; visible red, yellow, or purple arcs indicate faults, errors, or throttling respectively. To open a context menu with links for quick navigation to traces, metrics, and related views, click a service. For details on navigating, zooming, and interacting with the graph layout, refer to the [Node graph panel](https://grafana.com/docs/grafana//panels-visualizations/visualizations/node-graph/) documentation. To investigate a service with a high error rate, use TraceQL: ```traceql { resource.service.name = "" && status = error } ``` Replace _``_ with the name shown in the Service Graph node. Refer to [TraceQL query examples](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-query-examples/) for more queries you can run from Service Graph observations. ## Service Graph view The Service Graph view displays a table of RED metrics (rate, error rate, duration) alongside the node graph. The table uses a different set of metrics from the node graph: - `traces_spanmetrics_calls_total`: Rate and error rate columns. - `traces_spanmetrics_latency_bucket`, or `traces_spanmetrics_latency` for native histograms: Duration column. These span metrics must be present in your linked Prometheus data source. Span metrics generation must be enabled in your Tempo or Alloy configuration. Refer to [Enable service graphs](https://grafana.com/docs/tempo//metrics-from-traces/service_graphs/enable-service-graphs/) in the Tempo documentation. {{< figure src="/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph.png" class="docs-image--no-shadow" max-width="500px" alt="Screenshot of the Service Graph view showing a table of RED metrics alongside the node graph" >}} For details on the Service Graph view, refer to the [Service Graph view documentation](https://grafana.com/docs/tempo//metrics-generator/service-graph-view/). To open the Service Graph view, follow the same steps as [Display the Service Graph](#display-the-service-graph). The table appears alongside the node graph after you run the query. To open a query in Prometheus with the span name of that row automatically set in the query, click a row in the **rate**, **error rate**, or **duration** columns. ![Linked Prometheus query opened from the Service Graph view rate column](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph-prom.png) To open a query in Tempo with the span name of that row automatically set in the query, click a row in the **links** column. You can also use TraceQL to query the same data programmatically. Refer to [TraceQL query examples](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-query-examples/) to translate Service Graph observations into queries. ## Troubleshoot If the Service Graph isn't displaying data, the table is empty, or you see high cardinality warnings, refer to [Service graph issues](https://grafana.com/docs/grafana//datasources/tempo/troubleshooting/#service-graph-issues) in the troubleshooting guide. ## Next steps - [Enable service graphs](https://grafana.com/docs/tempo//metrics-from-traces/service_graphs/enable-service-graphs/) - Set up metric generation in Tempo or Alloy. - [Node graph panel](https://grafana.com/docs/grafana//panels-visualizations/visualizations/node-graph/) - Navigate and customize the graph layout. # Span filters When working with traces, especially those comprising a vast number of spans, pinpointing specific spans of interest can be a daunting task. This is where span filtering comes in. Located above the trace timeline viewer, the **Filters** bar lets you refine the spans displayed based on specific criteria. Whether you're looking to identify spans from a certain service, those exceeding a particular duration, or spans tagged with specific attributes, span filtering streamlines the process. The more filters you add, the more specific the results become. {{< admonition type="tip" >}} If you arrived from the [Service Graph](https://grafana.com/docs/grafana//datasources/tempo/service-graph/), use **service.name** and **status = error** filters to drill into the specific service nodes showing errors. {{< /admonition >}} ![The Filters section in the trace timeline viewer showing the filter drop-down, quick filter pills for Critical path, Errors, and High latency, navigation arrows, match count, and Show all spans toggle.](/media/docs/grafana/data-sources/tempo/screenshot-tempo-datasource-span-filters.png) ## Filter types The **Filters** text box lets you pick a key, choose an operator, and enter or select a value. Available filter keys include: - **Text search**: free-text search across span content - **duration**: filter by span duration (operators: `=`, `>=`, `<=`, `>`, `<`) - **service.name**: the service that produced the span - **span.name**: the operation name of the span - **status**: span status (`ok`, `error`, `unset`) - **kind**: span kind (for example, `server`, `client`, `internal`) - Any span tag, process tag, or log field key present in the trace For most attribute keys, the available operators are `=`, `!=`, `=~` (regular expression match), and `!~` (regular expression not match). {{< admonition type="note" >}} The same attributes available as filter keys (for example, `span.name`, `service.name`, `duration`) can be used directly in [TraceQL queries](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-editor/). {{< /admonition >}} ## Quick filter pills Below the text box, three quick-filter pills let you apply common filters with a single click: | Pill | Effect | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------- | | **Critical path** | Highlights spans in the critical path. The longest sequence of dependent tasks that determines the trace's minimum duration. | | **Errors** | Selects spans where `status = error`. | | **High latency** | Selects the 10% longest spans in the trace (p90 threshold). | Click a pill to toggle the filter on or off. ## Use span filters The **Filters** bar is always visible above the trace timeline viewer. To add a filter: 1. Click the **Filters** text box (placeholder text: _Filter by attribute or text_). 1. Select a key from the dropdown (for example, **service.name** or **duration**). 1. Choose an operator (for example, `=` or `>=`). 1. Select or type a value. The text box suggests values extracted from the current trace. The trace timeline viewer updates as you add filters, highlighting the matching spans. ### Show all spans By default, the **Show all spans** toggle is enabled, meaning all spans remain visible with matching spans highlighted. Turn off **Show all spans** to hide non-matching spans and display only the spans that meet your filter criteria. This is particularly useful when sifting through thousands of spans, allowing you to zero in on those that truly matter. The toggle is disabled when no filters are active. ### Navigate between matches Use the up and down arrow buttons next to the match count to jump between matching spans in the trace timeline. The match count displays the current position and total matches (for example, _2/5 matches_). ### Remove a filter To remove a filter, click the **×** on the filter chip below the **Filters** text box. ## Common use cases To find all errors from a specific service, select `service.name` = `checkout`, then click the **Errors** pill. To isolate slow database calls, select `span.name` =~ `db\..*`, then select `duration` > `200ms`. To narrow to the critical path only, click the **Critical path** pill, then turn off **Show all spans**. ## Next steps - [Service Graph](https://grafana.com/docs/grafana//datasources/tempo/service-graph/): Visualize service relationships and identify error-producing nodes. - [Query editor](https://grafana.com/docs/grafana//datasources/tempo/query-editor/): Build TraceQL queries using the same attributes available as span filter keys. - [Configure the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/): Set up your Tempo data source and trace correlations. # Additional settings The **Additional settings** section of the Tempo data source settings contains options that most users don't need to change. These settings are collapsed by default. Use the down arrow to expand the **Additional settings** section to view these options. This page covers the following settings: - **[Advanced HTTP settings](#advanced-http-settings)**: Configure cookie forwarding and request timeout - **[Service graph](#service-graph)**: Link a Prometheus data source for Service Graph visualization - **[Node graph](#node-graph)**: Enable the node graph above the trace view - **[Tempo search](#tempo-search)**: Show or hide the search option in Explore - **[TraceID query](#traceid-query)**: Narrow TraceID lookups to a time range for performance - **[Tags time range](#tags-time-range)**: Control the time window for tag and tag-value queries - **[Tag limit](#tag-limit)**: Set the maximum number of tags retrieved from Tempo - **[Span bar](#span-bar)**: Display duration or a custom tag in the span bar - **[Private data source connect](#private-data-source-connect)**: Connect to data sources in private networks ## Advanced HTTP settings The Grafana Proxy deletes forwarded cookies. Use the **Allowed cookies** field to specify cookies by name that should be forwarded to the data source. The **Timeout** field sets the HTTP request timeout in seconds. ## Service graph The **Service graph** setting configures the [Service Graph](https://grafana.com/docs/tempo//metrics-from-traces/service_graphs/enable-service-graphs/) data. You must link a Prometheus data source in the **Data source** setting for the [Service Graph](https://grafana.com/docs/grafana//datasources/tempo/service-graph/) to display data. Without this, Service Graph queries return empty results. The linked Prometheus data source must contain metrics generated by [Grafana Alloy](https://grafana.com/docs/tempo//configuration/grafana-alloy/) or the [Tempo metrics generator](https://grafana.com/docs/tempo//metrics-generator/service_graphs/#tempo). For the full list of prerequisites and troubleshooting steps, refer to [Service Graph and Service Graph view](https://grafana.com/docs/grafana//datasources/tempo/service-graph/). {{< admonition type="note" >}} If you cloned a provisioned data source (for example, the Grafana Cloud Traces data source) and the **Service graph** section doesn't appear in the cloned data source's settings, verify that the cloned data source has the same Tempo backend URL and authentication configured. The Service graph section only appears when the data source can connect to Tempo successfully. Refer to [Clone a provisioned data source for Grafana Cloud](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/#clone-a-provisioned-data-source-for-grafana-cloud) for setup steps. {{< /admonition >}} ## Node graph The **Node graph** setting enables the [node graph visualization](https://grafana.com/docs/grafana//panels-visualizations/visualizations/node-graph/), which isn't activated by default. After it's activated, Grafana displays the node graph above the trace view. ## Tempo search The **Search** setting configures [Tempo search](https://grafana.com/docs/tempo/latest/configuration/#search). You can configure the **Hide search** setting to hide the search query option in **Explore** if search is not configured in the Tempo instance. ## TraceID query The **TraceID query** setting modifies how TraceID queries are run. The time range can be used when there are performance issues or timeouts since it narrows down the search to the defined range. This setting is disabled by default. You can configure this setting as follows: | Name | Description | | ---------------------------------- | ----------------------------------------------------------- | | **Use time range in query** | Use a time range in the TraceID query. Default: `disabled`. | | **Time shift for start of search** | Time shift for start of search. Default: `30m`. | | **Time shift for end of search** | Time shift for end of search. Default: `30m`. | ## Tags time range The **Tags time range** feature controls how tag and tag-value queries are executed by specifying the time window applied to these requests. You can select one of the following options to constrain your queries: | Name | Description | | ------------------- | ---------------------------------- | | **Last 30 minutes** | Last 30 minutes of selected range. | | **Last 3 hours** | Last 3 hours of selected range. | | **Last 24 hours** | Last 24 hours of selected range. | | **Last 3 days** | Last 3 days of selected range. | | **Last 7 days** | Last 7 days of selected range. | ## Tag limit The **Tag limit** setting modifies the max number of tags and tag values to retrieve from Tempo. Default: 5000 ## Span bar The **Span bar** setting helps you display additional information in the span bar row. You can choose one of three options: | Name | Description | | ------------ | -------------------------------------------------------------------------------------------------------------------------------- | | **None** | Adds nothing to the span bar row. | | **Duration** | _(Default)_ Displays the span duration on the span bar row. | | **Tag** | Displays the span tag on the span bar row. You must also specify which tag key to use to get the tag value, such as `component`. | ## Private data source connect [//]: # 'Shared content for authentication section procedure in data sources' {{< docs/shared source="grafana" lookup="datasources/datasouce-private-ds-connect.md" leveloffset="+1" version="" >}} ## Next steps - [Configure the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/): Return to connection, authentication, and streaming settings. - [Provision the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/): Configure the Tempo data source using a YAML file. - [Service Graph and Service Graph view](https://grafana.com/docs/grafana//datasources/tempo/service-graph/): Visualize service dependencies and monitor RED metrics. # Configure trace to logs correlation Trace to logs correlation lets you navigate from a span in Tempo directly to the matching logs in Loki, and from a log line in Loki back to its trace in Tempo. When configured, a **Logs for this span** link appears in the trace view and a link to the configured tracing data source appears on log lines that contain a trace ID. This is a two data source configuration: - **Tempo data source:** Controls how Grafana queries logs when you click a span, including the time window, query, and tags to match. - **Loki data source:** Controls how trace IDs are extracted from log lines and linked back to traces. Both sides must be configured for the correlation to work in either direction. ![Trace to logs settings](/media/docs/grafana/data-sources/tempo/tempo-data-source-trace-to-logs.png) ## Before you begin To configure trace to logs correlation, you need: - A [Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/) configured in Grafana - A [Loki data source](https://grafana.com/docs/grafana//datasources/loki/configure/) configured in Grafana - Applications that emit traces and logs with a shared identifier such as a trace ID, service name, or both - Editor or Admin permissions in Grafana {{< admonition type="note" >}} You can't modify a provisioned data source from the Grafana UI. If you're using Grafana Cloud Traces (the pre-configured tracing data source in Grafana Cloud), its settings are read-only. To configure trace to logs, [clone the data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/#clone-a-provisioned-data-source-for-grafana-cloud) to create an editable copy, or update the provisioning file for self-managed instances. Refer to [Provision trace to logs settings](#provision-trace-to-logs-settings) for details. {{< /admonition >}} ## Configure the Tempo data source These settings control how Grafana queries your log data source when you click a span in the trace view. This guide uses Loki, but Trace to logs also supports Elasticsearch, Splunk, OpenSearch, Falcon LogScale, Google Cloud Logging, and VictoriaMetrics Logs. {{< admonition type="note" >}} To change the target log data source, for example from Elasticsearch to Loki, select the new data source from the **Data source** drop-down in the following steps and update the tags and custom query to match its query language. The reverse direction, from a log line back to a trace, is configured on the log data source itself, and the mechanism differs by type: Loki uses [Derived fields](https://grafana.com/docs/grafana//datasources/loki/configure/#derived-fields), while Elasticsearch and Splunk use [Data links](https://grafana.com/docs/grafana//datasources/elasticsearch/configure/#data-links). {{< /admonition >}} 1. Go to **Connections** > **Data sources** and select your Tempo data source. 1. Scroll to the **Trace to logs** section. 1. Select your Loki data source from the **Data source** drop-down list. 1. Configure the **Span start time shift** and **Span end time shift** fields. These fields expand the log query time window around the span's start and end timestamps. A span's timestamps are precise to the millisecond, but log lines are often written slightly before or after the span boundary. The default value of `0` can return no logs if timestamps don't align exactly. A common starting point is `-2s` for start shift and `2s` for end shift. You can increase this value for spans that represent slow or batch operations, for example `5s`. 1. Configure **Tags** to map span attributes to Loki label names. Grafana uses these tags to filter logs to the correct service, Pod, or cluster. The Loki label name you enter must exactly match the label name on your Loki streams. A mismatch silently breaks the correlation with no error shown. If you don't configure any tags, Grafana uses the following defaults: `cluster`, `hostname`, `namespace`, `pod`, `service.name` (remapped to `service_name`), and `service.namespace` (remapped to `service_namespace`). The automatic dot-to-underscore remapping only applies to these default tags. For custom tags, you must specify the remapped name explicitly. To customize the mapping, use the **Add tag** button to add each row. The following table shows common tags and their typical Loki label equivalents: | Span attribute | Loki label | | ------------------- | ------------------- | | `service.name` | `service_name` | | `service.namespace` | `service_namespace` | | `cluster` | `cluster` | | `namespace` | `namespace` | | `pod` | `pod` | | `hostname` | `hostname` | Choose labels with low cardinality. In Loki, every unique combination of label values creates a separate stream. Labels that take on many distinct values, such as `pod`, `host`, `thread`, `duration`, `traceId`, or `spanId`, can create hundreds of thousands or even millions of streams, causing slow queries, high memory usage, and log loss at ingest. The labels listed above (`service_name`, `namespace`, `cluster`) are good choices because they have a bounded, small set of values. If you need to query high-cardinality values like trace IDs or Pod names, store them as [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) instead. They are queryable without creating additional streams. For more information, refer to [Cardinality](https://grafana.com/docs/loki/latest/get-started/labels/cardinality/). {{< admonition type="note" >}} When using Loki, the **Logs for this span** link only appears on a span if at least one of the configured tags exists on that span. If none of the tags match, the link doesn't appear and no error is shown. {{< /admonition >}} 1. Configure the **Filter by trace ID** and **Filter by span ID** toggles. - **Filter by trace ID:** Filters Loki results to all logs for the entire trace, not just the selected span. This works well for most use cases. - **Filter by span ID:** Filters to logs for a single span. This only works if your logs include a span ID field. These toggles are disabled when **Use custom query** is enabled because the custom query controls filtering directly. Clicking **Logs for this span** filters by trace ID, not span ID, so results include all logs for the entire trace that contain a matching trace ID, not only logs emitted during that specific span. If your results look broader than expected, this is expected behavior. To narrow results to a specific span's activity, write a custom LogQL query that filters on span ID or span-specific attributes if your application logs them, for example `{service_name="my-service"} | json | spanId=""`. For this feature to work at all, your application must inject trace IDs into log lines, for example, using OpenTelemetry SDK structured logging. Without trace IDs in logs, the query falls back to a time-range filter. 1. Optional: Enable **Use custom query** and write a LogQL expression to replace the auto-generated query. Use `${__tags}` to inject the mapped tag filters automatically: ```logql {${__tags}} | logfmt | trace_id=`${__trace.traceId}` ``` If a tag like `pod` is stored as Loki [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) instead of an indexed label, it can't appear in the stream selector `{}`. Move it to a pipeline filter instead: ```logql {${__tags}} | pod=`${__span.tags["k8s.pod.name"]}` |= `${__trace.traceId}` ``` Refer to [Custom query variables](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/#custom-query-variables) for the full list of available variables. 1. Click **Save & test**. ## Configure the Loki data source These settings enable the **Tempo** link on Loki log lines, letting you navigate from a log entry to its trace. {{< admonition type="note" >}} Configuring the Tempo data source without also configuring the Loki data source is a common mistake. If you skip this step, clicking a log line doesn't open a trace. Both sides are required. {{< /admonition >}} 1. Go to **Connections** > **Data sources** and select your Loki data source. 1. In the **Additional settings** section, scroll to **Derived fields**. 1. Click **Add** to add a new derived field. 1. Enter a **Name** for the field, for example `TraceID`. 1. Select the **Type** and enter a pattern to extract the trace ID from log lines. If you select **Regex in log line**, enter a regular expression with one capture group. The following table shows common patterns: | Log format | Regex | | --------------------------------- | ------------------- | | `traceID=` | `traceID=(\w+)` | | `trace_id=` | `trace_id=(\w+)` | | JSON field `"traceId": ""` | `"traceId":"(\w+)"` | If you select **Label**, enter a regular expression that matches the label key, for example `trace[_]?id` to match both `traceid` and `trace_id`. {{< admonition type="tip" >}} If your applications use different trace ID field names, you need a separate derived field entry for each format. To simplify this, standardize on OpenTelemetry structured metadata so trace IDs are stored as a consistent label. {{< /admonition >}} 1. Enable the **Internal link** toggle and select your Tempo data source from the drop-down list. 1. In the **Query** field, enter `${__value.raw}`. This passes the extracted trace ID to Tempo when the link is clicked. The field shows `${__value.raw}` as placeholder text, but you must enter it as the actual value. 1. Click **Save & test**. For more details about derived field options, refer to [Derived fields](https://grafana.com/docs/grafana//datasources/loki/configure/#derived-fields) in the Loki data source documentation. ## Provision trace to logs settings You can provision the trace to logs configuration using the `tracesToLogsV2` block in your data source YAML file. For the full provisioning YAML example including all Tempo settings, refer to [Provision the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/). For general provisioning instructions, refer to [Provisioning data sources](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ## Example: NGINX service This example assumes: - Applications running in Kubernetes that emit OpenTelemetry traces to Tempo - Logs sent to Loki with labels `service_name`, `namespace`, and `pod` - Trace IDs written to logs as `trace_id=` The Tempo data source settings are: | Field | Value | | ------------------------- | ----------------------------------------------------------- | | **Data source** | Loki | | **Span start time shift** | `-2s` | | **Span end time shift** | `2s` | | **Tags** | `service.name` mapped to `service_name`, `namespace`, `pod` | | **Filter by trace ID** | Enabled | | **Filter by span ID** | Disabled | The Loki data source derived field settings are: | Field | Value | | ----------------- | --------------------- | | **Name** | TraceID | | **Type** | Regex in log line | | **Regex** | `trace_id=(\w+)` | | **Internal link** | Enabled, select Tempo | | **Query** | `${__value.raw}` | ### Expected result After configuring both data sources: - In **Explore** with the Tempo data source selected, click any span and then click **Logs for this span**. Loki opens filtered to `service_name=nginx` and `namespace=production` within the configured time window around the span. - In **Explore** with the Loki data source selected, any log line containing `trace_id=...` shows a link to your Tempo data source. Clicking it opens the full trace in Tempo. ## Troubleshooting If trace to logs links aren't appearing, return no data, or only work for some services, refer to [Trace to logs/metrics/profiles issues](https://grafana.com/docs/grafana//datasources/tempo/troubleshooting/#trace-to-logsmetricsprofiles-issues) in the troubleshooting guide. If the configuration fields are greyed out, your data source is provisioned. Refer to [Provision trace to logs settings](#provision-trace-to-logs-settings) for options. ## Next steps - [Configure trace to metrics correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/): Link spans to metrics queries in Prometheus. - [Configure trace to profiles correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/): Link spans to profiling data in Grafana Pyroscope. - [Configure the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/): Return to connection, authentication, and streaming settings. # Configure trace to metrics correlation Trace to metrics lets you navigate from a span in a trace to metrics in a Prometheus or other metrics data source. When configured, a **Metrics for this span** link appears in the trace view. Trace to metrics works with any Prometheus-compatible data source and doesn't require the Tempo metrics generator. The metrics generator creates _new_ metrics from trace data; trace to metrics links to metrics that _already exist_ in your metrics data source. {{< youtube id="TkapvLeMMpc" >}} ## Before you begin To configure trace to metrics, you need: - A [Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/) configured in Grafana - A Prometheus-compatible metrics data source configured in Grafana - Metrics that correspond to your traced services (for example, `requests_total`, `request_duration_seconds`) - Editor or Admin permissions in Grafana {{< admonition type="note" >}} You can't modify a provisioned data source from the Grafana UI. If you're using Grafana Cloud Traces (the pre-configured tracing data source in Grafana Cloud), its settings are read-only. To configure trace to metrics, [clone the data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/#clone-a-provisioned-data-source-for-grafana-cloud) to create an editable copy, or update the provisioning file for self-managed instances. Refer to the [Provisioning](#provisioning) section for details. {{< /admonition >}} There are two ways to configure the trace to metrics feature: - Use a basic configuration with a default query. - Configure one or more custom queries where you can use a [template language](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/) to interpolate variables from the trace or span. ![Trace to metrics settings in the Tempo data source](/media/docs/grafana/data-sources/tempo/tempo-data-source-trace-to-metrics.png) ## Set up a basic configuration To use a basic configuration, follow these steps: 1. Select a metrics data source from the **Data source** drop-down. 1. Optional: Change **Span start time shift** and **Span end time shift**. The placeholders show `-2m` (start) and `2m` (end), which are applied if you leave the fields empty. 1. Optional: Choose any tags to use in the query. Click **Add tag** to add a tag mapping. The tags you configure must be present in the spans attributes or resources for a trace to metrics span link to appear. You can optionally configure a new name for the tag. This is useful if the tag has dots in the name and the target data source doesn't allow using dots in labels. For example, you can remap `service.name` to `service_name`. 1. Don't select **Add query**. 1. Select **Save & test**. ## Set up custom queries To use custom queries, you need to configure the tags you'd like to include in the linked queries. For each tag, the key is the span attribute name. In cases where the attribute name would result in an invalid metrics query or doesn't exactly match the desired label name, you can enter the label name as the second value. For example, you could map the attribute `k8s.pod` to the label `pod`. You can interpolate the configured tags using the `$__tags` keyword. For example, when you configure the query `requests_total{$__tags}` with the tags `k8s.pod=pod` and `cluster`, it results in `requests_total{pod="nginx-554b9", cluster="us-east-1"}`. The label values are dynamically inserted based on the span attributes' values. You can link to any metric you'd like, but metrics for span durations, counts, and errors filtered by service or span are a great starting point. To use custom queries with the configuration, follow these steps: 1. Select a metrics data source from the **Data source** drop-down. 1. Optional: Choose any tags to use in the query. Click **Add tag** to add a tag mapping. These tags can be used in the custom query with the `${__tags}` variable. This variable interpolates the mapped tags as a list in an appropriate syntax for the data source and only includes the tags that were present in the span, omitting those that weren't present. You can optionally configure a new name for the tag. This is useful in cases where the tag has dots in the name and the target data source doesn't allow using dots in labels. For example, you can remap `service.name` to `service_name`. If you don't map any tags here, you can still use any tag in the query like this: `method="${__span.tags.method}"`. Refer to [Custom query variables](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/#custom-query-variables) for the full list of available variables. 1. Click **Add query** to add a custom query. 1. Specify a custom query to be used to query metrics data. Each linked query consists of: - **Link Label:** _(Optional)_ Descriptive label for the linked query. - **Query:** The query ran when navigating from a trace to the metrics data source. Interpolate tags using the `$__tags` keyword. For example, when you configure the query `requests_total{$__tags}` with the tags `k8s.pod=pod` and `cluster`, the result looks like `requests_total{pod="nginx-554b9", cluster="us-east-1"}`. 1. Select **Save & test**. ## Configuration options The following table describes options for configuring the **Trace to metrics** settings: | Setting name | Description | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Data source** | Defines the target data source. | | **Span start time shift** | Shifts the start time for the metrics query, based on the span's start time. You can use time units, such as `5s`, `1m`, `3h`. To extend the time to the past, use a negative value. Default: `-2m`. | | **Span end time shift** | Shifts the end time for the metrics query, based on the span's end time. You can use time units. Default: `2m`. | | **Tags** | Defines the tags used in linked queries. The key sets the span attribute name, and the optional value sets the corresponding metric label name. For example, you can map `k8s.pod` to `pod`. To interpolate these tags into queries, use the `$__tags` keyword. | | **Link Label** | _(Optional)_ Descriptive label for the linked query. | | **Query** | Input to write a custom query. Use variable interpolation to customize it with variables from span. | ## Provisioning You can provision the trace to metrics configuration using the `tracesToMetrics` block in your data source YAML file. For the full provisioning YAML example including all Tempo settings, refer to [Provision the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/). ## Link from metrics to traces To navigate in the reverse direction, from a metric to its associated trace, configure exemplars in your Prometheus data source. Refer to [Exemplars](https://grafana.com/docs/grafana//fundamentals/exemplars/) for setup instructions. ## Troubleshooting If trace to metrics links aren't appearing or return no data, refer to [Trace to logs/metrics/profiles issues](https://grafana.com/docs/grafana//datasources/tempo/troubleshooting/#trace-to-logsmetricsprofiles-issues) in the troubleshooting guide. If the configuration fields are greyed out, your data source is provisioned. Refer to the [Provisioning](#provisioning) section for how to update the configuration via YAML. ## Next steps - [Configure trace to logs correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/): Navigate from spans to related logs in Loki. - [Configure trace to profiles correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/): Link spans to profiling data in Grafana Pyroscope. - [Provision the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/): Configure the Tempo data source using a YAML file. # Configure trace to profiles correlation Trace to profiles lets you navigate from a span in a trace directly to profiling data in [Grafana Pyroscope](https://grafana.com/docs/pyroscope//). When configured, a **Profiles for this span** button and an embedded flame graph appear in the span details view. ## Before you begin To configure trace to profiles, you need: - A [Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/) configured in Grafana - A [Grafana Pyroscope data source](https://grafana.com/docs/grafana//datasources/grafana-pyroscope/) configured in Grafana - An application instrumented with the [OpenTelemetry span profiling bridge](https://grafana.com/docs/pyroscope//configure-client/trace-span-profiles/) so that spans include the `pyroscope.profile.id` attribute - Editor or Admin permissions in Grafana {{< admonition type="note" >}} You can't modify a provisioned data source from the Grafana UI. If you're using Grafana Cloud Traces (the pre-configured tracing data source in Grafana Cloud), its settings are read-only. To configure trace to profiles, [clone the data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/#clone-a-provisioned-data-source-for-grafana-cloud) to create an editable copy, or update the provisioning file for self-managed instances. Refer to the [Provisioning](#provisioning) section for details. {{< /admonition >}} [//]: # 'Shared content for Trace to profiles in the Tempo data source' {{< docs/shared source="grafana" lookup="datasources/tempo-traces-to-profiles.md" leveloffset="+1" version="" >}} ## Provisioning You can provision the trace to profiles configuration using the `tracesToProfiles` block in your data source YAML file. For the full provisioning YAML example including all Tempo settings, refer to [Provision the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/). ## Verify the integration After configuring trace to profiles, verify the integration: 1. Open **Explore** and select your Tempo data source. 1. Run a query to load a trace. 1. Expand a span and confirm that the **Profiles for this span** button appears in the span details. 1. Click the button and verify that an embedded flame graph loads with profiling data. ## Troubleshooting If trace to profiles links aren't appearing or the flame graph is empty, refer to [Trace to logs/metrics/profiles issues](https://grafana.com/docs/grafana//datasources/tempo/troubleshooting/#trace-to-logsmetricsprofiles-issues) in the troubleshooting guide. For additional diagnosis steps, refer to [Troubleshoot trace to profiles](https://grafana.com/docs/pyroscope//configure-client/trace-span-profiles/) in the Pyroscope documentation. If the configuration fields are greyed out, your data source is provisioned. Refer to the [Provisioning](#provisioning) section for how to update the configuration via YAML. ## Next steps - [Configure trace to logs correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/): Navigate from spans to related logs in Loki. - [Configure trace to metrics correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/): Link spans to metrics queries in Prometheus. - [Provision the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/): Configure the Tempo data source using a YAML file. # Provision the Tempo data source You can define and configure the Tempo data source in YAML files as part of the Grafana provisioning system. You can use version control, like Git, to track and manage file changes. Changes can be updated or rolled back as needed. {{< admonition type="note" >}} Provisioning via YAML is primarily used for self-managed Grafana instances. {{< /admonition >}} For more information about provisioning and available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ## Provisioned data sources You can't modify a provisioned data source from the Grafana UI. The settings form is read-only and the **Save & test** button is replaced by **Test** (you can test the connection but not save changes). To make changes, you can either: - **Clone the data source:** Create a new data source of the same type and copy the settings from the provisioned data source. Refer to [Clone a provisioned data source for Grafana Cloud](#clone-a-provisioned-data-source-for-grafana-cloud) for detailed steps. - **Update the provisioning file:** Edit the YAML configuration file and restart Grafana, or wait for the provisioning system to pick up the change. Any setting, including span time shifts on trace to logs, can be changed by editing the YAML. ## Example file This example provisioning YAML file sets up the equivalents of the options available in the Tempo data source UI. Replace `grafana-pyroscope-datasource` with the actual UID of your Pyroscope data source, and verify the other `datasource Uid` values match what's actually provisioned. ```yaml apiVersion: 1 datasources: - name: Tempo type: tempo uid: EbPG8fYoz url: http://localhost:3200 access: proxy basicAuth: false jsonData: tracesToLogsV2: # Field with an internal link pointing to a logs data source in Grafana. # datasourceUid value must match the uid value of the logs data source. datasourceUid: 'loki' spanStartTimeShift: '-1h' spanEndTimeShift: '1h' tags: [{ key: 'job' }, { key: 'instance' }, { key: 'pod' }, { key: 'namespace' }] filterByTraceID: false filterBySpanID: false customQuery: true query: 'method="$${__span.tags.method}"' tracesToMetrics: datasourceUid: 'prom' spanStartTimeShift: '-1h' spanEndTimeShift: '1h' tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }] queries: - name: 'Sample query' query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))' tracesToProfiles: datasourceUid: 'grafana-pyroscope-datasource' tags: [{ key: 'job' }, { key: 'instance' }, { key: 'pod' }, { key: 'namespace' }] profileTypeId: 'process_cpu:cpu:nanoseconds:cpu:nanoseconds' customQuery: true query: 'method="$${__span.tags.method}"' serviceMap: datasourceUid: 'prometheus' nodeGraph: enabled: true search: hide: false traceQuery: timeShiftEnabled: true spanStartTimeShift: '-1h' spanEndTimeShift: '1h' spanBar: type: 'Tag' tag: 'http.path' streamingEnabled: search: true metrics: true ``` For details about individual settings, refer to: - [Configure trace to logs correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/): `tracesToLogsV2` block - [Configure trace to metrics correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/): `tracesToMetrics` block - [Configure trace to profiles correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/): `tracesToProfiles` block - [Additional settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/additional-settings/): `serviceMap`, `nodeGraph`, `search`, `traceQuery`, `spanBar` blocks ## Clone a provisioned data source for Grafana Cloud If you have a data source that is provisioned by a configuration file in Grafana Cloud, you can clone that provisioned data source and then edit the new data source in the Grafana UI. For example, if you want to edit the trace to logs settings in your Tempo data source that is provisioned on Grafana Cloud, you can enable traceID and spanID filtering by cloning the data source. To clone a provisioned data source, follow these steps: 1. Create a viewer [Cloud Access Policy token](https://grafana.com/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-policies/) in the Grafana Cloud Portal, making sure it has read permissions at least for the data types you are trying to clone. 1. [Create a new data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/#add-a-new-data-source) of the same type you want to clone. 1. Copy all of the settings from the existing provisioned data source into the new data source while replacing the password with the API key you created. The easiest way to do this is to open separate browser windows with the provisioned data source in one and the newly created data source in another. After copying the HTTP and Auth section details, pasting the Cloud Access Policy token into the Password field, and changing any of the other options that you want, you can save and test the data source. ## Next steps - [Configure the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/): Return to connection, authentication, and streaming settings. - [Configure trace to logs correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/): Set up span-to-log navigation. - [Additional settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/additional-settings/): Configure Service graph, node graph, search, and other settings. # Trace correlations You can use Grafana [correlations](https://grafana.com/docs/grafana//administration/correlations/) to embed interactive correlation links in your trace view to jump from spans to related logs, metrics, profiles, or external systems. This guide explains how to configure and manage Trace correlations in Grafana. ## What are trace correlations? Trace correlations let you define rules that inject context-sensitive links into your trace spans. When viewing traces in Explore or the Traces panel, users can click these links to navigate directly to relevant queries or URLs. Correlations are similar but more flexible to the [trace to logs, metrics, and profiles links you can configure for the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/). {{< figure src="/media/docs/tempo/screenshot-trace-view-correlations.png" max-width="900px" class="docs-image--no-shadow" alt="Using correlations for a trace" >}} ## Before you begin To use trace correlations, you need: - Grafana 12 or later - A [Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/) configured in Grafana - Admin access to configuration settings or provisioning files in Grafana ## Set up a trace correlation 1. Log in to Grafana with an administrator account. 1. Go to **Administration** > **Plugins and data** > **Correlations**. 1. Select **Add correlation** or **Add new**. 1. On step 1, provide a **label** for the correlation, and an optional **description**. 1. On step 2, configure the correlation **target**. - Select the **Type** drop-down list and choose **Query** to link to another data source or choose **External** for a custom URL. - For a query **Target**, select the target drop-down list and select the data source that should be queried when the link is clicked. Define the target query. - For an external **Target**, enter the **External URL**. - For both query and external targets, you can use the following variables based on trace data. Object variables must be parsed into a value variable with a regular expression transformation. | Variable | Type | Description | | -------------- | ------ | ---------------------- | | `traceId` | String | Trace identifier | | `spanID` | String | Span identifier | | `parentSpanID` | String | Parent span identifier | | `serviceName` | String | Service name | | `serviceTags` | Object | Resource attributes | | `tags` | Object | Span attributes | | `logs` | Object | Trace events | | `references` | Object | Trace links | {{< figure src="/media/docs/tempo/screenshot-grafana-trace-correlations-loki-step-2.png" max-width="900px" class="docs-image--no-shadow" alt="Setting up a correlation for a Loki target using trace variables" >}} 1. On step 3, configure the correlation data source: - Select your Tempo data source in the **Source** drop-down list. - Enter the trace data variable you use for the correlation in the **Results field**. - Optionally, add one or more **Transformations** to parse the trace data into additional variables. You can use these variables to configure the correlation **Target**. {{< figure src="/media/docs/tempo/screenshot-grafana-trace-correlations-loki-step-3.png" max-width="900px" class="docs-image--no-shadow" alt="Setting up a correlation for a Loki data source" >}} 1. Select **Save** to save the correlation. ## Verify correlations in Explore 1. Open **Explore** and select your Tempo tracing source. 1. Run a query to load spans. 1. Hover over the span links menu or open the span details to reveal the correlation link buttons. {{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-correlations.png" max-width="900px" class="docs-image--no-shadow" alt="Using correlations for a trace" >}} 1. Click a correlation link to open a split view or navigate to your target system or query. ## Examples Below are several practical correlation configurations to get you started. ### Example 1: Trace to logs by service name and trace identifier In this example, you configure trace to logs by service name and a trace identifier. 1. On step 1, add a new correlation with the label **Logs for this service and trace** and an optional description. {{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-correlations-example-1-step-1.png" max-width="900px" class="docs-image--no-shadow" alt="Using correlations for a trace" >}} 1. On step 2, configure the correlation target: - Select the target type **Query** and select your Loki data source as **Target**. - Define the Loki query, using `serviceName` and `traceID` as variables derived from the span data: ``` {service_name="$serviceName"} | trace_id=`$traceID` |= `` ``` In this query, `service_name` is the only [stream label](https://grafana.com/docs/loki/latest/get-started/labels/) used inside `{}`. Stream labels should be low-cardinality values that describe the source of your logs. The `trace_id` field appears after the `|` pipe as a [pipeline filter](https://grafana.com/docs/loki/latest/query/log_queries/#line-filter-expression), which searches log content or [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) without creating additional streams. Don't use trace IDs or other high-cardinality values as stream labels because each unique value creates a separate stream, which degrades Loki performance. For more information, refer to [Label best practices](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/) and [Cardinality](https://grafana.com/docs/loki/latest/get-started/labels/cardinality/). {{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-correlations-example-1-step-2.png" max-width="900px" class="docs-image--no-shadow" alt="Using correlations for a trace" >}} 1. On step 3, configure the correlation source: - Select your Tempo data source as **Source**. - Use `traceID` as **Results field**. - Add a new transformation to extract the `serviceName` from the span `serviceTags` using the regular expression: ``` {(?=[^\}]*\bkey":"service.name")[^\}]*\bvalue":"(.*?)".*} ``` {{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-correlations-example-1-step-3.png" max-width="900px" class="docs-image--no-shadow" alt="Using correlations for a trace" >}} 1. Save the correlation. ### Example 2: Trace to custom URL In this example, you configure trace correlations with a custom URL. 1. On step 1, add a new correlation with the label **Open custom URL** and an optional description. 1. On step 2, configure the correlation target: - Select the target type **External**. - Define your target URL, using variables derived from the span data. In this example, we are using `serviceName` and `traceID`. ``` https://my-server.example.com/service=$serviceName&trace=$traceID ``` 1. On step 3, configure the correlation source: - Select your Tempo data source as **Source**. - Use `traceID` as **Results field**. - Add a new transformation to extract the `serviceName` from the span `serviceTags` using the regular expression: ``` {(?=[^\}]*\bkey":"service.name")[^\}]*\bvalue":"(.*?)".*} ``` 1. Save the correlation. ## Best practices - **Name clearly:** Use descriptive names indicating source and target. For example: **Trace to errors in logs**. - **Use low-cardinality stream labels:** When targeting Loki, use only low-cardinality values like `service_name`, `namespace`, or `cluster` inside the stream selector `{}`. Place high-cardinality values like trace IDs in [pipeline filters](https://grafana.com/docs/loki/latest/query/log_queries/#line-filter-expression) (after `|`) or store them as [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/). Using trace IDs as stream labels creates excessive streams and degrades Loki performance. For more information, refer to [Cardinality](https://grafana.com/docs/loki/latest/get-started/labels/cardinality/) and [Label best practices](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/). For detailed guidance on mapping span attributes to Loki labels, refer to [Configure trace to logs correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/). - **Template wisely:** Use multiple `$variable` tokens if you need to inject more than one field. ## Next steps - [Configure trace to logs correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/): Link spans to log queries in Loki with tag mapping and cardinality guidance. - [Configure trace to metrics correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/): Link spans to metrics queries in Prometheus. - [Configure trace to profiles correlation](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/): Link spans to profiling data in Grafana Pyroscope. - [Configure the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/): Return to connection, authentication, and streaming settings. # Write TraceQL queries with the editor Use the **TraceQL** editor when you need structural queries across parent and child spans, aggregations, or other features that the [Search query builder](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-search/) doesn't support. TraceQL queries follow the pattern `{ conditions } | pipeline`. Refer to [Construct a TraceQL query](https://grafana.com/docs/tempo//traceql/construct-traceql-queries/) for the full syntax. To get started, paste this query into the editor and select **Run query**: ```traceql { resource.service.name = "frontend" && span:status = error } ``` This returns all error spans from the `frontend` service. Replace `frontend` with your service name. For more examples, refer to [TraceQL query examples](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-query-examples/). If queries return no results, check that your [Tempo data source is configured and connected](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/). [//]: # 'Shared content for the TraceQL query editor' [//]: # 'This content is located in /docs/sources/shared/datasources/tempo-editor-traceql.md' {{< docs/shared source="grafana" lookup="datasources/tempo-editor-traceql.md" version="" >}} ## Next steps - [TraceQL query examples](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-query-examples/): Copy-paste query examples for common use cases - [Search traces using the query builder](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-search/): Build queries visually - [Construct a TraceQL query](https://grafana.com/docs/tempo//traceql/construct-traceql-queries/): Full TraceQL syntax reference - [Service Graph and Service Graph view](https://grafana.com/docs/grafana//datasources/tempo/service-graph/): Visualize service dependencies - [Span filters](https://grafana.com/docs/grafana//datasources/tempo/span-filters/): Refine results in the trace detail view # TraceQL query examples Use these TraceQL queries in Grafana **Explore** with a Tempo data source. Adjust service names, routes, and attribute values to match your environment. These examples use [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/general/trace/). If your instrumentation uses different attribute names, substitute them in the queries. For the full TraceQL syntax, refer to [Construct a TraceQL query](https://grafana.com/docs/tempo//traceql/construct-traceql-queries/). For an extended recipe collection, refer to the [TraceQL cookbook for Grafana Cloud Traces](https://grafana.com/docs/grafana-cloud/send-data/traces/traces-query-editor/traceql-cookbook/). {{< admonition type="tip" >}} If a query returns no results, widen the time range and confirm the correct Tempo data source is selected. Prefer trace-level intrinsic fields like `trace:duration` and `trace:rootService` for faster queries. {{< /admonition >}} ## Find error spans Filter by error status or HTTP error codes: ```traceql { status = error } ``` ```traceql { span.http.status_code >= 500 } ``` ```traceql { span.http.status_code > 399 } ``` ## Find slow requests Filter by span duration. Common thresholds are 1 second and 5 seconds: ```traceql { duration > 1s } ``` ```traceql { duration > 5s } ``` Filter a specific endpoint by duration: ```traceql { span.http.url = "/api/checkout" && duration > 2s } ``` ## Find exceptions by type Query exception events recorded on spans. Exception data uses the `event` scope because OpenTelemetry records exceptions as span events: ```traceql { event.exception.message =~ "context cancelled" } ``` ```traceql { event.exception.type = "NotFoundException" } ``` Find error spans that have an exception message: ```traceql { status = error && event.exception.message != "" } ``` ## Filter by service Select spans from a specific service or match services by pattern. You can use the [Service Graph view](https://grafana.com/docs/grafana//datasources/tempo/service-graph/) to identify which services have high error rates or latency, then query those services with TraceQL. ```traceql { resource.service.name = "checkout" } ``` ```traceql { resource.service.name =~ "payment.*" && status = error } ``` ## Find errors and slow spans together Combine error status and duration filters to identify likely root causes: ```traceql { resource.service.name = "api" && status = error && duration > 1s } ``` ## Analyze service dependencies Find errors on a specific API path: ```traceql { span.http.url =~ ".*/api/.*" && span.http.status_code >= 500 } ``` Use the descendant operator (`>>`) to find traces where a frontend service calls a downstream service that errors: ```traceql { resource.service.name = "frontend" } >> { resource.service.name = "database" && status = error } ``` ## Query rate and count metrics Run these in **Metrics** mode in Explore. TraceQL metrics queries have a default 24-hour time-range limit: ```traceql { } | rate() ``` ```traceql { status = error } | rate() ``` ```traceql { resource.service.name = "api" } | count_over_time() ``` ```traceql { duration > 1s } | rate() ``` ## Filter by region or environment Narrow results to a specific cloud region or deployment environment: ```traceql { resource.cloud.region = "us-east-1" && status = error } ``` ```traceql { resource.deployment.environment = "production" && duration > 2s } ``` ## Use dashboard variables Replace hard-coded values with [dashboard variables](https://grafana.com/docs/grafana//dashboards/variables/) when using these queries in panels. For example, using a `$service` variable: ```traceql { resource.service.name = "$service" && status = error } ``` ## More resources - [TraceQL cookbook for Grafana Cloud Traces](https://grafana.com/docs/grafana-cloud/send-data/traces/traces-query-editor/traceql-cookbook/) - Extended recipe collection with additional examples - [Construct a TraceQL query](https://grafana.com/docs/tempo//traceql/construct-traceql-queries/) - Full TraceQL syntax, scopes, and operators - [Query tracing data](https://grafana.com/docs/grafana//datasources/tempo/query-editor/) - Query editor modes and options - [Service Graph and Service Graph view](https://grafana.com/docs/grafana//datasources/tempo/service-graph/) - Visualize service dependencies # Search traces using the query builder The **Search** query builder lets you create TraceQL queries using drop-down lists and text fields instead of writing syntax directly. Each selection you make generates a TraceQL query behind the scenes. You can view the generated query and copy it to the [TraceQL editor](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-editor/) at any time. Use Search when you're exploring data or learning TraceQL patterns. For complex queries involving structural operators, aggregations, or features that Search doesn't support, use the [TraceQL editor](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-editor/). To learn the full query syntax, refer to [Construct a TraceQL query](https://grafana.com/docs/tempo//traceql/construct-traceql-queries/). If queries return no results, check that your [Tempo data source is configured and connected](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/). ## Availability The Search query builder is available by default in Grafana and Grafana Cloud. No feature toggle or additional configuration is required. [//]: # 'Shared content for the Search - TraceQL query builder' {{< docs/shared source="grafana" lookup="datasources/tempo-search-traceql.md" leveloffset="+1" version="" >}} ## Try these common searches The following examples show what to select in the Search builder and the TraceQL query each selection generates. Adjust service names and values to match your environment. **Find error spans for a service:** 1. Set **Service Name** to your service (for example, `frontend`). 1. Set **Status** to `error`. Generated query: ```traceql { resource.service.name = "frontend" && status = error } ``` **Find slow API calls:** 1. Set **Span Name** to the operation (for example, `POST /api/orders`). 1. Set **Duration** to `>` `500ms`. Generated query: ```traceql { name = "POST /api/orders" && duration > 500ms } ``` **Filter by HTTP status code:** 1. Select **Add tag**, choose **span**, and select `http.response.status_code`. 1. Set the operator to `>=` and the value to `500`. Generated query: ```traceql { span.http.response.status_code >= 500 } ``` For more query examples, refer to [TraceQL query examples](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-query-examples/). ## Next steps - [TraceQL query examples](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-query-examples/): Copy-paste query examples for common use cases - [Write TraceQL queries with the editor](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-editor/): For complex queries the Search builder doesn't support - [Construct a TraceQL query](https://grafana.com/docs/tempo//traceql/construct-traceql-queries/): Full TraceQL syntax reference - [Service Graph and Service Graph view](https://grafana.com/docs/grafana//datasources/tempo/service-graph/): Visualize service dependencies - [Span filters](https://grafana.com/docs/grafana//datasources/tempo/span-filters/): Refine results in the trace detail view # Troubleshoot Tempo data source issues This document provides solutions to common issues you may encounter when configuring or using the Tempo data source in Grafana. ## Scope of this guide This guide covers issues related to connecting Grafana to Tempo and using the data source features. It applies to the following setups: - **Self-managed Grafana + self-managed Tempo:** you manage both Grafana (OSS or Enterprise) and Tempo. - **Grafana Cloud + Cloud Traces:** you use Grafana Cloud with the managed Tempo backend (Grafana Cloud Traces). Some configuration (streaming, metrics generator) is handled automatically. - **Grafana Cloud + self-managed Tempo via PDC:** you use Grafana Cloud but connect to your own Tempo instance through [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). Where troubleshooting steps differ between these setups, the guide calls it out. Sections labeled _Grafana Cloud only_ or _self-managed Tempo_ apply only to those environments. {{< admonition type="note" >}} The error messages in this guide are representative examples. Recent versions of the Tempo data source return clearer, more descriptive messages that add context, such as the query time range or the specific reason a request failed, so the exact wording you see might differ from these examples. Match on the general error rather than the exact string. {{< /admonition >}} ### Resources for troubleshooting Tempo For issues with Tempo itself (not the data source), refer to the Tempo product documentation: - [Troubleshoot Tempo](https://grafana.com/docs/tempo//troubleshooting/) - General Tempo troubleshooting, including ingestion failures and server-side errors. - [Unable to find traces](https://grafana.com/docs/tempo//troubleshooting/querying/unable-to-see-trace/) - Traces missing due to ingestion, sampling, or retention issues. - [Too many requests error](https://grafana.com/docs/tempo//troubleshooting/querying/too-many-requests-error/) - Query capacity limits and 429 errors. - [Query issues](https://grafana.com/docs/tempo//troubleshooting/querying/) - Server-side query failures, bad blocks, and performance tuning. ### Resources for troubleshooting traces in Grafana Cloud Additional resources for Grafana Cloud: - [Troubleshoot Grafana Cloud Traces](https://grafana.com/docs/grafana-cloud/send-data/traces/troubleshoot/), which covers quick checks, ingestion issues, TraceQL and search, service graph, exemplars, and rate limiting and retry. - [Investigate traces with Grafana Assistant](https://grafana.com/docs/grafana-cloud/send-data/traces/investigate-traces-with-assistant/) - Use Grafana Assistant to help troubleshoot any issues. - [Troubleshoot traces collection with Alloy](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/traces-with-alloy/#troubleshoot) - [Troubleshoot errors with metrics-generator in Cloud Traces](https://grafana.com/docs/grafana-cloud/send-data/traces/configure/metrics-generator/#troubleshoot-errors) ## Connection errors These errors occur when Grafana cannot establish or maintain a connection to the Tempo instance. ### Failed to connect to Tempo **Error message:** `Failed to connect to Tempo` or `dial tcp: connection refused` **Cause:** Grafana can't reach the Tempo instance. This typically means the Tempo process isn't running, the URL in the data source settings is wrong, or a network-level rule is blocking the connection. **Solution:** 1. Verify that the Tempo instance is running and accessible from the Grafana server. 1. Check that the URL is correct in the [data source configuration](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/). The default Tempo HTTP port is `3200`. Common URL mistakes include: - Adding trailing path segments, for example, `/api` or `/tempo`, to self-managed Tempo URLs. The correct format is `http://:3200` with no path. The `/tempo` suffix is only used for Grafana Cloud Traces URLs. - Using the gRPC port (`9095`) instead of the HTTP port (`3200`). The Tempo data source URL must be the HTTP endpoint. Grafana derives the gRPC connection for streaming from the same URL. - Refer to the [Connection](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/) section for URL format examples by environment. 1. Ensure there are no firewall rules blocking the connection between Grafana and Tempo. 1. For Grafana Cloud users connecting to a self-managed Tempo instance, ensure you have configured [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) if Tempo isn't publicly accessible. 1. Verify the protocol (HTTP or HTTPS) matches your Tempo configuration. ### Connection timeout **Error message:** `Connection timed out` or `context deadline exceeded` **Cause:** The connection to Tempo timed out before receiving a response. This can happen when network latency is high, Tempo is overloaded, or an intermediary device (load balancer, proxy) terminates the connection before Tempo responds. **Solution:** 1. Check the network latency between Grafana and Tempo. 1. Verify that Tempo isn't overloaded or experiencing performance issues. If you have [Tempo mixin dashboards](https://grafana.com/docs/tempo//operations/monitor/) installed, check them for resource saturation or elevated error rates. 1. Increase the **Timeout** setting in the [data source configuration](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/additional-settings/) under **Additional settings**. 1. Check if any network devices (load balancers, proxies) are timing out the connection. 1. For large trace queries, consider reducing the time range or adding more specific filters. ### TLS/SSL connection failures **Error message:** `TLS handshake failed` or `x509: certificate signed by unknown authority` **Cause:** There's a mismatch between the TLS settings in Grafana and what the Tempo instance supports or requires. For example, Tempo may present a self-signed certificate that the Grafana server doesn't trust, or the certificate's hostname doesn't match the URL. **Solution:** 1. Verify that Tempo has a [valid TLS certificate if HTTPS is enabled](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/#authentication). 1. Check that the certificate is trusted by the Grafana server. 1. If using a self-signed certificate, configure the **TLS/SSL Auth Details** in the [data source settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/#authentication). 1. Ensure the certificate's Common Name (CN) or Subject Alternative Name (SAN) matches the hostname used in the URL. For additional information on setting up TLS encryption with Tempo, refer to [Configure TLS communication](https://grafana.com/docs/tempo//configuration/network/tls/). ## Authentication errors These errors occur when there are issues with authentication credentials or permissions. ### Authentication failed **Error message:** `401 Unauthorized` or `403 Forbidden` **Cause:** The authentication credentials are invalid or the user doesn't have permission to access Tempo. This error appears when you select **Save & test** or when running a query. **Solution:** 1. Verify that the authentication credentials (username/password or API key) are correct in the [data source settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/#authentication). 1. Check that the user or service account has the required permissions. 1. Ensure the authentication method selected in the data source matches what Tempo expects. 1. For Grafana Cloud, verify that your Cloud Access Policy token has the `traces:read` scope. Refer to [Create a Cloud Access Policy token](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/add-access-policy/) for setup instructions. To find your stack URL and credentials, refer to [Locate your stack URL, user, and password](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/locate-url-user-password/). 1. For self-managed Tempo, ensure the credentials match those configured in the Tempo `server` block or reverse proxy. ### Save & test returns 404 **Error message:** `Tempo echo endpoint returned status 404` when selecting **Save & test** **Cause:** The Grafana health check reaches the server but the `api/echo` endpoint returns a 404. The most common cause is incorrect path segments in the URL (such as `/api` or `/tempo` on a self-managed instance) that shift the health check request to a nonexistent path. A 404 can also occur when a reverse proxy in front of Tempo rejects requests that include unexpected basic authentication headers. **Solution:** 1. Verify that the URL doesn't include trailing path segments. For self-managed Tempo, the URL should be `http://:3200` with no path. Refer to the [Connection](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/) section for the correct format. 1. If a reverse proxy sits between Grafana and Tempo, check whether the proxy is returning the 404. Tempo itself ignores unexpected auth headers, but a proxy may reject them. 1. If Tempo doesn't require authentication, select **No authentication** in the [data source settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/#authentication). 1. Check that the authentication credentials are entered in the correct fields. For Grafana Cloud, the **User** field is the instance ID (not your Grafana login), and the **Password** is a Cloud Access Policy token. ### Multi-tenant authentication issues **Error message:** `Unauthorized` when using multi-tenant Tempo **Cause:** The tenant ID (`X-Scope-OrgID` header) is missing or incorrect. This header must be set on both the write path (in your Alloy or OpenTelemetry Collector configuration) and the read path (in the Grafana data source settings) to route data to the correct tenant. **Solution:** 1. In the Tempo data source settings, verify that the `X-Scope-OrgID` header is configured under **Authentication > HTTP Headers**. Refer to [Multi-tenancy](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/) for data source configuration steps. 1. On the write path, verify that the same `X-Scope-OrgID` header is set in your Alloy or OpenTelemetry Collector configuration so traces are ingested under the correct tenant. 1. Check that the authenticated user has access to the specified tenant. 1. For cross-tenant queries, ensure all specified tenants are accessible. For more information, refer to [Enable multi-tenancy](https://grafana.com/docs/tempo//operations/multitenancy/) and [Tenant IDs](https://grafana.com/docs/tempo//configuration/tenant-ids/). ## Configuration errors These issues occur when configuring the Tempo data source itself, before you run queries or set up correlations. ### Can't edit a provisioned data source (read-only) **Symptoms:** - The data source settings form is read-only and the button reads **Test** instead of **Save & test**. - You get a read-only error when you try to configure trace to logs correlation, enable **Filter by trace ID**, or change service graph or node graph settings. - Changes you make in the UI silently revert after Grafana restarts. **Cause:** The data source is provisioned through a YAML configuration file, Helm chart, or Terraform. By default, provisioned data sources are read-only in the Grafana UI, so the UI can't save changes. If the provisioning file sets `editable: true`, you can save changes in the UI, but provisioning overwrites them on the next restart or provisioning reload. **Solutions:** 1. **Edit the provisioning file to change settings permanently.** This is the recommended fix for self-managed Grafana, where you have access to the provisioning file. Any setting available in the UI can be set in the YAML file, including trace to logs, trace to metrics, **Filter by trace ID**, and service graph configuration. Edit the file, then restart Grafana or wait for the provisioning system to reload. Refer to [Provision the Tempo data source](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/) for the full YAML reference. 1. **Clone the data source to get an editable copy.** On Grafana Cloud, where you might not have direct access to the provisioning file, clone the provisioned data source to configure correlation, **Filter by trace ID**, or service graph settings through the UI. The clone isn't provisioned, so you can edit and save it normally. Refer to [Clone a provisioned data source for Grafana Cloud](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/provision/#clone-a-provisioned-data-source-for-grafana-cloud) for the steps. ## Query errors These errors occur when there are issues with TraceQL queries or trace lookups. ### Trace not found **Error message:** `Trace not found` or `trace ID not found` **Cause:** The specified trace ID doesn't exist in Tempo or has aged out of the configured retention period. Common reasons include: - **Sampling:** Head or tail sampling in your instrumentation pipeline (Alloy, OpenTelemetry Collector) may have dropped the trace before it reached Tempo. This is the most common cause of missing traces. - **Retention:** Traces are only available for the duration set in the Tempo retention configuration. For Grafana Cloud users, default retention periods apply unless you contact Support to change them. - **Ingestion failure:** The trace may not have been successfully ingested due to rate limits, errors, or misconfigured pipelines. **Solution:** 1. Verify the trace ID is correct and complete. 1. Check your sampling configuration in Alloy or OpenTelemetry Collector. If head or tail sampling is enabled, the trace may have been intentionally dropped. Refer to [Sampling strategies](https://grafana.com/docs/tempo//set-up-for-tracing/instrument-send/set-up-collector/tail-sampling/) for guidance. 1. Check that the trace is within the configured retention period for Tempo. 1. If using time range restrictions, expand the time range in the [**TraceID query** settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/additional-settings/#traceid-query). 1. Enable **Use time range in query** in the [TraceID query settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/additional-settings/#traceid-query) and adjust the time shift values to search a broader range. 1. Verify the trace was successfully ingested by Tempo. You can run a broad query to check for recent traces: ```traceql { } | count() > 0 ``` If this returns results, traces are being ingested but your specific trace ID may have been dropped by sampling or aged out. For more query examples, refer to the [TraceQL cookbook](https://grafana.com/docs/grafana-cloud/send-data/traces/traces-query-editor/traceql-cookbook/). 1. For Grafana Cloud users, refer to [Troubleshoot TraceQL and search](https://grafana.com/docs/grafana-cloud/send-data/traces/troubleshoot/#traceql-and-search) for TraceQL queries that can help investigate missing traces. ### TraceQL syntax errors **Error message:** `parse error` or `unexpected token` **Cause:** The TraceQL query contains syntax errors. **Solution:** 1. Verify the query follows [TraceQL syntax](https://grafana.com/docs/tempo//traceql/construct-traceql-queries/). 1. Check that all braces, parentheses, and quotes are balanced. 1. Ensure attribute names are correctly formatted with the correct scope. Use dot notation for span or resource attributes, for example, `span.http.status_code` or `resource.service.name`. 1. Use the **Search** query builder to generate valid TraceQL queries if you're unfamiliar with the syntax. 1. Verify operators are used correctly. Supported operators include `=`, `!=`, `=~`, `!~`, `>`, `<`, `>=`, `<=`. ### Query returns no results **Cause:** The query is valid but doesn't match any traces. **Solution:** 1. Expand the time range to search a broader period. 1. Verify the attribute names and values exist in your traces. 1. Check that the attribute names match exactly, as they're case-sensitive. 1. Use the **Search** query builder to explore available attributes and values. 1. Start with a broader query and progressively add filters to narrow results. 1. Try [Grafana Traces Drilldown](https://grafana.com/docs/grafana//explore/simplified-exploration/traces/), a queryless app that lets you explore tracing data using RED metrics without writing TraceQL. 1. Note that Tempo search is non-deterministic. Identical queries can return different results because Tempo scans in parallel and returns the first matching traces. Refer to [Understand search behavior](https://grafana.com/docs/grafana//datasources/tempo/query-editor/) for details. 1. Verify traces are being ingested into Tempo by querying for a known trace ID. ### Traces not appearing after successful connection **Cause:** **Save & test** succeeds, but running queries in Explore returns no traces. The data source connection is valid, but no trace data is reaching Tempo, or the queries don't match any ingested data. **Solution:** 1. Verify that your applications are instrumented and that your collector (Alloy, OpenTelemetry Collector) is forwarding spans to Tempo. Check collector logs for send errors, rejected spans, or drops. 1. For Grafana Cloud users who send traces with Alloy, refer to [Troubleshoot sending traces with Alloy](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/traces-with-alloy/#troubleshoot) for ingestion-side issues. 1. Confirm that the query time range covers the period when traces were generated. Trace data only appears for the window in which spans were ingested. 1. For multi-tenant setups, verify that the `X-Scope-OrgID` header is the same on both the write path (collector) and the read path (Grafana data source). A mismatch causes traces to be ingested under one tenant and queried under another. 1. Try a broad query, for example, `{}`, with a short, recent time range to confirm that any traces exist in Tempo. If this returns results, your original query filters are too narrow. ### Query timeout **Error message:** `context deadline exceeded` or `query timeout` **Cause:** The query took too long to execute. Trace searches scan raw trace data across the selected time range, so broad queries over long periods can exceed timeout limits. **Solution:** 1. Reduce the time range to limit the amount of data scanned. 1. Add more specific filters to narrow the search scope. 1. Use indexed attributes in your queries when possible for faster lookups. 1. For self-managed Tempo, increase the query timeout in the Tempo server configuration if appropriate. 1. If you need aggregate data (rates, counts, durations) rather than individual traces, consider using [TraceQL metrics queries](https://grafana.com/docs/tempo//traceql/metrics-queries/) (for example, `{ } | rate()`) or pre-computed metrics from the [metrics generator](https://grafana.com/docs/tempo//metrics-generator/), which are optimized for aggregate workloads. ### TraceQL metrics query fails **Error message:** `localblocks processor not found` or `metrics-generator not configured` **Cause:** TraceQL metrics queries (such as `{ } | rate()` or `{ } | count_over_time()`) require the `local-blocks` processor to be active in the Tempo metrics-generator configuration. This processor is separate from the `service-graphs` and `span-metrics` processors and must be enabled independently. **Solution:** 1. For self-managed Tempo, activate the `local-blocks` processor in the overrides configuration. The configuration path depends on your Tempo version and deployment method: - Standard: `overrides.metrics_generator_processors: ["local-blocks"]` - Newer versions and Helm: `overrides.defaults.metrics_generator.processors: [local-blocks]` Refer to [Configure TraceQL metrics](https://grafana.com/docs/tempo//metrics-from-traces/metrics-queries/configure-traceql-metrics/) for setup steps and required configuration. 1. For Grafana Cloud, the `local-blocks` processor is enabled by default. If you still see this error, contact [Grafana Support](https://grafana.com/contact/). 1. Verify that the metrics generator is running and healthy by checking the Tempo metrics-generator logs for errors. ### TraceQL metrics query exceeds the maximum time range **Error message:** One of the following, depending on whether streaming is enabled for metrics queries: - `The selected time range exceeds the maximum allowed duration. Please select a shorter time range.` - `metrics query time range exceeds the maximum allowed duration of ` **Symptoms:** - A TraceQL metrics query, such as `{ } | rate()`, fails when the selected time range is too wide. - Shorter time ranges run successfully, but range queries over long periods return the error. **Cause:** Tempo limits the time range for TraceQL metrics queries. The default maximum is 24 hours. Tempo enforces this limit, not the data source, so you can't change it from the Grafana UI. **Solutions:** 1. Reduce the query time range to within the configured maximum, which is 24 hours by default. 1. For self-managed Tempo, raise `query_frontend.metrics.max_duration` (default `24h`) in your Tempo configuration and restart Tempo, or set the per-tenant `max_metrics_duration` override. Refer to [Configure Tempo](https://grafana.com/docs/tempo//configuration/) in the Tempo documentation. 1. For Grafana Cloud Traces, contact [Grafana Support](https://grafana.com/contact/) to request a higher limit. You can't change this limit yourself. ## Streaming issues Streaming displays TraceQL query results as they become available. Without streaming, you don't see results until the query completes. For Grafana Cloud users, search streaming is enabled by default and metrics streaming is off by default. For self-managed Tempo, search streaming requires Tempo v2.2 or later, and metrics streaming requires Tempo v2.7 or later. Both require `stream_over_http_enabled: true` in the Tempo configuration. For more information, refer to [Streaming](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/#streaming). ### Streaming not working **Cause:** Results don't appear incrementally and only display after the full query completes. **Solution:** 1. For self-managed Tempo, verify that you're using the required Tempo version (v2.2 or later for search streaming, v2.7 or later for metrics streaming) and that streaming is enabled in your configuration (`stream_over_http_enabled: true`). Refer to [Tempo gRPC API](https://grafana.com/docs/tempo//api_docs/#tempo-grpc-api) in the Tempo documentation. For Grafana Cloud users, search streaming is enabled by default and metrics streaming is off by default. 1. Verify that **Streaming** is enabled in your [Tempo data source settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/#streaming). The **Search queries** and **Metrics queries** toggles are independent. Check that the toggle for the query type you're using is enabled. 1. If your Tempo instance is behind a load balancer or proxy that doesn't support gRPC or HTTP2, streaming may not work. Disable streaming and use standard HTTP queries instead. 1. If you're using [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) to reach Tempo, verify that the PDC tunnel is running and the Tempo instance is reachable from the PDC agent. PDC connectivity issues can present as streaming timeouts. ### gRPC transport security error **Error message:** `rpc error: code = Unavailable desc = credentials require transport level security` **Cause:** Grafana tries to use gRPC over an insecure (non-TLS) connection. Streaming uses gRPC, which may require TLS depending on your Tempo configuration. **Solution:** 1. Configure TLS between Grafana and Tempo. Refer to [Configure TLS communication](https://grafana.com/docs/tempo//configuration/network/tls/). 1. If TLS isn't an option, disable streaming in the [Tempo data source settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/#streaming) and use standard HTTP queries instead. ### Partial results or streaming errors **Error message:** `stream error` or incomplete results during streaming **Cause:** The streaming connection was interrupted before all results were delivered. **Solution:** 1. Check network stability between the browser and Grafana. 1. Verify that proxies or load balancers aren't terminating long-lived connections. 1. Disable streaming and use standard queries if issues persist. 1. Check Tempo server logs for any errors related to streaming. ## Service graph issues The Service Graph visualizes service dependencies and highlights request rate, error rate, and duration (RED metrics) across connections. It requires a linked Prometheus data source with service graph metrics generated by the Tempo metrics generator or Grafana Alloy. For more information, refer to [Service Graph and Service Graph view](https://grafana.com/docs/grafana//datasources/tempo/service-graph/). ### Service graph not displaying **Cause:** The Service Graph view shows no data or an empty graph. **Solution:** 1. Verify that a Prometheus data source is linked in the Tempo data source **Service Graph** settings under [Additional settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/additional-settings/). 1. For self-managed Tempo, check that the metrics generator is configured to produce service graph metrics. Refer to [Service graph metrics](https://grafana.com/docs/tempo//metrics-generator/service-graph-view/) in the Tempo documentation. 1. Ensure the following service graph metrics exist in Prometheus: - `traces_service_graph_request_total` - `traces_service_graph_request_failed_total` - `traces_service_graph_request_server_seconds_sum` 1. Verify that Grafana Alloy or the Tempo metrics generator is configured to produce these metrics. These are distinct from span metrics (`traces_spanmetrics_*`), which power the Service Graph view table. 1. To verify the metrics exist, run the following query in Explore with your Prometheus data source selected: ```promql {__name__=~"traces_service_graph_request.*"} ``` 1. Check the Prometheus data source connection is working. 1. For Grafana Cloud Traces users, refer to [Troubleshoot service graph and RED metrics](https://grafana.com/docs/grafana-cloud/send-data/traces/troubleshoot/#troubleshoot-service-graph-and-red-metrics). ### Service Graph view table is empty **Cause:** The Service Graph node graph renders correctly, but the RED metrics table alongside it shows no data. **Solution:** 1. The Service Graph view table uses a different set of metrics from the node graph. The Grafana frontend queries `traces_spanmetrics_calls_total` and `traces_spanmetrics_latency_bucket` (or `traces_spanmetrics_latency` for native histograms). Confirm that these metrics exist in your linked Prometheus data source. 1. Verify that span metrics generation is enabled in your Tempo or Alloy configuration. Refer to [Enable service graphs](https://grafana.com/docs/tempo//metrics-from-traces/service_graphs/enable-service-graphs/) in the Tempo documentation. ### High cardinality warnings **Cause:** Service graph or span metrics queries produce high-cardinality warnings or slow queries. **Solution:** 1. Review the label cardinality of your service graph and span metrics. High-cardinality labels (such as unique Pod names or request IDs) create many time series and degrade query performance. 1. Refer to [Cardinality](https://grafana.com/docs/tempo//metrics-from-traces/metrics-generator/cardinality/) in the Tempo documentation for guidance on managing label cardinality. ### Service graph shows incomplete data **Cause:** Some services or connections are missing from the graph. **Solution:** 1. Verify that all services are instrumented and sending spans. 1. Check that span names and service names are consistent. 1. Ensure the time range includes data from all expected services. 1. Verify the Prometheus data source has metrics for all services. ## Trace to logs/metrics/profiles issues These issues relate to the correlation features that link traces to other telemetry signals. For configuration details, refer to [Trace to logs](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/), [Trace to metrics](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/), and [Trace to profiles](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/). ### Trace to logs link not appearing **Cause:** Links to logs don't appear when viewing a trace. **Solution:** 1. Verify that a Loki or other log data source is configured in the [Trace to logs](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/) settings. 1. Check that the configured tags exist in your spans. 1. Ensure at least one mapped tag has a non-empty value in the span. 1. If using a custom query, verify the query syntax is valid for the target data source. 1. Check that **Filter by trace ID** or **Filter by span ID** aren't enabled when using a custom query. 1. If using **Filter by trace ID** with Loki derived fields, verify that the derived field regular expression matches the trace ID field name used in your logs. Different instrumentation libraries use different field names (`traceID`, `trace_id`, or `traceId`). Refer to [Trace to logs](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/) for common regular expression patterns by instrumentation format. ### Loki log lines don't show a trace link **Cause:** The reverse direction, navigating from a Loki log line to a trace, isn't working. No trace link appears on log lines. **Solution:** 1. Verify that a derived field is configured in the Loki data source under **Additional settings > Derived fields**. The derived field extracts the trace ID from log lines and creates a link to Tempo. 1. Check that the derived field regular expression matches the trace ID format in your logs. Use **Show example log message** in the derived field settings to test the regular expression against a sample log line. 1. Confirm the **Internal link** toggle is enabled and points to the correct Tempo data source. 1. If different services use different trace ID field names, you need a separate derived field entry for each format. Refer to [Trace to logs](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/) for regular expression patterns by format. ### Trace to metrics link not appearing **Cause:** Links to metrics don't appear when viewing a trace. **Solution:** 1. Verify that a Prometheus or other metrics data source is configured in the [Trace to metrics](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-metrics/) settings. 1. Check that the configured tags exist in your spans. 1. If using custom queries, verify they're valid PromQL. 1. Ensure the `$__tags` variable is correctly placed in custom queries. ### Trace to profiles link not appearing **Cause:** Links to profiling data don't appear when viewing a trace. **Solution:** 1. Verify that a Grafana Pyroscope data source is configured in the [Trace to profiles](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-profiles/) settings. 1. Check that the configured tags exist in your spans. 1. Verify that the configured profile type matches what Pyroscope has data for (for example, `process_cpu:cpu:nanoseconds:cpu:nanoseconds`). A mismatch between the configured profile type and the available data causes the link to not appear. 1. Tighten the **Span start time shift** and **Span end time shift** values. Profiles are sampled at intervals, so a narrow time window around the span is more likely to match profile data than a wide one. 1. Verify that the profiling data source is connected and returning data. ### Links appear but return no data **Cause:** The generated query doesn't match any data in the target data source. **Solution:** 1. Adjust the **Span start time shift** and **Span end time shift** to widen the time range. 1. Verify that log or metric labels match the span attributes. 1. Check that the tag mappings correctly translate attribute names between data sources. Span attributes use dots (for example, `service.name`) but Loki labels use underscores (for example, `service_name`). Ensure tag mappings account for this difference. Refer to [Trace to logs](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/) for tag mapping configuration. 1. If a tag like `pod` is stored as Loki [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) rather than an indexed label, the auto-generated stream selector `{pod="..."}` returns no results. Enable **Use custom query** and move the tag to a pipeline filter. Refer to [Trace to logs](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/configure-trace-to-logs/) for a custom query example. 1. Run the generated query directly in the target data source's Explore view to confirm it returns data outside of the trace context. 1. Use the Query Inspector to view the generated query and verify it's correct. ### Trace links break after renaming a provisioned data source **Symptoms:** - Log-to-trace or trace-to-log links stop working after you rename a provisioned Tempo or Loki data source. - Correlations or derived fields that previously resolved now point to a data source that no longer exists. **Cause:** Correlations, derived fields, and trace to logs, metrics, and profiles settings reference the target data source by its `uid`, not its name. When a provisioned data source doesn't set an explicit `uid`, Grafana derives one from the data source name. Renaming the data source in the provisioning file changes the derived `uid`, so every link that referenced the old `uid` breaks. **Solutions:** 1. Set an explicit, stable `uid` for each provisioned data source in the YAML file so the `uid` doesn't change when you rename the data source: ```yaml datasources: - name: Tempo type: tempo uid: tempo-prod ``` 1. After you add or correct the `uid`, update the correlations, derived fields, and trace to logs, metrics, and profiles settings that reference it to use the current `uid`. 1. To confirm a data source's `uid`, open its settings page. The `uid` appears in the page URL. ## Performance issues These issues relate to slow queries or high resource usage. ### Slow trace queries **Cause:** Queries take a long time to return results. **Solution:** 1. Reduce the dashboard or Explore time range. 1. Add more specific filters to narrow the search scope. 1. Use indexed attributes in TraceQL queries when possible. 1. Enable streaming to see results as they become available. 1. Consider using the **Search** query builder which can help optimize queries. ### High memory usage during queries **Cause:** Large traces or result sets consume excessive memory. **Solution:** 1. Reduce the **Limit** setting in query options to return fewer traces. 1. Reduce the **Span Limit** setting to return fewer spans per trace. 1. Add filters to reduce the result set size. 1. Avoid querying very large time ranges. ## Other common issues The following issues don't produce specific error messages but are commonly encountered. ### Node graph not displaying **Cause:** The node graph visualization doesn't appear above the trace view. **Solution:** 1. Verify that **Node graph** is enabled in the Tempo data source settings under [Additional settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/additional-settings/). 1. Check that the trace contains multiple spans with parent-child relationships. Spans must include `traceID` and `spanID`, and non-root spans must include `parentSpanID` to establish hierarchy. If these identifiers are missing, the trace data is incomplete at the instrumentation level and the graph can't be rendered. ### Autocomplete not working **Cause:** The TraceQL editor doesn't show suggestions for attributes or values. Autocomplete requires Tempo to enumerate tags, which requires a supported storage backend (`vParquet` or later). **Solution:** 1. For self-managed Tempo, verify that the storage backend supports tag search. Tempo requires `vParquet` format or later for tag-based search and autocomplete. Refer to [Parquet configuration](https://grafana.com/docs/tempo//configuration/parquet/) for details. Grafana Cloud Traces supports autocomplete by default. 1. Check the **Tags time range** setting in the [data source settings](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/additional-settings/) to ensure it covers a period with recent trace data. 1. Increase the **Tag limit** setting if you have many unique attributes. 1. Verify the Tempo data source connection is working by selecting **Save & test** in the data source settings. ### TraceQL alerting not available **Cause:** You can select the Tempo data source in the Grafana Alerting rule builder, but standard TraceQL search queries return trace data (spans and trace IDs), not numeric time series. Alert rules require numeric data to evaluate thresholds, so trace search queries don't produce usable alert conditions. **Solution:** TraceQL metrics queries (such as `{ } | rate()` or `{ status = error } | count_over_time()`) return numeric time series and can drive alert thresholds directly. However, TraceQL alerting requires the experimental `tempoAlerting` feature flag, which is disabled by default. To enable it, set the [`tempoAlerting` feature flag](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#feature_toggles) in your Grafana configuration. For Grafana Cloud, contact Support to enable it. For step-by-step setup, refer to [Trace-based alerts](https://grafana.com/docs/grafana//alerting/examples/trace-based-alerts/). {{< admonition type="caution" >}} The `tempoAlerting` feature flag is experimental. For production alerting, consider the Prometheus-based approach described below. {{< /admonition >}} Alternatively, use the Tempo metrics generator to produce Prometheus metrics from trace data, then create alert rules against the Prometheus data source that stores those metrics. 1. Configure the Tempo [metrics generator](https://grafana.com/docs/tempo//metrics-generator/) (using the `service-graphs`, `span-metrics`, or `local-blocks` processor) to produce metrics from traces and remote-write them to a Prometheus-compatible backend. 1. Create alert rules against the Prometheus data source that receives those metrics. For example, alert on high error rates using `traces_spanmetrics_calls_total` with a `status_code="STATUS_CODE_ERROR"` filter. 1. Refer to the feature support table in the [Tempo data source overview](https://grafana.com/docs/grafana//datasources/tempo/) for details on alerting support. ### Switching between multiple Tempo instances **Cause:** You have multiple Tempo data sources configured (for example, one per environment or per tenant), but there's no built-in UI dropdown in Explore or dashboards to switch between them dynamically. **Workaround:** 1. In dashboards, create a [data source template variable](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/#add-a-data-source-variable) filtered to `tempo` type. This adds a dropdown that lets viewers switch between Tempo instances without editing the dashboard. 1. In Explore, use the data source picker in the top bar to switch between configured Tempo data sources. 1. For multi-tenant setups on a single Tempo instance, consider using the `X-Scope-OrgID` header instead of separate data sources. Refer to [Multi-tenancy](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/) for configuration details. ## Private data source connect issues (_Grafana Cloud only_) These issues relate to connecting to a private Tempo instance using Private data source connect (PDC). PDC is only available in Grafana Cloud. {{< admonition type="note" >}} Private data source connect allows Grafana Cloud to securely connect to data sources in your private network without exposing them to the public internet. For setup instructions, refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). {{< /admonition >}} ### PDC agent not connected **Error message:** `No healthy PDC agents available` or `Private data source connect agent not connected` **Cause:** The PDC agent installed in your private network isn't running or can't connect to Grafana Cloud. **Solution:** 1. Verify the PDC agent is running in your private network. 1. Check the PDC agent logs for connection errors. 1. Ensure the PDC agent has outbound network access to Grafana Cloud endpoints. 1. Verify the PDC agent is using the correct token for your Grafana Cloud instance. 1. Check that firewall rules allow outbound HTTPS (port 443) connections from the PDC agent. 1. Restart the PDC agent if it appears stuck or unresponsive. For more information, refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). ### Connection through PDC failing **Error message:** `Failed to connect through private data source` or `Connection refused via PDC` **Cause:** The PDC agent is connected to Grafana Cloud but cannot reach the Tempo instance in your private network. **Solution:** 1. Verify the Tempo URL is correct and accessible from the machine running the PDC agent. 1. Check that the PDC agent can resolve the Tempo hostname (DNS resolution). 1. Ensure there are no firewall rules blocking connections from the PDC agent to Tempo. 1. Verify the Tempo port (default `3200`) is open and accessible. 1. Test connectivity from the PDC agent host to Tempo using `curl` or `telnet`. ### PDC timeout errors **Error message:** `Request timed out via private data source connect` **Cause:** Queries through the PDC tunnel are timing out. **Solution:** 1. Check network latency between the PDC agent and Tempo. 1. Verify the PDC agent host has sufficient resources (CPU, memory, network bandwidth). 1. Ensure no network devices between the PDC agent and Tempo are dropping long-running connections. 1. For large queries, consider reducing the time range or adding more specific filters. 1. Check if multiple data sources are sharing the same PDC agent, which may cause resource contention. ### PDC authentication mismatch **Error message:** `Unauthorized` or `403 Forbidden` when using PDC **Cause:** The authentication configured in the data source doesn't work with the PDC connection. **Solution:** 1. Verify that authentication credentials are configured correctly in the Grafana Cloud data source settings. 1. Check that the Tempo instance accepts the authentication method configured. 1. Ensure any required headers (such as `X-Scope-OrgID` for multi-tenant Tempo) are properly forwarded through PDC. 1. Test the connection directly from the PDC agent host to verify credentials work outside of PDC. ## Get additional help If you continue to experience issues after following this troubleshooting guide: 1. For Grafana Cloud users, try [Grafana Assistant](https://grafana.com/docs/grafana-cloud/send-data/traces/investigate-traces-with-assistant/) to investigate trace issues interactively (if enabled on your account). 1. For Grafana Cloud users who send traces with Alloy, refer to the [Alloy troubleshooting steps](https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/traces-with-alloy/#troubleshoot) for ingestion-side issues. 1. Ask questions in the [Grafana community forums](https://community.grafana.com/) or the [Grafana Community Slack](https://slack.grafana.com/). 1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs. 1. Review the [Tempo GitHub issues](https://github.com/grafana/tempo/issues) for Tempo-specific bugs. 1. Enable [debug logging](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/#log) in Grafana to capture detailed error information. 1. Check [Tempo server logs](https://grafana.com/docs/tempo//operations/monitor/) for additional error details. 1. Contact [Grafana Support](https://grafana.com/contact/) if you're an Enterprise, Cloud Pro, or Cloud Contracted user. When reporting issues, include: - Grafana version - Tempo version - Error messages (redact sensitive information) - Steps to reproduce - TraceQL query examples (redact sensitive data) - Relevant configuration settings # TestData alerting The TestData data source supports [Grafana Alerting](https://grafana.com/docs/grafana//alerting/). You can use it to prototype alert rules, test threshold conditions, and verify notification pipelines without connecting to an external data source. ## Before you begin - [Configure the TestData data source](https://grafana.com/docs/grafana//datasources/testdata/configure/). - Familiarize yourself with [alert rules](https://grafana.com/docs/grafana//alerting/fundamentals/alert-rules/) and how to [create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Choose a scenario for alerting TestData scenarios that return time-series data work with alert rule conditions. Choose a scenario based on the behavior you want to test. | Scenario | Alerting use case | | ---------------------------- | ------------------------------------------------------------------------------------------------ | | **Predictable Pulse** | Deterministic on/off pattern. Produces repeatable alert firing and resolving on a fixed cycle. | | **Random Walk** | Non-deterministic time series. Useful for quick prototyping and load testing alert evaluation. | | **CSV Metric Values** | Controlled, fixed values. Test exact threshold boundaries. | | **Predictable CSV Wave** | Custom repeating waveforms. Test complex threshold patterns with precise control. | | **Random Walk (with error)** | Returns both data and an error. Test how alerts handle partial failures. | | **Error with source** | Returns a plugin or downstream error. Test alert evaluation error handling. | | **Flaky Query** | Returns errors intermittently. Test how alerts handle a data source that fails part of the time. | | **No Data Points** | Returns empty results. Test no-data alert conditions. | | **Slow Query** | Introduces a configurable delay. Test alert evaluation timeouts. | Scenarios that produce logs, traces, or streaming data aren't suitable for threshold-based alert conditions. ## Create an alert rule To create an alert rule using TestData: 1. Navigate to **Alerting** > **Alert rules**. 1. Click **New alert rule**. 1. Enter a name for the rule. 1. Select the **TestData** data source. 1. Choose a scenario from the **Scenario** drop-down (for example, **Predictable Pulse**). 1. Configure the scenario options. 1. Add a **Reduce** expression to aggregate the series (for example, **Last** value). 1. Add a **Threshold** expression to define the condition (for example, **Is above** `0.5`). 1. Set the evaluation interval and pending period. 1. Configure notifications and labels. 1. Click **Save rule**. For detailed instructions, refer to [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/). ## Example: Predictable Pulse alert The Predictable Pulse scenario produces a repeating on/off wave that's ideal for testing alert rules because the firing pattern is deterministic and reproducible. To create a Predictable Pulse alert: 1. Select the **Predictable Pulse** scenario. 1. Configure the pulse options: | Field | Value | Effect | | ------------- | ----- | ------------------------------------------- | | **Step** | `60` | One data point every 60 seconds. | | **On Count** | `3` | Three consecutive points at the "on" value. | | **Off Count** | `6` | Six consecutive points at the "off" value. | | **On Value** | `1` | Value during the "on" phase. | | **Off Value** | `0` | Value during the "off" phase. | 1. Add a **Reduce** expression with function **Last**. 1. Add a **Threshold** expression: **Is above** `0.5`. With this configuration, the alert fires for 3 minutes (3 x 60s on-points) and then resolves for 6 minutes (6 x 60s off-points), repeating indefinitely. Adjust the step, on count, and off count to control the timing. ## Example: CSV Metric Values threshold alert The CSV Metric Values scenario lets you define an exact sequence of data points, making it useful for testing precise threshold boundaries. To create a threshold alert with fixed values: 1. Select the **CSV Metric Values** scenario. 1. Enter a comma-separated list of values in the **String Input** field. For example: ``` 10,45,70,95,50,20 ``` 1. Add a **Reduce** expression with function **Last**. 1. Add a **Threshold** expression: **Is above** `80`. With this input, the series cycles through the values `10, 45, 70, 95, 50, 20`. The alert fires when the reduced value exceeds `80` (the `95` data point) and resolves when it drops back below the threshold. This approach is helpful for validating that a threshold condition triggers at the exact boundary you expect. ## Test error and no-data conditions TestData includes scenarios designed for testing how alerts respond to failures. Use these to verify that your alert rules handle edge cases correctly before connecting to production data sources. ### No data Select the **No Data Points** scenario. The backend returns an empty result, which triggers the alert rule's **no data** state. Use this to verify that your rule is configured to alert, keep the last state, or resolve when no data is received. Configure this behavior in the alert rule's **No data and error handling** section. ### Errors Select **Error with source** and choose the error type: - **Plugin**: Simulates a failure in the plugin itself. The alert enters an error state. - **Downstream**: Simulates a failure in the data source or network. The alert enters an error state with a downstream classification. You can also use **Random Walk (with error)**, which returns valid data alongside an error, to test how the alerting engine handles partial failures. ### Intermittent errors Select **Flaky Query** to simulate a data source that fails part of the time. Set the **Error rate** to control how often requests fail, and use **Status code** and **Error source** to shape the error. Use this to verify how an alert rule behaves when evaluations succeed and fail across consecutive intervals, and how the **No data and error handling** settings respond to intermittent failures. ### Timeouts Select **Slow Query** and set the **String Input** field to a duration longer than your evaluation interval (for example, `30s` for a rule evaluated every `10s`). The default delay is `5s`. Use this to verify timeout behavior and confirm that alerts transition to the expected state when queries take too long. ## Limitations When using TestData with Grafana Alerting, be aware of the following limitations. ### Template variables aren't supported in alert queries Grafana evaluates alert rules on the backend without dashboard context. Template variables like `$varname` aren't resolved during alert evaluation. Use fixed values in the scenario options instead. ### Only time-series scenarios work with alert conditions Alert conditions require numeric time-series data. Scenarios that produce logs (Logs), traces (Trace), graphs (Node Graph, Flame Graph), annotations (Annotations), or tables (Table Static) can't be used as alert conditions. ### Streaming scenarios aren't evaluated by the alerting engine The alerting engine evaluates queries at fixed intervals and doesn't consume streaming data. Avoid using Streaming Client, Grafana Live, or Simulation with streaming enabled for alert rules. # TestData annotations The TestData data source can generate simulated [annotation](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/) events. Use it to test how dashboards and panels render annotation markers and overlays without connecting to an external source. The generated events are created in the browser and aren't persisted. ## Before you begin - [Configure the TestData data source](https://grafana.com/docs/grafana//datasources/testdata/configure/). - Familiarize yourself with [annotations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/) in Grafana. ## Add an annotation query To add a dashboard annotation query that uses TestData: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click **Settings**. 1. Select the **Annotations** tab. 1. Click **Add annotation query**. 1. Enter a name for the annotation. 1. From the **Data source** drop-down, select **TestData**. The drop-down only lists data sources that support annotations. 1. Set the **Count** field to the number of events you want to generate. Default: `10`. 1. Click **Save dashboard**. TestData distributes the generated events evenly across the current dashboard time range. Each event includes sample text with a link and the tags `text` and `server`. ## Use the Annotations scenario in a panel You can also generate annotation data as a panel query using the **Annotations** scenario. This scenario returns data on the annotations data topic, so Grafana renders the events as markers overlaid on the panel rather than plotting them as a series. Because the Annotations scenario doesn't return a visible series on its own, add a second query that returns data to give the markers something to overlay: 1. Add or edit a panel and select the **TestData** data source. 1. On the first query, choose a scenario that returns a series, such as **Random Walk**, from the **Scenario** drop-down. 1. Add a second query, then choose the **Annotations** scenario from its **Scenario** drop-down. 1. Set the **Count** field on the Annotations query to the number of events you want to generate. Default: `10`. 1. Click **Run queries**. Grafana plots the series from the first query and overlays the annotation markers on top. If you run the Annotations scenario by itself, the query returns annotation events but the panel doesn't display a visible series. For a reference of all TestData scenarios, refer to the [TestData query editor](https://grafana.com/docs/grafana//datasources/testdata/query-editor/). ## Example: Overlay annotations on a time series panel This example creates a time series panel with simulated data, then adds a TestData annotation query so you can verify how the panel renders annotation markers. 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add** icon and select **Visualization**. 1. Select the **TestData** data source. 1. Choose the **Random Walk** scenario from the **Scenario** drop-down. 1. Click **Run queries** to render the time series. 1. Click **Back to dashboard**, then click **Settings**. 1. Select the **Annotations** tab. 1. Click **Add annotation query**. 1. Enter a name for the annotation, for example `Test events`. 1. From the **Data source** drop-down, select **TestData**. 1. Set the **Count** field to `5`. 1. Click **Save dashboard**. Grafana draws five vertical annotation markers evenly across the time range on the time series panel. Hover over a marker to see the sample text and tags. Adjust the **Count** field to change how many markers appear. {{< admonition type="tip" >}} To test how a panel handles crowded overlays, increase the **Count** field to a high value, such as `100`. This produces many closely spaced markers, which is useful for checking marker rendering and tooltip behavior at high density. {{< /admonition >}} ## Limitations - **Events are synthetic.** TestData generates fixed sample text and tags. You can't customize the event content. - **Events aren't persisted.** The data source builds events in the browser each time the query runs. TestData doesn't store annotations or query an external source. - **Not evaluated by the alerting engine.** The Annotations scenario runs in the browser, so it can't be used with [Grafana Alerting](https://grafana.com/docs/grafana//datasources/testdata/alerting/) or in any context that requires server-side evaluation. # Configure the TestData data source This document explains how to configure the TestData data source. TestData requires no external connection or authentication, so configuration is minimal. For general information on managing data sources, refer to [Data source management](https://grafana.com/docs/grafana//administration/data-source-management/). ## Before you begin You must have the Organization administrator role to add and configure data sources. ## Add the data source To add the TestData data source: 1. Click **Connections** in the left-side menu. 1. Click **Add new connection**. 1. Type `TestData` in the search bar. 1. Select **TestData**. 1. Click **Add new data source**. Grafana takes you to the **Settings** tab. ## Configuration options TestData doesn't require any settings beyond the standard options common to all data sources. | Setting | Description | | ----------- | ------------------------------------------------------------------------ | | **Name** | Sets the name you use to refer to the data source in panels and queries. | | **Default** | Defines whether this data source is pre-selected for new panels. | ## Verify the connection Click **Save & test** to verify the data source. A successful test displays the message: **Data source is working** If the test fails, refer to [Troubleshoot TestData](https://grafana.com/docs/grafana//datasources/testdata/troubleshooting/) for common issues and solutions. ## Provision the data source You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, refer to [Provisioning data sources](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). TestData doesn't require any `jsonData` or `secureJsonData` fields. **YAML example** ```yaml apiVersion: 1 datasources: - name: TestData type: grafana-testdata-datasource access: proxy ``` ## Provision with Terraform To provision the data source with Terraform, use the [`grafana_data_source` resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source): ```hcl resource "grafana_data_source" "testdata" { name = "TestData" type = "grafana-testdata-datasource" } ``` For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). ## Next steps After configuring your TestData data source, you can: - [Write queries](https://grafana.com/docs/grafana//datasources/testdata/query-editor/) using the 33 available scenarios to generate simulated data. - [Use template variables](https://grafana.com/docs/grafana//datasources/testdata/template-variables/) to create dynamic, reusable dashboards. - [Set up alerting](https://grafana.com/docs/grafana//datasources/testdata/alerting/) to prototype and test alert rules with simulated data. - [Troubleshoot issues](https://grafana.com/docs/grafana//datasources/testdata/troubleshooting/) if you encounter problems with your data source. # TestData query editor Instead of a traditional query language, the TestData data source uses **scenarios** to generate simulated data. Each scenario produces a different type of data suited for testing specific visualizations, behaviors, or edge cases. TestData includes 33 scenarios covering time series, logs, traces, graphs, streaming, and error simulation. Use scenarios to: - **Prototype dashboards** without connecting to a real data source. - **Reproduce bugs** with controlled, deterministic data that other developers can replicate. - **Test panel behavior** with edge cases like empty results, timestamps outside the visible range, or mixed data and errors. - **Validate alerting pipelines** using predictable patterns that fire and resolve on a known schedule. - **Simulate streaming** to verify how panels handle real-time data updates. To build a query, select a scenario from the **Scenario** drop-down. The query editor updates to show fields specific to that scenario. Click **Run queries** or use the keyboard shortcut to execute. {{< admonition type="note" >}} Some scenarios run entirely in the browser: Streaming Client, Grafana Live, Grafana API, Steps, Annotations, Node Graph, Flame Graph, Trace, and Raw Frames. These scenarios don't send queries to the backend, which means they can't be used with [Grafana Alerting](https://grafana.com/docs/grafana//datasources/testdata/alerting/) or in any context that requires server-side evaluation. {{< /admonition >}} ## Scenario reference The scenarios are organized into the following categories. Use the table below to find the right scenario for your use case. | Scenario | Category | Purpose | | ------------------------------------------------------------------- | --------------------- | -------------------------------------------------- | | [Random Walk](#random-walk) | Data generation | Random walk time series (default scenario). | | [Random Walk Table](#random-walk-table) | Data generation | Random walk in table format with `Enum` state. | | [Random Walk (with error)](#random-walk-with-error) | Data generation | Random walk that also returns an error. | | [Predictable Pulse](#predictable-pulse) | Data generation | Repeating on/off wave based on absolute time. | | [Predictable CSV Wave](#predictable-csv-wave) | Data generation | Custom repeating waveforms from CSV values. | | [Simulation](#simulation) | Data generation | Continuous simulation engine (flight, sine, tank). | | [USA generated data](#usa-generated-data) | Data generation | Multi-dimensional data with US state dimensions. | | [CSV Content](#csv-content) | Manual input | Paste or type CSV data directly. | | [Steps](#steps) | Manual input | Step-function data from CSV input. | | [CSV File](#csv-file) | Manual input | Select from built-in CSV data files. | | [CSV Metric Values](#csv-metric-values) | Manual input | Time series from comma-separated values. | | [Raw Frames](#raw-frames) | Manual input | Define data frames in JSON format. | | [Load Apache Arrow Data](#load-apache-arrow-data) | Manual input | Render base64-encoded Arrow payloads. | | [Table Static](#table-static) | Manual input | Static table with predefined columns. | | [Logs](#logs) | Visualization testing | Simulated log data with random levels. | | [Node Graph](#node-graph) | Visualization testing | Data for node graph visualizations. | | [Flame Graph](#flame-graph) | Visualization testing | Data for flame graph visualizations. | | [Trace](#trace) | Visualization testing | Simulated distributed trace data. | | [Annotations](#annotations) | Visualization testing | Annotation data points. | | [Exponential heatmap bucket data](#exponential-heatmap-bucket-data) | Visualization testing | Heatmap data with exponential buckets. | | [Linear heatmap bucket data](#linear-heatmap-bucket-data) | Visualization testing | Heatmap data with linear buckets. | | [Exemplars](#exemplars) | Visualization testing | Exemplar markers to overlay on a sibling series. | | [Streaming Client](#streaming-client) | Streaming | Browser-side streaming (signal, logs, traces). | | [Grafana Live](#grafana-live) | Streaming | Server-side streaming via live channels. | | [Grafana API](#grafana-api) | Data retrieval | Fetch data from internal Grafana endpoints. | | [Conditional Error](#conditional-error) | Error testing | Configurable error or CSV data. | | [Error with source](#error-with-source) | Error testing | Error with plugin or downstream classification. | | [Flaky Query](#flaky-query) | Error testing | Intermittent errors with configurable rate. | | [Errors and notices](#errors-and-notices) | Error testing | Series with info, warning, and error notices. | | [No Data Points](#no-data-points) | Error testing | Empty result with no data. | | [Datapoints Outside Range](#datapoints-outside-range) | Error testing | Data point outside the visible time range. | | [Slow Query](#slow-query) | Error testing | Configurable delay before returning data. | | [Query Metadata](#query-metadata) | Metadata | Returns query context metadata. | ## Shared query options Every scenario displays the **Scenario** drop-down. Most scenarios also show an **Alias** field and, when the scenario defines a default, a **String Input** field. Some scenarios expose additional shared controls. Scenario-specific options appear below these fields. | Field | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Scenario** | Drop-down to select the data generation type. Scenarios are listed alphabetically. | | **Alias** | Optional display name for the series. Hidden for the Simulation and Annotations scenarios. Supports the special values `__server_names` and `__house_locations` to use built-in name sets. | | **String Input** | Text field shown when the selected scenario provides a default string value. Used differently by each scenario. | | **Labels** | Available for Random Walk and Predictable Pulse. Set labels using key=value syntax, for example `key="value", key2="value2"`. Use `$seriesIndex` in a value to insert the series index. | | **Drop percent** | Available for CSV Content and CSV File. Drops a random percentage (0-100) of data points from the result. | ## Data generation scenarios These scenarios produce time-series data with configurable parameters. ### Random Walk Generates random walk time-series data. This is the default scenario. | Field | Description | | ---------------- | ------------------------------------------------------- | | **Series count** | Number of series to generate. Default: `1`. | | **Start value** | Initial value for the walk. Default: auto (random). | | **Min** | Minimum value the walk can reach. Default: none. | | **Max** | Maximum value the walk can reach. Default: none. | | **Spread** | Controls how far each step can deviate. Default: `1`. | | **Noise** | Adds noise to each data point. Default: `0`. | | **Drop (%)** | Percentage of points to randomly exclude. Default: `0`. | ### Random Walk Table Generates random walk data in table format with columns for Time, Value, Min, Max, Info, and State. The State column uses an `enum` field with values Unknown, Up, and Down. Optionally includes null values when **withNil** is enabled. ### Random Walk (with error) Generates random walk time-series data and also returns an error in the response. Use this to test how panels handle data responses that contain both data and errors. ### Predictable Pulse Generates a predictable pulse wave based on absolute time from the epoch, making it reproducible across runs. | Field | Description | | ------------- | ------------------------------------------------------------------------------------- | | **Step** | Seconds between data points. Default: `60`. | | **On Count** | Number of data points at the start of each cycle that use the on value. Default: `3`. | | **Off Count** | Number of data points in each cycle that use the off value. Default: `3`. | | **On Value** | The value for "on" data points. Can be a number, `null`, or `nan`. Default: `2`. | | **Off Value** | The value for "off" data points. Can be a number, `null`, or `nan`. Default: `1`. | The wave cycles at `Step * (On Count + Off Count)` seconds. Timestamps align evenly on the step interval. With the defaults (Step=60, On Count=3, Off Count=3), the wave completes a full cycle every 6 minutes. The on value (`2`) holds for the first 3 data points, then the off value (`1`) holds for the next 3. ### Predictable CSV Wave Generates one or more predictable waves from CSV-defined values. Each wave cycles through its comma-separated values at a fixed time step. | Field | Description | | ---------- | ----------------------------------------------------------------------------------------------- | | **Values** | Comma-separated numeric values for the wave. Supports `null` and `nan`. Default: `0,0,2,2,1,1`. | | **Step** | Seconds between data points. Default: `60`. | | **Name** | Optional name for the series. | | **Labels** | Optional labels in key=value format. | Click the **+** button on the last wave row to add another wave. Click the **-** button on any other row to remove it. For example, to create two overlapping waves for comparison: - Wave 1: Values `0,1,2,3,4,5`, Step `60`, Name `rising` - Wave 2: Values `5,4,3,2,1,0`, Step `60`, Name `falling` ### Simulation Runs a simulation engine that generates data continuously. Simulation supports streaming data through Grafana Live. | Field | Description | | -------------- | -------------------------------------------------------------------------------------------------------------- | | **Simulation** | Simulation type: `Flight` (circling flight path), `Sine` (sine wave), or `Tank` (fill and drain a water tank). | | **Stream** | Toggle to stream data through Grafana Live instead of returning a static result. | | **Interval** | Tick frequency in Hz. Default: `10`. | | **Last** | Toggle to return only the most recent value instead of the full time range. | | **UID** | Optional unique identifier. Allows multiple instances of the same simulation type to run concurrently. | ### USA generated data Generates data with US state dimensions. Useful for testing geo-map visualizations and multi-dimensional data. | Field | Description | | ---------- | ------------------------------------------------------------------------------------------------------------------ | | **Mode** | Data format: `values-as-rows`, `values-as-fields`, `values-as-labeled-fields`, `timeseries`, or `timeseries-wide`. | | **Period** | Time period for data generation. Default: `30m`. | | **Fields** | Fields to include: `foo`, `bar`, `baz`. Default: all. | | **States** | US state codes to include (for example, `CA`, `NY`, `TX`). Default: all 50 states plus DC. | ## Manual input scenarios These scenarios let you provide your own data directly instead of generating it. ### CSV Content Provides a text editor where you paste or type CSV data directly. The first row is treated as headers. Use the **Drop percent** field to randomly exclude a percentage of data points. For example, enter the following to create a time series with two value columns: ``` Time,Temperature,Humidity 2024-01-01 00:00:00,22.5,45 2024-01-01 01:00:00,21.8,48 2024-01-01 02:00:00,20.1,52 ``` ### Steps Provides a CSV text area (the same editor as CSV Content) for defining step-function data. The default content is `a`, `b`, `c`. This scenario is handled entirely in the browser. ### CSV File Selects from a set of built-in CSV data files. Use the **Drop percent** field to randomly exclude a percentage of data points. Available files: - `flight_info_by_state.csv` - `population_by_state.csv` - `gdp_per_capita.csv` - `js_libraries.csv` - `ohlc_dogecoin.csv` - `weight_height.csv` - `browser_marketshare.csv` - `automobiles.csv` ### CSV Metric Values Generates time-series data from comma-separated values entered in the **String Input** field. Values are evenly distributed across the selected time range. Default: `1,20,90,30,5,0`. For example, with the default values and a 6-hour time range, the six values (`1`, `20`, `90`, `30`, `5`, `0`) are spread across the range at equal intervals. ### Raw Frames Provides a JSON editor for defining data frames directly. The editor accepts the Grafana data frame JSON format and includes paste helpers for panel JSON and raw query results. ### Load Apache Arrow Data Loads data from a base64-encoded Apache Arrow payload entered in the **String Input** text area. Use this to test Arrow format rendering. ### Table Static Generates a static table with predefined columns: Time, Message, Description, and Value. ## Visualization test scenarios These scenarios generate data for testing specific visualization types. ### Logs Generates simulated log data with random log levels, container IDs, and hostnames. | Field | Description | | --------- | ------------------------------------------------------------------------------------------ | | **Lines** | Number of log lines to generate. Default: `10`. Maximum: `10000`. | | **Level** | Toggle to include a separate level column in the data instead of embedding it in messages. | ### Node Graph Generates data for the [Node Graph visualization](https://grafana.com/docs/grafana//panels-visualizations/visualizations/node-graph/). | Field | Description | | ------------- | --------------------------------------------------------------------------------------------------------- | | **Data type** | Type of graph data: `random`, `response_small`, `response_medium`, `random edges`, or `feature_showcase`. | | **Count** | Number of nodes to generate. Available for `random` and `random edges` types. | | **Seed** | Seed value for reproducible random generation. Available for `random` and `random edges` types. | ### Flame Graph Generates data for the [Flame Graph visualization](https://grafana.com/docs/grafana//panels-visualizations/visualizations/flame-graph/). | Field | Description | | ---------------- | --------------------------------------------------------------------- | | **Diff profile** | Toggle to generate a differential flame graph comparing two profiles. | ### Trace Generates simulated distributed trace data. | Field | Description | | -------------- | --------------------------------------- | | **Span count** | Number of spans in the generated trace. | ### Annotations Generates synthetic annotation data points. These annotations are created entirely in the browser and are useful for testing how panels display annotation markers and overlays. They don't query an external source or persist data. For more information about using annotations with TestData, refer to [TestData annotations](https://grafana.com/docs/grafana//datasources/testdata/annotations/). For more information about annotations in Grafana, refer to [Annotate visualizations](https://grafana.com/docs/grafana//dashboards/build-dashboards/annotate-visualizations/). | Field | Description | | --------- | ------------------------------------------------- | | **Count** | Number of annotations to generate. Default: `10`. | ### Exponential heatmap bucket data Generates heatmap data with exponentially distributed bucket boundaries (1, 2, 4, 8, 16, ...). Use this to test heatmap panels with exponential distributions. ### Linear heatmap bucket data Generates heatmap data with linearly distributed bucket boundaries (0, 10, 20, 30, ...). Use this to test heatmap panels with linear distributions. ### Exemplars Generates only exemplars, the diamond markers a panel draws on top of a series, and no series of its own. Add a second query in the same panel, such as Random Walk, to give the exemplars a series to annotate. Values and timestamps are seeded from the dashboard time range, so the same range always returns the same exemplars. | Field | Description | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------- | | **Count** | Number of exemplars to generate. | | **Min** | Minimum value for the exemplars. When empty, the range is derived from a reference random walk over the same time range, padded by 10%. | | **Max** | Maximum value for the exemplars. When empty, the range is derived as described for **Min**. | | **Labels** | Exemplar labels shown in the tooltip. Each label defines a **Name**, a **Length**, and an optional **Link**. | To keep the exemplars aligned with a sibling series, pin **Min** and **Max**, or give the sibling query an explicit start value and spread. The legend filter matches on exemplar labels, so hiding a series in the legend also hides its markers. ## Streaming scenarios These scenarios produce real-time streaming data. Data updates continuously in the browser without requiring manual query execution. ### Streaming Client Generates streaming data directly from the browser client. | Field | Description | | -------- | ------------------------------------------------------------- | | **Type** | Stream type: `Signal`, `Logs`, `Fetch`, `Traces`, or `Watch`. | Additional fields depend on the selected type: - **Signal:** Speed (ms), Spread, Noise, Bands. - **Logs, Traces, Watch:** Speed (ms) only. - **Fetch:** URL of a remote CSV endpoint to stream incrementally. ### Grafana Live Connects to a Grafana Live channel that streams random data from the server. | Field | Description | | ----------- | ------------------------------------------ | | **Channel** | The live channel to subscribe to. Options: | | Channel | Description | | ----------------------- | --------------------------------------------- | | `random-2s-stream` | Random stream with points every 2s. | | `random-flakey-stream` | Stream that returns data at random intervals. | | `random-labeled-stream` | Value with moving labels. | | `random-20Hz-stream` | Random stream with points at 20 Hz. | ## Data retrieval scenarios These scenarios fetch data from internal Grafana endpoints. ### Grafana API Fetches data from internal Grafana API endpoints and returns the result as a data frame. This scenario runs in the browser. | Field | Description | | ------------ | ----------------------------------- | | **Endpoint** | The API endpoint to query. Options: | | Endpoint | Description | | ---------------- | --------------------------------- | | **Data Sources** | Lists configured data sources. | | **Search** | Returns dashboard search results. | | **Annotations** | Returns annotations. | ## Error and edge-case testing scenarios These scenarios help test how Grafana handles errors, empty data, and slow responses. ### Conditional Error Produces an error or data depending on the **String Input** and **Error type** fields. When the **String Input** field is empty, the scenario triggers the selected error type. When the field contains CSV values (default: `1,20,90,30,5,0`), it returns time-series data like the CSV Metric Values scenario, regardless of the error type. | Field | Description | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Error type** | Type of error to simulate: `Server panic` (backend panic), `Frontend exception` (uncaught error in the browser), or `Frontend observable` (error emitted through the query observable). | To trigger an error, clear the **String Input** field and select the error type you want to test. ### Error with source Returns an error with a configurable source classification. | Field | Description | | ---------------- | ---------------------------------------------------------------------------------------- | | **Error source** | Source of the error: `Plugin` (plugin error) or `Downstream` (downstream service error). | Use this to test how Grafana differentiates between plugin errors and downstream errors in alerting and error handling. ### Flaky Query Returns errors intermittently based on a configurable error rate, with optional delays. Use this to test how panels, alert rules, and error handling behave when a data source fails part of the time. | Field | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | **Error rate** | Percentage of requests that return an error. Default: `50`. | | **Status code** | HTTP status code returned with the error. Default: `400`. | | **Error source** | Source of the error: `Plugin` (plugin error) or `Downstream` (downstream service error). | | **Query delay** | Base delay applied to each request in Go duration syntax, for example `1s` or `500ms`. Default: `5s`. | | **Delay variability** | Randomizes the query delay by plus or minus this percentage. For example, `100` on a `1s` delay sleeps between 0 and 2 seconds. | | **Error message** | Message returned with the error. Default: `Flaky query error`. | ### Errors and notices Returns a random walk series with one query-result notice of each severity (info, warning, and error) attached to the frame metadata. Use this to preview how a panel displays query errors and notices. ### No Data Points Returns an empty result with no data points. Use this to test how panels display when there's no data. ### Datapoints Outside Range Returns a single data point with a timestamp one hour before the query time range. Use this to test how panels handle data outside the visible range. ### Slow Query Introduces a configurable delay before returning random walk data. Set the delay duration in the **String Input** field using Go duration syntax (for example, `5s`, `1m`, `500ms`). Default: `5s`. ## Metadata scenarios These scenarios return metadata rather than time-series data. ### Query Metadata Returns a table with query metadata including the current user's username. Use this to verify that query context information is available. # TestData template variables Use template variables with the TestData data source to test dynamic, reusable dashboards. TestData provides a hierarchical metric tree that you can query to populate variable drop-downs with simulated values. For an introduction to templating and template variables, refer to the [Templating](https://grafana.com/docs/grafana//dashboards/variables/) and [Add and manage variables](https://grafana.com/docs/grafana//dashboards/variables/add-template-variables/) documentation. ## Supported variable types | Variable type | Supported | | ------------- | --------- | | Query | Yes | | Custom | Yes | | Data source | Yes | ## Create a query variable To create a query variable using TestData: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Variable**. 1. Select **Query** as the variable type. 1. Enter a **Name** for your variable. 1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed. 1. Click **Open variable editor** to open the **Query Variable** dialog box. 1. Select the **TestData** data source. 1. Enter your query in the **Query** field. ## Metric tree queries TestData generates a hierarchical metric tree with nodes named using letter combinations. The tree has 6 levels of depth, with each level containing 8 nodes labeled A through H. The query string navigates this tree using dot-separated path segments. ### Query syntax Use `*` as a wildcard to return all children at a given level: | Query | Result | | -------- | --------------------------------------------------------------------------- | | `*` | Returns the top-level nodes: `A`, `B`, `C`, `D`, `E`, `F`, `G`, `H`. | | `A.*` | Returns children of node A: `AA`, `AB`, `AC`, `AD`, `AE`, `AF`, `AG`, `AH`. | | `A.AA.*` | Returns children of node AA: `AAA`, `AAB`, `AAC`, etc. | | `A.AB` | Returns children of the AB node: `ABA`, `ABB`, `ABC`, etc. | Use glob patterns with curly braces to match multiple nodes: | Query | Result | | ------------- | -------------------------------------------------------------------- | | `{A,B}.*` | Returns children of both A and B: `AA`, `AB`, ..., `BA`, `BB`, etc. | | `A.{AA,AB}.*` | Returns children of AA and AB: `AAA`, `AAB`, ..., `ABA`, `ABB`, etc. | Use `*` after a prefix to match nodes that start with the given string: | Query | Result | | ----- | ------------------------------------------------------------------------ | | `A*` | Returns all top-level nodes starting with A (only `A` at the top level). | ### Value queries If your query string starts with `value`, TestData returns the query string itself as the result. This is useful for creating variables with specific values for testing. For example, a query of `value_production` returns `value_production`. ## Example: Create a region variable This example walks through creating a query variable that populates a drop-down with simulated region values, then using it in a panel query. 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Variable**. 1. Select **Query** as the variable type. 1. Enter a **Name** for your variable (for example, `host`). 1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed. 1. Click **Open variable editor** to open the **Query Variable** dialog box. 1. Select the **TestData** data source. 1. Enter `*` in the **Query** field. This populates the variable with the top-level nodes (A through H). 1. Name the variable `region`. 1. Click **Close** when you've completed the query setup. 1. (Optional) Configure **Multi-value** to allow selecting multiple values. 1. (Optional) Configure **Include All value** to add an "All" option. 1. (Optional) Turn on the **Allow custom values** switch to allow users to enter custom variable values. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. To use the variable in a panel: 1. Add a new panel and select the **TestData** data source. 1. Choose the **Random Walk** scenario. 1. In the **Labels** field, enter `region=$region`. 1. The panel updates when you select a different value from the **region** drop-down. ## Example: Chain variables for cascading drop-downs You can create dependent variables where one variable's query references another, producing cascading drop-downs. 1. Create a variable named `region` with the query `*`. This returns the top-level nodes (A through H). 1. Create a second variable named `host` with the query `$region.*`. When the user selects `A` as the region, the host drop-down populates with `AA`, `AB`, `AC`, etc. Changing the region selection automatically refreshes the host drop-down. ## Use variables in queries TestData interpolates template variables in the following query fields: - **Labels** - **Alias** - **Scenario** - **String Input** - **CSV Content** - **Raw Frame Content** Use the standard Grafana variable syntax (`$varname` or `${varname}`) in any of these fields to insert variable values. ### Example With a variable named `env` set to `production`, entering `env=$env` in the **Labels** field produces a series labeled `env=production`. ## Limitations TestData doesn't support the use of [filters](https://grafana.com/docs/grafana//visualizations/dashboards/build-dashboards/filter-group-by/). Filters require the data source to implement tag key and value lookups, which TestData doesn't provide. # Troubleshoot TestData data source issues This document provides solutions to common issues you may encounter when using the TestData data source. TestData is a built-in data source with no external dependencies, so most issues relate to scenario configuration or data formatting. ## Query errors These errors occur when running TestData scenarios. ### "No data" or empty panels **Symptoms:** - Panel displays "No data" message. - Query executes without error but returns nothing. **Possible causes and solutions:** | Cause | Solution | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | No Data Points scenario selected | The No Data Points scenario returns empty results by design. Select a different scenario such as Random Walk. | | Datapoints Outside Range selected | This scenario returns a data point one hour before the query time range. Expand the time range or select a different scenario. | | Empty String Input for CSV scenarios | CSV Metric Values and CSV Content require input data. Add comma-separated values or CSV content. | | Time range doesn't contain data | Some scenarios generate data relative to the query time range. Expand the dashboard time range. | ### Exemplars don't appear **Symptoms:** - The Exemplars scenario runs without error but no diamond markers appear on the panel. - Markers disappear when you interact with the legend. **Possible causes and solutions:** | Cause | Solution | | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | No series query in the panel | Exemplars returns only markers and no series of its own. Add a second query, such as Random Walk, for the exemplars to annotate. | | Series hidden in the legend | The legend filter matches on exemplar labels, so hiding a series in the legend also hides its markers. Show the series to restore the markers. | | Markers fall outside the panel value range | With **Min** and **Max** empty, the value range is derived from a reference random walk and markers can land off-panel. Set **Min** and **Max** explicitly, or align them with the sibling series. | ### Wrong data type for the visualization **Symptoms:** - Panel shows "Data does not have a time field" or similar type mismatch errors. - Visualization renders but looks wrong (for example, a table when you expected a graph). **Solutions:** 1. Match the scenario to the visualization type. For example, use the Logs scenario with the Logs panel, Node Graph with the Node Graph panel, and Trace with the Traces panel. 1. Time-series panels require scenarios that produce time-series data (Random Walk, Predictable Pulse, CSV Metric Values, Predictable CSV Wave). 1. For table panels, use Table Static, Random Walk Table, or CSV File. ### Predictable Pulse pattern doesn't match expectations **Symptoms:** - The on/off pattern doesn't align with expected times. - Values appear at unexpected intervals. **Solutions:** 1. Predictable Pulse aligns timestamps to the step interval based on absolute time from the epoch, not from the start of the time range. This means the pattern starts at the same point regardless of when you load the dashboard. 1. Verify the cycle duration matches your intent: the full cycle is `Step * (On Count + Off Count)` seconds. 1. Check that **On Value** and **Off Value** are set correctly. The defaults are `2` and `1`, not `1` and `0`. ### Slow Query appears stuck **Symptoms:** - Query runs for an extended period. - Panel shows a loading spinner indefinitely. **Solutions:** 1. Check the **String Input** field, which controls the delay duration. The default is `5s`. 1. Reduce the value to a shorter duration (for example, `1s` or `500ms`). 1. The field accepts Go duration syntax: `5s` (seconds), `1m` (minutes), `500ms` (milliseconds). ### Unexpected error from Conditional Error **Symptoms:** - Query returns a server error or panic message. - Error appears immediately when running the query. **Solutions:** 1. The Conditional Error scenario triggers a server panic when the **String Input** field is empty. This is the intended behavior for error testing. 1. To return data instead of an error, populate the **String Input** field with comma-separated values (for example, `1,20,90,30,5,0`). 1. To change the error type, use the **Error type** drop-down to select between Server panic, Frontend exception, and Frontend observable. ### Error with source returns an error The Error with source scenario intentionally returns errors for testing how Grafana handles different error sources. This isn't a misconfiguration. - **Plugin** error source simulates an error originating from the plugin itself. - **Downstream** error source simulates an error from a downstream service. Use this scenario to test alerting rules, error displays, and error handling in custom panels. ### Flaky Query returns errors intermittently The Flaky Query scenario returns errors for a configurable percentage of requests. Intermittent errors are the intended behavior, not a misconfiguration. **Solutions:** 1. Adjust the **Error rate** field to control how often requests fail. Set it to `0` for no errors or `100` to fail every request. 1. Use the **Status code** and **Error source** fields to shape the returned error. 1. If you're using Flaky Query with an alert rule, be aware that intermittent failures can flip the rule between states across evaluations. For alerting behavior, refer to [TestData alerting](https://grafana.com/docs/grafana//datasources/testdata/alerting/). ## CSV and data input errors These errors occur with scenarios that accept user-provided data. ### CSV Content or CSV Metric Values return unexpected results **Symptoms:** - Data doesn't match expectations. - Fewer data points than expected. - Parsing errors. **Solutions:** 1. Verify CSV formatting: headers in the first row, commas as delimiters, numeric values unquoted. 1. For CSV Metric Values, enter only comma-separated numbers in the **String Input** field (for example, `1,20,90,30,5,0`). 1. The special values `null` and `nan` are supported in Predictable CSV Wave values. Use `null` for missing data points and `nan` for Not-a-Number values. 1. Check for trailing commas or extra whitespace in your input. ### Raw Frames JSON parse errors **Symptoms:** - Error message about invalid JSON. - Panel fails to render. **Solutions:** 1. Validate that the JSON matches the Grafana data frame format. 1. Use the paste helpers in the editor to import data from panel JSON or raw query results. 1. Verify that field types are consistent within each field array. ## Streaming errors These errors occur with the Streaming Client and Grafana Live scenarios. ### Streaming Client doesn't update **Symptoms:** - Panel shows initial data but doesn't update. - No new data points appear over time. **Solutions:** 1. Verify the **Speed** field is set to a reasonable value (in milliseconds). Lower values produce faster updates. 1. For the **Fetch** type, verify the URL is accessible from the browser. 1. Check the browser console for WebSocket or network errors. ### Grafana Live channel shows no data **Symptoms:** - Panel remains empty after selecting a channel. - No streaming data appears. **Solutions:** 1. Confirm the selected channel matches one of the available options: `random-2s-stream`, `random-flakey-stream`, `random-labeled-stream`, or `random-20Hz-stream`. 1. Verify that Grafana Live is enabled in your Grafana instance. 1. Check the Grafana server logs for Live connection errors. ## Template variable errors These errors occur when using TestData with [dashboard template variables](https://grafana.com/docs/grafana//datasources/testdata/template-variables/). ### Variable drop-down is empty **Symptoms:** - Variable drop-down shows no options. - Preview in the variable editor returns zero values. **Solutions:** 1. Verify the query string navigates valid nodes in the metric tree. Use `*` to return all top-level nodes. 1. Check that dot-separated path segments match the tree structure (for example, `A.*` returns children of node A). An invalid path like `Z.*` returns nothing because `Z` doesn't exist in the tree. 1. Verify the TestData data source is selected in the variable editor. 1. Click **Run query** in the variable editor preview to test. ### Chained variable doesn't update **Symptoms:** - A dependent variable (for example, `$region.*`) doesn't refresh when the parent variable changes. - The drop-down shows stale values from a previous selection. **Solutions:** 1. Set the dependent variable's **Refresh** option to **On dashboard load** or **On time range change** in the variable editor. 1. Verify the query uses the correct parent variable syntax (for example, `$region.*` not `region.*`). ### Variable value not interpolated in queries **Symptoms:** - Panel shows the literal string `$varname` instead of the variable value. - Labels or data don't change when the variable selection changes. **Solutions:** 1. Verify you're using the correct variable syntax: `$varname` or `${varname}`. 1. TestData interpolates variables in these fields only: **Labels**, **Alias**, **Scenario**, **String Input**, **CSV Content**, and **Raw Frame Content**. Other fields don't support variable substitution. ## Alerting errors These errors occur when using TestData with [Grafana Alerting](https://grafana.com/docs/grafana//datasources/testdata/alerting/). ### Alert rule shows "no data" unexpectedly **Symptoms:** - Alert rule status shows **No Data** even though the scenario works in a panel. - Alert never transitions to firing or normal. **Solutions:** 1. Verify you're using a backend-evaluated scenario. Browser-only scenarios (Streaming Client, Grafana Live, Grafana API, Steps, Annotations, Node Graph, Flame Graph, Trace, and Raw Frames) return empty results when evaluated by the alerting engine. Use a backend scenario like Random Walk or Predictable Pulse instead. 1. Confirm the scenario is returning time-series data. Scenarios that produce logs (Logs), traces (Trace), graphs (Node Graph, Flame Graph), or tables (Table Static) can't be used as alert conditions. 1. Check that the **String Input** field isn't empty for scenarios that require it (for example, CSV Metric Values). ### Alert rule never fires **Symptoms:** - Alert rule stays in **Normal** state. - Data is visible in the panel but the alert condition is never met. **Solutions:** 1. If using Random Walk, the values are non-deterministic and may not exceed your threshold. Use Predictable Pulse or CSV Metric Values with values that reliably cross the threshold. 1. Verify the **Reduce** expression is aggregating correctly. For example, **Last** returns the most recent value, while **Mean** averages the series. 1. Check the **Threshold** expression. Click **Preview** in the alert rule editor to see the evaluated value. 1. Ensure the evaluation interval is long enough for the scenario to produce data points. A Predictable Pulse with Step=60 needs at least a 60-second evaluation window. ### Template variables don't resolve in alert rules **Symptoms:** - Alert rule error mentions unresolved variables. - Alert query uses `$varname` syntax but the variable isn't replaced. **Solutions:** Grafana evaluates alert rules on the backend without dashboard context. Template variables like `$varname` aren't resolved during alert evaluation. Replace variable references with fixed values in the scenario options. For more information, refer to the [alerting limitations](https://grafana.com/docs/grafana//datasources/testdata/alerting/#template-variables-arent-supported-in-alert-queries). ### Alert evaluation timeout **Symptoms:** - Alert rule transitions to **Error** state. - Logs show a timeout during query evaluation. **Solutions:** 1. If you're using the Slow Query scenario, reduce the delay in the **String Input** field to a value shorter than the evaluation interval. 1. For other scenarios, verify that the query isn't producing an excessive number of data points. Reduce the time range or increase the step interval. ## Enable debug logging To capture detailed error information for troubleshooting: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location). 1. Look for entries containing `testdata` or `testdatasource` for scenario-specific details. 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you've tried the solutions in this document and still encounter issues: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs. 1. When reporting issues, include: - Grafana version - Selected scenario and its configuration - Error messages (redact sensitive information) - Steps to reproduce # Configure the Zipkin data source This document explains how to configure the Zipkin data source in Grafana, including connection settings, authentication, trace-to-logs and trace-to-metrics integrations, and provisioning with YAML or Terraform. ## Before you begin Before configuring the data source, ensure you have: - **Grafana permissions:** Organization administrator role - **Zipkin instance:** A running Zipkin instance accessible from your Grafana server ## Add the data source To add the Zipkin data source: 1. Click **Connections** in the left-side menu. 1. Click **Add new connection**. 1. Type `Zipkin` in the search bar. 1. Select **Zipkin**. 1. Click **Add new data source**. ## Configure settings The following table describes the basic configuration settings for the Zipkin data source. | Setting | Description | | ----------- | ---------------------------------------------------------------- | | **Name** | The name used to refer to the data source in panels and queries. | | **Default** | Toggle to make this the default data source for new panels. | | **URL** | The URL of the Zipkin instance, such as `http://localhost:9411`. | ## Authentication The Zipkin data source supports the following authentication methods: - **Basic authentication:** Provide a username and password to authenticate with the Zipkin instance. - **TLS client authentication:** Configure client certificates for mutual TLS. - **Forward OAuth identity:** Forward the user's OAuth token to the Zipkin instance. - **With credentials:** Send credentials (cookies, TLS client certificates) with cross-site requests. You can also configure custom headers and TLS settings in the **Advanced HTTP settings** section, which is located inside **Additional settings**. ## Trace to logs {{< figure src="/media/docs/tempo/tempo-trace-to-logs-9-4.png" max-width="800px" class="docs-image--no-shadow" caption="Trace to logs settings" >}} {{< admonition type="note" >}} If you use Grafana Cloud, open a [support ticket in the Cloud Portal](/profile/org#support) to access this feature. {{< /admonition >}} The **Trace to logs** setting configures the [trace to logs feature](https://grafana.com/docs/grafana//explore/trace-integration/) that is available when you integrate Grafana with Zipkin. This feature lets you navigate from a span in a trace view directly to the relevant logs. There are two ways to configure the trace to logs feature: - Use a simplified configuration with a default query, or - Configure a custom query where you can use a [template language](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/) to interpolate variables from the trace or span. ### Use a simple configuration 1. Select the target data source from the drop-down list. You can also click **Open advanced data source picker** to see more options, including adding a data source. 1. Set start and end time shift. Since the log timestamps may not exactly match the timestamps of the spans in the trace, you may need to widen or shift the time range to find the desired logs. 1. Select which tags to use in the logs query. The tags you configure must be present in the span attributes or resources for a trace to logs span link to appear. You can optionally configure a new name for the tag. This is useful if the tag has dots in the name and the target data source doesn't allow dots in labels. For example, you can remap `http.status` to `http_status`. 1. Optionally, switch on the **Filter by trace ID** and/or **Filter by span ID** setting to further filter the logs if your logs consistently contain trace or span IDs. ### Configure a custom query 1. Select the target data source from the drop-down list. You can also click **Open advanced data source picker** to see more options, including adding a data source. 1. Set start and end time shift. Since the log timestamps may not exactly match the timestamps of the spans in the trace, you may need to widen or shift the time range to find the desired logs. 1. Optionally, select tags to map. These tags can be used in the custom query with the `${__tags}` variable. This variable interpolates the mapped tags as a list in an appropriate syntax for the data source and only includes tags that are present in the span. You can optionally configure a new name for the tag. This is useful when the tag has dots in the name and the target data source doesn't allow dots in labels. For example, you can remap `http.status` to `http_status`. If you don't map any tags here, you can still use any tag in the query like this: `method="${__span.tags.method}"`. 1. Skip **Filter by trace ID** and **Filter by span ID** settings as these can't be used with a custom query. 1. Switch on **Use custom query**. 1. Specify a custom query to be used to query the logs. You can use various variables to make the query relevant for the current span. The link only appears if all the variables are interpolated with non-empty values to prevent creating an invalid query. ### Variables for custom queries To use a variable, wrap it in `${}`. For example: `${__span.name}`. | Variable name | Description | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **\_\_tags** | This variable uses the tag mapping from the UI to create a label matcher string in the specific data source syntax. The variable only uses tags that are present in the span. The link is still created even if only one of those tags is present in the span. You can use this if all tags aren't required for the query to be useful. | | **\_\_span.spanId** | The ID of the span. | | **\_\_span.traceId** | The ID of the trace. | | **\_\_span.duration** | The duration of the span. | | **\_\_span.name** | The name of the span. | | **\_\_span.tags** | Namespace for the tags in the span. To access a specific tag named `version`, you would use `${__span.tags.version}`. If the tag contains a dot, access it as `${__span.tags["http.status"]}`. | | **\_\_trace.traceId** | The ID of the trace. | | **\_\_trace.duration** | The duration of the trace. | | **\_\_trace.name** | The name of the trace. | ### Trace to logs settings The following table describes the trace to logs configuration options. | Setting name | Description | | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Data source** | Defines the target data source. You can select Loki, Elasticsearch, Splunk, OpenSearch, FalconLogScale, Google Cloud Logging, and VictoriaMetrics Logs data sources. | | **Span start time shift** | Shifts the start time for the logs query, based on the span's start time. You can use time units, such as `5s`, `1m`, `3h`. To extend the time to the past, use a negative value. Default: `0`. | | **Span end time shift** | Shifts the end time for the logs query, based on the span's end time. You can use time units. Default: `0`. | | **Tags** | Defines the tags to use in the logs query. Default: `cluster`, `hostname`, `namespace`, `pod`, `service.name`, `service.namespace`. You can change the tag name to remove dots from the name if they aren't allowed in the target data source. For example, map `http.status` to `http_status`. | | **Filter by trace ID** | Toggles whether to append the trace ID to the logs query. | | **Filter by span ID** | Toggles whether to append the span ID to the logs query. | | **Use custom query** | Toggles use of custom query with interpolation. | | **Query** | Input to write a custom query. Use variable interpolation to customize it with variables from the span. | ## Trace to metrics The **Trace to metrics** setting lets you navigate from a span in a trace view to a metrics query in a configured metrics data source. To configure trace to metrics: 1. Select the target data source from the drop-down list. You can also click **Open advanced data source picker** to see more options, including adding a data source. 1. Create any desired linked queries. | Setting name | Description | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Data source** | Defines the target data source. | | **Tags** | Defines the tags used in linked queries. The key sets the span attribute name, and the optional value sets the corresponding metric label name. For example, you can map `k8s.pod` to `pod`. To interpolate these tags into queries, use the `$__tags` keyword. | Each linked query consists of: - **Link Label:** _(Optional)_ Descriptive label for the linked query. - **Query:** The query run when navigating from a trace to the metrics data source. Interpolate tags using the `$__tags` keyword. For example, when you configure the query `requests_total{$__tags}` with the tags `k8s.pod=pod` and `cluster`, the result looks like `requests_total{pod="nginx-554b9", cluster="us-east-1"}`. ## Additional settings The **Additional settings** section is collapsible and contains optional settings for the node graph, span bar, advanced HTTP configuration, and secure SOCKS proxy (when enabled in your Grafana configuration). ### Node graph The **Enable node graph** toggle enables the [Node graph visualization](https://grafana.com/docs/grafana//panels-visualizations/visualizations/node-graph/), which is disabled by default. After you enable it, Grafana displays the node graph above the trace view. ### Span bar The **Span bar** setting lets you display additional information in the span bar row. The **Label** drop-down has three options: | Label | Description | | ------------ | ---------------------------------------------------------------------------------------------------------------------------- | | **None** | Adds nothing to the span bar row. | | **Duration** | _(Default)_ Displays the span duration on the span bar row. | | **Tag** | Displays the span tag on the span bar row. Enter the **Tag key** to specify which tag value to display, such as `component`. | ## Verify the connection Click **Save & test** to verify the connection. A successful connection displays the message **Data source is working**. If you encounter errors, refer to [Troubleshoot Zipkin data source issues](https://grafana.com/docs/grafana//datasources/zipkin/troubleshooting/). ## Provision the data source You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning and available configuration options, refer to [Provisioning Grafana](https://grafana.com/docs/grafana//administration/provisioning/#data-sources). ### Provisioning example ```yaml apiVersion: 1 datasources: - name: Zipkin type: zipkin uid: zipkin-ds url: http://localhost:9411 access: proxy basicAuth: true basicAuthUser: my_user readOnly: false isDefault: false jsonData: tracesToLogsV2: datasourceUid: 'loki' spanStartTimeShift: '1h' spanEndTimeShift: '-1h' tags: ['job', 'instance', 'pod', 'namespace'] filterByTraceID: false filterBySpanID: false customQuery: true query: 'method="$${__span.tags.method}"' tracesToMetrics: datasourceUid: 'prom' spanStartTimeShift: '1h' spanEndTimeShift: '-1h' tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }] queries: - name: 'Sample query' query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))' nodeGraph: enabled: true spanBar: type: 'None' secureJsonData: basicAuthPassword: my_password ``` ## Configure with Terraform You can configure the Zipkin data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation. ### Basic Terraform example The following example creates a basic Zipkin data source: ```hcl resource "grafana_data_source" "zipkin" { name = "Zipkin" type = "zipkin" url = "http://localhost:9411" } ``` ### Terraform example with trace to logs The following example includes a trace to logs configuration that links traces to a Loki data source: ```hcl resource "grafana_data_source" "zipkin" { name = "Zipkin" type = "zipkin" url = "http://localhost:9411" json_data_encoded = jsonencode({ tracesToLogsV2 = { datasourceUid = grafana_data_source.loki.uid spanStartTimeShift = "1h" spanEndTimeShift = "-1h" filterByTraceID = true filterBySpanID = false tags = [ { key = "service.name", value = "service" }, { key = "namespace" } ] } nodeGraph = { enabled = true } }) } ``` ### Terraform example with basic authentication The following example includes basic authentication: ```hcl resource "grafana_data_source" "zipkin" { name = "Zipkin" type = "zipkin" url = "http://localhost:9411" basic_auth_enabled = true basic_auth_username = "zipkin_user" secure_json_data_encoded = jsonencode({ basicAuthPassword = var.zipkin_password }) json_data_encoded = jsonencode({ nodeGraph = { enabled = true } }) } ``` For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). # Zipkin query editor This document explains how to use the Zipkin query editor to query and visualize traces. You can look up traces by ID, browse available traces using the cascading service and span selector, or upload a JSON trace file for visualization. For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/). ## Query by trace ID To query a specific trace by its ID: 1. Select the **TraceID** query type. 1. Enter the trace ID into the **Trace ID** field. 1. Press **Shift+Enter** to run the query. Zipkin trace IDs are 16 or 32 character hexadecimal strings, for example `efe9cb8857f68c8f` or `463ac35c9f6413ad48485a3953bb6124`. You can also use [template variables](https://grafana.com/docs/grafana//datasources/zipkin/template-variables/) in the trace ID field. For example, `${traceId}` is replaced with the variable's current value when the query runs. ## Query by trace selector You can browse and select traces using the **Traces** cascading selector instead of entering a trace ID manually. This is useful when you want to explore traces without knowing a specific trace ID. The trace selector has three levels: 1. **Service:** Select the service you're interested in, such as `frontend` or `api-gateway`. 1. **Span:** Select a specific operation within the selected service, such as `GET /api/users`. 1. **Trace:** Select a specific trace in which the selected operation occurred, represented by the root operation name and trace duration (for example, `get /api [23 ms]`). When you select a trace at the third level, the trace ID is automatically populated and the query runs. {{< admonition type="note" >}} The trace selector lists traces within the currently selected Explore time range. If you don't see expected traces, try expanding the time range. {{< /admonition >}} ## Upload a JSON trace file You can upload a JSON file that contains a single trace and visualize it. If the file has multiple traces, Grafana visualizes the first trace. To upload a trace file: 1. Click **Import trace**. 1. Drag and drop a JSON file or click to browse for a file. The file must contain a JSON array of spans in the [Zipkin v2 span format](https://zipkin.io/zipkin-api/#/). ### Single-span trace example ```json [ { "traceId": "efe9cb8857f68c8f", "id": "efe9cb8857f68c8f", "kind": "SERVER", "name": "get /api", "timestamp": 1627975249601797, "duration": 23457, "localEndpoint": { "serviceName": "backend", "ipv4": "127.0.0.1", "port": 9000 }, "tags": { "http.method": "GET", "http.path": "/api" } } ] ``` ### Multi-span trace example The following example shows a trace with a parent span and a child span, representing a frontend service calling a backend service: ```json [ { "traceId": "efe9cb8857f68c8f", "id": "efe9cb8857f68c8f", "kind": "SERVER", "name": "get /api", "timestamp": 1627975249601797, "duration": 23457, "localEndpoint": { "serviceName": "frontend", "ipv4": "127.0.0.1", "port": 8080 }, "tags": { "http.method": "GET", "http.path": "/api", "http.status_code": "200" } }, { "traceId": "efe9cb8857f68c8f", "parentId": "efe9cb8857f68c8f", "id": "8608dc6ce5cafe8e", "kind": "CLIENT", "name": "get /api/data", "timestamp": 1627975249602000, "duration": 18200, "localEndpoint": { "serviceName": "backend", "ipv4": "127.0.0.1", "port": 9000 }, "tags": { "http.method": "GET", "http.path": "/api/data", "http.status_code": "200" } } ] ``` ## View data mapping in the trace UI Zipkin annotations appear in the trace view as logs with the annotation value displayed under the annotation key. This lets you view timestamped events that occurred during a span, such as retry attempts or cache misses. ## Span filters {{< figure src="/media/docs/tempo/screenshot-grafana-tempo-span-filters-v10-1.png" max-width="800px" class="docs-image--no-shadow" caption="Span filters in the trace timeline viewer" >}} Using span filters, you can filter spans in the trace timeline viewer. The more filters you add, the more specific the filtered spans are. You can add one or more of the following filters: | Filter | Description | Example | | ---------------- | ------------------------------------------------- | ------------------------- | | **Service name** | Filter by the service that produced the span. | `frontend` | | **Span name** | Filter by the operation name. | `GET /api/users` | | **Duration** | Filter by minimum and/or maximum span duration. | Min: `10ms`, Max: `500ms` | | **Tags** | Filter by span tags, process tags, or log fields. | `http.status_code=500` | To only show the spans you have matched, toggle on **Show matches only**. ### Span filter examples - **Find slow database calls:** Set **Service name** to your database service and **Duration** minimum to `100ms`. - **Find errors:** Add a **Tags** filter for `error=true` or `http.status_code=500`. - **Isolate a specific service:** Set **Service name** and toggle on **Show matches only** to hide all other spans. ## Link to a trace ID from logs You can link to Zipkin traces from logs in Loki, Elasticsearch, Splunk, and other logs data sources by configuring an internal link. To configure this feature, refer to the [Derived fields](https://grafana.com/docs/grafana//datasources/loki/configure-loki-data-source/#derived-fields) section of the Loki data source docs or the [Data links](https://grafana.com/docs/grafana//datasources/elasticsearch/configure/#data-links) section of the Elasticsearch or Splunk data source docs. ## Link to a trace ID from metrics You can link to Zipkin traces from metrics in Prometheus data sources by configuring an exemplar. To configure this feature, refer to the [introduction to exemplars](https://grafana.com/docs/grafana//fundamentals/exemplars/) documentation. ## Troubleshoot query issues If you encounter issues with queries, refer to [Troubleshoot Zipkin data source issues](https://grafana.com/docs/grafana//datasources/zipkin/troubleshooting/). # Zipkin template variables Use template variables to create dynamic, reusable dashboards. Instead of hard-coding trace IDs, you can use variables to let dashboard viewers select or input trace IDs at view time, making it easier to share dashboards across teams. For an introduction to Grafana template variables, refer to [Variables](https://grafana.com/docs/grafana//dashboards/variables/). ## Supported variable types | Variable type | Description | Supported | | ------------- | ------------------------------------------------- | --------- | | Custom | Define a fixed list of trace IDs or other values. | Yes | | Text box | Let users type a trace ID at dashboard view time. | Yes | | Data source | Let users switch between Zipkin instances. | Yes | | Query | Populate options from a data source query. | No | The Zipkin data source doesn't support query-based variables, but you can use custom or text box variables to parameterize trace ID queries. ## Create a text box variable A text box variable lets dashboard viewers enter a trace ID directly. This is the most common variable type for Zipkin. To create a text box variable: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Variable**. 1. Select **Textbox** as the variable type. 1. Set the **Name** to `traceId`. 1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed. 1. (Optional) Set a default **Value** with a known trace ID for initial display. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. The dashboard displays a text input where viewers can paste a trace ID. ## Create a custom variable A custom variable provides a predefined list of values. This is useful when you want to give viewers a set of known trace IDs to choose from. To create a custom variable: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Variable**. 1. Select **Custom** as the variable type. 1. Set the **Name** to `traceId`. 1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed. 1. Click **Open variable editor** to open the **Custom options** dialog box. 1. Enter trace IDs in the **Values separated by comma** field: ``` efe9cb8857f68c8f,463ac35c9f6413ad48485a3953bb6124 ``` 1. Click **Apply**. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. The dashboard displays a drop-down where viewers can select from the predefined trace IDs. ## Create a data source variable A data source variable lets viewers switch between multiple Zipkin instances. This is useful when you have separate Zipkin deployments for different environments. To create a data source variable: 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Add new element** icon (blue plus sign). 1. Click **Variable**. 1. Select **Data source** as the variable type. 1. Set the **Name** to `zipkinDs`. 1. Select an option in the **Display** drop-down list to control where on the dashboard the variable is displayed. 1. Select **Zipkin** as the **Type**. 1. Click **Save**. 1. (Optional) Enter a description of the changes you've made. 1. Click **Save**. 1. Click **Exit edit**. Then, in your panel's query editor, select `${zipkinDs}` as the data source instead of a specific Zipkin instance. ## Use variables in queries You can use template variables in the **Trace ID** field of the query editor. Grafana replaces the variable with its current value when the query runs. For example, if you create a text box variable named `traceId`, enter `${traceId}` in the trace ID field. When the viewer enters a new trace ID, the query automatically runs with the new value. For more information about variable syntax, refer to [Variable syntax](https://grafana.com/docs/grafana//dashboards/variables/variable-syntax/). ## Troubleshoot template variable issues If template variables aren't working as expected, refer to [Troubleshoot Zipkin data source issues](https://grafana.com/docs/grafana//datasources/zipkin/troubleshooting/). # Troubleshoot Zipkin data source issues This document provides solutions to common issues you may encounter when configuring or using the Zipkin data source, including connection failures, query errors, JSON upload issues, and configuration problems. For configuration instructions, refer to [Configure Zipkin](https://grafana.com/docs/grafana//datasources/zipkin/configure/). ## Connection errors These errors occur when Grafana can't connect to the Zipkin instance. ### `error reading settings: url is empty` **Symptoms:** - **Save & test** fails immediately - Health check returns an error **Solutions:** 1. Open the data source settings and verify the **URL** field isn't empty. 1. Enter the full URL of your Zipkin instance, such as `http://localhost:9411`. ### "request failed: 404 Not Found" or other HTTP status errors **Symptoms:** - **Save & test** fails with an HTTP status error - Trace queries return errors **Possible causes and solutions:** | Cause | Solution | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | Incorrect URL or port | Verify the URL and port. The default Zipkin port is `9411`. Ensure you haven't accidentally used the Jaeger port (`16686`). | | Missing `/api/v2` path | The Zipkin data source calls `/api/v2/services`, `/api/v2/traces`, etc. Verify these endpoints are accessible at the configured URL. | | Reverse proxy misconfiguration | If using a reverse proxy, ensure it correctly forwards requests to the Zipkin API. | | Authentication failure | If the Zipkin instance requires authentication, verify your credentials are correct. | ### `error creating http client` or TLS errors **Symptoms:** - **Save & test** fails with a connection or TLS error - HTTPS connections are refused **Solutions:** 1. If using HTTPS, verify the TLS certificate is valid and trusted by the Grafana server. 1. Configure TLS settings in **Additional settings** > **Advanced HTTP settings**. 1. For self-signed certificates, add the CA certificate to the Grafana server trust store or toggle **Skip TLS Verify** (not recommended for production). ### Save & test times out **Symptoms:** - **Save & test** hangs and eventually times out - No error message is returned **Possible causes and solutions:** | Cause | Solution | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | Zipkin instance is down | Confirm the Zipkin instance is running and accessible from the Grafana server. | | Firewall or network rules | Ensure the Grafana server can reach the Zipkin instance. Check that firewall rules allow outbound traffic on the configured port. | | DNS resolution failure | Verify the hostname in the URL resolves correctly from the Grafana server. | For Grafana Cloud, if you're accessing a private Zipkin instance, configure [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/). ## Query editor errors These errors occur when using the query editor in Explore or dashboards. ### "Failed to load spans from Zipkin" **Symptoms:** - Error message appears in the query editor - The **Traces** cascading selector doesn't populate services, spans, or traces This error occurs when the frontend can't reach the Zipkin API endpoints (`/api/v2/services`, `/api/v2/spans`, or `/api/v2/traces`). **Solutions:** 1. Verify the Zipkin instance is running and accessible. 1. Test the endpoint directly: `curl http:///api/v2/services`. 1. Check that the Grafana server can reach the Zipkin API. 1. Review the Grafana server logs for more details. Look for `An error occurred while doing a resource call` in the logs. ### "An error occurred within the plugin" **Symptoms:** - A generic error appears when querying or browsing traces - No specific error details are shown in the UI This is a generic error returned by the Zipkin plugin backend when an internal error occurs. The actual error details are only available in the Grafana server logs. **Solutions:** 1. Enable [debug logging](#enable-debug-logging) to capture detailed error information. 1. Check the Grafana server logs for messages like `An error occurred while doing a resource call` or `An error occurred while processing response from resource call`. 1. Verify the Zipkin instance is healthy and responding to API requests. ### "No data" or empty results **Symptoms:** - Query runs without error but returns no data - Trace view shows "No data" - The cascading selector shows "No traces found" or "[No traces in time range]" **Possible causes and solutions:** | Cause | Solution | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | Invalid trace ID | Verify the trace ID is correct. Zipkin trace IDs are 16 or 32 character hex strings. | | Empty trace ID | Ensure the **Trace ID** field isn't empty. The backend rejects queries with `invalid/empty traceId`. | | Time range doesn't contain data | Expand the dashboard or Explore time range. Traces must fall within the selected time range for the cascading selector to find them. | | Trace has expired | Zipkin may have purged old trace data based on its retention settings. Check your Zipkin storage configuration. | | No services registered | If the cascading selector shows "No traces found" at the top level, your Zipkin instance may have no data. Verify data is being collected. | ## Upload errors These errors occur when importing a JSON trace file. ### "JSON is not valid Zipkin format" **Symptoms:** - Error appears after uploading a JSON trace file - The trace doesn't render **Possible causes and solutions:** | Cause | Solution | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Invalid JSON syntax | Validate the file with a JSON linter. Common issues include trailing commas, missing quotes, or unclosed brackets. | | Wrong JSON structure | The file must be a JSON array of span objects (`[{...}, {...}]`), not a single object. Refer to the [Zipkin v2 span format](https://zipkin.io/zipkin-api/#/). | | Wrong file type | Ensure you're uploading a `.json` file, not a different format. | ### `unsupported query type upload. only available in frontend mode` **Symptoms:** - Error when trying to use uploaded trace data in a dashboard panel or alert Upload queries only work in **Explore**. They can't be used in dashboard panels or alerting rules because the upload is processed in the browser, not the backend. **Solution:** Use the **TraceID** query type with the trace ID from your uploaded trace if you need to display it in a dashboard. This requires the trace to exist in your Zipkin instance. ## Configuration errors These errors relate to data source configuration settings. ### "Invalid time shift. See tooltip for examples." **Symptoms:** - Error appears in the trace to logs or trace to metrics configuration - The **Span start time shift** or **Span end time shift** field is highlighted **Solutions:** 1. Use valid time unit formats: `5s`, `1m`, `3h`, `-30m`. 1. Use a negative value to shift the time to the past (for example, `-1h`). 1. The value must be a number followed by a time unit (`s`, `m`, `h`). ### Trace to logs links don't appear **Symptoms:** - No log links appear in the trace view - Links appear for some spans but not others **Possible causes and solutions:** | Cause | Solution | | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tags not present in span | The tags configured in trace to logs settings must exist in the span attributes for the link to appear. Verify the span contains the expected tags. | | Logs data source not configured | Ensure the target logs data source (Loki, Elasticsearch, Splunk, OpenSearch, FalconLogScale, Google Cloud Logging, or VictoriaMetrics Logs) is configured and working. | | Custom query variables not resolved | If using a custom query, the link only appears when all variables resolve to non-empty values. Verify the span contains the referenced tags. | | Wrong data source type selected | The trace to logs data source must be a supported logs data source. Other data source types don't appear in the drop-down. | ### Trace to metrics links don't appear **Symptoms:** - No metrics links appear in the trace view **Solutions:** 1. Verify the target metrics data source is configured and working. 1. Ensure linked queries are defined with valid query syntax. 1. Check that the `$__tags` keyword in queries maps to tags that exist in the span. ## Template variable errors These errors relate to using template variables with the Zipkin data source. ### Variable value not substituted in query **Symptoms:** - The trace ID field shows the variable syntax (for example, `${traceId}`) instead of the actual value - Query returns no results when using a variable **Solutions:** 1. Navigate to the dashboard you want to update and click **Edit**. 1. Click the **Dashboard options** icon to open the sidebar. 1. Expand the **Variables** section. 1. Verify the variable is defined. 1. Check the variable name matches the syntax used in the query. Variable names are case-sensitive. 1. Ensure the variable has a value selected or entered. 1. For text box variables, verify the viewer has entered a value. ## Enable debug logging To capture detailed error information for troubleshooting: 1. Set the Grafana log level to `debug` in the configuration file: ```ini [log] level = debug ``` 1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location). 1. Look for Zipkin-specific entries that include request and response details, such as: - `Failed to close response body` - `An error occurred while doing a resource call` - `An error occurred while processing response from resource call` 1. Reset the log level to `info` after troubleshooting to avoid excessive log volume. ## Get additional help If you've tried these solutions and still encounter issues: 1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. 1. Review the [Grafana GitHub issues](https://github.com/grafana/grafana/issues) for known bugs. 1. Consult the [Zipkin documentation](https://zipkin.io/) for service-specific guidance. 1. Contact Grafana Support if you have a paid plan. [Open a support ticket](https://grafana.com/profile/org#support) for Grafana Cloud, or [contact support](https://grafana.com/contact/) for Grafana Enterprise. 1. When reporting issues, include: - Grafana version - Error messages (redact sensitive information) - Steps to reproduce - Relevant configuration (redact credentials)