Alerting

Improve alert rule quality with Alert Advisor
Alerts are easier to act on when they carry the right context.
A runbook URL helps you investigate an issue, and a team label helps you route your notifications.
Alert Advisor checks your Grafana-managed alert rules against a quality policy of required labels and annotations, so you can find and fix rules that are missing that context.
With Alert Advisor, you can:
Define which labels and annotations your Grafana-managed alert rules must have.
Find rules that are missing required labels or annotations and update them.
Optionally block supported provisioning writes that don't meet selected requirements.
Before you begin
Before you begin, check that you have the following permissions in your Grafana organization:
Rule access: Have permission to view the Grafana-managed alert rules you want to assess.
Policy permissions: To configure the quality policy, have the Admin role in your organization or a role with 
alert.rules.quality:write and 
alert.rules:read.
Rule updates: To fix findings, have permission to update the affected rules.
For provisioned rules, you also need access to their source configuration and permission to reapply it.
For more information about alerting permissions and folder access, refer to Configure RBAC.
Understand policy scope
A single quality policy applies to Grafana-managed alert rules across folders in your organization.
Data source-managed alert rules and recording rules are excluded.
Alert Advisor reports a finding when a required label or annotation is missing, empty, or contains only whitespace.
The checks do not validate the content itself, they only check for the presence of a particular field.
For example, requiring a runbook URL doesn't check whether the link works.
Detection and enforcement
You choose a mode for each requirement:
Mode
Behavior
Detect-only
Missing values appear as findings but don't block rule creation or updates.
Enforced
Missing values appear as findings and block rule creation or updates through supported provisioning paths.
Both modes contribute to the quality score.
Enabling enforcement doesn't stop existing rules from evaluating or sending notifications.
Supported enforcement paths
Enforcement applies to provisioned rules created or updated through Terraform and the Alerting provisioning HTTP API, with the following exceptions:
Provisioning API requests that set the 
X-Disable-Provenance header, including Terraform 
grafana_rule_group resources with 
disable_provenance = true, aren't enforced.
kubectl writes are enforced only when the manifest sets 
grafana.app/managedBy to 
kubectl or 
terraform together with a non-empty 
grafana.app/managerId, or sets 
grafana.com/provenance to 
api.
Writes from the Grafana rule editor aren't enforced, but the resulting rules can still have quality findings.
Configure a quality policy
Start with detect-only requirements to identify missing information before enabling enforcement.
The following example requires a runbook URL and a 
team label.
The Quality policy settings button is visible only if you have the policy permissions listed in Before you begin.
To configure a quality policy, follow these steps:
Go to Alerting > Alert rules and select Alert quality.
Click Quality policy settings (the cog icon) to open the Quality policy drawer.
Under Required annotations, turn on Runbook URL.
You can also require Summary and Description.
Under Required labels, enter 
team in Label keys and select the matching option.
You can select keys already used by your rules or add a new key.
Keep Enforce off for the requirements you add.
New requirements start in detect-only mode.
Click Save.
Grafana displays Policy saved to confirm the change.
The saved policy identifies rules with missing or empty 
runbook_url annotations or 
team labels.
A policy with no required annotations or labels checks nothing.
Review alert quality
In Alert Advisor, the Alert quality tab lists rules that don't meet your policy and shows an overall quality score.
To review the findings, follow these steps:
Go to Alerting > Alert rules and select Alert quality.
Review the Alert quality score and the number of rules that need attention.
Inspect the missing annotations and labels in the Findings list.
Each missing field appears as a badge. To check whether a requirement is enforced or detect-only, hover over its badge.
Understand the score
The score represents the proportion of assessed rules that meet every configured requirement, on a scale of 0 to 10.
For example, if 8 of 10 rules meet the policy, the score is 8.0.
Grafana rounds the score to one decimal place.
The score measures compliance with your policy, not alert accuracy, severity, or noise. Both detect-only and enforced requirements count toward the score.
Your permissions determine which rules you can view and assess.
Search filters change the findings' list, not the score or the total rule count.
Grafana also displays 10.0 when there are no rules to assess, so check the rule count alongside the score.
Resolve quality findings
Add meaningful values for the missing fields in each rule.
For the example policy, set 
runbook_url to the rule's runbook URL and 
team to the owning team's name.
Changing label values can affect notification routing.
Choose values that match your team's notification policies.
Update provisioned rules
For provisioned rules, make changes in the source configuration so they remain part of your provisioning workflow.
To resolve a finding on a provisioned rule, follow these steps:
Locate the rule in its source configuration, using the rule name, folder, and group shown in Alert quality.
Add values for the missing annotations and labels.
Reapply the configuration through your usual provisioning workflow.
Reload the Alert quality page and check the rule's findings.
A rule disappears from the findings list when it meets every configured requirement.
If other required fields are still missing, the rule remains in the list with those findings.
Update rules in Grafana
For rules that you can edit in Grafana, use the Edit action on the quality page.
To resolve a finding in the rule editor, follow these steps:
Click Edit for the affected rule.
Fill in the missing annotations and labels in the rule editor.
Click Save.
Return to Alert quality and reload the page to check the findings.
Optional: Enforce quality requirements
After reviewing and resolving findings, enable enforcement for the requirements you want to apply to supported provisioning writes.
You can enforce selected requirements while keeping others detect-only.
To enable enforcement, follow these steps:
On the Alert quality page, click Quality policy settings (the cog icon) to open the Quality policy drawer.
Turn on Enforce beside each requirement you want to enforce.
For example, enforce 
team while keeping Runbook URL detect-only.
To enforce all currently configured requirements, turn on Enforce all requirements instead.
Click Save.
Grafana displays Policy saved to confirm the change.
Disable enforcement or remove a requirement
To return a requirement to detect-only, turn off its Enforce control (or Enforce all requirements, if every requirement is enforced) and click Save.
The requirement remains in the policy and continues to affect the quality score.
To stop checking a field altogether, remove its requirement instead: turn off its switch under Required annotations, or remove its key from Label keys. Then click Save.
Troubleshoot quality findings and rejected writes
Use the following guidance when you can't access the feature, resolve a finding, or apply a rule change.
Symptom
Action
Alert quality or the Quality policy settings button isn't visible.
Ask your stack administrator to check your permissions and the feature's availability in your stack.
A rule has no Edit action.
If it has a Provisioned badge, update its source configuration. Otherwise, check your permission to update rules in that folder.
Grafana reports that no quality policy is configured.
Add at least one requirement in the Quality policy drawer and click Save. An empty policy doesn't assess any fields.
No findings match your search, but the score is below 10.0.
Clear Search to view all findings. Search filters don't change the score.
A finding remains after enforcement is disabled.
Detect-only requirements still produce findings. Fill in the missing value or remove the requirement from the policy.
A provisioning write is rejected by the quality policy.
Use the rule and field details in the error to update the source configuration, then reapply it.
A write succeeds despite a quality finding.
Check that the requirement is enforced and the policy is saved. Confirm that the write uses a supported enforcement path and isn't one of its exceptions.
Next steps
Use these guides to improve rule context, manage rules as code, and configure access:
Rule context: Labels and annotations.
Provisioning: Use Terraform to provision alerting resources.
API management: Use the HTTP API to manage alerting resources.
Permissions: Configure RBAC.

Import data source-managed rules to Grafana-managed rules
You can convert existing alert rules from data sources such as Mimir, Loki, and Prometheus into Grafana-managed alert rules. This enables you to operate and manage these rules using Grafana Alerting.
This guide explains two methods for importing data source-managed rules:
Using the Grafana Alerting user interface to import rules from connected data sources or Prometheus rule YAML files.
Using command-line tools like 
mimirtool and 
cortextool to import rules from YAML files.
Importing rules is a safe operation: the original data source–managed rules remain intact in their original location. During import, the rules are converted into Grafana-managed rules while preserving their configuration and behavior.
Choose the method that best fits your workflow. As a best practice, test and validate your import process before migrating all alert rules.
Import your notification configuration first
Rules only tell you about a problem if the notification reaches someone. Grafana-managed rules route through the Grafana Alertmanager, not through the Alertmanager your data source-managed rules used, so imported rules notify through whatever contact points and notification policies already exist in Grafana.
Consider importing your Alertmanager configuration before you import rules. Grafana imports the configuration as it is and runs it the way your Prometheus or Mimir Alertmanager does. You can then route the rules you import through the imported policy tree.
How it works
When you use any of the import methods, data source–managed rules are copied to another folder as Grafana-managed rules.
The original data source–managed rules remain intact in their original location.
The copied rules are converted to Grafana-managed rules, preserving their behavior by using equivalent Grafana-managed features. The following settings are applied during the conversion:
Unique UIDs
The newly created rules are assigned unique UIDs.
If you don’t want a UID to be auto-generated, you can specify one using the 
__grafana_alert_rule_uid__ label.
Query offset
A query offset is applied to each rule. For example, an offset of 
1m adjusts the query's time range to 
To: now-1m.
The rule query offset is taken from the 
query_offset value in the rule group configuration. If empty, it defaults to the 
rule_query_offset configuration setting, which is 
1m by default.
Rule query conversion
For alert rules, adds 
prometheus_math and 
threshold expressions to preserve Prometheus no data behavior, ensuring the alert stays in Normal state when 
query returns no data.
Missing series evaluations to resolve
The Missing series evaluations to resolve setting is set to 
1 to replicate Prometheus’s alert eviction behavior.
Rule group labels
Labels defined at the rule group level are added as labels to each imported rule within the group.
Sequential evaluation
Imported rules are evaluated sequentially within each rule group, mirroring Prometheus behavior. This differs from native Grafana-managed alert rules, where the evaluation order is not enforced. For more details, refer to evaluation strategies.
Feature compatibility
The rule group 
limit option and the 
query function within alert rule templates are not currently supported in Grafana-managed rules. If the 
limit option is present, the import fails. However, rules with 
query in templates are imported.
{{< admonition type="note" >}}
Rules with the label 
__grafana_origin are not included in rule imports. These rules are typically created by apps such as Kubernetes Monitoring, Synthetic Monitoring, and other Grafana plugins.
{{< /admonition >}}
Import rules with Grafana Alerting
You can use the Grafana Alerting user interface to import rules from the following sources:
Connected Mimir and Loki data sources with the ruler API enabled
Prometheus YAML rule files
Imported rules using this method are editable in the user interface. Like regular Grafana-managed rules, you can later export them for provisioning.
Before you begin
To use Grafana Alerting to migrate rules, you need the following RBAC permissions:
Alerting: 
Rules Writer, 
Set provisioning status.
Datasources: 
Reader.
Folders: 
Creator.
The Folders permission is optional and only necessary if you want to create new folders for your target namespace. If your account doesn't have permissions to view a namespace, the tool creates a new one.
To convert data source-managed alert rules to Grafana managed alerts:
Go to Alerting > Alert rules.
Navigate to the Data source-managed alert rules section and click Import to Grafana-managed rules.
Choose the Import source from which you want to import rules:
Select Existing data source-managed rules to import rules from connected Mimir or Loki data sources with the ruler API enabled.
Select Prometheus YAML file to import rules by uploading a Prometheus YAML rule file.
In the Data source dropdown, select the data source that the imported alert rules will query.
(Optional) In Additional settings, select a target folder or designate a new folder to import the rules into.
If you import the rules into an existing folder, don't choose a folder with existing alert rules, as they could get overwritten.
(Optional) Select a Namespace and/or Group to determine which rules are imported.
(Optional) Turn on Pause imported alerting rules.
Pausing stops alert rule evaluation and doesn’t create any alert instances for the newly created Grafana-managed alert rules.
(Optional) Turn on Pause imported recording rules.
Pausing stops alert rule evaluation behavior for the newly created Grafana-managed alert rules.
(Optional) In the Target data source of the Recording rules section, you can select the data source to which the imported recording rules will write metrics. By default, it is the data source selected in the Data source dropdown.
Click Import.
A preview shows the rules that will be imported. If your target folder contains folders with the same name of the imported folders, a warning displays to inform you. You can explore the warning to see a list of folders that might be overwritten.
Click Yes, import to import the rules.
Import rules with command-line tools
You can also use command-line tools to import data source-managed rules as Grafana-managed rules: use 
mimirtool for Mimir and Prometheus rules, and 
cortextool for Loki rules.
Both tools provide 
rules commands to import rules groups. For example:
rules load can import rule groups from files.
rules sync can read rule files and applies only the differences compared to existing Grafana-managed rules. This is useful for automation workflows and pipelines.
By default, rules imported using the API or command-line tools are Provisioned and not editable in the user interface. To make them editable, enable the X-Disable-Provenance header.
Before you begin
You need to have installed 
mimirtool or 
cortextool (version 
0.11.3 or later).
You need a service account with the following RBAC permissions:
Alerting: 
Rules Reader, 
Rules Writer, 
Set provisioning status.
Datasources: 
Reader.
Folders: 
Creator, 
Reader, 
Writer.
You need to a service account token with your service account. For more details, refer to service accounts and service account tokens.
mimirtool
To convert and import them into a Grafana instance, you can use the 
mimirtool rules load command:
MIMIR_ADDRESS=<GRAFANA_BASE_URL>/api/convert/ \
MIMIR_AUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN> \
MIMIR_TENANT_ID=1 \
mimirtool rules load rule_file.yaml \
  --extra-headers "X-Grafana-Alerting-Datasource-UID=<DATASOURCE_UID_QUERY_TARGET>"
This command imports Prometheus alert rules defined in 
rule_file.yaml as Grafana-managed alert rules. It's important to know that:
When using the 
<GRAFANA_BASE_URL>/api/convert/ endpoint, 
mimirtool interacts with Grafana—not with a Mimir instance. In this case, 
MIMIR_TENANT_ID must always be set to 
1.
The 
X-Grafana-Alerting-Datasource-UID header configures the data source that the imported alert rules will query. Use multiple 
--extra-headers flags to include other optional headers.
Similarly, the 
rules sync command can import and update Grafana-managed alert rules.
MIMIR_ADDRESS=<GRAFANA_BASE_URL>/api/convert/ \
MIMIR_AUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN> \
MIMIR_TENANT_ID=1 \
mimirtool rules sync rule_file.yaml \
  --extra-headers "X-Grafana-Alerting-Datasource-UID=<DATASOURCE_UID_QUERY_TARGET>" \
  --concurrency 1
The 
--concurrency flag must be set to 
1, as the default value of 
8 may cause API errors.
This 
sync command reads rules from the file, compares them with the existing Grafana-managed rules in the instance, and applies only the differences—creating, updating, or deleting rules as needed.
## Sync Summary: 0 Groups Created, 1 Groups Updated, 0 Groups Deleted
For more information other Mimirtool commands and options, see the Mimirtool documentation and the Mimir HTTP Rule API documentation.
cortextool
For Loki alert rules, use 
cortextool (version 
0.11.3 or later) with the 
--backend=loki flag. For example:
CORTEX_ADDRESS=<GRAFANA_BASE_URL>/api/convert/ \
CORTEX_AUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN> \
CORTEX_TENANT_ID=1 \
cortextool rules load loki_rules.yaml \
  --extra-headers "X-Grafana-Alerting-Datasource-UID=<LOKI_DATASOURCE_UID_QUERY_TARGET>" \
  --backend=loki
Optional Headers
Additional configuration headers for more granular import control include the following:
X-Disable-Provenance
When this header is set to 
true:
The imported rules are not marked as provisioned.
They can then be edited in the Grafana UI.
They are excluded from the 
GET and 
DELETE operations on the 
/api/convert endpoints.
Do not enable this header when using the 
rules sync command, as it relies on the 
GET and 
DELETE operations to detect and update existing rules.
X-Grafana-Alerting-Alert-Rules-Paused
Set to 
true to import alert rules in paused state.
X-Grafana-Alerting-Recording-Rules-Paused
Set to 
true to import recording rules in paused state.
X-Grafana-Alerting-Datasource-UID
The UID of the data source to use for alert rule queries.
If not specified in the header, Grafana uses the configured default from 
unified_alerting.prometheus_conversion.default_datasource_uid. If neither the header nor the configuration option is provided, the request fails.
X-Grafana-Alerting-Target-Datasource-UID
The UID of the target data source for recording rules. If not specified, the value from 
X-Grafana-Alerting-Datasource-UID is used.
X-Grafana-Alerting-Folder-UID
Enter the UID of the target destination folder for imported rules.
X-Grafana-Alerting-Notification-Settings
JSON-encoded 
AlertRuleNotificationSettings object that allows setting the contact point for the alert rules.
{{< collapse title="AlertRuleNotificationSettings object" >}}
AlertRuleNotificationSettings object
When you set 
X-Grafana-Alerting-Notification-Settings, the header value must be a JSON-encoded object with the following keys:
Field
Type
Required
Example
Description
receiver
string
Yes
"grafana-default"
Name of the contact point (receiver) to which alerts are routed. Must exist in Grafana before import.
group_by
[]string
No
["alertname","grafana_folder","cluster"]
Label set used by Alertmanager to aggregate alerts into a single notification.
group_wait
duration
No
"30s"
How long Alertmanager waits before sending the first notification for a new group.
group_interval
duration
No
"5m"
Time to wait before adding new alerts to an existing group's next notification.
repeat_interval
duration
No
"4h"
Minimum time before a previously-sent notification is repeated. Must not be less than 
group_interval.
mute_time_intervals
[]string
No
["maintenance"]
One or more mute time interval names that silence alerts during those windows.
active_time_intervals
[]string
No
["maintenance"]
List of active time interval names. Alerts are suppressed unless the current time matches one of them.
{{< /collapse >}}
Compatible endpoints
The API endpoints listed in this section are supported in Grafana and are used by 
mimirtool and 
cortextool, as shown earlier. These endpoints are compatible with Mimir HTTP API.
In these endpoints, a "namespace" corresponds to a folder title in Grafana.
The 
POST endpoints can be used to import data source–managed alert rules. They accept requests in both YAML and JSON. If no media type is specified, YAML is assumed.
Endpoint
Method
Summary
Mimir equivalent
POST
/convert/prometheus/config/v1/rules
Create or update multiple rule groups across multiple namespaces. Requires 
X-Grafana-Alerting-Datasource-UID.
None
POST
/convert/prometheus/config/v1/rules/:namespaceTitle
Create or update a single rule group in a namespace. Requires 
X-Grafana-Alerting-Datasource-UID.
Set rule group
The 
GET and 
DELETE endpoints work only with provisioned and imported alert rules. All 
GET endpoints support both JSON and YAML response formats based on the 
Accept header: use 
application/json for JSON responses, or 
application/yaml for YAML responses. YAML is the default format when no 
Accept header is specified.
Endpoint
Method
Summary
Mimir equivalent
GET
/convert/prometheus/config/v1/rules
Get all imported rule groups across all namespaces.
List rule groups
GET
/convert/prometheus/config/v1/rules/:namespaceTitle
Get imported rule groups in a specific namespace.
Get rule groups by namespace
GET
/convert/prometheus/config/v1/rules/:namespaceTitle/:group
Get imported rule group in a specific namespace.
Get rule group
DELETE
/convert/prometheus/config/v1/rules/:namespaceTitle
Delete all imported alert rules in a namespace.
Delete namespace
DELETE
/convert/prometheus/config/v1/rules/:namespaceTitle/:group
Delete a specific imported rule group.
Delete rule group
Create or update multiple rule groups
POST /convert/prometheus/config/v1/rules
Creates or updates multiple rule groups across multiple namespaces. This endpoint expects a request with a map of namespace titles to arrays of rule groups, and returns 
202 on success.
This endpoint has no Mimir equivalent and is Grafana-specific for bulk operations.
Example request body
namespace1:
  - name: MyGroupName1
    rules:
      - alert: MyAlertName1
        expr: up == 0
        labels:
          severity: warning
namespace2:
  - name: MyGroupName2
    rules:
      - alert: MyAlertName2
        expr: rate(http_requests_total[5m]) > 0.1
        labels:
          severity: critical

Configure data source-managed alert rules
{{< admonition type="caution" >}}
Pre-provisioned Loki and Prometheus data source-managed alerts have been deprecated in Grafana Cloud and can not be created in new stacks.
New Grafana Cloud stacks will use Grafana-managed alerting (GMA) by default. Datasource-managed alerting (DMA) is not provisioned in new stacks. Existing stacks are not affected.
This applies to the default Loki and Prometheus Grafana Cloud data sources managed by Grafana Labs and Cloud Alertmanager, which will not be available nor will Grafana provision the data source for it. If you add your own Mimir, Loki, or Alertmanager data sources, you can continue to use data source-managed alerts.
Cloud users can import DMA rules into GMA rules with the import tool. See the import data source-managed alerts to Grafana-managed alerts documentation for information on how to do this.
{{< /admonition >}}
Configure data source-managed alert rules
Data source-managed alert rules are alert rules that are stored in the data source, such as in Prometheus or Loki, rather than in Grafana.
In Grafana Alerting, you can:
Create and edit data source-managed rules for Grafana Mimir and Grafana Loki data sources.
View rules from Prometheus data sources when Manage alerts via Alerting UI is enabled. However, you cannot create or edit these rules in Grafana.
Import data source-managed rules from Loki, Mimir, and Prometheus into Grafana-managed rules.
{{< admonition type="note" >}}
Data source-managed rules are supported for horizontal scalability, but they can introduce more operational complexity than Grafana-managed alert rules.
We recommend using Grafana-managed alert rules whenever possible, as they provide a richer feature set and better integration with the full Grafana Alerting workflow.
{{< /admonition >}}
Comparison with Grafana-managed rules
The table below compares Grafana-managed and data source-managed alert rules.
Feature
Grafana-managed alert rule
Data source-managed alert rule
Supported data sources
All backend data sources enabling the 
alerting option
Only supports creating rules for Mimir and Loki data sources
Mix and match data sources
Yes
No
Add expressions to transform
 your data and set alert conditions
Yes
No
No data and error states
Yes
No
Stale alert instances
Yes
No
Images in alert notifications
Yes
No
Role-based access control
Yes
No
Alert state history
Yes
No
Alert version history
Yes
No
Terraform provisioning
Yes
No
Recording rules
Yes
Yes
Organization
Organize and manage access with folders
Use namespaces
Alert rule evaluation
Alert evaluation is done in Grafana
Alert rule evaluation is done in the data source and allow for horizontal scaling
Scaling
Alert rules are stored in the Grafana database.
Alert rules are stored within the data source and allow for horizontal scaling
The following diagram shows the architecture of a Mimir setup that uses data source-managed alert rules.
{{< figure src="/media/docs/alerting/mimir-managed-alerting-architecture-v2.png" max-width="750px" alt="Data source-managed alerting architecture based on Grafana Mimir" >}}
Create data source-managed alert rules
To create or edit data source-managed alert rules, follow these instructions.
Before you begin
Verify that you have write permission to the Mimir or Loki data source. Otherwise, you cannot create or update data source-managed alert rules.
Enable the Ruler API
For more information, refer to the Mimir Ruler API or Loki Ruler API.
Mimir - use the 
/prometheus prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the Query API and Ruler API are under the same URL. You cannot provide a separate URL for the Ruler API.
Loki - The 
local rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.
Permissions
Alert rules for Mimir or Loki instances can be edited or deleted by users with Editor or Admin roles.
If you do not want to manage alert rules for a particular data source, go to its settings and clear the Manage alerts via Alerting UI checkbox.
Set alert rule name
{{< docs/shared lookup="alerts/configure-alert-rule-name.md" source="grafana" version="<GRAFANA_VERSION>" >}}
Define query and condition
Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.
{{< admonition type="note" >}}
By default, new alert rules are Grafana-managed. To switch to Data source-managed, follow these instructions.
{{< /admonition >}}
Select a Prometheus-based data source from the drop-down list.
You can also click Open advanced data source picker to find more options.
Enter a PromQL or LogQL query, including the alert condition.
In the Rule type option, select Data source-managed.
Click Preview alerts.
Set alert evaluation behavior
Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.
Select a namespace or click + New namespace.
Select an evaluation group or click + New evaluation group.
If you are creating a new evaluation group, specify the interval for the group.
All rules within the same group are evaluated sequentially over the same time interval. You can reorder them from the Alert rules page.
Enter a pending period.
The pending period is the period in which an alert rule can be in breach of the condition until it fires.
After a condition is met, the alert goes into the Pending state. If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.
Configure labels and notifications
Add labels to your alert rules to set which notification policy should handle your firing alert instances.
All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.
Add labels if you want to change the way your notifications are routed.
Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
Configure notification message
Use annotations to add information to alert messages that can help respond to the alert.
Annotations are included by default in notification messages, and can use text or templates to display dynamic data from queries.
Grafana provides several optional annotations.
Optional: Add a summary.
Short summary of what happened and why.
Optional: Add a description.
Description of what the alert rule does.
Optional: Add a Runbook URL.
Webpage where you keep your runbook for the alert
Optional: Add a custom annotation.
Add any additional information that could help address the alert.
Optional: Link dashboard and panel.
Link the alert rule to a panel to facilitate alert investigation.
Click Save rule.

Configure Grafana-managed alert rules
{{< admonition type="caution" >}}
Pre-provisioned Loki and Prometheus data source-managed alerts have been deprecated in Grafana Cloud and can not be created in new stacks.
New Grafana Cloud stacks will use Grafana-managed alerting (GMA) by default. Datasource-managed alerting (DMA) is not provisioned in new stacks. Existing stacks are not affected.
This applies to the default Loki and Prometheus Grafana Cloud data sources managed by Grafana Labs and Cloud Alertmanager, which will not be available nor will Grafana provision the data source for it. If you add your own Mimir, Loki, or Alertmanager data sources, you can continue to use data source-managed alerts.
Cloud users can import DMA rules into GMA rules with the import tool. See the import data source-managed alerts to Grafana-managed alerts documentation for information on how to do this.
{{< /admonition >}}
Configure Grafana-managed alert rules
Grafana-managed alert rules are the default way to create alert rules in Grafana.
Grafana-managed rules inherit their model from Prometheus Alerting and extend it with greater flexibility—such as multi-data source queries, expression-based transformations, advanced alert conditions, images in notifications, custom states, and more.
To create or edit Grafana-managed alert rules, follow the instructions below.
{{< admonition type="tip" >}}
For quick-start tutorials on key alerting features, see Getting started with Grafana Alerting tutorials.
{{< /admonition  >}}
Before you begin
Before you create Grafana-managed alert rules, review the following requirements and options.
Supported data sources
Grafana-managed alert rules can query backend data sources when the data source's 
plugin.json file sets 
{"backend": true, "alerting": true}.
Before you create an alert rule, verify that the data sources you plan to query are compatible and properly configured.
You can find the public data sources that support alert rules in the Grafana Plugins directory.
Alert rule limits in Grafana Cloud
In Grafana Cloud, the number of Grafana-managed alert rules you can create depends on your Grafana Cloud plan.
Free Forever plan: You can create up to 500 free alert rules, with each alert rule having a maximum of 1000 alert instances.
All paid plans: They have a soft limit of 2000 alert rules and support unlimited alert instances. To increase the limit, open a support ticket from the Cloud portal.
Permissions
Only users with Edit permissions for the folder storing the rules can edit or delete Grafana-managed alert rules. Only admins can restore deleted Grafana-managed alert rules.
Provisioning
Note that if you delete an alert resource created in the UI, you can no longer retrieve it.
To backup and manage alert rules, you can provision alerting resources using options such as configuration files, Terraform, or the Alerting API.
Default versus Advanced options
Grafana Alerting provides multiple configuration options to support different alerting needs.
To simplify alert rule creation, some settings display a Default and an Advanced configuration view:
Define query and condition
Configure notifications
Use Default to create basic query conditions and route alerts directly to a contact point. Use Advanced for more complex query definitions, or to route alerts using notification policies.
You can switch between options at any time. After you create an alert rule, Grafana remembers your last selection and uses it as the default for the next rule.
Set alert rule name
{{< docs/shared lookup="alerts/configure-alert-rule-name.md" source="grafana" version="<GRAFANA_VERSION>" >}}
Define query and condition
Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.
Depending on the data source, the query editor offers a Builder and a Code option. Builder helps you construct a query with a visual interface. The Builder option is useful if you have limited experience with the query language, while Code lets you write the query directly for more control. Switch between them using the Builder and Code tabs on the query editor toolbar.
The Default option allows you to configure one query and one alert condition. The Advanced option allows you to use multiple queries and expressions for more complex rule definitions.
{{< collapse title="Default options" >}}
Select a data source.
Add a query.
Add an alert condition.
The When input includes the reducer function and the last input is the threshold.
Click Preview to verify.
{{< /collapse >}}
{{< collapse title="Advanced options" >}}
Select a data source.
From the Options drop-down menu, specify a time range.
Note that Grafana Alerting only supports fixed relative time ranges, for example, 
now-24hr: now.
It doesn't support absolute time ranges: 
2021-12-02 00:00:00 to 2021-12-05 23:59:592 or semi-relative time ranges: 
now/d to: now.
Add a query.
To add multiple queries, click Add query.
All alert rules are managed by Grafana by default. If you want to switch to a data source-managed alert rule, click Switch to data source-managed alert rule.
Add one or more expressions.
a. For each expression, select either Classic condition to create a single alert rule, or choose from the Math, Reduce, and Resample options to generate separate alert for each series.
When using Prometheus, you can use an instant vector and built-in functions, so you don't need to add additional expressions.
b. Click Preview to verify that the expression is successful.
To add a recovery threshold, enable the Custom recovery threshold toggle and enter a value that defines when the alert should recover—transition to 
Normal state from the 
Alerting or 
Pending state.
You can only add one recovery threshold, and it must be part of the alert condition.
Click Set as alert condition on the query or expression you want to set as your alert condition.
{{< /collapse >}}
You can switch between Default and Advanced options at any time.
Switching from Advanced to Default may result in queries and expressions that can't be converted. In this case, a warning message asks if you want to continue to reset to default settings.
If the Default versus Advanced options feature is not enabled in your Grafana instance, follow the Advanced options instructions.
Set folder and labels
Organize your alert rule with a folder and set of labels.
In the Labels section, you can optionally choose whether to add labels to organize your alert rules and their notifications. For more details, refer to alert rule labels.
Select a folder or click + New folder.
Add labels, if required.
Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
Configure alert evaluation behavior
Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.
To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.
Select an evaluation group or click + New evaluation group.
If you are creating a new evaluation group, specify the interval for the group.
All rules within the same group are evaluated concurrently over the same time interval.
Enter a pending period.
The pending period is the period in which an alert rule can be in breach of the condition until it fires.
After a condition is met, the alert goes into the Pending state.
If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.
Optionally, set the Keep firing for period.
You can set the minimum amount of time that an alert remains firing after the breached threshold expression no longer returns any results. This sets an alert to a "Recovering" state for the duration of time set here. The Recovering state can be used to reduce noise from flapping alerts. Select "none" to stop an alert from firing immediately after the breach threshold is cleared.
Turn on pause alert notifications, if required.
You can pause alert rule evaluation to prevent noisy alerting while tuning your alerts.
Pausing stops alert rule evaluation and doesn't create any alert instances.
This is different from mute timings, which stop notifications from being delivered, but still allow for alert rule evaluation and the creation of alert instances.
In Configure no data and error handling, you can define the alerting behavior and alerting state for two scenarios:
When the evaluation returns No data or all values are null.
When the evaluation returns Error or timeout.
Configure no data and error handling
Configure the behavior when the evaluation returns no data:
{{< docs/shared lookup="alerts/modify-nodata-state.md" source="grafana" version="<GRAFANA_VERSION>" >}}
You can also configure the behavior when the evaluation returns an error:
{{< docs/shared lookup="alerts/modify-error-state.md" source="grafana" version="<GRAFANA_VERSION>" >}}
For more details, refer to alert instance states and modify the no data or error state.
In Configure no data and error handling, you can also configure Missing series evaluations to resolve: how many consecutive evaluation intervals must pass without data before an alert instance is considered stale.
Configure notifications
Configure who receives notifications when the alert rule fires.
The Default option allows you to select a contact point to handle notifications for this alert rule. The Advanced option routes notifications through notification policies.
{{< collapse title="Default options" >}}
Select an existing contact point.
All notifications for this alert rule are sent to this contact point automatically and notification policies aren't used.
Optionally, configure additional notification settings like grouping or timing options.
{{< /collapse >}}
{{< collapse title="Advanced options" >}}
With this option, all notifications for this alert rule are managed by the notification policy tree, which routes alerts based on their labels.
Click Change to change the routing from the default policy to another policy.
You can preview which notification policy would handle notifications from this alert rule.
Click Preview routing after configuring the previous alert rule settings.
Based on the alert labels and query labels, Grafana displays the notification policies that would handle the notifications.
Click View route to view policy and routing details.
{{< /collapse >}}
You can switch between Default and Advanced options at any time.
If the Default versus Advanced options feature is not enabled in your Grafana instance, follow the Advanced options instructions.
Configure notification message
Use annotations to add information to alert messages that can help respond to the alert.
Annotations are included by default in notification messages, and can use text or templates to display dynamic data from queries.
Grafana provides several optional annotations.
Optional: Add a summary.
Short summary of what happened and why.
Optional: Add a description.
Description of what the alert rule does.
Optional: Add a Runbook URL.
Webpage where you keep your runbook for the alert
Optional: Add a custom annotation.
Add any additional information that could help address the alert.
Optional: Link dashboard and panel.
Link the alert rule to a panel to facilitate alert investigation.
Click Save rule.

Create and link alert rules to panels
Grafana allows you to link an alert rule to a dashboard panel. This can help you:
Inform alert responders about where to investigate and which data to examine.
Visualize the alert state directly from dashboards.
Include a screenshot of the panel in notification messages.
An alert rule is linked to a panel by setting the 
__dashboardUid__ and 
__panelId__ annotations. Both annotations must be set together.
Link alert rules to panels
When configuring the alert rule, you can set the dashboard and panel annotations as shown in this video.
Configure the alert rule.
In the Configure notification message section, click Link dashboard and panel.
Select an existing dashboard, then choose a panel from the selected dashboard.
Complete the alert rule configuration and click Save rule to initiate the alert rule.
You can then view the alert state on the panel.
By default, notification messages include a link to the dashboard panel. Additionally, you can enable displaying panel screenshots in notifications.
{{< figure src="/media/docs/alerting/panel-displays-alert-state.png" max-width="1200px" caption="A panel displaying the alert status and state changes." >}}
Create alert rules from panels
{{< shared id="create-alert-from-panel" >}}
To streamline alert creation, you can create an alert rule directly from a panel.
Navigate to a dashboard in the Dashboards section.
Hover over the top-right corner of a panel and click the panel menu icon.
From the dropdown menu, select More... > New alert rule.
This opens the Edit rule form and pre-fills some values:
Sets the annotations to the corresponding dashboard and panel.
Sets the alert rule query using the panel query.
Complete the alert rule configuration and click Save rule to initiate the alert rule.
{{< /shared >}}
You can then view the alert state on the panel.
By default, notification messages include a link to the dashboard panel. Additionally, you can enable displaying panel screenshots in notifications.
{{< admonition type="note" >}}
Changes to panel and alert rule queries aren't synchronized. If you change a query, you have to update it in both the panel and the alert rule.
{{< /admonition >}}
Access linked alert rules from panels
This option is available only in time series panels. To access alert rules associated to a time series panel, complete the following steps.
Hover over the top-right corner of the panel and click the panel menu icon.
Click Edit.
Click the Alert tab to view existing alert rules or create a new one.
{{< admonition type="tip" >}}
For a practical example, refer to our Getting started: Link alerts to visualizations tutorial.
{{< /admonition >}}

Configure rule sequences
{{< docs/public-preview product="Rule sequences" >}}
Configure rule sequences
Use rule sequences in Grafana Alerting when a recording rule or alert rule depends on the results of another recording rule. A rule sequence references existing Grafana-managed rules and defines their evaluation order and shared evaluation interval.
You can create and manage rule sequences through the Grafana Alerting API or Terraform. The API uses 
v0alpha1 and is subject to change.
Before you begin
Before you create a sequence, ensure you have the following:
Grafana version: Use Grafana Cloud or Grafana OSS or Enterprise 13.1 or later.
Rules: Create at least one Grafana-managed recording rule and any dependent recording rules or alert rules. Place all the rules in the same folder.
Recording rule storage: Ensure your recording rules write to a Prometheus-compatible data source. In Grafana Cloud, rules without an explicit target write to your managed Prometheus data source by default. In self-managed Grafana, configure a target data source or a default target. For details, refer to Create Grafana-managed recording rules.
API access: Use an authenticated account with permission to manage alert rules in the folder. For authentication options, refer to Authentication options for the HTTP API.
Understand evaluation order
For each evaluation interval, Grafana evaluates the rules in two stages:
Evaluate the recording rules in the order listed in 
spec.recordingRules.
Evaluate the alert rules in the order listed in 
spec.alertingRules, after the recording rules.
For example, you can evaluate a recording rule that calculates a request rate, then a recording rule that uses that metric to calculate an error ratio, and then an alert rule that queries the error ratio.
The sequence's 
spec.trigger.interval sets the evaluation interval for every rule in the sequence, overriding each rule's individual interval while it belongs to the sequence. You continue to manage each rule's queries, expressions, and other settings separately.
Rules outside the sequence retain their existing evaluation behavior. A sequence defines evaluation order, not a general-purpose organizational group.
Evaluation order doesn't guarantee that a preceding rule produces new data. Grafana continues to the next rule even if a recording rule fails or returns no data. A dependent rule can query older data or receive no data, depending on its query and the target data source. Configure its error and no-data handling accordingly.
Even when a recording rule succeeds, its samples might not be available to query immediately. Grafana waits for the write request to complete but doesn't check whether the target data source can return the samples before it evaluates the next rule. A sequence guarantees evaluation order, not read-after-write consistency.
Create a sequence with the API
Create a 
RuleSequence resource that references the rules by name. For alert rules and recording rules, use each resource's 
metadata.name, which corresponds to its rule UID.
Send a 
POST request to 
/apis/rules.alerting.grafana.app/v0alpha1/namespaces/{namespace}/rulesequences with the 
Content-Type: application/json header and the following body:
{
  "apiVersion": "rules.alerting.grafana.app/v0alpha1",
  "kind": "RuleSequence",
  "metadata": {
    "name": "request-error-sequence",
    "namespace": "<NAMESPACE>",
    "annotations": {
      "grafana.app/folder": "<FOLDER_UID>"
    }
  },
  "spec": {
    "trigger": {
      "interval": "1m"
    },
    "recordingRules": [{ "name": "<REQUEST_RATE_RULE_UID>" }, { "name": "<ERROR_RATIO_RULE_UID>" }],
    "alertingRules": [{ "name": "<ALERT_RULE_UID>" }]
  }
}
Replace the placeholders with the following values:
<NAMESPACE>: Use your Grafana instance's namespace in both the request path and body. For namespace formats, refer to API structure in Grafana.
<FOLDER_UID>: Use the UID of the folder that contains all referenced rules.
<REQUEST_RATE_RULE_UID>: Use the UID of the recording rule that calculates the request rate.
<ERROR_RATIO_RULE_UID>: Use the UID of the recording rule that queries the request-rate metric and calculates the error ratio.
<ALERT_RULE_UID>: Use the UID of the alert rule that queries the error-ratio metric.
The example evaluates the three rules in that order every minute. Configure the dependent rules to query the metrics produced by the preceding recording rules. Adding a rule to a sequence doesn't change its query or automatically connect it to another rule's output.
To retrieve the sequence, send a 
GET request to 
/apis/rules.alerting.grafana.app/v0alpha1/namespaces/{namespace}/rulesequences/{name}. Use 
request-error-sequence as 
{name} for this example.
Create a sequence with Terraform
Use the 
grafana_apps_rules_rulesequence_v0alpha1 resource to manage a sequence in Terraform. Set 
metadata.folder_uid to the folder that contains the rules, and reference their UIDs in 
spec.recording_rules and 
spec.alerting_rules.
For a complete example and resource fields, refer to grafana_apps_rules_rulesequence_v0alpha1.
Update or remove a sequence
To change the order, membership, or interval through the API, retrieve the current sequence and send the updated resource with a 
PUT request to 
/apis/rules.alerting.grafana.app/v0alpha1/namespaces/{namespace}/rulesequences/{name}. Preserve its 
metadata.resourceVersion from the response.
To remove a sequence, send a 
DELETE request to 
/apis/rules.alerting.grafana.app/v0alpha1/namespaces/{namespace}/rulesequences/{name}. Deleting a sequence doesn't delete the rules it references.
After Grafana picks up the change, rules removed from a sequence return to their stored evaluation interval and group settings.
If you manage the sequence with Terraform, update or remove it in your Terraform configuration instead.
Sequence constraints
When you create or update a sequence, follow these constraints:
Recording rules: Include at least one recording rule. Alert rules are optional.
Folder: Place the sequence and all referenced rules in the same folder.
Membership: Reference each rule only once. A rule can belong to only one sequence.
Rule changes: Remove a rule from its sequence before you delete the rule or move it to another folder.
Interval: Use a positive interval that's an integer multiple of your instance's base evaluation interval. For example, with a base interval of 
10s, you can use 
10s, 
30s, or 
1m, but not 
15s. The API rejects invalid intervals instead of adjusting them.
Self-managed evaluation jitter: In Grafana OSS and Enterprise, keep the 
jitterAlertRulesWithinGroups feature toggle disabled when you use rule sequences. This toggle disables sequential evaluation and is disabled by default.
Related resources
For more information about recording rules and evaluation behavior, refer to the following resources:
Create Grafana-managed recording rules
How rules are evaluated within a group

Create data source-managed recording rules
Recording rules allow you to periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.
Alert rules and dashboards can then query the new metric resulting from the recording rule. This is faster than querying real-time data and can help to reduce system load.
Data source-managed recording rules can query Prometheus-based data sources like Mimir or Loki. For more information on recording rules in Prometheus, refer to Defining recording rules in Prometheus.
Note that in data source-managed groups, the alert rules and recording rules within the same evaluation group are evaluated sequentially. This is useful to ensure that a recording rule is evaluated before any other alert rule queries the pre-computed metric.
Before you begin
Verify that you have write permission to the Prometheus or Loki data source. Otherwise, you will not be able to create or update Grafana Mimir managed alerting rules.
For Grafana Mimir and Loki data sources, enable the ruler API by configuring their respective services.
Loki - The 
local rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.
Mimir - use the 
/prometheus prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the Query API and Ruler API are under the same URL. You cannot provide a separate URL for the Ruler API.
Add new recording rule
To create a new data source-managed recording rule:
Click Alerts & IRM -> Alerting -> Alert rules.
At the top of the Alert rules page, click More -> New Data source recording rule.
Enter recording rule name
The recording rule name must be a Prometheus metric name and contain no whitespace.
Define recording rule
Select your data source and enter a query. The queries used in data source-managed recording rules always run as instant queries.
Add namespace and group
From the Namespace dropdown, select an existing rule namespace or add a new one.
Namespaces can contain one or more rule groups and only have an organizational purpose.
From the Group dropdown, select an existing group within the selected namespace or add a new one.
Rules within a group are run sequentially at a regular interval, with the same evaluation time.
Newly created rules are appended to the end of the group, and you can reorder them from the Alert rules page.
Add labels
Optionally, you can add custom labels to the resulting metric by selecting existing key-value pairs from the drop down or entering the new key or value.
Query the new metric in dashboards or alert rules
Click Save rule or Save rule and exit to save the rule.
Once saved, the new recording metric is available for use in dashboards and alert rules.

Create Grafana-managed recording rules
Recording rules allow you to periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.
Alert rules and dashboards can then query the new metric resulting from the recording rule. This is faster than querying real-time data and can help to reduce system load.
Grafana does not contain an embedded time-series database to store recording rule results. You must bring your own Prometheus-compatible database to store the series generated by recording rules.
Grafana-managed recording rules offer the same Prometheus-like semantics but allow you to query data sources supported by alerting. Additionally, you can use recording rules to import and map data from other data sources into Prometheus.
In Grafana Cloud, recording rules support writing to Prometheus-compatible data sources configured with Private Data Source Connect (PDC). This allows you to write recording rule results to a Prometheus or Mimir instance in a private network without opening that network to inbound traffic from Grafana Cloud.
{{< admonition type="note" >}}
Grafana-managed recording rules are enabled by default in Grafana Cloud.
In Grafana OSS and Enterprise, you must enable them by following the Before you begin instructions.
{{< /admonition >}}
To configure Grafana-managed recording rules, complete the following steps.
Before you begin
This section only applies to Grafana OSS and Grafana Enterprise.
If a rule does not explicitly specify a target data source for writing (for example, if it was created before Grafana 12.1), Grafana will fall back to the 
default_datasource_uid set in the configuration:
[recording_rules]
default_datasource_uid = my-uid
{{< admonition type="note" >}}
Grafana Cloud: If you leave Target data source blank when creating a recording rule, Grafana writes the results to your managed Prometheus data source named 
grafanacloud-prom by default. This may be different from the default data source you use in dashboards. To write to a different backend, explicitly select a target data source. In self-managed Grafana, you can set the default fallback with 
default_datasource_uid in the 
[recording_rules] section of the configuration.
{{< /admonition >}}
If you previously configured recording rules using the 
url and 
basic_auth_* configuration options, these are no longer supported. You must either:
Set 
default_datasource_uid in the 
[recording_rules] section of the configuration file to point to the target data source
Or, before upgrading to Grafana 12.1, enable the 
grafanaManagedRecordingRulesDatasources feature flag and update each recording rule individually to include a target data source
Add new recording rule
To create a new Grafana-managed recording rule:
Click Alerts & IRM -> Alerting ->
Alert rules.
At the top of the Alert rules page, click More -> New Grafana recording rule.
Enter the names to identify your recording rule and metric.
The metric name must be a Prometheus metric name and contain no whitespace. For details, refer to Prometheus metric names.
Define recording rule
Define a query to get the data you want to measure and set the recording rule output.
Select a data source.
From the Options dropdown, specify a time range.
{{< admonition type="note" >}}
Grafana Alerting only supports fixed relative time ranges, for example, 
now-24hr: now.
It does not support absolute time ranges: 
2021-12-02 00:00:00 to 2021-12-05 23:59:592 or semi-relative time ranges: 
now/d to: now.
{{< /admonition >}}
Add a query.
To add multiple queries, click Add query.
Add one or more expressions.
a. For each expression, select either Classic condition to create a single recording rule, or choose from the Math, Reduce, and Resample options.
When using Prometheus, you can use an instant vector and built-in functions, so you don't need to add additional expressions.
b. Click Preview to verify that the expression is successful.
Click Set as recording rule output on the query or expression you want to set as your rule output.
Set evaluation behavior
Use recording rule evaluation to determine how frequently a recording rule should be evaluated.
To do this, you need to make sure that your recording rule is in the right evaluation group with an interval that works best for your use case.
Select a folder or click + New folder.
Select an evaluation group or click + New evaluation group.
If you are creating a new evaluation group, specify the interval for the group.
All rules within the same group are evaluated concurrently over the same time interval. Every recording rule in a group uses the same evaluation time, meaning that all queries from the same group are always aligned with each other.
If another recording rule or alert rule depends on this rule's results, you can define an explicit evaluation order through the API or Terraform. For instructions, refer to Configure rule sequences.
Before or after creating a recording rule, you have the option to Pause evaluation if necessary.
Advanced configuration
min_interval sets the minimum interval to enforce between rule evaluations. The default value is 10s which equals the scheduler interval. Rules are adjusted if they are less than this value or if they are not multiple of the scheduler interval (10s). Higher values can help with resource management as fewer evaluations are scheduled over time.
This setting has precedence over each individual rule frequency. If a rule frequency is lower than this value, then this value is enforced.
This setting applies to both Grafana-managed alert and recording rules.
Add labels
Optionally, you can add custom labels to the resulting metric by selecting existing key-value pairs from the drop down or entering the new key or value.
Query the new metric in dashboards or alert rules
Click Save rule or Save rule and exit to save the rule.
Once saved, the new recording metric is available for use in dashboards and alert rules.

Create Grafana-managed recording rules for Tempo
{{< docs/private-preview product="Grafana-managed recording rules for Tempo" >}}
Grafana-managed recording rules let you pre-compute query results and store them as new time series metrics. These metrics can then be reused in dashboards, alerts, or downstream queries, improving performance and reliability.
This guide explains how to configure recording rules for the Tempo data source, including product-specific considerations, such as TraceQL metrics queries, time range alignment, and evaluation delays.
For additional information, refer to Grafana-managed recording rules.
Create a new Tempo recording rule
In Grafana, go to Alerts & IRM > Alerting > Alert rules.
From the More+ menu, select New Grafana recording rule.
Enter recording rule and metric name.
In the New recording rule window, configure the following:
Name: A human-readable identifier for the recording rule.
Metric: The Prometheus-compatible name of the new metric series that will be generated. For details, refer to Prometheus metric names.
Target data source: Must be a Prometheus data source with write permissions (to store the rule's results). Ensure your Prometheus or Mimir metrics backend allows Grafana to write the new series.
Define the recording rule.
Data source: Select the Tempo data source with read permissions.
In the Options dropdown, specify a time range.
{{< admonition type="note" >}}
Grafana Alerting only supports fixed relative time ranges, for example, 
now-5m: now-4m.
It does not support absolute time ranges: 
2021-12-02 00:00:00 to 2021-12-05 23:59:592 or semi-relative time ranges: 
now/d to: now.
{{< /admonition >}}
TraceQL metrics queries are executed as instant queries over a window of time.
The range you select defines the aggregation window. For example use 
now-5m: now-4m for a 1-minute recording-rule interval with a 4-minute delay. The TraceQL query will run over the selected interval.
Always include a delay of a few minutes (for example, 2–5 minutes) to account for traces still in flight. This avoids missing late-arriving spans.
Max data points and Interval are not relevant for TraceQL instant queries.
Query: Enter a valid TraceQL metrics query.
{ resource.service.name = "checkout" }  | count_over_time()
Expressions: Typically use Reduce > Last to take the most recent computed value.
{{< figure src="/media/docs/grafana/alerting/screenshot-recording-rule-tempo-definition-private-preview.png" alt="Defining the recording rule for Tempo." >}}
Organize the rule by selecting a folder and adding labels.
Labels can help group or filter rules later.
Folders provide UI organization and RBAC scoping.
Set evaluation behavior.
Recording rules are executed by evaluation groups at fixed intervals. Align this interval with the query's time range.
{{< admonition type="caution" >}}
The Evaluation interval of the evaluation group should equal the size of the query window. For example, if your query covers 
now-5m: now-4m (1-minute window with 4-minute delay), use a 1-minute evaluation interval. This ensures each evaluation produces a new, non-overlapping sample.
{{< /admonition >}}
{{< figure src="/media/docs/grafana/alerting/screenshot-recording-rule-tempo-evaluation-group-private-preview.png" alt="Defining the recording rule evaluation group for Tempo." max-width="400px" >}}
Best practices
Always include a delay of at least 2 minutes in the query range to avoid losing in-flight spans.
Align the evaluation interval with your query window size.
The time series generated by a recording rule will be time-shifted based on the selected query time range. To align it back with the original timeline, you can apply the 
offset modifier in your Prometheus queries.

Migrate recorded queries
Users can transpose their now-deprecated recorded queries into Grafana-managed recording rules in a few easy steps. The query PromQL for each recorded query has been exposed on the recorded queries list along with the existing data source, time range, and interval values to simplify the migration process.
Migrate your recorded queries to Grafana-managed alert rules
Navigate to Administration -> Plugins and Data -> Recorded queries.
Note the data source, query PromQL, interval, and time range, and copy them somewhere accessible.
{{< figure alt="Example of relevant recorded query information"  src="/media/docs/alerting/rec-query-example.png" max-width="800px" >}}
Now navigate to Alerting -> Alert rules.
At the top of the Alert rules page, click More -> New Grafana recording rule.
Add a name for your Recording Rule and a name for the new metric that the recording rule generates.
Select your data source and paste your ratio query PromQL into the query builder.
Click Options and validate that the Time Range is the same as your recorded query.
Select the Folder you would like the rule to be created in.
Add any labels to the rule.
Select or create an evaluation group. Set your evaluation group’s evaluation interval to the interval of your recorded query.
Review your rule and click Save rule and exit when you are finished.
Update any areas where the recorded query was referenced to use the new Grafana-managed recording rule.

Labels and annotations template examples
Templating allows you to add dynamic data from queries to alert labels and annotations. Dynamic data enhances alert context, making it easier for responders to quickly assess and address the issue.
This page provides common examples for templating labels and annotations. For more information on templating, refer to:
Template annotations and labels
Annotation and label template reference
Alerting template language
Annotation example
Annotations add extra details to alert instances and are often used to provide helpful information for identifying the issue and guiding the response.
A common use case for annotations is to display the specific query value or threshold that triggered the alert.
For example, you can display the query value from the 
$values variable to inform about the CPU value that triggered the alert.
CPU usage has exceeded 80% ({{ $values.A.value }}) for the last 5 minutes.
Alternatively, you can use the 
index() function to retrieve the query value as follows.
CPU usage has exceeded 80% ({{ index $values "A" }}) for the last 5 minutes.
CPU usage has exceeded 80% (81.2345) for the last 5 minutes.
Include labels for extra details
To provide additional context, you can include labels from the query using the 
$labels variable.
For instance, the previous case could also include the affected 
instance.
CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) for the last 5 minutes.
CPU usage for Instance 1 has exceeded 80% (81.2345) for the last 5 minutes.
You can incorporate any labels returned by the query into the template. For instance, the following template includes information about the environment and region where the alert occurred.
Alert triggered in {{ $labels.environment }} on {{ $labels.region }} region.
Alert triggered in production on AMER region.
Print a range query
To print the value of an instant query you can print its Ref ID using the 
index function or the 
$values variable:
{{ $values.A.Value }}
For range queries, reduce them from a time series to an instant vector using a reduce expression. You can then print the result by referencing its Ref ID. For example, if the reduce expression averages 
A with the Ref ID 
B, you would then print 
$values.B:
{{ $values.B.Value }}
Humanize the value of a query
To print the humanized value of an instant query, use the 
humanize function:
{{ humanize $values.A.Value }}
Alternatively:
{{ humanize (index $values "A").Value }}
554.9
To print the value of an instant query as a percentage, use the 
humanizePercentage function:
{{ humanizePercentage $values.A.Value }}
10%
For additional functions to display or format data, refer to:
Annotation and label template functions
Template language functions
Label example
Labels determine how alerts are routed and managed, ensuring that notifications reach the right teams at the right time. If the labels returned by your queries don’t fully capture the necessary context, you can create a new label and sets its value based on query data.
Based on query value
Here’s an example of creating a 
severity label based on a query value:
{{- if (gt $values.A.Value 90.0) -}}critical
{{- else if (gt $values.A.Value 80.0) -}}high
{{- else if (gt $values.A.Value 60.0) -}}medium
{{- else -}}low
{{- end -}}
In this example, the 
severity label is determined by the query value:
critical for values above 90,
high for values above 80,
medium for values above 60,
and 
low for anything below.
You can then use the 
severity label to control how alerts are handled. For instance, you could send 
critical alerts immediately, while routing 
low severity alerts to a team for further investigation.
Note: An alert instance is uniquely identified by its set of labels.
Avoid displaying query values in labels, as this can create numerous alert instances—one for each distinct label set. Instead, use annotations for query values.
If a templated label's value changes, it maps to a different alert instance, and the previous instance is considered stale. Learn all the details in this example using dynamic labels.
Based on query label
You can use labels to differentiate alerts coming from various environments (e.g., production, staging, dev). For example, you may want to add a label that sets the environment based on the instance’s label. Here’s how you can template it:
{{- if eq $labels.instance "prod-server-1" -}}
production
{{- else if eq $labels.instance "staging-server-1" -}}
staging
{{- else -}}
development
{{- end -}}
This would print:
For instance 
prod-server-1, the label would be 
production.
For 
staging-server-1, the label would be 
staging.
All other instances would be labeled 
development.
To make this template more flexible, you can use a regular expression that matches the instance name with the instance name prefix using the 
match() function:
{{- if match "^prod-server-.*" $labels.instance -}}
production
{{- else if match "^staging-server-.*" $labels.instance -}}
staging
{{- else -}}
development
{{- end -}}
{{< collapse title="Legacy Alerting templates" >}}
Legacy Alerting templates
For users working with Grafana's legacy alerting system, templates can still be utilized to extract useful information from alert conditions. However, it's important to note that you cannot use 
$labels to print labels from the query if you are using classic conditions, and must use 
$values instead. The reason for this is classic conditions discard these labels to enforce uni-dimensional behavior (at most one alert per alert rule). If classic conditions didn't discard these labels, then queries that returned many time series would cause alerts to flap between firing and resolved constantly as the labels would change every time the alert rule was evaluated.
Instead, the 
$values variable contains the reduced values of all time series for all conditions that are firing. For example, if you have an alert rule with a query A that returns two time series, and a classic condition B with two conditions, then 
$values would contain 
B0, 
B1, 
B2 and 
B3. If the classic condition B had just one condition, then 
$values would contain just 
B0 and 
B1.
Print all labels from a classic condition
To print all labels of all firing time series use the following template (make sure to replace 
B in the regular expression with the Ref ID of the classic condition if it's different):
{{ range $k, $v := $values -}}
{{ if (match "B[0-9]+" $k) -}}
{{ $k }}: {{ $v.Labels }}{{ end }}
{{ end }}
For example, a classic condition for two time series exceeding a single condition would print:
B0: instance=server1
B1: instance=server2
If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then its labels will be duplicated for each condition that is exceeded:
B0: instance=server1
B1: instance=server2
B2: instance=server1
B3: instance=server2
If you need to print unique labels you should consider changing your alert rules from uni-dimensional to multi-dimensional instead. You can do this by replacing your classic condition with reduce and math expressions.
Print all values from a classic condition
To print all values from a classic condition take the previous example and replace 
$v.Labels with 
$v.Value:
{{ range $k, $v := $values -}}
{{ if (match "B[0-9]+" $k) -}}
{{ $k }}: {{ $v.Value }}{{ end }}
{{ end }}
For example, a classic condition for two time series exceeding a single condition would print:
B0: 81.2345
B1: 84.5678
If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then 
$values will contain the values of all conditions:
B0: 81.2345
B1: 92.3456
B2: 84.5678
B3: 95.6789
{{< /collapse >}}

Alerting template language
Notification templates and alert rule templates, such as annotations and labels, both use the Go template language, text/template.
Both types of templates can use the same keywords, functions, and comparison operators of the Go template language, such as 
range, 
if, 
and, 
index, 
eq, and more.
However, it's important to note that because notifications and alert rules operate in distinct contexts, some additional variables and functions are only available for either notification or alert rule templates. Refer to:
Annotation and label template reference
Notification template reference
This documentation provides an overview of the functions and operators of the Go template language that are available for both notification and alert rule templates.
Print
To print the value of something, use 
{{ and 
}}. You can print the value of a variable, a field of a variable, the result of a function, or the value of dot.
{{ $values }}
{{ $values.A.Value }}
{{ humanize 1000.0 }}
{{ .Alerts }}
Dot
In 
text/template, there is a special cursor called dot, written as 
.. You can think of this cursor as a variable whose value changes depending on where in the template it is used.
At the start of notification templates, dot (
.) refers to Notification Data.
{{ .Alerts }}
In annotation and label templates, dot (
.) is initialized with all alert data. It’s recommended to use the 
$labels and 
$values variables instead to directly access the alert labels and query values.
{{< admonition type="note" >}}
Dot (
.) might refer to something else when used in a range, a with, or when writing templates used in other templates.
{{< /admonition >}}
{{< docs/shared lookup="alerts/template-language.md" source="grafana" version="<GRAFANA_VERSION>" >}}

Annotation and label template reference
Annotations and labels in alert rules can be defined using plain text. However, you can also define templates to customize their values with dynamic data from alert rule queries.
For example, you can template the 
summary annotation to include information from query values, providing relevant alert context for responders. Refer to Template annotations and labels for various use cases.
In templates, variables represent dynamic values from queries, while functions perform actions to transform or format this data.
Variables
Variables represent dynamic values from alert rule queries that can be displayed or accessed in your templates.
The 
$ and 
. symbols are used to reference variables and their properties. You can reference variables directly in your alert rule definitions using the 
$ symbol followed by the variable name. Similarly, you can access properties of variables using the dot (
.) notation in alert rule templates.
{{ $values.A.Value }}
Templates are based on the Go templating system. Refer to Template language for additional information.
The following variables are available when templating annotations and labels:
Variables
Description
$labels
Contains all labels from the query, only query labels.
$values
Contains the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs.
$value
A string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule. When a single data source is used, it returns the value of the query. It is generally recommended to use $values.
$labels
The 
$labels variable contains all labels from the query. It excludes user-configured and reserved labels, containing only query labels.
{{< figure src="/media/docs/alerting/query-labels-and-values.png" max-width="1200px" caption="An alert rule displaying labels and value from a query." >}}
For example, suppose you have a query that returns CPU usage for all of your servers, and you have an alert rule that fires when any of your servers have exceeded 80% CPU usage for the last 5 minutes. You want to add a summary annotation to the alert that tells you which server is experiencing high CPU usage. With the 
$labels variable you can write a template that prints a human-readable sentence such as:
CPU usage for {{ $labels.instance }} has exceeded 80% for the last 5 minutes
The outcome of this template would be:
CPU usage for server1 has exceeded 80% for the last 5 minutes
If you are using a classic condition then 
$labels will not contain any labels from the query. Classic conditions discard these labels to enforce uni-dimensional behavior (at most one alert per alert rule). If you want to use labels from the query in your template then use the example here.
$values
The 
$values variable is a table containing the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs (e.g, 
A, 
B, 
C, etc.). It does not contain the results of range queries, as they can return hundreds or thousands of rows.
Each Ref IDs, such as 
$values.A, has the following properties
Property
Type
Description
Value
Float
The value returned by the instant query or expression.
Labels
Key/value pairs
The labels associated with the instant query or expression.
To print labels from a specific query or expression, use its Ref ID and the 
Labels property:
{{ $values.A.Labels.instance }}
$values is indexed by Ref ID (
A, 
B, 
C), not by the alert name.
For most alert messages, use 
$labels instead. 
$labels contains the query labels for the alert instance, so you don't need the Ref ID:
{{ $labels.instance }}
Here's the previous example printing now the value of the instant query with Ref ID 
A:
{{ $values.A.Value }} CPU usage for {{ $labels.instance }} over the last 5 minutes.
If the alert has the label 
instance=server1 and the query returns 
81.2345, the template would print:
81.2345 CPU usage for instance1 over the last 5 minutes.
If the query in Ref ID 
A is a range query rather than an instant query then add a reduce expression with Ref ID 
B and replace 
$values.A.Value with 
$values.B.Value:
{{ $values.B.Value }} CPU usage for {{ $labels.instance }} over the last 5 minutes.
Alternatively, you can use the 
index() function to retrieve the query value:
{{ index $values "B" }} CPU usage for {{ index $labels "instance" }} over the last 5 minutes.
{{< admonition type="note" >}}
Variable names that start with a number (for example, 
1B) are not valid identifiers in Go templates.
To access a value or label whose key starts with a number, use the 
index function:
{{ index $values "1B" }} CPU usage for {{ index $labels "1instance" }} over the last 5 minutes.
Using 
{{ $values.1B.Value }} is invalid and causes the template code to render as plain text.
{{< /admonition  >}}
$value
The 
$value variable is a string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule.
When a single data source is used in the alert rule, 
$value will return the query value directly.
This example prints the 
$value variable:
{{ $value }}: CPU usage has exceeded 80% for the last 5 minutes.
When using multiple data sources, it would display something like this:
[ var='A' labels={instance=instance1} value=81.234, , [ var='B' labels={instance=instance2} value=1 ] ]: CPU usage has exceeded 80% for the last 5 minutes.
But with a single data source, it would display just the value of the query:
81.234: CPU usage has exceeded 80% for the last 5 minutes.
Instead, we recommend using $values, which contains the same information as 
$value but is structured in an easier-to-use table format.
Functions
Functions can perform actions in templates such as transforming or formatting data.
Note that the functions provided by Go's template language, such as 
index, 
and, 
printf, and 
len, are available, along with many others.
In addition, the following functions are also available for templating annotations and labels:
Numbers
Name
Arguments
Returns
Description
humanize
number or string
string
Humanizes decimal numbers.
humanize1024
number or string
string
Like 
humanize, but uses 1024 as the base rather than 1000.
humanizeDuration
number or string
string
Humanizes a duration in seconds.
humanizePercentage
number or string
string
Humanizes a ratio value to a percentage.
humanizeTimestamp
number or string
string
Humanizes a Unix timestamp.
toTime
number or string
time
Converts a Unix timestamp in seconds to time.
Strings
Name
Arguments
Returns
Description
title
string
string
Capitalizes the first character of each word.
toUpper
string
string
Returns all text in uppercase.
toLower
string
string
Returns all text in lowercase.
stripPort
string
string
Returns only host.
match
pattern, text
boolean
Matches the text against a regular expression pattern.
reReplaceAll
pattern, replacement, text
string
Replaces text matching the regular expression.
graphLink
expr
string
Returns the path to the graphical view in 
Explore for the given expression and data source.
tableLink
expr
string
Returns the path to the tabular view in 
Explore for the given expression and data source.
parseDuration
string
float
Parses a duration string such as "1h" into the number of seconds it represents.
stripDomain
string
string
Returns the result of removing the domain part of a FQDN.
Others
Name
Arguments
Returns
Description
args
[]interface{}
map[string]interface{}
Translates a list of objects to a map with keys arg0, arg1 etc.
safeHtml
string
string
Marks string as HTML not requiring auto-escaping.
externalURL
none
string
Returns the external URL of the Grafana server as configured in the ini file(s).
pathPrefix
none
string
Returns the path of the Grafana server as configured in the ini file(s).
For further context on these functions, note that templating in Grafana is based on the Prometheus template implementation, enabling the use of these functions and Prometheus-like templates for formatting alert messages within Grafana.
humanize
The 
humanize function humanizes decimal numbers:
{{ humanize 1000.0 }}
1k
humanize1024
The 
humanize1024 function works similarly to 
humanize but uses 1024 as the base rather than 1000:
{{ humanize1024 1024.0 }}
1ki
humanizeDuration
The 
humanizeDuration function humanizes a duration in seconds:
{{ humanizeDuration 60.0 }}
1m 0s
humanizePercentage
The 
humanizePercentage function humanizes a ratio value between 0 and 1 to a percentage:
{{ humanizePercentage 0.2 }}
20%
humanizeTimestamp
The 
humanizeTimestamp function humanizes a Unix timestamp:
{{ humanizeTimestamp 1577836800.0 }}
2020-01-01 00:00:00 +0000 UTC
toTime
The 
toTime function converts a Unix timestamp in seconds to time.:
{{ toTime 1727802106 }}
2024-10-01 17:01:46 +0000 UTC
title
The 
title function capitalizes the first character of each word:
{{ title "hello, world!" }}
Hello, World!
toUpper
The 
toUpper function returns all text in uppercase:
{{ toUpper "Hello, world!" }}
HELLO, WORLD!
toLower
The 
toLower function returns all text in lowercase:
{{ toLower "Hello, world!" }}
hello, world!
stripPort
The 
stripPort splits string into host and port, then returns only host:
{{ stripPort "example.com:8080" }}
example.com
match
The 
match function matches the text against a regular expression pattern:
{{ match "a.*" "abc" }}
true
reReplaceAll
The 
reReplaceAll function replaces text matching the regular expression:
{{ reReplaceAll "localhost:(.*)" "example.com:$1" "localhost:8080" }}
example.com:8080
graphLink
The 
graphLink function returns the path to the graphical view in Explore for the given expression and data source:
{{ graphLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }}
/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":false,"range":true}]
parseDuration
The 
parseDuration function parses a duration string such as "1h" into the number of seconds it represents.
{{ parseDuration "1h" }}
3600
stripDomain
The 
stripDomain removes the domain part of a FQDN, leaving port untouched:
{{ stripDomain "example.com:8080" }}
example:8080
tableLink
The 
tableLink function returns the path to the tabular view in Explore for the given expression and data source:
{{ tableLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }}
/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":true,"range":false}]
args
The 
args function translates a list of objects to a map with keys arg0, arg1 etc. This is intended to allow multiple arguments to be passed to templates:
{{define "x"}}{{.arg0}} {{.arg1}}{{end}}{{template "x" (args 1 "2")}}
1 2
safeHtml
The 
safeHtml function marks string as HTML not requiring auto-escaping:
{{ safeHtml "<b>Text</b>"}}
<b>Text</b>
externalURL
The 
externalURL function returns the external URL of the Grafana server as configured in the ini file(s):
{{ externalURL }}
https://example.com/grafana
pathPrefix
The 
pathPrefix function returns the path of the Grafana server as configured in the ini file(s):
{{ pathPrefix }}
/grafana
Differences with notification templates
Both notification templates and alert rule templates use the Go templating system. However, the functions and variables available in notification templates differ from those used in annotations and labels templates, which are described in this documentation.
Annotation and label templates operate in the context of an individual alert instance, while notification templates apply to a notification that includes a group of alert(s).
For example, notification templates provide the 
.Alerts variable, which includes the list of all firing and resolved alerts in the notification. This variable is not available in alert rule templates, which operate within the context of a single alert instance.
Additionally, you cannot reuse templates for labels and annotations as you can with notification templates. Instead, you need to write each template inline within the label or annotation fields and manually copy them wherever you want to reuse the templates.

Configure notification policies
Notification policies determine how alerts are routed to contact points.
Policies have a tree structure. Each policy can have one or more child policies and a set of label matchers.
Each alert (or alert instance) is evaluated by the default policy and subsequently by each child policy. Alerts are routed to the appropriate notification policy by matching alert labels with the policy's label matchers. For further details, refer to label matching and routing in notification policies.
{{< figure src="/media/docs/alerting/get-started-notification-policy-tree-combo.png" max-width="750px" alt="A diagram displaying how the notification policy tree routes alerts" >}}
When an alert instance is assigned to a notification policy, the notification policy is responsible for:
Grouping similar alerts to minimize the number of notifications.
Controlling when notifications are sent using the timing options.
Determining the contact points that receive the alert notification.
{{< admonition type="note" >}}
The default notification policy and its child policies are assigned to a specific Alertmanager, and they cannot use contact points or mute timings from other Alertmanagers.
{{< /admonition >}}
Edit the default notification policy
You can change the contact point, grouping, and timing options for this policy. Mute timings and active time intervals aren't available on the root policy. Add them on a child policy instead.
In the left-side menu, click Alerts & IRM and then Alerting.
Click Notification configuration, then select the Notification policies tab.
From the Choose Alertmanager dropdown, select an external Alertmanager. By default, the Grafana Alertmanager is selected.
In the Default policy section, click ... -> Edit.
In Default contact point, update the contact point for where to send notifications when alert rules do not match any specific policy.
In Group by, choose labels to group alerts. If multiple alerts match this policy, they are grouped by the selected labels, and notifications are sent per group. For more details on using this option, refer to Group notifications.
In Timing options, set the timing options to configure when notifications are sent.
Group wait: the time to wait before sending the first notification for a new group of alerts. Default is 30 seconds.
Group interval: the time to wait before sending a notification about changes in the alert group. Default is 5 minutes.
Repeat interval: the time to wait before sending a notification if the group has not changed since the last notification. Default is 4 hours.
Click Save to save your changes.
Add a child policy
You can create a child policy under a default policy or under an existing child policy.
If you want to choose where to position your policy, see the section on Add a sibling policy.
In the left-side menu, click Alerts & IRM and then Alerting.
Click Notification configuration, then select the Notification policies tab.
From the Choose Alertmanager dropdown, select an Alertmanager. By default, the Grafana Alertmanager is selected.
Click +New child policy from the default policy or an existing child policy.
In the Matching labels section of the child policy, add one or more matching label rules to narrow down a specific case from the parent policy.
For instance, a child policy of the default policy handles 
team=security alerts, or a child policy handles only the 
severity=critical alerts of a parent policy.
In the Contact point dropdown, select the contact point to send notifications. If left empty, the contact point of the parent policy is inherited.
Optionally, enable Continue matching subsequent sibling nodes to continue matching sibling policies even after the alert matched the current policy. If enabled, multiple policies can handle the same alert.
Optionally, enable Override grouping to set different grouping than the parent policy. If disabled, the grouping of the parent policy is inherited.
Optionally, enable Override general timings to set different timing options than the parent policy. If disabled, the timing options of the parent policy are inherited.
Optionally, in Mute timings and Active timings, select time intervals that apply to this policy. These fields aren't available on the root policy.
Click Save policy to save your changes.
Add a sibling policy
In the left-side menu, click Alerts & IRM and then Alerting.
Click Notification configuration, then select the Notification policies tab.
Find the child policy you want to create a sibling for.
Click Add new policy -> New sibling above or New sibling below.
It's important to determine which policy receives the alert first and to set the correct order of sibling and child policies.
Policies are evaluated from top to bottom. If a matching policy is found, the system continues to evaluate its child policies in the order they are displayed. For more details, refer to notification policies routing.
Follow the instructions from step 5 onward in adding a child policy.
Search for policies
Grafana allows you to search within the tree of policies by the following:
Label matchers
Contact Points
To search by contact point simply select a contact point from the Search by contact point dropdown. The policies that use that contact point are highlighted in the user interface.
To search by label matchers simply enter a valid matcher in the Search by matchers input field. Multiple matchers can be combined with a comma (
,).
It is important to note that all matched policies are exact matches. Grafana supports regular expressions for creating label matchers. It does not support regular expression or partial matching in the search for policies.
Mute timings
Mute timings pause notifications during a recurring period. Active time intervals allow notifications only during a recurring period.
You can assign them only on child notification policies, not on the root policy. They aren't inherited from a parent notification policy. For instructions, refer to Configure mute timings.
Manage multiple notification policy trees
By default, Grafana uses a single notification policy tree for all alert routing. As organizations grow in size and complexity, managing a single global tree can become difficult. Multiple notification policy trees allow you to split routing logic into separate, independently managed routing trees.
Each notification policy tree has its own name, root policy, and child policies which function similarly to the default policy tree. You can create additional routing trees to organize routing logic by team, service, or domain.
The Default Policy can't be deleted. You can, however, reset the policy to clear it.
For provisioning instructions, refer to Enable multiple notification policy trees via Terraform.
Example
An example of an alert configuration.
Create a "default" contact point for slack notifications, and set it on the default policy.
Edit the default policy grouping to group alerts by 
cluster, 
namespace and 
severity so that you get a notification per alert rule and specific Kubernetes cluster and namespace.
Create specific route for alerts coming from the development cluster with an appropriate contact point.
Create a specific route for alerts with "critical" severity with a more invasive contact point integration, like pager duty notification.
Create specific routes for particular teams that handle their own on-call rotations.

Configure silences
Silences stop notifications from being created for a specified time window but do not interrupt alert evaluation. Use them to temporarily prevent alert notifications, such as during incident response or a maintenance window.
{{< admonition type="note" >}}
Silences are assigned to a specific Alertmanager and only suppress notifications for alerts managed by that Alertmanager.
In Grafana Cloud, the standalone Alertmanager UI is deprecated and no longer available. Manage silences here in the Grafana Alerting UI instead. For details, refer to Configure Alertmanagers.
{{< /admonition >}}
Mute timings and active time intervals vs silences
Mute timings and active time intervals and silences are distinct methods to suppress notifications. They do not prevent alert rules from being evaluated or stop alert instances from appearing in the user interface; they only prevent notifications from being created.
The following table highlights the key differences between mute timings and silences.
Mute timing
Silence
Setup
Created and then added to notification policies
Matches alerts using labels to determine whether to silence them
Period
Uses time interval definitions that can repeat periodically
Has a fixed start and end time
Add silences
To add a silence, complete the following steps.
In the left-side menu, click Alerts & IRM and then Alerting.
Click Silences.
From the Alertmanager dropdown, select an external Alertmanager to create and manage silences for the external data source. Otherwise, keep the default option of Grafana.
Click Create silence to open the Create silence page.
In Silence start and end, select the start and end date to indicate when the silence should go into effect and expire.
Optionally, in Duration, specify how long the silence is enforced. This automatically updates the end time in the Silence start and end field.
In the Label and Value fields, enter one or more Matching Labels to determine which alerts the silence applies to.
{{< collapse title="How label matching works" >}}
Use labels and label matchers to link alert rules to notification policies and silences. This allows for a flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.
A label matcher consists of 3 distinct parts, the label, the value and the operator.
The Label field is the name of the label to match. It must exactly match the label name.
The Value field matches against the corresponding value for the specified Label name. How it matches depends on the Operator value.
The Operator field is the operator to match against the label value. The available operators are:
Operator
Description
=
Select labels that are exactly equal to the value.
!=
Select labels that are not equal to the value.
=~
Select labels that regex-match the value.
!~
Select labels that do not regex-match the value.
{{< admonition type="note" >}}
If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match in order to link a rule to a policy.
{{< /admonition >}}
Label matching example
If you define the following set of labels for your alert:
{ foo=bar, baz=qux, id=12 }
then:
A label matcher defined as 
foo=bar matches this alert rule.
A label matcher defined as 
foo!=bar does not match this alert rule.
A label matcher defined as 
id=~[0-9]+ matches this alert rule.
A label matcher defined as 
baz!~[0-9]+ matches this alert rule.
Two label matchers defined as 
foo=bar and 
id=~[0-9]+ match this alert rule.
Exclude labels
You can also write label matchers to exclude labels.
Here is an example that shows how to exclude the label 
Team. You can choose between any of the values below to exclude labels.
Label
Operator
Value
team
=
""
team
!~
.+
team
=~
^$
{{< /collapse >}}
Any matching alerts (in the firing state only) display under Affected alert instances. Grafana evaluates the silence against each instance's current labels, so if a label value changes, that instance is treated as a new one and might no longer match.
In Comment, add details about the silence.
Click Submit.
Edit silences
To edit a silence, complete the following steps.
In the left-side menu, click Alerts & IRM and then Alerting.
Click Silences to view the list of existing silences.
Find the silence you want to edit, then click Edit (pen icon).
Make the desired changes, then click Submit to save your changes.
Remove silences
To remove a silence, complete the following steps.
In the left-side menu, click Alerts & IRM and then Alerting.
Click Silences to view the list of existing silences.
Select the silence you want to end, then click Unsilence.
{{< admonition type="note" >}}
Clicking Unsilence ends the silence immediately, which is the only way to end a silence before its configured end time. Silences cannot be permanently deleted manually.
{{< /admonition >}}
Silence duration and retention
Duration limits
There is no maximum duration for a silence. You can create a silence for any length of time. However, administrators can configure the following limits:
alertmanager_max_silences_count: Maximum number of active and pending silences per tenant (default: 0, no limit), expired silences are also counted.
alertmanager_max_silence_size_bytes: Maximum size of a silence in bytes (default: 0, no limit).
Expired silence retention
Expired silences are automatically deleted after 5 days. This retention period is not configurable. The cleanup process runs automatically every 15 minutes to remove expired silences that are older than the retention period.
Rule-specific silences
Rule-specific silences are silences that apply only to a specific alert rule. They're created when you silence an alert rule directly using the Silence notifications action in the UI.
As opposed to general silences, rule-specific silence access is tied directly to the alert rule they act on. They can be created manually by including the specific label matcher: 
__alert_rule_uid__=<alert rule UID>.
URL link to a silence form
Default notification messages often include a link to silence alerts.
In custom notification templates, you can use 
.Alert.SilenceURL to redirect users to the UI where they can silence the given alert.
If 
.Alert.SilenceURL doesn’t fit your specific use case, you can also create a custom silence link for your custom templates.
{{< collapse title="Create a custom silence link" >}}
When linking to a silence form, provide the default matching labels and comment via 
matcher and 
comment query parameters. The 
matcher parameter should be in the following format 
[label][operator][value] where the 
operator parameter can be one of the following: 
= (equals, not regular expression), 
!= (not equals, not regular expression), 
=~ (equals, regular expression), 
!~ (not equals, regular expression).
The URL can contain many query parameters with the key 
matcher.
For example, to link to silence form with matching labels 
severity=critical & 
cluster!~europe-.* and comment 
Silence critical EU alerts, create a URL 
https://mygrafana/alerting/silence/new?matcher=severity%3Dcritical&matcher=cluster!~europe-*&comment=Silence%20critical%20EU%20alert.
To link to a new silence page for an external Alertmanager, add a 
alertmanager query parameter with the Alertmanager data source name.
{{< /collapse >}}
Inhibition rules
Inhibition rules suppress notifications for target alerts when a related source alert is already firing. For more information, refer to Configure inhibition rules.

Import Alertmanager configuration to Grafana-managed notifications
You can import an existing Prometheus or Mimir Alertmanager configuration into Grafana Alerting. Grafana runs the imported receivers as Mimir Alertmanager v0 integrations and surfaces the configuration as Grafana notification resources: contact points, a notification policy tree, notification templates, time intervals, and inhibition rules. You operate your notification setup from Grafana.
Importing is a safe operation. The source Alertmanager keeps its configuration, and Grafana never writes back to it.
The import happens in two stages:
Stage: Grafana keeps the imported configuration as a unit of its own and combines it with your Grafana resources at runtime.
Promote: Grafana merges the imported configuration permanently into your Grafana configuration. Every imported resource becomes a normal, editable Grafana resource, and you can no longer revert the import in one action.
Because Grafana keeps the staged configuration separate, you can revert or re-import it all at once instead of resource by resource. Its resources are read-only, and no Grafana resource can reference them. The imported policy tree is the exception: an alert rule can route to it by name. Until a rule does, your existing notifications are unchanged.
{{< admonition type="note" >}}
Importing Alertmanager configuration is in public preview. The API is behind the 
alertingImportAlertmanagerAPI feature toggle and the user interface is behind 
alertingMigrationWizardUI. Both are disabled by default. In Grafana Cloud, contact Support to enable them.
{{< /admonition >}}
Before you begin
Before you import an Alertmanager configuration, make sure you have the following:
A source Grafana can read: A configuration YAML file with its template files, or a configured Mimir Alertmanager data source. Prometheus Alertmanager data sources aren't supported.
Permissions: Creating, reading, updating, or deleting an import requires the corresponding scoped Alertmanager import permissions. Promoting an import requires read and delete permissions for the import, create permissions for contact points and notification policy trees, and write permissions for any notification templates, time intervals, and inhibition rules in the configuration. The Admin role has these permissions by default. For more details, refer to RBAC permissions.
Not every Alertmanager configuration can be imported as-is. Refer to limitations before you start.
How it works
Grafana imports the configuration as it is and evaluates it the way your source Alertmanager does, for configurations from supported versions. Receivers keep their Alertmanager fields, and they notify with the same logic and message format, because Grafana runs them as Mimir-compatible integrations instead of rewriting them into native Grafana ones.
Mimir-compatible integrations don't offer what a native Grafana integration adds on top, such as images in notifications. To use those features, promote the import and then rebuild the contact point as a Grafana one.
What gets imported
Grafana imports every field of the configuration: receivers, the routing tree, template files, time intervals, and inhibition rules. Fields keep their names and values, apart from the renames described in name conflicts. Both 
time_intervals and the deprecated 
mute_time_intervals become time intervals.
The 
global section is the one part with no place of its own. Grafana resolves its values into the settings of each integration that relies on them as the configuration is parsed, so the defaults you set globally still apply.
Routing
Grafana adds the imported routing tree as a named policy tree of its own, alongside your default notification policy and any other named policy trees. Alerts reach it only when an alert rule routes to it by name. The matchers inside the imported tree then route those alerts the way they did in your source Alertmanager. For more details on serving more than one policy tree, refer to manage multiple notification policies.
If the imported root route leaves 
group_wait, 
group_interval, or 
repeat_interval unset, Grafana uses the default notification timing values.
You choose the tree name when you import. This name is also the identifier of the import, so pick something you recognize, such as 
mimir-prod. The name can't be 
default or match another named policy tree. It must be a valid DNS subdomain name and is length-limited; for more details, refer to limitations.
Name conflicts
Contact points, time intervals, and notification templates are identified by name, and the imported configuration may reuse names that already exist. Rather than fail or overwrite, Grafana renames the incoming resource:
Grafana appends 
_ and the import identifier to the name, for example 
default becomes 
default_mimir-prod.
If that name is also taken, Grafana appends a number, for example 
default_mimir-prod_01.
All references to a renamed resource are updated throughout the imported configuration, so routing still points at the right contact point.
Templates count conflicts differently. An imported template never conflicts with a Grafana-managed template of the same name, only with a template from another imported configuration. For more details, refer to limitations.
Staged resources are read-only
While an import is staged, its resources appear with an Imported status in the Grafana Alerting user interface and in the notification APIs, and you can't edit or delete them individually. The deprecated provisioning API is the exception: it doesn't return them at all.
You also can't reference an imported resource from a Grafana resource. For example, you can't pick an imported contact point in a Grafana notification policy. The imported policy tree is the exception: an alert rule can route to it by name while the import is staged.
To make imported resources editable and available for reference, promote the import.
One import at a time
Grafana stores one imported configuration per organization. Before you stage another configuration, promote or revert the existing one. You can also explicitly replace the existing configuration during import.
Import with the Grafana Alerting user interface
The Grafana Alerting user interface imports notification resources and alert rules in one flow. It requires both the 
alertingImportAlertmanagerAPI and 
alertingMigrationWizardUI feature toggles.
Go to Alerting > Alert rules.
In the More menu, click Import to Grafana Alerting.
Choose how the resources are added:
Stage brings the configuration in as a read-only, reversible copy.
Promote merges the configuration into your live configuration immediately. This can't be undone. To reverse it, you have to delete each resulting resource by hand.
Click Next.
On the Import notification resources step, choose the Import source:
Alertmanager config YAML uploads a configuration file. Optionally, upload the template files the configuration references. Each file is imported as a template named after the file.
Alertmanager data source reads the configuration from a configured Alertmanager data source.
Enter a Policy tree name.
Grafana validates the configuration as you fill in the form and reports any conflicts. The form lists the resources that Grafana renames before you import.
Click Next, and either configure the alert rules import or skip the step.
Review the summary, then click Start import.
Auto-sync an Alertmanager configuration
Auto-sync keeps a staged notification configuration up to date with a Mimir or Cortex Alertmanager data source. Grafana periodically reads the source configuration and applies changes to the staged resources without writing back to the source. Auto-sync doesn't import or synchronize alert rules.
{{< admonition type="note" >}}
Auto-sync is in private preview. The 
alerting.syncExternalAlertmanager feature toggle is disabled by default and requires a Grafana restart when enabled. To access Settings > Import, also enable 
alertingMigrationWizardUI. In Grafana Cloud, contact Support to request access.
{{< /admonition >}}
Enable auto-sync
Before you begin, configure a Mimir or Cortex Alertmanager data source. Auto-sync supports only these data sources. If a staged configuration with a different identifier already exists, promote or revert it first.
You need the following permissions, scoped to 
configs:uid:default:
notifications.alerting.grafana.app/configs:get to read the configuration and sync status.
notifications.alerting.grafana.app/configs:update to enable or disable auto-sync.
Organization administrators have these permissions by default. You can also grant them through RBAC.
Go to Alerting > Settings.
Click the Import tab.
In Auto-sync configuration, select the Alertmanager data source.
Click Save.
Grafana uses the data source UID as the import identifier and notification policy tree name. The imported resources remain read-only while staged. To send notifications through the imported policy tree, route your Grafana-managed alert rules to that tree. Use the data source UID as the routing tree name.
The Auto-sync configuration section shows the sync status and any errors from the last sync attempt. If Grafana can't fetch or validate an update, it keeps the configuration from the last successful sync.
Stop auto-sync or promote the configuration
While auto-sync is configured, you can't manually import another notification configuration, and the user interface hides Revert for the sync-managed configuration. You can still import alert rules separately.
To stop synchronization, click Disable sync in Auto-sync configuration and confirm. Disabling sync doesn't delete the staged configuration. You can then review, promote, or revert it.
Promoting the sync-managed configuration stops synchronization from that data source.
If Auto-sync configuration shows Managed by operator, you can't change the setting through the user interface. In Grafana Cloud, contact Support to change or disable auto-sync.
Import with the API
The Alertmanager import endpoints are compatible with the Mimir Alertmanager HTTP API, so you can use 
mimirtool or plain HTTP requests.
In these endpoints, an import is addressed by its identifier, which is set with the 
X-Grafana-Alerting-Config-Identifier header and defaults to 
imported.
Method
Endpoint
Summary
POST
/api/convert/api/v1/alerts
Import an Alertmanager configuration, optionally promoting it in the same request.
GET
/api/convert/api/v1/alerts
Get the staged configuration. Secrets are masked.
DELETE
/api/convert/api/v1/alerts
Delete the staged configuration. The Grafana configuration is unaffected.
POST
/api/convert/api/v1/alerts/{Identifier}/promote
Promote a staged configuration into the Grafana configuration.
The 
POST endpoint accepts YAML and JSON. If no media type is specified, YAML is assumed. The request body has the same shape as the Mimir Alertmanager configuration API:
template_files:
  default.tmpl: '{{ define "custom" }}Custom message{{ end }}'
alertmanager_config: |
  route:
    receiver: webhook
  receivers:
    - name: webhook
      webhook_configs:
        - url: 'https://example.com/webhook'
A successful import returns the merge result:
{
  "status": "success",
  "stats": {
    "added_route": "mimir-prod",
    "added_receivers": ["webhook"],
    "added_templates": ["default.tmpl"]
  }
}
Configure auto-sync and read its status
Auto-sync uses a dedicated notification configuration API. Use credentials with the auto-sync permissions.
{{< admonition type="note" >}}
Enable the 
alerting.syncExternalAlertmanager feature toggle before using this API. For availability and restart requirements, refer to Auto-sync an Alertmanager configuration.
{{< /admonition >}}
The configuration is a per-organization resource named 
default. Grafana creates it automatically.
Read the configuration and status with 
GET /apis/notifications.alerting.grafana.app/v0alpha1/namespaces/{namespace}/configs/default:
curl \
  -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  "<GRAFANA_BASE_URL>/apis/notifications.alerting.grafana.app/v0alpha1/namespaces/<NAMESPACE>/configs/default"
Replace the following placeholders in these examples:
<GRAFANA_BASE_URL>: The base URL of your Grafana instance.
<SERVICE_ACCOUNT_TOKEN>: A service account token with the required configuration permissions.
<NAMESPACE>: The API namespace for your Grafana organization.
<DATASOURCE_UID>: The UID of the Mimir or Cortex Alertmanager data source to synchronize.
Enable auto-sync with 
PATCH /apis/notifications.alerting.grafana.app/v0alpha1/namespaces/{namespace}/configs/default. Set 
spec.externalAlertmanagerSync.datasourceUid with a JSON patch:
curl -X PATCH \
  -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  -H "Content-Type: application/json-patch+json" \
  "<GRAFANA_BASE_URL>/apis/notifications.alerting.grafana.app/v0alpha1/namespaces/<NAMESPACE>/configs/default" \
  --data '[{"op":"add","path":"/spec/externalAlertmanagerSync","value":{"datasourceUid":"<DATASOURCE_UID>"}}]'
A successful patch saves the setting; it doesn't mean the first sync has completed. Read the resource again to inspect these status fields:
status.externalAlertmanagerSync.datasourceUid: The data source UID used on the last sync attempt.
status.externalAlertmanagerSync.origin: Whether the source is configured through the API or managed by an operator. For an operator-managed source in Grafana Cloud, contact Support to change or disable it.
status.conditions: Find the condition with 
type set to 
ExternalAlertmanagerSynced. Its 
status is 
True after a successful sync, 
False after a failed attempt, or 
Unknown when no sync is configured. Read 
reason and 
message for details. The 
MergeCommitted reason means the configuration was promoted and synchronization has stopped.
Status updates are asynchronous and can lag behind configuration changes.
To disable an API-configured sync, clear the setting with the same PATCH endpoint:
curl -X PATCH \
  -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  -H "Content-Type: application/json-patch+json" \
  "<GRAFANA_BASE_URL>/apis/notifications.alerting.grafana.app/v0alpha1/namespaces/<NAMESPACE>/configs/default" \
  --data '[{"op":"add","path":"/spec/externalAlertmanagerSync","value":{}}]'
Clearing the setting stops future synchronization but doesn't delete the staged notification resources. It doesn't override an operator-managed source. Read the status again to confirm that synchronization is no longer configured. Don't delete the 
default configuration resource or write its status; Grafana manages both its lifecycle and status.
Optional headers
Use these headers for more granular import control:
X-Grafana-Alerting-Config-Identifier
The identifier of the import, which is also the name of the notification policy tree it creates. It must be a valid DNS subdomain name of at most 40 characters, using only lowercase alphanumeric characters, 
-, and 
.. Defaults to 
imported.
X-Grafana-Alerting-Dry-Run
Set to 
true to validate the configuration and report the merge result without saving anything. Use this to preview renames before importing.
X-Grafana-Alerting-Promote
Set to 
true to promote the configuration in the same request instead of staging it. Combine it with 
X-Grafana-Alerting-Dry-Run to preview what a promotion merges into the live configuration.
X-Grafana-Alerting-Config-Force-Replace
Set to 
true to replace an existing staged configuration that has a different identifier. Without it, importing a second configuration fails.
mimirtool
Use 
mimirtool alertmanager load to import a configuration and its template files:
MIMIR_ADDRESS=<GRAFANA_BASE_URL>/api/convert/ \
MIMIR_AUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN> \
MIMIR_TENANT_ID=1 \
mimirtool alertmanager load alertmanager.yaml default.tmpl \
  --extra-headers "X-Grafana-Alerting-Config-Identifier=mimir-prod"
Replace the following placeholders:
<GRAFANA_BASE_URL>: The base URL of your Grafana instance.
<SERVICE_ACCOUNT_TOKEN>: A service account token with permission to import notification resources.
When the address points at 
<GRAFANA_BASE_URL>/api/convert/, 
mimirtool talks to Grafana rather than to a Mimir instance, so 
MIMIR_TENANT_ID must always be 
1.
mimirtool alertmanager get and 
mimirtool alertmanager delete read and remove the staged configuration in the same way.
Promote a staged configuration
To promote a configuration that's already staged, call the promote endpoint with its identifier:
curl -X POST \
  -H "Authorization: Bearer <SERVICE_ACCOUNT_TOKEN>" \
  <GRAFANA_BASE_URL>/api/convert/api/v1/alerts/mimir-prod/promote
After promotion, the staged configuration no longer exists. Its resources are part of the Grafana configuration. You can edit them through the regular notification APIs and user interface.
Review, promote, or revert a staged configuration
Staged configurations are managed in Grafana Alerting settings.
Go to Alerting > Settings.
Click the Import tab.
The Staged configuration section lists the contact points, notification policies, templates, time intervals, and inhibition rules the import contains, and links to each resource so you can inspect it before promoting.
To discard the import and everything it added, click Revert. Your Grafana configuration is unaffected. For a configuration managed by auto-sync, disable auto-sync before reverting. The user interface hides Revert while auto-sync is active. Deleting the staged configuration through the API doesn't disable auto-sync: Grafana can import it again when the source configuration changes or Grafana restarts.
{{< admonition type="warning" >}}
Reverting deletes the imported notification policy tree. Alert rules that route to that tree lose their target, and their alerts fall back to the root of your default notification policy. Update those rules to point elsewhere before you revert.
{{< /admonition >}}
Limitations
Consider the following when you import an Alertmanager configuration:
Mimir version: Grafana supports notification configuration imports from Grafana Mimir up to 3.0.x. Later Mimir versions can use integrations and fields that the imported receivers don't support.
Prometheus Alertmanager version: Grafana supports notification configuration imports from Prometheus Alertmanager up to v0.27. Later versions add integrations and fields that the imported receivers don't support.
Unsupported receiver fields: Integration fields that read their values from elsewhere, including every 
*_file and 
*_ref variant, have no equivalent in Grafana. Grafana rejects a configuration that contains one, so nothing is imported. Supply the value inline in the matching field instead, and drop the 
_file or 
_ref variant.
Data source support: Grafana reads a configuration from a Mimir Alertmanager data source. Upstream Prometheus Alertmanager doesn't expose a configuration API, so those data sources aren't supported. Import a configuration YAML file instead.
Global settings: After you promote an import, the 
global section no longer exists as a section of its own. Grafana has resolved its values into each integration setting that relies on them, so a promoted contact point carries the resolved value rather than a reference to 
global.
Policy tree name: The name of the imported policy tree, which is also the import identifier, must be a valid DNS subdomain name of at most 40 characters, using only lowercase alphanumeric characters, 
-, and 
..
One staged configuration at a time: Grafana stores one staged configuration per organization. Before you stage another configuration, promote or revert the existing one. You can also explicitly replace the existing configuration during import.
Template name conflicts across imports: Grafana renames colliding template files, but the templates defined inside those files share one namespace. Say an earlier import defines a template called 
default.email that renders 
X, and a new import defines 
default.email again, in a file under a different name, rendering 
Y. Which definition wins isn't deterministic, and every contact point that uses that template gets the winner. As a result, contact points from the earlier import can start sending the wrong content. Check the names of the templates you define, not just the filenames, before you import a second configuration.
Deprecated provisioning API: A staged configuration isn't visible in the deprecated provisioning API, which doesn't return its contact points, templates, or time intervals. Use the Grafana Alerting notification API (
notifications.alerting.grafana.app) instead.
Inhibition rules: Imported inhibition rules are supported through the API only. There's no user interface for creating or editing them. For more details, refer to configure inhibition rules.
Next steps
After you import your notification configuration:
Import your data source-managed alert rules and route them through the imported policy tree.
Review the imported contact points, notification policies, templates, and time intervals.

Configure inhibition rules
{{< admonition type="note" >}}
Available in Grafana 13 or higher.
{{< /admonition >}}
An inhibition rule suppresses notifications for target alerts when source alerts with matching label values are already firing. This lets you reduce noise when a root-cause alert makes dependent alerts redundant.
For example, if a node is down (the source), you can inhibit all alerts for services running on that node (the target). This prevents your team from receiving individual alerts for each affected service when the underlying cause is already captured in the source alert.
{{< admonition type="note" >}}
Inhibition rules are assigned to a specific Alertmanager and only suppress notifications for alerts managed by that Alertmanager.
{{< /admonition >}}
{{< admonition type="caution" >}}
Inhibition rules are intended for compatibility with configurations imported from Prometheus Alertmanager or Mimir. They have no dedicated management UI in Grafana by design.
If not carefully configured, inhibition rules can silently suppress alerts and make issues harder to detect. Consider silences or mute timings for most suppression use cases.
{{< /admonition >}}
Inhibition rules vs silences
Both inhibition rules and silences suppress alert notifications. The key difference is that inhibition rules suppress alerts automatically based on the presence of another alert, while silences suppress alerts for a fixed time window regardless of other alerts.
Inhibition rule
Silence
Trigger
Active when a matching source alert is firing
Active during a configured time window
Duration
Lasts as long as the source alert fires
Has a fixed start and end time
Setup
Defined as a persistent configuration
Created manually per occurrence
Manage inhibition rules
You can manage inhibition rules by using the Grafana App Platform API. There is no dedicated UI for creating or editing inhibition rules.
The API resource is:
Group: 
notifications.alerting.grafana.app
Version: 
v1beta1
Resource: 
inhibitionrules
{{< admonition type="caution" >}}
The inhibition rules API is in beta (
v1beta1) and is subject to change.
{{< /admonition >}}
Inhibition rules are also supported in the Prometheus Alertmanager. Refer to Configure Alertmanager to set up an external Alertmanager.
Inhibition rule schema
Each inhibition rule has the following fields:
Field
Required
Description
name
Yes
Unique name for the inhibition rule. Immutable after creation.
source_matchers
Yes
One or more matchers that identify source (inhibiting) alerts.
target_matchers
Yes
One or more matchers that identify target (inhibited) alerts.
equal
No
Labels that must have equal values in both source and target alerts for the inhibition to take effect.
Matcher format
Both 
source_matchers and 
target_matchers are lists of structured matcher objects. Each matcher has:
Field
Description
label
The name of the label to match against.
type
The matching operator. One of 
=, 
!=, 
=~, 
!~.
value
The value to match against. For 
=~ and 
!~, this is a regular expression.
Operator
Description
=
Select alerts where the label equals the value.
!=
Select alerts where the label does not equal the value.
=~
Select alerts where the label matches the regular expression.
!~
Select alerts where the label does not match the regular expression.
The 
equal field
When 
equal is specified, the inhibition only applies when the source and target alerts have the same value for each listed label. For example, setting 
equal: [cluster] ensures that an alert in one cluster only inhibits alerts in the same cluster.
If 
equal is omitted, any firing source alert matching 
source_matchers inhibits all matching target alerts regardless of their label values.
Missing labels and empty-value labels are treated as equivalent for the purpose of 
equal matching.
Example: suppress warning alerts
The following example defines an inhibition rule that suppresses all 
warning-severity alerts in a cluster when a 
critical-severity alert is already firing for the same cluster.
{
  "name": "critical-inhibits-warnings",
  "source_matchers": [{ "label": "severity", "type": "=", "value": "critical" }],
  "target_matchers": [{ "label": "severity", "type": "=", "value": "warning" }],
  "equal": ["cluster", "namespace"]
}
In this example:
The source matcher selects 
critical alerts. When one fires, the rule becomes active.
The target matcher selects 
warning alerts. The rule suppresses these while the source alert fires.
The 
equal field ensures suppression only applies when source and target share the same 
cluster and 
namespace label values.
Inhibition rules and managed routes
An Alertmanager can serve multiple routing trees: the default Grafana-managed route and any routes imported from Prometheus Alertmanager or Mimir configurations. Inhibition rules behave differently depending on how each rule is created:
Grafana-managed rules are created through the API and apply across the entire Alertmanager. They can match alerts handled by any route, including imported routes.
Imported rules come from an imported Alertmanager configuration and are scoped to the alerts handled by their imported route. Grafana adds an internal label matcher that limits each imported rule to its own imported route, so an imported rule can't suppress alerts that flow through other routes.
To suppress alerts across multiple imported configurations, create a Grafana-managed rule through the API instead of relying on the inhibition rules embedded in an imported configuration.

Configure mute timings and active time intervals
Mute timings and active time intervals let you determine how your alert notifications are handled during designated periods of time. After you create a time interval, you can apply it as either a mute timing or active time interval for your notification policies.
A mute timing is a recurring interval that stops notifications for one or multiple notification policies during a specified period. It suppresses notifications but does not interrupt alert evaluation.
Use mute timings to temporarily pause notifications for a specific recurring period, such as a regular maintenance window or weekends.
Active time intervals provide the opposite functionality, where alerts handled by a notification policy are suppressed unless the notification happens at a time that matches the time interval. Use active time intervals for periods where you want to reduce alert noise. Mute timings take precedence over active time intervals when they overlap.
{{< admonition type="note" >}}
Mute timings and active time intervals are assigned to a specific Alertmanager and only suppress notifications for alerts managed by that Alertmanager.
{{< /admonition >}}
Mute timings and active time intervals vs silences
Mute timings and active time intervals and silences are distinct methods to suppress notifications. They do not prevent alert rules from being evaluated or stop alert instances from appearing in the user interface; they only prevent notifications from being created.
The following table highlights the key differences between mute timings and silences.
Mute timing
Silence
Setup
Created and then added to notification policies
Matches alerts using labels to determine whether to silence them
Period
Uses time interval definitions that can repeat periodically
Has a fixed start and end time
Add time intervals
In the left-side menu, click Alerts & IRM, and then Alerting.
Click Notification configuration, then select the Time intervals tab.
From the Alertmanager dropdown, select an external Alertmanager. By default, the Grafana Alertmanager is selected.
Click + New time interval.
Fill out the form to create a time interval to match against for your mute timing or active time interval.
Save your changes.
Assign a time interval to a notification policy
After you create a time interval, assign it as a mute timing or an active time interval on a child notification policy. You can't assign these settings to the root policy. They aren't inherited from a parent policy.
In the left-side menu, click Alerts & IRM, and then Alerting.
Click Notification configuration, then select the Notification policies tab.
Find a child notification policy and click ... -> Edit.
From Mute timings or Active timings, choose the time intervals to add to the policy.
Save your changes.
Time intervals
A time interval is a specific duration during which alerts are suppressed. The duration typically consists of a specific time range and the days of the week, month, or year.
A mute timing or active time interval can contain multiple time intervals.
Supported time interval options are:
Time range: The time inclusive of the start and exclusive of the end time (in UTC if no location has been selected, otherwise local time).
Location: Depending on the location you select, the time range is displayed in local time.
Days of the week: The day or range of days of the week. Example: 
monday:thursday.
Days of the month: The date 1-31 of a month. Negative values can also be used to represent days that begin at the end of the month. For example: 
-1 for the last day of the month.
Months: The months of the year in either numerical or the full calendar month. For example: 
1, may:august.
Years: The year or years for the interval. For example: 
2021:2024.
All fields are lists; to match the field, at least one list element must be satisfied. Fields also support ranges using 
: (e.g., 
monday:thursday).
If a field is left blank, any moment of time matches the field. For an instant of time to match a complete time interval, all fields must match.
If you want to specify an exact duration, specify all the options.
Example
If you wanted to create a time interval for the first Monday of the month, for March, June, September, and December, between the hours of 12:00 and 24:00 UTC your time interval specification would be:
Time range:
Start time: 
12:00
End time: 
24:00
Days of the week: 
monday
Months: 
3, 6, 9, 12
Days of the month: 
1:7

Configure an Alertmanager contact point
Use the Alertmanager integration in a contact point to send specific alerts to a different Alertmanager.
If you have an existing Alertmanager running in your infrastructure, you can use a contact point to forward Grafana alerts to your Alertmanager.
For example, a team might run its own Alertmanager to manage notifications from other alerting systems. That team can create alerts with Grafana and configure an Alertmanager contact point to forward only their alerts to their existing Alertmanager.
This setup avoids duplicating Alertmanager configurations for better maintenance.
{{< admonition type="note" >}}
To send all Grafana-managed alerts to an Alertmanager, add it as a data source and enable it to receive all alerts. With this setup, you can configure multiple Alertmanagers to receive all alerts.
For setup instructions, refer to Configure Alertmanagers.
{{< /admonition >}}
Configure an Alertmanager for a contact point
To create a contact point with Alertmanager integration, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a name for the contact point.
From the Integration list, select Alertmanager.
In the URL field, enter the URL of the Alertmanager.
(Optional) Configure optional settings.
Click Save contact point.
For more details on contact points, including how to test them and enable notifications, refer to Configure contact points.
Alertmanager settings
Option
Description
URL
The Alertmanager URL. This field is protected from modification in Grafana Cloud.
Optional settings
Option
Description
Basic Auth User
Username for HTTP Basic Authentication.
Basic Auth Password
Password for HTTP Basic Authentication.
Optional notification settings
Option
Description
Disable resolved message
Enable this option to prevent notifications when an alert resolves.

Configure Amazon SNS notifications
Use the Amazon SNS integration in a contact point to send alert notifications to a SNS topic. Then, configure the SNS topic to forward notifications to distinct subscriber channels used in your SNS account.
Before you begin
Before you begin, ensure you have the following:
AWS SNS Topic: An SNS topic to send notifications to.
AWS IAM Identity with necessary access: An IAM identity (e.g. user, role) with the necessary permissions to publish messages to the SNS topic.
For a minimal setup, refer to Example using an Access Key.
Configure Amazon SNS for a contact point
To create a contact point with a SNS integration, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a name for the contact point.
From the Integration list, select AWS SNS.
Set up the required settings for your SNS configuration.
Click Save contact point.
For more details on contact points, including how to test them and enable notifications, refer to Configure contact points.
SNS Settings
The Amazon SNS API URL: (Optional) The SNS API URL, e.g., 
https://sns.us-east-2.amazonaws.com. If not specified, the SNS API URL from the SNS SDK will be used.
Signature Version (sigv4): Configures AWS's Signature Verification 4 signing process to sign requests.
Region: (Optional) The AWS region. If blank, the region from the default credentials chain is used.
Access Key : (Optional) The AWS API access key.
Secret Key: (Optional) The AWS API secret key.
Both 
Access Key and 
Secret Key must be provided together or left blank together.
If left blank, Grafana searches for credentials using the default credentials chain, including environment variables (
AWS_ACCESS_KEY_ID and 
AWS_SECRET_ACCESS_KEY), the shared credential file, and EC2 instance roles.
Profile: (Optional) Named AWS profile used to authenticate.
SNS topic ARN: (Optional) If you don't specify this value, you must specify a value for the 
Phone number or 
Target ARN. If you are using a FIFO SNS topic you should set a message group interval longer than 5 minutes to prevent messages with the same group key being deduplicated by the SNS default deduplication window.
Phone number: (Optional) Phone number if message is delivered via SMS in E.164 format. If you don't specify this value, you must specify a value for the 
SNS topic ARN or 
Target ARN.
Target ARN: (Optional) The mobile platform endpoint ARN if message is delivered via mobile notifications. If you don't specify this value, you must specify a value for the 
SNS topic ARN or 
Phone number.
Subject: (Optional) Customize the subject. This field supports notification templates and, by default, uses the default title template (
default.title). It cannot be an empty string.
Message: (Optional) Customize the message. This field supports notification templates and, by default, uses the default message template (
default.message).
Attributes: (Optional) Add any SNS message attributes.
Example using an Access Key
This section outlines a minimal setup to configure Amazon SNS with Alerting.
1. Create an SNS Topic and Email Subscriber
Navigate to SNS in AWS Console:
Go to the Amazon SNS Console.
Create a new topic:
On the Topics page, choose "Create topic".
Select "Standard" as the type.
Enter a Name for your topic, e.g., 
My-Topic.
Encryption: Leave disabled for this minimal setup.
Click "Create topic".
(Optional) Add an email subscriber to help test:
Within your newly created topic, click on "Create subscription".
Protocol: Choose 
Email.
Endpoint: Enter your email address to receive test notifications.
Click "Create subscription".
Confirm Subscription: Check your email and confirm the subscription by clicking the provided link.
2. Create an IAM Policy, User, and Access Key
Navigate to IAM in AWS Console:
Go to the IAM Console.
Create a new policy:
On the Policies page, choose "Create policy".
Switch to the "JSON" tab and paste the following policy, replacing 
Resource with your SNS topic ARN:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["sns:Publish", "sns:GetTopicAttributes"],
      "Resource": "arn:aws:sns:<region>:<account_id>:<topic_name>"
    }
  ]
}
Click "Next", name it (e.g., 
SNSPublishPolicy), and click "Create policy".
Create a new IAM user and assign the policy
In the IAM Console, on the Users page, choose "Create user".
Enter a User name, e.g., 
alerting-sns-user.
Click "Next".
In Set permissions, select "Attach policies directly".
Search for the policy you created earlier (
SNSPublishPolicy) and select it.
Click "Next" , and click "Create user".
Create an Access Key:
Within your newly created user, click on "Create access key".
Select an appropriate use-case, e.g., 
Application running outside AWS.
Click "Next" , and click "Create access key".
Save Credentials: Note the Access key ID and Secret access key that are required in the next step.
3. Configure the SNS Contact Point in Grafana
Follow the steps in configure Amazon SNS for a contact point, using the settings below and replacing the placeholders with the SNS and IAM values created in the previous steps.
The Amazon SNS API URL: 
https://sns.<region>.amazonaws.com
Signature Version (sigv4):
Region: 
<region>
Access Key: 
<YOUR_ACCESS_KEY>.
Secret Key: 
<YOUR_SECRET_ACCESS_KEY>
SNS topic ARN: 
arn:aws:sns:<region>:<account_id>:<topic_name>
Test the contact point to ensure it's working, or enable notifications for it.
Additional Resources
Configure contact points
Amazon SNS Documentation
Amazon IAM Documentation
Prometheus Alertmanager SNS Configuration

Configure Discord for Alerting
Use the Grafana Alerting - Discord integration to receive alert notifications in your Discord channels when your Grafana alert rules are triggered and resolved.
Before you begin
Create a Webhook to enable Grafana to send alert notifications to Discord channels.
To create a Webhook in Discord, complete the following steps.
Follow the steps in the Intro to Webhooks guide.
Copy the Webhook URL.
Procedure
To create your Discord integration in Grafana Alerting, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a contact point name.
From the Integration list, select Discord.
In the Webhook URL field, paste in your Webhook URL.
Click Test to check that your integration works.
** For Grafana Alertmanager only.**
A test alert notification should be sent to the Discord channel that you associated with the Webhook.
Click Save contact point.
Next steps
The Discord contact point is ready to receive alert notifications.
To add this contact point to your alert, complete the following steps.
In Grafana, navigate to Alerting > Alert rules.
Edit or create a new alert rule.
Scroll down to the Configure labels and notifications section.
Under Notifications click Select contact point.
From the drop-down menu, select the previously created contact point.
Click Save rule and exit.

Configure email for alert notifications
Use the email integration to send alert notifications to one or more addresses.
You can customize the subject and main section of the email body. By default, the subject and body are generated from the alert data included in the notification.
Before you begin
In Grafana OSS, you must configure SMTP settings before you can enable email notifications.
{{}}
In Grafana Cloud, SMTP configuration is not required.
{{}}
Open the configuration file. The file is typically named 
grafana.ini or 
custom.ini and located in the 
conf directory of your Grafana installation.
Configure the SMTP settings for your email server in the 
[smtp] section.
Save your changes and restart Grafana.
Test email notifications by creating a contact point.
Configure Email for a contact point
To create a contact point with a email integration, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + New contact point.
Enter a name for the contact point.
From the Integration list, select Email.
Set up the required settings for your Email configuration.
Click Save contact point.
For more details on contact points, including how to test them and enable notifications, refer to Configure contact points.
Email settings
Option
Description
Addresses
The list of email addresses to send the notifications. Email addresses are case sensitive.
Optional settings
Option
Description
Single email
Send a single email to all email addresses rather than individual emails.
Optional settings using templates
{{}}
You can customize the email subject and main section of the email body, but you can't edit HTML or CSS for visual changes.
In Grafana OSS and Enterprise, you can edit the full email template. However, this is not officially supported because it's an internal API that may change without prior notice.
{{}}
Option
Description
Subject
Sets the email subject, replacing the default template. Supports notification templates.
Message
Sets the main section of the email body, replacing the default template. Supports notification templates.
{{< figure src="/media/docs/alerting/custom-email-message5.png" caption="Email notification with custom message." max-width="750px" >}}
Optional notification settings
Option
Description
Disable resolved message
Enable this option to prevent notifications when an alert resolves.

Configure Google Chat for Alerting
Use the Grafana Alerting - Google Chat integration to receive alert notifications in your Google Chat space when your Grafana alert rules are triggered and resolved.
Before you begin
Create a Webhook to enable Grafana to send alert notifications to a Google Chat space.
To create a Webhook in Google Chat space, complete the following steps.
Follow the steps in Google's Chat app guide.
Copy the Webhook URL.
Procedure
To create your Google Chat integration in Grafana Alerting, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a contact point name.
From the Integration list, select Google Chat.
In the URL field, paste in your Webhook URL.
Click Test to check that your integration works.
** For Grafana Alertmanager only.**
A test alert notification should be sent to the Google Chat space that you associated with the Webhook.
Click Save contact point.
Next steps
The Google Chat contact point is ready to receive alert notifications.
To add this contact point to your alert, complete the following steps.
In Grafana, navigate to Alerting > Alert rules.
Edit or create a new alert rule.
Scroll down to the Configure labels and notifications section.
Under Notifications click Select contact point.
From the drop-down menu, select the previously created contact point.
Click Save rule and exit.

Configure Grafana IRM for Alerting
Configure Grafana IRM for Alerting
In Grafana OSS and Grafana Enterprise, you can use a webhook contact point to send alerts to Grafana IRM. Grafana IRM can then route alerts based on escalation chains for your team's workflows.
The alert notification flow is as follows:
* Grafana OSS/Enterprise (send webhook alerts)->Grafana Cloud IRM (route via escalation chains)
{{< admonition type="note" >}}
The Free Forever plan in Grafana Cloud IRM includes 3 IRM active users per month.
These instructions apply only to Grafana OSS and Grafana Enterprise. To configure IRM for Grafana Cloud Alerting, refer to the Grafana Cloud documentation.
{{< /admonition >}}
Configure an integration in Grafana IRM
First, enable an integration in IRM to accept alerts from Grafana Alerting. You can either create a new integration or use an existing Alertmanager or Webhook integration in IRM.
To create the integration, follow the same steps as described in Configure an OnCall integration in IRM:
Navigate to Alerts & IRM -> IRM -> Integrations.
Click + New integration.
Select either Alertmanager or Webhook integration from the list.
Alertmanager integration – Includes preconfigured IRM templates for processing Grafana and Prometheus alerts.
Webhook integration – Uses default IRM templates for general alert processing.
Provide a title, description, and assign it to a team, then click Create Integration.
On the integration details page, copy the HTTP Endpoint URL to use in the next section.
Configure a webhook contact point in Grafana Alerting
Next, configure the contact point in Grafana Alerting to send alert notifications to Grafana IRM.
In Grafana OSS and Grafana Enterprise, you need to create a Webhook contact point using the HTTP endpoint URL you copied in the previous step.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a name for the contact point.
From the Integration list, select Webhook.
In the URL field, enter the HTTP endpoint URL of the IRM integration.
Click Save contact point.
After configuring the contact point in Grafana Alerting and the integration in Grafana IRM, you can start testing the Alerting-to-IRM integration.
{{< figure src="/media/docs/alerting/test-alert-group-on-irm.png" max-width="750px" caption="Example of a test alert group in Grafana IRM" >}}
For more information, see:
Configure contact points – Learn how to test the integration and enable notifications in Alerting.
Webhook contact point – Learn the format of the webhook payload and additional settings in Alerting.
Configure IRM alert templates – Learn how to process the incoming webhook messages in IRM.
Enable heartbeat monitoring in Grafana IRM (optional)
Enabling the heartbeat in the IRM integration acts as a monitoring for your Grafana Alerting setup. If Grafana Alerting stops sending alerts—due to downtime or misconfiguration—Grafana IRM creates a new alert group to notify you.
To set up heartbeat monitoring, you must enable the heartbeat in IRM and create an alert rule that continuously sends alerts to the heartbeat endpoint.
Enable the heartbeat for the IRM integration
In IRM, select the IRM integration you configured earlier.
Click the ⋮ (three dots) on top right, then select Heartbeat Settings.
Copy the Endpoint URL—this will be used in a new contact point in Grafana Alerting.
Set the heartbeat interval, time period after which Grafana IRM starts a new alert group if it doesn’t receive a heartbeat.
Create another Webhook contact point in Grafana Alerting
Follow the same steps as before to create a webhook contact point in Grafana Alerting:
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a name for the contact point.
From the Integration list, select Webhook.
In the URL field, enter the endpoint URL of the heartbeat.
Click Save contact point.
You can now click the Test button to send an alert to the heartbeat endpoint. In IRM, verify the heartbeat status in the Heartbeat column on the Integrations page.
{{< figure src="/media/docs/alerting/view-heartbeat-status-on-irm.png" max-width="750px" caption="Heartbeat status column in the Grafana IRM Integrations page" >}}
Create an alert rule in Grafana Alerting
Create a Grafana-managed alert rule with the following settings:
Always firing – Use a query and alert condition that constantly fire. For example, select a Prometheus data source and set the query to 
vector(1) > 0.
Configure a pending period that is shorter than the heartbeat interval.
Choose the webhook contact point you created for the heartbeat to forward alerts.
Adjust timing options in the alert rule or notification policy to ensure alerts are forwarded before the heartbeat interval elapses:
Group wait: 
0s
Group interval: 
1s
Repeat interval: shorter than the heartbeat interval.
After it's created, the alert rule acts as a heartbeat, verifying that Grafana Alerting is running and sending alerts to Grafana IRM.

Configure Jira for Alerting
Use the Jira integration in a contact point to create issues in your Jira instance when alerts fire. The integration supports both Jira Cloud and Jira Server/Data Center installations.
Before you begin
Before you begin, ensure you have the following:
A Jira instance (Cloud, Server, or Data Center)
API access credentials for Jira
Appropriate permissions to create issues in your target Jira project
Configure Jira for a contact point
To create a contact point with a Jira integration, complete the following steps:
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a name for the contact point.
From the Integration list, select Jira.
Set up the required settings for your Jira configuration.
Click Save contact point.
For more details on contact points, including how to test them and enable notifications, refer to Configure contact points.
Required Settings
Key
Description
URL
The URL of the REST API of your Jira instance. Supported versions: 
2 and 
3 (e.g., 
https://your-domain.atlassian.net/rest/api/3). This field is protected from modification in Grafana Cloud.
Basic Auth User
Username for authentication. For Jira Cloud, use your email address.
Basic Auth Password
Password or personal token. For Jira Cloud, you need to obtain a personal token here and use it as the password.
API Token
An alternative to basic authentication, a bearer token is used to authorize the API requests. See Jira documentation for more information.
Project Key
The project key identifying the project where issues will be created. Project keys are unique identifiers for a project.
Issue Type
The type of issue to create (e.g., 
Task, 
Bug, 
Incident). Make sure that you specify a type that is available in your project.
Optional Settings
Key
Description
Summary
The issue title. Supports templating. Max length is 255 characters.
Description
The description of the issue. Depending on version of the API, it can be a text, Markdown, or JSON (v3 API only). Maximum size of the field is 32kb.
Note: JSON is not limited by the client, and if it exceeds the size, the API is likely to reject the request.
Labels
Custom labels can be added to organize and filter issues created in Jira. Supports templating, allowing dynamic label generation based on alert data.
Priority
The priority level of the issue (e.g., 
Low, 
Medium, 
High, 
Critical). 
Ensure that the priority value matches the available options for your Jira instance. You can customize priority levels in Jira here.
Resolve Transition
The transition name to move the issue to a resolved state when an alert is resolved. Ensure that the value matches a valid transition available in your Jira workflow for the specified issue type. If this field is empty, the issue will not be transitioned to Done.
Reopen Transition
The transition name to move the issue back to an open state when an alert reoccurs. Ensure that the value matches a valid transition available in your Jira workflow for the specified issue type. If this field is empty, the issue will not be reopened.
Reopen Duration
The time duration (in minutes) to control whether to reopen an issue that was closed within this duration or create a new one. If not specified, the most recent issue that matches the deduplication key will be updated and reopened (if reopen transition is specified).
"Won't fix" Transition
Specify a resolution status that should be ignored when searching for existing issues. For example, issues with this resolution will not be reopened or updated by subsequent alerts.
Deduplication Key Field
Custom field to store the deduplication key. Must be a text field. 
 If not specified, the deduplication key is added to labels in the format of 
ALERT(hash sum). See Jira documentation for how to create custom fields.
Fields
Extra Jira issue fields, including standard and custom fields. For custom fields, use 
customfield_<FIELD_ID>, where 
<FIELD_ID> is the numeric field ID in your Jira instance. For JSON parsing and templating behavior, refer to Custom field data.
Custom field data
Grafana attempts to parse each string field value as JSON when it loads the contact point. Grafana sends values parsed as objects or arrays as structured JSON, such as 
{"value": "High"}, but doesn't evaluate templates inside them. Grafana evaluates values that remain strings as notification templates, but doesn't parse their rendered output as JSON. Templates therefore can't generate dynamic object or array field values, such as those required by Jira select-list fields.
To set different structured values for different alerts, create a Jira contact point for each static value and route alerts with notification policies. Or use a Webhook contact point with a custom payload to send structured JSON that includes alert data.

Configure MQTT notifications
Use the MQTT integration in contact points to send alert notifications to your MQTT broker.
Configure MQTT for a contact point
To create a contact point with MQTT integration, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a name for the contact point.
From the Integration list, select MQTT.
Enter your broker URL in the Broker URL field. Supports 
tcp, 
ssl, 
mqtt, 
mqtts, 
ws, 
wss schemes. For example: 
tcp://127.0.0.1:1883.
Enter the MQTT topic name in the Topic field.
(Optional) Configure additional settings.
Click Save contact point.
For more details on contact points, including how to test them and enable notifications, refer to Configure contact points.
Required Settings
Option
Description
Broker URL
The URL of the MQTT broker. This field is protected from modification in Grafana Cloud.
Topic
The topic to which the message will be sent.
Optional Settings
Option
Description
Message format
If set to 
json (default), the notification message uses the default JSON payload. 
 If set to 
text, the notification message is fully customizable.
Message
Depends on the Message format option. 
 In 
json format, defines only the 
message field of the default JSON payload. 
 In 
text format, defines the entire custom payload. 
 This field supports notification templates.
Client ID
The client ID to use when connecting to the MQTT broker. If blank, a random client ID is used.
Username
The username to use when connecting to the MQTT broker.
Password
The password to use when connecting to the MQTT broker.
QoS
The quality of service to use when sending the message. Options are 
At most once, 
At least once, and 
Exactly once.
Retain
If set to true, the message will be retained by the broker.
TLS
TLS configuration options, including CA certificate, client certificate, and client key, and disable certificate verification.
Disable resolved message
Enable this option to prevent notifications when an alert resolves.
Default JSON payload
If the Message format option is 
json (the default), the payload is like this example.
{
  "receiver": "My MQTT integration",
  "status": "firing",
  "orgId": 1,
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High memory usage",
        "team": "blue",
        "zone": "us-1"
      },
      "annotations": {
        "description": "The system has high memory usage",
        "runbook_url": "https://myrunbook.com/runbook/1234",
        "summary": "This alert was triggered for zone us-1"
      },
      "startsAt": "2021-10-12T09:51:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/1afz29v7z/edit",
      "fingerprint": "c6eadffa33fcdf37",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1",
      "dashboardURL": "",
      "panelURL": "",
      "values": {
        "B": 44.23943737541908,
        "C": 1
      }
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "High CPU usage",
        "team": "blue",
        "zone": "eu-1"
      },
      "annotations": {
        "description": "The system has high CPU usage",
        "runbook_url": "https://myrunbook.com/runbook/1234",
        "summary": "This alert was triggered for zone eu-1"
      },
      "startsAt": "2021-10-12T09:56:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/d1rdpdv7k/edit",
      "fingerprint": "bc97ff14869b13e3",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1",
      "dashboardURL": "",
      "panelURL": "",
      "values": {
        "B": 44.23943737541908,
        "C": 1
      }
    }
  ],
  "groupLabels": {},
  "commonLabels": {
    "team": "blue"
  },
  "commonAnnotations": {},
  "externalURL": "https://play.grafana.org/",
  "version": "1",
  "groupKey": "{}:{}",
  "message": "**Firing**\n\nLabels:\n - alertname = T2\n - team = blue\n - zone = us-1\nAnnotations:\n - description = This is the alert rule checking the second system\n - runbook_url = https://myrunbook.com\n - summary = This is my summary\nSource: https://play.grafana.org/alerting/1afz29v7z/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1\n\nLabels:\n - alertname = T1\n - team = blue\n - zone = eu-1\nAnnotations:\nSource: https://play.grafana.org/alerting/d1rdpdv7k/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1\n"
}
Body
If the Message format option is 
json (the default), the payload contains the following fields.
Key
Type
Description
receiver
string
Name of the contact point
status
string
Current status of the alert, 
firing or 
resolved
orgId
number
ID of the organization related to the payload
alerts
array of alerts
Alerts that are triggering
groupLabels
object
Labels that are used for grouping, map of string keys to string values
commonLabels
object
Labels that all alarms have in common, map of string keys to string values
commonAnnotations
object
Annotations that all alarms have in common, map of string keys to string values
externalURL
string
External URL to the Grafana instance sending this webhook
version
string
Version of the payload
groupKey
string
Key that is used for grouping
message
string
Custom message configured in Message (Optional Settings). 
 Supports notification templates; the output is formatted as a string.
{{< admonition type="note" >}}
When using the 
json Message format, only the message field of the JSON payload is customizable, and its output is formatted as a string.
To customize the full payload in text or JSON format, use the 
text format and define a custom payload.
{{< /admonition >}}
Alert object
The Alert object represents an alert included in the notification group, as provided by the 
alerts field.
{{< docs/shared lookup="alerts/table-for-json-alert-object.md" source="grafana" version="<GRAFANA_VERSION>" >}}
Custom payload
When you set the Message format option to 
text, you can customize the entire payload of the MQTT message.
In this mode, the Message option defines the entire payload. It supports notification templates and can generate notification messages in plain text, JSON, or any custom format.
For examples of templates that produce plain text or JSON messages, refer to notification template examples.

Configure Opsgenie for Alerting
Use the Grafana Alerting - Opsgenie integration to receive alert notifications in your Opsgenie alert dashboard when your Grafana alert rules are triggered and resolved.
Before you begin
Create an API key to enable Grafana to send alert notifications to Opsgenie alert dashboard.
To create an API key in Opsgenie, complete the following steps.
Follow the steps in the API integration guide.
Make sure you turn on the integration.
Copy the API key.
Procedure
To create your Opsgenie integration in Grafana Alerting, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a contact point name.
From the Integration list, select Opsgenie.
In the API key field, paste in your API key.
Configure the Alert API URL.
For Grafana Alertmanager, enter 
https://api.opsgenie.com/v2/alerts.
For other Alertmanagers, enter the host for sending Opsgenie API requests, depending on the hosted region.
Click Test to check that your integration works.
For Grafana Alertmanager only.
A test alert notification is sent to the Alerts page in Opsgenie.
Click Save contact point.
Next steps
The Opsgenie contact point is ready to receive alert notifications.
To add this contact point to your alert rule, complete the following steps:
In Grafana, navigate to Alerting > Alert rules.
Edit or create a new alert rule.
Scroll down to the Configure labels and notifications section.
Under Notifications, click Select contact point.
From the drop-down menu, select the previously created contact point.
Click Save rule and exit.

Configure Slack for Alerting
Use the Grafana Alerting - Slack integration to send Slack notifications when your alerts are firing.
There are two ways of integrating Slack into Grafana Alerting.
Use a Slack API token
Enable your app to access the Slack API. If, for example, you are interested in more granular control over permissions, or your project is expected to regularly scale, resulting in new channels being created, this is the best option.
Use a Webhook URL
Webhooks is the simpler way to post messages into Slack. Slack automatically creates a bot user with all the necessary permissions to post messages to one particular channel of your choice.
Note that you can only setup one Slack channel per contact point.
You can customize the 
title and 
body of the Slack message using notification templates; however, you cannot modify its visual appearance with custom blocks. Additional optional settings are available to customize bot appearance, mentions, and message formatting. Refer to the Optional settings section for more details.
Before you begin
Slack API Token
If you are using a Slack API Token, complete the following steps.
Follow step 1 of the Slack API Quickstart to create the app.
Continue onto the second step of the Slack API Quickstart and add the chat:write.public scope as described to give your app the ability to post in all public channels without joining.
In OAuth Tokens for Your Workspace, copy the Bot User OAuth Token.
Open your Slack workplace.
Right click the channel you want to receive notifications in.
Click View channel details.
Scroll down and copy the Channel ID.
Note that while going through these steps, Slack may prompt you to Reinstall your app in order for the changes to take effect.
Webhook URL
If you are using a Webhook URL, complete the following steps.
Follow the Creating an app section of the Slack API Quickstart to create the app.
Follow the Create an incoming webhook section of the Slack Incoming Webhooks guide to enable incoming webhooks for your app and create a webhook for your workspace.
Copy the Webhook URL that starts with 
https://hooks.slack.com/services/. You need this when setting up your contact point integration in Grafana Alerting.
Procedure
To create your Slack integration in Grafana Alerting, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a contact point name.
From the Integration list, select Slack.
If you are using a Slack API token:
In the Recipient field, copy in the channel ID.
In the Token field, copy in the Bot User OAuth Token that starts with “xoxb-”.
If you are using a Webhook URL, in the Webhook field, copy in your Slack app Webhook URL.
Click Test to check that your integration works.
** For Grafana Alertmanager only.**
Click Save contact point.
Optional settings
You can configure additional settings to customize your Slack notifications beyond the required fields.
Customize bot appearance
You can customize how the bot appears in Slack channels:
Option
Description
Username
Override the default bot username that appears in Slack. Set the 
username field to customize the display name.
Icon emoji
Provide an emoji to use as the icon for the message. Set the 
icon_emoji field (for example, 
:rocket:). This overrides the icon URL.
Icon URL
Provide a URL to an image to use as the icon for the message. Set the 
icon_url field with the image URL.
Configure mentions
You can automatically mention users, groups, or the entire channel when notifications are sent:
Option
Description
Mention users
Mention one or more specific users in the notification. Set the 
mentionUsers field with a comma-separated list of user IDs (for example, 
U024BE7LH,U024BE7LJ). To find a user's ID, refer to Slack's documentation on finding member IDs.
Mention groups
Mention one or more user groups in the notification. Set the 
mentionGroups field with a comma-separated list of group IDs. You can copy group IDs from the group's Slack profile URL.
Mention channel
Mention the entire channel or active members. Set the 
mentionChannel field to one of the following:
- Empty or omit the field to disable channel mentions
- 
here to mention every active channel member
- 
channel to mention every channel member
Customize message content
You can customize the notification message using notification templates:
Option
Description
Title
Set a custom title for the Slack message. Use the 
title field with template variables (for example, 
{{ template "slack.default.title" . }}).
Text body
Set a custom body text for the Slack message. Use the 
text field with template variables (for example, 
{{ template "slack.default.text" . }}).
Color
Set the color of the vertical bar on the left side of the message attachment. Use the 
color field.
Advanced configuration
For specialized environments, you can override the default Slack API endpoint:
Option
Description
Endpoint URL
Specify a custom Slack API endpoint for non-webhook requests. Set the 
endpointUrl field to your custom endpoint. The default is 
https://slack.com/api/chat.postMessage.
Next steps
The Slack contact point is ready to receive alert notifications.
To add this contact point to your alert, complete the following steps.
In Grafana, navigate to Alerting > Alert rules.
Edit or create a new alert rule.
Scroll down to the Configure labels and notifications section.
Under Notifications click Select contact point.
From the drop-down menu, select the previously created contact point.
Click Save rule and exit.

Configure Microsoft Teams for Alerting
Use the Grafana Alerting - Microsoft Teams integration to receive notifications in your team’s channel when your alerts are firing.
Note that you can customize the 
title and 
message of the notification using notification templates; however, you cannot modify its visual appearance with adaptive cards.
Before you begin
To set up Microsoft Teams for integration with Grafana Alerting, create a new workflow that accepts Webhook requests. This allows Grafana to send alert notifications to Microsoft Teams channels.
Create a workflow in Microsoft Teams
To create a new workflow, follow the steps in Create flows in Microsoft Teams.
Microsoft provides a template library. You can use the template Post to a channel when a webhook request is received.
At the end of workflow creation wizard, copy the URL that is provided.
Note
If you chose a private channel for the target of the workflow, you need to edit workflow before using it. Expand the step "Send each adaptive card", and then expand action "Post your own adaptive card as the Flow bot to a channel". Change "Post as" to User, and save the workflow.
Procedure
To create your MS Teams integration in Grafana Alerting, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a contact point name.
From the Integration list, select Microsoft Teams.
In the URL field, copy in your Webhook URL.
Click Test to check that your integration works.
** For Grafana Alertmanager only.**
A test alert notification should be sent to the MS Team channel.
Click Save contact point.
Next steps
The Microsoft Teams contact point is ready to receive alert notifications.
To add this contact point to your alert, complete the following steps.
In Grafana, navigate to Alerting > Alert rules.
Edit or create a new alert rule.
Scroll down to the Configure labels and notifications section.
Under Notifications click Select contact point.
From the drop-down menu, select the previously created contact point.
Click Save rule and exit.
Troubleshooting
If Grafana reports that the notification was sent successfully but it was not delivered to the channel, check the workflow's run history. You can find it in the workflow details page.

Configure Telegram for Alerting
Use the Grafana Alerting - Telegram integration to send Telegram notifications when your alerts are firing.
Before you begin
Telegram limitation
Telegram messages are limited to 4096 UTF-8 characters. If you use a 
parse_mode other than 
None, truncation may result in an invalid message, causing the notification to fail.
For longer messages, we recommend using an alternative contact method.
Telegram bot API token and chat ID
To integrate Grafana with Telegram, you need to get a Telegram bot API token and a chat ID (the ID of the Telegram chat where you want to receive the alert notifications). To complete the integration, use the browser version of Telegram.
Set up your Telegram bot
Create a Telegram bot. You can associate this bot to your chats and perform different actions with it, such as receiving alerts from Grafana.
To set up the bot, complete the following steps.
Open the Telegram app on your device.
Find the Telegram bot named BotFather.
Type or press 
/newbot.
Choose a name for the bot. It must end in bot or _bot. E.g. "my_bot".
Copy the API token.
Chat ID
Add the bot to a group chat by following the steps below. Once the bot is added to the chat, you will be able to route your alert notifications to that group.
In the Telegram app, open a group or start a new one.
Search and add the bot to the group.
Copy the chat ID from the URL in your browser's address bar. It should look like this: 
https://web.telegram.org/a/#-4266674385.
The chat ID is the sequence of numbers that follows the 
# symbol. For example: 
-4266674385.
Procedure
To create your Telegram integration in Grafana Alerting, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a contact point name.
From the Integration list, select Telegram.
In the BOT API Token field, copy in the bot API token.
In the Chat ID field, copy in the chat ID.
Click Test to check that your integration works.
** For Grafana Alertmanager only.**
Click Save contact point.
Next steps
The Telegram contact point is ready to receive alert notifications.
To add this contact point to your alert, complete the following steps.
In Grafana, navigate to Alerting > Alert rules.
Edit or create a new alert rule.
Scroll down to the Configure labels and notifications section.
Under Notifications click Select contact point.
From the drop-down menu, select the previously created contact point.
Click Save rule and exit.

Configure PagerDuty for Alerting
Use the Grafana Alerting - PagerDuty integration to receive notifications in PagerDuty when your alerts are firing.
Before you begin
To set up PagerDuty for integration with Grafana Alerting, you need to create a PagerDuty account. There are several set up steps to perform within PagerDuty before you set up the integration in Grafana Alerting.
Create a Service
In PagerDuty, a service represents a component, microservice, or infrastructure element that a team oversees, manages, and monitors.
Refer to PagerDuty’s services and integrations guide.
Follow steps 1 to 5 under Create a Service.
{{< admonition type="note" >}}
In step 5, choose Create a service without an integration.
{{< /admonition >}}
Obtain a PagerDuty integration key
Once the service is created, click Integrations within the Service options.
Click + Add an integration.
Select Events API V2.
Click Add.
Click the drop-down arrow to display the integration details.
Copy the Integration Key.
Procedure
To create your PagerDuty integration in Grafana Alerting, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a contact point name.
From the Integration list, select PagerDuty.
In the Integration Key field, copy in your integration key.
Click Test to check that your integration works.
** For Grafana Alertmanager only.**
An incident should display in the Service’s Activity tab in PagerDuty.
Click Save contact point.
Next steps
The PagerDuty contact point is ready to receive alert notifications.
To add this contact point to your alert, complete the following steps.
In Grafana, navigate to Alerting > Alert rules.
Edit or create a new alert rule.
Scroll down to the Configure labels and notifications section.
Under Notifications click Select contact point.
From the drop-down menu, select the previously created contact point.
Click Save rule and exit.

Configure webhook notifications
Use the webhook integration in contact points to send alert notifications to your webhook.
The webhook integration is a flexible way to integrate alerts into your system. When a notification is triggered, it sends a JSON request with alert details and additional data to the webhook endpoint.
Configure webhook for a contact point
To create a contact point with webhook integration, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Click + Add contact point.
Enter a name for the contact point.
From the Integration list, select Webhook.
In the URL field, copy in your Webhook URL.
(Optional) Configure additional settings.
Click Save contact point.
For more details on contact points, including how to test them and enable notifications, refer to Configure contact points.
Webhook settings
Option
Description
URL
The Webhook URL. This field is protected from modification in Grafana Cloud.
{{< admonition type="caution" >}}
The URL is not a secured field. It's stored in plain text and isn't masked when the contact point is read back. Include secrets such as tokens or passwords in the URL at your own risk.
To authenticate to your webhook endpoint, use the authorization fields in the optional settings instead, such as Basic Authentication or the Authorization request header. These fields are stored securely and are masked on read.
{{< /admonition >}}
Optional settings
Option
Description
HTTP Method
Specifies the HTTP method to use: 
POST or 
PUT.
Basic Authentication Username
Username for HTTP Basic Authentication.
Basic Authentication Password
Password for HTTP Basic Authentication.
Authentication Header Scheme
Scheme for the 
Authorization Request Header. Default is 
Bearer.
Authentication Header Credentials
Credentials for the 
Authorization Request header.
Extra Headers
Additional HTTP headers to include in the request. You can also override the default 
Content-Type: application/json header to specify a different content type for the request payload.
Max Alerts
Maximum number of alerts to include in a notification. Any alerts exceeding this limit are ignored. 
0 means no limit.
TLS
TLS configuration options, including CA certificate, client certificate, and client key.
HMAC Signature
HMAC signature configuration options.
OAuth2
OAuth2 configuration options for authenticating the webhook request. Refer to OAuth2 authentication for details.
{{< admonition type="note" >}}
You can configure either HTTP Basic Authentication or the Authorization request header, but not both.
{{< /admonition >}}
HMAC signature
You can secure your webhook notifications using HMAC signatures to verify the authenticity and integrity of the requests. When enabled, Grafana signs the webhook payload with a shared secret using HMAC-SHA256.
Option
Description
Secret
The shared secret key used to generate the HMAC signature.
Header
The HTTP header where the signature will be set. Default is 
X-Grafana-Alerting-Signature.
Timestamp Header
Optional header to include a timestamp in the signature calculation. When specified, Grafana will set a Unix timestamp in this header and include it in the HMAC calculation. This provides protection against replay attacks.
When HMAC signing is configured, Grafana generates a signature using HMAC-SHA256 with your secret key. If a timestamp header is specified, a Unix timestamp is included in the signature calculation. The signature is calculated as:
HMAC(timestamp + ":" + body)
The timestamp is sent in the specified header. If no timestamp header is specified, the signature is calculated just from the request body. The signature is sent as a hex-encoded string in the specified signature header.
Validate a request
To validate incoming webhook requests from Grafana, follow these steps:
Extract the signature from the header (default is 
X-Grafana-Alerting-Signature).
If you configured a timestamp header, extract the timestamp value and verify it's recent to prevent replay attacks.
Calculate the expected signature:
Create an HMAC-SHA256 hash using your shared secret
If using timestamps, include the timestamp followed by a colon (
:) before the request body
Hash the raw request body
Convert the result to a hexadecimal string
Compare the calculated signature with the one in the request header.
OAuth2 authentication
You can authenticate webhook requests using OAuth2 client credentials. When configured, Grafana requests an access token from the token URL using the client ID and secret, then includes the token as a bearer token in the 
Authorization header of the webhook request.
{{< admonition type="note" >}}
If you configure OAuth2 authentication, it takes precedence over HTTP Basic Authentication and the Authorization request header for the webhook request.
{{< /admonition >}}
Option
Description
Token URL
The URL of the OAuth2 token endpoint. This field is protected from modification in Grafana Cloud.
Client ID
The OAuth2 client ID.
Client Secret
The OAuth2 client secret.
Scopes
Optional list of OAuth2 scopes to request when obtaining an access token.
Endpoint Parameters
Optional key-value pairs to include in the access token request.
TLS
Optional TLS configuration used only for the OAuth2 token request, including CA certificate, client certificate, and client key.
Proxy
Optional HTTP proxy configuration used only for the OAuth2 token request. The proxy URL is protected from modification in Grafana Cloud.
Optional settings using templates
Use the following settings to include custom data within the JSON payload. Both options support using notification templates.
Option
Description
Title
Sends the value as a string in the 
title field of the JSON payload. Supports notification templates.
Message
Sends the value as a string in the 
message field of the JSON payload. Supports notification templates.
Custom Payload
Optionally override the default payload format with a custom template.
Optional notification settings
Option
Description
Disable resolved message
Enable this option to prevent notifications when an alert resolves.
Default JSON payload
The following example shows the payload of a webhook notification containing information about two firing alerts:
{
  "receiver": "My Super Webhook",
  "status": "firing",
  "orgId": 1,
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High memory usage",
        "team": "blue",
        "zone": "us-1"
      },
      "annotations": {
        "description": "The system has high memory usage",
        "runbook_url": "https://myrunbook.com/runbook/1234",
        "summary": "This alert was triggered for zone us-1"
      },
      "startsAt": "2021-10-12T09:51:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/1afz29v7z/edit",
      "fingerprint": "c6eadffa33fcdf37",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1",
      "dashboardURL": "",
      "panelURL": "",
      "values": {
        "B": 44.23943737541908,
        "C": 1
      }
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "High CPU usage",
        "team": "blue",
        "zone": "eu-1"
      },
      "annotations": {
        "description": "The system has high CPU usage",
        "runbook_url": "https://myrunbook.com/runbook/1234",
        "summary": "This alert was triggered for zone eu-1"
      },
      "startsAt": "2021-10-12T09:56:03.157076+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://play.grafana.org/alerting/d1rdpdv7k/edit",
      "fingerprint": "bc97ff14869b13e3",
      "silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1",
      "dashboardURL": "",
      "panelURL": "",
      "values": {
        "B": 44.23943737541908,
        "C": 1
      }
    }
  ],
  "groupLabels": {},
  "commonLabels": {
    "team": "blue"
  },
  "commonAnnotations": {},
  "externalURL": "https://play.grafana.org/",
  "version": "1",
  "groupKey": "{}:{}",
  "truncatedAlerts": 0,
  "title": "[FIRING:2]  (blue)",
  "state": "alerting",
  "message": "**Firing**\n\nLabels:\n - alertname = T2\n - team = blue\n - zone = us-1\nAnnotations:\n - description = This is the alert rule checking the second system\n - runbook_url = https://myrunbook.com\n - summary = This is my summary\nSource: https://play.grafana.org/alerting/1afz29v7z/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1\n\nLabels:\n - alertname = T1\n - team = blue\n - zone = eu-1\nAnnotations:\nSource: https://play.grafana.org/alerting/d1rdpdv7k/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1\n"
}
Body
The JSON payload of webhook notifications includes the following key-value pairs:
Key
Type
Description
receiver
string
Name of the contact point.
status
string
Current status of the alert, 
firing or 
resolved.
orgId
number
ID of the organization related to the payload.
alerts
array of alerts
Alerts that are triggering.
groupLabels
object
Labels that are used for grouping, map of string keys to string values.
commonLabels
object
Labels that all alarms have in common, map of string keys to string values.
commonAnnotations
object
Annotations that all alarms have in common, map of string keys to string values.
externalURL
string
External URL to the Grafana instance sending this webhook.
version
string
Version of the payload structure.
groupKey
string
Key that is used for grouping.
truncatedAlerts
number
Number of alerts that were truncated.
state
string
State of the alert group (either 
alerting or 
ok).
The following key-value pairs are also included in the JSON payload and can be configured in the webhook settings using notification templates.
Key
Type
Description
title
string
Custom title. Configurable in webhook settings using notification templates.
message
string
Custom message. Configurable in webhook settings using notification templates.
Alert
The Alert object represents an alert included in the notification group, as provided by the 
alerts field.
{{< docs/shared lookup="alerts/table-for-json-alert-object.md" source="grafana" version="<GRAFANA_VERSION>" >}}
Custom Payload
The 
Custom Payload option allows you to completely customize the webhook payload using templates. This gives you full control over the structure and content of the webhook request.
For detailed information about how to create and manage notification templates, refer to notification templates.
{{< admonition type="note" >}}
When using Custom Payload, the 
Title and 
Message fields are ignored as the entire payload structure is determined by your template.
Custom Payload is not yet generally available in Grafana Cloud.
{{< /admonition >}}
Option
Description
Payload Template
Notification template that defines the structure of the webhook payload.
Payload Variables
Key-value pairs that define additional variables available in the template under 
.Vars.<variable_name>.
Example of a custom payload template that includes variables:
{
  "alert_name": "{{ .CommonLabels.alertname }}",
  "status": "{{ .Status }}",
  "environment": "{{ .Vars.environment }}",
  "custom_field": "{{ .Vars.custom_field }}"
}
JSON Template Functions
When creating custom payloads, several template functions are available to help generate valid JSON structures. These include functions for creating dictionaries (
coll.Dict), arrays (
coll.Slice, 
coll.Append), and converting between JSON strings and objects (
data.ToJSON, 
data.JSON).
For detailed information about these and other template functions, refer to notification template functions.
Example using JSON helper functions:
{{< docs/shared lookup="alerts/example-custom-json-payload.md" source="grafana" version="<GRAFANA_VERSION>" >}}

Notification template examples
Notification templates allow you to change the default notification messages.
You can modify the content and format of notification messages. For example, you can customize the content to show only specific information or adjust the format to suit a particular contact point, such as Slack or Email.
{{< admonition type="note" >}}
Avoid adding extra information about alert instances in notification templates, as this information is only visible in the notification message.
Instead, you should use annotations or labels to add information directly to the alert, ensuring it's also visible in the alert state and alert history within Grafana. You can then print the new alert annotation or label in notification templates.
{{< /admonition >}}
This page provides various examples illustrating how to template common notification messages. For more details about notification templates, refer to:
Template notifications
Select, create, and preview a notification template
Notification template reference
Basic examples
Notification templates can access the notification data using the dot (
.). The following examples demonstrate some basic uses of the template language.
Text formatting examples
Text formatting in notification templates varies by contact point type. Each platform has its own formatting syntax:
Slack formatting
For Slack contact points, you can use:
{{ define "slack_formatted_message" -}}
*This text will be bold in Slack*
_This text will be italic in Slack_
Regular text without formatting
{{ end }}
Execute the template:
{{ template "slack_formatted_message" . }}
*This text will be bold in Slack*
_This text will be italic in Slack_
Regular text without formatting
Additional contact point types
{{< admonition type="note" >}}
Each contact point type has its own formatting requirements:
Some support Markdown-style formatting
Others use HTML tags
Some may not support formatting at all
Check your specific contact point's documentation for supported formatting syntax.
{{< /admonition >}}
Template logic examples
Beyond text formatting, you can use template logic to create dynamic notification content. The following examples demonstrate basic template operations:
Conditional logic
To check if there are common labels (
.CommonLabels) for all alerts in the notification, use 
if:
{{ define "custom_message" -}}
{{ if .CommonLabels }}
Alerts have common labels
{{ else }}
There are no common labels
{{ end }}
{{ end }}
Iterating over alerts
To iterate on the alerts in the notification and print a specific label, use 
range and 
index:
{{ define "custom_message" -}}
{{ range .Alerts }}
The name of the alert is {{ index .Labels "alertname" }}
{{ end }}
{{ end }}
Alternatively, you can use the 
. notation to print the value of the key.
{{ define "custom_message" -}}
{{ range .Alerts }}
The name of the alert is {{ .Labels.alertname }}
{{ end }}
{{ end }}
You can then use the template by passing the notification data (dot 
.):
{{ template "custom_message" . }}
The name of the alert is InstanceDown
The name of the alert is CpuOverload
Print alerts with summary and description
Here's an example that displays the summary and description annotations for each alert in the notification.
{{ define "custom.alerts" -}}
{{ len .Alerts }} alert(s)
{{ range .Alerts -}}
  {{ template "alert.summary_and_description" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.summary_and_description" }}
  Summary: {{.Annotations.summary}}
  Status: {{ .Status }}
  Description: {{.Annotations.description}}
{{ end -}}
In this example:
A template (
alert.summary_and_description) is defined to print the 
summary, 
status, and 
description of one alert.
The main template 
custom.alerts iterates the list of alerts (
.Alerts) in notification data, executing the 
alert.summary_and_description template to print the details of each alert.
Execute the template by passing the dot (
.):
{{ template "custom.alerts" . }}
2 alert(s)
  Summary: The database server db1 has exceeded 75% of available disk space.
  Status: firing
  Description: This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption.
  Summary: The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.
  Status: resolved
  Description: This alert fires when a web server responds with more 5xx errors than expected. This could be an issue with the web server or a backend service.
Print firing and resolved alerts
The following example is similar to the previous one, but it separates firing and resolved alerts.
{{ define "custom.firing_and_resolved_alerts" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
  {{ template "alert.summary_and_description" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
  {{ template "alert.summary_and_description" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.summary_and_description" }}
  Summary: {{.Annotations.summary}}
  Status: {{ .Status }}
  Description: {{.Annotations.description}}
{{ end -}}
Instead of 
.Alerts, the template accesses 
.Alerts.Firing and 
.Alerts.Resolved separately to print details for each alert.
Run the template by passing the dot (
.):
{{ template "custom.firing_and_resolved_alerts" . }}
1 resolved alert(s)
  Summary: The database server db1 has exceeded 75% of available disk space.
  Status: resolved
  Description: This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption.
1 firing alert(s)
  Summary: The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.
  Status: firing
  Description: This alert fires when a web server responds with more 5xx errors than expected. This could be an issue with the web server or a backend service.
Print common labels and annotations
This example displays only the labels and annotations that are common to all alerts in the notification.
{{ define "custom.common_labels_and_annotations" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ len .Alerts.Firing }} firing alert(s)
Common labels: {{ len .CommonLabels.SortedPairs }}
{{ range .CommonLabels.SortedPairs -}}
- {{ .Name }} = {{ .Value }}
{{ end }}
Common annotations: {{ len .CommonAnnotations.SortedPairs }}
{{ range .CommonAnnotations.SortedPairs }}
- {{ .Name }} = {{ .Value }}
{{ end }}
{{ end -}}
Note that 
.CommonAnnotations and 
.CommonLabels are part of notification data.
Execute the template by passing the dot (
.) as argument:
{{ template "custom.common_labels_and_annotations" . }}
1 resolved alert(s)
1 firing alert(s)
Common labels: 2
- grafana_folder = server_alerts
- team = server_admin
Common annotations: 0
Print individual labels and annotations
This example displays all labels and annotations for each alert in the notification.
{{ define "custom.alert_labels_and_annotations" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
  {{ template "alert.labels_and_annotations" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
  {{ template "alert.labels_and_annotations" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.labels_and_annotations" }}
Alert labels: {{ len .Labels.SortedPairs }}
{{ range .Labels.SortedPairs -}}
- {{ .Name }} = {{ .Value }}
{{ end -}}
Alert annotations: {{ len .Annotations.SortedPairs }}
{{ range .Annotations.SortedPairs -}}
- {{ .Name }} = {{ .Value }}
{{ end -}}
{{ end -}}
In this example:
The 
custom.alert_labels_and_annotations template iterates over the list of resolved and firing alerts, similar to previous examples. It then executes 
alert.labels_and_annotations for each alert.
The 
alert.labels_and_annotations template prints all the alert labels and annotations by accessing 
.Labels.SortedPairs and 
.Annotations.SortedPairs.
Run the template by passing the dot (
.):
{{ template "custom.alert_labels_and_annotations" . }}
1 resolved alert(s)
Alert labels: 4
- alertname = db_server_disk_space
- grafana_folder = server_alerts
- server = db1
- team = server_admin
Alert annotations: 2
- summary = The database server db1 has exceeded 75% of available disk space.
- description = This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption.
1 firing alert(s)
Alert labels: 4
- alertname = web_server_http_errors
- grafana_folder = server_alerts
- server = web1
- team = server_admin
Alert annotations: 2
- summary = The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.
- description = This alert fires when a web server responds with more 5xx errors than expected. This could be an issue with the web server or a backend service.
Print URLs for runbook and alert data in Grafana
Note that the following example works only for Grafana-managed alerts. It displays some alert data such as 
DashboardURL, 
PanelURL, and 
SilenceURL, which are exclusive to Grafana-managed alerts.
{{ define "custom.alert_additional_details" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
  {{ template "alert.additional_details" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
  {{ template "alert.additional_details" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.additional_details" }}
- Dashboard: {{ .DashboardURL }}
- Panel: {{ .PanelURL }}
- AlertGenerator: {{ .GeneratorURL }}
- Silence: {{ .SilenceURL }}
- RunbookURL: {{ .Annotations.runbook_url}}
{{ end -}}
Pass the dot (
.) to execute the template:
{{ template "custom.alert_additional_details" . }}
1 resolved alert(s)
- Dashboard: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395
- Panel: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395&viewPanel=31
- AlertGenerator: ?orgId=1
- Silence: https://example.com/alerting/silence/new
- RunbookURL: https://example.com/on-call/db_server_disk_space
1 firing alert(s)
- Dashboard: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395
- Panel: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395&viewPanel=31
- AlertGenerator: ?orgId=1
- Silence: https://example.com/alerting/silence/new
- RunbookURL: https://example.com/on-call/web_server_http_errors
Print a notification title or subject
A title or subject provides a one-line summary of the notification content.
Here’s a basic example that displays the number of firing and resolved alerts in the notification.
{{ define "custom_title" -}}
{{ if gt (.Alerts.Firing | len) 0 }}🚨 {{ .Alerts.Firing | len }} firing alerts. {{ end }}{{ if gt (.Alerts.Resolved | len) 0 }}✅ {{ .Alerts.Resolved | len }} resolved alerts.{{ end }}
{{ end -}}
Execute the template by passing the dot (
.) as argument:
{{ template "custom_title" . }}
🚨 1 firing alerts. ✅ 1 resolved alerts.
The next example is a copy of the default title/subject template used in Grafana.
{{ define "copy_of_default_title" -}}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ if gt (.Alerts.Resolved | len) 0 }}, RESOLVED:{{ .Alerts.Resolved | len }}{{ end }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}
{{ end }}
This is a more advanced example:
Prints the number of firing and resolved alerts in the notification.
Outputs 
.GroupLabels, the labels used to group multiple alerts in one notification.
Prints 
CommonLabels, excluding labels in 
.GroupLabels.
Execute the template by passing the dot (
.):
{{ template "copy_of_default_title" . }}
[FIRING:1, RESOLVED:1] api warning (sql_db)
Print a link to a dashboard with time range
You can include a link to a dashboard or panel in your alert notifications. This is useful when the alert rule is created from a dashboard panel or monitors a target visualized in an existing dashboard.
Including a dashboard link in the notification helps responders quickly navigate to the relevant context for investigation.
Use one of the following methods to include a dashboard link with the correct time range in the alert notification:
You can link the alert rule to a panel. This includes the dashboard and panel URLs via 
{{.Alert.DashboardURL}} and 
{{.Alert.PanelURL}}.
{{ define "custom.link_to_dashboard" -}}
{{ range .Alerts -}}
  Dashboard: {{.DashboardURL}}
  Panel: {{ .PanelURL }}
{{ end -}}
{{ end -}}
Run the template using:
{{ template "custom.link_to_dashboard" . }}
Dashboard: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395
Panel: https://example.com/d/uiyahbsdaubsd?from=1740070380000&orgId=1&to=1740074106395&viewPanel=31
These URLs include a time range based on the alert’s timing:
from: One hour before the alert started.
to: The current time if the alert is firing, or the alert’s end time if resolved.
Alternatively, you can use a custom annotation to set the dashboard URL and build the full URL using the 
from and 
to query parameters derived from 
{{.Alert.StartsAt}} and 
{{.Alert.EndsAt}}.
{{ define "custom.my_dashboard_url_annotation" -}}
{{ range .Alerts -}}
  {{/* StartsAt - 1h */}}
  {{- $from := (.StartsAt.Add -3600000000000).UnixMilli }}
  {{- $to := "" }}
  {{- if eq .Status "resolved" }}
     {{- $to = (.EndsAt).UnixMilli }}
  {{- else -}}
    {{/* Use current time if alert is firing */}}
    {{- $to = (time.Now).UnixMilli }}
  {{- end -}}
  Dashboard: {{.Annotations.MyDashboardURL}}?from={{$from}}&to={{$to}}
{{ end }}
{{ end }}
To use this template, define a custom annotation named 
MyDashboardURL that contains the base dashboard URL without 
from and 
to parameters. For example: 
http://localhost:3000/d/uiyahbsdaubsd.
Run the template using:
{{ template "custom.my_dashboard_url_annotation" . }}
Dashboard: http://localhost:3000/d/uiyahbsdaubsd?from=1740070380000&to=1740071880000
Custom JSON payload
The custom payload option in the webhook contact point allows you to customize the payload of webhook notifications using a custom template.
The following example generates a custom JSON payload by executing other templates with 
tmpl.Exec, and using functions like 
coll.Dict and 
data.ToJSON to process and format JSON data.
{{< docs/shared lookup="alerts/example-custom-json-payload.md" source="grafana" version="<GRAFANA_VERSION>" >}}
{
 "alerts": [
  {
   "endsAt": "0001-01-01T00:00:00Z",
   "labels": {
    "alertname": "InstanceDown",
    "grafana_folder": "Test Folder",
    "instance": "instance1"
   },
   "startsAt": "2025-04-21T10:19:46.179Z",
   "status": "firing"
  },
  {
   "endsAt": "2025-04-22T10:19:46.179Z",
   "labels": {
    "alertname": "CpuUsage",
    "grafana_folder": "Test Folder",
    "instance": "instance1"
   },
   "startsAt": "2025-04-22T06:19:46.179Z",
   "status": "resolved"
  }
 ],
 "allVariables": {},
 "commonAnnotations": {},
 "commonLabels": {
  "grafana_folder": "Test Folder",
  "instance": "instance1"
 },
 "externalURL": "http://localhost:3000/",
 "groupKey": "",
 "groupLabels": {
  "group_label": "group_label_value"
 },
 "message": "**Firing**\n\nValue: B=22, C=1\nLabels:\n - alertname = InstanceDown\n - grafana_folder = Test Folder\n - instance = instance1\nAnnotations:\n - summary = Instance instance1 has been down for more than 5 minutes\nSource: http://grafana.com/alerting/grafana/cdeqmlhvflz40f/view?orgId=1\nSilence: http://localhost:3000/alerting/silence/new?alertmanager=grafana\u0026matcher=alertname%3DInstanceDown\u0026matcher=grafana_folder%3DTest+Folder\u0026matcher=instance%3Dinstance1\u0026orgId=1\nDashboard: http://localhost:3000/d/dashboard_uid?from=1745227186179\u0026orgId=1\u0026to=1745317189058\nPanel: http://localhost:3000/d/dashboard_uid?from=1745227186179\u0026orgId=1\u0026to=1745317189058\u0026viewPanel=1\n\n\n**Resolved**\n\nValue: B=22, C=1\nLabels:\n - alertname = CpuUsage\n - grafana_folder = Test Folder\n - instance = instance1\nAnnotations:\n - summary = CPU usage above 90%\nSource: http://grafana.com/alerting/grafana/oZSMdGj7z/view?orgId=1\nSilence: http://localhost:3000/alerting/silence/new?alertmanager=grafana\u0026matcher=alertname%3DCpuUsage\u0026matcher=grafana_folder%3DTest+Folder\u0026matcher=instance%3Dinstance1\u0026orgId=1\nDashboard: http://localhost:3000/d/dashboard_uid?from=1745299186179\u0026orgId=1\u0026to=1745317186179\nPanel: http://localhost:3000/d/dashboard_uid?from=1745299186179\u0026orgId=1\u0026to=1745317186179\u0026viewPanel=1\n",
 "orgId": 1,
 "receiver": "TestReceiver",
 "state": "alerting",
 "status": "firing",
 "title": "[FIRING:1, RESOLVED:1] group_label_value (Test Folder instance1)",
 "truncatedAlerts": null,
 "version": "1"
}

Use images in notifications
{{< admonition type="note" >}}
Grafana Cloud users can request this feature by opening a support ticket in the Cloud Portal.
{{< /admonition >}}
Images in notifications help recipients of alert notifications better understand why an alert has fired or resolved by including a screenshot of the panel associated with the alert.
{{< admonition type="note" >}}
This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager.
{{< /admonition >}}
When an alert is fired or resolved, Grafana takes a screenshot of the panel associated with the alert. This is determined via the Dashboard UID and Panel ID annotations of the rule. Grafana cannot take a screenshot for alerts that are not associated with a panel.
Grafana takes at most two screenshots for each alert: once when the alert fires and again when the alert is resolved. Screenshots are not re-taken over the lifetime of the alert. Instead, you should open the panel in Grafana to follow the data in real time. In addition, depending on how alerts are grouped in your notification policies, Grafana might send a notification with many screenshots of the same panel. This happens because Grafana does not know how your alerts are grouped at the time a screenshot is taken, and so acts conservatively by taking a screenshot for every alert.
After a screenshot has been taken, Grafana can either upload it to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; upload the screenshot to its internal web server; or upload it to the service that is receiving the notification, such as Slack. Which option you should choose depends on how your Grafana is managed and which integrations you use. More information on this can be found in Requirements.
Refer to the table at the end of this page for a list of contact points and their support for images in notifications.
Requirements
To use images in notifications, Grafana OSS and Enterprise must be set up to use the Grafana image renderer service.
When a screenshot is taken, it is saved to the [data][paths] folder, even if Grafana is configured to upload screenshots to a cloud storage service. Grafana must have write-access to this folder, otherwise screenshots cannot be saved to disk and an error is logged for each failed screenshot attempt.
You should use a cloud storage service unless sending alerts to Discord, email, Pushover, Slack or Telegram. These integrations support either embedding screenshots in the email or attaching screenshots to the notification, while other integrations must link screenshots uploaded to a cloud storage bucket. If a cloud storage service has been configured then integrations that support both link screenshots from the cloud storage bucket instead of embedding or attaching screenshots to the notification.
If uploading screenshots to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; and accessing screenshots in the bucket requires authentication, logging into a VPN or corporate network; then image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN. If this happens, we recommend using integrations which support uploading images or disabling images in notifications altogether.
When uploading screenshots to a cloud storage service Grafana uses a random 20 character (30 characters for Azure Blob Storage) filename for each image. This makes URLs hard to guess but not impossible.
Grafana does not delete screenshots from cloud storage. We recommend configuring a retention policy with your cloud storage service to delete screenshots older than 1 month.
If Grafana is configured to upload screenshots to its internal web server, and accessing Grafana requires logging into a VPN or corporate network; image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN. If this happens, we recommend using integrations which support uploading images or disabling images in notifications altogether.
Grafana does not delete screenshots uploaded to its internal web server. To delete screenshots from 
static_root_path/images/attachments after a certain amount of time, we recommend setting up a CRON job.
Note you cannot adjust the number and size of images or their placement in notifications.
Configuration
{{< admonition type="note" >}}
Grafana Cloud users can request this feature by opening a support ticket in the Cloud Portal.
{{< /admonition >}}
Having set up Grafana to use the image renderer service, set 
capture in 
[unified_alerting.screenshots] to 
true:
# Enable screenshots in notifications. You must have either installed the Grafana image rendering
# plugin, or set up Grafana to use a remote rendering service.
# For more information on configuration options, refer to [rendering].
capture = false
If screenshots should be uploaded to cloud storage then 
upload_external_image_storage should also be set to 
true:
# Uploads screenshots to the local Grafana server or remote storage such as Azure, S3 and GCS. Please
# see [external_image_storage] for further configuration options. If this option is false, screenshots
# are persisted to disk for up to temp_data_lifetime.
upload_external_image_storage = false
Restart Grafana for the changes to take effect.
Advanced configuration
We recommend that 
max_concurrent_screenshots is less than or equal to 
concurrent_render_request_limit. The default value for both 
max_concurrent_screenshots and 
concurrent_render_request_limit is 
5:
# The maximum number of screenshots that can be taken at the same time. This option is different from
# concurrent_render_request_limit as max_concurrent_screenshots sets the number of concurrent screenshots
# that can be taken at the same time for all firing alerts where as concurrent_render_request_limit sets
# the total number of concurrent screenshots across all Grafana services.
max_concurrent_screenshots = 5
Supported contact points
Grafana supports a wide range of contact points with varied support for images in notifications. The table below shows the list of all contact points supported in Grafana and their support for uploading screenshots to the receiving service and referencing screenshots that have been uploaded to a cloud storage service.
Name
Upload from disk
Reference from cloud storage
DingDing
No
No
Discord
Yes (Maximum of 10 per notification)
Yes (Maximum of 10 per notification)
Email
Yes (Embedded in the email)
Yes
Google Chat
No
Yes
Kafka
No
No
Line
No
No
MQTT
No
No
Microsoft Teams
No
Yes
Opsgenie
No
Yes
PagerDuty
No
Yes
Prometheus Alertmanager
No
No
Pushover
Yes (Maximum of 1 per notification)
No
Sensu Go
No
No
Slack
Yes (when using Bot tokens, maximum of 5 per notification)
Yes (when using webhooks, maximum of 1 per notification)
Telegram
Yes
No
Threema
No
No
VictorOps
No
No
Webhook
No
Yes
Limitations
This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager.
This feature is not supported when using custom templates in email notifications.
A number of contact points support at most one image per notification. In this case, just the first image is either uploaded to the receiving service or referenced from cloud storage per notification.
When multiple alerts are sent in a single notification a screenshot might be included for each alert. The order the images are shown is random.
If uploading screenshots to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; and accessing screenshots in the bucket requires authentication, logging into a VPN or corporate network; image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN.
Troubleshooting
If Grafana has been set up to send images in notifications, however notifications are still being received without them, follow the troubleshooting steps below:
Check that images in notifications has been set up as per the instructions.
Enable debug logging in Grafana and look for logs with the logger 
ngalert.image.
If the alert is not associated with a dashboard there are logs for 
Cannot take screenshot for alert rule as it is not associated with a dashboard.
If the alert is associated with a dashboard, but no panel in the dashboard, there are logs for 
Cannot take screenshot for alert rule as it is not associated with a panel.
If images cannot be taken because of mis-configuration or an issue with image rendering, there are logs for 
Failed to take an image including the Dashboard UID, Panel ID, and the error message.
Check that the contact point supports images in notifications and whether it supports uploading images to the receiving service or referencing images that have been uploaded to a cloud storage service.
Monitor
Grafana provides the following metrics to observe the performance and failure rate of images in notifications.
For example, if a screenshot could not be taken within the expected time (10 seconds) then the counter 
grafana_screenshot_failures_total is updated.
grafana_alerting_image_cache_hits_total
grafana_alerting_image_cache_misses_total
grafana_screenshot_duration_seconds
grafana_screenshot_failures_total
grafana_screenshot_successes_total
grafana_screenshot_upload_failures_total
grafana_screenshot_upload_successes_total

Alerting template language
Notification templates and alert rule templates, such as annotations and labels, both use the Go template language, text/template.
Both types of templates can use the same keywords, functions, and comparison operators of the Go template language, such as 
range, 
if, 
and, 
index, 
eq, and more.
However, it's important to note that because notifications and alert rules operate in distinct contexts, some additional variables and functions are only available for either notification or alert rule templates. Refer to:
Annotation and label template reference
Notification template reference
This documentation provides an overview of the functions and operators of the Go template language that are available for both notification and alert rule templates.
Print
To print the value of something, use 
{{ and 
}}. You can print the value of a variable, a field of a variable, the result of a function, or the value of dot.
{{ $values }}
{{ $values.A.Value }}
{{ humanize 1000.0 }}
{{ .Alerts }}
Dot
In 
text/template, there is a special cursor called dot, written as 
.. You can think of this cursor as a variable whose value changes depending on where in the template it is used.
At the start of notification templates, dot (
.) refers to Notification Data.
{{ .Alerts }}
In annotation and label templates, dot (
.) is initialized with all alert data. It’s recommended to use the 
$labels and 
$values variables instead to directly access the alert labels and query values.
{{< admonition type="note" >}}
Dot (
.) might refer to something else when used in a range, a with, or when writing templates used in other templates.
{{< /admonition >}}
{{< docs/shared lookup="alerts/template-language.md" source="grafana" version="<GRAFANA_VERSION>" >}}

Manage notification templates
In contact points, you can select notification templates to customize the notification messages sent.
By default, Grafana provides a template for the notification title (
{{define "default.title"}}) and a template for the notification message (
{{define "default.message"}}). Both default templates display common alert details.
You can also create custom templates to customize the content and format of notification messages, which can then be applied to one or more contact points. In Grafana, a custom notification template is created within a notification template group.
This documentation provides step-by-step instructions for selecting templates in contact points, previewing templates, and creating notification template groups using the Grafana UI.
Select a notification template for a contact point
To add an existing notification template to your contact point, complete the following steps.
Click an existing contact point or create a new one.
In Optional settings, click any field that contains templates.
For example, if you are creating an email contact point integration, click Message or Subject.
Click Edit.
A dialog box opens where you can select notification templates.
Click Select notification template or Enter custom message to customize a template or message
You can select an existing notification template and preview it using the default payload.
You can also copy the notification template and use it in the Enter custom message tab.
Click Save contact point.
You can create custom notification templates using the Enter custom message tab. For reusable and consistent notifications, a best practice is to custom notification templates as described in the following section.
Create a notification template and notification template group
Create notification templates to customize notification messages and reuse them in contact points.
In Grafana, custom notification templates (
{{define "<NAME>"}}) are created within a notification template group, allowing you to test and implement multiple templates together.
Your notification template name (
{{define "<NAME>"}}) must be unique. You cannot have two templates with the same name in the same notification template group or in different notification template groups. Therefore, avoid using names already defined as default templates, such as: 
__subject, 
__text_values_list, 
__text_alert_list, 
default.title and 
default.message.
In Grafana Cloud, you can create up to 30 notification templates by default. Template changes can take 1 to 2 minutes to take effect while a background job synchronizes them with storage.
To create a notification template in Grafana, complete the following steps.
Click Alerts & IRM -> Contact points.
Click the Notification Templates tab and then + New notification template.
Enter a name for the notification template group.
Write the content of the template in the content field.
Save your changes.
If 
{{ define }} is not included in the content, 
{{ define "<NOTIFICATION_TEMPLATE_NAME>" }} and 
{{ end }} is automatically added to the start and end.
To create a notification template group that contains more than one notification template, complete the following steps.
Click + New notification template.
Enter a name for the notification template group.
Write each template in the Content field, including 
{{ define "<NOTIFICATION_TEMPLATE_NAME>" }} and 
{{ end }} at the start and end of each template.
Save your changes.
For more details on how to write notification templates, refer to the template reference and examples.
Preview a notification template
Preview how your notification templates should look before using them in your contact points, helping you understand the result of the template you are creating as well as enabling you to fix any errors before saving it.
{{< admonition type="note" >}}
Notification template preview is only for Grafana Alertmanager.
{{< /admonition >}}
To preview your notification templates:
Navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Templates tab.
Click + New notification template or edit an existing template group.
Add or update your template content.
Default data is provided and you can add or edit alert data to it as well as alert instances. You can add alert data directly in the Payload data window itself or click Select alert instances or Add custom alerts.
Optional: To add alert data from existing alert instances:
a. Click Select alert instances.
b. Hover over the alert instances to view more information on each alert instance.
c. Click Confirm to add the alert instance(s) to the payload.
Optional: To add alert data using the Alert data editor, click Add custom data:
a. Add annotations, custom labels and/or set a dashboard or a panel.
b. Toggle Firing/resolved depending on whether you want to add firing or resolved alerts to your notification.
c. Click Add alert data.
d. Click Refresh preview to see what your template content should look like and the corresponding payload data.
If there are any errors in your template, they are displayed in the Preview and you can correct them before saving.
Save your changes.

Notification template reference
By default, Grafana provides predefined templates to format notification messages.
You can also customize your notifications with custom templates, which are based on the Go template language.
This documentation lists the data available for use in notification templates.
Notification Data
In notification templates, dot (
.) is initialized with the following data:
Name
Type
Description
Receiver
string
The name of the contact point sending the notification
Status
string
The status is 
firing if at least one alert is firing, otherwise 
resolved.
Alerts
[]Alert
List of all firing and resolved alerts in this notification.
Alerts.Firing
[]Alert
List of all firing alerts in this notification.
Alerts.Resolved
[]Alert
List of all resolved alerts in this notification.
GroupLabels
KV
The labels that group these alerts in this notification based on the 
Group by option.
CommonLabels
KV
The labels common to all alerts in this notification.
CommonAnnotations
KV
The annotations common to all alerts in this notification.
ExternalURL
string
A link to Grafana, or the Alertmanager that sent this notification if using an external Alertmanager.
GroupKey
string
The key used to identify this alert group.
TruncatedAlerts
integer
The number of alerts, if any, that were truncated in the notification. Supported by Webhook and OnCall.
It's important to remember that a single notification can group multiple alerts to reduce the number of alerts you receive. 
Alerts is an array that includes all the alerts in the notification.
Here's an example that prints all available notification data from dot (
.):
{{ define "custom_template" }}
  {{ .Receiver }}
  {{ .Status }}
  There are {{ len .Alerts }} alerts
  There are {{ len .Alerts.Firing }} firing alerts
  There are {{ len .Alerts.Resolved }} resolved alerts
  {{ .GroupLabels }}
  {{ .CommonLabels }}
  {{ .CommonAnnotations }}
  {{ .ExternalURL }}
{{ end }}
You can execute this template by passing the dot (
.):
{{ template "custom_template" . }}
Alert
Alert contains data for an individual alert:
Name
Type
Description
Status
string
Firing or resolved.
Labels
KV
The labels associated with this alert. 
 It includes all types of labels, but only query labels used in the alert condition.
Annotations
KV
The annotations for this alert.
StartsAt
Time
The time the alert fired
EndsAt
Time
Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received.
GeneratorURL
string
A link to Grafana, or the source of the alert if using an external alert generator.
Fingerprint
string
A unique string that identifies the alert.
Grafana-managed alerts include these additional properties:
Name
Type
Description
DashboardURL
string
A link to the Grafana Dashboard if the alert has a Dashboard UID annotation, with time range from 
1h before alert start to end (or now if firing).
PanelURL
string
A link to the panel if the alert has a Panel ID annotation, with time range from 
1h before alert start to end (or now if firing).
SilenceURL
string
A link to silence the alert.
Values
KV
The values of expressions used to evaluate the alert condition. Only relevant values are included.
ValueString
string
A string that contains the labels and value of each reduced expression in the alert.
OrgID
integer
The ID of the organization that owns the alert.
This example iterates over the list of firing and resolved alerts (
.Alerts) in the notification and prints the data for each alert:
{{ define "custom_template" }}
{{ range .Alerts }}
  {{ .Status }}
  {{ .Labels }}
  {{ .Annotations }}
  {{ .StartsAt }}
  {{ .EndsAt }}
  {{ .GeneratorURL }}
  {{ .Fingerprint }}
  {{/* Only available for Grafana-managed alerts */}}
  {{ .DashboardURL }}
  {{ .PanelURL }}
  {{ .SilenceURL }}
  {{ .Values }}
  {{ .ValueString }}
{{ end }}
{{ end }}
You can run this template by passing the dot (
.):
{{ template "custom_template" . }}
KV
KV is a set of key value pairs, where each key and value is a string.
Similarly to accessing variable properties, you can use 
. to retrieve the value of a value. For example:
{{ define "custom_template" }}
  {{ .CommonLabels.grafana_folder }}
{{ end }}
If a KV happens to contain numbers or bools then these are string representations of the numeric or boolean value.
Additionally, KV provides methods to sort the pairs, remove keys, and iterate over just the keys or values:
Method name
Description
Arguments
Returns
SortedPairs
Returns a sorted list of key/value pairs.
Pairs
Remove
Returns a copy of the KV with the keys removed
[]string
KV
Names
Return the names of the label names
[]string
Values
Return a list of the values
[]string
Here's an example of using these methods:
{{ define "custom_template" }}
  {{ .CommonLabels.SortedPairs }}
  {{ .CommonLabels.Names }}
  {{ .CommonLabels.Values }}
  {{ .CommonLabels.Remove (stringSlice "grafana_folder") }}
{{ end }}
Time
Some template functions and properties return a 
Time object, which refers to the type 
Time in Go's time package.
When accessing a 
Time object, you can use various 
Time functions in your templates as follows.
{{ define "custom_template" }}
  {{ range .Alerts }}
    {{ .StartsAt  }}
    {{ .StartsAt.Add 6000000000000  }}
    {{ .StartsAt.Add -6000000000000  }}
    {{ .StartsAt.AddDate 1 0 0  }}
    {{ .StartsAt.Year   }}/{{ .StartsAt.Month   }}/{{ .StartsAt.Day   }}
    {{ .StartsAt.Hour   }}:{{ .StartsAt.Minute   }}:{{ .StartsAt.Second   }}
    {{ .StartsAt.YearDay   }}-{{ .StartsAt.Weekday   }}
    {{ .StartsAt.Unix }} {{ .StartsAt.UnixMilli }}
  {{ end}}
{{ end }}
Functions
Functions can perform actions in templates such as transforming or formatting data.
Note that the functions provided by Go's template language, such as 
index, 
and, 
printf, and 
len, are available, along with many others.
In addition, the following functions are also available for templating notifications:
Name
Arguments
Returns
Description
title
string
string
Capitalizes the first character of each word.
toUpper
string
string
Returns all text in uppercase.
toLower
string
string
Returns all text in lowercase.
trimSpace
string
string
Removes leading and trailing white spaces.
match
pattern, text
boolean
Matches the text against a regular expression pattern.
reReplaceAll
pattern, replacement, text
string
Replaces text matching the regular expression.
join
sep string, s []string
string
Concatenates the elements of s to create a single string. The separator string sep is placed between elements in the resulting string.
safeHtml
string
string
Marks string as HTML not requiring auto-escaping.
stringSlice
...string
string
Returns the passed strings as a slice of strings. auto-escaping.
date
string, Time
string
Format a time in the specified format. For format options, refer to Go's time format documentation.
tz
string, Time
Time
Returns the time in the specified timezone, such as 
Europe/Paris. For available timezone options, refer to the list of tz database time zones.
Here's an example using some functions to format text:
{{ define "custom_template" }}
  {{ title "hello, world!" }}
  {{ toUpper "Hello, world!" }}
  {{ toLower "Hello, world!" }}
  {{ trimSpace "Hello, world!" }}
  {{ match "a.*" "abc" }}
  {{ reReplaceAll "localhost:(.*)" "example.com:$1" "localhost:8080" }}
  {{ join "-" (stringSlice "a" "b" "c") }}
  {{ safeHtml "<b>Text</b>"}}
  {{ stringSlice "a" "b" "c" }}
{{ end }}
date and 
tz can format times. For example, to print the time an alert fired in the format 
15:04:05 MST:
{{ define "custom_template" }}
  {{ range .Alerts }}
    {{ .StartsAt | date "15:04:05 MST" }}
  {{ end}}
{{ end }}
You can then use 
tz to change the timezone from UTC to local time, such as 
Europe/Paris.
{{ define "custom_template" }}
  {{ range .Alerts }}
    {{ .StartsAt | tz "Europe/Paris" }}
    {{ .StartsAt | tz "Europe/Paris" | date "15:04:05 MST" }}
  {{ end}}
{{ end }}
2024-10-30 21:01:45.227 +0100 CET
21:01:45 CET
Namespaced Functions
{{< admonition type="note" >}}
Namespaced Functions are not yet generally available in Grafana Cloud.
{{< /admonition >}}
In addition to the top-level functions, the following namespaced functions are also available:
Collection Functions
Name
Arguments
Returns
Description
coll.Dict
key string, value any, ...
map
Creates a map with string keys from key/value pairs. All keys are converted to strings. If an odd number of arguments is provided, the last key will have an empty string value.
coll.Slice
...any
[]any
Creates a slice (array/list) from the provided arguments. Useful for creating lists that can be iterated over with 
range.
coll.Append
value any, list []any
[]any
Creates a new list by appending a value to the end of an existing list. Does not modify the original list.
Example using collection functions:
{{ define "collection.example" }}
{{- /* Create a dictionary of alert metadata */ -}}
{{- $metadata := coll.Dict
    "severity" "critical"
    "team" "infrastructure"
    "environment" "production"
-}}
{{- /* Create a slice of affected services */ -}}
{{- $services := coll.Slice "database" "cache" "api" -}}
{{- /* Append a new service to the list */ -}}
{{- $services = coll.Append "web" $services -}}
{{- /* Use the collections in a template */ -}}
Affected Services: {{ range $services }}{{ . }},{{ end }}
Alert Metadata:
{{- range $k, $v := $metadata }}
  {{ $k }}: {{ $v }}
{{- end }}
{{ end }}
Output:
Affected Services: database,cache,api,web,
Alert Metadata:
  environment: production
  severity: critical
  team: infrastructure
Data Functions
Name
Arguments
Returns
Description
data.JSON
jsonString string
any
Parses a JSON string into an object that can be manipulated in the template. Works with both JSON objects and arrays.
data.ToJSON
obj any
string
Serializes any object (maps, arrays, etc.) into a JSON string. Useful for creating webhook payloads.
data.ToJSONPretty
indent string, obj any
string
Creates an indented JSON string representation of an object. The first argument specifies the indentation string (e.g., spaces).
Example using data functions:
{{ define "data.example" }}
{{- /* First, let's create some alert data as a JSON string */ -}}
{{ $jsonString := `{
  "service": {
    "name": "payment-api",
    "environment": "production",
    "thresholds": {
      "error_rate": 5,
      "latency_ms": 100
    }
  }
}` }}
{{- /* Parse the JSON string into an object we can work with */ -}}
{{ $config := $jsonString | data.JSON }}
{{- /* Create a new alert payload */ -}}
{{ $payload := coll.Dict
    "service" $config.service.name
    "environment" $config.service.environment
    "status" .Status
    "errorThreshold" $config.service.thresholds.error_rate
}}
{{- /* Output the payload in different JSON formats */ -}}
Compact JSON: {{ $payload | data.ToJSON }}
Pretty JSON with 2-space indent:
{{ $payload | data.ToJSONPretty "  " }}
{{ end }}
Output:
Compact JSON: {"environment":"production","errorThreshold":5,"service":"payment-api","status":"resolved"}
Pretty JSON with 2-space indent:
{
  "environment": "production",
  "errorThreshold": 5,
  "service": "payment-api",
  "status": "resolved"
}
Template Functions
Name
Arguments
Returns
Description
tmpl.Exec
name string, [context any]
string
Executes a named template and returns the result as a string. Similar to the 
template action but allows for post-processing of the result.
tmpl.Inline
template string, context any
string
Renders a string as a template.
{{ define "template.example" -}}
{{ coll.Dict
    "info" (tmpl.Exec `info` . | data.JSON)
    "severity" (tmpl.Inline `{{ print "critical" | toUpper }}` . )
    | data.ToJSONPretty " "}}
{{- end }}
{{- /* Define a sub-template */ -}}
{{ define "info" -}}
{{coll.Dict
    "team" "infrastructure"
    "environment" "production" | data.ToJSON }}
{{- end }}
Output:
{
  "info": {
    "environment": "production",
    "team": "infrastructure"
  },
  "severity": "CRITICAL"
}
Time Functions
Name
Arguments
Returns
Description
time.Now
Time
Returns the current local time as a time.Time object. Can be formatted using Go's time formatting functions.
Example using time functions:
{{ define "time.example" }}
{{- /* Get current time in different formats */ -}}
Current Time (UTC): {{ (time.Now).UTC.Format "2006-01-02 15:04:05 MST" }}
Current Time (Local): {{ (time.Now).Format "Monday, January 2, 2006 at 15:04:05" }}
{{- /* Compare alert time with current time */ -}}
{{ $timeAgo := (time.Now).Sub .StartsAt }}
Alert fired: {{ $timeAgo }} ago
{{ end }}
Output:
Current Time (UTC): 2025-03-08 18:14:27 UTC
Current Time (Local): Saturday, March 8, 2025 at 14:14:27
Alert fired: 25h49m32.78574723s ago
Differences with annotation and label templates
In the alert rule, you can also template annotations and labels to include additional information. For example, you might add a 
summary annotation that displays the query value triggering the alert.
Annotation and label templates add relevant information to individual alert instances, while notification templates inform about a group of alert instances.
Since both types of templates operate in distinct contexts, the functions and variables available in annotation and label templates differ from those used in notification templates.

Example of dynamic labels in alert instances
Labels are essential for scaling your alerting setup. They define metadata like 
severity, 
team, 
category, or 
environment, which you can use for alert routing.
A label like 
severity="critical" can be set statically in the alert rule configuration, or dynamically based on a query value such as the current free disk space. Dynamic labels adjust label values at runtime, allowing you to reuse the same alert rule across different scenarios.
This example shows how to define dynamic labels based on query values, along with key behavior to keep in mind when using them.
First, it's important to understand how Grafana Alerting treats labels.
Alert instances are defined by labels
Each alert rule creates a separate alert instance for every unique combination of labels.
This is called multi-dimensional alerts: one rule, many instances—one per unique label set.
For example, a rule that queries CPU usage per host might return multiple series (or dimensions):
{alertname="ServerHighCPU", instance="prod-server-1" }
{alertname="ServerHighCPU", instance="prod-server-2" }
{alertname="ServerHighCPU", instance="prod-server-3" }
Each unique label combination defines a distinct alert instance, with its own evaluation state and potential notifications.
The full label set of an alert instance can include:
Labels from the query result (e.g., 
instance)
Auto-generated labels (e.g., 
alertname)
User-defined labels from the rule configuration
User-defined labels
As shown earlier, alert instances automatically include labels from the query result, such as 
instance or 
job. To add more context or control alert routing, you can define user-defined labels in the alert rule configuration:
{{< figure src="/media/docs/alerting/example-dynamic-labels-edit-labels-v3.png" max-width="750px" alt="Edit labels UI in the alert rule configuration." >}}
User-defined labels can be either:
Fixed labels: These have the same value for every alert instance. They are often used to include common metadata, such as team ownership.
Templated labels: These calculate their values based on the query result at evaluation time.
Templated labels
Templated labels evaluate their values dynamically, based on the query result. This allows the label value to vary per alert instance.
Use templated labels to inject additional context into alerts. To learn about syntax and use cases, refer to Template annotations and labels.
You can define templated labels that produce either:
A fixed value per alert instance.
A dynamic value per alert instance that changes based on the last query result.
Fixed values per alert instance
You can use a known label value to enrich the alert with additional metadata not present in existing labels. For example, you can map the 
instance label to an 
env label that represents the deployment environment:
{{- if eq $labels.instance "prod-server-1" -}}production
{{- else if eq $labels.instance "stag-server-1" -}}staging
{{- else -}}development
{{- end -}}
This produces alert instances like:
{alertname="ServerHighCPU", instance="prod-server-1", env="production"}
{alertname="ServerHighCPU", instance="stag-server-1", env="staging"}
In this example, the 
env label is fixed for each alert instance and does not change during its lifecycle.
Dynamic values per alert instance
You can define a label whose value depends on the numeric result of a query—mapping it to a predefined set of options. This is useful for representing 
severity levels within a single alert rule.
Instead of defining three separate rules like:
CPU ≥ 90 → 
severity=critical
CPU ≥ 80 → 
severity=warning
CPU ≥ 70 → 
severity=minor
You can define a single rule and assign 
severity dynamically using a template:
{{/* $values.B.Value refers to the numeric result from query B */}}
{{- if gt $values.B.Value 90.0 -}}critical
{{- else if gt $values.B.Value 80.0 -}}warning
{{- else if gt $values.B.Value 70.0 -}}minor
{{- else -}}none
{{- end -}}
This pattern lets you express multiple alerting scenarios in a single rule, while still routing based on the 
severity label value.
Example overview
In the previous severity template, you can set the alert condition to 
$B > 70 to prevent firing when 
severity=none, and then use the 
severity label to route distinct alert instances to different contact points.
For example, configure a notification policy that matches 
alertname="ServerHighCPU" with the following children policies:
severity=critical → escalate to an incident response and management solution (IRM).
severity=warning → send to the team's Slack channel.
severity=minor → send to a non-urgent queue or log-only dashboard.
The resulting alerting flow might look like this:
Time
$B query
Alert instance
Routed to
t1
65
{alertname="ServerHighCPU", severity="none"}
Not firing
t2
75
{alertname="ServerHighCPU", severity="minor"}
Non-urgent queue
t3
85
{alertname="ServerHighCPU", severity="warning"}
Team Slack channel
t4
95
{alertname="ServerHighCPU", severity="critical"}
IRM escalation chain
This alerting setup allows you to:
Use a single rule for multiple severity levels.
Route alerts dynamically using the label value.
Simplify alert rule maintenance and avoid duplication.
However, dynamic labels can introduce unexpected behavior when label values change. The next section explains this.
Caveat: a label change affects a distinct alert instance
Remember: alert instances are defined by their labels.
If a dynamic label changes between evaluations, this new value affects a separate alert instance.
Here's what happens if 
severity changes from 
minor to 
warning:
The instance with 
severity="minor" disappears → it becomes a missing series.
A new instance with 
severity="warning" appears → it starts from scratch.
After two evaluations without data, the 
minor instance is resolved and evicted.
Here’s a sequence example:
Time
Query value
Instance 
severity="none"
Instance 
severity="minor"
Instance 
severity="warning"
t0
t1
75
🔴 📩
t2
85
⚠️ MissingSeries
🔴 📩
t3
85
⚠️ MissingSeries
🔴
t4
50
🟢
📩 Resolved and evicted
⚠️ MissingSeries
t5
50
🟢
⚠️ MissingSeries
t6
50
🟢
📩 Resolved and evicted
Learn more about this behavior in Stale alert instances.
In this example, the 
minor and 
warning alerts likely represent the same underlying issue, but Grafana treats them as distinct alert instances. As a result, this scenario generates two firing notifications and two resolved notifications, one for each instance.
This behavior is important to keep in mind when dynamic label values change frequently.
It can lead to multiple notifications firing and resolving in short intervals, resulting in noisy and confusing notifications.
Try it with TestData
You can replicate this scenario using the TestData data source to simulate an unstable signal—like monitoring a noisy sensor.
This setup reproduces label flapping and shows how dynamic label values affect alert instance behavior.
Add the TestData data source through the Connections menu.
Create an alert rule.
Navigate to Alerting → Alert rules and click New alert rule.
Simulate a query (
$A) that returns a noisy signal.
Select TestData as the data source and configure the scenario.
Scenario: Random Walk
Series count: 1
Start value: 51
Min: 50, Max: 100
Spread: 100 (ensures large changes between consecutive data points)
Add an expression.
Type: Reduce
Input: A
Function: Last (to get the most recent value)
Name: B
Define the alert condition.
Use a threshold like 
$B >= 50 (it always fires).
Click Edit Labels to add a dynamic label.
Create a new label 
severity and set its value to the following:
{{/* $values.B.Value refers to the numeric result from query B */}}
{{- if gt $values.B.Value 90.0 -}}P1
{{- else if gt $values.B.Value 80.0 -}}P2
{{- else if gt $values.B.Value 70.0 -}}P3
{{- else if gt $values.B.Value 60.0 -}}P4
{{- else if gt $values.B.Value 50.0 -}}P5
{{- else -}}none
{{- end -}}
Set evaluation behavior.
Set a short evaluation interval (e.g., 
10s) to observe quickly label flapping and alert instance transitions in the history.
Preview alert routing to verify the label template.
In Configure notifications, toggle Advanced options.
Click Preview routing and check the value of the 
severity label:
{{< figure src="/media/docs/alerting/example-dynamic-labels-preview-label.png" max-width="750px" caption="Preview routing multiple times to verify how label values change over time." >}}
Observe alert state changes.
Click Save rule and exit, and open the alert history view to see how changes in 
severity affect the state of distinct alert instances.
{{< figure src="/media/docs/alerting/example-dynamic-labels-alert-history-page.png" max-width="750px" caption="You can find multiple transitions over time as the label value fluctuates." >}}
{{< admonition type="tip" >}}
You can explore this alerting example in Grafana Play.
Open the example to view alert evaluation results, generated alert instances, the alert history timeline, and alert rule details.
{{< /admonition >}}
Considerations
Dynamic labels lets you reuse a single alert rule across multiple escalation scenarios—but it also introduces complexity. When the label value depends on a noisy metric and changes frequently, it can lead to flapping alert instances and excessive notifications.
These alerts often require tuning to stay reliable and benefit from continuous review. To get the most out of this pattern, consider the following:
Tune evaluation settings and queries for stability
Increase the evaluation interval and pending period to reduce the frequency of state changes. Additionally, consider smoothing metrics with functions like 
avg_over_time to reduce flapping.
Use wider threshold bands
Define broader ranges in your label template logic to prevent label switching caused by small value changes.
Disable resolved notifications
When labels change frequently and alerts resolve quickly, you can reduce the number of notifications by disabling resolved notifications at the contact point.
Disable the Missing series evaluations setting
The Missing series evaluations setting (default: 2) defines how many intervals without data are allowed before resolving an instance. Consider disabling it if it's unnecessary for your use case, as it can complicate alert troubleshooting.
Preserve context across related alerts
Ensure alert metadata includes enough information to help correlate related alerts during investigation.
Use separate alert rules and static labels when simpler
In some cases, defining separate rules with static labels may be easier to manage than one complex dynamic rule. This also allows you to customize alert queries for each specific case.
Learn more
Here's a list of additional resources related to this example:
Multi-dimensional alerting example – Explore how Grafana creates separate alert instances for each unique set of labels.
Labels – Learn about the different types of labels and how they define alert instances.
Template labels in alert rules – Use templating to set label values dynamically based on query results.
Stale alert instances – Understand how Grafana resolves and removes stale alert instances.
Handle missing data – Learn how Grafana distinguishes between missing series and 
NoData.
Notification policies and routing – Create multiple notification policies to route alerts based on label values like 
severity or 
team.
Dynamic label example in Grafana Play - View this example in Grafana Play to explore alert instances and state transitions with dynamic labels.

Example of dynamic thresholds per dimension
In Grafana Alerting, each alert rule supports only one condition expression.
That's enough in many cases—most alerts use a fixed numeric threshold like 
latency > 3s or 
error_rate > 5% to determine their state.
As your alerting setup grows, you may find that different targets require different threshold values.
Instead of duplicating alert rules, you can assign a different threshold value to each target—while keeping the same condition. This simplifies alert maintenance.
This example shows how to do that using multi-dimensional alerts and a Math expression.
Example overview
You're monitoring latency across multiple API services. Initially, you want to get alerted if the 95th percentile latency (
p95_api_latency) exceeds 3 seconds, so your alert rule uses a single static threshold:
p95_api_latency > 3
But the team quickly finds that some services require stricter thresholds. For example, latency for payment APIs should stay under 1.5s, while background jobs can tolerate up to 5s. The team establishes different thresholds per service:
p95_api_latency{service="checkout-api"}: must stay under 
1.5s.
p95_api_latency{service="auth-api"}: also strict, 
1.5s.
p95_api_latency{service="catalog-api"}: less critical, 
3s.
p95_api_latency{service="async-tasks"}: background jobs can tolerate up to 
5s.
You want to avoid creating one alert rule per service—this is harder to maintain.
In Grafana Alerting, you can define one alert rule that monitors multiple similar components like this scenario. This is called multi-dimensional alerts: one alert rule, many alert instances—one per unique label set.
But there's an issue: Grafana supports only one alert condition per rule.
One alert rule
├─ One condition ( e.g., $A > 3)
│  └─ Applies to all returned series in $A
│     ├─ {service="checkout-api"}
│     ├─ {service="auth-api"}
│     ├─ {service="catalog-api"}
│     └─ {service="async-tasks"}
To evaluate per-service thresholds, you need a distinct threshold value for each returned series.
Dynamic thresholds using a Math expression
You can create a dynamic alert condition by operating on two queries with a Math expression.
$A for query results (e.g., 
p95_api_latency).
$B for per-service thresholds (from CSV data or another query).
$A > $B is the Math expression that defines the alert condition.
Grafana evaluates the Math expression per series, by joining series from 
$A and 
$B based on their shared labels before applying the expression.
Here’s an example of an arithmetic operation:
{{< docs/shared lookup="alerts/math-example.md" source="grafana" version="<GRAFANA_VERSION>" >}}
In practice, you must align your threshold input with the label sets returned by your alert query.
The following table illustrates how a per-service threshold is evaluated in the previous example:
$A: p95 latency query
$B: threshold value
$C: $A>$B
State
{service="checkout-api"} 3
{service="checkout-api"} 1.5
{service="checkout-api"} 1
Firing
{service="auth-api"} 1
{service="auth-api"} 1.5
{service="auth-api"} 0
Normal
{service="catalog-api"} 2
{service="catalog-api"} 3
{service="catalog-api"} 0
Normal
{service="sync-work"} 3
{service="sync-work"} 5
{service="sync-work"} 0
Normal
In this example:
$A comes from the 
p95_api_latency query.
$B is manually defined with a threshold value for each series in 
$A.
The alert condition compares 
$A>$B using a Math relational operator (e.g., 
>, 
<, 
>=, 
<=, 
==, 
!=) that joins series by matching labels.
Grafana evaluates the alert condition and sets the firing state where the condition is true.
The Math expression works as long as each series in 
$A can be matched with exactly one series in 
$B. They must align in a way that produces a one-to-one match between series in 
$A and 
$B.
{{< admonition type="caution" >}}
If a series in one query doesn’t match any series in the other, it’s excluded from the result and a warning message is displayed:
1 items dropped from union(s): ["$A > $B": ($B: {service=payment-api})]
{{< /admonition >}}
Labels in both series don’t need to be identical. If labels are a subset of the other, they can join. For example:
$A returns series 
{host="web01", job="event"} 30 and 
{host="web02", job="event"} 20.
$B returns series 
{host="web01"} 10 and 
{host="web02"} 0.
$A + 
$B returns 
{host="web01", job="event"} 40 and 
{host="web02", job="event"} 20.
Try it with TestData
You can use the TestData data source to replicate this example:
Add the TestData data source through the Connections menu.
Create an alert rule.
Navigate to Alerting → Alert rules and click New alert rule.
Simulate a query (
$A) that returns latencies for each service.
Select TestData as the data source and configure the scenario.
Scenario: Random Walk
Alias: latency
Labels: service=api-$seriesIndex
Series count: 4
Start value: 1
Min: 1, Max: 4
This uses 
$seriesIndex to assign unique service labels: 
api-0, 
api-1, etc.
{{< figure src="/media/docs/alerting/example-dynamic-thresholds-latency-series-v2.png" max-width="750px" alt="TestData data source returns 4 series to simulate latencies for distinct API services." >}}
Define per-service thresholds with static data.
Add a new query (
$B) and select TestData as the data source.
From Scenario, select CSV Content and paste this CSV:
 service,value
 api-0,1.5
 api-1,1.5
 api-2,3
 api-3,5
The 
service column must match the labels from 
$A.
The 
value column is a numeric value used for the alert comparison.
For details on CSV format requirements, see table data examples.
Add a new Reduce expression (
$C).
Type: Reduce
Input: A
Function: Mean
Name: C
This calculates the average latency for each service: 
api-0, 
api-1, etc.
Add a new Math expression.
Type: Math
Expression: 
$C > $B
Set this expression as the alert condition.
This fires if the average latency (
$C) exceeds the threshold from 
$B for any service.
Preview the alert.
{{< figure src="/media/docs/alerting/example-dynamic-thresholds-preview-v3.png" max-width="750px" caption="Alert preview evaluating multiple series with distinct threshold values" >}}
{{< admonition type="tip" >}}
You can explore this alerting example in Grafana Play.
Open the example to view alert evaluation results, generated alert instances, the alert history timeline, and alert rule details.
{{< /admonition >}}
Other use cases
This example showed how to build a single alert rule with different thresholds per series using multi-dimensional alerts and Math expressions.
This approach scales well when monitoring similar components with distinct reliability goals.
By aligning series from two queries, you can apply a dynamic threshold—one value per label set—without duplicating rules.
While this example uses static CSV content to define thresholds, the same technique works in other scenarios:
Dynamic thresholds from queries or recording rules: Fetch threshold values from a real-time query, or from custom recording rules.
Combine multiple conditions: Build more advanced threshold logic by combining multiple conditions—such as latency, error rate, or traffic volume.
For example, you can define a PromQL expression that sets a latency threshold which adjusts based on traffic—allowing higher response times during periods of high-load.
(
  // Fires when p95 latency > 2s during usual traffic (≤ 1000 req/s)
  service:latency:p95 > 2 and service:request_rate:rate1m <= 1000
)
or
(
  // Fires when p95 latency > 4s during high traffic (> 1000 req/s)
  service:latency:p95 > 4 and service:request_rate:rate1m > 1000
)

Examples of high-cardinality alerts
In Prometheus and Mimir, metrics are stored as time series, where each unique set of labels defines a distinct series.
A large number of unique series (high cardinality) can overload your metrics backend, slow down dashboard and alert queries, and quickly increase your observability costs or exceed the limits of your Grafana Cloud plan.
These examples show how to detect and alert on early signs of high cardinality:
Total active series near limits: detect when your Prometheus, Mimir, or Grafana Cloud Metrics instance approaches soft or hard limits.
Series increase per metric or scope: fine-tune detection to identify growth in a particular metric or domain.
Sudden series growth: detect runaway cardinality increases caused by misconfigured exporters or new deployments.
High ingestion rate: detect when too many samples per second are being ingested, even if the total series count is stable.
Use these alert patterns to act on high-cardinality growth, and consider implementing Adaptive Metrics recommendations to keep your observability costs under control.
Choose metrics to monitor active series
First, identify which metric reports the number of active time series.
Prometheus, Mimir, and Grafana Cloud expose this information differently:
Environment
Metric
Description
Prometheus
prometheus_tsdb_head_series
Reports the number of active series currently stored in memory (the head block) of a single Prometheus instance. It includes series that have stopped receiving samples for up to one hour.
Grafana Cloud
grafanacloud_instance_active_series
Tracks the number of active series in your Grafana Cloud Metrics backend (Mimir).
Prometheus or Mimir
count({__name__!=""})
Counts the number of series with recent samples by scanning the TSDB index. This query is expensive and should be exposed through a recording rule.
Detect total active series near limits
A high number of active series increases memory usage and can impact performance. Grafana Cloud enforces usage limits to prevent your instance from running into these performance issues.
In Prometheus, you can alert when the total number of active series exceeds a threshold:
prometheus_tsdb_head_series > 1.5e6
This fires when the instance exceeds 1.5 million active series.
Adjust the threshold based on the capacity of your Prometheus host and observed load.
In Grafana Cloud, use the 
grafanacloud_instance_active_series metric to monitor active series across your managed Mimir backend:
grafanacloud_instance_active_series > 1.5e6
Grafana Cloud also provides account-level limits through the 
grafanacloud_instance_metrics_limits metric.
For more robust alerting, you can compare your current usage to the 
max_global_series_per_user limit:
(
  grafanacloud_instance_active_series
  / on (id)
    grafanacloud_instance_metrics_limits{limit_name="max_global_series_per_user"}
)
* on (id) group_left(name) grafanacloud_instance_info
> 0.9
grafanacloud_instance_active_series
Returns the current number of active series per your Mimir (Prometheus) data source instance (
id).
/ on (id) grafanacloud_instance_metrics_limits{limit_name="max_global_series_per_user"}
Divides current usage by the account limit to calculate a utilization ratio between 0 and 1 (where 
1 means the limit is reached).
* on (id) group_left(name) grafanacloud_instance_info
Joins instance metadata to display the instance name.
> 0.9
Defines the threshold condition to fire when usage exceeds 90% of the limit.
Adjust this value according to your alert goal. Alternatively, you can set the threshold as a Grafana Alerting expression in the UI.
Detect high-cardinality per metric
Instead of monitoring the total number of active series, you can fine-tune alerts to detect high cardinality within a specific scope — for example, by filtering on certain namespaces, services, or metrics known to generate many label combinations.
Multi-dimensional alerts let you evaluate each metric independently, so you can identify which metric is responsible for the label explosion instead of only tracking the overall total.
You can apply label filters, or use 
{__name__=~"regex"} to select specific metrics. Then, use 
count by (__name__) to group results per metric name.
Because the 
__name__ selector queries the entire TSDB index, it’s recommended to query this using a recording rule:
# Only HTTP/RPC-style metrics
active_series_per_metric:http_rpc =
label_replace(
  count by (__name__) ({__name__=~"http_.*|rpc_.*"}),
  "metric", "$1", "__name__", "(.*)"
)
This recording rule stores the number of active series per metric name.
count by (__name__) ({__name__=~"http_.*|rpc_.*"})
Counts the number of active series per metric matching the 
http_.* or 
rpc_.* regex.
label_replace(..., "metric", "$1", "__name__", "(.*)")
Copies the metric name into a new label called 
metric.
This enables generating one alert instance per metric because 
__name__ is not treated as a regular label.
Adjust the threshold and recording rule scope based on the label usage and normal behavior of your observed metrics.
After the recording rule is available, you can define this multi-dimensional alert rule as follows:
active_series_per_metric:http_rpc > 100
Grafana Alerting evaluates each row (or time series) returned by the 
active_series_per_metric:http_rpc recording rule as a separate alert instance, producing independent alert instance states:
Alert instance
Value
State
{metric="http_requests_total"}
320
Firing
{metric="rpc_client_calls_total"}
45
Normal
{metric="rpc_server_errors_total"}
110
Firing
Each metric name (
__name__) becomes a separate alert instance, so you immediately see which metric exceeds the expected limit.
Detect sudden cardinality growth
Even if the number of active series stays within proper limits, a sudden increase can signal a misbehaving exporter, a new deployment, or an unexpected label explosion. These peaks can help you prevent potential issues, or just inform you of deployment changes that might need adjustments.
You can use any of the metrics from the previous examples to track short-term changes in the total active series:
delta(active_series_metric[10m]) > 1000
This alert fires when the number of active series increases by more than 1000 within the last 10 minutes.
Adjust the time window (for example, 
[5m] or 
[30m]) and threshold to match your environment’s normal variability.
Detect high ingestion rate
Even if label cardinality remains under control, a high ingestion rate can affect Prometheus performance or increase observability costs.
In Prometheus, this usually happens when scrapes occur too frequently or when exporters generate large numbers of samples in short intervals.
To find an appropriate threshold to be alerted, start by monitoring normal ingestion peaks and set the threshold to a value that stays below the point where scrapes or WAL operations begin to slow down.
In Prometheus, use the 
prometheus_tsdb_head_samples_appended_total metric to measure the number of samples appended per scrape:
rate(prometheus_tsdb_head_samples_appended_total[10m]) > 1e5
The alert rule query returns the average ingestion rate per second over the last 10 minutes and fires when the value exceeds 100 000 samples per second.
In Grafana Cloud, use the 
grafanacloud_instance_samples_per_second metric to monitor total ingestion rate of your Mimir instances:
grafanacloud_instance_samples_per_second
  * on (id) group_left(name) grafanacloud_instance_info
> 1e5
Alternatively, Grafana Cloud metrics limits are based on data points per minute (DPM): the number of samples sent per minute across all your active series.
To monitor when your actual data-point rate approaches your DPM limit, you can compare total ingestion to your plan’s DPM limit:
(
  (grafanacloud_instance_samples_per_second * 60)
  / grafanacloud_org_metrics_included_dpm_per_series
)
> 0.9
(grafanacloud_instance_samples_per_second * 60)
Converts ingestion rate from data points per second to minutes.
/ grafanacloud_org_metrics_included_dpm_per_series
Divides current DPM usage by the DPM limit to calculate a utilization ratio between 0 and 1 (where 
1 means the limit is reached).
> 0.9
Defines the threshold condition to fire when the usage exceeds 90% of the DPM limit.
This alert helps you detect when your organization is ingesting data faster and approaching its limits. Use ingestion rate alerts to detect workload spikes, exporter misconfigurations, or rapid increases in ingestion volume and cost.
Learn more
Here’s list of additional resources related to this example:
Multi-dimensional alerting example – Learn how Grafana creates separate alert instances for each unique label set.
Understand Grafana Cloud active series and DPM– See how active series and data points per minute (DPM) are used to calculate metrics usage in Grafana Cloud.
Create Grafana Cloud usage alerts – Set up alerts when your usage or costs approach your predefined limits.
Plan capacity for Mimir– Learn how to plan ingestion rate and memory capacity for Mimir or Prometheus environments.
Adaptive Metrics recommendations – Use Adaptive Metrics to automatically reduce high-cardinality metrics and control observability costs.

Example of multi-dimensional alerts on time series data
This example shows how a single alert rule can generate multiple alert instances — one for each label set (or time series). This is called multi-dimensional alerting: one alert rule, many alert instances.
In Prometheus, each unique combination of labels defines a distinct time series. Grafana Alerting uses the same model: each label set is evaluated independently, and a separate alert instance is created for each series.
This pattern is common in dynamic environments when monitoring a group of components like multiple CPUs, containers, or per-host availability. Instead of defining individual alert rules or aggregated alerts, you alert on each dimension — so you can detect particular issues and include that level of detail in notifications.
For example, a query returns one series per CPU:
cpu label value
CPU percent usage
cpu-0
95
cpu-1
30
cpu-2
85
With a threshold of 
> 80, this would trigger two alert instances for 
cpu-0 and one for 
cpu-2.
Examples overview
Imagine you want to trigger alerts when CPU usage goes above 80%, and you want to track each CPU core independently.
You can use a Prometheus query like this:
sum by(cpu) (
  rate(node_cpu_seconds_total{mode!="idle"}[1m])
)
This query returns the active CPU usage rate per CPU core, averaged over the past minute.
CPU core
Active usage rate
cpu-0
95
cpu-1
30
cpu-2
85
This produces one series for each existing CPU.
When Grafana Alerting evaluates the query, it creates an individual alert instance for each returned series.
Alert instance
Value
{cpu="cpu-0"}
95
{cpu="cpu-1"}
30
{cpu="cpu-2"}
85
With a threshold condition like 
$A > 80, Grafana evaluates each instance separately and fires alerts only where the condition is met:
Alert instance
Value
State
{cpu="cpu-0"}
95
Firing
{cpu="cpu-1"}
30
Normal
{cpu="cpu-2"}
85
Firing
Multi-dimensional alerts help you surface issues on individual components—problems that might be missed when alerting on aggregated data (like total CPU usage).
Each alert instance targets a specific component, identified by its unique label set. This makes alerts more specific and actionable. For example, you can set a 
summary annotation in your alert rule that identifies the affected CPU:
High CPU usage on {{$labels.cpu}}
In the previous example, the two firing alert instances would display summaries indicating the affected CPUs:
High CPU usage on 
cpu-0
High CPU usage on 
cpu-2
Try it with TestData
You can quickly experiment with multi-dimensional alerts using the TestData data source, which can generate multiple random time series.
Add the TestData data source through the Connections menu.
Go to Alerting and create an alert rule
Select TestData as the data source.
Configure the TestData scenario
Scenario: Random Walk
Labels: 
cpu=cpu-$seriesIndex
Series count: 3
Min: 70, Max: 100
Spread: 2
{{< figure src="/media/docs/alerting/testdata-random-series-v2.png" max-width="750px" alt="Generating random time series data using the TestData data source" >}}
Reduce time series data for comparison
The example returns three time series like shown above with values across the selected time range.
To alert on each series, you need to reduce the time series to a single value that the alert condition can evaluate and determine the alert instance state.
Grafana Alerting provides several ways to reduce time series data:
Data source query functions. The earlier example used the Prometheus 
sum function to sum the rate results by 
cpu,producing a single value per CPU core.
Reduce expression. In the query and condition section, Grafana provides the 
Reduce expression to aggregate time series data.
In Default mode, the When input selects a reducer (like 
last, 
mean, or 
min), and the threshold compares that reduced value.
In Advanced mode, you can add the Reduce expression (e.g., 
last(), 
mean()) before defining the threshold (alert condition).
For demo purposes, this example uses the Advanced mode with a Reduce expression:
Toggle Advanced mode in the top right section of the query panel to enable adding additional expressions.
Add the Reduce expression using a function like 
mean() to reduce each time series to a single value.
Define the alert condition using a Threshold like 
$reducer > 80
Click Preview to evaluate the alert rule.
{{< figure src="/media/docs/alerting/using-expressions-with-multiple-series.png" max-width="750px" caption="The alert condition evaluates the reduced value for each alert instance and shows whether each instance is Firing or Normal." alt="Alert preview using a Reduce expression and a threshold condition" >}}
{{< admonition type="tip" >}}
You can explore this alerting example in Grafana Play.
Open the example to view alert evaluation results, generated alert instances, the alert history timeline, and alert rule details.
{{< /admonition >}}
Learn more
This example shows how Grafana Alerting implements a multi-dimensional alerting model: one rule, many alert instances and why reducing time series data to a single value is required for evaluation.
For additional learning resources, check out:
Get started tutorial – Create multi-dimensional alerts and route them
Example of alerting on tabular data
Update the interval of a rule group or modify the rules of the group.

Example of alerting on tabular data
Not all data sources return time series data. SQL databases, CSV files, and some APIs often return results as rows or arrays of columns or fields — commonly referred to as tabular data.
This example shows how to create an alert rule using data in table format. Grafana treats each row as a separate alert instance, as long as the data meets the expected format.
How Grafana Alerting evaluates tabular data
When a query returns data in table format, Grafana transforms each row into a separate alert instance.
To evaluate each row (alert instance), it expects:
Only one numeric column. This is the value used for evaluating the alert condition.
Non-numeric columns. These columns defines the label set. The column name becomes a label name; and the cell value becomes the label value.
Unique label sets per row. Each row must be uniquely identifiable by its labels. This ensures each row represents a distinct alert instance.
{{< admonition type="caution" >}}
These three conditions must be met—otherwise, Grafana can’t evaluate the table data and the rule will fail.
{{< /admonition >}}
Example overview
Imagine you store disk usage in a 
DiskSpace table and you want to trigger alerts when the available space drops below 5%.
Time
Host
Disk
PercentFree
2021-06-07
web1
/etc
3
2021-06-07
web2
/var
4
2021-06-07
web3
/var
8
To calculate the free space per Host and Disk in this case, you can use 
$__timeFilter to filter by time but without returning the date to Grafana:
SELECT
  Host,
  Disk,
  AVG(PercentFree) AS PercentFree
FROM DiskSpace
WHERE $__timeFilter(Time)
GROUP BY Host, Disk
This query returns the following table response:
Host
Disk
PercentFree
web1
/etc
3
web2
/var
4
web3
/var
8
When Alerting evaluates the query response, the data is transformed into three alert instances as previously detailed:
The numeric column becomes the value for the alert condition.
Additional columns define the label set for each alert instance.
Alert instance
Value
{Host="web1", Disk="/etc"}
3
{Host="web2", Disk="/var"}
4
{Host="web3", Disk="/var"}
8
Finally, an alert condition that checks for less than 5% of free space (
$A < 5) would result in two alert instances firing:
Alert instance
Value
State
{Host="web1", Disk="/etc"}
3
Firing
{Host="web2", Disk="/var"}
4
Firing
{Host="web3", Disk="/var"}
8
Normal
Try it with TestData
To test this quickly, you can simulate the table using the TestData data source:
Add the TestData data source through the Connections menu.
Go to Alerting and create an alert rule
Select TestData as the data source.
From Scenario, select CSV Content and paste this CSV:
host, disk, percentFree
web1, /etc, 3
web2, /var, 4
web3, /var, 8
Set a condition like 
$A < 5 and Preview the alert.
Grafana evaluates the table data and fires the two first alert instances.
{{< figure src="/media/docs/alerting/example-table-data-preview.png" max-width="750px" alt="Alert preview with tabular data using the TestData data source" >}}
{{< admonition type="tip" >}}
You can explore this alerting example in Grafana Play.
Open the example to view alert evaluation results, generated alert instances, the alert history timeline, and alert rule details.
{{< /admonition >}}
CSV data with Infinity
Note that when the Infinity plugin fetches CSV data, all the columns are parsed and returned as strings. By default, this causes the query expression to fail in Alerting.
To make it work, you need to format the CSV data as expected by Grafana Alerting.
In the query editor, specify the column names and their types to ensure that only one column is treated as a number.
{{< figure src="/media/docs/alerting/example-table-data-infinity-csv-data.png" max-width="750px" alt="Using the Infinity data source plugin to fetch CSV data in Alerting" >}}
Differences with time series data
Working with time series is similar—each series is treated as a separate alert instance, based on its label set.
The key difference is the data format:
Time series data contains multiple values over time, each with its own timestamp.
To evaluate the alert condition, alert rules must reduce each series to a single number using a function like 
last(), 
avg(), or 
max().
Tabular data doesn’t require reduction, as each row contains only a single numeric value used to evaluate the alert condition.
For comparison, see the multi-dimensional time series data example.

Examples of trace-based alerts
Metrics are the foundation of most alerting systems. They are usually the first signal that something is wrong, but they don’t always indicate where or why a failure occurs.
Traces fill that gap by showing the complete path a request takes through your system. They map the workflows across services, indicating where the request slows down or fails.
{{< figure src="/media/docs/alerting/screenshot-traces-visualization-11.5.png" max-width="750px" alt="Trace view" >}}
Traces report duration and errors directly to specific services and spans, helping to find the affected component and service scope. With this additional context, alerting on tracing data can help you identify root causes faster.
You can create trace-based alerts in Grafana Alerting using two main approaches:
Querying metrics generated from tracing data.
Using TraceQL, a query language for traces available in Grafana Tempo.
This guide provides introductory examples and distinct approaches for setting up trace-based alerts in Grafana. Tracing data is commonly collected using OpenTelemetry (OTel) instrumentation. OTel allows you to integrate trace data from a wide range of applications and environments into Grafana.
Alerting on span metrics
OpenTelemetry provides processors that convert tracing data into Prometheus-style metrics.
The service graph and span metrics processors are the standard options in Alloy and Tempo to generate Prometheus metrics from traces. They can generate the rate, error, and duration (RED) metrics from sampled spans.
You can then create alert rules that query metrics derived from traces.
{{< figure src="/media/docs/alerting/why-trace-based-metrics.png" max-width="750px" alt="Why metrics if you have traces?" >}}
Service graph metrics focus on inter-service communication and dependency health. They measure the calls between services, helping Grafana to infer the service topology. However, they measure only the interaction between two services—they don’t include the internal processing time of the client service.
You can use service graph metrics to detect infrastructure issues such as network degradation or service mesh problems.
For trace-based alerts, use span metrics.
Span metrics measure the total processing time of a service request: capturing what happens inside the service, not just the communication between services. They include the time spent on internal processing and waiting on downstream calls, providing an end-to-end picture of service performance.
Depending on how you create span metrics, the following span metrics are generated:
Span metrics generator
Metric name
Prometheus metric type
Description
Alloy and OTEL span metrics connector
traces_span_metrics_calls_total
Counter
Total count of the span
traces_span_metrics_duration_seconds
Histogram (native or classic)
Duration of the span
Tempo and Grafana Cloud Application Observability
traces_spanmetrics_calls_total
Counter
Total count of the span
traces_spanmetrics_latency
Histogram (native or classic)
Duration of the span
traces_spanmetrics_size_total
Counter
Total size of spans ingested
Each metric includes by default the following labels: 
service, 
span_name, 
span_kind, 
status_code, 
status_message, 
job, and 
instance.
In the metrics generator, you can customize how traces are converted into metrics by configuring histograms, exemplars, metric dimensions, and other options.
The following examples assume that span metrics have already been generated using one of these options or an alternative.
Detect slow span operations
This example shows how to define an alert rule that detects when operations handled by a service become slow.
Before looking at the query, it’s useful to review a few trace elements that shape how it works:
A trace represents a single request or transaction as it flows through multiple spans and services. A span refers to a specific operation within a service.
Each span includes the operation name (
span_name) and its duration (the metric value), as well as additional fields like span status (
status_code) and span kind (
span_kind).
A server span represents work performed on the receiving side of a request, while a client span represents the outbound call (parent span) waiting for a response (client → server).
To detect slow inbound operations within a specific service, you can define an alert rule that detects when the percentile latency of server spans exceeds a threshold. For example:
Detect when 95% of requests (excluding errors) do not complete faster than 2 seconds.
Using native histograms
The following PromQL query uses the 
traces_span_metrics_duration_seconds native histogram metric to define the alert rule query.
histogram_quantile(0.95,
 sum by (span_name) (
   rate(traces_span_metrics_duration_seconds{
     service_name="<SERVICE_NAME>",
     span_kind="SPAN_KIND_SERVER",
     status_code!="STATUS_CODE_ERROR"
   }[10m])
 )
) > 2
Here’s the query breakdown:
traces_span_metrics_duration_seconds
It’s a native histogram produced from spans using Alloy or the OTEL collector. The metric is filtered by:
service_name="<SERVICE_NAME>" targets a particular service.
span_kind="SPAN_KIND_SERVER" selects spans handling inbound requests.
status_code!="STATUS_CODE_ERROR" excludes spans that ended with errors.
You should query 
traces_spanmetrics_latency when using other span metric generators.
rate(...[10m])
Converts the histogram into a per-second histogram over the last 10 minutes (the distribution of spans per second during that period).
This makes the time window explicit and ensures latencies can be calculated over the last 10 minutes using 
histogram_* functions.
sum by (span_name)( … )
Merges all series that share the same 
span_name. This creates a multidimensional alert that generates one alert instance per span name (operation).
histogram_quantile(0.95, ...)
Calculates p95 latency from the histogram after applying the rate.
The query runs as an instant Prometheus query, returning a single value for the 10-minute window.
> 2
Defines the threshold condition. It returns only series whose p95 latency exceeds 2 seconds.
Alternatively, you can set this threshold as a Grafana Alerting expression in the UI, as shown in the following screenshot.
{{< figure src="/media/docs/alerting/trace-based-alertrule-screenshot.png" max-width="750px" caption="Alert rule querying span metrics and using threshold expression" >}}
Using classic histograms
Native histograms are stable in Prometheus since v3.8.0. Your span metric generator may therefore create classic histograms for latency span metrics, either 
traces_span_metrics_duration_seconds or 
traces_spanmetrics_latency.
When using classic histograms, the metric is the same but the metric format changes. A classic histogram represents a histogram with fixed buckets and exposes three metrics:
_bucket: cumulative buckets of the observations.
_sum: total sum of all observed values.
_count: count of observed values.
To calculate percentiles accurately, especially exceeding a particular threshold (e.g. 
`2s`), you have to configure the classic histogram with the explicit bucket, such as:
["100ms", "250ms", "1s", "2s", "5s"]
The 
otelcol.connector.spanmetrics can configure the buckets using the 
explicit block. The metric-generator in Tempo can configure the 
span_metrics.histogram_buckets setting.
Here's the equivalent PromQL for classic histograms:
histogram_quantile(0.95,
 sum by (span_name, le) (
   rate(traces_span_metrics_duration_seconds_bucket{
     service_name="<SERVICE_NAME>",
     span_kind="SPAN_KIND_SERVER",
     status_code!="STATUS_CODE_ERROR"
   }[10m])
 )
) > 2
Key differences compared with the native histograms example:
You must configure a histogram bucket matching the desired threshold (for example, 
2s).
You must query the 
_bucket metric, not the base metric.
You must include 
le in the 
sum by (…) grouping for 
histogram_quantile calculation.
Everything else remains the same.
{{< admonition type="note" >}}
The alert rules in these examples create multi-dimensional alerts: one alert instance for each distinct span name.
Dynamic span routes such as 
/product/1234 can create separate metric dimensions and alerts for each unique span, which can significantly impact metric costs and performance for large volumes.
To prevent high-cardinality data, normalize dynamic routes like 
/product/{id} using semantic attributes such as 
http.route and 
url.template, and limit dimensions to low-cardinality fields such as 
service_name, 
status_code, or 
http_method.
{{< /admonition >}}
Detect high error rate
This example defines an alert rule that detects when the error rate for any operation exceeds 20%. You can use this error rate alerts to identify increases in request errors, such as 5xx responses or internal failures.
The following query calculates the fraction of failed server spans for each service and operation.
(
  sum by (service, span_name) (
    rate(traces_span_metrics_calls_total{
      span_kind="SPAN_KIND_SERVER",
      status_code="STATUS_CODE_ERROR"
    }[10m])
  )
/
  sum by (service, span_name) (
    rate(traces_span_metrics_calls_total{
      span_kind="SPAN_KIND_SERVER"
    }[10m])
  )
) > 0.2
Here’s the query breakdown
traces_span_metrics_calls_total
A counter metric produced from spans that tracks the number of completed span operations.
span_kind="SPAN_KIND_SERVER" selects spans handling inbound requests.
status_code="STATUS_CODE_ERROR" selects only spans that ended in error.
Omitting the 
status_code filter in the denominator includes all spans, returning the total span count.
Check whether your metric generator instead creates the 
traces_spanmetrics_calls_total metric, and adjust the metric name.
rate(...[10m])
Converts the cumulative histogram into a per-second histogram over the last 10 minutes (the distribution of spans per second during that period).
This makes the time window explicit and ensures counters can be calculated over the last 10 minutes.
sum by (service, span_name)( … )
Aggregates per service and operation, creating one alert instance for each 
(service, span_name) combination.
This is a multidimensional alert that applies to all services, helping identify which service and corresponding operation is failing.
sum by () (...) / sum by () (...)
Divides failed spans by total spans to calculate the error rate per operation.
The result is a ratio between 
0 and 
1, where 
1 means all operations failed.
The query runs as an instant Prometheus query, returning a single value for the 10-minute window.
> 0.2
Defines the threshold condition. It returns only series whose error rate is higher than 20% of spans.
Alternatively, you can set this threshold as a Grafana Alerting expression in the UI.
Enable traffic guardrails
When the traffic is very low, even a single slow or failing request can trigger the alerts.
To avoid these types of false positives during low-traffic periods, you can include a minimum traffic condition in your alert rule queries. For example:
sum by (service, span_name)(
  increase(traces_span_metrics_calls_total{
    span_kind="SPAN_KIND_SERVER"
  }[10m])
) > 300
This query returns only spans that handled more than 300 requests in the 10-minute period.
This minimum level of traffic helps prevent false positives, ensuring the alert evaluates a significant number of spans before triggering.
You can combine this traffic condition with the error-rate query to ensure alerts fire only when both conditions are met:
((
  sum by (service, span_name) (
    rate(traces_span_metrics_calls_total{
      span_kind="SPAN_KIND_SERVER",
      status_code="STATUS_CODE_ERROR"
    }[10m])
  )
/
  sum by (service, span_name) (
    rate(traces_span_metrics_calls_total{
      span_kind="SPAN_KIND_SERVER"
    }[10m])
  )
) > 0.2)
and
(
    sum by (service, span_name)(
    increase(traces_span_metrics_calls_total{
      span_kind="SPAN_KIND_SERVER"
    }[10m])
) > 300 )
For a given span, the alert fires when:
The error rate exceeds 20% over the last 10 minutes.
The span handled at least 300 requests over the last 10 minutes.
Alternatively, you can split the alert into separate queries and combine them using a math expression as the threshold. In the example below, 
$ErrorRateCondition is the Grafana reference for the error-rate query, and 
$TrafficCondition is the reference for the traffic query.
{{< figure src="/media/docs/alerting/traffic-guardrail-with-separate-queries.png" max-width="500px" alt="Alert rule with threshold based on two queries" >}}
In this case, you must ensure both queries group by the same labels.
The advantage of this approach is that you can observe the results of both independent queries. You can then access the query results through the 
$values variable and display them in notifications or use them in custom labels.
A potential drawback of splitting queries is that each query runs separately. This increases backend load and can affect query performance, especially in environments with a large number of active alerts.
You can apply this traffic guardrail pattern to any alert rule.
Consider sampling {#consider-sampling}
Sampling is a technique used to reduce the amount of collected spans for cost-saving purposes. There are two main strategies which can be combined:
Head sampling: The decision to record or drop a span is made when the trace begins. The condition can be configured probabilistically (a percentage of traces) or by filtering out certain operations.
Tail sampling: The decision is made after the trace completes. This allows sampling more interesting operations, such as slow or failing requests.
With head sampling, alerting on span metrics should be done with caution, since span metrics will represent only a subset of all traces.
With tail sampling, it’s important to generate span metrics before a sampling decision is made. Grafana Cloud Adaptive Traces handle this automatically. With Alloy or the OpenTelemetry Collector, make sure the SpanMetrics connector runs before the filtering or tail sampling processor.
Using TraceQL
TraceQL is a query language for searching and filtering traces in Grafana Tempo, which uses a syntax similar to 
PromQL and 
LogQL.
With TraceQL, you can skip converting tracing data into span metrics and query raw trace data directly. It provides a more flexible filtering based on the trace structure, attributes, or resource metadata, and can detect issues faster as it doesn't wait for metric generation.
TraceQL isn't suitable for all scenarios. For example:
Inadequate for long-term analysis
Trace data has a significantly shorter retention period than metrics. For historical monitoring, it’s recommended to convert key tracing data into metrics to ensure the persistence of important data.
Inadequate for alerting after sampling
TraceQL can only query traces that are actually stored in Tempo. If sampling drops a large portion of traces, TraceQL-based alerts may miss real issues. Refer to consider sampling for guidance on how to generate span metrics before sampling.
{{< admonition type="caution" >}}
TraceQL alerting is available in Grafana v12.1 or higher, supported as a private preview feature.
Engineering and on-call support isn't available. No SLA is provided.
While TraceQL can be powerful for exploring and detecting issues directly from trace data, alerting with TraceQL shouldn't be used in production environments yet. Use it for testing and experimentation at this moment.
{{< /admonition >}}
The following example demonstrates how to recreate the previous alert rule that detected slow span operations using TraceQL.
Follow these steps to create the alert:
Enable TraceQL alerting
To use TraceQL in alerts, you must enable the 
tempoAlerting feature flag in your Grafana configuration.
If you use Grafana Cloud, contact Support to enable TraceQL alerting.
Configure the alert query
In your alert rule, select the Tempo data source, then convert the original PromQL query into the equivalent TraceQL query:
{status != error && kind = server && .service.name = "<SERVICE_NAME>"}
| quantile_over_time(duration, .95) by (name)
For a given service, this query calculates the p95 latency for all server spans, excluding errors, and groups them by span name.
Configure the time range
Currently, TraceQL alerting supports only range queries.
To define the time window, set the query time range to the last 10 minutes.
From: 
now-10m
To: 
now
{{< figure src="/media/docs/alerting/traceql-alert-configure-time-range.png" max-width="750px" alt="Time range configuration for TraceQL alert rule" >}}
Add a reducer expression.
Range queries return time series data, not a single value. The alert rule must then reduce time series data to a single numeric value before comparing it against a threshold.
Add a Reduce expression to convert the query results into a single value.
Set the threshold condition.
Create a Threshold expression to fire when the p95 latency exceeds 2 seconds: $B > 2.
{{< figure src="/media/docs/alerting/traceql-alert-configure-threshold.png" max-width="750px" alt="Alert rule configuration showing reducer and threshold expressions for TraceQL query" >}}
This final alert detects when 95% of the server spans for a particular service (excluding errors) take longer than 2 seconds to complete, using raw trace data instead of span metrics.
Additional resources
To explore related topics and expand the examples in this guide, see the following resources:
Trace structure: Learn how traces and spans are structured.
Grafana Tempo documentation: Full reference for Grafana’s open source tracing backend.
Span metrics using the metrics generator in Tempo: Generate span metrics directly from traces with Tempo’s built-in metrics generator.
Span metrics using Grafana Alloy: Configure Alloy to export span metrics from OpenTelemetry (OTel) traces.
Multi-dimensional alerts: Learn how to trigger multiple alert instances per alert rule like in these examples.
Grafana SLO documentation: Use span metrics to define Service Level Objectives (SLOs) in Grafana.
Trace sampling: explore strategies and configuration in Grafana Tempo.
{{< admonition type="note" >}}
OpenTelemetry instrumentations can record metrics independently of spans.
These OTEL metrics are not derived from traces and are not affected by sampling. They can serve as an alternative to span-derived metrics.
{{< /admonition >}}

Grafana Alerting tutorials
This section provides step-by-step tutorials to help you learn Grafana Alerting and explore key features through practical, easy-to-follow examples.
Get started with Grafana Alerting
Create and receive your first alert
Create multi-dimensional alerts and route them
Group alert notifications
Template your alert notifications
Additional tutorials
Route alerts using dynamic labels
Link alerts to visualizations
Create alerts with log data
Create alerts with InfluxDB and Flux queries

Templates
Use templating to customize, format, and reuse alert notification messages. Create more flexible and informative alert notification messages by incorporating dynamic content, such as metric values, labels, and other contextual information.
In Grafana, you have various options to template your alert notification messages:
Alert rule annotations
Annotations add extra information, like 
summary and 
description, to alert instances for notification messages.
Template annotations to display query values that are meaningful to the alert, for example, the server name or the threshold query value.
Alert rule labels
Labels are used to differentiate an alert instance from all other alert instances.
Template labels to add an additional label based on a query value, or when the labels from the query are incomplete or not descriptive enough.
Avoid displaying query values in labels as this can create numerous alert instances—use annotations instead.
Notification templates
Notification templates are used by contact points for consistent messaging in notification titles and descriptions.
Template notifications when you want to customize the appearance and information of your notifications.
Avoid using notification templates to add extra information to alert instances—use annotations instead.
{{< admonition type="tip" >}}
For a practical example of templating, refer to our Getting Started with Templating tutorial.
{{< /admonition  >}}
This diagram illustrates the entire templating process, from querying labels and templating the alert summary and notification to the final alert notification message.
{{< figure src="/media/docs/alerting/how-notification-templates-works.png" max-width="1200px" caption="How templating works" >}}
In this diagram:
The alert rule query returns 
12345, along with the values of the 
instance and 
job labels.
This query result breaches the alert rule condition, firing the alert instance.
The alert instance generates an annotation summary, defined by the template used in the alert rule summary. In this case, it displays the value of the 
instance label: 
server1.
The Alertmanager receives the firing alert instance, including the final annotation summary, and determines the contact point that will process the alert.
The Alertmanager uses the contact point's notification template to format the message, then sends the notification to the configured destination(s)—an email address.
Template annotations
Annotations can be defined in the alert rule to add extra information to alert instances.
When creating an alert rule, Grafana suggests several optional annotations, such as 
description, 
summary, and 
runbook_url, which help identify and respond to alerts. You can also create custom annotations.
Annotations are key-value pairs, and their values can contain a combination of text and template code that is evaluated when the alert fires.
Annotations can contain plain text, but you should template annotations if you need to display query values that are relevant to the alert, for example:
Show the query value that triggers the alert.
Include labels returned by the query that identify the alert.
Format the annotation message depending on a query value.
Here’s an example of templating an annotation, which explains where and why the alert was triggered. In this case, the alert triggers when CPU usage exceeds a threshold, and the 
summary annotation provides the relevant details.
CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) for the last 5 minutes.
The outcome of this template would be:
CPU usage for Instance 1 has exceeded 80% (81.2345) for the last 5 minutes.
Implement annotations that provide meaningful information to respond to your alerts. Annotations are displayed in the Grafana alert detail view and are included by default in notifications.
For more details on how to template annotations, refer to Template annotations and labels.
Template labels
Labels are used to differentiate one alert instance from all other alert instances, as the set of labels uniquely identifies an alert instance. Notification policies and silences use labels to handle alert instances.
You can also template labels based on query results. This is helpful if the labels you get from your query aren't detailed enough. For instance:
Add a new label to change how alerts are identified and grouped into different alert groups.
Add a new label used by notification policies or silences to manage how the alert is handled.
Here’s an example of templating a new 
env label based on the value of a query label:
{{- if eq $labels.instance "prod-server-1" -}}
production
{{- else if eq $labels.instance "staging-server-1" -}}
staging
{{- else -}}
development
{{- end -}}
For more details on how to template labels, refer to Template annotations and labels.
Template notifications
Notification templates allow you to customize the content of your notifications, such as the subject of an email or the body of a Slack message.
Notification templates differ from templating annotations and labels in the following ways:
Notification templates are assigned to the Contact point, rather than the alert rule.
If not specified, the contact point uses a default template that includes relevant alert information.
The same template can be shared across multiple contact points, making it easier to maintain and ensuring consistency.
Notification templates should not be used to add additional information to individual alerts—use annotations for that purpose.
While both annotation/label templates and notification templates use the same templating language, the available variables and functions differ. For more details, refer to the notification template reference and annotation/label template reference.
Here is an example of a notification template that summarizes all firing and resolved alerts in a notification group:
{{ define "alerts.message" -}}
  {{ if .Alerts.Firing -}}
    {{ len .Alerts.Firing }} firing alert(s)
    {{ template "alerts.summarize" .Alerts.Firing }}
  {{- end }}
  {{- if .Alerts.Resolved -}}
    {{ len .Alerts.Resolved }} resolved alert(s)
    {{ template "alerts.summarize" .Alerts.Resolved }}
  {{- end }}
{{- end }}
{{ define "alerts.summarize" -}}
  {{ range . -}}
  - {{ index .Annotations "summary" }}
  {{ end }}
{{ end }}
The notification message to the contact point would look like this:
1 firing alert(s)
- The database server db1 has exceeded 75% of available disk space. Disk space used is 76%, please resize the disk size within the next 24 hours.
1 resolved alert(s)
- The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.
For more details, refer to Template notifications.

Alert rule state and health
Each alert rule can generate one or more alert instances—one alert instance for each series or dimension, as shown in the multi-dimensional alert example.
Each alert instance of the same alert rule represents a different target and can be in a different state; for example, one alert instance may be Normal while another is Alerting.
{{< figure src="/media/docs/alerting/alert-rule-example-multiple-alert-instances.png" max-width="750px" alt="Multi dimensional alert rule. The alert rule state and alert rule health are determined by the state of the alert instances." >}}
The alert rule state and alert rule health are determined by the state of the alert instances.
Alert rule states
An alert rule can be in either of the following states:
State
Description
Normal
None of the alert instances returned by the evaluation engine is in a 
Pending or 
Alerting state.
Pending
At least one alert instances returned by the evaluation engine is 
Pending.
Firing
At least one alert instances returned by the evaluation engine is 
Alerting.
Alert rule health
An alert rule can have one of the following health statuses:
State
Description
Ok
No error when evaluating the alert rule.
Error
An error occurred when evaluating the alert rule.
No Data
The alert rule query returns no data.
{status}, KeepLast
The rule would have received another status but was configured to keep the last state of the alert rule.

How rules are evaluated within a group
Each evaluation group contains an evaluation interval that determines how frequently the rule is evaluated. For instance, the evaluation may occur every 
10s, 
30s, 
1m, 
10m, etc.
Rules in different evaluation groups can be evaluated simultaneously.
Rules within the same evaluation group can be evaluated simultaneously or sequentially, depending on the rule type:
Grafana-managed rules within the same group are evaluated concurrently—they are evaluated at different times over the same evaluation interval but display the same evaluation timestamp.
Data source-managed rules within the same group are evaluated sequentially, one after the other—this is useful to ensure that recording rules are evaluated before alert rules.
Grafana-managed rules imported from data source-managed rules are also evaluated sequentially, in the order they appear in the original rule group. Grafana runs them as an evaluation sequence: each rule starts after the previous one finishes. This preserves the evaluation order when a rule depends on a preceding recording rule.
For Grafana-managed rules that depend on recording rule results, you can define an explicit evaluation order with a rule sequence. The sequence evaluates recording rules first, followed by alert rules, at a shared interval. For instructions, refer to Configure rule sequences.

No Data and Error states
Grafana Alerting implements the No Data and Error states to handle common scenarios when evaluating alert rules, and you can modify their behavior.
An alert instance can transition to these special states:
No Data state occurs when the alert rule query runs successfully but returns no data points.
Error state occurs when the alert rule fails to evaluate its query or queries successfully.
{{< admonition type="note" >}}
No Data and Error states are supported only for Grafana-managed alert rules.
{{< /admonition  >}}
{{< admonition type="tip" >}}
For common examples and practical guidance on handling Error, No Data, and stale alert scenarios, refer to the Handle connectivity errors and Handle missing data guides.
{{< /admonition  >}}
Error state
The Error state is triggered when the alert rule fails to evaluate its query or queries successfully.
This can occur due to evaluation timeouts (default: 
30s) or repeated failures (default: 
3) when querying the data source. The 
evaluation_timeout and 
max_attempts options control these settings.
The Error state honors the pending period: the alert instance transitions 
Normal → Pending → Error. Set the pending period to 
0 to skip the Pending state and enter Error immediately.
When an alert instance enters the Error state, Grafana, by default, triggers a new 
DatasourceError alert. You can control this behavior based on the desired outcome of your alert rule in Modify the 
No Data or 
Error state.
No Data state
The No Data state occurs when the alert rule query runs successfully but returns no data points at all.
The No Data state honors the pending period: the alert instance transitions 
Normal → Pending → No Data. Set the pending period to 
0 to skip the Pending state and enter No Data immediately.
When an alert instance enters the No Data state, Grafana, by default, triggers a new 
DatasourceNoData alert. You can control this behavior based on the desired outcome of your alert rule in Modify the 
No Data or 
Error state.
Modify the 
No Data or 
Error state
{{< figure src="/media/docs/alerting/screenshot-configure-no-and-error-handling.png" alt="A screenshot of the configure no data and error handling section in the alert rule form." max-width="750px" >}}
In Configure no data and error handling, you can configure the behavior when the evaluation returns no data or all values are null:
{{< docs/shared lookup="alerts/modify-nodata-state.md" source="grafana" version="<GRAFANA_VERSION>" >}}
You can also configure the behavior when the evaluation returns an error or timeout:
{{< docs/shared lookup="alerts/modify-error-state.md" source="grafana" version="<GRAFANA_VERSION>" >}}
Note that when you configure the No Data or Error behavior to set the Alerting or Normal state, Grafana attempts to keep a stable set of fields under notification 
Values. If your query returns no data or an error, Grafana re-uses the latest known set of fields in 
Values, but will use 
-1 in place of the measured value.
Keep last state
The "Keep Last State" option helps mitigate temporary data source issues, preventing alerts from unintentionally firing, resolving, and re-firing.
However, in situations where strict monitoring is critical, relying solely on the "Keep Last State" option may not be appropriate. Instead, consider using an alternative or implementing additional alert rules to ensure that issues with prolonged data source disruptions are detected.
No Data and 
Error alerts
When an alert rule evaluation results in a 
No Data or 
Error state, Grafana Alerting creates a new alert instance with the following additional labels:
alertname: Either 
DatasourceNoData or 
DatasourceError depending on the state.
datasource_uid: The UID of the data source that caused the state.
rulename: The name of the alert rule that originated the alert.
Note that 
DatasourceNoData and 
DatasourceError alert instances are independent from the original alert instance. They have different labels, which means existing silences, mute timings, and notification policies applied to the original alert may not apply to them.
You can manage these alerts like regular ones by using their labels to apply actions such as adding a silence, routing via notification policies, and more.
If the alert rule is configured to send notifications directly to a selected contact point (instead of using notification policies), the 
DatasourceNoData and 
DatasourceError alerts are also sent to that contact point. Any additional notification settings defined in the alert rule, such as muting or grouping, are preserved.
Reduce 
No Data or 
Error alerts
To minimize the number of No Data or Error state alerts received, try the following.
Use the Keep last state option. For more information, refer to the section below. This option allows the alert to retain its last known state when there is no data available, rather than switching to a No Data state.
For No Data alerts, you can optimize your alert rule by expanding the time range of the query. However, if the time range is too big, it affects the performance of the query and can lead to errors due to timeout.
To minimize timeouts resulting in the Error state, reduce the time range to request less data every evaluation cycle.
Consider increasing the pending period. Since the No Data and Error states honor the pending period, a longer period can suppress 
DatasourceNoData and 
DatasourceError notifications for transient issues.
To reduce multiple notifications from Error alerts, define a notification policy to handle all related alerts with 
alertname=DatasourceError, and filter and group errors from the same data source using the 
datasource_uid label.
Change the evaluation timeout (default: 
30s) or the retry mechanism (
max_attempts) settings. This should be a last resort, as it can affect the performance of all alert rules and may cause missed evaluations if the timeout is too long. For Grafana Cloud, open a support ticket from the Cloud Portal.
{{< admonition type="tip" >}}
For common examples and practical guidance on handling Error, No Data, and stale alert scenarios, refer to the Handle connectivity errors and Handle missing data guides.
{{< /admonition  >}}
grafana_state_reason for troubleshooting
Occasionally, an alert instance may be in a state that isn't immediately clear to everyone. For example:
If "no data" handling is configured to transition to a state other than 
No Data.
If "error" handling is configured to transition to a state other than 
Error.
If the alert rule is deleted, paused, or updated in some cases, the alert instance also transitions to the 
Normal state.
Stale alert instances in the 
Alerting state transition to the 
Normal state when the series disappear.
In these situations, the evaluation state may differ from the alert state, and it might be necessary to understand the reason for being in that state when receiving the notification.
The 
grafana_state_reason annotation is included in these situations, providing the reason that explains why the alert instance transitioned to its current state. For example:
If "no data" or "error" handling transitions to the 
Normal state, the 
grafana_state_reason annotation is included with the value No Data or Error, respectively.
If the alert rule is deleted or paused, the 
grafana_state_reason is set to Paused or RuleDeleted. For some updates, it is set to Updated.
Stale alert instances in the 
Normal state include the 
grafana_state_reason annotation with the value MissingSeries.

Stale alert instances
An alert instance is considered stale if the alert rule query returns data but its dimension (or series) has disappeared for a number of consecutive evaluation intervals (2 by default).
This is different from the No Data state, which occurs when the alert rule query runs successfully but returns no dimensions (or series) at all.
A stale alert instance transitions to the Normal (MissingSeries) state as Resolved, and is then evicted:
Eval. Interval
1
2
3
4
Alert instance A
✔
✔
✔
✔
Alert instance B
✔
MissingSeries
️
Normal(MissingSeries) 📩*
{{< admonition type="note" >}}
Stale alert instances are supported only for Grafana-managed alert rules.
{{< /admonition  >}}
How Grafana handles stale alert instances
The process for handling stale alert instances is as follows:
The alert rule runs and returns data for some label sets.
An alert instance that previously existed is now missing.
Grafana keeps the previous state of the alert instance for the number of evaluation intervals specified in Missing series evaluations to resolve.
If it remains missing after the specified number of evaluation intervals (2 by default), it transitions to the Normal state and sets MissingSeries in the 
grafana_state_reason annotation.
Stale alert instances in the Alerting, No Data, or Error states transition to the Normal state as Resolved, and are routed for notifications like other resolved alerts.
The alert instance is removed from the UI.
{{< admonition type="tip" >}}
For common examples and practical guidance on handling No Data and stale alert scenarios, see Handling missing data.
{{< /admonition  >}}
Configure Missing series evaluations to resolve
In Configure no data and error handling > Missing series evaluations to resolve, you can set how many consecutive evaluation intervals must pass without data for a given dimension before the alert instance is marked as stale and resolved.
If you don't specify a value, Grafana uses the default of 2 evaluation intervals.

Labels and annotations
Labels and annotations add additional information about an alert using key/value pairs:
Labels are used to differentiate an alert from all other alerts and decide how to manage them.
Annotations provide extra details for alert responders to help them understand and address potential issues.
Labels
{{< shared id="labels-basics" >}}
Labels are unique identifiers of an alert instance. You can use them for searching, silencing, and routing notifications.
Examples of labels are 
server=server1 or 
team=backend. Each alert rule can have more than one label and the complete set of labels for an alert rule is called its label set. It is this label set that identifies the alert.
{{< /shared >}}
For example, an alert instance might have the label set 
{alertname="High CPU usage",server="server1"} while another alert instance might have the label set 
{alertname="High CPU usage",server="server2"}. These are two separate alert instances because although their 
alertname labels are the same, their 
server labels are different.
{{< figure alt="Image shows an example of an alert instance and the labels used on the alert instance." src="/static/img/docs/alerting/unified/multi-dimensional-alert.png" >}}
Labels are a fundamental component of alerting:
The complete set of labels for an alert is what uniquely identifies an alert instance.
The alerting UI shows labels for every alert instance generated during evaluation of that rule.
Notification policies and silences use labels to match alert instances and route them to contact points or stop their notifications.
Contact points can include information from labels in notification messages.
Label types
An alert's label set can contain three types of labels:
User-configured labels
Labels that you manually configure in the alert rule to identify the generated alert instances and manage the alerts. Common custom labels, depending on the use case, are: 
severity, 
priority, 
team, and 
service.
Additionally, you can use a template to customize the label value and generate dynamic values from query data.
Query labels
Query labels are labels returned by the data source query.
{{< figure src="/media/docs/alerting/query-labels-and-values.png" max-width="1200px" caption="An alert rule query returning labels from the query." >}}
Query labels can generate multiple alert instances from the same alert rule, helping to distinguish alerts from different data. In this example, the 
instance label generates an alert instance for each server.
Reserved labels
Reserved labels are automatically added by Grafana:
alertname: the name of the alert rule.
grafana_folder: the title of the folder containing the alert.
Labels prefixed with 
grafana_ are reserved by Grafana for special use. You can disable reserved labels via the 
unified_alerting.reserved_labels option.
{{}}
Two alert rules cannot produce alert instances with the same labels. If two alert rules have the same labels such as 
foo=bar,bar=baz and 
foo=bar,bar=baz then one of the generated alert instances is discarded.
Ensure the label set for an alert does not have two or more labels with the same name.
If a configured label has the same name as a data source query label, it replaces the data source label.
If a configured label has the same name as a reserved label, it is omitted.
{{}}
{{< collapse title="Label key format" >}}
Grafana has a built-in Alertmanager that supports both Unicode label keys and values. If you are using an external Prometheus Alertmanager, label keys must be compatible with their data model.
This means that label keys must only contain ASCII letters, numbers, and underscores.
Label keys must also be matched by the regular expression 
[a-zA-Z_][a-zA-Z0-9_]*.
Any invalid characters are removed or replaced by the Grafana alerting engine before being sent to the external Alertmanager according to the following rules:
Whitespace is removed.
ASCII characters are replaced with 
_.
All other characters are replaced with their lower-case hex representation.
If this is the first character, it's prefixed with 
_.
Example: A label key/value pair 
Alert! 🔔="🔥" will become 
Alert_0x1f514="🔥".
If multiple label keys are sanitized to the same value, the duplicates have a short hash of the original label appended as a suffix.
{{< /collapse >}}
Annotations
{{< shared id="annotations-basics" >}}
Annotations add additional information to alert instances, helping responders identify and address potential issues.
Create clear and self-explanatory annotations so that first responders can investigate without needing deeper knowledge of the alert setup.
Annotations are displayed in Grafana and are included by default in notifications. Grafana provides several optional annotations that you can edit:
summary: A short summary of what the alert has detected and why.
description: A detailed description of what happened and what the alert does.
runbook_url: The runbook page to guide operators managing a potential incident.
__dashboardUid__ and 
__panelId__: Link the alert to a dashboard and panel to facilitate alert investigation.
{{< /shared >}}
Unlike labels, annotations do not affect notification policies or silences. For example, if 
team=backend exists only in an annotation, a child policy that matches 
team=backend does not receive the alert; the Default notification policy handles it instead.
You can edit annotations such as 
summary to explain why the alert was triggered:
CPU usage has exceeded 80% for the last 5 minutes.
Edit the 
description annotation to provide more context and how to respond:
The web server's CPU has exceeded 80% for more than 5 minutes.
This indicates that the system is under heavy load and may result in an outage.
Consider scaling the server's resources and investigating bottlenecks.
Like labels, annotations can use a template to include dynamic data from queries.

Queries and conditions
In Grafana, queries fetch and transform data from data sources, which include databases like MySQL or PostgreSQL, time series databases like Prometheus or InfluxDB, and services like Amazon CloudWatch or Azure Monitor.
An alert rule defines the following components:
A query that specifies the data to retrieve from a data source, with the syntax depending on the type of data source used.
A condition that must be met before the alert rule fires.
Optional expressions to perform transformations on the retrieved data.
Alerting periodically runs the queries and expressions, evaluating the condition. If the condition is breached, an alert instance is triggered for each time series.
Data source queries
Alerting queries are the same as the queries used in Grafana panels, but Grafana-managed alerts are limited to querying data sources that have Alerting enabled.
Queries in Grafana can be applied in various ways, depending on the data source and query language being used. Each data source’s query editor provides a customized user interface to help you write queries that take advantage of its unique capabilities. For details about query editors and syntax in Grafana, refer to Query and transform data.
Alerting can work with two types of data:
Time series data — The query returns a collection of time series, where each series must be reduced to a single numeric value for evaluating the alert condition.
Tabular data — The query must return data in a table format with only one numeric column. Each row must have a value in that column, used to evaluate the alert condition. See a tabular data example.
Each time series or table row is evaluated as a separate alert instance.
{{< figure src="/media/docs/alerting/alerting-query-conditions-default-options.png" max-width="750px" caption="Alert query using the Prometheus query editor and alert condition" >}}
Alert condition
The alert condition is the query or expression that determines whether the alert fires or not depending whether the value satisfies the specified comparison. There can be only one condition which determines the triggering of the alert.
If the queried data meets the defined condition, Grafana fires the alert.
When using Default options, the 
When input reduces the query data, and the last input defines the threshold condition.
When using Advanced options, you have to choose one of your queries or expressions as the alert condition.
Advanced options: Expressions
Expressions are only available for Grafana-managed alerts and when the Advanced options are enabled.
In Grafana, expressions allow you to perform calculations, transformations, or aggregations on queried data. They modify existing metrics through mathematical operations, functions, or logical expressions.
With expression queries, you can perform tasks such as calculating the percentage change between two values, applying functions like logarithmic or trigonometric functions, aggregating data over specific time ranges or dimensions, and implementing conditional logic to handle different scenarios.
{{< figure src="/media/docs/alerting/alert-rule-expressions.png" max-width="750px" caption="Alert rule expressions" >}}
The following expressions are available:
Reduce
Aggregates time series values within the selected time range into a single number.
Reduce takes one or more time series and transforms each series into a single number, which can then be compared in the alert condition.
The following aggregations functions are included: 
Min, 
Max, 
Mean, 
Median, 
Sum, 
Count, and 
Last. For more details, refer to the Reduce documentation.
Math
Performs free-form math functions/operations on time series data and numbers. For example, 
$A + 1 or 
$A * 100.
If queries being compared have multiple series in their results, series from different queries are matched(joined) if they have the same labels. For example:
{{< docs/shared lookup="alerts/math-example.md" source="grafana" version="<GRAFANA_VERSION>" >}}
In this case, only series with matching labels are joined, and the operation is calculated between them.
For additional scenarios on how Math handles different data types, refer to the Math documentation.
You can also use a Math expression to define the alert condition. For example:
$B > 70 should fire if the value of B (query or expression) is more than 70.
$B < $C * 100 should fire if the value of B is less than the value of C multiplied by 100.
Compare matching series from two queries, as shown in the dynamic threshold example.
Resample
Realigns a time range to a new set of timestamps. This is useful when comparing time series data from different data sources where the timestamps would otherwise not align.
For more details, refer to the Resample documentation.
SQL
Transforms the results of previous queries or expressions using MySQL-like syntax, which is useful for filtering rows, joining data from multiple queries, and aggregating results before evaluating the alert condition.
For more details, refer to the SQL expressions documentation.
Threshold
Compares single numbers from previous queries or expressions (e.g., 
$A, 
$B) to a specified condition. It's often used to define the alert condition.
The threshold expression allows the comparison between two single values. Available threshold functions are:
Is above: 
$A > 5
Is below: 
$B < 3
Is equal to: 
$A == 2
Is not equal to: 
$B != 4
Is above or equal to: 
$A >= 8
Is below or equal to: 
$B <= 16
Is within range: 
$A > 0 AND $A < 10
Is outside range: 
$B < 0 OR $B > 100
Is within range included: 
$A >= 0 AND $A <= 10
Is outside range included: 
$B <= 0 OR $B >= 100
A threshold returns 
0 when the condition is false and 
1 when true.
If the threshold is set as the alert condition, the alert fires when the threshold returns 
1.
Recovery threshold
To reduce the noise from flapping alerts, you can set a recovery threshold so that the alert returns to the 
Normal or 
Recovering state only after the recovery threshold is crossed.
Flapping alerts occur when the query value repeatedly crosses above and below the alert threshold, causing frequent state changes. This results in a series of firing-resolved-firing notifications and a noisy alert state history.
For example, if you have an alert for latency with a threshold of 1000ms and the number fluctuates around 1000 (say 980 -> 1010 -> 990 -> 1020, and so on), then each of those might trigger a notification:
980 -> 1010 triggers a firing alert.
1010 -> 990 triggers a resolving alert.
990 -> 1020 triggers a firing alert again.
To prevent this, you can set a recovery threshold to define two thresholds instead of one:
An alert transitions to the 
Pending or 
Alerting state when it crosses the alert threshold.
It then transitions to the 
Recovering or 
Normal state only when it crosses the recovery threshold.
In the previous example, setting the recovery threshold to 900ms means the alert only resolves when the latency falls below 900ms:
980 -> 1010 triggers a firing alert.
1010 -> 990 does not resolve the alert, keeping it in the firing state.
990 -> 1020 keeps the alert in the firing state.
The recovery threshold mitigates unnecessary alert state changes and reduces alert noise.
{{< collapse title="Classic condition (legacy)" >}}
Classic condition (legacy)
Classic conditions exist mainly for compatibility reasons and should be avoided if possible.
Classic condition checks if any time series data matches the alert condition. It always produce one alert instance only, no matter how many time series meet the condition.
Condition operators
How it works
and
Two conditions before and after must be true for the overall condition to be true.
or
If one of conditions before and after are true, the overall condition is true.
logic-or
If the condition before 
logic-or is true, the overall condition is immediately true, without evaluating subsequent conditions. For instance, 
TRUE and TRUE logic-or FALSE and FALSE evaluate to 
TRUE, because the preceding condition returns 
TRUE.
The following aggregation functions are also available to further refine your query.
Function
What it does
avg
Displays the average of the values
min
Displays the lowest value
max
Displays the highest value
sum
Displays the sum of all values
count
Counts the number of values in the result
last
Displays the last value
median
Displays the median value
diff
Displays the difference between the newest and oldest value
diff_abs
Displays the absolute value of diff
percent_diff
Displays the percentage value of the difference between newest and oldest value
percent_diff_abs
Displays the absolute value of 
percent_diff
count_non_null
Displays a count of values in the result set that aren't 
null
{{< /collapse >}}

Contact points
Contact points contain the configuration for sending alert notifications. You can assign a contact point either in the alert rule or notification policy options.
A contact point includes one or more contact point integrations for sending alert notifications, such as:
{{< column-list >}}
Alertmanager
Amazon SNS
Cisco Webex Teams
DingDing
Discord
Email
Google Chat
Grafana IRM
Jira
Kafka REST Proxy
Line
Microsoft Teams
MQTT
Opsgenie
PagerDuty
Pushover
Sensu Go
Slack
Telegram
Threema Gateway
VictorOps
Webhook
WeCom
{{< /column-list >}}
For example, a contact point could contain a PagerDuty integration; an email and Slack integration; or a PagerDuty integration, a Slack integration, and two email integrations. You can also configure a contact point with no integrations; in which case no notifications are sent.
Each contact point integration can also define the notification message to be sent, which can use the predefined message, a custom message, or notification templates.
For a complete list of supported integrations and more details about contact points, refer to Configure contact points.

Group alert notifications
Grouping in Grafana Alerting allows you to batch relevant alerts together into a smaller number of notifications. This is particularly important if notifications are delivered to first-responders, such as engineers on-call, where receiving lots of notifications in a short period of time can be overwhelming. In some cases, it can negatively impact a first-responder's ability to respond to an incident. For example, consider a large outage where many of your systems are down. In this case, grouping can be the difference between receiving 1 phone call and 100 phone calls.
{{< admonition type="tip" >}}
For a practical example of grouping, refer to our Getting Started with Grouping tutorial.
{{< /admonition  >}}
Group notifications
Grouping combines similar alert instances within a specific period into a single notification, reducing alert noise.
Grafana Alerting uses alert group for this temporary notification batch. In Grafana IRM, the same term refers to a persistent object with a status, responders, and an escalation chain.
In the notification policy, you can configure how to group multiple alerts into a single notification:
The 
Group by option specifies the criteria for grouping incoming alerts within the policy. The default is by alert rule.
Timing options determine when and how often to send the notification.
{{< figure src="/media/docs/alerting/alerting-notification-policy-diagram-with-labels-v3.png" max-width="750px" alt="A diagram about the components of a notification policy, including labels and groups" >}}
Alert instances are grouped together if they have the same exact label values for the labels configured in the 
Group by option.
For example, given the 
Group by option set to the 
team label:
alertname:foo, team=frontend, and 
alertname:bar, team=frontend are in one group.
alertname:foo, team=frontend, and 
alertname:qux, team=backend are in another group.
Group by alert rule or labels
By default, notification policies in Grafana group alerts by the alert rule. Specifically, they are grouped using the 
alertname and 
grafana_folder labels, as alert rule names are not unique across folders.
If you want to group alerts by other labels, something other than the alert rule, change the 
Group by option to any other combination of labels.
A single group for all alerts
If you want to group all alerts handled by the notification policy in a single group (without grouping notifications by alert rule or other labels), leave 
Group by empty in the Default policy.
Disable grouping
If you want to receive every alert as a separate notification, you can do so by grouping by a special label called 
..., ensuring that other labels are not present.
Timing options
In the notification policy, you can also configure how often notifications are sent for each group of alerts. There are three distinct timers applied to groups within the notification policy:
Group wait: the time to wait before sending the first notification for a new group of alerts.
Group interval: the time to wait before sending a notification about changes in the alert group.
Repeat interval: the time to wait before sending a notification if the group has not changed since the last notification.
These timers reduce the number of notifications sent. By delaying the delivery of notifications, incoming alerts can be grouped into just one notification instead of many.
{{< figure src="/media/docs/alerting/alerting-timing-options-flowchart-v2.png" max-width="750px" alt="A basic sequence diagram of the notification policy timers" caption="A basic sequence diagram of the notification policy timers" >}}
Group wait
Default: 30 seconds
Group wait is the duration Grafana waits before sending the first notification for a new group of alerts.
This option helps reduce the number of notifications sent for related alerts occurring within a short time frame. The longer the group wait, the more time other alerts have to be included in the initial notification of the new group. The shorter the group wait, the earlier the first notification is sent, but at the risk of not including some alerts.
If an alert is resolved before the duration elapses, no notification is sent for that alert. This reduces noise from flapping alerts.
Example
Consider a notification policy that:
Matches all alert instances with the 
team label—matching labels equals to 
team=~.+.
Groups notifications by the 
team label—one group for each distinct 
team.
Sets the Group wait timer to 
30s.
Time
Incoming alert instance
Notification policy group
Number of instances
00:00
alertname=f1 
team=frontend
frontend
1
Starts the group wait timer of the 
frontend group.
00:10
alertname=f2 
team=frontend
frontend
2
00:20
alertname=b1 
team=backend
backend
1
Starts the group wait timer of the 
backend group.
00:30*
frontend
2
Group wait elapsed. 
 Send initial notification reporting 2 alerts.
00:35
alertname=b2 
team=backend
backend
2
00:40
alertname=b3 
team=backend
backend
3
00:50*
backend
3
Group wait elapsed. 
 Send initial notification reporting 3 alerts.
Group interval
Default: 5 minutes
If an alert was too late to be included in the first notification due to group wait, it is included in subsequent notifications after group interval.
Group interval is the duration to wait before sending notifications about group changes. A group change occurs when:
A new firing alert is added to the group.
An existing alert is resolved.
Example
Here are the related excerpts from the previous example:
Time
Incoming alert instance
Notification policy group
Number of instances
00:30*
frontend
2
Group wait elapsed and starts Group interval timer. 
 Send initial notification reporting 2 alerts.
00:50*
backend
3
Group wait elapsed and starts Group interval timer. 
 Send initial notification reporting 3 alerts.
And below is the continuation of the example setting the Group interval timer to 5 minutes:
Time
Incoming alert instance
Notification policy group
Number of instances
01:30
alertname=f3 
team=frontend
frontend
3
02:30
alertname=f4 
team=frontend
frontend
4
05:30*
frontend
4
Group interval elapsed and resets timer. 
 Send one notification reporting 4 alerts.
05:50*
backend
3
Group interval elapsed and resets timer. 
 No group changes, and do not send notification.
08:00
alertname=f4 
team=backend
backend
4
10:30*
frontend
4
Group interval elapsed and resets timer. 
 No group changes, and do not send notification.
10:50*
backend
4
Group interval elapsed and resets timer. 
 Send one notification reporting 4 alerts.
How it works
Once the first notification has been sent for a new group of alerts, the group interval timer starts.
When the group interval timer elapses, the system resets the group interval timer and sends a notification only if there were group changes. This process repeats until there are no more alerts.
It's important to note that an alert instance exits the group after being resolved and notified of its state change.
When the group interval timer elapses and no alerts remain, the group is deleted. The group wait timer will then start again the next time a new alert arrives.
Repeat interval
Default: 4 hours
Repeat interval acts as a reminder that alerts in the group are still firing.
The repeat interval timer decides how often notifications are sent (or repeated) if the group has not changed since the last notification.
How it works
Repeat interval is evaluated every time the group interval resets. If the alert group has not changed and the time since the last notification was longer than the repeat interval, then a notification is sent as a reminder that the alerts are still firing.
Repeat interval must not only be greater than or equal to group interval, but also must be a multiple of Group interval. If Repeat interval is not a multiple of group interval it is coerced into one. For example, if your Group interval is 5 minutes, and your Repeat interval is 9 minutes, the Repeat interval is rounded up to the nearest multiple of 5 which is 10 minutes.
The maximum duration of the repeat interval is 5 days, constrained by the default value of the 
notification_log_retention setting in Grafana.
Example
Here are the related excerpts from the previous example:
Time
Incoming alert instance
Notification policy group
Number of instances
05:30*
frontend
4
Group interval resets. 
 Send the last notification.
10:50*
backend
4
Group interval resets. 
 Send the last notification.
And below is the continuation of the example setting the Repeat interval timer to 4 hours:
Time
Incoming alert instance
Notification policy group
Number of instances
04:05:30
frontend
4
Group interval resets. The time since the last notification was no longer than the repeat interval.
04:10:30
frontend
4
Group interval resets. The time since the last notification was longer than the repeat interval. 
 Send one notification reminding the 4 firing alerts.
04:10:50
backend
4
Group interval resets. The time since the last notification was no longer than the repeat interval.
04:15:50
backend
4
Group interval resets. The time since the last notification was longer than the repeat interval. 
 Send one notification reminding the 4 firing alerts.

Notification policies
Notification policies provide you with a flexible way of designing how to handle notifications and minimize alert noise.
Using label matchers, alert instances are routed to notification policies. The notification policy can then group multiple alert instances into a single notification and deliver it to the contact point.
{{< figure src="/media/docs/alerting/how-alerting-works.png" max-width="750px" alt="How Alerting works" >}}
Notification policies are not a list, but rather are structured according to a tree structure:
The root of the notification policy tree is the Default notification policy.
Each policy can have child policies.
Each policy can have sibling policies, sharing the same parent and hierarchical level.
Each policy consists of a set of label matchers (0 or more) that specify which alerts they are or aren't interested in handling. A matching policy refers to a notification policy with label matchers that match the alert instance’s labels.
{{< collapse title="How label matching works" >}}
Use labels and label matchers to link alert rules to notification policies and silences. This allows for a flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.
A label matcher consists of 3 distinct parts, the label, the value and the operator.
The Label field is the name of the label to match. It must exactly match the label name.
The Value field matches against the corresponding value for the specified Label name. How it matches depends on the Operator value.
The Operator field is the operator to match against the label value. The available operators are:
Operator
Description
=
Select labels that are exactly equal to the value.
!=
Select labels that are not equal to the value.
=~
Select labels that regex-match the value.
!~
Select labels that do not regex-match the value.
{{< admonition type="note" >}}
If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match to link a rule to a policy.
{{< /admonition >}}
By default, Grafana uses a single notification policy tree for all alert routing. You can also create multiple notification policy trees to split routing logic into separate, independently managed trees.
Label matching example
If you define the following set of labels for your alert:
{ foo=bar, baz=qux, id=12 }
then:
A label matcher defined as 
foo=bar matches this alert rule.
A label matcher defined as 
foo!=bar does not match this alert rule.
A label matcher defined as 
id=~[0-9]+ matches this alert rule.
A label matcher defined as 
baz!~[0-9]+ matches this alert rule.
Two label matchers defined as 
foo=bar and 
id=~[0-9]+ match this alert rule.
Exclude labels
You can also write label matchers to exclude labels.
Here is an example that shows how to exclude the label 
Team. You can choose between any of the values below to exclude labels.
Label
Operator
Value
team
=
""
team
!~
.+
team
=~
^$
{{< /collapse >}}
{{< figure src="/media/docs/alerting/notification-routing.png" max-width="750px" caption="Matching alert instances with notification policies" alt="Example of a notification policy tree" >}}
{{< admonition type="tip" >}}
For a practical example of routing with notification policies, refer to our Getting Started with Alert Instances and Notification Routing tutorial.
{{< /admonition  >}}
Routing
To determine which notification policies handle an alert instance, the system looks for matching policies starting from the top of the tree—beginning with the default notification policy.
If a matching policy is found, the system continues to evaluate its child policies in the order they are displayed. If a child policy matches the alert, the system then evaluates its child policies recursively until no more matching child policies are found. In this case, only the deepest matching child policy handles the alert instance.
By default, once a matching policy is found, the system does not continue to look for sibling policies. If you want sibling policies of one matching policy to handle the alert instance as well, then enable Continue matching siblings on the particular matching policy.
{{< admonition type="note" >}}
The default notification policy matches all alert instances. It always handles alert instances if there are no child policies or if none of the child policies match the alert instance's labels—this prevents any alerts from being missed.
If alerts use multiple labels, these labels must also be present in a notification policy to match and route notifications to a specific contact point.
{{< /admonition >}}
{{< collapse title="Routing example" >}}
Here's a breakdown of the previous example:
Pod stuck in CrashLoop does not have a 
severity label, so none of its child policies are matched. It does have a 
team=operations label, so the first policy is matched.
The 
team=security policy is not a match and Continue matching siblings was not configured for that policy.
Disk Usage – 80% has both a 
team and 
severity label, and matches a child policy of the operations team.
{{< admonition type="note" >}}
When an alert matches both a parent policy and a child policy (like it does in this case), the routing follows the child policy (
severity) as it provides a more specific match.
{{< /admonition >}}
Unauthorized log entry has a 
team label but does not match the first policy (
team=operations) since the values are not the same, so it will continue searching and match the 
team=security policy. It does not have any child policies, so the additional 
severity=high label is ignored.
{{< /collapse >}}
This routing and tree structure makes it convenient to organize and handle alerts for dedicated teams, while also narrowing down specific cases within the team by applying additional labels.
Inheritance
In addition to child policies being a useful concept for routing alert instances, they also inherit properties from their parent policy. This also applies to child policies of the default notification policy.
By default, a child policy inherits the following notification properties from its parent:
Contact point
Grouping options
Timing options
Then, each policy can overwrite these properties if needed.
The inheritance of notification properties, together with the routing process, is an effective method for grouping related notifications and handling specific cases through child policies.
Inheritance example
{{< figure src="/media/docs/alerting/notification-inheritance.png" max-width="750px" alt="Simple example inhering notification settings" >}}
This example shows how the notification policy tree from the previous example allows the child policies of the 
team=operations to inherit its contact point. In this way, you can avoid specifying the same contact point multiple times for each child policy.

Alerting on forecasts
A forecast predicts the future state of a signal, such as expected disk space utilization over the next month. Common forecast use cases include:
Capacity planning: Detect when resources such as disk space, memory, CPU, or database connections are expected to approach their limits.
Demand and business planning: Analyze expected demand, costs, production, or other business metrics.
Standard alert rules evaluate the current state of a signal. A forecast-based alert compares a predicted future value with a threshold condition, allowing you to act before this happens.
{{< figure src="/media/docs/alerting/forecast-vs-standard-alert2.svg" alt="Comparison of standard and forecast-based alerts." >}}
Different forecasting tools provide different forecasting approaches, which are often chosen based on the behavior of the signal.
Linear forecasts are used for signals that follow a relatively steady trend.
Seasonal forecasts are better suited to signals with recurring patterns, such as daily or weekly traffic.
The following examples show how to create alerts using both types of forecasts.
Linear forecast with 
predict_linear
When a signal follows a relatively steady trend, you can use the PromQL 
predict_linear function to predict its future value.
predict_linear performs a linear regression over the values in a range vector and extrapolates the resulting trend into the future. The function takes two arguments:
predict_linear(<range-vector>, <seconds>)
<range-vector> provides the historical data used to calculate the trend.
<seconds> specifies how far into the future to predict, in seconds.
For example:
predict_linear(<metric>[14d], 7 * 24 * 3600)
This uses the last 
14 days of data to calculate a linear trend and predicts the value 
7 days from the current evaluation time.
How 
predict_linear works
The calculation is based on a simple linear regression. It calculates the slope from the historical data and extrapolates that slope into the future:
predicted_value = value_at_evaluation_time + slope * time
The result is a single predicted value at the requested future time, rather than a forecast series covering the entire future period.
When to use 
predict_linear
predict_linear is useful when a signal follows a relatively steady trend, such as disk utilization that is consistently increasing.
{{< figure src="/media/docs/alerting/alerting_on_forecast_trend_signal2.png" alt="A stable trend signal suitable for predict_linear forecasts." >}}
It is not suitable for signals with strong recurring patterns. For example, if a signal follows daily or weekly cycles, a straight-line projection might produce values that do not reflect the signal's normal behavior.
Another consideration is that 
predict_linear assumes the current trend continues. A sudden signal change can make the extrapolated trend inaccurate and lead to an inaccurate prediction.
Alert on a future predicted value
You can compare the predicted value with a threshold directly in an alert rule:
predict_linear(<metric>[14d], 7 * 24 * 3600) > 85
This evaluates whether the predicted value seven days from now exceeds 
85.
The following example uses 
predict_linear to forecast disk utilization for a PostgreSQL persistent volume claim:
predict_linear(
  max by (persistentvolumeclaim) (
    100 * (
      1 - kubelet_volume_stats_available_bytes{persistentvolumeclaim="<id>"}
      /
      kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="<id>"}
    )
  )[14d:1h],
  7 * 24 * 3600
)
The query works as follows:
It calculates the disk utilization percentage for the volume.
predict_linear uses the previous 
14 days of disk utilization to predict the value 
7 days into the future.
The subquery 
[14d:1h] evaluates the historical data at 
1-hour intervals. This resolution is sufficient for a forecast over a long period.
The alert rule uses a threshold condition that fires when the predicted utilization exceeds 
85%.
The alert rule evaluates every 
12 hours because disk utilization changes relatively slowly.
{{< admonition type="tip" >}}
You can explore this alert example in Grafana Play.
Alert description: The PVC is forecast to reach more than 85% used capacity within the next 7 days based on the last 14 days of usage. Review disk growth and plan capacity expansion before the volume reaches the threshold.
{{< /admonition >}}
Seasonal forecast with Grafana Cloud Machine Learning
When a signal follows recurring patterns, such as daily or weekly traffic cycles, a linear forecast cannot accurately represent its future behavior.
{{< figure src="/media/docs/alerting/seasonal_signal_and_linear_forecast2.png" caption="A signal with seasonal patterns (green line) and its linear forecast (yellow line)." >}}
In these cases, you can use Grafana Machine Learning forecasting to generate a forecast that accounts for trends and seasonality.
Unlike 
predict_linear, which returns a single predicted value at a specific point in the future, a Grafana Machine Learning forecast produces a series of predicted values over the forecast period. It also provides upper and lower bounds that represent the uncertainty of the prediction.
Each forecast produces these Prometheus metrics:
<forecast_metric_name>:predicted{ml_forecast="yhat"}: The predicted value, shown as the blue line.
<forecast_metric_name>:predicted{ml_forecast="yhat_upper"}: The upper bound of the prediction interval, shown as the upper edge of the blue band.
<forecast_metric_name>:predicted{ml_forecast="yhat_lower"}: The lower bound of the prediction interval, shown as the lower edge of the blue band.
<forecast_metric_name>:actual: The current value of the signal, shown as the green line.
{{< figure src="/media/docs/grafana-cloud/machine-learning/screenshot-traffic-latency-example.png" caption="A forecast example in Grafana Machine Learning." >}}
Because the forecast is stored in the 
grafanacloud-ml-metrics Prometheus data source, you can use the forecast metrics directly in alert rules to detect:
When the forecast is expected to breach a threshold at any point in the future.
When the current signal deviates from the expected forecast range.
Alert on a future prediction
Like the 
predict_linear example, a forecast alert can detect whether a signal is expected to exceed a threshold within a future time window.
In this example, the alert query uses the upper forecast bound (
yhat_upper) rather than the central prediction. This accounts for prediction uncertainty and provides a more conservative capacity-planning signal.
<forecast_metric_name>:predicted{ml_forecast="yhat_upper"} over the next 7 days > 80%
{{< figure src="/media/docs/grafana-cloud/machine-learning/screenshot-db-utilization-forecast-alert-example.png" alt="A screenshot of capacity planning forecast alert." >}}
In this example, the forecast provides up to 15 days of future prediction data. The maximum function value is used so that the alert fires if the forecast is expected to exceed the threshold at any point during the next 7 days.
{{< admonition type="tip" >}}
You can explore this alert example in Grafana Play.
{{< /admonition >}}
Alert when the actual state is outside the forecast
A forecast-based alert can also detect when the current signal deviates from the prediction.
In the previous examples, forecast alerts detect potential future situations: they alert when a signal is expected to exceed a threshold at some point in the future. The key differences in this case are:
The alert does not look for a future threshold breach. Instead, it compares the current value with the forecast range and alerts when the actual value falls outside the forecast range.
The forecast acts as an adaptive baseline. While a standard alert rule evaluates the signal against a fixed threshold, a forecast-based alert rule evaluates the signal against the forecast, which can change over time based on seasonal patterns.
{{< figure src="/media/docs/alerting/forecast-deviation-alert-v3.svg" alt="A forecast-based alert rule can evaluate the signal against the forecast." >}}
For alerting, each forecast produces an anomalous metric (
<forecast_metric_name>:anomalous) that returns 
0 when the actual value is within the prediction interval. You can use this metric directly in a forecast alert:
# Fires when the actual value is outside the prediction interval
<forecast_metric_name>:anomalous
Alert only on unexpectedly high values
You can also configure an alert to fire only when the actual value exceeds the predicted upper bound.
{{< figure src="/media/docs/alerting/forecast-deviation-alert-v2.svg" alt="A forecast-based alert rule configured to fire only for unexpectedly high values." >}}
In this case, the alert rule can also query 
<forecast_metric_name>:anomalous, which returns 
1 when the actual value is above the upper bound and 
-1 when it is below the lower bound:
<forecast_metric_name>:anomalous == 1
Alternatively, you can compare the actual value directly with the predicted upper bound:
<forecast_metric_name>:actual >
ignoring (ml_forecast) <forecast_metric_name>:predicted{ml_forecast="yhat_upper"}
{{< admonition type="tip" >}}
You can explore a similar alert example in Grafana Play.
{{< /admonition >}}
{{< admonition type="note" >}}
Adaptive alerts are useful when a signal varies over time and a fixed threshold would miss meaningful deviations or generate alerts during normal changes in behavior.
For example, the traffic might be higher during business hours and lower overnight. Comparing the current value with the forecast range adapts to these recurring patterns.
{{< /admonition >}}
Learn more
For more information about forecasting and alerting with Grafana Cloud Machine Learning, refer to:
Get started with forecasting
Alerting on forecasts
Forecasting examples

Alerting best practices
Designing and configuring an effective alerting system takes time. This guide focuses on building alerting systems that scale with real-world operations.
The practices described here are intentionally high-level and apply regardless of tooling. Whether you use Prometheus, Grafana Alerting, or another stack, the same constraints apply: complex systems, imperfect signals, and humans on call.
Alerting is never finished. It evolves with incidents, organizational changes, and the systems it’s meant to protect.
{{< shared id="alert-planning-fundamentals" >}}
Prioritize symptoms, but don’t ignore infrastructure signals
Alerts should primarily detect user-facing failures, not internal component behavior. Users don't care that a pod restarted; they care when the application is slow or failing. Symptom-based alerts tie directly to user impact.
Reliability metrics that impact users—latency, errors, availability—are better paging signals than infrastructure events or internal errors.
That said, infrastructure signals still matter. They can act as early warning indicators and are often useful when alerting maturity is low. A sustained spike in CPU or memory usage might not justify a page, but it can help explain or anticipate symptom-based failures.
Infrastructure alerts tend to be noisy and are often ignored when treated like paging signals. They are usually better suited for lower-severity channels such as dashboards, alert lists, or non-paging destinations like a dedicated Slack channel, where they can be monitored without interrupting on-call.
The key is balance as your alerting matures. Use infrastructure alerts to support diagnosis and prevention, not as a replacement for symptom-based alerts.
Escalate priority based on confidence
Alert priority is often tied to user impact and the urgency to respond, but confidence should determine when escalation is necessary.
In this context, escalation defines how responders are notified as confidence grows. This can include increasing alert priority, widening notification, paging additional responders, or opening an incident once intervention is clearly required.
Early signals are often ambiguous, and confidence in a non-transient failure is usually low. Paging too early creates noise; paging too late means users are impacted for longer before anyone acts. A small or sudden increase in latency may not justify immediate action, but it can indicate a failure in progress.
Confidence increases as signals become stronger or begin to correlate.
Escalation is justified when issues are sustained or reinforced by multiple signals. For example, high latency combined with a rising error rate, or the same event firing over a sustained period. These patterns reduce the chance of transient noise and increase the likelihood of real impact.
Use confidence in user impact to drive escalation and avoid unnecessary pages.
Scope alerts for scalability and actionability
In distributed systems, avoid creating separate alert rules for every host, service, or endpoint. Instead, define alert rules that scale automatically using multi-dimensional alert rules. This reduces rule duplication and allows alerting to scale as the system grows.
Start simple. Default to a single dimension such as 
service or 
endpoint to keep alerts manageable. Add dimensions only when they improve actionability. For example, when missing a dimension like 
region hides failures or doesn't provide enough information to act quickly.
Additional dimensions like 
region or 
instance can help identify the root cause, but more isn't always better.
Design alerts for first responders and clear actions
Alerts should be designed for the first responder, not the person who created the alert. Anyone on call should be able to understand what's wrong and what to do next without deep knowledge of the system or alert configuration.
Avoid vague alerts that force responders to spend time figuring out context. Every alert should clearly explain why it exists, what triggered it, and how to investigate. Use annotations to link to relevant dashboards and runbooks, which are essential for faster resolution.
Alerts should indicate a real problem and be actionable, even if the impact is low. Informational alerts add noise without improving reliability.
If no action is possible, it shouldn't be an alert—consider using a dashboard instead. Over time, alerts behave like technical debt: easy to create, costly to maintain, and hard to remove.
Review alerts often and remove those that don’t lead to action.
Alerts should have an owner and system scope
Alerts without ownership are often ignored. Every alert must have an owner: a team responsible for maintaining the alert and responding when it fires.
Alerts must also define a system scope, such as a service or infrastructure component. Scope provides organizational context and connects alerts with ownership. Defining clear scopes is easier when services are treated as first-class entities, and organizations are built around service ownership.
Service Center in Grafana Cloud can help operate a service-oriented view of your system and align alert scope with ownership.
After scope, ownership, and alert priority are defined, routing determines where alerts go and how they escalate. Notification routing is as important as the alerts.
Alerts should be delivered to the right team and channel based on priority, ownership, and team workflows. Use notification policies to define a routing tree that matches the context of your service or scope:
Define a parent policy for default routing within the scope.
Define nested policies for specific cases or higher-priority issues.
Prevent notification overload with alert grouping
Without alert grouping, responders can receive many notifications for the same underlying problem.
For example, a database failure can trigger several alerts at the same time like increased latency, higher error rates, and internal errors. Paging separately for each symptom quickly turns into notification spam, even though there is a single root cause.
Notification grouping consolidates related alerts into a single notification. Instead of receiving multiple pages for the same issue, responders get one alert that represents the incident and includes all related firing alerts.
Grouping should follow operational boundaries such as service or owner, as defined by notification policies. Downstream or cascading failures should be grouped together so they surface as one issue rather than many.
There is also a limit to how many notifications can be sent out per second. You can configure your Contact Point settings to reduce the email expenditure by setting up single email notifications rather than individual email notifications. To configure this, go to your desired email Contact Point and enable Single email in the optional settings.
Mitigate flapping alerts
Short-lived failure spikes often trigger alerts that auto-resolve quickly. Alerting on transient failures creates noise and leads responders to ignore them.
Require issues to persist before alerting. Set a pending period to define how long a condition must remain true before firing. For example, instead of alerting immediately on high error rate, require it to stay above the threshold for some minutes.
Also, stabilize alerts by tuning query ranges and aggregations. Using raw data makes alerts sensitive to noise. Instead, evaluate over a time window and aggregate the data to smooth short spikes.
# Reacts to transient spikes. Avoid this.
cpu_usage > 90
# Smooth fluctuations.
avg_over_time(cpu_usage[5m]) > 90
For latency and error-based alerts, percentiles are often more useful than averages:
quantile_over_time(0.95, http_duration_seconds[5m]) > 3
Finally, avoid rapid resolve-and-fire notifications by using 
keep_firing_for or recovery thresholds to keep alerts active briefly during recovery. Both options reduce flapping and unnecessary notifications.
Graduate symptom-based alerts into SLOs
When a symptom-based alert fires frequently, it usually indicates a reliability concern that should be measured and managed more deliberately. This is often a sign that the alert could evolve into a service level objective (SLO).
Traditional alerts create pressure to react immediately, while error budgets introduce a buffer of time to act, changing how urgency is handled. Alerts can then be defined in terms of error budget burn rate rather than reacting to every minor deviation.
SLOs also align distinct teams around common reliability goals by providing a shared definition of what "good" looks like. They help consolidate multiple symptom alerts into a single user-facing objective.
For example, instead of several teams alerting on high latency, a single SLO can be used across teams to capture overall API performance.
For other use cases, refer to when alerts need an SLO.
Integrate alerting into incident post-mortems
Every incident is an opportunity to improve alerting. After each incident, evaluate whether alerts helped responders act quickly or added unnecessary noise.
Assess which alerts fired, and how they influenced incident response. Review whether alerts triggered too late, too early, or without enough context, and adjust thresholds, priority, or escalation based on what actually happened.
Use silences during active incidents to reduce repeated notifications, but scope them carefully to avoid silencing unrelated alerts.
Post-mortems should evaluate alerts with root causes and lessons learned. If responders lacked key information during the incident, enrich alerts with additional context, dashboards, or better guidance.
Alerts should be continuously improved
Alerting is an iterative process. Alerts that aren’t reviewed and refined lose effectiveness as systems and traffic patterns change.
Schedule regular reviews of existing alerts. Remove alerts that don’t lead to action, and tune alerts or thresholds that fire too often without providing useful signal. Reduce false positives to combat alert fatigue.
Prioritize clarity and simplicity in alert design. Simpler alerts are easier to understand, maintain, and trust under pressure. Favor fewer high-quality, actionable alerts over a large number of low-value ones.
Use dashboards and observability tools for investigation, not alerts.
{{< /shared >}}

Handle connectivity errors in alerts
Connectivity issues are a common cause of misleading alerts or unnoticed failures.
There could be a number of reasons for these errors. Maybe your target went offline, or Prometheus couldn't scrape it. Or maybe your alert query failed because its target timed out or the network went down. These situations might look similar, but require different considerations in your alerting setup.
This guide walks through how to detect and handle these types of failures, whether you're writing alert rules in Prometheus, using Grafana Alerting, or combining both. It covers both availability monitoring and alert query failures, and outlines strategies to improve the reliability of your alerts.
Understand connectivity issues in alerts
Typically, connectivity issues fall into a few common scenarios:
Servers or containers crashed or were shut down.
Service overload or timeout.
Misconfigured authentication or incorrect permissions.
Network issues like DNS problems or ISP outages.
When we talk about connectivity errors in alerting, we’re usually referring to one of two use cases:
Your target is down or unreachable.
The service crashed, the host was down, or a firewall or DNS issue blocked the connection. These are availability problems.
Your alert query failed.
The alert couldn’t evaluate its query—maybe because the data source timed out or an invalid query. These are execution errors.
It helps to separate these cases early, because they behave differently and require different strategies.
Keep in mind that most alert rules don’t hit the target directly. They query metrics from a monitoring system like Prometheus, which scrapes data from your actual infrastructure or application. That gives us two typical alerting setups where connectivity issues can show up:
Alert rule → Target
For example, an alert rule querying an external data source like a database.
Alert rule → Prometheus ← Target
More common in observability stacks. For instance, Prometheus scrapes a node or container, and the alert rule queries the metrics later.
In this second setup, you can run into connectivity issues on either side. If Prometheus fails to scrape the target, your alert rule might not fire, even though something is likely wrong.
Detect target availability with the Prometheus 
up metric
Prometheus scrapes metrics from its targets regularly, following the 
scrape_interval period. The default scrape interval is 60 seconds, which is generally considered common practice.
Prometheus provides a built-in metric called 
up for every scrape target, a simple method to indicate whether scraping is successful:
up == 1: Your target is reachable; Prometheus collected the target metrics as expected.
up == 0: Prometheus couldn't reach your target—indicating possible downtime or network errors.
A typical PromQL expression for an alert rule to detect when a target becomes unreachable is:
up == 0
But this alert rule might result in noisy alerts as one single scrape failure will fire the alert. To reduce noise, you should add a delay:
up == 0 for: 5m
The 
for option in Prometheus (or pending period in Grafana) delays the alert until the condition has been true for the full duration.
In this example, waiting for 5 minutes means the single scrape error won't result in a fired alert. Since Prometheus scrapes metrics every minute by default, the alert only fires after five consecutive failures.
However, this kind of 
up alert has a few potential downfalls:
Failures can slip between scrape intervals: An outage that starts and ends between two evaluations go undetected. You could shorten the 
for duration, but this might lead to scrape failures that trigger false alarms.
Intermittent recoveries reset the 
for timer: A single successful scrape resets the alert timer, which masks intermittent outages.
Brief connectivity drops are common in real-world environments, so expect some flakiness in 
up alerts. For example:
Scrape result (
up)
Alert rule evaluation
00:00 
up == 0
Timer starts
01:00 
up == 0
Timer continues
02:00 
up == 0
Timer continues
03:00 
up == 1
Successful scrape resets timer
04:00 
up == 0
Timer starts again
05:00 
up == 0
No alert yet; timer hasn’t reached the 
for duration
The longer the period, the more likely this is to happen.
A single recovery resets the alert, that’s why 
up == 0 for: 5m can sometimes be unreliable. Even if the target is down most of the time, the alert didn't fire, leaving you unaware of a potential persistent issue.
Use 
avg_over_time to smooth signal
One way to work around these issues is to smooth the signal by averaging the 
up metric over a similar or longer period:
avg_over_time(up[10m]) < 0.8
This alert rule fires when the target is unreachable for more than 20% of the last 10 minutes, rather than looking for consecutive scrape failures. With a one minute scrape interval, three or more failed scrapes within the last 10 minutes now triggers the alert.
Since this query uses a threshold and time window to control accuracy, you can now lower the 
for duration (or pending period in Grafana) to something shorter—
0m or 
1m—so the alert fires faster.
This approach gives you more flexibility in detecting real crashes or network issues. As always, adjust the threshold and period based on your noise tolerance and how critical the target is.
Use synthetic checks to monitor external availability
Prometheus often runs inside the same network as the target it monitors. That means Prometheus might be able to reach the target, but doesn’t ensure it’s reachable to users on the outside.
Firewalls, DNS misconfigurations, or other network issues might block public traffic while Prometheus scrapes 
up successfully.
This is where synthetic monitoring helps. Tools like the Blackbox Exporter let you continuously verify whether a service is available and reachable from outside your network—not just internally.
The Blackbox Exporter exposes the results of these checks as metrics, which Prometheus can scrape like any other target. For example, the 
probe_success metric reports whether the probe was able to reach the service. The setup looks like this:
Alert rules → Prometheus ← Blackbox Exporter (external probe) → Target
To detect when a service isn’t reachable externally, you can define an alert using the 
probe_success metric:
probe_success == 0 for: 5m
This alert fires when the probe has failed continuously for 5 minutes—indicating that the service couldn’t be reached from the outside.
You can then combine internal and external checks to make the detection of connectivity errors more reliable. This alert catches when the internal scrape fails or the service is externally unreachable.
up == 0 or probe_success == 0
As with the 
up metric, you might want to smooth this out using 
avg_over_time() for more robust detection. The smooth version might look like:
avg_over_time(up[10m]) < 0.8 or avg_over_time(probe_success[10m]) < 0.8
This alert fires when Prometheus couldn't scrape the target successfully for more than 20% of the past 10 minutes, or when the external probes have been failing more than 20% of the time. This smoothing technique can be applied to any binary availability signal.
Manage offline hosts
In many setups, Prometheus scrapes multiple hosts under the same target, such as a fleet of servers or containers behind a common job label. It’s common for one host to go offline while the others continue to report metrics normally.
If your alert only checks the general 
up metric without breaking it down by labels (like 
instance, 
host, or 
pod), you might miss when a host stops reporting. For example, an alert that looks only at the aggregated status of all instances will likely fail to catch when individual instances go missing.
This isn't a connectivity error in this context — it’s not that the alert or Prometheus can't reach anything, it’s that one or more specific targets have gone silent. These kinds of problems aren’t caught by 
up == 0 alerts.
For these cases, see the complementary guide on handling missing data — it covers common scenarios where the alert queries return no data at all, or where only some targets stop reporting. These aren't full availability failures or execution errors, but they can still lead to blind spots in alert detection.
Handle query errors in Grafana Alerting
Not all connectivity issues come from targets going offline. Sometimes, the alert rule fails when querying its target. These aren’t availability problems—they’re query execution errors: maybe the data source timed out, the network dropped, or the query was invalid.
These errors lead to broken alerts. But they come from a different part of the stack: between the alert rule and the data source, not between the data source (for example, Prometheus) and its target.
This difference matters. Availability issues are typically handled using metrics like 
up or 
probe_success but execution errors require a different setup.
Grafana Alerting has built-in handling for execution errors, regardless of the data source. That includes Prometheus, and others like Graphite, InfluxDB, PostgreSQL, etc. By default, Grafana Alerting automatically handles query errors so you don’t miss critical failures. When an alert rule fails to execute, Grafana fires a special 
DatasourceError alert.
You can configure this behavior depending on how critical the alert is and on whether you already have other alerts detecting the issue. In Configure no data and error handling, click Alert state if execution error or timeout, and choose the desired option for the alert:
Error (default): Triggers a separate 
DatasourceError alert. This default ensures alert rules always inform about query errors but can create noise.
Alerting: Treats the error as if the alert condition is firing. Grafana transitions all existing instances for that rule to the 
Alerting state.
Normal: Ignores the query error and transitions all alert instances to the 
Normal state. This is useful if the error isn’t critical or if you already have other alerts detecting connectivity issues.
Keep Last State: Keeps the previous state until the query succeeds again. Suitable for unstable environments to avoid flapping alerts.
{{< figure src="/media/docs/alerting/alert-rule-configure-no-data-and-error-v2.png" alt="A screenshot of the 
Configure error handling option in Grafana Alerting." max-width="500px" >}}
This applies even when alert rules query Prometheus itself—not just external data sources.
Design alerts for connectivity errors
In practice, start by deciding if you want to create explicit alert rules — for example, using 
up or 
probe_success — to detect when a target is down or has connectivity issues.
Then, for each alert rule, choose the error-handling behavior based on whether you already have dedicated connectivity alerts, the stability of the target, and how critical the alert is. Prioritize alerts based on symptom severity rather than just infrastructure signals that might not impact users.
Build a dedicated data source health alert
Any alert rule query can fail to reach its data source, including one built specifically to check reachability. Prometheus-style data sources expose metrics such as 
up and 
probe_success for target-level availability, but those are still queries, and they fail the same way if the data source itself becomes unreachable. In any case, detecting that the data source itself is unreachable uses a different mechanism: the execution-error handling built into Grafana Alerting.
Create a separate alert rule dedicated to detecting when the data source itself is unreachable, distinct from any application-specific rules that happen to query it:
Reuse a query you already know returns data reliably when the data source is healthy. Any small, cheap query works, since the result doesn't matter—only whether the query executes.
In Configure no data and error handling, set Alert state if execution error or timeout to Alerting.
Add a pending period, for example 
5m, so the rule fires only once the data source has stayed unreachable for a sustained period, not for a single blip.
Set the same option to Normal or Keep Last State on your application-specific rules instead, so a transient error from the data source doesn't also fire a 
DatasourceError alert alongside the condition you're actually trying to detect. Which value to use isn't a one-time default—decide it per rule, based on whether that rule already has a dedicated connectivity alert covering it.
Reduce redundant error notifications
A single data source error can lead to multiple alerts firing simultaneously, sometimes bombarding you with many alerts and generating too much noise.
As described previously, you can control the error-handling behavior for Grafana alerts. The Keep Last State or Normal option prevents alerts from firing and helps avoid redundant alerts, especially for services already covered by 
up or 
probe_success alerts.
When using the default behavior, a single connectivity error will likely trigger multiple 
DatasourceError alerts.
Consider not treating these alerts in the same way as the original alerts, and implement dedicated strategies for their notifications:
Reduce duplicate notifications by grouping 
DatasourceError alerts. Use the 
datasource_uid label to group errors from the same data source.
Route 
DatasourceError alerts separately, sending them to different teams or channels depending on their impact and urgency.
For details on how to configure grouping and routing, refer to handling notifications and 
No Data and 
Error alerts documentation.
Conclusion
Connectivity issues are one of the common causes of noisy or misleading alerts. This guide covered two distinct types:
Availability issues, where the target itself is down or unreachable (e.g., due to a crash or network failure).
Query execution errors, where the alert rule can't reach its data source (e.g., due to timeouts, invalid queries, or data source outages).
These problems come from different parts of your stack, and require their own techniques. Prometheus and Grafana allow you to detect them, and combining distinct techniques can make your alerts more resilient.
With Prometheus, avoid relying solely on 
up == 0. Smooth queries to account for intermittent failures, and use synthetic monitoring to detect reachability issues from outside your network.
In Grafana Alerting, configure error handling explicitly. Not all alerts are equal or have the same urgency. Tune the error-handling behavior based on the reliability and severity of the alerts and whether you already have alerts dedicated to connectivity problems.
And don’t forget the third case: missing data. If only one host from a fleet silently disappears, you might not get alerted. If you're dealing with individual instances that stopped reporting data, see the Guide on handling missing data to continue exploring this topic.

Handle missing data in Grafana Alerting
Missing data from when a target stops reporting metric data can be one of the most common issues when troubleshooting alerts. In cloud-native environments, this happens all the time. Pods or nodes scale down to match demand, or an entire job quietly disappears.
When this happens, alerts won’t fire, and you might not notice the system has stopped reporting.
Sometimes it's just a lack of data from a few instances. Other times, it's a connectivity issue where the entire target is unreachable.
This guide covers different scenarios where the underlying data is missing and shows how to design your alerts to act on those cases. If you're troubleshooting an unreachable host or a network failure, see the Handle connectivity errors documentation as well.
No Data vs. Missing Series
There are a few common causes when an instance stops reporting data, similar to connectivity errors:
Host crash: The system is down, and Prometheus stops scraping the target.
Temporary network failures: Intermittent scrape failures cause data gaps.
Deployment changes: Decommissioning, Kubernetes pod eviction, or scaling down resources.
Ephemeral workloads: Metrics intentionally stop reporting.
And more.
The first thing to understand is the difference between a query failure (or connectivity error), No Data, and a Missing Series.
Alert queries often return multiple time series — one per instance, pod, region, or label combination. This is known as a multi-dimensional alert, meaning a single alert rule can trigger multiple alert instances (alerts).
For example, imagine a recorded metric, 
http_request_latency_seconds, that reports latency per second in the regions where the application is deployed. The query returns one series per region — for instance, 
region1 and 
region2 — and generates only two alert instances. In this scenario, you may experience:
Connectivity Error if the alert rule query fails.
No Data if the query runs successfully but returns no data at all.
Missing Series if one or more specific series, which previously returned data, are missing, but other series still return data.
In both No Data and Missing Series cases, the query still technically "works", but the alert won’t fire unless you explicitly configure it to handle these situations.
The following tables illustrate both scenarios using the previous example, with an alert that triggers if the latency exceeds 2 seconds in any region: 
avg_over_time(http_request_latency_seconds[5m]) > 2.
No Data Scenario: The query returns no data for any series:
Time
region1
region2
Alert triggered
00:00
1.5s 🟢
1s 🟢
✅ No Alert
01:00
No Data ⚠️
No Data ⚠️
⚠️ No Alert (Silent Failure)
02:00
1.4s 🟢
1s 🟢
✅ No Alert
MissingSeries Scenario: Only a specific series (
region2) disappears:
Time
region1
region2
Alert triggered
00:00
1.5s 🟢
1s 🟢
✅ No Alert
01:00
1.6s 🟢
Missing Series ⚠️
⚠️ No Alert (Silent Failure)
02:00
1.4s 🟢
1s 🟢
✅ No Alert
In both cases, something broke silently.
Detect missing data in Prometheus
Prometheus doesn't fire alerts when the query returns no data. It simply assumes there was nothing to report, like with query errors. Missing data won’t trigger existing alerts unless you explicitly check for it.
In Prometheus, a common way to catch missing data is by to use the 
absent_over_time function.
absent_over_time(http_request_latency_seconds[5m]) == 1
This triggers when all series for 
http_request_latency_seconds are absent for 5 minutes — catching the No Data case when the entire metric disappears.
However, 
absent_over_time() can’t detect which specific series are missing since it doesn’t preserve labels. The alert won’t tell you which series stopped reporting, only that the query returns no data.
If you want to check for missing data per-region or label, you can specify the label in the alert query as follows:
# Detect missing data in region1
absent_over_time(http_request_latency_seconds{region="region1"}[5m]) == 1
# Detect missing data in region2
absent_over_time(http_request_latency_seconds{region="region2"}[5m]) == 1
But this doesn't scale well. It is unreliable to have hard-coded queries for each label set, especially in dynamic cloud environments where instances can appear or disappear at any time.
To detect when a specific target has disappeared, see below Evict alert instances for missing series for details on how Grafana handles this case and how to set up detection.
Manage No Data issues in Grafana alerts
While Prometheus provides functions like 
absent_over_time() to detect missing data, not all data sources — like Graphite, InfluxDB, PostgreSQL, and others — available to Grafana alerts support a similar function.
To handle this, Grafana Alerting implements a built-in 
No Data state logic, so you don’t need to detect missing data with 
absent_* queries. Instead, you can configure in the alert rule settings how alerts behave when no data is returned.
Similar to error handling, Grafana triggers a special No data alert by default and lets you control this behavior. In Configure no data and error handling, click Alert state if no data or all values are null, and choose one of the following options:
No Data (default): Triggers a new 
DatasourceNoData alert, treating No data as a specific problem.
Alerting: Transition each existing alert instance into the 
Alerting state when data disappears.
Normal: Ignores missing data and transitions all instances to the 
Normal state. Useful when receiving intermittent data, such as from experimental services, sporadic actions, or periodic reports.
Keep Last State: Leaves the alert in its previous state until the data returns. This is common in environments where brief metric gaps happen regularly, like with flaky exporters or noisy environments.
{{< figure src="/media/docs/alerting/alert-rule-configure-no-data.png" alt="A screenshot of the 
Configure no data handling option in Grafana Alerting." max-width="500px" >}}
Manage DatasourceNoData notifications
When Grafana triggers a NoData alert, it creates a distinct alert instance, separate from the original alert instance. These alerts behave differently:
They use a dedicated 
alertname: DatasourceNoData.
They don’t inherit all the labels from the original alert instances.
Because of this, 
DatasourceNoData alerts might require a dedicated setup to handle their notifications. For general recommendations, see Reduce redundant DatasourceError alerts — similar practices can apply to NoData alerts.
Evict alert instances for missing series
MissingSeries occurs when only some series disappear but not all. This case is subtle, but important.
Grafana marks missing series as stale after two evaluation intervals and triggers the alert instance eviction process. Here’s what happens under the hood:
Alert instances with missing data keep their last state for two evaluation intervals.
If the data is still missing after that:
Grafana adds the annotation 
grafana_state_reason: MissingSeries.
The alert instance transitions to the 
Normal state.
A resolved notification is sent if the alert was previously firing.
The alert instance is removed from the Grafana UI.
If an alert instance becomes stale, you’ll find it in the alert history as 
Normal (Missing Series) before it disappears. This table shows the eviction process from the previous example:
Time
region1
region2
Alert triggered
00:00
1.5s 🟢
1s 🟢
🟢🟢 No Alerts
01:00
3s 🔴 
 
Alerting
3s 🔴 
 
Alerting
🔴🔴 Alert instances triggered for both regions
02:00
1.6s 🟢
(MissingSeries)⚠️ 
 
Alerting ️
🟢🔴 Region2 missing, state maintained.
03:00
1.4s 🟢
(MissingSeries) 
 
Normal
🟢🟢 
region2 was resolved, 📩 notification sent, and instance evicted.
04:00
1.4s 🟢
—
🟢 No Alerts. 
region2 was evicted.
Why doesn’t MissingSeries match No Data behavior?
In dynamic environments, such as autoscaling groups, ephemeral pods, spot instances, series naturally come and go. MissingSeries normally signals infrastructure or deployment changes.
By default, No Data triggers an alert to indicate a potential problem.
The eviction process for MissingSeries is designed to prevent alert flapping when a pod or instance disappears, reducing alert noise.
In environments with frequent scale events, prioritize symptom-based alerts over individual infrastructure signals and use aggregate alerts unless you explicitly need to track individual instances.
Handle MissingSeries notifications
A stale alert instance triggers a resolved notification if it transitions from a firing state (such as 
Alerting, 
No Data, or 
Error) to 
Normal, and the 
grafana_state_reason annotation is set to MissingSeries to indicate that the alert wasn’t resolved by recovery but evicted because the series data went missing.
Recognizing these notifications helps you handle them appropriately. For example:
Display the 
grafana_state_reason annotation to clearly identify MissingSeries alerts.
Or use the 
grafana_state_reason annotation to process these alerts differently.
Also, review these notifications to confirm whether something broke or if the alert was unnecessary. To reduce noise:
Silence or mute alerts during planned maintenance or rollouts.
Adjust alert rules to avoid triggering on series you expect to come and go, and use aggregated alerts instead.
Detect missing series in Prometheus
Previously, an example showed how to detect missing data for a specific label, such as 
region:
# Detect missing data in region1
absent_over_time(http_request_latency_seconds{region="region1"}[5m]) == 1
# Detect missing data in region2
absent_over_time(http_request_latency_seconds{region="region2"}[5m]) == 1
However, this approach doesn’t scale well because it requires hardcoding all possible 
region values.
As an alternative, you can create an alert rule that detects missing series dynamically using the 
present_over_time function:
present_over_time(http_request_latency_seconds{}[24h])
unless
present_over_time(http_request_latency_seconds{}[10m])
Or, if you want to group by a label such as region:
group(present_over_time(http_request_latency_seconds{}[24h])) by (region)
unless
group(present_over_time(http_request_latency_seconds{}[10m])) by (region)
This query finds regions (or other targets) that were present at any time in the past 24 hours but have not been present in the past 10 minutes. The alert rule then triggers an alert instance for each missing region. You can apply the same technique to any label or target dimension.
Conclusion
Missing data isn’t always a failure. It’s a common scenario in dynamic environments when certain targets stop reporting.
Grafana Alerting handles distinct scenarios automatically. Here’s how to think about it:
Understand 
DatasourceNoData and 
MissingSeries notifications, since they don’t behave like regular alerts.
Use Grafana’s No Data handling options to define what happens when a query returns nothing.
When NoData is not an issue, consider rewriting the query to always return data — for example, in Prometheus, use 
your_metric_query OR on() vector(0) to return 
0 when 
your_metric_query returns nothing.
Use 
absent_over_time() or 
present_over_time in Prometheus to detect when a metric or target disappears.
If data is frequently missing due to scrape delays, use techniques to account for data delays:
Adjust the Time Range query option in Grafana to evaluate slightly behind real time (e.g., set To to 
now-1m) to account for late data points.
In Prometheus, you can use 
last_over_time(metric_name[10m]) to pick the most recent sample within a given window.
Don’t alert on every instance by default. In dynamic environments, it’s better to aggregate and alert on symptoms — unless a missing individual instance directly impacts users.
If you’re getting too much noise from disappearing data, consider adjusting alerts, using 
Keep Last State, or routing those alerts differently.
For connectivity issues involving alert query failures, see the sibling guide: Handling connectivity errors in Grafana Alerting.

When SLOs reduce alert noise
As your alerting setup grows in production, not every alert works as expected. Sometimes they fire too often, or they resolve immediately before anyone responds. You start wondering if the problem is the alert.
When this happens, your instinct may be to tune the query or threshold, or maybe just delete it. Sometimes the alert is telling you that you're measuring reliability wrong, and implementing service level objectives (SLOs) could be the right answer.
This guide covers the three most common patterns where SLOs can help reduce alert noise and improve reliability coverage, and shows how to recognize the need for SLOs in your alerting setup.
How SLOs differ from threshold alerts
Threshold alerts evaluate a condition on its evaluation interval, for example, every 
5m, 
30m, 
1h.
If the query result crosses the threshold at that moment, the alert fires. It continues in a firing state on the next evaluation if the condition still holds, or returns to normal once resolved.
In this sense, the alert state is binary: firing or not firing.
{{< figure src="/media/docs/alerting/slog-guide-alert-state-lifecycle.png" max-width="750px" alt="States of alert instances over time" >}}
SLOs don't focus primarily on whether the service is healthy right now. They measure whether your service is reliable over a long period, for a week or several weeks, not at a point in time. For instance:
An alert asks: over the past 10 minutes, did 50% of requests respond in under 3s?
An SLO asks a similar question over a longer window: over the past 4 weeks, did 95% of requests respond in under 3 seconds?
Alerts can also use long evaluation periods, but they reset when the service recovers.
SLOs track failures against the error budget even when the service recovers. They record how much degradation has accumulated, and report how much failure is still allowed.
{{< figure src="/media/docs/alerting/slo-guide-remaining-error-budget-panel.png" max-width="750px" alt="SLO error budget" >}}
Another key difference is how SLO alerts work. Rather than firing when the error budget is consumed and the objective is breached, they fire when the objective is at risk.
SLO alerts page you when the system is degrading quickly, so you have time to act before breaching the SLO.
SLOs and threshold-based alerts are complementary. In some cases, tracking the same reliability signal in both alerts and SLOs makes sense: alerts for immediate response and SLOs for long-term visibility. In others, graduating alerts to SLOs is the right call. The following cases are the most common situations where teams start adopting SLOs from existing alerts.
The alert has no actionable response
Alerts should be actionable. An alert nobody acts on is operational noise. If the alert consistently triggers no response, but the metric still matters to your team, this is a reliability concern.
This reliability metric could indicate service health over time. It's worth tracking in operational dashboards, but it's not designed to trigger immediate action.
Ask the team: will we commit to an objective on the health of this service and act if performance degrades? If the answer is yes, an SLO is the right fit.
In practice, this question often surfaces whether the metric is actually measurable as a service level indicator (SLI).
{{< figure src="/media/docs/alerting/slo-guide-slo-performance.png" max-width="750px" alt="Service level indicator and error budget performance" >}}
Defining the SLO objective aligns the team around a shared target and determines when accumulated degradation is severe enough to require action. This is frequently the hardest part. Expect a few iterations before the objective reflects reality.
Sometimes deleting the alert is the right call if there's no commitment or SLI to anchor to a valid SLO. But if the same metric keeps surfacing in post-mortems across teams, that's a different signal that points to a shared reliability concern.
The alert reflects a shared reliability concern
Sometimes, multiple teams maintain alerts for the same user-facing metric such as service response times, each applying their own thresholds and internal metrics for their services, but lacking an objective to evaluate the end-to-end user experience.
This sometimes results in overlapping alerts covering the same signal.
When distinct teams define their own thresholds for service performance or user-experience metrics, this reveals a shared user-experience concern that hasn't been made explicit.
In other cases, the SLO is driven by the business, such as a customer commitment or service level agreement (SLA). An SLO formalizes it as a shared objective tracked against what customers expect.
Before creating a shared SLO, teams need to agree on clear ownership.
In both cases, the shift is often from internal metrics to user-facing reliability metrics, as recommended in the best practice to prioritize symptoms over causes.
Creating a shared SLO should not eliminate existing related alerts. This is a good moment to evaluate whether existing alerts trigger immediate action. If you notice these alerts are less actionable, and that the underlying reliability metric is better measured as an SLO, then migrate the existing alert to a team-scoped SLO.
A team-scoped SLO is the safer default. Move to a shared SLO only once teams agree on ownership.
The alert fires on transient conditions
The alert fires and resolves without human intervention, a pattern commonly referred to as flapping.
{{< figure src="/media/docs/alerting/slo-guide-flapping-alert-timeline.png" max-width="750px" caption="Alert instance states for a flapping alert" >}}
Alerting best practices recommend tuning away flapping alerts by tweaking alert rule settings to avoid detecting short spikes and transient issues. Tuning reduces noise, but can remove the signal entirely.
{{< figure src="/media/docs/alerting/slo-guide-non-flapping-alert-timeline.png" max-width="750px" caption="Alert instance states after mitigating flapping behavior" >}}
In this case, the flapping alert has surfaced a reliability issue. Individual firings look harmless, but the accumulated degradation over time impacts users. A threshold-based alert can mask these errors.
The SLO error budget captures this pattern. Each transient failure consumes a fraction of the budget, making cumulative impact visible over the SLO window.
{{< figure src="/media/docs/alerting/slo-guide-remaining-error-budget-panel.png" max-width="750px" alt="Error budget burndown" >}}
This pattern isn't limited to flapping alerts. Any alert with acceptable transient failures faces the same tradeoff: tune for less noise and lose visibility, or keep the alert and learn to ignore it.
Tracking reliability signals in dashboards and alerting on long evaluation periods can be the solution. SLOs are the better fit: the error budget doesn't reset and tracks every failure within the SLO window.
Conclusion
Not every alert needs to be tuned or silenced when it's not effective. Some are measuring something worth tracking differently.
Three common patterns signal an alert is ready for an SLO:
No actionable response: the signal indicates service reliability, but doesn't trigger immediate action.
Shared reliability concern: teams track similar user-facing metrics, but lack a shared objective.
Transient conditions: short-lived failures are often ignored, but still impact users.
In all three cases, the shift is from evaluating a threshold over a recent period to evaluating a reliability objective tracked over a long period. This reduces operational noise and focuses alerting on whether your service meets its reliability goals. SLOs don't replace your existing alerts; they bring a distinct method for measuring service reliability.
To get started, refer to introduction to SLOs in Grafana Cloud.

Alerts overview page
{{< docs/public-preview product="Alerts page" >}}
Alert view
Grafana Alerting provides a consolidated snapshot of your firing and pending Grafana-managed alerts in a simplified view. For users with complex deployments, it can be difficult to monitor and prioritize critical incidents among a large volume of firing or pending alerts. With the Alert page, you have a view where you can quickly explore and sort your recent alert history and see what alerts require review or action.
To see your firing and pending alerts in the alert page, go to Alerts & IRM > Alerting > Alert activity.
{{< figure src="/media/docs/alerting/alerts-page.png" max-width="750px" alt="Filter your firing and pending alerts in the Alert view." >}}
How it works
The Alerts page only shows alerts from Grafana-managed alert rules. Grafana uses a metric called 
GRAFANA_ALERTS, which is recorded in the default Mimir data source that is provisioned for cloud users.
{{< admonition type="note" >}}
OSS users need to manually configure this. To configure alert state history for OSS, refer to the configure Loki and Prometheus for alert state documentation.
{{< /admonition >}}
Required permissions
To see alerts in the Alerts page, you need both of the following permissions:
alert.rules:read: Read the alert rules in the folders that contain them.
datasources:query: Query the data source that records alert state history. In Grafana Cloud, this is the provisioned Prometheus data source that stores the 
GRAFANA_ALERTS metric.
The page queries the alert state history data source directly, so it can't display any alerts without the 
datasources:query permission for that data source. This applies even when you can read the alert rules and view them in the Alert rules page.
For more information about alerting permissions, refer to Configure role-based access control. To find out which data source records alert state history in your Grafana instance, refer to Configure alert state history.
Filter alerts in the Alerts page
The page displays the count of alert rules and instances that are both firing and pending. Use the filters to group your alerts and find specific alerts by label.
To group alerts by label, click the Group by field and select the labels from the dropdown menu. You can select multiple labels for alert grouping.
To filter by specific label values, click the Filters field and select a label to filter by, followed by an expression of what label value you want to use. You can enter multiple label values in your search.
You can also select a time range from the time picker to further adjust your results. Click the time range field and enter an absolute time range or select a period from the quick ranges list to apply a new time window to the display results.

Declare incidents from firing alerts
Declare an incident from a firing alert to streamline your alert to incident workflow.
Before you begin
Ensure you have Grafana Incident installed.
You must have a firing alert.
Procedure
To declare an incident from a firing alert, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Alert rules.
From the Alert rules page, click the Firing filter to display firing alerts. Find the firing alert that you want to declare an incident for.
Click More -> Declare Incident.
Alternatively, you can declare an incident from the Alert details page.
The Declare Incident pop-up opens in the Grafana Incident application.
In the Declare Incident pop-up, enter the What's going on? field.
This field is pre-filled with the name of the alert rule, but you can edit it as required.
The alert rule is also linked to the incident.
Select a severity.
Add labels, as required.
Click More options to include a channel prefix and status.
Click Automated options to enable automated actions configured by admins.
Click Declare Incident.
Next steps
View and manage the incident in the Grafana Incident application.
{{< figure src="/media/docs/alerting/incident-response-management-timeline-ui.png" max-width="750px" alt="Incident timeline view in Grafana Incident" >}}
For more information, refer to the Grafana Incident documentation.

View active notifications
The Active notifications page lists groups of alerts (or alert instances) that are actively triggering notifications.
By default, Grafana Alerting groups similar alerts into a single notification.
In this view, you can:
Find alert groups and the state of their notifications.
Filter for alert instances that match specific criteria.
The Active notifications view is useful for debugging and verifying how notifications are grouped based on your notification policy settings.
To review notifications that were already sent, refer to View notification history.
View alert groups and notification state
To view alert groups, complete the following steps.
Click Alerts & IRM -> Alerting -> Alert activity.
Select the Active notifications tab to view the list of groups firing notifications.
{{< figure src="/media/docs/alerting/active-notifications-view2.png" max-width="750px" alt="Active notifications view in Grafana Alerting" >}}
By default, alert groups are grouped by the notification policies grouping.
Each group displays its label set, contact point, and the number of alert instances (or alerts).
Then, click on a group to access its alert instances. You can find alert instances by their label set and view their notification state.
Notification states
The notification state of an alert instance can be in one of the following states:
Unprocessed: The alert is received but its notification has not been processed yet.
Suppressed: The alert has been silenced.
Active: The alert notification has been handled. The alert is still firing and continues to be managed.
Filter alerts
You can filter by label, state, or Alertmanager:
By label: In Search, enter an existing label to view alerts matching the label. For example, 
environment=production,region=~US|EU,severity!=warning.
By state: In States, select from Active, Suppressed, or Unprocessed states to view alerts matching your selected state. All other alerts are hidden.
By Alertmanager: In the Alertmanager dropdown, select an external Alertmanager to view only alert groups for that specific Alertmanager. By default, the 
Grafana Alertmanager is selected.
Custom group
From Custom group by dropdown, select a combination of labels to view a grouping other than the default. This helps validate the grouping settings of your notification policies.
If an alert does not contain labels specified either in the grouping of the default policy or the custom grouping, then the alert is added to a catch-all group with a header of 
No grouping.
View notification errors
{{< admonition type="note" >}}
Notification errors are only available with pre-configured Grafana Alertmanagers.
{{< /admonition >}}
Notification errors provide information about why they failed to be sent or were not received.
To view notification errors, navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab.
Each contact point displays a message about the status of their latest notification deliveries.
If a contact point is failing, a red message indicates that there are errors delivering notifications. Hover over the error message to see the notification error details.
To review past delivery attempts across contact points, refer to View notification history.

View alert rules
The Alert rules list view page lists all existing recording and alert rules. The rules are organized in a hierarchical structure, from folder -> rule group -> rules.
To access the Alert rules page, click Alerts & IRM -> Alerting -> Alert rules.
Select a group to expand it and view the list of alert rules within that group.
The list view includes a number of filters to simplify managing large volumes of alerts.
Filter and save searches
Click the Filter button to open the filter popup. You can filter by name, label, folder/namespace, evaluation group, data source, contact point, rule source, rule state, rule type, and the health of the alert rule from the popup menu. Click Apply at the bottom of the filter popup to enact the filters as you search.
Click the Saved searches button to open the list of previously saved searches, or click + Save current search to add your current search to the saved searches list. You can also rename a saved search or set it as a default search. When you set a saved search as the default search, the Alert rules page opens with the search applied.
{{< figure src="/media/docs/alerting/alerting-saved-searches.png" max-width="750px" alt="Alert rule filter options" >}}
Search syntax
The search input accepts a structured 
key:value syntax. The key on the left of the colon is the filter type, and the value on the right is what you want to match. Use quotation marks around values that contain spaces, for example 
rule:"High CPU usage".
The following filter types are supported:
datasource
namespace
label
group
rule
state
type
health
For example:
label:severity filters rules by the 
severity label.
health:error filters rules whose health is in an error state.
rule:"High CPU usage" filters rules by name.
You can combine multiple filters in a single search. Any text entered without a 
key: prefix is treated as a query that filters alert rules by name.
Match labels with regular expressions
The 
label filter accepts Prometheus-style matchers, so you can match label values with regular expressions instead of exact strings. The other filter types don't support regular expressions.
A label matcher uses the form 
label:<KEY><OPERATOR><VALUE>. The following operators are supported:
=: matches rules whose label value is exactly equal to the value.
!=: matches rules whose label value isn't equal to the value.
=~: matches rules whose label value matches the regular expression.
!~: matches rules whose label value doesn't match the regular expression.
For example:
label:severity=~crit.* matches rules where the 
severity label matches the regular expression, such as 
critical or 
crit-high.
label:team!~fe.* matches rules where the 
team label doesn't start with 
fe.
Wrap the matcher in quotation marks when the key or value contains spaces or special characters, for example 
label:"team=~fe.*devs".
Regular expressions use RE2 syntax and are fully anchored, so the pattern must match the entire label value. Use 
.* or 
.+ to match part of a value, for example 
label:severity=~.*crit.*. Matching is case-sensitive by default. To match without case sensitivity, add the 
(?i) inline flag, for example 
label:severity=~(?i)critical.
Label matchers apply to both the alert rule's labels and the labels on its alert instances, so a rule matches when either set of labels satisfies the matcher. For labels whose values are set by Go templates, the matcher runs against the rendered static value, not the template expression, so you can't use a template query to find a match.
The search input and the Filter popup are kept in sync, so changes made in either place are reflected in the other.
Change alert rules list view
You can also change how the rule list is displayed using the View as option.
Grouped: Displays rules grouped in a hierarchical structure, from folder/namespace, to evaluation group, to the individual rules. This is the default view.
List: Displays all rules in a flat, unpaginated list.
{{< figure src="/media/docs/alerting/view-alert-rule-list-with-actions-2.png" max-width="750px" alt="View alert rule state and alert rule health in Grafana Alerting" >}}
For details on how rule states and alert instance states are displayed, refer to View alert state.
View, compare and restore alert rules versions.
To view, compare, or restore previous versions for an alert rule, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Alert rules.
Select an alert rule and click View.
Click the Versions tab.
The page displays a list of the previous rule versions.
On the Alert rule's Versions page you can view, compare and restore the previous rule versions.
{{< figure src="/media/docs/alerting/screenshot-grafana-alerting-version-history-v3.png" max-width="750px" alt="View alert rule history to compare and restore previous alert rules." >}}
{{< admonition type="note" >}}
The alert rule does not guarantee sequential version increases.
In Grafana OSS and Enterprise, the number of alert rule versions can be limited using the 
rule_version_record_limit option. In Grafana Cloud, free users are allowed a maximum of 10 alert rule versions, while paid users have a maximum of 100 stored alert rule versions.
{{< /admonition >}}
Bulk pause or resume alert rules evaluations within a folder
Admin users can pause or resume all of the alert rules evaluations within a folder. To pause or resume all the alert rules evaluations in a folder, click the menu icon and select Pause all rule evaluation or Resume all rule evaluation.
Bulk delete all alert rules within a folder
Admin users can delete all of the alert rules within a folder. To delete all the alert rules in a folder, click the menu icon and select Delete. Then type "Delete" into the field and click Delete to confirm the bulk deletion.
Permanently delete or restore deleted alert rules
Only users with an Admin role can restore deleted Grafana-managed alert rules. After an alert rule is restored, it is restored with a new, different UID from the one it had before.
Go to Alerts & IRM -> Alerting -> Alert rules, then click the Recently deleted tab.
Click the Restore button to restore the alert rule or click Delete permanently to delete the alert rule.
{{< admonition type="note" >}}
Deleted alert rules are stored for 30 days. Grafana Enterprise and OSS users can adjust the length of time for which the rules are stored in the Grafana configuration file's 
[unified_alerting].deleted_rule_retention field. For an example of how to modify the Grafana configuration file, refer to the documentation example here.
{{< /admonition >}}

View alert state history
View a history of all alert events generated by your Grafana-managed alert rules from one centralized page. This helps you see patterns in your alerts over time, observe trends, make predictions, and even debug alerts that might be firing too often.
An alert event is displayed each time an alert instance changes its state over a period of time. All alert events are displayed regardless of whether silences or mute timings are set, so you’ll see a complete set of your data history even if you’re not necessarily being notified.
{{< admonition type="note" >}}
Grafana OSS and Grafana Enterprise users must configure alert state history in Loki to view the History page and State history view.
{{< /admonition >}}
View from the History page
The History page shows the history and state changes of all Grafana-managed alert rules. You can filter by labels and alert states.
The History page also includes a Notifications tab for notification history.
Users can only view the history of alert rules they have permission to access (RBAC).
To access the History page, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> History.
{{< figure src="/media/docs/alerting/alert-history-page.png" max-width="750px" alt="History page in Grafana Alerting" >}}
An Alert Events chart shows your alert activity within a specific time frame. Each alert event represents an alert instance that changed its state at a particular point in time. The history of the data is displayed over a period of time.
Filter by labels either by entering a label or clicking a label of an instance in the list of events.
Filter by current state and previous state by selecting a state from the drop-down or by clicking the states from the list of events.
Zoom in by dragging on the chart or use the time picker.
{{< admonition type="note" >}}
If you exceed the 5000 alerts limit, you may see data missing from the chart. To see complete results, narrow the time frame.
{{< /admonition >}}
Under the chart, there is a list of events. Each event represents a state change on an alert instance. Expand a row to see the number of transitions for the alert instance, a state graph, and the value in the transition.
Click the alert rule name to jump to the History tab in the Alert Rule view.
{{< figure src="/media/docs/alerting/alerting-alert-history-tab.png" max-width="750px" alt="Alert History tab in Grafana Alerting" >}}
Use Grafana Assistant to analyze alert state history
{{< admonition type="note" >}}
This feature is available in Grafana Cloud when Grafana Assistant is enabled.
{{< /admonition >}}
The Analyze with Assistant button provides AI-powered analysis of your alert history to help you understand and troubleshoot alert patterns. Located in the top-right corner of the History page event list, this button uses Grafana Assistant to analyze the events displayed in your current view.
When you click the AI Triage button, the Grafana Assistant analyzes:
Alert state transitions over the selected time range
Alert instance patterns and frequency
Common labels and characteristics of firing alerts
Temporal patterns in alert behavior
The AI assistant can help you:
Identify root causes of alert storms
Detect patterns in alert firing behavior
Understand correlations between different alert instances
Get suggestions for improving alert configurations
To use the Analyze with Assistant feature:
Navigate to the History page as described above.
Filter the events to focus on the alerts you want to analyze using labels, states, or time range.
Click the Analyze with Assistant button in the top-right corner of the event list.
Review the AI-generated analysis and recommendations.
The AI analysis is based on the currently displayed events, so filtering your view to specific alerts or time periods will result in more focused insights.
View from the State history view
Use the State history view to get insight into how your individual alert instances behave over time.
View information on when a state change occurred, what the previous state was, the current state, any other alert instances that changed their state at the same time as well as what the query value was that triggered the change.
To access the State history view, complete the following steps.
Navigate to Alerts & IRM -> Alerting -> Alert rules.
Click an alert rule.
Select Show state history.
{{< figure src="/media/docs/alerting/alerting-state-history-view.png" max-width="750px" alt="State history view in Grafana Alerting" >}}
The timeline view at the top displays a timeline of changes for the past hour, so you can track how your alert instances are behaving over time.
The bottom part shows the alert instances, their previous and current state, the value of each part of the expression and a unique set of labels.
Common labels are displayed at the top to make it easier to identify different alert instances.
From the timeline view, hover over a time to get an automatic display of all the changes that happened at that particular moment.
These changes are displayed in real time in the timestamp view at the bottom of the page. The timestamp view is a list of all the alert instances that changed state at that point in time. The visualization only displays 12 instances by default.
The value shown for each instance is for each part of the expression that was evaluated.
Click the labels to filter and narrow down the results.
Explore and query the alert history
You can also use Grafana Explore to query the data sources that store alert history.
For details on querying these data sources, refer to Alerting Meta monitoring.

View alert state
An alert rule and its corresponding alert instances can transition through distinct states during the alert rule evaluation.
{{< figure src="/media/docs/alerting/alert-rule-evaluation-full-statediagram_v3.svg" alt="A diagram of the distinct alert instance states and transitions." max-width="750px" >}}
There are three key components that helps us understand the behavior of our alerts:
Alert Instance State: Refers to the state of the individual alert instances.
Alert Rule State: Determined by the "worst state" among its alert instances.
Alert Rule Health: Indicates the status in cases of 
Error or 
NoData events.
View alert rule and instance states
To view the details of your alert rules and the status of alert instances:
Click Alerts & IRM -> Alerting.
Click Alert rules to view the list of existing alert rules.
{{< figure src="/media/docs/alerting/view-alert-rule-list-with-actions2.png" max-width="750px" alt="View alert rule state and alert rule health in Grafana Alerting" >}}
Each alert rule shows its state, summary, and available actions such as Pause evaluation, Silence notifications, Export, Delete, and more.
Click on an alert rule to view additional details and its resulting alert instances.
View from the alert rule details page
To view more alert rule details, complete the following steps.
Click Alerts & IRM -> Alerting -> Alert rules.
Click to expand an alert rule.
Click the alert name to go to the alert details view.
{{< figure src="/media/docs/alerting/alert-rule-view-page-with-breadcrumb.png" max-width="750px" alt="Alert rule view page in Grafana Alerting" >}}
The namespace and group are shown in the breadcrumb navigation. They are interactive and can be used to filter rules by namespace or group.
The rest of the alert detail content is split up into tabs:
Query and conditions
View the details of the query that is used for the alert rule, including the expressions and intermediate values for each step of the expression pipeline. A graph view is included for range queries and data sources that return time series-like data frames.
Instances
Explore each alert instance, its status, labels and various other metadata for multi-dimensional alert rules.
Use Search by label to enter search criteria using label selectors. For example, 
environment=production,region=~US|EU,severity!=warning.
History
Explore the recorded history for an alert rule. You can also filter by alert state.
Details
Debug or audit using the alert rule metadata and view the alert rule annotations.
View alert state on panels
When an alert rule is linked to a time series panel, the time series panel displays the alert state and alert events.
A heart icon near the panel title shows the current alert state:
A broken red heart when the alert is in 
Alerting state.
A green heart when the alert is in 
Normal state.
Colored annotations indicate changes in alert state, such as pending, alerting, and resolving.
{{< figure src="/media/docs/alerting/panel-displays-alert-state.png" max-width="1200px" alt="A panel with a firing alert and annotations that display the pending and alerting state changes." >}}
Additionally, Grafana provides an alert list panel that you can add to a dashboard to display a list of alerts and their states.
{{< figure src="/static/img/docs/alert-list-panel/alert-list-panel.png" max-width="850px" alt="Two alert list panels displaying distinct lists of alerts." >}}
You can configure the alert list panel with various visualization options and filters to control how alerts are displayed. For more details, refer to the Alert list documentation.
{{< docs/play title="this demo dashboard with alert list panels and linked alert rules" url="https://play.grafana.org/d/000000074/alerting?tech=docs&pg=alerting-demo&plcmt=callout-play&cta=alert-demo-dashboard" >}}

View notification history
View notifications that Grafana Alerting sent for your Grafana-managed alert rules from one page. Confirm delivery, see which contact point received each notification, and debug failures.
Grafana records a notification event each time it attempts to send a notification to a contact point for a group of alerts. Both firing and resolved notifications are recorded, so you'll see a complete delivery history even when an alert has already recovered.
Notification history is different from active notifications. Active notifications show alerts that are currently grouped and waiting to be sent. Notification history shows notifications that were already sent.
The History page also includes an Alert events tab for alert state history.
{{< admonition type="note" >}}
Grafana OSS and Grafana Enterprise users must store notification history in Loki. Set 
enabled = true in the 
[unified_alerting.notification_history] section and configure the connection to Loki. Refer to Before you begin.
{{< /admonition >}}
Before you begin
Notification history stores Alertmanager notification logs in Loki. In Grafana OSS and Grafana Enterprise, configure the connection to Loki in the 
[unified_alerting.notification_history] section of your Grafana configuration file:
[unified_alerting.notification_history]
# Enable the notification history functionality in Unified Alerting.
# Alertmanager notification logs will be stored in Loki.
enabled = true
# URL of the Loki instance.
loki_remote_url = http://localhost:3100
# Optional tenant ID to attach to requests sent to Loki.
loki_tenant_id =
# Optional basic authentication for requests sent to Loki.
loki_basic_auth_username =
loki_basic_auth_password =
Replace 
http://localhost:3100 with the URL of your Loki instance.
You can attach extra labels to every notification history record with the 
[unified_alerting.notification_history.external_labels] section.
In Grafana Cloud, notification history is managed for you.
View from the History page
The History page shows notifications sent for all Grafana-managed alert rules. Filter by labels, status, contact point, and delivery outcome.
You can only view notification history for alert rules you have permission to access.
To view notification history from the History page, follow these steps.
Click Alerts & IRM > Alerting > History.
Select the Notifications tab.
Each row represents a notification sent to a contact point. The list shows the following columns:
Time: When the notification was sent.
Status: Whether the notification was for a Firing or Resolved alert.
Group Labels: The labels that identify the group of alerts included in the notification.
Contact point: The contact point and integration that received the notification.
Failed notifications are marked with a Failed badge. Hover over the badge to see the error details.
Filter the list to narrow down the results:
Labels: Enter a label or click a label on a notification in the list.
Status: Filter by Firing or Resolved.
Delivery outcome: Show only successful or failed notifications.
Contact point: Filter by contact point.
Time range: Adjust the time range with the time picker.
Expand a row to see the alerts included in the notification, grouped into Firing Alerts and Resolved Alerts. Each alert shows its labels, annotations, summary, and description, and links to the alert rule that generated it.
View the details of a single notification
Select View on a notification row to open its detail page.
The detail page shows the full context of the notification, including the contact point and integration used, the delivery outcome, any error returned, and the complete list of firing and resolved alerts that were included.
View notification history for an alert rule
You can review notifications sent for a specific alert rule without leaving the rule.
To view notification history for an alert rule, follow these steps.
Click Alerts & IRM > Alerting > Alert rules.
Click an alert rule.
Select the Notifications tab.
The tab shows the same notification list as the History page, pre-filtered to alerts generated by that rule.
Explore and query notification history
Because notification history is stored in Loki, you can query it directly in Loki through Grafana Explore. Use Explore to build custom dashboards or run free-form investigations of your notification delivery.
Next steps
Refer to View alert state history to review alert state changes on the Alert events tab of the History page.
Refer to View active notifications to inspect alerts that are currently grouped and waiting to be sent.
Refer to Manage contact points to configure where notifications are delivered.

Meta monitoring
Monitor your alerting metrics to ensure you identify potential issues before they become critical.
Meta monitoring is the process of monitoring your monitoring system and alerting when your monitoring is not working as it should.
In order to enable you to meta monitor, Grafana provides predefined metrics.
Identify which metrics are critical to your monitoring system (i.e. Grafana) and then set up how you want to monitor them.
You can use meta-monitoring metrics to understand the health of your alerting system in the following ways:
Optional: Create a dashboard in Grafana that uses this metric in a panel (just like you would for any other kind of metric).
Optional: Create an alert rule in Grafana that checks this metric regularly (just like you would do for any other kind of alert rule).
Optional: Use the Explore module in Grafana.
Metrics for Grafana-managed alerts
To meta monitor Grafana-managed alerts, you can collect two types of metrics in a Prometheus instance:
State history metric (
GRAFANA_ALERTS) — Exported by Grafana Alerting as part of alert state history.
Scraped metrics — Exported by Grafana’s 
/metrics endpoint to monitor alerting activity and performance.
You need a Prometheus-compatible server to collect and store these metrics.
GRAFANA_ALERTS metric
If you have configured Prometheus for alert state history, Grafana writes alert state changes to the 
ALERTS metric:
GRAFANA_ALERTS{alertname="", alertstate="", grafana_alertstate="", grafana_rule_uid="", <additional alert labels>}
This 
GRAFANA_ALERTS metric is compatible with the 
ALERTS metric used by Prometheus Alerting and includes two additional labels:
A new 
grafana_rule_uid label for the UID of the Grafana rule.
A new 
grafana_alertstate label for the Grafana alert state, which differs slightly from the equivalent Prometheus state included in the 
alertstate label.
Alert labels are automatically converted before being written to Prometheus to ensure compatibility. Prometheus requires label names to start with a letter or underscore (
_), followed only by letters, numbers, or additional underscores. Invalid characters are replaced during conversion. For example, 
1my-label becomes 
_my_label.
Grafana state
alertstate
grafana_alertstate
Alerting
firing
alerting
Recovering
firing
recovering
Pending
pending
pending
Error
firing
error
NoData
firing
nodata
Normal
(no metric emitted)
(no metric emitted)
You can then query this metric like any other Prometheus metric:
{{< code >}}
GRAFANA_ALERTS{grafana_alertstate="alerting"}
GRAFANA_ALERTS{grafana_alertstate="recovering"}
GRAFANA_ALERTS{grafana_alertstate="pending", severity="critical"}
{{< /code >}}
Scraped metrics
To collect scraped Alerting metrics, configure Prometheus to scrape metrics from Grafana.
- job_name: grafana
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  static_configs:
    - targets:
        - grafana:3000
The Grafana ruler, which is responsible for evaluating alert rules, and the Grafana Alertmanager, which is responsible for sending notifications of firing and resolved alerts, provide a number of metrics that let you observe them.
grafana_alerting_alerts
This metric is a counter that shows you the number of 
normal, 
pending, 
alerting, 
nodata and 
error alerts. For example, you might want to create an alert that fires when 
grafana_alerting_alerts{state="error"} is greater than 0.
grafana_alerting_schedule_alert_rules
This metric is a gauge that shows you the number of alert rules scheduled. An alert rule is scheduled unless it is paused, and the value of this metric should match the total number of non-paused alert rules in Grafana.
grafana_alerting_schedule_periodic_duration_seconds_bucket
This metric is a histogram that shows you the time it takes to process an individual tick in the scheduler that evaluates alert rules. If the scheduler takes longer than 10 seconds to process a tick then pending evaluations start to accumulate such that alert rules might later than expected.
grafana_alerting_schedule_query_alert_rules_duration_seconds_bucket
This metric is a histogram that shows you how long it takes the scheduler to fetch the latest rules from the database. If this metric is elevated, 
schedule_periodic_duration_seconds is also evaluated.
grafana_alerting_scheduler_behind_seconds
This metric is a gauge that shows you the number of seconds that the scheduler is behind where it should be. This number increases if 
schedule_periodic_duration_seconds is longer than 10 seconds, and decrease when it is less than 10 seconds. The smallest possible value of this metric is 0.
grafana_alerting_notification_latency_seconds_bucket
This metric is a histogram that shows you the number of seconds taken to send notifications for firing and resolved alerts. This metric lets you observe slow or over-utilized integrations, such as an SMTP server that is being given emails faster than it can send them.
grafana_alerting_state_history_writes_failed_total
This metric is a counter that shows you the number of failed writes to the configured alert state history backend. It includes a 
backend label to distinguish between different backends (such as 
loki or 
prometheus).
For example, you might want to create an alert that fires when 
grafana_alerting_state_history_writes_failed_total{backend="prometheus"} is greater than 0 to detect when Prometheus remote write is failing.
Logs for Grafana-managed alerts
If you have configured Loki for alert state history, logs related to state changes in Grafana-managed alerts are stored in the Loki data source.
You can use Grafana Explore and the Loki query editor to search for alert state changes.
{{< code >}}
{from="state-history"} | json
{from="state-history"} | json | previous=~"Normal.*" | current=~"Alerting.*"
{from="state-history"} | json | current=~"Error.*"
{{< /code >}}
In the Logs view, you can review details for individual alerts by selecting fields such as:
previous: previous alert instance state.
current: current alert instance state.
ruleTitle: alert rule title.
ruleID and 
ruleUID.
labels_alertname, 
labels_new_label, and 
labels_grafana_folder.
Additional available fields.
Alternatively, you can access the History page in Grafana to visualize and filter state changes for individual alerts or all alerts.
If you have enabled notification history, you can also refer to View notification history to review sent notifications and delivery outcomes in the History page Notifications tab.
Metrics for Mimir-managed alerts
To meta monitor Grafana Mimir-managed alerts, open source and on-premise users need a Prometheus/Mimir server, or another metrics database to collect and store metrics exported by the Mimir ruler.
rule_evaluation_failures_total
This metric is a counter that shows you the total number of rule evaluation failures.
Metrics for Alertmanager
To meta monitor the Alertmanager, you need a Prometheus/Mimir server, or another metrics database to collect and store metrics exported by Alertmanager.
For example, if you are using Prometheus you should add a 
scrape_config to Prometheus to scrape metrics from your Alertmanager.
- job_name: alertmanager
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  static_configs:
    - targets:
        - alertmanager:9093
The following is a list of available metrics for Alertmanager.
alertmanager_alerts
This metric is a counter that shows you the number of active, suppressed, and unprocessed alerts in Alertmanager. Suppressed alerts are silenced alerts, and unprocessed alerts are alerts that have been sent to the Alertmanager but have not been processed.
alertmanager_alerts_invalid_total
This metric is a counter that shows you the number of invalid alerts that were sent to Alertmanager. This counter should not exceed 0, and so in most cases, create an alert that fires if whenever this metric increases.
alertmanager_notifications_total
This metric is a counter that shows you how many notifications have been sent by Alertmanager. The metric uses a label "integration" to show the number of notifications sent by integration, such as email.
alertmanager_notifications_failed_total
This metric is a counter that shows you how many notifications have failed in total. This metric also uses a label "integration" to show the number of failed notifications by integration, such as failed emails. In most cases, use the 
rate function to understand how often notifications are failing to be sent.
alertmanager_notification_latency_seconds_bucket
This metric is a histogram that shows you the amount of time it takes Alertmanager to send notifications and for those notifications to be accepted by the receiving service. This metric uses a label "integration" to show the amount of time by integration. For example, you can use this metric to show the 95th percentile latency of sending emails.
Metrics for Alertmanager in high availability mode
If you are using Alertmanager in high availability mode there are a number of additional metrics that you might want to create alerts for.
alertmanager_cluster_members
This metric is a gauge that shows you the current number of members in the cluster. The value of this gauge should be the same across all Alertmanagers. If different Alertmanagers are showing different numbers of members then this is indicative of an issue with your Alertmanager cluster. You should look at the metrics and logs from your Alertmanagers to better understand what might be going wrong.
alertmanager_cluster_failed_peers
This metric is a gauge that shows you the current number of failed peers.
alertmanager_cluster_health_score
This metric is a gauge showing the health score of the Alertmanager. Lower values are better, and zero means the Alertmanager is healthy.
alertmanager_cluster_peer_info
This metric is a gauge. It has a constant value 
1, and contains a label called "peer" containing the Peer ID of each known peer.
alertmanager_cluster_reconnections_failed_total
This metric is a counter that shows you the number of failed peer connection attempts. In most cases you should use the 
rate function to understand how often reconnections fail as this may be indicative of an issue or instability in your network.

Configure alert state history
Alerting can record all alert rule state changes for your Grafana managed alert rules in a Loki or Prometheus instance, or in both.
With Prometheus, you can query the 
GRAFANA_ALERTS metric for alert state changes in Grafana Explore.
With Loki, you can query and view alert state changes in Grafana Explore and the Grafana Alerting History views.
Configure Loki for alert state
The following steps describe a basic configuration:
Configure Loki
The default Loki settings might need some tweaking as the state history view might query up to 30 days of data.
The following change to the default configuration should work for most instances, but look at the full Loki configuration settings and adjust according to your needs.
limits_config:
  split_queries_by_interval: '24h'
  max_query_parallelism: 32
As this might impact the performances of an existing Loki instance, use a separate Loki instance for the alert state history.
Configure Grafana
The following Grafana configuration instructs Alerting to write alert state history to a Loki instance:
[unified_alerting.state_history]
enabled = true
backend = loki
# The URL of the Loki server
loki_remote_url = http://localhost:3100
Configure the Loki data source in Grafana
Add the Loki data source to Grafana.
If everything is set up correctly, you can access the History view and History page to view and filter alert state history. You can also use Grafana Explore to query the Loki instance, see Alerting Meta monitoring for details.
Configure Prometheus for alert state (GRAFANA_ALERTS metric)
You can also configure a Prometheus instance to store alert state changes for your Grafana-managed alert rules. However, this setup does not enable the Grafana Alerting History views, as Loki does.
Instead, Grafana Alerting writes alert state data to the 
GRAFANA_ALERTS metric-similar to how Prometheus Alerting writes to the 
ALERTS metric.
GRAFANA_ALERTS{alertname="", alertstate="", grafana_alertstate="", grafana_rule_uid="", <additional alert labels>}
The following steps describe a basic configuration:
Configure Prometheus
Enable the remote write receiver in your Prometheus instance by setting the 
--web.enable-remote-write-receiver command-line flag. This enables the endpoint to receive alert state data from Grafana Alerting.
Configure the Prometheus data source in Grafana
Add the Prometheus data source to Grafana.
In the Prometheus data source configuration options, set the Prometheus type to match your Prometheus instance type. Grafana Alerting uses this option to identify the remote write endpoint.
Configure Grafana
The following Grafana configuration instructs Alerting to write alert state history to a Prometheus instance:
[unified_alerting.state_history]
enabled = true
backend = prometheus
# Target data source UID for writing alert state changes.
prometheus_target_datasource_uid = <DATA_SOURCE_UID>
# (Optional) Metric name for the alert state metric. Default is "GRAFANA_ALERTS".
# prometheus_metric_name = GRAFANA_ALERTS
# (Optional)  Timeout for writing alert state data to the target data source. Default is 10s.
# prometheus_write_timeout = 10s
Grafana reads alert state history from the same data source, so users need the 
datasources:query permission for it to view the Alerts page.
You can then use Grafana Explore to query the alert state metric. For details, refer to Alerting Meta monitoring.
GRAFANA_ALERTS{alertstate='firing'}
Configure Loki and Prometheus for alert state
You can also configure both Loki and Prometheus to record alert state changes for your Grafana-managed alert rules.
Start with the same setup steps as shown in the previous Loki and Prometheus sections. Then, adjust your Grafana configuration as follows:
[unified_alerting.state_history]
enabled = true
backend = multiple
primary = loki
# URL of the Loki server.
loki_remote_url = http://localhost:3100
secondaries = prometheus
# Target data source UID for writing alert state changes.
prometheus_target_datasource_uid = <DATA_SOURCE_UID>

Manage access using folders or data sources
You can extend the access provided by a role to alert rules and rule-specific silences by assigning permissions to individual folders or data sources.
{{< admonition type="note" >}}
Data source query permissions are required to create an alert rule or modify an alert rule's query using that data source.
{{< /admonition >}}
This allows different users, teams, or service accounts to have customized access to modify or silence alert rules in specific folders or using specific data sources.
Details on the additional access provided by folder permissions are below.
Folder permission
Additional Access
View
Read access to all general silences. Read access to alert rules and their rule-specific silences only in the given folder and subfolders.
Edit
View access and write access to alert rules and their rule-specific silences only in the given folder and subfolders.
Admin
Same additional access as Edit.
Some example combinations:
Role
Folder permission
Access
No Basic Role
-
None. No access to alert rules or rule-specific silences in the folder.
No Basic Role
View
Read access to all general silences. Read access to alert rules and their rule-specific silences only in the given folder and subfolders.
No Basic Role
Edit
Above access and write access to alert rules (given necessary data source query permissions) and their rule-specific silences only in the given folder and subfolders.
Viewer
-
Read access to alert rules and silences in all folders.
Viewer
Edit
Read access to alert rules and silences in all folders. Write access to alert rules and their rule-specific silences only in the given folder and subfolders.
Editor
View
Read and write access to alert rules and silences in all folders. Access can only be extended.
{{< admonition type="note" >}}
You can't use folders to customize access to notification resources.
{{< /admonition >}}
Folder permissions
To manage folder permissions, complete the following steps.
In the left-side menu, click Dashboards.
Choose the folder you want to add permissions for.
{{< admonition type="note" >}}It doesn’t matter which tab you’re on (Dashboards, Panels, or Alert rules); the folder permission you set applies to all.{{< /admonition >}}
Click Manage permissions from the Folder actions menu.
Update or add permissions as required.
Data source permissions
By default, users with the basic roles Admin, Editor, and Viewer roles have query access to data sources for Grafana Alerting.
If you used fixed roles or custom roles, you need to update data source permissions to create rules or update rule queries.
Alternatively, an administrator can assign the role Datasource Reader, which grants the user access to all data sources.
To manage data source permissions, complete the following steps.
In the left-side menu, click Connections > Data sources.
Click the data source you want to change the permissions for.
Click the Permissions tab.
In the Permission column, update the permission, or remove it by clicking X.

Manage access using roles
In Grafana Enterprise and Grafana Cloud there are Basic, Fixed, and Custom roles.
Basic roles
There are four basic roles: 
Admin, 
Editor, 
Viewer, and 
None. Each basic role contains a number of fixed roles.
Details of the basic roles and the access they provide for Grafana Alerting are below.
Role
Access
Admin
Write access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), alert enrichments, and provisioning.
Editor
Write access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), alert enrichments, and provisioning.
Viewer
Read access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), and alert enrichments.
None
A blank canvas to assign fixed or custom roles and craft permissions more precisely. For example, if you want to give a user the ability to see alert rules, but not notification settings, add No basic role and then the fixed role Rules reader.
Fixed roles
A fixed role is a group of multiple permissions.
Fixed roles provide users more granular access to create, view, and update Alerting resources than you would have with basic roles alone.
Details of the fixed roles and the access they provide for Grafana Alerting are below.
Display name in UI / Fixed role
Permissions
Description
Silences Writer: 
fixed:alerting.instances:writer
All permissions from 
fixed:alerting.instances:reader and
 
alert.instances:create
alert.instances:write for organization scope 
 
alert.instances.external:write for scope 
datasources:*
Add and update silences in Grafana and external providers.
Instances and Silences Reader: 
fixed:alerting.instances:reader
alert.instances:read for organization scope 
 
alert.instances.external:read for scope 
datasources:*
Read alert instances and silences in Grafana and external providers.
Notifications Writer: 
fixed:alerting.notifications:writer
All permissions from 
fixed:alerting.notifications:reader,
 
fixed:alerting.receivers:writer,
 
fixed:alerting.templates:writer,
 
fixed:alerting.time-intervals:writer,
 
fixed:alerting.routes:writer,
 
fixed:alerting.inhibition-rules:writer,
 
fixed:alerting.notifications.external:writer
Add, update, and delete notification policies and contact points in Grafana and external providers.
Notifications Reader: 
fixed:alerting.notifications:reader
All permissions from 
fixed:alerting.receivers:reader,
 
fixed:alerting.templates:reader,
 
fixed:alerting.time-intervals:reader,
 
fixed:alerting.routes:reader,
 
fixed:alerting.inhibition-rules:reader,
 
fixed:alerting.notifications.external:reader
Read notification policies and contact points in Grafana and external providers.
Rules Writer: 
fixed:alerting.rules:writer
All permissions from 
fixed:alerting.rules:reader and 
 
alert.rule:create 
 
alert.rule:write 
 
alert.rule:delete 
 
alert.silences:create 
 
alert.silences:write for scope 
folders:* 
 
alert.rules.external:write for scope 
datasources:*
Create, update, and delete all alert rules and manage rule-specific silences.
Rules Reader: 
fixed:alerting.rules:reader
alert.rule:read, 
alert.silences:read for scope 
folders:* 
 
alert.rules.external:read for scope 
datasources:* 
 
alert.notifications.time-intervals:read 
 
alert.notifications.receivers:list
Read all alert rules and rule-specific silences in Grafana and external providers.
Full access: 
fixed:alerting:writer
All permissions from 
fixed:alerting.rules:writer 
fixed:alerting.instances:writer
fixed:alerting.notifications:writer
Add, update, and delete alert rules, silences, contact points, and notification policies in Grafana and external providers.
Full read-only access: 
fixed:alerting:reader
All permissions from 
fixed:alerting.rules:reader 
fixed:alerting.instances:reader
fixed:alerting.notifications:reader
Read alert rules, alert instances, silences, contact points, and notification policies in Grafana and external providers.
Read via Provisioning API + Export Secrets: 
fixed:alerting.provisioning.secrets:reader
alert.provisioning:read and 
alert.provisioning.secrets:read
Read alert rules, alert instances, silences, contact points, and notification policies using the provisioning API and use export with decrypted secrets.
Access to alert rules provisioning API: 
fixed:alerting.provisioning:writer
alert.provisioning:read and 
alert.provisioning:write
Manage all alert rules, notification policies, contact points, templates, in the organization using the provisioning API.
Set provisioning status: 
fixed:alerting.provisioning.provenance:writer
alert.provisioning.provenance:write
Set provisioning rules for Alerting resources. Should be used together with other regular roles (Notifications Writer and/or Rules Writer.)
Contact Point Reader: 
fixed:alerting.receivers:reader
alert.notifications.receivers:read for scope 
receivers:*
Read all contact points.
Contact Point Creator: 
fixed:alerting.receivers:creator
alert.notifications.receivers:create 
 
alert.notifications.receivers.test:create for scope 
receivers:uid:-
Create a new contact point. The user is automatically granted full access to the created contact point.
Contact Point Writer: 
fixed:alerting.receivers:writer
All permissions from 
fixed:alerting.receivers:creator 
 
alert.notifications.receivers:read, 
alert.notifications.receivers:write, 
alert.notifications.receivers:delete and 
alert.notifications.receivers.test:create for scope 
receivers:*
Create a new contact point and manage all existing contact points.
Templates Reader: 
fixed:alerting.templates:reader
alert.notifications.templates:read
Read all notification templates.
Templates Writer: 
fixed:alerting.templates:writer
alert.notifications.templates:read, 
alert.notifications.templates:write, 
alert.notifications.templates:delete, 
alert.notifications.templates.test:write
Create new and manage existing notification templates. Test templates with custom payloads.
Time Intervals Reader: 
fixed:alerting.time-intervals:reader
alert.notifications.time-intervals:read
Read all time intervals.
Time Intervals Writer: 
fixed:alerting.time-intervals:writer
alert.notifications.time-intervals:read, 
alert.notifications.time-intervals:write, 
alert.notifications.time-intervals:delete
Create new and manage existing time intervals.
Notification Policies Reader: 
fixed:alerting.routes:reader
notifications.alerting.grafana.app/routingtrees:get for scope 
notifications.alerting.grafana.app/routingtrees:*
Read all notification policies.
Notification Policies Writer: 
fixed:alerting.routes:writer
All permissions from 
fixed:alerting.routes:reader and
notifications.alerting.grafana.app/routingtrees:create, and
notifications.alerting.grafana.app/routingtrees:update, 
notifications.alerting.grafana.app/routingtrees:delete for scope 
notifications.alerting.grafana.app/routingtrees:*
Create new and manage existing notification policies.
Inhibition Rules Reader: 
fixed:alerting.inhibition-rules:reader
alert.notifications.inhibition-rules:read for scope 
inhibition-rules:*
Read all inhibition rules.
Inhibition Rules Writer: 
fixed:alerting.inhibition-rules:writer
All permissions from 
fixed:alerting.inhibition-rules:reader and
alert.notifications.inhibition-rules:write, 
alert.notifications.inhibition-rules:delete for scope 
inhibition-rules:*
Create, update, and delete all inhibition rules.
Full admin access: 
fixed:alerting:admin
All permissions from 
fixed:alerting:writer and
alert.notifications.receivers.permissions:read, 
alert.notifications.receivers.permissions:write, 
alert.notifications.receivers:readSecrets, 
alert.notifications.receivers:updateProtected for scope 
receivers:*
Full write access in Grafana and all external providers, including their permissions, protected fields and secrets.
Enrichments Reader: 
fixed:alerting.enrichments:reader
alert.enrichments:read
Read per-rule alert enrichment configurations. Alert enrichment is a Grafana Cloud feature; global enrichment settings require the Admin basic role.
Enrichments Writer: 
fixed:alerting.enrichments:writer
alert.enrichments:read
alert.enrichments:write
Create and manage per-rule alert enrichment configurations. Alert enrichment is a Grafana Cloud feature; global enrichment settings require the Admin basic role.
External Notifications Reader: 
fixed:alerting.notifications.external:reader
alert.notifications.external:read for scope 
datasources:*
Read notification policies and contact points in external providers.
External Notifications Writer: 
fixed:alerting.notifications.external:writer
All permissions from 
fixed:alerting.notifications.external:reader and
alert.notifications.external:write for scope 
datasources:*
Add, update, and delete contact points and notification policies in external providers.
Create custom roles
Create custom roles of your own to manage permissions. Custom roles contain unique combinations of permissions, actions and scopes. Create a custom role when basic roles and fixed roles do not meet your permissions requirements.
For more information on creating custom roles, refer to Create custom roles.
{{< admonition type="note" >}}
It is not recommended to create custom roles that include 
alerting.notifications.receiver actions with a scope other than 
receivers:*. The UID used in the scope is not stable and changes whenever a contact point is renamed.
{{< /admonition >}}
Examples
The following examples give you an idea of how you can combine permissions for Grafana Alerting.
A custom role for read access to alert rules in folder F:
PUT access-control/roles
{
	"name": "custom:alert_rules_reader",
	"displayName": "Alert rule reader in folder F",
	"description": "Read access to rules in folder F that use DS1 and DS2",
	"permissions": [
    	{
        	"action": "alert.rules:read",
        	"scope": "folders:uid:UID_F"
    	},
    	{
        	"action": "folders:read",
        	"scope": "folders:uid:UID_F"
    	}
	]
}
A custom role for write access to alert rules that uses simplified routing:
PUT access-control/roles
{
	"name": "custom:alert_rules_updater",
	"displayName": "Alert rules editor in folder F",
	"description": "Edit access to rules in folder F that use DS1 and DS2",
	"permissions": [
    	{
        	"action": "alert.rules:read",
        	"scope": "folders:uid:UID_F"
    	},
    	{
        	"action": "alert.rules:read",
        	"scope": "folders:uid:UID_F"
    	},
    	{
        	"action": "alert.rules:write",
        	"scope": "folders:uid:UID_F"
    	},
    	{
        	"action": "alert.rules:create",
        	"scope": "folders:uid:UID_F"
    	},
    	{
        	"action": "alert.notifications.receivers:list",
    	},
{
        	"action": "alert.notifications.time-intervals:read",
    	},
	]
}
{{< admonition type="note" >}}
Delete the last two permissions if you aren’t using simplified notification routing.
{{< /admonition >}}
Assign roles
To assign roles, complete the following steps.
Navigate to Administration > Users and access > Users, Teams, or Service Accounts.
Search for the user, team or service account you want to add a role for.
Select the role you want to assign.

Configure silence access
This article describes how Grafana Alerting controls access to silences and how to configure it for your teams. Use this article when you need to give some users full silence management while limiting others to silencing only their own rules.
Before you begin, ensure you have:
Admin access to the Grafana organization
Familiarity with Grafana RBAC and fixed roles
Silence types
Grafana Alerting has two types of silences:
General silences — not attached to any specific alert rule. They can match alerts from any rule in the organization. Only users with org-scoped silence permissions can create them.
Rule-specific silences — linked to a single alert rule via its UID. Access is controlled by the folder that contains the rule.
When you silence a firing alert directly from the alert list, Grafana creates a rule-specific silence automatically by adding the label matcher 
__alert_rule_uid__=<rule UID>. You can also create a rule-specific silence manually by including this matcher.
Silence permissions
Grafana has two independent sets of permissions for silences. They serve different purposes and apply to different scopes.
Org-scoped permissions
These permissions apply across the whole organization and cover both silence types.
Action
Description
alert.instances:read
Read all silences — general and rule-specific
alert.instances:create
Create any silence, including general silences
alert.instances:write
Update or expire any silence
The built-in 
Editor role includes all three via the 
fixed:alerting.instances:writer fixed role.
Folder-scoped permissions
These permissions are scoped to a specific folder. Users with only these permissions can manage rule-specific silences for rules in the granted folder, but can't create general silences.
Action
Description
alert.silences:read
Read rule-specific silences for rules in the folder, plus all general silences org-wide
alert.silences:create
Create rule-specific silences for rules in the folder
alert.silences:write
Update or expire rule-specific silences for rules in the folder
{{< admonition type="note" >}}
alert.silences:read always includes all general silences, regardless of the folder scope. General silences can suppress notifications for any rule, so users managing rules in a folder need to see silences that may affect their alerts.
{{< /admonition >}}
Fixed roles
The following fixed roles are relevant to silence management.
Fixed role
Permissions
What it allows
fixed:alerting.instances:writer (Silences Writer)
alert.instances:create, 
alert.instances:write (org scope)
Create and manage general and rule-specific silences org-wide
fixed:alerting.instances:reader (Instances and Silences Reader)
alert.instances:read (org scope)
Read all silences
fixed:alerting.rules:writer (Rules Writer)
alert.silences:create, 
alert.silences:write for 
folders:*
Create and manage rule-specific silences
fixed:alerting.rules:reader (Rules Reader)
alert.silences:read for 
folders:*
Read rule-specific silences and all general silences
Folder permission mapping in Grafana OSS
In Grafana OSS, custom RBAC isn't available. Instead, folder permissions automatically grant the corresponding silence access.
Folder permission
Silence access
View
Read all general silences; read rule-specific silences for rules in the folder
Edit
View access, plus create and update rule-specific silences for rules in the folder
Admin
Same as Edit
In Grafana Enterprise and Grafana Cloud, folder permissions grant the same silence access as in OSS, but you can go further by assigning folder-scoped RBAC permissions independently of rule access — for example, granting silence write access without granting rule write access.
Scenario: restrict users to rule-specific silences only
Use this setup when a team should be able to silence their own rules but must not create general silences that affect the whole organization.
Grant the following folder-scoped permissions for the team's folder:
Permission
Scope
alert.silences:read
folders:<FOLDER_UID>
alert.silences:create
folders:<FOLDER_UID>
alert.silences:write
folders:<FOLDER_UID>
These permissions are included in 
fixed:alerting.rules:writer and are also granted automatically by the folder Edit permission.
Do not grant 
alert.instances:create or 
alert.instances:write at org scope — these allow creating general silences. They're bundled into 
fixed:alerting.instances:writer and the built-in 
Editor role, so avoid assigning either to users who should be folder-restricted.
With this configuration:
Users can create silences only for rules in their folder.
Users can read all general silences.
Any attempt to create a general silence is rejected.
Scenario: grant full silence access
Use this setup for on-call engineers or administrators who need full control over all silences.
Assign the following fixed roles:
Fixed role
Grants
fixed:alerting.instances:writer
Create and manage general and rule-specific silences org-wide
fixed:alerting.rules:writer
Create and manage rule-specific silences across all folders
Alternatively, assign the built-in 
Editor role, which includes both.
With this configuration:
Users can create general silences for org-wide maintenance windows.
Users can create and expire rule-specific silences for any rule.
Users see all silences across the organization.
Security considerations
Keep the following in mind when configuring silence access.
Org-scoped write actions are powerful. Granting 
alert.instances:create or 
alert.instances:write at org scope — even to a user who otherwise has only folder-level access — allows them to create or modify silences that affect the entire organization. Review all custom and inherited roles that include these actions.
General silences are always visible to folder users. You can't hide general silences from users who have any folder-scoped 
alert.silences:read permission. If general silences carry sensitive context (such as the name of an active incident), use clear, consistent naming conventions.
Use rule-specific silences for team isolation. In multi-team or multi-tenant deployments, rule-specific silences are the recommended pattern. They respect folder boundaries, which general silences don't.
Next steps
Configure silences — create, edit, and expire silences
Manage access using roles — full reference for fixed and custom roles
Manage access using folders — folder-based access for alert rules and silences

Configure roles and permissions
This guide explains how to configure roles and permissions for Grafana Alerting for Grafana OSS users. You'll learn how to manage access using roles, folder permissions, and contact point permissions.
A user is any individual who can log in to Grafana. Each user is associated with a role that includes permissions. Permissions determine the tasks a user can perform in the system. For example, the Admin role includes permissions for an administrator to create and delete users.
For more information, refer to Organization roles.
Manage access using roles
Grafana OSS has four basic roles: 
Admin, 
Editor, 
Viewer, and 
None.
The following table describes the access each role provides for Grafana Alerting.
Role
Access
Admin
Write access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), and provisioning, as well as assign roles.
Editor
Write access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences), and provisioning.
Viewer
Read access to alert rules, notification resources (notification API, contact points, templates, time intervals, notification policies, and silences).
None
No access to alert rules or notification resources. Limit access to only the required resources by assigning fixed or custom roles, or granting folder or contact point permissions.
Assign roles
To assign roles, an admin needs to complete the following steps.
Navigate to Administration > Users and access > Users, Teams, or Service Accounts.
Search for the user, team or service account you want to add a role for.
Add the role you want to assign.
Manage access using folder permissions
You can extend the access provided by a role to alert rules and rule-specific silences by assigning permissions to individual folders.
This allows different users, teams, or service accounts to have customized access to modify or silence alert rules in specific folders.
Refer to the following table for details on the additional access provided by folder permissions:
Folder permission
Additional Access
Admin
Same additional access as Edit.
Edit
Write access to alert rules and their rule-specific silences only in the given folder and subfolders.
View
No additional access: all permissions already contained in Viewer role.
{{< admonition type="note" >}}
You can't use folders to customize access to notification resources.
{{< /admonition >}}
To manage folder permissions, complete the following steps:
In the left-side menu, click Dashboards.
Hover your mouse cursor over a folder and click Go to folder.
Click Manage permissions from the Folder actions menu.
Update or add permissions as required.
Manage access to contact points
Extend or limit the access provided by a role to contact points by assigning permissions to individual contact points.
This allows different users, teams, or service accounts to have customized access to read or modify specific contact points.
Refer to the following table for details on the additional access provided by contact point permissions.
Contact point permission
Additional Access
Admin
Same additional access as Edit and manage permissions for the contact point. User should have additional permissions to read users and teams.
Edit
Update or delete the contact point
View
View and export contact point as well as select it on the Alert rule edit page
Assign contact point permissions
To manage contact point permissions, complete the following steps:
In the left-side menu, click Notification configuration, then select the Contact points tab.
Hover your mouse cursor over a contact point and click More.
Click Manage permissions from the actions menu.
Update or add permissions as required.

Performance considerations and limitations
Grafana Alerting supports multi-dimensional alerting, where one alert rule can generate many alerts. For example, you can configure an alert rule to fire an alert every time the CPU of individual virtual machines max out. This topic discusses performance considerations resulting from multi-dimensional alerting.
Evaluating alerting rules consumes RAM and CPU to compute the output of an alerting query, and network resources to send alert notifications and write the results to the Grafana SQL database. The configuration of individual alert rules affects the resource consumption and, therefore, the maximum number of rules a given configuration can support.
The following section provides a list of alerting performance considerations.
Frequency of rule evaluation consideration. The "Evaluate Every" property of an alert rule controls the frequency of rule evaluation. It is recommended to use the lowest acceptable evaluation frequency to support more concurrent rules.
Cardinality of the rule's result set. For example, suppose you are monitoring API response errors for every API path, on every virtual machine in your fleet. This set has a cardinality of n number of paths multiplied by v number of VMs. You can reduce the cardinality of a result set - perhaps by monitoring errors-per-VM instead of for each path per VM.
Complexity of the alerting query consideration. Queries that data sources can process and respond to quickly consume fewer resources. Although this consideration is less important than the other considerations listed above, if you have reduced those as much as possible, looking at individual query performance could make a difference.
Each evaluation of an alert rule generates a set of alert instances; one for each member of the result set. The state of all the instances is written to the 
alert_instance table in the Grafana SQL database. This number of write-heavy operations can cause issues when using SQLite.
Grafana Alerting exposes a metric, 
grafana_alerting_rule_evaluations_total that counts the number of alert rule evaluations. To get a feel for the influence of rule evaluations on your Grafana instance, you can observe the rate of evaluations and compare it with resource consumption. In a Prometheus-compatible database, you can use the query 
rate(grafana_alerting_rule_evaluations_total[5m]) to compute the rate over 5 minute windows of time. It's important to remember that this isn't the full picture of rule evaluation. For example, the load is unevenly distributed if you have some rules that evaluate every 10 seconds, and others every 30 minutes.
These factors all affect the load on the Grafana instance, but you should also be aware of the performance impact that evaluating these rules has on your data sources. Alerting queries are often the vast majority of queries handled by monitoring databases, so the same load factors that affect the Grafana instance affect them as well.
Limited rule sources support
Grafana Alerting can retrieve alerting and recording rules stored in most available Prometheus, Loki, Mimir, and Alertmanager compatible data sources.
It does not support reading or writing alerting rules from any other data sources but the ones previously mentioned at this time.
Prometheus version support
The latest two minor versions of both Prometheus and Alertmanager are supported. We cannot guarantee that older versions work.
As an example, if the current Prometheus version is 
2.31.1, >= 
2.29.0 is supported.
The Grafana Alertmanager can only receive Grafana managed alerts
Grafana cannot be used to receive external alerts. You can only send alerts to the Grafana Alertmanager using Grafana managed alerts.
You have the option to send Grafana-managed alerts to an external Alertmanager, you can find this option in the Admin tab on the Alerting page.
For more information, refer to this GitHub issue.
High load on database caused by a high number of alert instances
If you have a high number of alert rules or alert instances, the load on the database can get very high.
Grafana performs one SQL update per alert rule after each evaluation. This update groups all alert instances belonging to the rule together and compresses them into a single protobuf-based row, which keeps database overhead low even for alert rules with many instances.
{{< admonition type="warning" >}}
Earlier versions of Grafana wrote alert instance state to the database uncompressed, one row per instance, and let you opt in to compressed storage with the 
alertingSaveStateCompressed feature toggle. Grafana 13.2 removes that feature toggle and the uncompressed storage path entirely.
If you're upgrading from a Grafana version where you had explicitly disabled 
alertingSaveStateCompressed, enable it and let Grafana run for at least one evaluation cycle before you upgrade to Grafana 13.2 or later. Otherwise, any alert state still held in the old uncompressed format isn't read after the upgrade. This doesn't cause data loss beyond the alert state itself: Grafana re-evaluates every rule and rebuilds instance state from scratch, but you might see a brief, one-time gap in alert history and current state immediately after the upgrade.
{{< /admonition >}}
Save state periodically
You can also reduce database load by writing states periodically instead of after every evaluation.
Enable the 
alertingSaveStatePeriodic feature toggle to save alert states at the interval specified by 
state_periodic_save_interval instead of after every rule evaluation. Grafana groups all alert instances by rule UID and compresses them together for efficient storage.
By default, Grafana saves the states every 5 minutes and on each shutdown. Note that 
state_periodic_save_batch_size and 
state_periodic_save_jitter_enabled don't apply here, since Grafana groups instances by rule UID rather than by batch.
[unified_alerting]
state_periodic_save_interval = 1m
If Grafana crashes or is force killed, then the database can be up to 
state_periodic_save_interval seconds out of date.
When Grafana restarts, the UI might show incorrect state for some alerts until the alerts are re-evaluated.
In some cases, alerts that were firing before the crash might fire again.
If this happens, Grafana might send duplicate notifications for firing alerts.
Limit on the number of results per alert rule
Each alert rule evaluation generates one alert instance per series in the rule's query result set. Rules with high-cardinality result sets consume more CPU, memory, network, and database resources, and can produce a large number of alert instances (refer to High load on database caused by a high number of alert instances).
To prevent this, Grafana can limit the number of query evaluation results a single alert rule produces in one evaluation. When a rule's query returns more results than the limit, the evaluation fails with an error similar to the following:
query evaluation returned too many results: 12345 (limit: 10000)
The alert rule enters the Error state and produces no alert instances for that evaluation until you reduce its result set below the limit.
In self-managed Grafana, set this limit using the 
alerting_rule_evaluation_results option in the 
[quota] section. The default is 
-1 (unlimited). In Grafana Cloud, Grafana Labs manages this limit.
To resolve the error, reduce the cardinality of the rule's result set:
Aggregate the query so it returns fewer series. For example, sum or average by fewer labels.
Add label filters so the query returns only the series you need to alert on.
Split a single high-cardinality rule into several rules with narrower queries.
Alert rule migrations for Grafana 11.6.0
When you upgrade to Grafana 11.6.0, a migration is performed on the 
alert_rule_versions table. If you experience a 11.6.0 upgrade that causes a migration failure, then your 
alert_rule_versions table has too many rows. To fix this, you need to truncated the 
alert_rule_versions table for the migration to complete.

Export alerting resources
Export your alerting resources, such as alert rules, contact points, and notification policies for provisioning, automatically importing single folders and single groups.
There are distinct methods to export your alerting resources:
Grafana UI exports in Terraform format and YAML or JSON formats for file provisioning.
HTTP Alerting API exports in JSON API format used by the HTTP Alerting API.
HTTP Alerting API - Export endpoints exports in YAML or JSON formats for file provisioning.
{{< admonition type="note" >}}
Alerting resources imported through file provisioning cannot be edited in the Grafana UI. This prevents changes made in the UI from being overridden by file provisioning during Grafana restarts.
If you need to modify provisioned alerting resources in Grafana, refer to edit HTTP API alerting resources in the Grafana UI or to edit Terraform alerting resources in the Grafana UI.
{{< /admonition >}}
Export from the Grafana UI
The export options listed below enable you to download resources in YAML, JSON, or Terraform format, facilitating their provisioning through configuration files or Terraform.
Export alert rules
To export alert rules from the Grafana UI, complete the following steps.
Click Alerts & IRM -> Alert rules.
To export all Grafana-managed rules, click Export rules.
To export a folder, change the View as to List.
Select the folder you want to export and click the Export rules folder icon.
To export a group, change the View as to Grouped.
Find the group you want to export and click the Export rule group icon.
Choose the format to export in.
The exported alert rule data appears in different formats - YAML, JSON, Terraform.
Click Copy Code or Download.
Modify alert rule and export rule group without saving changes
{{< admonition type="note" >}} This feature is for Grafana-managed alert rules only. It is available to Admin, Viewer, and Editor roles. {{< /admonition >}}
Use the Modify export mode to edit and export an alert rule without updating it. The exported data includes all alert rules within the same alert group.
To export a modified alert rule without saving the modifications, complete the following steps from the Grafana UI.
Click Alerts & IRM -> Alert rules.
Locate the alert rule you want to edit and click More -> Modify Export to open the Alert Rule form.
From the Alert Rule form, edit the fields you want to change. Changes made are not applied to the alert rule.
Click Export.
Choose the format to export in.
The exported alert rule group appears in different formats - YAML, JSON, Terraform.
Click Copy Code or Download.
Export a new alert rule definition without saving changes
{{< admonition type="note" >}} You can only export in Terraform (HCL) format. {{< /admonition >}}
Add a new alert rule definition to an existing provisioned rule group rather than creating the code manually. You can then copy it to your Terraform pipeline, and quickly deploy and manage alert rules as part of your infrastructure as code.
An alert rule definition differs from an alert rule in the sense that you define your alert rule, but you do not save it. It only becomes a saved alert rule once you provision it using Terraform.
To export your alert rule definition into Terraform (HCL) format, complete the following steps.
Click Alerts & IRM -> Alert rules.
Click Export rule definition.
Fill out the alert rule details.
Choose a provisioned folder and group to add your alert rule definition to.
Click Export.
Copy and paste the code into your Terraform pipeline to create your new alert rule.
Export contact points
To export contact points from the Grafana UI, complete the following steps.
Click Alerts & IRM -> Contact points.
Find the contact point you want to export and click More -> Export.
Choose the format to export in.
The exported contact point appears in different formats - YAML, JSON, Terraform.
Click Copy Code or Download.
Export notification template groups
Grafana currently doesn't offer an Export UI or Export endpoint for notification template groups, unlike other Alerting resources presented in this documentation.
However, you can export it by manually copying the content and name of the notification template group from the Grafana UI.
Click Alerts & IRM -> Contact points -> Notification templates tab.
Find the notification template group you want to export.
Copy the content and name.
Adjust it for the file provisioning format or Terraform resource.
Export the notification policy tree
All notification policies are provisioned through a single resource: the root of the notification policy tree.
{{< docs/shared lookup="alerts/warning-provisioning-tree.md" source="grafana" version="<GRAFANA_VERSION>" >}}
To export the notification policy tree from the Grafana UI, complete the following steps.
Click Alerts & IRM -> Notification policies.
In the Default notification policy section, click ... -> Export.
Choose the format to export in.
The exported notification policy tree appears in different formats - YAML, JSON, Terraform.
Click Copy Code or Download.
You can also export individual named routing trees.
Click Alerts & IRM -> Notification policies.
Find the routing tree you want to export in the list.
Click the More actions menu item and select Export.
Choose the export format and click Copy Code or Download.
Export mute timings
To export mute timings from the Grafana UI, complete the following steps.
Click Alerts & IRM -> Notification policies, and then the Mute timings tab.
Find the mute timing you want to export and click Export.
Choose the format to export in.
The exported contact point appears in different formats - YAML, JSON, Terraform.
Click Copy Code or Download.
HTTP Alerting API
You can use the Alerting HTTP API to return existing alerting resources in JSON and import them to another Grafana instance using the same endpoint.
Resource
URI
Alert rules
/api/v1/provisioning/alert-rules
Contact points
/api/v1/provisioning/contact-points
Notification policy tree
/api/v1/provisioning/policies
Notification template groups
/api/v1/provisioning/templates
Mute timings
/api/v1/provisioning/mute-timings
However, note the standard endpoints return a JSON format that is not compatible for provisioning through configuration files or Terraform, except the 
/export endpoints listed below.
Export API endpoints
The Alerting HTTP API provides specific endpoints for exporting alerting resources in YAML or JSON, facilitating provisioning via configuration files, or Terraform (HCL).
Resource
Method / URI
Summary
Alert rules
GET /api/v1/provisioning/alert-rules/export
Export all alert rules in provisioning file format.
Alert rules
GET /api/v1/provisioning/folder/:folderUid/rule-groups/:group/export
Export an alert rule group in provisioning file format.
Alert rules
GET /api/v1/provisioning/alert-rules/:uid/export
Export an alert rule in provisioning file format.
Contact points
GET /api/v1/provisioning/contact-points/export
Export all contact points in provisioning file format.
Notification policy tree
GET /api/v1/provisioning/policies/export
Export the notification policy tree in provisioning file format.
Mute timings
GET /api/v1/provisioning/mute-timings/export
Export all mute timings in provisioning file format.
Mute timings
GET /api/v1/provisioning/mute-timings/:name/export
Export a mute timing in provisioning file format.
These endpoints accept a 
download parameter to download a file containing the exported resources.

Use configuration files to provision alerting resources
Manage your alerting resources using configuration files that can be version controlled. When Grafana starts, it provisions the resources defined in your configuration files. Provisioning can create, update, or delete existing resources in your Grafana instance.
This guide outlines the steps and references to provision alerting resources using YAML files. For a practical demo, you can clone and try this example using Grafana OSS and Docker Compose.
{{< admonition type="note" >}}
Provisioning Grafana with configuration files is not available in Grafana Cloud.
You cannot edit provisioned resources from files in Grafana. You can only change the resource properties by changing the provisioning file and restarting Grafana or carrying out a hot reload. This prevents changes from being made to the resource that would be overwritten if a file is provisioned again or a hot reload is carried out.
Provisioning using configuration files takes place during the initial set up of your Grafana system, but you can re-run it at any time using the Grafana Admin API.
Importing an existing alerting resource results in a conflict. First, when present, remove the resources you plan to import.
{{< /admonition >}}
Details on how to set up the files and which fields are required for each object are listed below depending on which resource you are provisioning.
Import alert rules
Create or delete alert rules using provisioning files in your Grafana instance(s).
Find the alert rule group in Grafana.
Export and download a provisioning file for your alert rules.
Copy the contents into a YAML or JSON configuration file and add it to the 
provisioning/alerting directory of the Grafana instance you want to import the alerting resources to.
Example configuration files can be found below.
Restart your Grafana instance (or reload the provisioned files using the Admin API).
Here is an example of a configuration file for creating alert rules.
# config file version
apiVersion: 1
# List of rule groups to import or update
groups:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the rule group
    name: my_rule_group
    # <string, required> name of the folder the rule group will be stored in
    folder: my_first_folder
    # <duration, required> interval that the rule group should be evaluated at
    interval: 60s
    # <list, required> list of rules that are part of the rule group
    rules:
      # <string, required> unique identifier for the rule. Should not exceed 40 symbols. Only letters, numbers, - (hyphen), and _ (underscore) allowed.
      - uid: my_id_1
        # <string, required> title of the rule that will be displayed in the UI
        title: my_first_rule
        # <string, required> which query should be used for the condition
        condition: A
        # <list, required> list of query objects that should be executed on each
        #                  evaluation - should be obtained through the API
        data:
          - refId: A
            datasourceUid: '__expr__'
            model:
              conditions:
                - evaluator:
                    params:
                      - 3
                    type: gt
                  operator:
                    type: and
                  query:
                    params:
                      - A
                  reducer:
                    type: last
                  type: query
              datasource:
                type: __expr__
                uid: '__expr__'
              expression: 1==0
              intervalMs: 1000
              maxDataPoints: 43200
              refId: A
              type: math
        # <string> UID of a dashboard that the alert rule should be linked to
        dashboardUid: my_dashboard
        # <int> ID of the panel that the alert rule should be linked to
        panelId: 123
        # <string> the state the alert rule will have when no data is returned
        #          possible values: "NoData", "Alerting", "OK", default = NoData
        noDataState: Alerting
        # <string> the state the alert rule will have when the query execution
        #          failed - possible values: "Error", "Alerting", "OK"
        #          default = Alerting
        execErrState: Alerting
        # <duration, required> for how long should the alert fire before alerting
        for: 60s
        # <map<string, string>> a map of strings to pass around any data
        annotations:
          some_key: some_value
        # <map<string, string> a map of strings that can be used to filter and
        #                      route alerts
        labels:
          team: sre_team_1
Here is an example of a configuration file for deleting alert rules.
# config file version
apiVersion: 1
# List of alert rule UIDs that should be deleted
deleteRules:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> unique identifier for the rule
    uid: my_id_1
Import contact points
Create or delete contact points using provisioning files in your Grafana instance(s).
Find the contact point in Grafana.
Export and download a provisioning file for your contact point.
Copy the contents into a YAML or JSON configuration file and add it to the 
provisioning/alerting directory of the Grafana instance you want to import the alerting resources to.
Example configuration files can be found below.
Restart your Grafana instance (or reload the provisioned files using the Admin API).
Here is an example of a configuration file for creating contact points.
# config file version
apiVersion: 1
# List of contact points to import or update
contactPoints:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the contact point
    name: cp_1
    receivers:
      # <string, required> unique identifier for the receiver. Should not exceed 40 symbols. Only letters, numbers, - (hyphen), and _ (underscore) allowed.
      - uid: first_uid
        # <string, required> type of the receiver
        type: prometheus-alertmanager
        # <bool, optional> Disable the additional [Incident Resolved] follow-up alert, default = false
        disableResolveMessage: false
        # <object, required> settings for the specific receiver type
        settings:
          url: http://test:9000
Here is an example of a configuration file for deleting contact points.
# config file version
apiVersion: 1
# List of receivers that should be deleted
deleteContactPoints:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> unique identifier for the receiver
    uid: first_uid
Settings
Here are some examples of settings you can use for the different
contact point integrations.
{{< collapse title="Alertmanager" >}}
Alertmanager
type: prometheus-alertmanager
settings:
  # <string, required>
  url: http://localhost:9093
  # <string>
  basicAuthUser: abc
  # <string>
  basicAuthPassword: abc123
{{< /collapse >}}
{{< collapse title="DingDing" >}}
DingDing
type: dingding
settings:
  # <string, required>
  url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx
  # <string> options: link, actionCard
  msgType: link
  # <string>
  message: |
    {{ template "default.message" . }}
{{< /collapse >}}
{{< collapse title="Discord" >}}
Discord
type: discord
settings:
  # <string, required>
  url: https://discord/webhook
  # <string>
  avatar_url: https://my_avatar
  # <bool>
  use_discord_username: false
  # <string>
  message: |
    {{ template "default.message" . }}
{{< /collapse >}}
{{< collapse title="E-Mail" >}}
E-Mail
type: email
settings:
  # <string, required>
  addresses: me@example.com;you@example.com
  # <bool>
  singleEmail: false
  # <string>
  message: my optional message to include
  # <string>
  subject: |
    {{ template "default.title" . }}
{{< /collapse >}}
{{< collapse title="Google Chat" >}}
Google Chat
type: googlechat
settings:
  # <string, required>
  url: https://google/webhook
  # <string>
  message: |
    {{ template "default.message" . }}
{{< /collapse >}}
{{< collapse title="Kafka" >}}
Kafka
type: kafka
settings:
  # <string, required>
  kafkaRestProxy: http://localhost:8082
  # <string, required>
  kafkaTopic: topic1
{{< /collapse >}}
{{< collapse title="LINE" >}}
LINE
type: line
settings:
  # <string, required>
  token: xxx
{{< /collapse >}}
{{< collapse title="MQTT" >}}
MQTT
type: mqtt
settings:
  # <string, required>
  brokerUrl: tcp://127.0.0.1:1883
  # <string>
  clientId: grafana
  # <string, required>
  topic: grafana/alerts
  # <string>
  messageFormat: json
  # <string>
  username: grafana
  # <string>
  password: password1
  # <string>
  qos: 0
  # <bool>
  retain: false
  # <map>
  tlsConfig:
    # <bool>
    insecureSkipVerify: false
    # <string>
    clientCertificate: certificate in PEM format
    # <string>
    clientKey: key in PEM format
    # <string>
    caCertificate: CA certificate in PEM format
{{< /collapse >}}
{{< collapse title="Microsoft Teams" >}}
Microsoft Teams
type: teams
settings:
  # <string, required>
  url: https://ms_teams_url
  # <string>
  title: |
    {{ template "default.title" . }}
  # <string>
  sectiontitle: ''
  # <string>
  message: |
    {{ template "default.message" . }}
{{< /collapse >}}
{{< collapse title="OpsGenie" >}}
OpsGenie
type: opsgenie
settings:
  # <string, required>
  apiKey: xxx
  # <string, required>
  apiUrl: https://api.opsgenie.com/v2/alerts
  # <string>
  message: |
    {{ template "default.title" . }}
  # <string>
  description: some descriptive description
  # <bool>
  autoClose: false
  # <bool>
  overridePriority: false
  # <string> options: tags, details, both
  sendTagsAs: both
{{< /collapse >}}
{{< collapse title="PagerDuty" >}}
PagerDuty
type: pagerduty
settings:
  # <string, required> the 32-character Events API key https://support.pagerduty.com/docs/api-access-keys#events-api-keys
  integrationKey: XXX
  # <string> options: critical, error, warning, info
  severity: critical
  # <string>
  class: ping failure
  # <string>
  component: Grafana
  # <string>
  group: app-stack
  # <string>
  summary: |
    {{ template "default.message" . }}
{{< /collapse >}}
{{< collapse title="Pushover" >}}
Pushover
type: pushover
settings:
  # <string, required>
  apiToken: XXX
  # <string, required>
  userKey: user1,user2
  # <string>
  device: device1,device2
  # <string> options (high to low): 2,1,0,-1,-2
  priority: '2'
  # <string>
  retry: '30'
  # <string>
  expire: '120'
  # <string> the number of seconds before a message expires and is deleted automatically. Examples: 10s, 5m30s, 8h.
  ttl:
  # <string>
  sound: siren
  # <string>
  okSound: magic
  # <string>
  message: |
    {{ template "default.message" . }}
{{< /collapse >}}
{{< collapse title="Slack" >}}
Slack
type: slack
settings:
  # <string, required>
  recipient: alerting-dev
  # <string, required>
  token: xxx
  # <string>
  username: grafana_bot
  # <string>
  icon_emoji: heart
  # <string>
  icon_url: https://icon_url
  # <string>
  mentionUsers: user_1,user_2
  # <string>
  mentionGroups: group_1,group_2
  # <string> options: here, channel
  mentionChannel: here
  # <string> Optionally provide a Slack incoming webhook URL for sending messages, in this case the token isn't necessary
  url: https://some_webhook_url
  # <string>
  endpointUrl: https://custom_url/api/chat.postMessage
  # <string>
  color: {{ if eq .Status "firing" }}#D63232{{ else }}#36a64f{{ end }}
  # <string>
  title: |
    {{ template "slack.default.title" . }}
  text: |
    {{ template "slack.default.text" . }}
{{< /collapse >}}
{{< collapse title="Sensu Go" >}}
Sensu Go
type: sensugo
settings:
  # <string, required>
  url: http://sensu-api.local:8080
  # <string, required>
  apikey: xxx
  # <string>
  entity: default
  # <string>
  check: default
  # <string>
  handler: some_handler
  # <string>
  namespace: default
  # <string>
  message: |
    {{ template "default.message" . }}
{{< /collapse >}}
{{< collapse title="Telegram" >}}
Telegram
type: telegram
settings:
  # <string, required>
  bottoken: xxx
  # <string, required>
  chatid: some_chat_id
  # <string>
  message: |
    {{ template "default.message" . }}
{{< /collapse >}}
{{< collapse title="Threema Gateway" >}}
Threema Gateway
type: threema
settings:
  # <string, required>
  api_secret: xxx
  # <string, required>
  gateway_id: A5K94S9
  # <string, required>
  recipient_id: A9R4KL4S
{{< /collapse >}}
{{< collapse title="VictorOps" >}}
VictorOps
type: victorops
settings:
  # <string, required>
  url: XXX
  # <string> options: CRITICAL, WARNING
  messageType: CRITICAL
{{< /collapse >}}
{{< collapse title="Webhook" >}}
Webhook
type: webhook
settings:
  # <string, required>
  url: https://endpoint_url
  # <string> options: POST, PUT
  httpMethod: POST
  # <string>
  username: abc
  # <string>
  password: abc123
  # <string>
  authorization_scheme: Bearer
  # <string>
  authorization_credentials: abc123
  # <string>
  maxAlerts: '10'
  # <map>
  tlsConfig:
    # <bool>
    insecureSkipVerify: false
    # <string>
    clientCertificate: certificate in PEM format
    # <string>
    clientKey: key in PEM format
    # <string>
    caCertificate: CA certificate in PEM format
  hmacConfig:
    #<string>
    secret: secret-key
    #<string>
    header: X-Grafana-Alerting-Signature
    #<string>
    timestampHeader: X-Grafana-Alerting-Signature-Timestamp
{{< /collapse >}}
{{< collapse title="WeCom" >}}
WeCom
type: wecom
settings:
  # <string, required>
  url: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxxx
  # <string>
  message: |
    {{ template "default.message" . }}
  # <string>
  title: |
    {{ template "default.title" . }}
{{< /collapse >}}
Import notification template groups
Create or delete notification template groups using provisioning files in your Grafana instance(s).
Find the notification template group in Grafana.
Export a template group by copying the template content and name.
Copy the contents into a YAML or JSON configuration file and add it to the 
provisioning/alerting directory of the Grafana instance you want to import the alerting resources to.
Example configuration files can be found below.
Restart your Grafana instance (or reload the provisioned files using the Admin API).
Here is an example of a configuration file for creating notification template groups.
# config file version
apiVersion: 1
# List of templates to import or update
templates:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the template group, must be unique
    name: my_first_template
    # <string, required> content of the template group
    template: |
      {{ define "my_first_template" }}
        Custom notification message
      {{ end }}
Here is an example of a configuration file for deleting notification template groups.
# config file version
apiVersion: 1
# List of alert rule UIDs that should be deleted
deleteTemplates:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the template group, must be unique
    name: my_first_template
Import notification policies
Create or reset the notification policy tree using provisioning files in your Grafana instance(s).
In Grafana, the entire notification policy tree is considered a single, large resource. Add new specific policies as sub-policies under the root policy. Since specific policies may depend on each other, you cannot provision subsets of the policy tree; the entire tree must be defined in a single place.
{{< docs/shared lookup="alerts/warning-provisioning-tree.md" source="grafana" version="<GRAFANA_VERSION>" >}}
Find the notification policy tree in Grafana.
Export and download a provisioning file for your notification policy tree.
Copy the contents into a YAML or JSON configuration file and add it to the 
provisioning/alerting directory of the Grafana instance you want to import the alerting resources to.
Example configuration files can be found below.
Restart your Grafana instance (or reload the provisioned files using the Admin API).
Here is an example of a configuration file for creating notification policies.
# config file version
apiVersion: 1
# List of notification policies
policies:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string> name of the contact point that should be used for this route
    receiver: grafana-default
    # <list> The labels by which incoming alerts are grouped together. For example,
    #        multiple alerts coming in for cluster=A and alertname=LatencyHigh would
    #        be batched into a single group.
    #
    #        To aggregate by all possible labels use the special value '...' as
    #        the sole label name, for example:
    #        group_by: ['...']
    #        This effectively disables aggregation entirely, passing through all
    #        alerts as-is. This is unlikely to be what you want, unless you have
    #        a very low alert volume or your upstream notification system performs
    #        its own grouping.
    group_by: ['...']
    # <list> a list of prometheus-like matchers that an alert rule has to fulfill to match the node (allowed chars
    #        [a-zA-Z_:])
    matchers:
      - alertname = Watchdog
      - service_id_X = serviceX
      - severity =~ "warning|critical"
    # <list> a list of grafana-like matchers that an alert rule has to fulfill to match the node
    object_matchers:
      - ['alertname', '=', 'CPUUsage']
      - ['service_id-X', '=', 'serviceX']
      - ['severity', '=~', 'warning|critical']
    # <list> Times when the route should be muted. These must match the name of a
    #        mute time interval.
    #        Additionally, the root node cannot have any mute times.
    #        When a route is muted it will not send any notifications, but
    #        otherwise acts normally (including ending the route-matching process
    #        if the `continue` option is not set)
    mute_time_intervals:
      - abc
    # <duration> How long to initially wait to send a notification for a group
    #            of alerts. Allows to collect more initial alerts for the same group.
    #            (Usually ~0s to few minutes), default = 30s
    group_wait: 30s
    # <duration> How long to wait before sending a notification about new alerts that
    #            are added to a group of alerts for which an initial notification has
    #            already been sent. (Usually ~5m or more), default = 5m
    group_interval: 5m
    # <duration>  How long to wait before sending a notification again if it has already
    #             been sent successfully for an alert. (Usually ~3h or more), default = 4h
    repeat_interval: 4h
    # <list> Zero or more child policies. The schema is the same as the root policy.
    # routes:
    #   # Another recursively nested policy...
    #   - receiver: another-receiver
    #     matchers:
    #       - ...
    #     ...
Here is an example of a configuration file for resetting the policy tree back to its default value:
# config file version
apiVersion: 1
# List of orgIds that should be reset to the default policy
resetPolicies:
  - 1
Import mute timings
Create or delete mute timings via provisioning files using provisioning files in your Grafana instance(s).
Find the mute timing in Grafana.
Export and download a provisioning file for your mute timing.
Copy the contents into a YAML or JSON configuration file and add it to the 
provisioning/alerting directory of the Grafana instance you want to import the alerting resources to.
Example configuration files can be found below.
Restart your Grafana instance (or reload the provisioned files using the Admin API).
Here is an example of a configuration file for creating mute timings.
# config file version
apiVersion: 1
# List of mute time intervals to import or update
muteTimes:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the mute time interval, must be unique
    name: mti_1
    # <list> time intervals that should trigger the muting
    #        refer to https://prometheus.io/docs/alerting/latest/configuration/#time_interval-0
    time_intervals:
      - times:
          - start_time: '06:00'
            end_time: '23:59'
        location: 'UTC'
        weekdays: ['monday:wednesday', 'saturday', 'sunday']
        months: ['1:3', 'may:august', 'december']
        years: ['2020:2022', '2030']
        days_of_month: ['1:5', '-3:-1']
Here is an example of a configuration file for deleting mute timings.
# config file version
apiVersion: 1
# List of mute time intervals that should be deleted
deleteMuteTimes:
  # <int> organization ID, default = 1
  - orgId: 1
    # <string, required> name of the mute time interval, must be unique
    name: mti_1
Template variable interpolation
Provisioning interpolates environment variables using the 
$variable syntax.
contactPoints:
  - orgId: 1
    name: My Contact Email Point
    receivers:
      - uid: 1
        type: email
        settings:
          addresses: $EMAIL
In this example, provisioning replaces 
$EMAIL with the value of the 
EMAIL environment variable or an empty string if it is not present. For more information, refer to Using environment variables in the Provision documentation.
In alerting resources, most properties support template variable interpolation, with a few exceptions:
Alert rule annotations: 
groups[].rules[].annotations
Alert rule time range: 
groups[].rules[].relativeTimeRange
Alert rule query model: 
groups[].rules[].data.model
Mute timings name: 
muteTimes[].name
Mute timings time intervals: 
muteTimes[].time_intervals[]
Notification template group name: 
templates[].name
Notification template group content: 
templates[].template
Note for properties that support interpolation, you may unexpectedly substitute template variables when not intended. To avoid this, you can escape the 
$variable with 
$$variable.
For example, when provisioning a 
subject property in a 
contactPoints.receivers.settings object that is meant to use the 
$labels variable.
subject: '{{ $labels }}' will interpolate, incorrectly defining the subject as 
subject: '{{ }}'.
subject: '{{ $$labels }}' will not interpolate, correctly defining the subject as 
subject: '{{ $labels }}'.
More examples
For more examples on the concept of this guide:
Try provisioning alerting resources in Grafana OSS with YAML files through a demo project using Docker Compose or Kubernetes deployments.
Review the distinct options about how Grafana provisions resources in the Provision Grafana documentation.
For Helm support, review the examples provisioning alerting resources in the Grafana Helm Chart documentation.

Use Terraform to provision alerting resources
Use Terraform’s Grafana Provider to manage your alerting resources and provision them into your Grafana system. Terraform provider support for Grafana Alerting makes it easy to create, manage, and maintain your entire Grafana Alerting stack as code.
This guide outlines the steps and references to provision alerting resources with Terraform. For a practical demo, you can clone and try this example using Grafana OSS and Docker Compose.
To create and manage your alerting resources using Terraform, you have to complete the following tasks.
Create an API key to configure the Terraform provider.
Create your alerting resources in Terraform format by
exporting configured alerting resources
or writing the Terraform Alerting schemas.
By default, you cannot edit provisioned resources. Enable 
disable_provenance in the Terraform resource to allow changes in the Grafana UI.
Run 
terraform apply to provision your alerting resources.
Before you begin, you should have available a Grafana instance and Terraform installed on your machine.
Create an API key and configure the Terraform provider
You can create a service account token to authenticate Terraform with Grafana. To create an API key for provisioning alerting resources, complete the following steps.
Create a new service account.
Assign the role or permission to access the Alerting provisioning API.
Create a new service account token.
Name and save the token for use in Terraform.
You can now move to the working directory for your Terraform configurations, and create a file named 
main.tf like:
terraform {
    required_providers {
        grafana = {
            source = "grafana/grafana"
            version = ">= 4.47.0"
        }
    }
}
provider "grafana" {
    url = <grafana-url>
    auth = <api-key>
}
Replace the following values:
<grafana-url> with the URL of the Grafana instance.
<api-key> with the API token previously created.
This Terraform configuration installs the Grafana Terraform provider and authenticates against your Grafana instance using an API token. For other authentication alternatives including basic authentication, refer to the 
auth option documentation.
For Grafana Cloud, refer to the instructions to manage a Grafana Cloud stack with Terraform. For role-based access control, refer to Provisioning RBAC with Terraform and the alerting provisioning roles (
fixed:alerting.provisioning.*).
Create Terraform configurations for alerting resources
Grafana Terraform provider enables you to manage the following alerting resources.
Alerting resource
Terraform resource
Alert rules
grafana_rule_group
Contact points
grafana_contact_point
Notification templates
grafana_message_template
Notification policy tree
grafana_notification_policy
Mute timings
grafana_mute_timing
In this section, we'll create Terraform configurations for each alerting resource and demonstrate how to link them together.
Add alert rules
Alert rules enable you to receive alerts by querying any backend Grafana data sources.
First, create a data source to query and a folder to store your rules in.
In this example, the TestData data source is used.
resource "grafana_data_source" "<terraform_data_source_name>" {
    name = "TestData"
    type = "testdata"
}
resource "grafana_folder" "<terraform_folder_name>" {
    title = "My Rule Folder"
}
Replace the following field values:
<terraform_data_source_name> with the terraform name of the data source.
<terraform_folder_name> with the terraform name of the folder.
Create or find an alert rule you want to import in Grafana.
Export the alert rule group in Terraform format. This exports the alert rule group as 
grafana_rule_group Terraform resource.
You can edit the exported resource, or alternatively, consider creating the resource from scratch.
resource "grafana_rule_group" "<terraform_rule_group_name>" {
    name = "My Alert Rules"
    folder_uid = grafana_folder.<terraform_folder_name>.uid
    interval_seconds = 60
    org_id = 1
    rule {
        name = "My Random Walk Alert"
        condition = "C"
        for = "0s"
        // Query the datasource.
        data {
            ref_id = "A"
            relative_time_range {
                from = 600
                to = 0
            }
            datasource_uid = grafana_data_source.<terraform_data_source_name>.uid
            // `model` is a JSON blob that sends datasource-specific data.
            // It's different for every datasource. The alert's query is defined here.
            model = jsonencode({
                intervalMs = 1000
                maxDataPoints = 43200
                refId = "A"
            })
        }
        // The query was configured to obtain data from the last 60 seconds. Let's alert on the average value of that series using a Reduce stage.
        data {
            datasource_uid = "__expr__"
            // You can also create a rule in the UI, then GET that rule to obtain the JSON.
            // This can be helpful when using more complex reduce expressions.
            model = <<EOT
{"conditions":[{"evaluator":{"params":[0,0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["A"]},"reducer":{"params":[],"type":"last"},"type":"avg"}],"datasource":{"name":"Expression","type":"__expr__","uid":"__expr__"},"expression":"A","hide":false,"intervalMs":1000,"maxDataPoints":43200,"reducer":"last","refId":"B","type":"reduce"}
EOT
            ref_id = "B"
            relative_time_range {
                from = 0
                to = 0
            }
        }
        // Now, let's use a math expression as our threshold.
        // We want to alert when the value of stage "B" above exceeds 70.
        data {
            datasource_uid = "__expr__"
            ref_id = "C"
            relative_time_range {
                from = 0
                to = 0
            }
            model = jsonencode({
                expression = "$B > 70"
                type = "math"
                refId = "C"
            })
        }
    }
}
Replace the following field values:
<terraform_rule_group_name> with the name of the alert rule group.
Note that the distinct Grafana resources are connected through 
uid values in their Terraform configurations. The 
uid value will be randomly generated when provisioning.
To link the alert rule group with its respective data source and folder in this example, replace the following field values:
<terraform_data_source_name> with the terraform name of the previously defined data source.
<terraform_folder_name> with the terraform name of the previously defined folder.
Continue to add more Grafana resources or use the Terraform CLI for provisioning.
Add contact points
Contact points are the receivers of alert notifications.
Create or find the contact points you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.
Export the contact point in Terraform format. This exports the contact point as 
grafana_contact_point Terraform resource—edit it if necessary.
In this example, notifications are muted on weekends.
 resource "grafana_contact_point" "<terraform_contact_point_name>" {
     name = "My contact point email"
     email {
         addresses               = ["<email_address>"]
     }
 }
Replace the following field values:
<terraform_contact_point_name> with the terraform name of the contact point. It will be used to reference the contact point in other Terraform resources.
<email_address> with the email to receive alert notifications.
Continue to add more Grafana resources or use the Terraform CLI for provisioning.
Add and enable notification templates
Notification templates allow customization of alert notifications across multiple contact points.
Create or find the notification template group you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.
Export the notification template group as 
grafana_message_template Terraform resource.
This example creates a notification template group named 
custom_emails that defines a 
custom_email.message template.
 resource "grafana_message_template" "<terraform_message_template_name>" {
     name = "custom_emails"
     template = <<EOT
 {{ define "custom_email.message" }}
 Lorem ipsum - Custom alert!
 {{ end }}
 EOT
 }
This enables contact points to use the notification templates (
{{ define "<NAME>"}}) within the notification template group.
In the previous contact point, enable the template by setting the 
email.message property as follows.
 resource "grafana_contact_point" "<terraform_contact_point_name>" {
     name = "My contact point email"
     email {
         addresses               = ["<email_address>"]
         message                 = "{{ template \"custom_email.message\" .}}"
     }
 }
Continue to add more Grafana resources or use the Terraform CLI for provisioning.
Add mute timings
Mute timings pause alert notifications during predetermined intervals.
Create or find the mute timings you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.
Export the mute timing in Terraform format. This exports the mute timing as 
grafana_mute_timing Terraform resource—edit it if necessary.
This example turns off notifications on weekends.
 resource "grafana_mute_timing" "<terraform_mute_timing_name>" {
     name = "No weekends"
     intervals {
         weekdays = ["saturday", "sunday"]
     }
 }
Replace the following field values:
<terraform_mute_timing_name> with the name of the Terraform resource. It will be used to reference the mute timing in the Terraform notification policy tree.
Continue to add more Grafana resources or use the Terraform CLI for provisioning.
Add the notification policy tree
Notification policies defines how to route alert instances to your contact points.
{{< docs/shared lookup="alerts/warning-provisioning-tree.md" source="grafana" version="<GRAFANA_VERSION>" >}}
Find the default notification policy tree. Alternatively, consider writing the resource in code as demonstrated in the example below.
Export the notification policy tree in Terraform format. This exports it as 
grafana_notification_policy Terraform resource—edit it if necessary.
resource "grafana_notification_policy" "my_policy_tree" {
contact_point = grafana_contact_point.<terraform_contact_point_name>.name
...
policy {
    contact_point = grafana_contact_point.<terraform_contact_point_name>.name
    matcher {...}
    mute_timings = [grafana_mute_timing.<terraform_mute_timing_name>.name]
}
}
To configure the mute timing and contact point previously created in the notification policy tree, replace the following field values:
<terraform_data_source_name> with the terraform name of the previously defined contact point.
<terraform_folder_name> with the terraform name of the previously defined mute timing.
Continue to add more Grafana resources or use the Terraform CLI for provisioning.
Enable multiple notification policy trees
By default, Grafana routes all alerts to the default notification policy tree using the 
grafana_notification_policy resource as shown above.
Multiple notification policy trees let you split routing logic into separate routing trees, for example, one per team. Each routing tree is provisioned using its own 
grafana_apps_notifications_routingtree_v1beta1 Terraform resource.
Create a routing tree for each scope.
resource "grafana_apps_notifications_routingtree_v1beta1" "team_platform" {
    metadata {
        uid = "platform-routing-tree"
    }
    spec {
         # ...
    }
}
metadata.uid sets the routing tree's unique identifier. Alert rules reference this value to route into this tree instead of the default policy tree.
For routing tree settings, refer to the 
grafana_apps_notifications_routingtree_v1beta1 Terraform resource.
To route notifications from an alert rule to a specific tree, set the 
notification_settings.policy option in the alert rule to the tree's 
metadata.uid.
resource "grafana_rule_group" "<terraform_rule_group_name>" {
     # ...
     rule {
         name = "My Random Walk Alert"
         # ...
         notification_settings {
             policy = grafana_apps_notifications_routingtree_v1beta1.team_platform.metadata.uid
         }
     }
     # ...
}
Alert rules that don't set 
notification_settings.policy continue to route through the default notification policy tree.
Continue to add more Grafana resources or use the Terraform CLI for provisioning.
Enable editing resources in the Grafana UI
By default, you cannot edit resources provisioned via Terraform in Grafana. This ensures that your alerting stack always stays in sync with your Terraform code.
To make provisioned resources editable in the Grafana UI, enable the 
disable_provenance attribute on alerting resources.
resource "grafana_contact_point" "my_contact_point" {
  name = "My Contact Point"
  disable_provenance = true
}
resource "grafana_message_template" "custom_notification_template_group" {
  name     = "custom_notification_template_group"
  template = "{{define \"template1\" }}Say{{ end }}{{define \"template2\" }}Hi!{{ end }}"
  disable_provenance = true
}
...
Provision Grafana resources with Terraform
To create the previous alerting resources in Grafana with the Terraform CLI, complete the following steps.
Initialize the working directory containing the Terraform configuration files.
terraform init
This command initializes the Terraform directory, installing the Grafana Terraform provider configured in the 
main.tf file.
Apply the Terraform configuration files to provision the resources.
terraform apply
Before applying any changes to Grafana, Terraform displays the execution plan and requests your approval.
 Plan: 4 to add, 0 to change, 0 to destroy.
 Do you want to perform these actions?
 Terraform will perform the actions described above.
 Only 'yes' will be accepted to approve.
 Enter a value:
Once you have confirmed to proceed with the changes, Terraform will create the provisioned resources in Grafana!
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
You can now access Grafana to verify the creation of the distinct resources.
More examples
For more examples on the concept of this guide:
Try the demo provisioning alerting resources in Grafana OSS using Terraform and Docker Compose.
Review all the available options and examples of the Terraform Alerting schemas in the Grafana Terraform Provider documentation.
Review the tutorial to manage a Grafana Cloud stack using Terraform.