Tutorials

Get started with Grafana Alerting - Create and receive your first alert

In this tutorial, we walk you through the process of setting up your first alert in just a few minutes. You'll witness your alert in action with real-time data, as well as sending alert notifications.

{{< youtube id="YVhKNbQNNss" >}}

In this tutorial you will:

{{< admonition type="tip" >}}

After you have completed Part 1, don’t forget to explore the advanced but essential alerting topics in Multi-dimensional alerts and how to route them.

{{< /admonition >}}

{{< docs/ignore >}}

After you have completed Part 1, don’t forget to explore the advanced but essential alerting topics in Multi-dimensional alerts and how to route them.

{{< /docs/ignore >}}

Before you begin

There are different ways you can follow along with this tutorial.

Set up the Grafana stack (OSS users)

To demonstrate the observation of data using the Grafana stack, download and run the following files.

  1. Clone the tutorial environment repository.

    git clone https://github.com/grafana/tutorial-environment.git

  2. Change to the directory where you cloned the repository:

    cd tutorial-environment

  3. Run the Grafana stack:

    docker compose up -d

    {{< docs/ignore >}}

    docker-compose up -d

    {{< /docs/ignore >}}

    The first time you run docker compose up -d, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

    {{< admonition type="note" >}} If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again. {{< /admonition >}}

    {{< docs/ignore >}}

    NOTE:

    If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

    {{< /docs/ignore >}}

Create a contact point

Besides being an open-source observability tool, Grafana has its own built-in alerting service. This means that you can receive notifications whenever there is an event of interest in your data, and even see these events graphed in your visualizations.

In this step, we set up a new contact point. This contact point uses the webhook integration. In order to make this work, we also need an endpoint for our webhook integration to receive the alert. We can use Webhook.site to quickly set up that test endpoint. This way we can make sure that our alert is actually sending a notification somewhere.

  1. In your browser, sign in to your Grafana Cloud account.

    OSS users: To log in, navigate to http://localhost:3000, where Grafana is running.

  2. In another tab, go to Webhook.site.

  3. Copy Your unique URL.

Your webhook endpoint is now waiting for the first request.

Next, let's configure a contact point in Grafana's Alerting UI to send notifications to our webhook endpoint.

  1. Return to Grafana. In Grafana's sidebar, hover over the Alerting (bell) icon and then click Contact points.

  2. Click + New contact point.

  3. In Name, write Webhook.

  4. In Integration, choose Webhook.

  5. In URL, paste the endpoint to your webhook endpoint.

  6. Click Test, and then click Send test notification to send a test alert to your webhook endpoint.

  7. Navigate back to Webhook.site. On the left side, there's now a POST / entry. Click it to see what information Grafana sent.

    {{< figure src="/media/docs/alerting/alerting-webhook-detail.png" max-width="1200px" caption="A POST entry in Webhook.site" >}}

  8. Return to Grafana and click Save contact point.

We have created a dummy Webhook endpoint and created a new Alerting contact point in Grafana. Now, we can create an alert rule and link it to this new integration.

Create an alert

Next, we establish an alert rule within Grafana Alerting to notify us whenever alert rules are triggered and resolved.

  1. In Grafana, navigate to Alerts & IRM > Alerting > Alert rules. Click on + New alert rule.

  2. Enter alert rule name for your alert rule. Make it short and descriptive as this appears in your alert notification. For instance, database-metrics

Define query and alert condition

In this section, we use the default options for Grafana-managed alert rule creation. The default options let us define the query, a expression (used to manipulate the data -- the WHEN field in the UI), and the condition that must be met for the alert to be triggered (in default mode is the threshold).

Grafana includes a test data source that creates simulated time series data. This data source is included in the demo environment for this tutorial. If you're working in Grafana Cloud or your own local Grafana instance, you can add the data source through the Connections menu.

  1. Select the TestData data source from the drop-down menu.

  2. In the Alert condition section:

    • Keep Random Walk as the Scenario.
    • Keep Last as the value for the reducer function (WHEN), and IS ABOVE 0 as the threshold value. This is the value above which the alert rule should trigger.

  3. Click Preview alert rule condition to run the query.

    It should return random time series data. The alert rule state should be Firing.

    {{< figure src="/media/docs/alerting/random-walk-firing-alert-rule.png" max-width="1200px" caption="A preview of a firing alert" >}}

Add folders and labels

  1. In Folder, click + New folder and enter a name. For example: metric-alerts . This folder contains our alert rules.

Set evaluation behavior

The alert rule evaluation defines the conditions under which an alert rule triggers, based on the following settings:

To set up the evaluation:

  1. In the Evaluation group and interval, enter a name. For example: 1m-evaluation.
  2. Choose an Evaluation interval (how often the alert are evaluated). For example, every 1m (1 minute).
  3. Set the pending period to, 0s (zero seconds), so the alert rule fires the moment the condition is met.
  4. Set Keep firing for to, 0s, so the alert stops firing immediately after the condition is no longer true. Use this when you want alerts to be resolved as soon as the system is healthy again.

Configure notifications

Choose the contact point where you want to receive your alert notifications.

  1. Under Contact point, select Webhook from the drop-down menu.
  2. Click Save rule and exit at the top right corner.

Trigger and resolve an alert

Now that the alert rule has been configured, you should receive alert notifications in the contact point whenever alerts trigger and get resolved.

Trigger an alert

Since the alert rule that you have created has been configured to always fire, once the evaluation interval has concluded, you should receive an alert notification in the Webhook endpoint.

{{< figure src="/media/docs/alerting/alerting-webhook-firing-alert.png" max-width="1200px" caption="Firing alert notification details" >}}

The alert notification details show that the alert rule state is Firing , and it includes the value that made the rule trigger by exceeding the threshold of the alert rule condition. The notification also includes links to see the alert rule details, and another link to add a Silence to it.

Resolve an alert

To see how a resolved alert notification looks like, you can modify the current alert rule threshold.

To edit the Alert rule:

  1. Navigate to Alerting > Alert rules.
  2. Click on the metric-alerts folder to display the alert that you created earlier
  3. Click the edit button on the right hand side of the screen
  4. Increment the Threshold expression to 1.
  5. Click Save rule and exit.

By incrementing the threshold, the condition is no longer met, and after the evaluation interval has concluded (1 minute approx.), you should receive an alert notification with status “Resolved”.

Learn more in Grafana Alerting: Multi-dimensional alerts and how to route them

{{< admonition type="tip" >}}

In Grafana Alerting: Multi-dimensional alerts and how to route them you'll learn to use alert instances and route notifications by labels to contacts

{{< /admonition >}}

{{< docs/ignore >}}

In Grafana Alerting: Multi-dimensional alerts and how to route them you'll learn to use alert instances and route notifications by labels to contacts.

{{< /docs/ignore >}}

Get started with Grafana Alerting - Multi-dimensional alerts and how to route them

This tutorial is a continuation of the Grafana Alerting - Create and receive your first alert tutorial.

In this guide, we dig into more complex yet equally fundamental elements of Grafana Alerting: alert instances and notification policies.

{{< youtube id="nI-_MEnFBQs" >}}

After introducing each component, you will learn how to:

Learning about alert instances and notification policies is useful if you have more than one contact point in your organization, or if your alert rule returns a number of metrics that you want to handle separately by routing each alert instance to a specific contact point. The tutorial will introduce each concept, followed by how to apply both concepts in a real-world scenario.

{{< docs/ignore >}}

Set up the Grafana stack

{{< /docs/ignore >}}

Before you begin

There are different ways you can follow along with this tutorial.

Set up the Grafana stack (OSS users)

To demonstrate the observation of data using the Grafana stack, download and run the following files.

  1. Clone the tutorial environment repository.

    git clone https://github.com/grafana/tutorial-environment.git

  2. Change to the directory where you cloned the repository:

    cd tutorial-environment

  3. Run the Grafana stack:

    docker compose up -d

    {{< docs/ignore >}}

    docker-compose up -d

    {{< /docs/ignore >}}

    The first time you run docker compose up -d, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

    {{< admonition type="note" >}} If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again. {{< /admonition >}}

    {{< docs/ignore >}}

    NOTE:

    If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

    {{< /docs/ignore >}}

Alert instances

An alert instance is an event that matches a metric returned by an alert rule query.

Let's consider a scenario where you're monitoring website traffic using Grafana. You've set up an alert rule to trigger an alert instance if the number of page views exceeds a certain threshold (more than 1000 page views) within a specific time period, say, over the past 5 minutes.

If the query returns more than one time-series, each time-series represents a different metric or aspect being monitored. In this case, the alert rule is applied individually to each time-series.

{{< figure alt="Screenshot displaying alert instances in the context of an alert rule, highlighting the specific alerts triggered by the rule and their respective statuses" src="/media/docs/alerting/alert-instance-flow.jpg" max-width="1200px" caption="Alert Instances in the Context of an Alert Rule" >}}

In this scenario, each time-series is evaluated independently against the alert rule. It results in the creation of an alert instance for each time-series. The time-series corresponding to the desktop page views meets the threshold and, therefore, results in an alert instance in Firing state for which an alert notification is sent. The mobile alert instance state remains Normal.

Notification policies

Notification policies route alerts to different communication channels, reducing alert noise and providing control over when and how alerts are sent. For example, you might use notification policies to ensure that critical alerts about server downtime are sent immediately to the on-call engineer. Another use case could be routing performance alerts to the development team for review and action.

Key Characteristics:

{{< figure alt="Screenshot illustrating the routing of alerts with notification policies, including the configuration and flow of alerts through different notification channels" src="/media/docs/alerting/get-started-notification-policy-tree-combo.png" max-width="1200px" caption="Routing of alerts with notification policies" >}}

In the above diagram, alert instances and notification policies are matched by labels. For instance, the label team=operations matches the alert instance “Pod stuck in CrashLoop” and “Disk Usage -80%” to child policies that send alert notifications to a particular contact point (operations@grafana.com).

Create notification policies

Create a notification policy if you want to handle metrics returned by alert rules separately by routing each alert instance to a specific contact point.

  1. In your browser, sign in to your Grafana Cloud account.

    OSS and interactive learning environment users: To log in, navigate to http://localhost:3000, where Grafana should be running.

  2. Navigate to Alerts & IRM > Alerting > Notification policies.

  3. In the Default policy, click + New child policy.

  4. In the field Label enter device, and in the field Value enter desktop.

  5. From the Contact point drop-down, choose Webhook. {{< admonition type="note" >}} If you don’t have any contact points, add a Contact point. {{</ admonition >}}

  6. Enable Continue matching:

    Turn on Continue matching subsequent sibling nodes so the evaluation continues even after one or more labels (i.e. device label) match.

  7. Click Save Policy.

    This new child policy routes alerts that match the label device=desktop to the Webhook contact point.

  8. Repeat the steps above to create a second child policy to match another alert instance. For labels use: device=mobile. Use the Webhook integration for the contact point. Alternatively, experiment by using a different Webhook endpoint or a different integration.

{{< docs/ignore >}}

  1. Visit http://localhost:3000, where Grafana should be running

  2. Navigate to Alerts & IRM > Alerting > Notification policies.

  3. In the Default policy, click + New child policy.

  4. In the field Label enter device, and in the field Value enter desktop.

  5. From the Contact point drop-down, choose Webhook.

    If you don’t have any contact points, add a Contact point.

  6. Click Save Policy.

    This new child policy routes alerts that match the label device=desktop to the Webhook contact point.

  7. Repeat the steps above to create a second child policy to match another alert instance. For labels use: device=mobile. Use the Webhook integration for the contact point. Alternatively, experiment by using a different Webhook endpoint or a different integration.

{{< /docs/ignore >}}

Create an alert rule that returns alert instances

The alert rule that you are about to create is meant to monitor web traffic page views. The objective is to explore what an alert instance is and how to leverage routing individual alert instances by using label matchers and notification policies.

Create an alert rule

  1. Navigate to Alerts & IRM > Alerting > Alert rules.
  2. Click + New alert rule.

Enter an alert rule name

Make it short and descriptive as this will appear in your alert notification. For instance, web-traffic.

Define query and alert condition

In this section, we use the default options for Grafana-managed alert rule creation. The default options let us define the query, a expression (used to manipulate the data -- the WHEN field in the UI), and the condition that must be met for the alert to be triggered (in default mode is the threshold).

Grafana includes a test data source that creates simulated time series data. This data source is included in the demo environment for this tutorial. If you're working in Grafana Cloud or your own local Grafana instance, you can add the data source through the Connections menu.

  1. Select TestData data source from the drop-down menu.

  2. From Scenario select CSV Content.

  3. Copy in the following CSV data:

    device,views
desktop,1200
mobile,900

    The above CSV data simulates a data source returning multiple time series, each leading to the creation of an alert instance for that specific time series. Note that the data returned matches the example in the Alert instance section.

  4. In the Alert condition section:

    • Keep Last as the value for the reducer function (WHEN), and IS ABOVE 1000 as the threshold value. This is the value above which the alert rule should trigger.

  5. Click Preview alert rule condition to run the queries.

It should return two series.desktop in Firing state, and mobile in Normal state. The values 1, and 0 mean that the condition is either true or false.

{{< figure alt="Screenshot showing a preview of a query in Grafana that returns two alert instances, including the query results and relevant alert details" src="/media/docs/alerting/firing-instances.png" max-width="1200px" caption="Preview of a query returning two alert instances in Grafana." >}}

Add folders and labels

  1. In Folder, click + New folder and enter a name. For example: web-traffic-alerts . This folder contains our alert rules.

Set evaluation behavior

In the life cycle of alert instances, when an alert condition (threshold) is not met, the alert instance state is Normal. Similarly, when the condition is breached (for longer than the pending period, which in this tutorial will be 0), the alert instance state switches back to Alerting, which means that the alert rule state is Firing, and a notification is sent.

To set up evaluation behavior:

  1. In the Evaluation group and interval, enter a name. For example:  1m-evaluation.
  2. Choose an Evaluation interval (how often the alert will be evaluated). Choose 1m.
  3. Set the pending period to 0s (zero seconds), so the alert rule fires the moment the condition is met.
  4. Set Keep firing for to, 0s, so the alert stops firing immediately after the condition is no longer true.

Configure notifications

In this section, you can select how you want to route your alert instances. Since we want to route by notification policy, we need to ensure that the labels match the alert instance.

  1. Toggle the Advanced options button to display matching Notification policies.

  2. Click Preview routing. Based on the existing labels, you should see a preview of what policies are matching with the alerts. There should be two alert instances matching the labels that were previously setup in each notification policy: device=desktop, device=mobile.

    These types of labels are generated by the data source query and they can be leveraged to match our notification policies without needing to manually add them to the alert rule.

    {{< figure alt="Screenshot showing a routing preview of matched notification policies, detailing how alerts are matched and routed to specific notification channels" src="/media/docs/alerting/get-started-alert-instace-routing-prev.png" max-width="1200px" caption="Routing preview of matched notification policies" >}}

    {{< admonition type="note" >}} Even if both labels match the policies, only the alert instance in Firing state produces an alert notification. {{</ admonition >}}

    {{< docs/ignore >}} Even if both labels match the policies, only the alert instance in Firing state produces an alert notification. {{< /docs/ignore >}}

  3. Click Save rule and exit.

Now that we have set up the alert rule, it’s time to check the alert notification.

Receive alert notifications

Now that the alert rule has been configured, you should receive alert notifications in the contact point whenever the alert triggers and gets resolved. In our example, each alert instance should be routed separately as we configured labels to match notification policies. Once the evaluation interval has concluded (1m), you should receive an alert notification in the Webhook endpoint.

{{< figure alt="Screenshot showing the exploration of alert notification details in a webhook endpoint, displaying the content and structure of the alert payload received by the endpoint" src="/media/docs/alerting/get-started-webhook-alert-isntance.png" max-width="1200px" caption="Exploring alert notification details in webhook endpoint" >}}

The alert notification details show that the alert instance corresponding to the website views from desktop devices was correctly routed through the notification policy to the Webhook contact point. The notification also shows that the instance is in Firing state, as well as it includes the label device=desktop, which makes the routing of the alert instance possible.

Feel free to change the CSV data in the alert rule to trigger the routing of the alert instance that matches the label device=mobile.

Summary

In this tutorial, you have learned how Grafana Alerting can route individual alert instances using the labels generated by the data-source query and match these labels with notification policies, which in turn routes alert notifications to specific contact points.

If you run into any problems, you are welcome to post questions in our Grafana Community forum.

Learn more in Grafana Alerting: Group alert notifications

{{< admonition type="tip" >}}

In Get started with Grafana Alerting: Group alert notifications you learn how to group alert notifications effectively.

{{< /admonition >}}

{{< docs/ignore >}}

In Get started with Grafana Alerting: Group alert notifications you learn how to group alert notifications effectively.

{{< /docs/ignore >}}

Get started with Grafana Alerting - Group alert notifications

This tutorial is a continuation of the Get started with Grafana Alerting - Alert routing tutorial.

{{< youtube id="WZ8gqKIQ5Oc" >}}

Grouping in Grafana Alerting reduces notification noise by combining related alert instances into a single, concise notification. This is useful for on-call engineers, ensuring they focus on resolving incidents instead of sorting through a flood of notifications.

Grouping is configured using labels in the notification policy. These labels reference those generated by alert instances or configured by the user.

Notification policies also allow you to define how often notifications are sent for each group of alert instances.

In this tutorial, you will:

{{< docs/ignore >}}

Set up the Grafana stack

{{< /docs/ignore >}}

Before you begin

There are different ways you can follow along with this tutorial.

Set up the Grafana stack (OSS users)

To demonstrate the observation of data using the Grafana stack, download and run the following files.

  1. Clone the tutorial environment repository.

    git clone https://github.com/grafana/tutorial-environment.git

  2. Change to the directory where you cloned the repository:

    cd tutorial-environment

  3. Run the Grafana stack:

    docker compose up -d

    {{< docs/ignore >}}

    docker-compose up -d

    {{< /docs/ignore >}}

    The first time you run docker compose up -d, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

    {{< admonition type="note" >}} If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again. {{< /admonition >}}

    {{< docs/ignore >}}

    NOTE:

    If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

    {{< /docs/ignore >}}

How alert rule grouping works

Alert notification grouping is configured with labels and timing options:

{{< figure src="/media/docs/alerting/alerting-notification-policy-diagram-with-labels-v3.png" max-width="750px" alt="A diagram about the components of a notification policy, including labels and groups" >}}

Types of Labels

Reserved labels (default):

User-configured labels:

Query labels:

Timing Options

Group wait: Time before sending the first notification. Group interval: Time between notifications for a group. Repeat interval: Time before resending notifications for an unchanged group.

Alerts sharing the same label values are grouped together, and timing options determine notification frequency.

For more details, see:

A real-world example of alert grouping in action

Scenario: monitoring a distributed application

You’re monitoring metrics like CPU usage, memory utilization, and network latency across multiple regions. Some of these alert rules include labels such as region: us-west and region: us-east. If multiple alert rules trigger across these regions, they can result in notification floods.

How to manage grouping

To group alert rule notifications:

  1. Define labels: Use region, metric, or instance labels to categorize alerts.
  2. Configure Notification policies:
    • Group alerts by the query label "region".
    • Example:
      • Alert notifications for region: us-west go to the West Coast team.
      • Alert notifications for region: us-east go to the East Coast team.
  3. Specify the timing options for sending notifications to control their frequency.
    • Example:
      • Group interval: setting determines how often updates for the same alert group are sent. By default, this interval is set to 5 minutes, but you can customize it to be shorter or longer based on your needs.

Setting up alert rule grouping

Notification Policy

Following the above example, notification policies are created to route alert instances, which have a region label, to a specific contact point. The goal is to receive one consolidated notification per region. To demonstrate how grouping works, alert notifications for the East Coast team are not grouped. Regarding timing, a specific schedule is defined for that region. This setup overrides the parent's settings to fine-tune the behavior for specific labels (i.e., regions).

  1. Sign in to Grafana:

  2. Navigate to Notification Policies:

    • Go to Alerts & IRM > Alerting > Notification Policies.

  3. Add a child policy:

    • In the Default policy, click + New child policy.

    • Label: region

    • Operator: =

    • Value: us-west

      This label matches alert rules where the region label is us-west.

  4. Choose a Contact point:

    • Select Webhook.

    If you don’t have any contact points, add a Contact point.

  5. Enable Continue matching:

    • Turn on Continue matching subsequent sibling nodes so the evaluation continues even after one or more labels (i.e. region label) match.

  6. Override grouping settings:

    • Toggle Override grouping.

    • Group by: Add region as label. Remove any existing labels.

      Group by consolidates alerts that share the same grouping label into a single notification. For example, all alerts with region=us-west will be combined into one notification, making it easier to manage and reducing alert fatigue.

  7. Set custom timing:

    • Toggle Override general timings.

    • Group interval: 2m. This ensures follow-up notifications for the same alert group will be sent at intervals of 2 minutes. While the default is 5 minutes, we chose 2 minutes here to provide faster feedback for demonstration purposes.

      Timing options control how often notifications are sent and can help balance timely alerting with minimizing noise.

  8. Save and repeat:

    • Repeat the steps above for region = us-east but without overriding grouping and timing options. Use a different webhook endpoint as the contact point.

    {{< figure src="/media/docs/alerting/notificaiton-policies-region.png" max-width="750px" alt="Two nested notification policies to route and group alert notifications" >}}

    These nested policies should route alert instances where the region label is either us-west or us-east. Only the us-west region team should receive grouped alert notifications.

    {{< admonition type="note" >}} Label matchers are combined using the AND logical operator. This means that all matchers must be satisfied for a rule to be linked to a policy. If you attempt to use the same label key (e.g., region) with different values (e.g., us-west and us-east), the condition will not match, because it is logically impossible for a single key to have multiple values simultaneously.

    However, region!=us-east && region=!us-west can match. For example, it would match a label set where region=eu-central. Alternatively, for identical label keys use regular expression matchers (e.g., region=~us-west|us-east). {{</ admonition >}}

{{< docs/ignore >}}

  1. Visit http://localhost:3000, where Grafana should be running

  2. Navigate to Alerts & IRM > Alerting > Notification policies.

  3. In the Default policy, click + New child policy.

    • In the Default policy, click + New child policy.
    • Label: region
    • Operator: =
    • Value: us-west

    This label matches alert rules where the region label is us-west

  4. Choose a Contact point:

    • Select Webhook.

    If you don’t have any contact points, add a Contact point.

  5. Enable Continue matching:

    • Turn on Continue matching subsequent sibling nodes so the evaluation continues even after one or more labels (i.e. region label) match.

  6. Override grouping settings:

    • Toggle Override grouping.

    • Group by: region.

      Group by consolidates alerts that share the same grouping label into a single notification. For example, all alerts with region=us-west will be combined into one notification, making it easier to manage and reducing alert fatigue.

  7. Set custom timing:

    • Toggle Override general timings.

    • Group interval: 2m. This ensures follow-up notifications for the same alert group will be sent at intervals of 2 minutes. While the default is 5 minutes, we chose 2 minutes here to provide faster feedback for demonstration purposes.

      Timing options control how often notifications are sent and can help balance timely alerting with minimizing noise.

  8. Save and repeat:

    • Repeat for region = us-east with a different webhook or a different contact point.

      Note: Label matchers are combined using the AND logical operator. This means that all matchers must be satisfied for a rule to be linked to a policy. If you attempt to use the same label key (e.g., region) with different values (e.g., us-west and us-east), the condition will not match, because it is logically impossible for a single key to have multiple values simultaneously.

      However, region!=us-east && region=!us-west can match. For example, it would match a label set where region=eu-central.Alternatively, for identical label keys use regular expression matchers (e.g., region=~us-west|us-east).

{{< /docs/ignore >}}

Create an alert rule

In this section we configure an alert rule based on our application monitoring example.

  1. Navigate to Alerts & IRM > Alerting > Alert rules.
  2. Click + New alert rule.

Enter an alert rule name

Make it short and descriptive as this appears in your alert notification. For instance, High CPU usage - Multi-region.

Define query and alert condition

In this section, we use the default options for Grafana-managed alert rule creation. The default options let us define the query, a expression (used to manipulate the data -- the WHEN field in the UI), and the condition that must be met for the alert to be triggered (in default mode is the threshold).

Grafana includes a test data source that creates simulated time series data. This data source is included in the demo environment for this tutorial. If you're working in Grafana Cloud or your own local Grafana instance, you can add the data source through the Connections menu.

  1. From the drop-down menu, select TestData data source.

  2. From Scenario select CSV Content.

  3. Copy in the following CSV data:

    • Select TestData as the data source.

    • Set Scenario to CSV Content.

    • Use the following CSV data:

      region,cpu-usage,service,instance
us-west,35,web-server-1,server-01
us-west,81,web-server-1,server-02
us-east,79,web-server-2,server-03
us-east,52,web-server-2,server-04
us-west,45,db-server-1,server-05
us-east,77,db-server-2,server-06
us-west,82,db-server-1,server-07
us-east,93,db-server-2,server-08

    The returned data simulates a data source returning multiple time series, each leading to the creation of an alert instance for that specific time series.

  4. In the Alert condition section:

    • Keep Last as the value for the reducer function (WHEN), and IS ABOVE 75 as the threshold value. This is the value above which the alert rule should trigger.

  5. Click Preview alert rule condition to run the queries.

    It should return 5 series in Firing state, two firing instances from the us-west region, and three from the us-east region.

    {{< figure src="/media/docs/alerting/regions-alert-instance-preview.png" max-width="750px" alt="Preview of a query returning alert instances." >}}

Add folders and labels

  1. In Folder, click + New folder and enter a name. For example: Multi-region alerts . This folder contains our alert rules.

Set evaluation behavior

Every alert rule is assigned to an evaluation group. You can assign the alert rule to an existing evaluation group or create a new one.

  1. In the Evaluation group and interval, enter a name. For example: Multi-region group.
  2. Choose an Evaluation interval (how often the alert are evaluated). Choose 1m.
  3. Set the pending period to 0s (zero seconds), so the alert rule fires the moment the condition is met (this minimizes the waiting time for the demonstration).
  4. Set Keep firing for to, 0s, so the alert stops firing immediately after the condition is no longer true.

Configure notifications

Select who should receive a notification when an alert rule fires.

  1. Select Use notification policy.

  2. Click Preview routing to ensure correct matching.

    {{< figure src="/media/docs/alerting/region-notification-policy-routing-preview.png" max-width="750px" alt="Preview of alert instance routing with the region label matcher" >}}

    The preview should show that the region label from our data source is successfully matching the notification policies that we created earlier thanks to the label matcher that we configured.

  3. Click Save rule and exit.

Create a second alert rule

Repeat the steps above to create a second alert rule that alerts on high memory usage.

  1. Duplicate the alert rule by clicking on More > Duplicate.

  2. Name it High Memory usage - Multi-region.

  3. Use the below CSV data to simulate a data source returning memory usage.

    region,memory-usage,service,instance
us-west,42,cache-server-1,server-09
us-west,88,cache-server-1,server-10
us-east,74,api-server-1,server-11
us-east,90,api-server-1,server-12
us-west,53,analytics-server-1,server-13
us-east,81,analytics-server-2,server-14
us-west,77,analytics-server-1,server-15
us-east,94,analytics-server-2,server-16

  4. Click Save rule and exit.

Receiving grouped alert notifications

Now that the alert rules have been configured, you should receive alert notifications in the contact point(s) whenever alerts trigger.

When the configured alert rule detects CPU or memory usage higher than 75% across multiple regions, it will evaluate the metric every minute. If the condition persists, notifications will be grouped together, with a Group wait of 30 seconds before the first alert is sent. Follow-up notifications for the same alert group will be sent at intervals of 2 minutes (US-west alert instances only), increasing the frequency of the grouped alert notifications. US-east instances follow-up notifications should be sent at the default interval of 5 minutes. If the condition continues for an extended period, a Repeat interval of 4 hours ensures that the alert is only resent if the issue persists.

As a result, our notification policies should route three notifications: one grouped notification grouping both CPU and memory alert instances from the us-west region and two separate notifications with alert instances from the us-east region.

Grouped notifications example:

{
  "receiver": "US-West-Alerts",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High CPU usage - Multi-region",
        "grafana_folder": "Multi-region alerts",
        "instance": "server-05",
...
  {
    "status": "firing",
      "labels": {
        "alertname": "High Memory usage - Multi-region",
        "grafana_folder": "Multi-region alerts",
        "instance": "server-10",
      },

...}

Detail of CPU and memory alert instances grouped into a single notification for us-west contact point.

{
  "receiver": "US-East-Alerts",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High CPU usage - Multi-region",
        "grafana_folder": "Multi-region alerts",
        "instance": "server-03",
        "region": "us-east",
        "service": "web-server-2"
...}}}

Detail of CPU alert instances grouped into a separate notification for us-east contact point.

{
  "receiver": "US-East-Alerts",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High memory usage - Multi-region",
        "grafana_folder": "Multi-region memory alerts",
        "instance": "server-12",
        "region": "us-east"
...}}}

Detail of memory alert instances grouped into a separate notification for us-east contact point.

Conclusion

By configuring notification policies and using labels (such as region), you can group alert notifications based on specific criteria and route them to the appropriate teams. Fine-tuning timing options—including group wait, group interval, and repeat interval—further can reduce noise and ensures notifications remain actionable without overwhelming on-call engineers.

Learn more in Grafana Alerting: Template your alert notifications

{{< admonition type="tip" >}}

In Get started with Grafana Alerting: Template your alert notifications you learn how to use templates to create customized and concise notifications.

{{< /admonition >}}

{{< docs/ignore >}}

In Get started with Grafana Alerting: Template your alert notifications you learn how to use templates to create customized and concise notifications.

{{< /docs/ignore >}}

Get started with Grafana Alerting - Template your alert notifications

This tutorial is a continuation of the Get started with Grafana Alerting - Grouping notifications tutorial.

{{< youtube id="9CSrQGKeZwQ" >}}

In this tutorial, you will learn:

{{< docs/ignore >}}

Set up the Grafana stack

{{< /docs/ignore >}}

Before you begin

There are different ways you can follow along with this tutorial.

Note: Some of the templating features in Grafana Alerting discussed in this tutorial are currently available in Grafana Cloud but have not yet been released to the Open Source (OSS) version.

Set up the Grafana stack (OSS users)

To demonstrate the observation of data using the Grafana stack, download and run the following files.

  1. Clone the tutorial environment repository.

    git clone https://github.com/grafana/tutorial-environment.git

  2. Change to the directory where you cloned the repository:

    cd tutorial-environment

  3. Run the Grafana stack:

    docker compose up -d

    {{< docs/ignore >}}

    docker-compose up -d

    {{< /docs/ignore >}}

    The first time you run docker compose up -d, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

    {{< admonition type="note" >}} If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again. {{< /admonition >}}

    {{< docs/ignore >}}

    NOTE:

    If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

    {{< /docs/ignore >}}

How templating works

In Grafana, you can use templates to dynamically pull in specific data about the alert rule. This results in more flexible and informative alert notification messages. You can template either alert rule labels and annotations, or notification templates. Both use the Go template language.

{{< figure src="/media/docs/alerting/how-notification-templates-works.png" max-width="1200px" caption="How templating works" >}}

Templating alert rule labels and annotations

Labels and annotations are key fields where templates are applied. One of the main advantages of using templating in annotations is the ability to incorporate dynamic data from queries, allowing alerts to reflect real-time information relevant to the triggered condition. By using templating in annotations, you can customize the content of each alert instance, such as including instance names and metric values, so the notification becomes more informative.

Notification templates

The real power of templating lies in how it helps you format notifications with dynamic alert data. Notification templates let you pull in details from annotations to create clear and consistent messages. They also make it simple to reuse the same format across different contact points, saving time and effort.

Notification templates allow you to customize how information is presented in each notification. For example, you can use templates to organize and format details about firing or resolved alerts, making it easier for recipients to understand the status of each alert at a glance—all within a single notification.

This particular notification template pulls in summary and description annotations for each alert instance and organizes them into separate sections, such as "firing" and "resolved." This way, instead of getting a long list of individual alert notifications, users can receive one well-structured message with all the relevant details grouped together.

This approach is helpful when you want to reduce notification noise, especially in situations where multiple instances of an alert are firing at the same time (e.g., high CPU usage across several instances). You can leverage templates to create a unified, easy-to-read notification that includes all the pertinent details.

Step 1: Template labels and annotations

Now that we've introduced how templating works, let’s move on to the next step. We guide you through creating an alert rule with a summary and description annotation. In doing so, we incorporate CPU usage and instance names, which we later use in our notification template.

Create an alert rule

  1. Sign in to Grafana:

  2. Create an alert rule that includes a summary and description annotation:

    • Navigate to Alerts & IRM > Alerting > Alert rules.
    • Click + New alert rule.
    • Enter an alert rule name. Name it High CPU usage

  3. Define query an alert condition section:

    • Select TestData data source from the drop-down menu.

      TestData is included in the demo environment. If you’re working in Grafana Cloud or your own local Grafana instance, you can add the data source through the Connections menu.

    • From Scenario select CSV Content.

    • Copy in the following CSV data:

      region,cpu-usage,service,instance
us-west,88,web-server-1,server-01
us-west,81,web-server-1,server-02
us-east,79,web-server-2,server-03
us-east,52,web-server-2,server-04

      This dataset simulates a data source returning multiple time series, with each time series generating a separate alert instance.

  4. Alert condition section:

    • Keep Last as the value for the reducer function (WHEN), and IS ABOVE 75 as the threshold value, representing CPU usage above 75% .This is the value above which the alert rule should trigger.
    • Click Preview alert rule condition to run the queries.

    It should return 3 series in Firing state, and 1 in Normal state.

    {{< figure src="/media/docs/alerting/part-4-firing-instances-preview.png" max-width="1200px" caption="Preview of a query returning alert instances" >}}

  5. Add folders and labels section:

    • In Folder, click + New folder and enter a name. For example: System metrics . This folder contains our alert rules.

      Note: while it's possible to template labels here, in this tutorial, we focus on templating the summary and annotations fields instead.

  6. Set evaluation behaviour section:

    • In the Evaluation group and interval, enter a name. For example: High usage.
    • Choose an Evaluation interval (how often the alert will be evaluated). Choose 1m.
    • Set the pending period to 0s (zero seconds), so the alert rule fires the moment the condition is met (this minimizes the waiting time for the demonstration.).
    • Set Keep firing for to, 0s, so the alert stops firing immediately after the condition is no longer true.

  7. Configure notifications section:

    Select who should receive a notification when an alert rule fires.

    • Select a Contact point. If you don’t have any contact points, click View or create contact points.

  8. Configure notification message section:

    In this step, you’ll configure the summary and description annotations to make your alert notifications informative and easy to understand. These annotations use templates to dynamically include key information about the alert.

    • Summary annotation: Enter the following code as the value for the annotation.:

      {{- "\n" -}}
Instance: {{ index $labels "instance" }}
{{- "\t" -}} Usage: {{ index $values "A"}}%{{- "\n" -}}

      This template automatically adds the instance name (from the $labels data) and its current CPU usage (from $values["A"]) into the alert summary. \t: Adds a tab space between the instance name and the value. And, \n: Inserts a new line after the value.

      Output example:

      server-01	88

      This output helps you quickly see which instance is affected and its usage level.

  9. Optional: Add a description to help the on-call engineer to better understand what the alert rule is about. Eg. This alert monitors CPU usage across instances and triggers if any instance exceeds a usage threshold of 75%.

  10. Click Save rule and exit.

Now that we’ve configured an alert rule with dynamic templates for the summary annotation, the next step is to customize the alert notifications themselves. While the default notification message includes the summary annotation and works well, it can often be too verbose.

{{< figure src="/media/docs/alerting/templated-annotation-alert.png" max-width="1200px" caption="Default email alert notification with templated annotation" >}}

To make our alert notifications more concise and tailored to our needs, we’ll create a custom notification template that references the summary annotation we just set up. Notification templates are especially useful because they can be reused across multiple contact points, ensuring consistent alert messages.

Step 2: Template notifications

In this step, we use a built-in notification template to format alert notifications in a clear and organized way. Notification templates allow us to customize the structure of alert messages, making them easier to read and more relevant.

Without a notification template, the alert messages would include the default Grafana formatting (default.message, see image above).

Adding a notification template:

  1. Navigate to Alerts & IRM > Alerting > Contact points.
  2. Select the Notification Templates tab.
  3. Click + New notification template.
  4. Enter a name. E.g instance-cpu-summary.
  5. From the Add example dropdown menu, choose Print firing and resolved alerts.

This template prints out alert instances into two sections: firing alerts and resolved alerts, and includes only the key details for each. In addition, it adds our summary and description annotations.

{{- /* Example displaying firing and resolved alerts separately in the notification. */ -}}
{{- /* Edit the template name and template content as needed. */ -}}
{{ define "custom.firing_and_resolved_alerts" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
  {{ template "alert.summary_and_description" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
  {{ template "alert.summary_and_description" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.summary_and_description" }}
  Summary: {{.Annotations.summary}}
  Status: {{ .Status }}
  Description: {{.Annotations.description}}
{{ end -}}

{{< docs/ignore >}} Note: Your notification template name ({{define "<NAME>"}}) must be unique. You cannot have two templates with the same name in the same notification template group or in different notification template groups. {{< /docs/ignore >}}

{{< admonition type="note" >}} Your notification template name ({{define "<NAME>"}}) must be unique. You cannot have two templates with the same name in the same notification template group or in different notification template groups. {{< /admonition >}}

Here’s a breakdown of the template:

In the Preview area, you can see a sample of how the notification would look. Since we’ve already created our alert rule, you can take it a step further by previewing how an actual alert instance from your rule would appear in the notification.

  1. Click Edit payload.

  2. Click Use existing alert instance.

    You should see our alert rule listed on the left.

  3. Click the alert rule.

  4. Select an instance.

  5. Click Add alert data to payload.

    The alert instance is added to the bottom of the preview.

    {{< figure src="/media/docs/alerting/alert-instance-preview-in-template.png" max-width="1200px" caption="Preview of an alert instance in a notification template" >}}

  6. Click Save.

With the notification template ready, the next step is to apply it to your contact point to see it in action.

Apply the template to your contact point

  1. Apply the template to your contact point.
    • Navigate to Alerts & IRM > Alerting > Contact points.
    • Edit your contact point.
  2. Optional [email] settings section:
    • Click Edit Message.
    • Under Select notification template, search custom.firing_and_resolved_alerts.
    • Click Save.
  3. Save your contact point.

Receiving notifications

Now that the template has been applied to the contact point, you should receive notifications in the specified contact point.

Note: you might need to pause the alert rule evaluation and resume it to trigger the notification.

{{< figure src="/media/docs/alerting/templated-notification-cpu.png" max-width="1200px" caption="Templated email notification for CPU and memory usage" >}}

In the screen capture, you can see how the notification template groups the alert instances into two sections: firing alerts and resolved alerts. Each section includes only the key details for each alert, ensuring the message remains concise and focused. Additionally, the summary and description annotations we created earlier are included, providing affected instance and CPU usage.

Conclusion

In this tutorial, we learned how to use templating in Grafana Alerting to create dynamic and actionable alert notifications. We explored how to configure alert rules with annotations, design custom notification templates, and apply them to contact points to enhance the clarity and efficiency of alert messages. By organizing alert instances into concise notifications, you can reduce noise and ensure on-call engineers quickly understand and address critical issues.

To deepen your understanding of Grafana’s templating, explore the following resources:

Learn more in Grafana Alerting: Route alerts using dynamic labels

{{< admonition type="tip" >}}

In Get started with Grafana Alerting: Route alerts using dynamic labels you learn how to dynamically route alerts and link them to dashboards.

{{< /admonition >}}

{{< docs/ignore >}}

In Get started with Grafana Alerting: Route alerts using dynamic labels you learn how to dynamically route alerts and link them to dashboards.

{{< /docs/ignore >}}

Get started with Grafana Alerting - Route alerts using dynamic labels

The Get started with Grafana Alerting - Dynamic routing tutorial is a continuation of the Get started with Grafana Alerting - Templating tutorial.

{{< youtube id="hSejnv1cdYY" >}}

Imagine you are managing a web application or a fleet of servers, tracking critical metrics such as CPU, memory, and disk usage. While monitoring is essential, managing alerts allows your team to act on issues without necessarily feeling overwhelmed by the noise.

In this tutorial you will learn how to:

Before you begin

Set up the Grafana stack

To observe data using the Grafana stack, download and run the following files.

  1. Clone the tutorial environment repository.

    git clone https://github.com/tonypowa/grafana-prometheus-alerting-demo.git

  2. Change to the directory where you cloned the repository:

    cd grafana-prometheus-alerting-demo

  3. Build the Grafana stack:

    docker compose build

    {{< docs/ignore >}}

    docker-compose build

    {{< /docs/ignore >}}

  4. Bring up the containers:

    docker compose up -d

    {{< docs/ignore >}}

    docker-compose up -d

    {{< /docs/ignore >}}

    The first time you run docker compose up -d, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

{{< admonition type="note" >}} If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again. {{< /admonition >}}

{{< docs/ignore >}}

NOTE:

If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

{{< /docs/ignore >}}

Use case: monitoring and alerting for system health with Prometheus and Grafana

In this use case, we focus on monitoring the system's CPU, memory, and disk usage as part of a monitoring setup. This example is based on the Grafana Prometheus Alerting Demo, which collects and visualizes system metrics via Prometheus and Grafana.

Your team is responsible for ensuring the health of your servers, and you want to leverage advanced alerting features in Grafana to:

Scenario

In the provided demo setup, you're monitoring:

You have a mixture of critical alerts (e.g., CPU usage over 75%) and warning alerts (e.g., memory usage over 60%).

This Flask-based Python script simulates a service that:

Objective

Use templates to dynamically populate a custom label that matches a notification policy, and therefore routes alerts to the correct contact point.

We'll automatically determine the environment associated with each firing alert by inspecting system metrics (e.g., CPU, memory) and extracting keywords using regular expressions with the Go templating language.

Step 1: Create Notification Policies

Notification policies route alert instances to contact points via label matchers. Since we know what labels our application returns (e.g., job, instance, deployment), we can use them to match alert rules and define appropriate notification routing.

Although our application doesn't explicitly include an environment label, we can rely on other labels like instance or deployment, which may contain keywords (like prod or staging) that indicate the environment.

  1. Sign in to Grafana:

  2. Navigate to Alerts & IRM > Alerting > Notification Policies.

  3. Add a child policy:

    • In the Default policy, click + New child policy.
    • Label: environment.
    • Operator: =.
    • Value: production.
    • This label matches alert rules where the environment label is prod.

  4. Choose a contact point:

    For a quick test, you can use a public webhook from webhook.site to capture and inspect alert notifications. If you choose this method, select Webhook from the drop-down menu in contact points.

  5. Enable continue matching:

    • Turn on Continue matching subsequent sibling nodes so the evaluation continues even after one or more labels (i.e., environment labels) match.

  6. Save and repeat

    • Create another child policy by following the same steps.
    • Use environment = staging as the label/value pair.
    • Feel free to use a different contact point.

Now that the labels are defined, we can create alert rules for CPU and memory metrics. These alert rules will use the labels from the collected and stored metrics in Prometheus.

Step 2: Create alert rules to monitor CPU and memory usage

Follow these steps to manually create alert rules and link them to a visualization.

Create an alert rule for CPU usage

  1. Navigate to Alerts & IRM > Alerting > Alert rules from the Grafana sidebar.
  2. Click + New alert rule rule to create a new alert.

Enter alert rule name

Make it short and descriptive, as this will appear in your alert notification. For instance, cpu-usage .

Define query and alert condition

  1. Select Prometheus data source from the drop-down menu.

  2. In the query section, enter the following query:

    ** switch to Code mode if not already selected **

    flask_app_cpu_usage{}

  3. Alert condition section:

    • Enter 75 as the value for WHEN QUERY IS ABOVE to set the threshold for the alert.
    • Click Preview alert rule condition to run the queries.

    {{< figure src="/media/docs/alerting/flask-app-metrics.png" max-width="1200px" caption="Preview of a query returning alert instances in Grafana." >}}

    Among the labels returned for flask_app_cpu_usage, the labels instance and deployment contain values that include the term prod and staging. We will create a template later to detect these keywords, so that any firing alert instances are routed to the relevant contact points (e.g., alerts-prod, alerts-staging).

Add folders and labels

In this section we add a templated label based on query value to map to the notification policies.

  1. In Folder, click + New folder and enter a name. For example: app-metrics . This folder contains our alerts.

  2. Click + Add labels.

  3. Key field: environment .

  4. In the value field copy in the following template:

    {{- $env := reReplaceAll ".*([pP]rod|[sS]taging|[dD]ev).*" "${1}" $labels.instance -}}
{{- if eq $env "prod" -}}
production
{{- else if eq $env "staging" -}}
staging
{{- else -}}
development
{{- end -}}

    This template uses a regular expression to extract prod, staging, or dev from the instance label ($labels.instance) and maps it to a more readable label (like "production" for "prod").

As result, when alerts exceed a threshold, the template checks the labels, such as instance="flask-prod:5000", instance="flask-staging:5000", or custom labels like deployment="prod-us-cs30", and assigns a value of production, staging or development to the custom environment environment label.

This label is then used by the alert notification policy to route alerts to the appropriate team, so that notifications are delivered efficiently, and reducing unnecessary noise.

Set evaluation behaviour

  1. Click + New evaluation group. Name it system-usage.
  2. Choose an Evaluation interval (how often the alert will be evaluated). Choose 1m.
  3. Set the pending period to 0s (zero seconds), so the alert rule fires the moment the condition is met (this minimizes the waiting time for the demonstration.).
  4. Set Keep firing for to, 0s, so the alert stops firing immediately after the condition is no longer true.

Configure notifications

Select who should receive a notification when an alert rule fires.

  1. Toggle the Advance options button.

  2. Click Preview routing. The preview should display which firing alerts are routed to contact points based on notification policies that match the environment label.

    {{< figure src="/media/docs/alerting/dynamic-routing-preview-prod-staging.png" max-width="1200px" caption="Notification policies matched by the environment label matcher." >}}

    The environment label matcher should map to the notification policies created earlier. This makes sure that firing alert instances are routed to the appropriate contact points associated with each policy.

Step 3: Create a second alert rule for memory usage

  1. Duplicate the existing alert rule (More > Duplicate), or create a new alert rule for memory usage, defining a threshold condition (e.g., memory usage exceeding 60%).
  2. Give it a name. For example: memory-usage
  3. Query: flask_app_memory_usage{}
  4. Link to the same visualization to obtain memory usage annotations whenever the alert rule triggers or resolves.

Now that the CPU and memory alert rules are set up, they are linked to the notification policies through the custom label matcher we added. The value of the label dynamically changes based on the environment template, using $labels.instance. This ensures that the label value will be set to production, staging, or development, depending on the environment.

Done! Your alerts are now dynamically routed

Based on your query's instance label values (which contain keywords like prod or staging ), Grafana dynamically assigns the value production, staging or development to the custom environment label using the template. This dynamic label then matches the label matchers in your notification policies, which route alerts to the correct contact points.

To see this in action go to Alerts & IRM > Alerting > Active notifications

This page shows grouped alerts that are currently triggering notifications. If you click on any alert group to view its label set, contact point, and number of alert instances. Notice that the environment label has been dynamically populated with values like production.

{{< figure src="/media/docs/alerting/routing-active-notification-detail.png" max-width="1200px" caption="Expanded alert in Active notifications section" >}}

Finally, you should receive notifications at the contact point associated with either prod or staging.

Feel free to experiment by changing the template to match other labels that contain any of the watched keywords. For example, you could reference:

$labels.deployment

The template should be flexible enough to capture the target keywords (e.g., prod, staging) by adjusting which label the$labels is referencing.

Conclusion

By using notification policies, you can route alerts based on query values, directing them to the appropriate teams.

Learn more in Grafana Alerting - Link alerts to visualizations

{{< admonition type="tip" >}}

In Grafana Alerting - Link alerts to visualizations you will create alerts using Prometheus data and link them to your graphs.

{{< /admonition >}}

{{< docs/ignore >}}

In Grafana Alerting - Link alerts to visualizations you will create alerts using Prometheus data and link them to your graphs.

{{< /docs/ignore >}}

Get started with Grafana Alerting - Link alerts to visualizations

This tutorial is a continuation of the Get started with Grafana Alerting - Route alerts using dynamic labels tutorial.

{{< youtube id="mqj_hN24zLU" >}}

In this tutorial you will learn how to:

Before you begin

Set up the Grafana stack

To observe data using the Grafana stack, download and run the following files.

  1. Clone the tutorial environment repository.

    git clone https://github.com/tonypowa/grafana-prometheus-alerting-demo.git

  2. Change to the directory where you cloned the repository:

    cd grafana-prometheus-alerting-demo

  3. Build the Grafana stack:

    docker compose build

    {{< docs/ignore >}}

    docker-compose build

    {{< /docs/ignore >}}

  4. Bring up the containers:

    docker compose up -d

    {{< docs/ignore >}}

    docker-compose up -d

    {{< /docs/ignore >}}

    The first time you run docker compose up -d, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

{{< admonition type="note" >}} If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again. {{< /admonition >}}

{{< docs/ignore >}}

NOTE:

If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

{{< /docs/ignore >}}

Use case: monitoring and alerting for system health with Prometheus and Grafana

In this use case, we focus on monitoring the system's CPU, memory, and disk usage as part of a monitoring setup. The demo app, launches a stack that includes a Python script to simulate metrics, which Grafana collects and visualizes as a time-series visualization.

The script simulates random CPU and memory usage values (10% to 100%) every 10 seconds and exposes them as Prometheus metrics.

Objective

You'll build a time series visualization to monitor CPU and memory usage, define alert rules with threshold-based conditions, and link those alerts to your dashboards to display real-time annotations when thresholds are breached.

Step 1: Create a visualization to monitor metrics

To keep track of these metrics you can set up a visualization for CPU usage and memory consumption. This will make it easier to see how the system is performing.

The time-series visualization supports alert rules to provide more context in the form of annotations and alert rule state. Follow these steps to create a visualization to monitor the application’s metrics.

  1. Log in to Grafana:

  2. Create a time series panel:

    • Navigate to Dashboards.
    • Click + Create dashboard.
    • Click + Add visualization.
    • Select Prometheus as the data source (provided with the demo).
    • Enter a title for your panel, e.g., CPU and Memory Usage.

  3. Add queries for metrics:

    • In the query area, copy and paste the following PromQL query:

      ** switch to Code mode if not already selected **

      flask_app_cpu_usage{instance="flask-prod:5000"}

    • Click Run queries.

    This query should display the simulated CPU usage data for the prod environment.

  4. Add memory usage query:

    • Click + Add query.

    • In the query area, paste the following PromQL query:

      flask_app_memory_usage{instance="flask-prod:5000"}

    {{< figure src="/media/docs/alerting/cpu-mem-dash.png" max-width="1200px" caption="Time-series panel displaying CPU and memory usage metrics in production." >}}

  5. Click Save dashboard. Name it: cpu-and-memory-metrics.

We have our time-series panel ready. Feel free to combine metrics with labels such as flask_app_cpu_usage{instance=“flask-staging:5000”}, or other labels like deployment.

Step 2: Create alert rules

Follow these steps to manually create alert rules and link them to a visualization.

Create an alert rule for CPU usage

  1. Navigate to Alerts & IRM > Alerting > Alert rules from the Grafana sidebar.
  2. Click + New alert rule rule to create a new alert.

Enter alert rule name

Make it short and descriptive, as this will appear in your alert notification. For instance, cpu-usage .

Define query and alert condition

  1. Select Prometheus data source from the drop-down menu.

  2. In the query section, enter the following query:

    ** switch to Code mode if not already selected **

    flask_app_cpu_usage{instance="flask-prod:5000"}

  3. Alert condition

    • Enter 75 as the value for WHEN QUERY IS ABOVE to set the threshold for the alert.

    • Click Preview alert rule condition to run the queries.

      {{< figure src="/media/docs/alerting/alert-condition-details-prod.png" max-width="1200px" caption="Preview of a query returning alert instances in Grafana." >}}

    The query returns the CPU usage of the Flask application in the production environment. In this case, the usage is 86.01%, which exceeds the configured threshold of 75%, causing the alert to fire.

Add folders and labels

  1. In Folder, click + New folder and enter a name. For example: system-metrics . This folder contains our alert rules.

Set evaluation behaviour

  1. Click + New evaluation group. Name it system-usage.
  2. Choose an Evaluation interval (how often the alert will be evaluated). Choose 1m.
  3. Set the pending period to 0s (None), so the alert rule fires the moment the condition is met (this minimizes the waiting time for the demonstration.).
  4. Set Keep firing for to, 0s, so the alert stops firing immediately after the condition is no longer true.

Configure notifications

Configure notification message

You have successfully linked this alert rule to your visualization!

When the CPU usage exceeds the defined threshold, an annotation should appear on the graph to mark the event. Similarly, when the alert is resolved, another annotation is added to indicate the moment it returned to normal.

Try adding a second alert rule using the memory usage metric (flask_app_memory_usage{instance="flask-prod:5000"}) to see how combining multiple alerts can enhance your dashboard.

Check how your dashboard looks now that your alert has been linked to your dashboard panel.

Step 3: Visualizing metrics and alert annotations

After the alert rules are linked to visualization, they should appear as health indicators (colored heart icons: a red heart when the alert is in Alerting state, and a green heart when in Normal state) on the linked panel. In addition, annotations provide helpful context, such as the time the alert was triggered.

{{< figure src="/media/docs/alerting/alert-in-panel.png" max-width="1200px" caption="Time series panel displaying health indicators and annotations." >}}

Step 4: Receiving notifications

Finally, as part of the alerting process, you should receive notifications at the associated contact point. If you're receiving alerts via email, the default email template will include two buttons:

{{< figure src="/media/docs/alerting/email-notification-w-url.png" max-width="1200px" caption="Alert notification with links to panel and dashboard." >}}

Clicking either button opens Grafana with a pre-applied time range relevant to the alert.

By default, this URL includes from and to query parameters that reflect the time window around the alert event (one hour before and after the alert). This helps you land directly in the time window where the alert occurred, making it easier to analyze what happened.

If you want to define a more intentional time range, you can customize your notifications using a notification template. With a template, you can explicitly set from and to values for more precise control over what users see when they follow the dashboard link. The final URL is constructed using a custom annotation (e.g., MyDashboardURL) along with the from and to parameters, which are calculated in the notification template.

Conclusion

You’ve now linked Prometheus-based alert rules to your Grafana visualizations, giving your dashboards real-time context with alert annotations and health indicators. By visualizing alerts alongside metrics, responders can quickly understand what’s happening and when. You also saw how alert notifications can include direct links to the affected dashboard or panel, helping teams jump straight into the right time window for faster troubleshooting.

Have feedback or ideas to improve this tutorial? Let us know.

How to create Grafana alerts with InfluxDB and the Flux query language

Grafana Alerting represents a powerful new approach to systems observability and incident response management. While the alerting platform is perhaps best known for its strong integrations with Prometheus, the system works with numerous popular data sources including InfluxDB. In this tutorial we will learn how to create Grafana alerts using InfluxDB and the newer Flux query language. We will cover five common scenarios from the most basic to the most complex. Together, these five scenarios will provide an excellent guide for almost any type of alerting query that you wish to create using Grafana and Flux.

Before we dive into our alerting scenarios, it is worth considering the development of InfluxDB's two popular query languages: InfluxQL and Flux. Originally, InfluxDB used InfluxQL as their query language, which uses a SQL-like syntax. But beginning with InfluxDB v1.8, the company introduced Flux, "an open source functional data scripting language designed for querying, analyzing, and acting on data." "Flux," its official documentation goes on to state, "unifies code for querying, processing, writing, and acting on data into a single syntax. The language is designed to be usable, readable, flexible, composable, testable, contributable, and shareable."

In the following five examples we will see just how powerful and flexible the new Flux query language can be. We will also see just how well Flux pairs with Grafana Alerting.

Example 1: Create an alert when a value is above or below a set threshold

Our first example uses a common real-world scenario for InfluxDB and Grafana Alerting. Popular with IoT and edge applications, InfluxDB excels at on-site, real-time observability. In this example, and in fact for many of the following examples, we will consider the hypothetical scenario where we are monitoring a number of fluid tanks in a manufacturing plant. This scenario, based on an actual application of InfluxDB and Alerting, will allow us to work through Grafana's various alerting setups, progressing from the simplest to the most complex.

For Example 1, let's consider the following scenario: we are monitoring one tank, A5, for which we are storing real-time temperature data. We need to make sure that the temperature in this tank is always greater than 30 °C and less than 60 °C.

We want to write a Grafana alert that will trigger whenever the temperature in tank A5 crosses the lower threshold of 30 °C or the upper threshold of 60 °C.

To do this, we'll: create a Grafana alert rule, add a Flux query, and then add expressions to the alert rule.

Create a Grafana Alert rule

  1. Open the Grafana alerting menu and select Alert rules.
  2. Click New alert rule.
  3. Give your alert rule a name and then select Grafana managed alert. For InfluxDB, you will always create a Grafana managed rule.

Add an initial Flux query to the alert rule

Still in the Step 2 section of the Alert rule page, you will see three boxes: a query editor (A), and then two sections labelled B and C. You will use these three sections to construct your rule. Let's move through them one by one.

First, we want to query the data in our imaginary InfluxDB instance to obtain a time series graph of the temperature of tank A5. For this you would choose your InfluxDB data source from the dropdown and then write a query like this:

```
 from(bucket: "RetroEncabulator")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "TemperatureData")
|> filter(fn: (r) => r["Tank"] == "A5")
|> filter(fn: (r) => r["_field"] == "Temperature")
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: "mean")
```

This is a fairly typical Flux query. Let's go through it function by function. We begin using the from() function to choose the correct bucket where our tank data resides. Then we use a range() function to filter our rows based on time constraints. Then we pass our data through three filter() functions to narrow our results. We choose a specific measurement (a special keyword in InfluxDB), then our tank in question (A5), and then a specific field (another special keyword in InfluxDB). After this we pass the data into an aggregateWindow() function, which downsamples our data into specific periods of time, and then finally a yield() function, which specifies which final result we want: mean.

This Flux query will yield a time-series graph like this:

grafana alerts from flux queries

Add expressions to your Grafana Alert rule

With data now appearing in our rule setup, our next step is to create an expression. Move to section B. For this scenario, we want to create a Reduce expression that will reduce the above to a single value. In this image, you can see that we have chosen to reduce our time-series data the Last value from input A. In this case, it returns a value 53 degrees celsius for Tank A5:

grafana alerts from flux queries

Finally, we need to create a math expression that Grafana will alert on. In our case we will write an expression with two conditions separated by the OR || operator. We want to trigger an alert any time our result in section B is less than 30 or more than 60. This looks like $B < 30 || $B > 60:

grafana alerts from flux queries

Set the alert condition to C - expression. We can now preview our alert. Here is a preview of this alert when the state is Normal:

grafana alerts from flux queries

And here is a preview of this alert when the state is Alerting:

grafana alerts from flux queries

Note that the Reduce expression above is needed. Without it, when previewing the results, Grafana would display invalid format of evaluation results for the alert definition B: looks like time series data, only reduced data can be alerted on.

💡Tip: In case your locale is still stubbornly using Fahrenheit, we can modify the above Flux query by adding (before the aggregateWindow statement) a map() function to to convert (or map) the values from °C to °F. Note that we are not creating a new field. We are simply remapping the existing value.

|> map(fn: (r) => ({r with _value: r._value * 1.8 + 32.0}))

Conclusion

Using these three steps you can create a Flux-based Grafana Alert that will trigger on either of two thresholds from a single data source. But what if you need to trigger an alert based on multiple conditions and from multiple time-series? In example two we will cover this very scenario.

Example 2: how to create a Grafana alert from two queries and two conditions

Let's mix things up a bit for example two and leave our imaginary manufacturing plant. Imagine you're an assistant to the great Dr. Emmett Brown from Back to the Future, and Doc has tasked you with the following challenge: "I want an alert sent to me every time both conditions for time travel are met: when the velocity of a vehicle reaches 88 miles per hour and an object generates 1.21 jigowatts of electricity."

Let's assume we are tracking this data in InfluxDB and Grafana. Let's also assume that each of the above data sources comes from different buckets. How do we alert on this? How do we use Grafana and Flux to alert on two distinct conditions originating from two distinct data sources?

Add two Flux queries to your Grafana Alert rule

Like we did in example 1, let's first mock up our queries. Our query for our vehicle data is very similar to our last query. We use a from(), range(), and a sequence of filter() functions. We then use AggregateWindow() and yield() to narrow our data even more. In this case, the result is a time series tracking the velocity of our 1983 DeLorean:

from(bucket: "vehicles")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "VehicleData")
|> filter(fn: (r) => r["VehicleType"] == "DeLorean")
|> filter(fn: (r) => r["VehicleYear"] == "1983")
|> filter(fn: (r) => r["_field"] == "velocity")
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: "mean")

Our second query will trigger an alert whenever our electricity resource (the lightning strike on the Hill Valley clocktower) reaches the needed 1.21 jigowatts. A query like this would look very similar to our vehicle velocity query:

from(bucket: "HillValley")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "ElectricityData")
|> filter(fn: (r) => r["Location"] == "clocktower")
|> filter(fn: (r) => r["Source"] == "lightning")
|> filter(fn: (r) => r["_field"] == "power")
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: "mean")

We are now ready to modify this data using expressions.

Add expressions to your Grafana Alert rule

  1. Let's now use the same steps to reduce each query to the last (most recent) value. Reducing Query A to a single value might look like this:

    grafana alerts from flux queries

  2. And here we are reducing query B:

    grafana alerts from flux queries

  3. Now, in section C we need to create a math expression to be alerted on. In this case we will use the AND && operator to specify that two conditions must be met: the value of C (the reduced value from query A) must be greater than 88.0 while the value of D (the reduced value from query B) must be greater than 1.21. We write this as $C > 88.0 && $D > 1.21

    grafana alerts from flux queries

And here is a preview of our alerts:

grafana alerts from flux queries

💡Tip: If your data in InfluxDB happens to have an unnecessarily large number of digits to the right of the decimal (such as 1.2104705741732575 shown above), and you want your Grafana alerts to be more legible, try using {{ printf "%.2f" $values.D.Value }}. For example, in the annotation Summary, we could write the following:

{{  $values.D.Labels.Source }} at the {{  $values.D.Labels.Location }} has generated {{ printf "%.2f"  $values.D.Value }} jigowatts.`

This will display as follows: grafana alerts from flux queries)

You can reference our documentation on alert message templating to learn more about this powerful feature.

Conclusion

In this example we showed how to create a Flux-based alert that uses two distinct conditions from two distinct queries that use data from two distinct data sources. For example three we will switch gears and tackle another popular alerting scenario: how to create an alert based on an aggregated (per day) value.

Example 3: how to create a Grafana Alert based on an aggregated (per-day) value

One of the most common requests in Grafana's community forum involves graphing daily electrical consumption and production. This sort of data is very often stored in InfluxDB. In this example we will see how to aggregate time series data into a per-day value and then alert on it.

Let’s assume our electricity meter sends a reading to InfluxDB once per hour and contains the total kWh used for that hour. We want to write a query that will aggregate these per-hour values into a per-day value, then create an alert that triggers when the power consumption (kWh) exceeds 5,000 kWh per day.

Add an initial Flux query to your Grafana Alert rule

  1. Let's begin by examining a typical query and the resulting time graph for our hourly data across a 7-day period. A query like this is shown below:

    from(bucket: "RetroEncabulator")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "ElectricityData")
|> filter(fn: (r) => r["Location"] == "PlantD5")
|> filter(fn: (r) => r["_field"] == "power_consumed")
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: "power")

    We can see the same pattern of Flux functions here that we say in examples 1 and 2. A query like this would produce a graph similar to the following:

    grafana alerts from flux queries

  2. Now let's adjust our query to calculate daily usage. With many datasources, this can be a rather complex operation. But with Flux, by simply changing the aggregateWindow function parameters we can calculate the daily usage over the same 7-day period:

    from(bucket: "RetroEncabulator")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "ElectricityData")
  |> filter(fn: (r) => r["Location"] == "PlantD5")
  |> filter(fn: (r) => r["_field"] == "power_consumed")
  |> aggregateWindow(every: 1d, fn: sum)
  |> yield(name: "power")

    Note how we've adjusted our aggregateWindow() function to aggregateWindow(every: 1d, fn: sum). This results in a graph like so:

    grafana alerts from flux queries

  3. Add expressions to your Grafana Alert rule.

    Now that we have our per-day query correct, we can continue using the same pattern as before, adding expressions to reduce and perform math on our results.

    As before, let's reduce our query to a single value:

    grafana alerts from flux queries

    Now create a math expression to be alerted on and set the evaluation behavior. In this case we want to write $B > 5000:

    grafana alerts from flux queries

    And now we are alerting on our daily electricity consumption whenever we exceed 5000 kWh. Here is preview of our alert:

    grafana alerts from flux queries

Conclusion

Plotting and aggregating electrical consumption is a common use case for combining InfluxDB and Grafana. Using Flux, we saw just how easy it can be to group our data by day and then alert on that daily value. In our next two examples we will examine the more complex form of Grafana Alert: multidimensional alerts.

Example 4: create a dynamic (multidimensional) Grafana Alert using Flux

Let’s return to our fluid tanks from example 1, but this time let’s assume we have 5 tanks (A5, B4, C3, D2, and E1). We are now tracking the temperature in five tanks: A5, B4, C3, D2, and E1.

We want to create one multidimensional alert that will notify us whenever the temperature in any tank is less than 30 °C or greater than 60 °C.

Add an initial Flux query to your Grafana Alert rule

We begin, as always, by writing our initial query. This is very similar to our query in example 1, but note how our third filter() function captures the data from all five tanks and not just A5:

from(bucket: "HyperEncabulator")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "TemperatureData")
|> filter(fn: (r) => r["MeasType"] == "actual")
|> filter(fn: (r) => r["Tank"] == "A5" or r["Tank"] == "B4" or r["Tank"] == "C3" or r["Tank"] == "D2" or r["Tank"] == "E1")
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: "mean")

💡Tip: If the tanks were shut down every night from 23:00 to 07:00, they would possibly fall below the 30 °C threshold. If one did not want to receive alerts during those hours, one can use the Flux function hourSelection() which filters rows by time values in a specified hour range.

|> hourSelection(start: 7, stop: 23)`

A query like the one above will produce a time series graph like this:

grafana alerts from flux queries

Add expressions to your Grafana Alert rule

  1. We create a Reduce expression that will reduce the time series for each tank to a single value. This gives us five distinct temperatures:

    grafana alerts from flux queries)

  2. Create a math expression to be alerted on. This is the exact same expression from example 1, $B < 30 || $B > 60:

    grafana alerts from flux queries

As we can see three tanks are within the acceptable thresholds while two tanks have crossed the upper boundary. This would trigger an alert for tanks D2 and E1.

Conclusion

With multidimensional alerts we can avoid repeating ourselves. But what if the scenario were even more complex? In the next and final example, we will examine how to use multidimensional alerts to create the most dynamic alerts possible.

Example 5: how to create a dynamic (multidimensional) Grafana Alert using multiple queries and multiple thresholds with Flux

For this final example let's continue with our five fluid tanks and their five datasets.Let’s assume again that each tank has a temperature controller with a setpoint value that is stored in InfluxDB. Let’s mix things up and assume that each tank has a different setpoint, where we always need to be within 3 degrees of the setpoint.

We want to create one multidimensional alert that will cover each unique scenario for each tank, triggering an alert whenever any tank's temperature moves beyond its unique allowable range.

To better visualize this challenge, here is a table representing our five tanks, their temperature setpoints, and their allowable range:

Tank Setpoint Allowable Range (±3)
A5 45 42 to 48
B4 55 52 to 58
C3 60 57 to 63
D2 72 69 to 75
E1 80 77 to 83

With Grafana Alerting, we can create a single multidimensional rule to cover all 5 tanks, and we can use Flux to compare the setpoint and actual value for each tank. In other words, one multidimensional alert can monitor 5 separate tanks, each with different setpoints and actual values, but all with one common "allowable threshold" (i.e. a temperature difference of ±3 degrees).

Add an initial Flux query to your Grafana Alert rule

Let's begin with our data query. It is similar to our past queries, only now more complex. We must add extra functions to get our data into the proper format, including a pivot(), map(), rename(), keep(), and drop() function:

from(bucket: "HyperEncabulator")
 |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
 |> filter(fn: (r) => r["_measurement"] == "TemperatureData")
 |> filter(fn: (r) => r["MeasType"] == "actual" or r["MeasType"] == "setpoint")
 |> filter(fn: (r) => r["Tank"] == "A5" or r["Tank"] == "B4" or r["Tank"] == "C3" or r["Tank"] == "D2" or r["Tank"] == "E1")
 |> filter(fn: (r) => r["_field"] == "Temperature")
 |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
 |> pivot(rowKey:["_time"], columnKey: ["MeasType"], valueColumn: "_value")
 |> map(fn: (r) => ({ r with _value: (r.setpoint - r.actual)}))
 |> rename(columns: {_value: "difference"})
 |> keep(columns: ["_time", "difference", "Tank"])
 |> drop(columns: ["actual", "setpoint"])
 |> yield(name: "mean")

Note in the above that we are calculating the difference between the actual and the setpoint. The way Grafana parses the result from InfluxDB is that if a _value column is found, it is assumed to be a time-series. The quick workaround is to add the following rename() function:

 |> rename(columns: {_value: "something"})

The above query results in this time series:

grafana alerts from flux queries

Add expressions to your Grafana Alert rule

  1. Again, we create a Reduce expression for the above query to reduce each of the above to a single value. This value represents the temperature differential between each tank's setpoint and its actual real-time temperature:

    grafana alerts from flux queries

  2. Now we create a math expression to be alerted on. This time we will create a condition that checks if the absolute value of our reduce calculation is greater than 3, abs($(B))>3.0:

    grafana alerts from flux queries

We can now see that two tanks, D2 and E1, are evaluating to true. When we preview the alert we can see that those two tanks will trigger a notification and change their state from Normal to Alerting:

grafana alerts from flux queries grafana alerts from flux queries

Conclusion

Flux queries and Grafana Unified Alerting are a powerful combination to identify practically any alertable conditions in your dataset, or across your entire system. For more information on Grafana Alerting, visit the documentation here. For more information on the Flux query language, you can visit that documentation as well.

How to create alert rules with log data

How to create alert rules with log data

Loki stores your logs and only indexes labels for each log stream. Using Loki with Grafana Alerting is a powerful way to keep track of what’s happening in your environment. You can create metric alert rules based on content in your log lines to notify your team. What’s even better is that you can add label data from the log message directly into your alert notification.

In this tutorial, you'll:

{{< admonition type="tip" >}} In Get started with Grafana Alerting - Part 2 you can advance your skills by exploring alert instances and notification routing. {{< /admonition >}}

{{< docs/ignore >}}

In Get started with Grafana Alerting - Part 2 you can advance your skills by exploring alert instances and notification routing.

{{< /docs/ignore >}}

Before you begin

There are different ways you can follow along with this tutorial.

Set up the Grafana stack

To demonstrate the observation of data using the Grafana stack, download and run the following files.

  1. Create a folder for this tutorial, then download and save the Docker Compose file, the Alloy configuration, and the Grafana data source provisioning file for this tutorial. Also create a logs folder, which Alloy reads the sample logs from.

    mkdir alerts-with-logs && cd alerts-with-logs
wget https://raw.githubusercontent.com/grafana/tutorial-environment/master/loki-logs/docker-compose.yaml
wget https://raw.githubusercontent.com/grafana/tutorial-environment/master/loki-logs/config.alloy
mkdir -p provisioning/datasources logs
wget -P provisioning/datasources https://raw.githubusercontent.com/grafana/tutorial-environment/master/loki-logs/provisioning/datasources/datasources.yaml

  2. Run the Grafana stack.

    docker compose up -d

The first time you run docker compose up -d, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

{{< admonition type="note" >}}

If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

{{< /admonition >}}

{{< docs/ignore >}}

If you already have Grafana, Loki, or Prometheus running on your system, you might see errors, because the Docker image is trying to use ports that your local installations are already using. If this is the case, stop the services, then run the command again.

{{< /docs/ignore >}}

Generate sample logs

To demonstrate how to create alert rules based on logs, you’ll use a script that generates realistic log entries to simulate typical monitoring data in Grafana. Running this script outputs logs continuously, each containing a timestamp, HTTP method (either GET or POST), status code (200 for success or 500 for failures), and request duration in milliseconds.

  1. Download and save a Python file that generates logs.

    wget https://raw.githubusercontent.com/grafana/tutorial-environment/master/app/loki/web-server-logs-simulator.py

  2. Execute the log-generating Python script.

    python3 ./web-server-logs-simulator.py | tee -a logs/web_requests.log

Troubleshooting the script

If you don't see the sample logs in Explore:

Create a contact point

Besides being an open-source observability tool, Grafana has its own built-in alerting service. This means that you can receive notifications whenever there is an event of interest in your data, and even see these events graphed in your visualizations.

In this step, we set up a new contact point. This contact point uses the webhook integration. This contact point uses the webhooks integration. In order to make this work, we also need an endpoint for our webhook integration to receive the alert. We can use Webhook.site to quickly set up that test endpoint. This way we can make sure that our alert is actually sending a notification somewhere.

  1. In your browser, sign in to your Grafana Cloud account.

    OSS users: To log in, navigate to http://localhost:3000, where Grafana should be running.

  2. In another tab, go to Webhook.site.

  3. Copy Your unique URL.

{{< docs/ignore >}}

  1. Navigate to http://localhost:3000, where Grafana should be running.
  2. In another tab, go to Webhook.site.
  3. Copy Your unique URL. {{< /docs/ignore >}}

Your webhook endpoint is now waiting for the first request.

Next, let's configure a contact point in the Grafana Alerting UI to send notifications to our webhook endpoint.

  1. Return to Grafana. In the Grafana sidebar, click Alerting > Notification configuration, and then select the Contact points tab.

  2. Click + New contact point.

  3. In Name, write Webhook.

  4. In Integration, choose Webhook.

  5. In URL, paste the endpoint to your webhook endpoint.

  6. Click Test, and then click Send test notification to send a test alert to your webhook endpoint.

  7. Navigate back to Webhook.site. On the left side, there's now a POST / entry. Click it to see what information Grafana sent.

    {{< figure src="/media/docs/alerting/alerting-webhook-detail.png" max-width="1200px" caption="A POST entry in Webhook.site" >}}

  8. Return to Grafana and click Save contact point.

We have created a dummy Webhook endpoint and created a new Alerting contact point in Grafana. Now, we can create an alert rule and link it to this new integration.

Create an alert rule

Next, we'll establish an alert rule within Grafana Alerting to notify us whenever alert rules are triggered and resolved.

  1. In Grafana, navigate to Alerting > Alert rules.
  2. Click on New alert rule.
  3. Enter alert rule name for your alert rule. Make it short and descriptive as this appears in your alert notification. For instance, web-requests-logs

Define query and alert condition

In this section, we use the default options for Grafana-managed alert rule creation. The default options let us define the query, an expression (used to manipulate the data -- the WHEN field in the UI), and the condition that must be met for the alert to be triggered (in default mode is the threshold).

  1. Select the Loki data source from the drop-down.

  2. The query editor opens in Code mode by default. If it's showing Builder mode instead, click Code on the right.

  3. Paste the query below.

    sum by (message)(count_over_time({filename="/var/log/web_requests.log"} != "status=200" | pattern "<_> <message> duration<_>" [10m]))

    This query counts the number of log lines with a status code that is not 200 (OK), then sum the result set by message type using an instant query and the time interval indicated in brackets. It uses the LogQL pattern parser to add a new label called message that contains the level, method, url, and status from the log line.

    You can use the explain query toggle button for a full explanation of the query syntax. The optional log-generating script creates a sample log line similar to the one below:

    2023-04-22T02:49:32.562825+00:00 level=info method=GET url=test.com status=200 duration=171ms

    {{< admonition type="note" >}} If you're using your own logs, modify the LogQL query to match your own log message. Refer to the Loki docs to understand the pattern parser. {{% / admonition %}}

    {{< docs/ignore >}} If you're using your own logs, modify the LogQL query to match your own log message. Refer to the Loki docs to understand the pattern parser. {{< /docs/ignore >}}

  4. In the Alert condition section, confirm the condition reads WHEN QUERY IS ABOVE 0. This is the default, and 0 is the threshold above which the alert rule should trigger.

  5. Click Preview alert rule condition to run the query.

    It should return alert instances from log lines with a status code that is not 200 (OK), and that has met the alert condition. The condition for the alert rule to fire is any occurrence that goes over the threshold of 0. Since the Loki query has returned more than zero alert instances, the alert rule is Firing.

    {{< figure src="/media/docs/alerting/firing-loki-alert-rule.png" max-width="1200px" caption="Preview of a firing alert instances" >}}

Set evaluation behavior

The alert rule evaluation defines the conditions under which an alert rule triggers, based on the following settings:

To set up the evaluation:

  1. In Folder, click + New folder and enter a name. For example: web-server-alerts. This folder contains our alerts.
  2. In Evaluation group and interval, click + New evaluation group.
  3. In Evaluation group name, enter 1m-evaluation.
  4. Keep Evaluation interval set to 1m (1 minute), so the alert rule is evaluated every minute, and then click Create.
  5. Set the pending period to, 0s (zero seconds), so the alert rule fires the moment the condition is met.
  6. Set Keep firing for to, 0s, so the alert stops firing immediately after the condition is no longer true. Use this when you want alerts to be resolved as soon as the system is healthy again.

Configure labels and notifications

Choose the contact point where you want to receive your alert notifications.

  1. Under Contact point, select Webhook from the drop-down menu.
  2. Click Save at the bottom of the page.

Trigger the alert rule

Since the Python script continues to generate log data that matches the alert rule condition, once the evaluation interval has concluded, you should receive an alert notification in the Webhook endpoint.

{{< figure src="/media/docs/alerting/alerting-webhook-firing-alert.png" max-width="1200px" caption="Firing alert notification details" >}}

{{< admonition type="tip" >}} In Get started with Grafana Alerting - Part 2 you can advance your skills by exploring alert instances and notification routing. {{< /admonition >}}

{{< docs/ignore >}}

In Get started with Grafana Alerting - Part 2 you can advance your skills by exploring alert instances and notification routing.

{{< /docs/ignore >}}

Create users and teams

This tutorial is for admins or anyone that wants to learn how to manage users in Grafana. You'll add multiple local users, organize them into teams, and make sure they're only able to access the resources they need.

Scenario

Graphona, a fictional telemarketing company, has asked you to configure Grafana for their teams.

In this scenario, you'll:

{{% class "prerequisite-section" %}}

Prerequisites

Add users

In Grafana, all users are granted an organization role that determines what resources they can access.

There are three types of organization roles in Grafana. The Grafana Admin is a global role, the default admin user has this role.

{{< admonition type="note" >}} You can also configure Grafana to allow anonymous access, to make dashboards available even to those who don't have a Grafana user account. That's how Grafana Labs made https://play.grafana.org publicly available. {{< /admonition >}}

Exercise

Graphona has asked you to add a group of early adopters that work in the Marketing and Engineering teams. They'll need to be able to edit their own team's dashboards, but want to have view access to dashboards that belong to the other team.

Name Email Username
Almaz Russom almaz.russom@example.com almaz.russom
Brenda Tilman brenda.tilman@example.com brenda.tilman
Mada Rawdha Tahan mada.rawdha.tahan@example.com mada.rawdha.tahan
Yuan Yang yuan.yang@example.com yuan.yang

Add users

Repeat the following steps for each of the employees in the table above to create the new user accounts:

  1. Log in as a user that has the Server Admin role.
  2. On the sidebar, click the Server Admin (shield) icon.
  3. Choose Users from the menu drop-down, then click New User.
  4. Enter the Name, Email, Username, and Password from the table above.
  5. Click the Create User button to create the account.

When you create a user they are granted the Viewer role by default, which means that they won't be able to make any changes to any of the resources in Grafana. That's ok for now, you'll grant more user permissions by adding users to teams in the next step.

Assign users to teams

Teams let you grant permissions to a group of users, instead of granting permissions to individual users one at a time.

Teams are useful when onboarding new colleagues. When you add a user to a team, they get access to all resources assigned to that team.

Exercise

In this step, you'll create two teams and assign users to them.

Username Team
brenda.tilman Marketing
mada.rawdha.tahan Marketing
almaz.russom Engineering
yuan.yang Engineering

Create a team

Create the Marketing and Engineering teams.

  1. In the sidebar, hover your mouse over the Configuration (gear) icon and then click Teams.
  2. Click New team.
  3. In Name, enter the name of the team: either Marketing or Engineering. You do not need to enter an email.
  4. Click Create.
  5. Click on the Teams link at the top of the page to return to teams page and create the second team.

Add a user to a team

Repeat these steps for each user to assign them to their team. Refer to the table above for team assignments.

  1. Click the team name Marketing or Engineering to add members to that team.
  2. Click Add member.
  3. In the Add team member box, click the drop-down arrow to choose the user you want to add to the team .
  4. Click Add to team.

When you're done, you'll have two teams with two users assigned to each.

Manage resource access with folders

It's a good practice to use folders to organize collections of related dashboards. You can assign permissions at the folder level to individual users or teams.

Exercise

The Marketing team is going to use Grafana for analytics, while the Engineering team wants to monitor the application they're building.

You'll create two folders, Analytics and Application, where each team can add their own dashboards. The teams still want to be able to view each other's dashboards.

Folder Team Permissions
Analytics Marketing Edit
Engineering View
Application Marketing View
Engineering Edit

Repeat the following steps for each folder. You'll move through all three steps for each folder before moving on to the next one.

Add a folder for each team

  1. In the sidebar, hover your cursor over the Dashboards (four squares) icon and then click Browse.
  2. To create a folder, click New Folder.
  3. In Name, enter the folder name.
  4. Click Create.
  5. Stay in the folder view and move on to the next sections to edit permissions for this folder.

Remove the viewer role from folder permissions

By default, when you create a folder, all users with the Viewer role are granted permission to view the folder.

In this example, Graphona wants to explicitly grant teams access to folders. To support this, you need to remove the Viewer role from the list of permissions:

  1. Go to the Permissions tab.
  2. Remove the Viewer role from the list, by clicking the red button on the right.
  3. Stay in the permissions tab and move on to the next section to grant folder permissions for each team.

Grant folder permissions to a team:

  1. Click Add Permission.
  2. In the Add Permission For dialog, make sure "Team" is selected in the first box.
  3. In the second box, select the team to grant access to.
  4. In the third box, select the access you want to grant.
  5. Click Save.
  6. Repeat for the other team.
  7. Click the Dashboards link at the top of the page to return to the dashboard list.

When you're finished, you'll have two empty folders, the contents of which can only be viewed by members of the Marketing or Engineering teams. Only Marketing team members can edit the contents of the Analytics folder, only Engineering team members can edit the contents of the Application folder.

Define granular permissions

By using folders and teams, you avoid having to manage permissions for individual users.

However, there are times when you need to configure permissions on a more granular level. For these cases, Grafana allows you to override permissions for specific dashboards.

Exercise

Graphona has hired a consultant to assist the Marketing team. The consultant should only be able to access the SEO dashboard in the Analytics folder.

Name Email Username
Luc Masson luc.masson@exampleconsulting.com luc.masson

Add a new user

  1. In the sidebar, click the Server Admin (shield) icon.
  2. In the Users tab, click New user.
  3. In Name, enter the name of the user.
  4. In E-mail, enter the email of the user.
  5. In Username, enter the username that the user will use to log in.
  6. In Password, enter a password. The user can change their password once they log in.
  7. Click Create user to create the user account.

Create a dashboard

  1. In the sidebar, click the Create (plus) icon to create a new dashboard.
  2. In the top right corner, click the cog icon to go to Dashboard settings.
  3. In Name, enter SEO.
  4. Click Save Dashboard.
  5. In the Save dashboard as... pop-up, choose the Analytics folder from the drop-down and click Save.

Grant a user permission to view dashboard

  1. In the top right corner of your dashboard, click the cog icon to go to Dashboard settings.
  2. Go to the Permissions tab, and click Add Permission.
  3. In the Add Permission For dialog, select User in the first box.
  4. In the second box, select the user to grant access to: Luc Masson.
  5. In the third box, select View.
  6. Click Save.
  7. Click Save dashboard.
  8. Add a note about giving Luc Masson Viewer permission for the dashboard and then click Save.

You've created a new user and given them unique permissions to view a single dashboard within a folder.

Check your work

You can repeat these steps to log in as the other users you've created see the differences in the viewer and editor roles.

For this example, you can log in as the user luc.masson to see that they can only access the SEO dashboard.

  1. Click the profile (avatar) button in the bottom left corner, choose Sign out.
  2. Enter luc.masson as the username.
  3. Enter the password you created for Luc.
  4. Click Log in.
  5. In the sidebar, hover your cursor over the Dashboards (four squares) icon and then click Browse.
  6. You'll notice that you won't see the Analytics folder in the folder view because we did not give Luc folder permission.
  7. Click on the list icon (3 lines) to see the dashboard list.
  8. Click on the SEO dashboard, there shouldn't be any editing permissions since we assigned Luc the viewer role.

Summary

In this tutorial, you've configured Grafana for an organization:

Learn more

Grafana fundamentals

Introduction

In this tutorial, you'll learn how to use Grafana to set up a monitoring solution for your application, and:

Alternatively, you can also watch our Grafana for Beginners series where we discuss fundamental concepts to help you get started with Grafana.

{{% class "prerequisite-section" %}}

Prerequisites

{{% /class %}}

{{< admonition type="tip" >}} Alternatively, you can try out this example in our interactive learning environment: Grafana Fundamentals.

It's a fully configured environment with all the dependencies already installed.

Interactive

Provide feedback, report bugs, and raise issues in the Grafana Killercoda repository. {{< /admonition >}}

Set up the sample application

This tutorial uses a sample application to demonstrate some of the features in Grafana. To complete the exercises in this tutorial, you need to download the files to your local machine.

In this step, you'll set up the sample application, as well as supporting services, such as Loki.

Note: Prometheus, a popular time series database (TSDB), has already been configured as a data source as part of this tutorial.

  1. Clone the github.com/grafana/tutorial-environment repository.

    git clone https://github.com/grafana/tutorial-environment.git

  2. Change to the directory where you cloned this repository:

    cd tutorial-environment

  3. Make sure Docker is running:

    docker ps

    No errors means it is running. If you get an error, then start Docker and then run the command again.

  4. Start the sample application:

    docker-compose up -d

    The first time you run docker-compose up -d, Docker downloads all the necessary resources for the tutorial. This might take a few minutes, depending on your internet connection.

    Note: If you already have Grafana, Loki, or Prometheus running on your system, then you might see errors because the Docker image is trying to use ports that your local installations are already using. Stop the services, then run the command again.

  5. Ensure all services are up-and-running:

    docker-compose ps

    In the State column, it should say Up for all services.

  6. Browse to the sample application on http://localhost:8081.

Grafana News

The sample application, Grafana News, lets you post links and vote for the ones you like.

To add a link:

  1. In Title, enter Example.

  2. In URL, enter https://example.com.

  3. Click Submit to add the link.

    The link appears in the list under the Grafana News heading.

To vote for a link, click the triangle icon next to the name of the link.

Open Grafana

Grafana is an open source platform for monitoring and observability that lets you visualize and explore the state of your systems.

  1. Open a new tab.
  2. Browse to http://localhost:3000.

{{< docs/ignore >}}

  1. Browse to http://localhost:3000. {{< /docs/ignore >}}

{{< admonition type="note" >}} This demo does not require a login page or credentials. However, if you choose to install Grafana locally, you need to log in and provide credentials. In that case, the default username and password is admin. {{< /admonition >}}

{{< docs/ignore >}}

This demo does not require a login page or credentials. However, if you choose to install Grafana locally, you need to log in and provide credentials. In that case, the default username and password is admin. {{< /docs/ignore >}}

The first thing you see is the Home dashboard, which helps you get started.

In the top left corner, you can see the menu icon. Clicking it opens the sidebar, the main menu for navigating Grafana.

Explore your metrics

Grafana Explore is a workflow for troubleshooting and data exploration. In this step, you'll be using Explore to create ad-hoc queries to understand the metrics exposed by the sample application. Specifically, you'll explore requests received by the sample application.

Ad-hoc queries are queries that are made interactively, with the purpose of exploring data. An ad-hoc query is commonly followed by another, more specific query.

  1. Click the menu icon and, in the sidebar, click Explore. A dropdown menu for the list of available data sources is on the upper-left side. The Prometheus data source should already be selected. If not, choose Prometheus.

  2. Confirm that you're in code mode by checking the Builder/Code toggle at the top right corner of the query panel.

  3. In the query editor, where it says Enter a PromQL query…, enter tns_request_duration_seconds_count and then press Shift + Enter. A graph appears.

  4. In the top right corner, click the dropdown arrow on the Run Query button, and then select 5s. Grafana runs your query and updates the graph every 5 seconds.

    You just made your first PromQL query! PromQL is a powerful query language that lets you select and aggregate time series data stored in Prometheus.

    tns_request_duration_seconds_count is a counter, a type of metric whose value only ever increases. Rather than visualizing the actual value, you can use counters to calculate the rate of change, i.e. how fast the value increases.

  5. Add the rate function to your query to visualize the rate of requests per second. Enter the following in the query editor and then press Shift + Enter.

    rate(tns_request_duration_seconds_count[5m])

    Immediately below the graph there's an area where each time series is listed with a colored icon next to it. This area is called the legend.

    PromQL lets you group the time series by their labels, using the sum aggregation operator.

  6. Add the sum aggregation operator to your query to group time series by route:

    sum(rate(tns_request_duration_seconds_count[5m])) by(route)

  7. Go back to the sample application and generate some traffic by adding new links, voting, or just refresh the browser.

  8. Back in Grafana, in the upper-right corner, click the time picker, and select Last 5 minutes. By zooming in on the last few minutes, it's easier to see when you receive new data.

Depending on your use case, you might want to group on other labels. Try grouping by other labels, such as status_code, by changing the by(route) part of the query to by(status_code).

Add a logging data source

Grafana supports log data sources, like Loki. Just like for metrics, you first need to add your data source to Grafana.

  1. Click the menu icon and, in the sidebar, click Connections and then Data sources.
  2. Click + Add new data source.
  3. In the list of data sources, click Loki.
  4. In the URL box, enter http://loki:3100
  5. Scroll to the bottom of the page and click Save & Test to save your changes.

You should see the message "Data source successfully connected." Loki is now available as a data source in Grafana.

Explore your logs

Grafana Explore not only lets you make ad-hoc queries for metrics, but lets you explore your logs as well.

  1. Click the menu icon and, in the sidebar, click Explore.

  2. In the data source list at the top, select the Loki data source.

  3. Confirm that you're in code mode by checking the Builder/Code toggle at the top right corner of the query panel.

  4. Enter the following in the query editor, and then press Shift + Enter:

    {filename="/var/log/tns-app.log"}

  5. Grafana displays all logs within the log file of the sample application. The height of each bar in the graph encodes the number of logs that were generated at that time.

  6. Click and drag across the bars in the graph to filter logs based on time.

Not only does Loki let you filter logs based on labels, but on specific occurrences.

Let's generate an error, and analyze it with Explore.

  1. In the sample application, post a new link without a URL to generate an error in your browser that says empty url.

  2. Go back to Grafana and enter the following query to filter log lines based on a substring:

    {filename="/var/log/tns-app.log"} |= "error"

  3. Click the log line that says level=error msg="empty url" to see more information about the error.

    Note: If you're in Live mode, clicking logs does not show more information about the error. Instead, stop and exit the live stream, then click the log line there.

Logs are helpful for understanding what went wrong. Later in this tutorial, you'll see how you can correlate logs with metrics from Prometheus to better understand the context of the error.

Build a dashboard

A dashboard gives you an at-a-glance view of your data and lets you track metrics through different visualizations.

Dashboards consist of panels, each representing a part of the story you want your dashboard to tell.

Every panel consists of a query and a visualization. The query defines what data you want to display, whereas the visualization defines how the data is displayed.

  1. Click the menu icon and, in the sidebar, click Dashboards.

  2. On the Dashboards page, click New in top right corner and select New Dashboard in the drop-down.

  3. Click + Add visualization.

  4. In the modal that opens, select the Prometheus data source that you just added.

  5. In the Query tab below the graph, enter the query from earlier and then press Shift + Enter:

    sum(rate(tns_request_duration_seconds_count[5m])) by(route)

  6. In the panel editor on the right, under Panel options, change the panel title to "Traffic".

  7. Click Apply in the top-right corner to save the panel and go back to the dashboard view.

  8. Click the Save dashboard (disk) icon at the top of the dashboard to save your dashboard.

  9. Enter a name in the Dashboard name field and then click Save.

    You should now have a panel added to your dashboard.

    {{< figure src="/media/tutorials/grafana-fundamentals-dashboard.png" alt="A panel in a Grafana dashboard" caption="A panel in a Grafana dashboard" >}}

Annotate events

When things go bad, it often helps if you understand the context in which the failure occurred. Time of last deploy, system changes, or database migration can offer insight into what might have caused an outage. Annotations allow you to represent such events directly on your graphs.

In the next part of the tutorial, we simulate some common use cases that someone would add annotations for.

  1. To manually add an annotation, click anywhere on a graph line to open the data tooltip, then click Add annotation. You can also press Ctrl or Command and click anywhere in the graph to open the Add annotation dialog box. Note: you might need to save the dashboard first.

  2. In Description, enter Migrated user database.

  3. Click Save.

    Grafana adds your annotation to the graph. Hover your mouse over the base of the annotation to read the text.

Grafana also lets you annotate a time interval, with region annotations.

Add a region annotation:

  1. Press Ctrl (or Cmd on macOS) and hold, then click and drag across the graph to select an area.
  2. In Description, enter Performed load tests.
  3. In Tags, enter testing.
  4. Click Save.

Using annotations to correlate logs with metrics

Manually annotating your dashboard is fine for those single events. For regularly occurring events, such as deploying a new release, Grafana supports querying annotations from one of your data sources. Let's create an annotation using the Loki data source we added earlier.

  1. At the top of the dashboard, click the Dashboard settings (gear) icon.

  2. Go to Annotations and click Add annotation query.

  3. In Name, enter Errors.

  4. In Data source, select Loki.

  5. In Query, enter the following query:

    {filename="/var/log/tns-app.log"} |= "error"

  6. Click Apply. Grafana displays the Annotations list, with your new annotation.

  7. Click your dashboard name to return to your dashboard.

  8. At the top of your dashboard, there is now a toggle to display the results of the newly created annotation query. Press it if it's not already enabled.

  9. Click the Save dashboard (disk) icon to save the changes.

  10. To test the changes, go back to the sample application, post a new link without a URL to generate an error in your browser that says empty url.

The log lines returned by your query are now displayed as annotations in the graph.

{{< figure src="/media/tutorials/annotations-grafana-dashboard.png" alt="A panel in a Grafana dashboard with log queries from Loki displayed as annotations" caption="Displaying log queries from Loki as annotations" >}}

Being able to combine data from multiple data sources in one graph allows you to correlate information from both Prometheus and Loki.

Annotations also work very well alongside alert rules. In the next and final section, we set up an alert rules for our app grafana.news and then we trigger it. This provides a quick intro to our new Alerting platform.

Create a Grafana-managed alert rule

Alert rules allow you to identify problems in your system moments after they occur. By quickly identifying unintended changes in your system, you can minimize disruptions to your services.

Grafana's new alerting platform debuted with Grafana 8. A year later, with Grafana 9, it became the default alerting method. In this step we create a Grafana-managed alert rule. Then we trigger our new alert rule and send a test message to a dummy endpoint.

The most basic alert rule consists of two parts:

  1. A Contact point - A Contact point defines how Grafana delivers an alert instance. When the conditions of an alert rule are met, Grafana notifies the contact points, or channels, configured for that alert rule.

    {{< admonition type="note" >}} An alert instance is a specific occurrence that matches a condition defined by an alert rule, such as when the rate of requests for a specific route suddenly increases. {{< /admonition >}}

    {{< docs/ignore >}}

    An alert instance is a specific occurrence that matches a condition defined by an alert rule, such as when the rate of requests for a specific route suddenly increases. {{< /docs/ignore >}}

    Some popular channels include:

  2. An Alert rule - An Alert rule defines one or more conditions that Grafana regularly evaluates. When these evaluations meet the rule's criteria, the alert rule is triggered.

To begin, let's set up a webhook contact point. Once we have a usable endpoint, we write an alert rule and trigger a notification.

Create a contact point for Grafana-managed alert rules

In this step, we set up a new contact point. This contact point uses the webhooks channel. In order to make this work, we also need an endpoint for our webhook channel to receive the alert notification. We can use Webhook.site to quickly set up that test endpoint. This way we can make sure that our alert manager is actually sending a notification somewhere.

  1. Browse to Webhook.site.
  2. Copy Your unique URL.

Your webhook endpoint is now waiting for the first request.

Next, let's configure a Contact Point in Grafana's Alerting UI to send notifications to our webhook endpoint.

  1. Return to Grafana. In Grafana's sidebar, hover over the Alerting (bell) icon and then click Manage Contact points.

  2. Click + Add contact point.

  3. In Name, write Webhook.

  4. In Integration, choose Webhook.

  5. In URL, paste the endpoint to your webhook endpoint.

  6. Click Test, and then click Send test notification to send a test alert notification to your webhook endpoint.

  7. Navigate back to the webhook endpoint you created earlier. On the left side, there's now a POST / entry. Click it to see what information Grafana sent.

  8. Return to Grafana and click Save contact point.

We have now created a dummy webhook endpoint and created a new Alerting Contact Point in Grafana. Now we can create an alert rule and link it to this new channel.

Add an alert rule to Grafana

Now that Grafana knows how to notify us, it's time to set up an alert rule:

  1. In Grafana's sidebar, hover over the Alerting (bell) icon and then click Alert rules.

    In this tutorial, we use the advanced options for Grafana-managed alert rule creation. The advanced options let us define queries, expressions (used to manipulate the data), and the condition that must be met for the alert to be triggered (the default condition is the threshold).

  2. Click + New alert rule.

  3. For Section 1, name the rule fundamentals-test.

  4. For Section 2, toggle the Advanced options button.

  5. Find the query A box, and choose your Prometheus data source.

  6. Enter the same Prometheus query that we used in our earlier panel:

    sum(rate(tns_request_duration_seconds_count[5m])) by(route)

  7. Keep expressions B and C as they are. These expressions (Reduce and Threshold, respectively) are included by default when creating a new rule. Enter 0.2 as threshold value. You can read more about queries and conditions here.

  8. Scroll down to bottom of section #2 and click the Preview button. You should see some data returned.

  9. In Section 3, in Folder, create a new folder, by clicking New folder and typing a name for the folder. This folder contains our alert rules. For example: fundamentals. Then, click create.

  10. In the Evaluation group, repeat the above step to create a new one. Name it fundamentals too.

  11. Choose an Evaluation interval (how often the alert rule are evaluated). For example, every 10s (10 seconds).

  12. Set the pending period. This is the time that a condition has to be met until the alert instance enters in Firing state and a notification is sent. Enter 0s. For the purposes of this tutorial, the evaluation interval is intentionally short. This makes it easier to test. This setting makes Grafana wait until an alert instance has fired for a given time before Grafana sends the notification.

  13. In Section 4, choose Webhook as the Contact point.

  14. Click Save rule and exit at the top of the page.

Trigger a Grafana-managed alert rule

We have now configured an alert rule and a contact point. Now let's see if we can trigger a Grafana-managed alert rule by generating some traffic on our sample application.

  1. Browse to localhost:8081.
  2. Add a new title and URL, repeatedly click the vote button, or refresh the page to generate a traffic spike.

Once the query sum(rate(tns_request_duration_seconds_count[5m])) by(route) returns a value greater than 0.2 Grafana triggers our alert rule. Browse to the webhook endpoint we created earlier and find the sent Grafana alert notification with details and metadata.

{{< admonition type="note" >}} The alert rule may be triggered by the /metrics endpoint which is frequently accessed by Grafana when pulling metrics from the application. If this happens, you can increase the Threshold value in Section 2 for testing purposes. {{< /admonition >}}

{{< docs/ignore >}}

The alert rule may be triggered by the /metrics endpoint which is frequently accessed by Grafana when pulling metrics from the application. If this happens, you can increase the Threshold value in Section 2 for testing purposes. {{< /docs/ignore >}}

Display Grafana-managed alert rules to your dashboard

In most cases, it's also valuable to display Grafana alert instances as annotations to your dashboard. Check out the video tutorial below to learn how to display alerting to your dashboard.

{{< youtube id="ClLp-iSoaSY" >}}

Let's see how we can configure this.

  1. In Grafana's sidebar, hover over the Alerting (bell) icon and then click Alert rules.

  2. Expand the fundamentals > fundamentals folder to view our created alert rule.

  3. Click the Edit icon and scroll down to Section 5.

  4. Click the Link dashboard and panel button and select the dashboard and panel to which you want the alert instance to be added as an annotation.

  5. Click Confirm and Save rule and exit to save all the changes.

  6. In Grafana's sidebar, navigate to the dashboard by clicking Dashboards and selecting the dashboard you created.

  7. To test the changes, follow the steps listed to trigger a Grafana-managed alert rule.

    You should now see a red, broken heart icon beside the panel name, indicating that the alert rule has been triggered. An annotation for the alert instance, represented as a vertical red line, is also displayed.

    {{< figure src="/media/tutorials/grafana-alert-on-dashboard.png" alt="A panel in a Grafana dashboard with alerting and annotations configured" caption="Displaying Grafana-managed alert rules on a dashboard" >}}

{{< admonition type="tip" >}}

In Get started with Grafana Alerting - Part 2 you can advance your skills by exploring alert instances and notification routing.

{{< /admonition >}}

{{< docs/ignore >}}

In Get started with Grafana Alerting - Part 2 you can advance your skills by exploring alert instances and notification routing.

{{< /docs/ignore >}}

Summary

In this tutorial you learned about fundamental features of Grafana. To do so, we ran several Docker containers on your local machine. When you are ready to clean up this local tutorial environment, run the following command:

docker-compose down -v

{{< docs/ignore >}}

In this tutorial you learned about fundamental features of Grafana.

{{< /docs/ignore >}}

Learn more

Check out the links below to continue your learning journey with Grafana's LGTM stack.

Use IIS with URL Rewrite as a reverse proxy

If you want Grafana to be a subpath/subfolder under a website in IIS then the Application Request Routing (ARR) and URL Rewrite modules for ISS can be used to support this.

Example:

Grafana as a subpath: http://yourdomain.com:8080/grafana

Other Examples:

Setup

Install the URL Rewrite module for IIS.

You will also need the Application Request Routing (ARR) module for IIS for proxy forwarding

Grafana Config

The Grafana config can be set by creating a file named/editing the existing file named custom.ini in the conf subdirectory of your Grafana installation. See the installation instructions for more details.

Using the example from above, if the subpath is grafana (you can set this to whatever is required) and the parent site is yourdomain.com:8080, then you would add this to the custom.ini config file:

[server]
domain = yourdomain.com:8080
root_url = %(protocol)s://%(domain)s/grafana/

Restart the Grafana server after changing the config file.

Configured address to serve Grafana: http://yourdomain.com:8080/grafana

If you already have a subpath on your domain, configure it as follows:

[server]
domain = yourdomain.com/existingsubpath
root_url = %(protocol)s://%(domain)s/grafana/

Restart the Grafana server after changing the config file.

Configured address to serve Grafana: http://yourdomain.com/existingsubpath/grafana

IIS Config

Step 1: Forward Proxy

  1. Open the IIS Manager and click on the server
  2. In the admin console for the server, double click on the Application Request Routing option:
  3. Click the Server Proxy Settings action on the right-hand pane
  4. Select the Enable proxy checkbox so that it is enabled
  5. Click Apply and proceed with the URL Rewriting configuration

Note: If you don't enable the Forward Proxy, you will most likely get 404 Not Found if you only apply the URL Rewrite rule

Step 2: URL Rewriting

  1. In the IIS Manager, click on the website that grafana will run under. For example, select the website that is bound to the http://yourdomain.com domain.
  2. In the admin console for this website, double click on the URL Rewrite option:

{{< figure src="/static/img/docs/tutorials/IIS_admin_console.png" max-width="800px" >}}

  1. Click on the Add Rule(s)... action
  2. Choose the Blank Rule template for an Inbound Rule

{{< figure src="/static/img/docs/tutorials/IIS_add_inbound_rule.png" max-width="800px" >}}

  1. Create an Inbound Rule for the website with the following settings:

{{< figure src="/static/img/docs/tutorials/IIS_url_rewrite.png" max-width="800px" >}}

  1. If your version of Grafana is greater than 8.3.5, you also need to configure the reverse proxy to preserve host headers.

Finally, navigate to http://yourdomain.com:8080/grafana and you should come to the Grafana login page.

Troubleshooting

404 error

When navigating to the Grafana URL (http://yourdomain.com:8080/grafana) and a HTTP Error 404.0 - Not Found error is returned, then either:

Grafana Website only shows text with no images or css

{{< figure src="/static/img/docs/tutorials/IIS_proxy_error.png" max-width="800px" >}}

  1. The root_url setting in the Grafana config file does not match the parent URL with subpath. This could happen if the root_url is commented out by mistake (; is used for commenting out a line in .ini files):

    ; root_url = %(protocol)s://%(domain)s/grafana/

  2. or if the subpath in the root_url setting does not match the subpath used in the pattern in the Inbound Rule in IIS:

    root_url = %(protocol)s://%(domain)s/grafana/

    pattern in Inbound Rule: wrongsubpath(/)?(.*)

  3. or if the Rewrite URL in the Inbound Rule is incorrect.

    The Rewrite URL should not include the subpath.

    The Rewrite URL should contain the capture group from the pattern matching that returns the part of the URL after the subpath. The pattern used above returns three capture groups and the third one {R:2} returns the part of the URL after http://yourdomain.com:8080/grafana/.

You see an 'Error updating options: origin not allowed' error

Install Grafana on Raspberry Pi

The Raspberry Pi is a tiny, affordable, yet capable computer that can run a range of different applications. Even Grafana!

Many people are running Grafana on Raspberry Pi as a way to monitor their home, for things like indoor temperature, humidity, or energy usage.

In this tutorial, you'll:

{{% class "prerequisite-section" %}}

Prerequisites

Set up your Raspberry Pi

Before we can install Grafana, you first need to set up your Raspberry Pi.

For this tutorial, you'll configure your Raspberry Pi to be headless. This means you don't need to connect a monitor, keyboard, or mouse to your Raspberry Pi. All configuration is done from your regular computer.

Download and install Raspberry Pi Imager

Before we get started, you need to download and install the Raspberry Pi Imager.

We'll use the Raspberry Pi Imager to flash the operating system image to the SD card. You download the imager directly from the official Raspberry Pi website and it's available for Ubuntu Linux, macOS, and Windows.

Follow the directions on the website to download and install the imager.

Install Raspberry Pi OS

Now it's time to install Raspberry Pi OS.

  1. Insert the SD card into your regular computer from which you plan to install Raspberry Pi OS.
  2. Run the Raspberry Pi Imager that you downloaded and installed.
  3. To select an operating system, click Choose OS in the imager. You will be shown a list of available options.
  4. From the list, select Raspberry Pi OS (other) and then select Raspberry Pi OS Lite, which is a Debian-based operating system for the Raspberry Pi. Since you're going to run a headless Raspberry Pi, you won't need the desktop dependencies.
  5. To select where you want to put the operating system image, click Choose Storage in the imager and then select the SD card you already inserted into your computer.
  6. The final step in the imager to click Write. When you do, the imager will write the Raspberry Pi OS Lite image to the SD card and verify that it has been written correctly.
  7. Eject the SD card from your computer, and insert it again.

While you could fire up the Raspberry Pi now, we don't yet have any way of accessing it.

  1. Create an empty file called ssh in the boot directory. This enables SSH so that you can log in remotely.

    The next step is only required if you want the Raspberry Pi to connect to your wireless network. Otherwise, connect the it to your network by using a network cable.

  2. (Optional) Create a file called wpa_supplicant.conf in the boot directory:

    ctrl_interface=/var/run/wpa_supplicant
update_config=1
country=<Insert 2 letter ISO 3166-1 country code here>

network={
 ssid="<Name of your WiFi>"
 psk="<Password for your WiFi>"
}

All the necessary files are now on the SD card. Let's start up the Raspberry Pi.

  1. Eject the SD card and insert it into the SD card slot on the Raspberry Pi.
  2. Connect the power cable and make sure the LED lights are on.
  3. Find the IP address of the Raspberry Pi. Usually you can find the address in the control panel for your WiFi router.

Connect remotely via SSH

  1. Open up your terminal and enter the following command:

    ssh pi@<ip address>

  2. SSH warns you that the authenticity of the host can't be established. Type "yes" to continue connecting.

  3. When asked for a password, enter the default password: raspberry.

  4. Once you're logged in, change the default password:

    passwd

Congratulations! You've now got a tiny Linux machine running that you can hide in a closet and access from your normal workstation.

Install Grafana

Now that you've got the Raspberry Pi up and running, the next step is to install Grafana.

  1. Add the APT key used to authenticate packages:

    sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

  2. Add the Grafana APT repository:

    echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

  3. Install Grafana:

    sudo apt-get update
sudo apt-get install -y grafana

Grafana is now installed, but not yet running. To make sure Grafana starts up even if the Raspberry Pi is restarted, we need to enable and start the Grafana Systemctl service.

  1. Enable the Grafana server:

    sudo /bin/systemctl enable grafana-server

  2. Start the Grafana server:

    sudo /bin/systemctl start grafana-server

    Grafana is now running on the machine and is accessible from any device on the local network.

  3. Open a browser and go to http://<ip address>:3000, where the IP address is the address that you used to connect to the Raspberry Pi earlier. You're greeted with the Grafana login page.

  4. Log in to Grafana with the default username admin, and the default password admin.

  5. Change the password for the admin user when asked.

Congratulations, Grafana is now running on your Raspberry Pi. If the Raspberry Pi is ever restarted or turned off, Grafana will start up whenever the machine regains power.

Summary

If you want to use Grafana without having to go through a full installation process, check out Grafana Cloud, which is designed to get users up and running quickly and easily. Grafana Cloud offers a forever free plan that is genuinely useful for hobbyists, testing, and small teams.

Learn more

Integrate Hubot with Grafana

Grafana 2.0 shipped with a great feature that enables it to render any graph or panel to a PNG image.

No matter what data source you are using, the PNG image of the Graph will look the same as it does in your browser.

This guide shows you how to install and configure the Hubot-Grafana plugin. This plugin allows you to tell Hubot to render any dashboard or graph right from a channel in Slack, Basecamp, or any other supported Hubot adapter. The bot will respond with an image of the graph and a link that will take you to the graph.

{{< figure src="/static/img/docs/tutorials/hubot_grafana.png" max-width="800px" >}}

What is Hubot?

Hubot is an universal and extensible chat bot that can be used with many chat services and has a huge library of third party plugins that allow you to automate anything from your chat rooms.

Install Hubot

Hubot is very easy to install and host. If you do not already have a bot up and running please read the official Getting Started With Hubot guide.

Install the Hubot-Grafana script

In your Hubot project repo install the Grafana plugin using npm:

npm install hubot-grafana --save

Edit the file external-scripts.json, and add hubot-grafana to the list of plugins.

["hubot-pugme", "hubot-shipit", "hubot-grafana"]

Configure

The Hubot-Grafana plugin requires two environment variables to be set in order to work properly.

export HUBOT_GRAFANA_HOST=https://play.grafana.org
export HUBOT_GRAFANA_API_KEY=abcd01234deadbeef01234

There are additional environment variables that you can set to control the appearance of the graphs.

Grafana server side rendering

The hubot plugin will take advantage of the Grafana server side rendering feature that can render any panel on the server using phantomjs. Grafana ships with a phantomjs binary (Linux only).

To verify that this feature works try the Direct link to rendered image link in the panel share dialog. If you do not get an image when opening this link verify that the required font packages are installed for phantomjs to work.

Grafana API Key

{{< figure src="/static/img/docs/v2/orgdropdown_api_keys.png" max-width="150px" class="docs-image--right">}}

You need to set the environment variable HUBOT_GRAFANA_API_KEY to a Grafana API Key. You can add these from the API Keys page which you find in the Organization dropdown.

Image uploading

There are several approaches to uploading the rendered graphs. If you are using Slack, Rocket.Chat, or Telegram, the adapter's native uploader will take care of sending it through their respective API. If your Hubot is hosted on a platform that doesn't support uploads (such as IRC), you can use the built-in S3 uploader. Note if you configure S3, it will not use the adapter's upload features.

Hubot commands

Aliases

Some of the hubot commands above can lengthy and you might have to remember the dashboard slug (url id). If you have a few favorite graphs you want to be able check up on often (let's say from your mobile) you can create hubot command aliases with the hubot script hubot-alias.

Install it:

npm i --save hubot-alias

Now add hubot-alias to the list of plugins in external-scripts.json and restart hubot.

Now you can add an alias like this:

{{< figure src="/static/img/docs/tutorials/hubot_grafana2.png" max-width="800px" >}}

Summary

Grafana is going to ship with integrated Slack and Hipchat features some day but you do not have to wait for that. Grafana 2 shipped with a very clever server side rendering feature that can render any panel to a png using phantomjs. The hubot plugin for Grafana is something you can install and use today!

Provision dashboards and data sources

Learn how you can reuse dashboards and data sources across multiple teams by provisioning Grafana from version-controlled configuration files.

In this tutorial, you'll:

{{% class "prerequisite-section" %}}

Prerequisites

Configuration as code

Configuration as code is the practice of storing the configuration of your system as a set of version controlled, human-readable configuration files, rather than in a database. These configuration files can be reused across environments to avoid duplicated resources.

As the number of dashboards and data sources grows within your organization, manually managing changes can become tedious and error-prone. Encouraging reuse becomes important to avoid multiple teams redesigning the same dashboards.

Grafana supports configuration as code through provisioning. The resources that currently supports provisioning are:

Set the provisioning directory

Before you can start provisioning resources, Grafana needs to know where to find the provisioning directory. The provisioning directory contains configuration files that are applied whenever Grafana starts and continuously updated while running.

By default, Grafana looks for a provisioning directory in the configuration directory (grafana > conf) on the system where Grafana is installed. However, if you are a Grafana Administrator, then you might want to place the config files in a shared resource like a network folder, so you would need to change the path to the provisioning directory.

You can set a different path by setting the paths.provisioning property in the main config file:

[paths]
provisioning = <path to config files>

For more information about configuration files, refer to Configuration in the Grafana documentation.

The provisioning directory assumes the following structure:

provisioning/
  datasources/
    <yaml files>
  dashboards/
    <yaml files>

Next, we'll look at how to provision a data source.

Provision a data source

Each data source provisioning config file contains a manifest that specifies the desired state of a set of provisioned data sources.

At startup, Grafana loads the configuration files and provisions the data sources listed in the manifests.

Let's configure a TestData data source that you can use for your dashboards.

Create a data source manifest

  1. In the provisioning/datasources/ directory, create a file called default.yaml with the following content:

    apiVersion: 1

datasources:
  - name: TestData
    type: testdata

  2. Restart Grafana to load the new changes.

  3. In the sidebar, hover the cursor over the Configuration (gear) icon and click Data Sources. TestData appears in the list of data sources.

The configuration options can vary between different types of data sources. For more information on how to configure a specific data source, refer to Data sources.

Provision a dashboard

Each dashboard config file contains a manifest that specifies the desired state of a set of dashboard providers.

A dashboard provider tells Grafana where to find the dashboard definitions and where to put them.

Grafana regularly checks for changes to the dashboard definitions (by default every 10 seconds).

Let's define a dashboard provider so that Grafana knows where to find the dashboards we want to provision.

Define a dashboard provider

In the provisioning/dashboards/ directory, create a file called default.yaml with the following content:

apiVersion: 1

providers:
  - name: Default # A uniquely identifiable name for the provider
    folder: Services # The folder where to place the dashboards
    type: file
    options:
      path:
        <path to dashboard definitions>
        # Default path for Windows: C:/Program Files/GrafanaLabs/grafana/public/dashboards
        # Default path for Linux is: /var/lib/grafana/dashboards

For more information on how to configure dashboard providers, refer to Dashboards.

Create a dashboard definition

  1. In the dashboard definitions directory you specified in the dashboard provider, i.e. options.path, create a file called cluster.json with the following content:

    {
  "__inputs": [],
  "__requires": [],
  "annotations": {
    "list": []
  },
  "editable": false,
  "gnetId": null,
  "graphTooltip": 0,
  "hideControls": false,
  "id": null,
  "links": [],
  "panels": [
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "TestData",
      "fill": 1,
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 2,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "rightSide": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "repeat": null,
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "CPU Usage",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ]
    }
  ],
  "refresh": "",
  "rows": [],
  "schemaVersion": 16,
  "tags": ["kubernetes"],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Cluster",
  "version": 0
}

  2. Restart Grafana to provision the new dashboard or wait 10 seconds for Grafana to automatically create the dashboard.

  3. In the sidebar, hover the cursor over Dashboards (squares) icon, and then click Manage. The dashboard appears in a Services folder.

If you don't specify an id in the dashboard definition, then Grafana assigns one during provisioning. You can set the id yourself if you want to reference the dashboard from other dashboards. Be careful to not use the same id for multiple dashboards, as this will cause a conflict.

Summary

In this tutorial you learned how you to reuse dashboards and data sources across multiple teams by provisioning Grafana from version-controlled configuration files.

Dashboard definitions can get unwieldy as more panels and configurations are added to them. There are a number of open source tools available to make it easier to manage dashboard definitions:

Learn more

Run Grafana behind a reverse proxy

In this tutorial, you'll configure Grafana to run behind a reverse proxy.

When running Grafana behind a proxy, you need to configure the domain name to let Grafana know how to render links and redirects correctly.

[server]
domain = example.com

Configure reverse proxy

Configure nginx

nginx is a high performance load balancer, web server, and reverse proxy.

# This is required to proxy Grafana Live WebSocket connections.
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

upstream grafana {
  server localhost:3000;
}

server {
  listen 80;
  root /usr/share/nginx/html;
  index index.html index.htm;

  location / {
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }

  # Proxy Grafana Live WebSocket connections.
  location /api/live/ {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }
}

For Grafana Live which uses WebSocket connections you may have to raise the nginx value for worker_connections option which is 512 by default. The default value limits the number of possible concurrent connections with Grafana Live.

Also, be aware that the preceding configuration only works when the proxy_pass value for location / is a literal string. If you want to use a variable here, you must instead use a rewrite rule. For more information, refer to the GitHub issue #18299.

To configure nginx to serve Grafana under a sub path, update the location block:

# This is required to proxy Grafana Live WebSocket connections.
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

upstream grafana {
  server localhost:3000;
}

server {
  listen 80;
  root /usr/share/nginx/www;
  index index.html index.htm;

  location /grafana/ {
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }

  # Proxy Grafana Live WebSocket connections.
  location /grafana/api/live/ {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }
}

Add a rewrite rule to each location block:

 rewrite  ^/grafana/(.*)  /$1 break;

{{< admonition type="note" >}} If nginx is performing TLS termination, then you must set the root_url and protocol configuration accordingly. If you're serving Grafana from https://example.com/grafana/ then the root_url is https://example.com/grafana/ or https://%(domain)s/grafana/ with the corresponding domain configuration value set to example.com in the server section of the Grafana configuration file. Set protocol to http. {{< /admonition >}}

Configure HAProxy

To configure HAProxy to serve Grafana under a sub path:

frontend http-in
  bind *:80
  use_backend grafana_backend if { path /grafana } or { path_beg /grafana/ }

backend grafana_backend
  server grafana localhost:3000
  # Requires haproxy >= 1.6
  http-request set-path %[path,regsub(^/grafana/?,/)]
  # Works for haproxy < 1.6
  # reqrep ^([^\ ]*\ /)grafana[/]?(.*) \1\2

  server grafana localhost:3000

Configure IIS

{{< admonition type="note" >}} To use IIS, you must have the URL Rewrite module installed. {{< /admonition >}}

To configure IIS to serve Grafana under a sub path, create an Inbound Rule for the parent website in IIS Manager with the following settings:

This is the rewrite rule that's generated in the web.config:

  <rewrite>
      <rules>
          <rule name="Grafana" enabled="true" stopProcessing="true">
              <match url="grafana(/)?(.*)" />
              <action type="Rewrite" url="http://localhost:3000/{R:2}" logRewrittenUrl="false" />
          </rule>
      </rules>
  </rewrite>

For more detailed instruction, refer to the tutorial on IIS URL Rewrites.

Configure Apache

To use Apache as a proxy, ensure its proper installation and configuration.

  1. Ensure that the Apache proxy module mod_proxy is installed and enabled. To enable, run the following commands:
a2enmod proxy
a2enmod proxy_http
  1. To configure the proxy, edit the site configuration file. To do so, inside the <VirtualHost> section, add the following code:
  ProxyPreserveHost on
  ProxyPass / http://your_grafana_server:3000
  ProxyPassReverse / http://your_grafana_server:3000
  1. Finally, restart Apache for the settings to take effect.

After you've restarted, navigate to your Apache server on port 80 and you will be redirected to Grafana.

To configure Grafana hosted in a sub path, replace the sub path with the following code (assuming your Grafana instance is on the sub path your_path):

  ProxyPreserveHost on
  ProxyPass /your_path http://your_grafana_server:3000
  ProxyPassReverse /your_path http://your_grafana_server:3000
  ProxyPass / http://your_grafana_server:3000/your_path
  ProxyPassReverse / http://192.168.250.5:3000/your_path

Note that the lines containing your_path must come before the lines referencing root path (/) in order for this to work correctly.

Configure Traefik

Traefik Cloud Native application proxy.

Using the Docker provider and the following labels configures the router and service for a domain or subdomain routing.

labels:
  traefik.http.routers.grafana.rule: Host(`grafana.example.com`)
  traefik.http.services.grafana.loadbalancer.server.port: 3000

To deploy on a sub path:

labels:
  traefik.http.routers.grafana.rule: Host(`example.com`) && PathPrefix(`/grafana`)
  traefik.http.services.grafana.loadbalancer.server.port: 3000

Examples using the file provider.

http:
  routers:
    grafana:
      rule: Host(`grafana.example.com`)
      service: grafana
  services:
    grafana:
      loadBalancer:
        servers:
          - url: http://192.168.30.10:3000

http:
  routers:
    grafana:
      rule: Host(`example.com`) && PathPrefix(`/grafana`)
      service: grafana
  services:
    grafana:
      loadBalancer:
        servers:
          - url: http://192.168.30.10:3000

Alternative for serving Grafana under a sub path

{{< admonition type="note" >}} You only need this if you don't handle the sub path serving via your reverse proxy configuration. {{< /admonition >}}

If you don't want or can't use the reverse proxy to handle serving Grafana from a sub path, you can set the configuration variable serve_from_sub_path to true.

  1. Include the sub path at the end of the root_url.
  2. Set serve_from_sub_path to true:
[server]
domain = example.com
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/
serve_from_sub_path = true

Stream metrics from Telegraf to Grafana

Grafana v8 introduced streaming capabilities – a way to push data to UI panels in near real-time. In this tutorial we show how Grafana real-time streaming capabilities can be used together with Telegraf to instantly display system measurements.

In this tutorial, you'll:

{{% class "prerequisite-section" %}}

Prerequisites

Run Grafana and create admin token

  1. Run Grafana following installation instructions for your operating system.
  2. Log in and go to Configuration -> API Keys.
  3. Press "Add API key" button and create a new API token with Admin role.

Configure and run Telegraf

Telegraf is a plugin-driven server agent for collecting and sending metrics and events from databases, systems, and IoT sensors.

You can install it following official installation instructions.

In this tutorial we will be using Telegraf HTTP output plugin to send metrics in Influx format to Grafana. We can use a configuration like this:

[agent]
  interval = "1s"
  flush_interval = "1s"

[[inputs.cpu]]
  percpu = false
  totalcpu = true

[[outputs.http]]
  url = "http://localhost:3000/api/live/push/custom_stream_id"
  data_format = "influx"
  [outputs.http.headers]
    Authorization = "Bearer <Your API Key>"

Make sure to replace <Your API Key> placeholder with your actual API key created in the previous step. Save this config into telegraf.conf file and run Telegraf pointing to this config file. Telegraf will periodically (once in a second) report the state of total CPU usage on a host to Grafana (which is supposed to be running on http://localhost:3000). Of course you can replace custom_stream_id to something more meaningful for your use case.

Inside Grafana Influx data is converted to Grafana data frames and then frames are published to Grafana Live channels. In this case, the channel where CPU data will be published is stream/custom_stream_id/cpu. The stream scope is constant, the custom_stream_id namespace is the last part of API URL set in Telegraf configuration (http://localhost:3000/api/live/push/telegraf) and the path is cpu - the name of a measurement.

The only thing left here is to create a dashboard with streaming data.

Create dashboard with streaming data

  1. Click Dashboards in the left-side menu.
  2. Click New and select New Dashboard.
  3. On the empty dashboard, click + Add visualization.
  4. In the modal that opens, select the -- Grafana -- data source.
  5. Select Live Measurements query type.
  6. Find and select stream/custom_stream_id/cpu measurement for Channel field.
  7. Save dashboard changes.

After making these steps Grafana UI should subscribe to the channel stream/custom_stream_id/cpu and you should see CPU data updates coming from Telegraf in near real-time.

Stream using WebSocket endpoint

If you aim for a high-frequency update sending then you may want to use the WebSocket output plugin of Telegraf (introduced in Telegraf v1.19.0) instead of the HTTP output plugin we used above. Configure WebSocket output plugin like this:

[agent]
  interval = "500ms"
  flush_interval = "500ms"

[[inputs.cpu]]
  percpu = false
  totalcpu = true

[[outputs.websocket]]
  url = "ws://localhost:3000/api/live/push/custom_stream_id"
  data_format = "influx"
  [outputs.websocket.headers]
    Authorization = "Bearer <Your API Key>"

WebSocket avoids running all Grafana HTTP middleware on each request from Telegraf thus reducing Grafana backend CPU usage significantly.

Summary

In this tutorial you learned how to use Telegraf to stream live metrics to Grafana.