As Code

Create and manage your Grafana Cloud stack using Ansible

This guide shows you how to create a Grafana Cloud stack and add a data source, dashboard, and folder using the Ansible Collection for Grafana. You'll manage your Grafana infrastructure through Ansible playbooks.

Before you begin

Before you begin, make sure you have the following available:

Install the Grafana Ansible collection

Install the Grafana Ansible collection:

ansible-galaxy collection install grafana.grafana

This collection provides all the modules needed to manage Grafana Cloud stacks and resources.

Create a Cloud stack

First, create a Grafana Cloud Access Policy and get a token. You'll need this for the Ansible playbook to be able to create a Grafana Cloud stack. Refer to Create a Grafana Cloud Access Policy.

Next, create an Ansible playbook file. This Ansible playbook creates a Grafana Cloud stack using the Cloud stack module.

To do so, create a file named cloud-stack.yml and add the following:

- name: Create Grafana Cloud stack
  connection: local
  hosts: localhost

  vars:
    grafana_cloud_api_key: '<CLOUD_ACCESS_POLICY_TOKEN>'
    stack_name: '<STACK_NAME>'
    org_name: '<ORG_NAME>'

  tasks:
    - name: Create a Grafana Cloud stack
      grafana.grafana.cloud_stack:
        name: '{{ stack_name }}'
        stack_slug: '{{ stack_name }}'
        cloud_api_key: '{{ grafana_cloud_api_key }}'
        org_slug: '{{ org_name }}'
        delete_protection: true
        state: present
      register: stack_result

    - name: Display stack URL
      debug:
        msg: 'Stack created at: {{ stack_result.url }}'

Replace the placeholders with your values:

The playbook registers the stack creation result and displays the stack URL, which you'll need for subsequent resource management.

Create an API key in your Grafana stack

Create an API key in the Grafana stack. You'll need this key to configure Ansible to create data sources, folders, and dashboards.

  1. Log into your Grafana Cloud instance.
  2. Click Administration and select API keys.
  3. Click Add API key.
  4. In Key name, enter a name for your API key.
  5. In Role, select Admin or Editor to associate the role with this API key.
  6. Click Copy to save it for later use.

Add resources using playbooks

Add a data source

The following steps use the InfluxDB data source. The required arguments vary depending on the type of data source you select.

Create a file named data-source.yml:

- name: Add/Update data source
  connection: local
  hosts: localhost

  vars:
    grafana_url: 'https://<STACK_NAME>.grafana.net'
    grafana_api_key: '<GRAFANA_API_KEY>'
    data_source_config:
      name: '<DATA_SOURCE_NAME>'
      type: 'influxdb'
      url: '<DATA_SOURCE_URL>'
      user: '<USERNAME>'
      secureJsonData:
        password: '<PASSWORD>'
      database: '<DATABASE_NAME>'
      uid: '<UID>'
      access: 'proxy'

  tasks:
    - name: Create/Update Data source
      grafana.grafana.datasource:
        dataSource: '{{ data_source_config }}'
        grafana_url: '{{ grafana_url }}'
        grafana_api_key: '{{ grafana_api_key }}'
        state: present

Replace the placeholders with your values:

Add a folder

This playbook creates a folder in your Grafana instance using the Folder module.

Create a file named folder.yml:

- name: Add/Update Folders
  connection: local
  hosts: localhost

vars:
    grafana_url: 'https://<STACK_NAME>.grafana.net'
    grafana_api_key: '<GRAFANA_API_KEY>'
    folders:
      - title: '<FOLDER_NAME>'
        uid: '<UID>'

  tasks:
    - name: Create/Update a Folder in Grafana
      grafana.grafana.folder:
        title: '{{ item.title }}'
        uid: '{{ item.uid }}'
        grafana_url: '{{ grafana_url }}'
        grafana_api_key: '{{ grafana_api_key }}'
        state: present
      loop: '{{ folders }}'

Replace the placeholders with your values:

Add a dashboard to the folder

This playbook iterates through the dashboard JSON source code files in the folder referenced in dashboards_path and adds them to the Grafana instance using the Dashboard module.

Create a file named dashboard.yml:

- name: Add/Update Dashboards
  connection: local
  hosts: localhost

  vars:
    grafana_url: 'https://<STACK_NAME>.grafana.net'
    grafana_api_key: '<GRAFANA_API_KEY>'
    dashboards_path: '<PATH_TO_DASHBOARD_FILES>' # Example "./dashboards"

  tasks:
    - name: Find dashboard files
      find:
        paths: '{{ dashboards_path }}'
        file_type: file
        recurse: true
        patterns: '*.json'
      register: files_matched
      no_log: true

    - name: Create list of dashboard file names
      set_fact:
        dashboard_file_names: '{{ dashboard_file_names | default([]) + [item.path] }}'
      loop: '{{ files_matched.files }}'
      no_log: true

    - name: Create/Update a dashboard
      grafana.grafana.dashboard:
        dashboard: "{{ lookup('ansible.builtin.file', item) }}"
        grafana_url: '{{ grafana_url }}'
        grafana_api_key: '{{ grafana_api_key }}'
        state: present
      loop: '{{ dashboard_file_names }}'

Replace the placeholders with your values:

Run the Ansible playbooks

In a terminal, run the following commands from the directory where all of the Ansible playbooks are located.

Create the Grafana Cloud stack:

ansible-playbook cloud-stack.yml

Add a data source to the Grafana stack:

ansible-playbook data-source.yml

Add a folder to the Grafana stack:

ansible-playbook folder.yml

Add a dashboard to the folder in your Grafana stack:

ansible-playbook dashboard.yml

Validate your configuration

After you've run the Ansible playbooks, you can verify the following:

Next steps

You've successfully created a Grafana Cloud stack along with a data source, a folder, and a dashboard using Ansible. Your Grafana infrastructure is now managed through code.

To learn more about managing Grafana with Infrastructure as code:

Automate dashboard provisioning with CI/CD

Managing Grafana dashboards manually can be inefficient and error-prone. With the Grafana Foundation SDK you can define dashboards using strongly typed code, commit them to version control systems, and automatically deploy them using GitHub Actions.

Why automate?

Automating Grafana dashboard deployment eliminates the need for manual dashboard creation and updates, ensuring that dashboards remain consistent across environments.

By defending dashboards as code and managing them through CI/CD such as GitHub Actions, you will gain full version control, making it easy to track changes over time and roll back if needed. This also prevents duplication, as the workflow intelligently checks whether a dashboard exists before deciding to create or update it.

With this fully automated CI/CD pipeline, you can focus on improving your dashboards rather than manually uploading JSON files to Grafana.

{{< youtube id="cFnO8kVOaAI" >}}

You can find the full example source code in the Introduction to the Foundation SDK GitHub repository.

Overview

This guide shows you how to:

  1. Generate a Grafana dashboard as code and format it for Kubernetes-style deployment
  2. Use GitHub Actions to deploy, verify, and update the dashboard

By the end, you'll be able to provision your dashboard as code, and every change to your dashboard will be automatically created or updated in your Grafana instance without manual intervention.

1. Generate the dashboard JSON

Before deploying a dashboard, define it as code using the Grafana Foundation SDK.

Since Grafana exposes a Kubernetes resource compatible API, you need to make some changes to the code to output the dashboard JSON in the appropriate format.

This script:

{{< code >}}

package main

import (
	"encoding/json"
	"log"
	"os"

	"github.com/grafana/grafana-foundation-sdk/go/cog"
	"github.com/grafana/grafana-foundation-sdk/go/common"
	"github.com/grafana/grafana-foundation-sdk/go/dashboard"
)

type DashboardWrapper struct {
	APIVersion string              `json:"apiVersion"`
	Kind       string              `json:"kind"`
	Metadata   Metadata            `json:"metadata"`
	Spec       dashboard.Dashboard `json:"spec"`
}

type Metadata struct {
	Name string `json:"name"`
}

func main() {
	builder := dashboard.NewDashboardBuilder("My Dashboard").
		Uid("my-dashboard").
		Tags([]string{"generated", "foundation-sdk", "go"}).
		Refresh("5m").
		Time("now-1h", "now").
		Timezone(common.TimeZoneBrowser).
		WithRow(dashboard.NewRowBuilder("Overview"))

	dashboard, err := builder.Build()
	if err != nil {
		log.Fatalf("failed to build dashboard: %v", err)
	}

	dashboardWrapper := DashboardWrapper{
		APIVersion: "dashboard.grafana.app/v1",
		Kind:       "Dashboard",
		Metadata: Metadata{
			Name: *dashboard.Uid,
		},
		Spec: dashboard,
	}

	dashboardJson, err := json.MarshalIndent(dashboardWrapper, "", "  ")
	if err != nil {
		log.Fatalf("failed to marshal dashboard: %v", err)
	}

	err = os.WriteFile("dashboard.json", dashboardJson, 0644)
	if err != nil {
		log.Fatalf("failed to write dashboard to file: %v", err)
	}

	log.Printf("Dashboard JSON:\n%s", dashboardJson)
}

import { DashboardBuilder, RowBuilder } from '@grafana/grafana-foundation-sdk/dashboard';
import * as fs from 'fs';

// Generate the dashboard JSON
const dashboard = new DashboardBuilder('My Dashboard')
  .uid('my-dashboard')
  .tags(['generated', 'foundation-sdk', 'typescript'])
  .refresh('5m')
  .time({ from: 'now-1h', to: 'now' })
  .timezone('browser')
  .withRow(new RowBuilder('Overview'))
  .build();

// Convert to Kubernetes-style format
const dashboardWrapper = {
  apiVersion: "dashboard.grafana.app/v1",
  kind: "Dashboard",
  metadata: {
    name: dashboard.uid!
  },
  spec: dashboard
};

// Save the formatted JSON to a file
const dashboardJSON = JSON.stringify(dashboardWrapper, null, 2);
fs.writeFileSync('dashboard.json', dashboardJSON, 'utf8');

console.log(`Dashboard JSON:\n${}`);

{{< /code >}}

2. Automate deployment with GitHub Actions

Next, set up GitHub Actions to automate the deployment of a Grafana dashboard using the Foundation SDK and the gcx CLI tool to:

{{< admonition type="note" >}} The following GitHub Action configuration assumes you are using a Go-based dashboard generator. If you're using one of the other languages that the Foundation SDK supports, modify the Generate Dashboard JSON step accordingly. {{< /admonition >}}

The .github/workflows/deploy-dashboard.yml deploy workflow looks like:

name: Deploy Grafana Dashboard

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: 1.24.6

      - name: Verify Go version
        run: go version

      - name: Download and Extract gcx
        run: |
          curl -L -o gcx-x86_64.tar.gz "https://github.com/grafana/gcx/releases/download/${{ vars.GCX_VERSION }}/gcx_Linux_x86_64.tar.gz"
          tar -xzf gcx-x86_64.tar.gz
          chmod +x gcx
          sudo mv gcx /usr/local/bin/gcx

      - name: Generate Dashboard JSON
        working-directory: ./github-actions-example
        run: go run main.go

      - name: Deploy Dashboard with gcx
        env:
          GRAFANA_SERVER: ${{ vars.GRAFANA_SERVER }}
          GRAFANA_STACK_ID: ${{ vars.GRAFANA_STACK_ID }}
          GRAFANA_TOKEN: ${{ secrets.GRAFANA_TOKEN }}
        run: |
          if [ -f dashboard.json ]; then
            echo "dashboard.json exists, deploying dashboard."
            gcx resources push dashboards --path ./dashboard.json
          else
            echo "dashboard.json does not exist."
            exit 1
          fi
        working-directory: ./github-actions-example

1. Checkout and set up Go

To set up Go:

2. Download and install gcx

Next, download the gcx CLI from GitHub using the version defined in vars.GCX_VERSION. It unpacks the tarball, makes it executable, and moves it to a location in the system PATH.

3. Generate the dashboard JSON

Next, run the dashboard generator (main.go) from the ./github-actions-example director to produce a dashboard.json file that contains the Grafana dashboard definition.

4. Deploy the dashboard with gcx

If dashboard.json already exists, it is deployed to your Grafana instance using:

gcx resources push dashboards --path ./dashboard.json

This command authenticates against Grafana using the following environment variables:

GitHub variables and secrets used

Verify these variables are configured in your repository under Settings > Security > Secrets and variables > Actions:

This action ensures that every push to main will regenerate and deploy your latest dashboard definition to Grafana.

Foundation SDK key concepts

The Grafana Foundation SDK is built around a few core concepts that make your dashboards structured, reusable, and strongly typed.

Builders

The SDK follows a builder pattern, which lets you compose dashboards step-by-step using chained method calls. Almost every piece of the dashboard, including dashboards, panels, rows, queries, and variables, has its own Builder class.

Here are a few you've already seen:

Builders can be chained, so you can fluently compose dashboards in a readable, structured way:

{{< code >}}

stat.NewPanelBuilder().
  Title("Version").
  Datasource(testdataRef).
  ReduceOptions(common.NewReduceDataOptionsBuilder().
    Calcs([]string{"lastNotNull"}).
    Fields("/.*/")).
  WithTarget(
    testdata.NewDataqueryBuilder().
      ScenarioId("csv_content").
      CsvContent("version\nv1.2.3"),
  )

new stat.PanelBuilder()
  .title('Version')
  .reduceOptions(new common.ReduceDataOptionsBuilder().calcs(['lastNotNull']).fields('/.*/'))
  .datasource(testdataRef)
  .withTarget(new testdata.DataqueryBuilder().scenarioId('csv_content').csvContent('version\nv1.2.3'));

{{< /code >}}

Types

The Grafana Foundation SDK uses strong types under the hood to help catch mistakes before you deploy a broken dashboard.

For example:

This helps you:

Strong typing also makes it easier to build reusable patterns and components with confidence, especially in large codebases or teams.

Options

Most builder methods accept simple values like strings or numbers, but others expect more structured option objects. These are used for things like:

Example using options:

{{< code >}}

stat.NewPanelBuilder().
  ReduceOptions(common.NewReduceDataOptionsBuilder().
    Calcs([]string{"lastNotNull"}).
    Fields("/.*/"))
  )

new stat.PanelBuilder().reduceOptions(new common.ReduceDataOptionsBuilder().calcs(['lastNotNull']).fields('/.*/'));

{{< /code >}}

By using option builders, you don't need to manually construct deeply nested configuration objects. Instead, the SDK gives you a typed and guided API that mirrors a dashboards internal structure, making it easier to configure complex options without guesswork or referring back to the JSON schema.

Git Sync key concepts

{{< admonition type="note" >}}

Git Sync functionalities are constantly evolving. Contact Grafana for support or to report any issues you encounter and help us improve this feature.

{{< /admonition >}}

Key Git Sync components

Before you start using Git Sync, understand how the key Git Sync components relate:

Grafana instance

A Grafana instance is a running Grafana server. Multiple instances can:

Git repository

A Git repository is the external storage you want to sync your Grafana instance with. You can organize your Git repository in several ways:

Flexible configuration for your Git Sync repositories

Git Sync repositories support different combinations of repository URL, branch, and path:

Git Sync repository resource

A repository resource is a Grafana configuration object that defines the connection between a group of repositories and your Grafana instance via Git Sync.

Each repository resource creates bidirectional synchronization between a Grafana instance and a specific location in Git.

Git Sync connection resource

A connection is the authentication setup between Grafana and an external Git provider, required when you're not using a Personal Access Token or a static token to authenticate. Use it to authorize access to your external repositories, and to generate or refresh the credentials for Git Sync. A single connection can be reused across multiple repositories.

For example, if you're using GitHub App to authenticate, the connection represents the app installation. Grafana uses this connection to authenticate with GitHub, create access tokens, and authorize repository access on your behalf.

The connection resource includes:

How does Git Sync behave?

Git Sync is bidirectional, and syncs a repository resource with your Grafana instance. You can modify provisioned resources both from the Grafana UI or from the synced GitHub repository, and changes will be reflected in both places:

  1. Grafana monitors the specified Git location (repository, branch, and path).
  2. Grafana creates a folder in Dashboards (typically named after the repository).
  3. Grafana creates dashboards from dashboard JSON files in Git within this folder.
  4. Grafana commits dashboard changes made in the UI back to Git.
  5. Grafana pulls dashboard changes made in Git and updates dashboards in the UI.
  6. Synchronization occurs at regular intervals (configurable), or instantly if you use webhooks.

You can find the provisioned dashboards organized in folders under Dashboards.

Sync targets

With Git Sync you can place synced resources in Grafana in two ways:

Use folder sync to keep each repository's resources grouped together under a dedicated folder. Use folderless sync when you want provisioned resources to appear at the top of your Dashboards view instead of nested inside a repository folder.

Both modes can coexist with each other and with resources that aren't managed by Git Sync.

The following examples use the same repository to show how the same files appear with each mode.

The repository grafana-manifests syncs from the path grafana/:

your-org/grafana-manifests/
└── grafana/
    ├── cpu-metrics.json
    └── team-platform/
        ├── _folder.json
        └── memory-usage.json

The _folder.json file stores the team-platform folder's stable UID and display name, so the folder keeps its identity and permissions if you move or rename it in the repository. Refer to The Git Sync folder metadata file for details.

The instance also has content that isn't managed by Git Sync: a manually created Ops folder and an Ad-hoc dashboard.

With folder sync, a repository folder wraps the synced resources, alongside the unprovisioned content:

Dashboards
├── 📁 grafana-manifests/      ← managed by Git Sync
│   ├── CPU Metrics Dashboard
│   └── 📁 team-platform/
│       └── Memory Usage Dashboard
├── 📁 Ops/                    ← not managed by Git Sync
│   └── Ops dashboard
└── Ad-hoc dashboard           ← not managed by Git Sync

With folderless sync, the same files map to the top level, next to the unprovisioned content:

Dashboards
├── CPU Metrics Dashboard      ← managed by Git Sync
├── 📁 team-platform/          ← managed by Git Sync
│   └── Memory Usage Dashboard
├── 📁 Ops/                    ← not managed by Git Sync
│   └── Ops dashboard
└── Ad-hoc dashboard           ← not managed by Git Sync

Folderless sync only manages the resources it provisions. The unprovisioned Ops folder and Ad-hoc dashboard are left untouched.

Multiple folderless repositories

Because folderless sync doesn't create a wrapper folder, several folderless repositories can sync to the top level at the same time. Each repository manages only the resources it provisions:

Dashboards
├── CPU Metrics Dashboard      ← managed by grafana-manifests
├── 📁 team-platform/          ← managed by grafana-manifests
│   └── Memory Usage Dashboard
├── Billing Overview           ← managed by finance-dashboards
├── 📁 invoices/               ← managed by finance-dashboards
│   └── Monthly Invoices
└── Ad-hoc dashboard           ← not managed by Git Sync

Git Sync states

Your Grafana instance can be in one of the following Git Sync states:

Example: Relationship between repository, branch, and path

Here's an example showing how the repository, branch, and path concepts work together:

Configuration:

In Git (on branch main):

your-org/grafana-manifests/
├── .git/
├── README.md
├── team-platform/
│   └── grafana/
│       ├── cpu-metrics.json       ← Synced
│       ├── memory-usage.json      ← Synced
│       └── disk-io.json           ← Synced
├── team-data/
│   └── grafana/
│       └── pipeline-stats.json    ← Not synced (different path)
└── other-files.txt                ← Not synced (outside path)

In the Grafana Dashboards view:

Dashboards
└── 📁 grafana-manifests/
    ├── CPU Metrics Dashboard
    ├── Memory Usage Dashboard
    └── Disk I/O Dashboard

Key takeaways:

Configure Git repository protection

For Git Sync to work correctly you need to configure permissions at two layers:

When you use Git Sync, your dashboard configurations are stored as code in a Git repository. You can configure Git repository protection to control who can access this source code and who can modify it. Repository protection works as an additional security layer after Grafana internal permissions.

Required permissions at the Git provider level

Git Sync authentication credentials must have specific permissions at your Git provider:

Required for all configurations:

Required for writing changes:

Optional for instant synchronization:

Refer to the Git Sync setup documentation for detailed instructions on configuring authentication for your Git provider.

Control access to the dashboards source code

You can access dashboard source code through two paths, each with its own protection mechanism:

Protect both access points to secure your dashboard configurations.

Manage access to dashboard source code via the API

View dashboard source code

You can view dashboard source code through the files endpoint if you have Viewer permission or higher on the dashboard or its parent folder. This allows you to:

Edit dashboard source code

You can modify dashboard source code through the files endpoint if you have Editor or Admin permission on the dashboard or its parent folder. This allows you to:

When you save changes through this endpoint, Git Sync commits those changes to the Git repository (or creates pull requests if branch protection is enabled), subject to Git repository permissions.

For detailed information about configuring folder and dashboard permissions in Grafana, refer to Git Sync permissions and access control.

Control access to your Git repository

Access control at the Git provider level determines who can view and modify your dashboard source code. Configure repository access based on your security and compliance requirements.

Read access (repository visibility)

Read access controls who can view the dashboard source code stored in your repository. This includes dashboard JSON files, folder structure, and any other files in the repository.

Public repositories: Anyone can view the repository contents, including dashboard configurations and any data or queries they contain. Only use public repositories if your dashboards contain no sensitive information.

Private repositories: Only authorized users can view repository contents. This protects dashboard configurations, queries, and any embedded credentials or sensitive data from public access.

For Git Sync to function, the authentication credentials configured in Grafana must have read access to pull dashboard changes from Git to Grafana.

Write access (push permissions)

Write access controls who can push changes to your repository. This determines who can modify dashboard source code, either directly or through pull requests.

Direct write access: Users with write permission can push commits directly to branches. For Git Sync to push dashboard changes from Grafana to Git, the authentication credentials must have write access to the repository.

Protected branches: Branch protection rules can restrict direct writes and require changes to go through pull requests with review and approval, even for users with write access.

Refer to the Git Sync setup documentation for detailed instructions on configuring authentication for your Git provider.

Control how changes are written in Git Sync

Git Sync supports different modes for writing dashboard changes to your repository, from allowing direct commits to requiring formal review processes.

Read-only mode

Configure the repository as read-only in Grafana to prevent any writes to Git from the Grafana UI. Dashboards sync from Git to Grafana, but users cannot save changes back to Git.

Use when: Git is the single source of truth and all changes must be made through direct Git commits or CI/CD processes.

Direct commit mode

When the repository allows writes and branch protection is not enabled, Git Sync commits dashboard changes directly to the configured branch without review.

Use when: Rapid iteration is needed and changes don't require formal review, such as in development environments.

Pull request mode

When branch protection is enabled at your Git provider, Git Sync prompts you to go create pull requests in your Git repository instead of committing directly. Changes require review and approval before merging to the main branch.

Use when: Changes require review and approval, such as in production environments or when multiple teams collaborate on dashboards.

Protect your branch

Branch protection rules at your Git provider enforce how changes are made to specific branches. When enabled on the branch that Git Sync targets, these rules require Git Sync to create pull requests instead of pushing commits directly.

Common use cases for branch protection:

Branch protection can enforce various controls such as requiring pull requests before merging, setting reviewer approval requirements, running automated validation checks, preventing force pushes, and restricting who can push directly to protected branches.

Refer to your Git provider's documentation for specific instructions on configuring branch protection rules.

Code review assignments

Many Git providers support CODEOWNERS files that automatically assign reviewers to pull requests based on which files are changed. When Git Sync creates a pull request, the Git provider can use the CODEOWNERS file to assign the appropriate team or users for review.

This ensures dashboard changes are reviewed by the teams responsible for those dashboards, based on folder path or file patterns.

Refer to your Git provider's documentation for instructions on configuring CODEOWNERS files.

Troubleshooting

For permissions troubleshooting refer to Troubleshoot permissions in the Git Sync permissions documentation.

Git Sync permissions and access control

For Git Sync to work correctly you need to configure permissions at two layers:

Grafana role-based permissions

Git Sync integrates with the Grafana standard role-based permission model, which has three levels:

  1. Organization-level: Default permissions of the Admin, Editor, or Viewer role. Refer to Roles and permissions for more details.
  2. Folder-level: They also apply to all dashboards within it. Refer to Folder permissions for more details.
  3. Dashboard-level: Refer to Dashboard permissions for more details.

{{< admonition type="note" >}}

Dashboard-level permissions override folder-level permissions, which override organization-level roles.

{{< /admonition >}}

The following applies for Git Sync:

Use org-level permissions in Git Sync

If you apply org-level permissions, users can do the following with Git Sync:

Admin users

If Grafana OSS/Enterprise, you need to have the Admin role to set up and manage Git Sync repositories and connections. In Grafana Cloud, the equivalent role is Grafana Cloud Admin or Admin at the organization level.

As an Admin you can:

Editor users

Users with the Editor role can work with provisioned dashboards and folders. Their specific capabilities depend on the folder-level and dashboard-level permissions assigned to them.

Organization-level capabilities:

{{< admonition type="note" >}} Only Admins can trigger a manual sync (pull from Git) and orphan-resource clean-up. Editors can push their own changes but can't pull from Git on demand. Refer to Job actions and required permissions for more details. {{< /admonition >}}

Resource access depends on folder/dashboard permissions:

Editors don't need access to the Provisioning admin UI or repository configuration. Refer to Configure folder and dashboard permissions and Configure fine-grained access control (RBAC) for details.

Viewer users

Users with the Viewer role can view provisioned resources. Their access to specific dashboards and folders depends on the permissions assigned to them.

Organization-level capabilities:

Resource access depends on folder and dashboard permissions:

Configure folder and dashboard permissions

Folder-level role permissions determine who can view, edit, or delete provisioned resources.

These roles grant Grafana permissions (dashboards:read, dashboards:write, folders:create...) that are checked when users interact with provisioned resources through the Git Sync files endpoint. Dashboards within a provisioned folder inherit the folder's permissions.

When Git Sync creates a provisioned folder, it assigns these default permissions:

Grafana Role Folder Permission
Admin Admin
Editor Editor
Viewer Viewer

Folder-level Viewer users:

Folder-level Editor users:

Folder-level Admin users:

Modify folder-level permissions

Folder permissions in Grafana are attached to a folder's UID, not to its path in the Git repository. Git Sync stores this UID in the folder's _folder.json metadata file (in the metadata.name field). For the metadata file schema and why it exists, refer to The Git Sync folder metadata file.

Because permissions reference the folder UID, keep the following in mind:

{{< admonition type="note" >}} To safely modify permissions, each provisioned folder must include a _folder.json metadata file with the folder's UID. Without this file, the folder's UID is derived from its repository path, so folder permissions can be lost if the folder is moved to a different path in the Git repository.

For folders created from the Grafana UI, the metadata file is added automatically. If your folder is missing the metadata file, the UI shows a warning with instructions on how to add it. Refer to The Git Sync folder metadata file for details about this file and why it exists. {{< /admonition >}}

You can set folder permissions manually or as code. In both cases, apply them only after Git Sync has created the folders.

Set permissions manually

Set permissions with Terraform

You can manage folder permissions as code with the Grafana Terraform provider. Because Git Sync—not Terraform—creates the folders, reference each folder by the UID from its _folder.json file (the metadata.name value). Apply the permissions only after Git Sync has synced the folders, so the folder UIDs already exist.

Use grafana_folder_permission_item to manage a single permission entry. This is additive, so it only manages the entries you declare and leaves other permissions untouched:

# The UID comes from the folder's _folder.json metadata file ("metadata.name").
locals {
  team_platform_folder_uid = "team-platform-abc123"
}

resource "grafana_folder_permission_item" "team_platform_editor" {
  folder_uid = local.team_platform_folder_uid
  team       = grafana_team.platform.id # an existing team
  permission = "Edit"
}

resource "grafana_folder_permission_item" "team_platform_viewer" {
  folder_uid = local.team_platform_folder_uid
  role       = "Viewer"
  permission = "View"
}

Alternatively, use grafana_folder_permission to declare the full set of permissions for a folder in a single resource. This resource is authoritative: any permission that isn't listed is removed when you apply the configuration.

{{< admonition type="note" >}} Don't manage these folders with the grafana_folder resource—they're owned by Git Sync. To avoid hardcoding UIDs, read them from the _folder.json files in your repository, or look them up with a grafana_folder data source. {{< /admonition >}}

Configure fine-grained access control (RBAC)

If you're a Grafana Enterprise or Grafana Cloud user with RBAC enabled, Git Sync provides fine-grained permissions that allow more granular control over Git Sync operations. You can create custom roles that combine Git Sync permissions in different ways. Refer to Role-Based Access Control documentation for instructions on creating and managing custom roles.

How basic roles map to RBAC permissions

Understanding which permissions each basic role receives helps you create custom roles or understand exactly what access users have. The tables below show which Git Sync permissions are granted to each role by default.

{{< admonition type="note" >}} The provisioning.settings:read permission is granted to all roles (Viewer and above) and allows viewing Git Sync system settings, which is necessary for the UI to display properly. This does not grant access to modify settings or manage repositories. {{< /admonition >}}

Admin role

Users with the Admin role receive full access to Git Sync infrastructure:

Permission Category Specific Permissions What This Allows
Repositories provisioning.repositories:create
provisioning.repositories:read
provisioning.repositories:write
provisioning.repositories:delete		 Create new repositories
View repository configurations
Update repository settings (branch, path, interval)
Delete repositories
Connections provisioning.connections:create
provisioning.connections:read
provisioning.connections:write
provisioning.connections:delete		 Create Git provider connections
View connection details
Update connection settings
Delete connections
Jobs provisioning.jobs:create
provisioning.jobs:read
provisioning.jobs:write
provisioning.jobs:delete		 Trigger manual syncs
View sync jobs
Modify sync job settings
Cancel/delete sync jobs
History & Monitoring provisioning.historicjobs:read
provisioning.stats:read		 View sync job history
View Git Sync statistics and metrics
Settings provisioning.settings:read View Git Sync system settings

Editor role

Users with the Editor role can manage sync operations but not infrastructure configuration:

Permission Category Specific Permissions What This Allows
Jobs provisioning.jobs:create
provisioning.jobs:read
provisioning.jobs:write
provisioning.jobs:delete		 Push changes and open pull requests
View sync jobs
Modify sync job settings
Cancel/delete sync jobs
Read-Only Access provisioning.repositories:read
provisioning.settings:read		 View repository configurations
View Git Sync settings

{{< admonition type="note" >}} provisioning.jobs:create allows editors to create jobs to push their changes, subject to the relevant dashboards:* and folders:* permissions. However, it doesn't allow admin-only job actions such as a manual pull from Git or orphan-resource cleanup, which require the provisioning.repositories:write permission. Refer to Job actions and required permissions for more details. {{< /admonition >}}

Editors can access resources based on the folder/dashboard assignments:

Viewer role

Users with Viewer role have read-only access to Git Sync:

Permission Category Specific Permissions What This Allows
Read-Only Access provisioning.repositories:read
provisioning.settings:read		 View repository configurations
View Git Sync settings (required for UI)

Viewers can access resources based on the folder/dashboard assignments:

None role

The None basic role, also called No basic role in the UI, doesn't have any Git Sync permissions by default, as it doesn't inherit the default permissions of Viewer, Editor, or Admin. Without explicit grants, as a None user you can't read Git Sync settings or repository configurations, configure repositories, submit jobs, or read and modify provisioned dashboards and folders. In any case, you can use RBAC to assign specific permissions to your None users.

RBAC for dashboards and folders

Provisioned dashboards and folders use the Grafana standard permission model. To modify provisioned resources, you will need permissions over your dashboard and folder.

The following applies for Git Sync:

Job actions and required permissions

Git Sync operations run as jobs. You need the provisioning.jobs:create permission to create any job. Moreover, some job actions are restricted to administrators and require the provisioning.repositories:write permission. This prevents editors from triggering repository-wide operations even though they hold provisioning.jobs:create.

Job action Required permission Who can run it
Push changes / open pull request provisioning.jobs:create plus the relevant dashboards:* / folders:* permissions Editors and Admins, on the resources they can modify
Migrate resources provisioning.jobs:create plus read/write on the affected resource types Editors and Admins with the required resource permissions
Manual sync (pull from Git) provisioning.repositories:write Admins only
Release / delete orphaned resources provisioning.repositories:write Admins only

Repository subresource access

The repository API exposes several subresources. The following table shows the permission each one is gated on.

The refs subresource lists the repository's branches and commits, and two distinct flows legitimately need it:

Because the repositories resource has no Editor tier (repositories:read is granted to Viewer and above, while write, create, and delete are admin-only), refs accepts either of these checks, and viewers satisfy neither.

Subresource Purpose Required permission Who can access it
files Read and write provisioned resource files Authenticated access, then standard dashboards:* / folders:* checks All authenticated users
refs List repository branches and commits provisioning.repositories:write or provisioning.jobs:create Admins and Editors
resources, history, status Repository management and inspection views provisioning.repositories:write Admins only

{{< admonition type="note" >}} The management and inspection views (resources, history, status) are gated on provisioning.repositories:write rather than repositories:read, because there's no admin-only read action on the repositories resource. As a result, only users who can manage a repository can inspect its management views; Viewers and Editors can't access these views. {{< /admonition >}}

Configure Git repository protection

After you've configured your Grafana permissions, set up the appropriate permissions at your Git provider to write changes. Repository protection settings control write access, branch protection rules, and code review requirements.

For detailed information about configuring repository write access and branch protection, refer to Repository protection for Git Sync.

Troubleshoot permissions

"Permission denied" when saving a dashboard

Cause: User lacks Editor or Admin permission on the provisioned folder.

Solution:

  1. Verify the user's folder-level permissions in Grafana
  2. Navigate to Folder settings > Permissions
  3. Grant the user or their team Editor or Admin role

Git Sync fails with "403 Forbidden" or "Unauthorized"

Cause: The Git provider authentication credentials lack the required repository permissions.

Solution:

  1. Verify the authentication credentials (GitHub App, Personal Access Token) have read and write permissions on the repository
  2. If using a GitHub App or OAuth app, verify it is installed and authorized for the target repository
  3. Check that the credentials have permission to create pull requests (if branch protection is enabled)
  4. Verify authentication credentials haven't expired
  5. For expired or revoked tokens, generate new credentials and update the Git Sync connection configuration

Dashboard changes commit directly instead of creating pull requests

Cause: Branch protection is not configured at the Git provider.

Solution: Enable branch protection rules at your Git provider to enforce pull request workflows or in your Grafana repository settings. Refer to your Git provider's documentation for instructions on configuring branch protection.

Dashboard changes commit directly without review

Cause: Branch protection is not configured at the Git provider.

Solution:

  1. Enable branch protection on the target branch at your Git provider
  2. Configure the branch to require pull requests before merging
  3. Verify the branch name in protection rules matches the branch configured in Grafana

Pull requests not created when expected

Cause: Branch protection is not enabled or the authentication credentials lack pull request creation permission.

Solution:

  1. Verify branch protection is enabled on the correct branch
  2. Check that the credentials have permission to create pull requests
  3. Ensure the branch name in Git Sync settings matches the protected branch exactly

Work with provisioned dashboards in Git Sync

{{< admonition type="note" >}}

Git Sync functionalities are constantly evolving. Contact Grafana for support or to report any issues you encounter and help us improve this feature.

{{< /admonition >}}

Using provisioning, you can choose to store your dashboard JSON files in either Git repositories using Git Sync or a local path, and manage them through the Grafana interface. Dashboards and folders synchronized using Git Sync or a local path are referred to as provisioned resources. Git Sync is the recommended method for provisioning your dashboards.

Manage dashboards provisioned with Git Sync

Using Git Sync, you can manage your dashboards in the UI and synchronize them with your configured Git repository. If you push a change in the repository, those changes are mirrored in your Grafana instance.

Refer to Key concepts for more details.

Create a new dashboard

{{< admonition type="note" >}}

It may take a few minutes for your changes to reflect on your screen. If they don't, refresh the UI manually.

{{< /admonition >}}

You have the following options to add new dashboards in your Git Sync connection:

If you want to add an existing dashboard to your provisioned resources, refer to Add non-provisioned resources from Grafana.

Add a new dashboard from the Grafana UI

To add a new dashboard from the Grafana UI:

  1. On the Dashboards tab, click on New on the top right corner and chose one of these options:
    • New dashboard. When you create a new dashboard in a provisioned folder associated with a Git repository, you follow the same process you use for any new dashboard. Refer to Create a dashboard for more information.
    • Import dashboard. Refer to Add a dashboard using Import dashboards for more details.
    • Use template. Create a dashboard using a Grafana template.

After you create the dashboard, follow these steps:

  1. Click Save dashboard.
  2. On the Provisioned dashboard panel, choose the options you want to use:
    • Title: The title of the dashboard.
    • Description: Description of the dashboard.
    • Target folder: The folder where you want to store the new dashboard.
    • Branch: Specify the branch name in your Git provider (for example, main).
    • Repository folder: The folder inside the repository. Leave empty for the repository root.
    • Filename: The name of the dashboard file.
    • Comment: Add a comment describing your changes for you commit.
  3. Optional: Select the Changes tab to view the differences between the updates you made and the original resource.
  4. Select Save, and the updated dashboard will load.
  5. On the prompt, select Open a pull request in GitHub to open a new PR to your repository.
  6. Follow your usual Git workflow to save and merge the PR to your repository.

Edit dashboards

When you edit a provisioned resource, you're prompted to save or discard those changes. Saving changes requires opening a pull request in your Git repository.

To save dashboard changes:

  1. Select Edit to update a provisioned dashboard.
  2. Make your desired changes, and click Save dashboard.
  3. On the Provisioned dashboard panel, choose the options you want to use:
    • Branch: Specify the branch name in your Git provider (for example, main).
    • Repository folder: The folder inside the repository. Leave empty for the repository root.
    • Filename: The name of the dashboard file.
    • Comment: Add a comment describing your changes for your commit.
  4. Optional: Select the Changes tab to view the differences between the updates you made and the original resource.
  5. Select Save, and the updated dashboard will load.
  6. On the prompt, select Open a pull request in GitHub to open a new PR to your repository.
  7. Follow your usual Git workflow to save and merge the PR to your repository.

Preview banner

This section applies to Git repository connections (GitHub, GitHub Enterprise, GitLab, Bitbucket, and plain Git). It doesn't apply to local file provisioning, which has no branches or pull requests.

Whenever you're viewing a provisioned dashboard loaded from a branch other than your repository's configured branch, Grafana shows a preview banner at the top of the page. You'll see it right after you create, edit, delete, or move a dashboard from the Grafana UI, as well as when you open a dashboard from a pull request notification.

The banner's title reflects what changed:

In every case, the rest of your organization keeps seeing the version on the configured branch until the branch is merged — what you're looking at is a private preview of your own change. The banner also shows the branch your change was pushed to and the branch it targets.

From the banner you can:

If the branch behind the preview no longer exists in the repository, for example, the pull request was already merged or closed and its source branch deleted, selecting Open a pull request can't take you to a live pull request. Grafana instead shows This branch no longer exists and offers two ways forward:

{{< admonition type="note" >}}

How much of the banner is interactive depends on your repository type:

Provisioned folders show a related banner, This resource is behind the branch, when the folder itself is out of sync with its configured branch. Select Open in repository to view the latest changes.

{{< /admonition >}}

Remove dashboards

You can remove a provisioned dashboard either from the Grafana UI or from the repository:

To restore a deleted dashboard, revert the deletion (or re-add the file) directly in your Git repository and raise a PR as usual. Restoring resources from the UI isn't possible at the moment.

Document folders with a README

You can document the contents or any other relevant piece of information of your provisioned folder in a README.md file stored alongside its resources in the repository. Grafana renders the README inline on the folder page, below the list of dashboards, so your team can see what's in the folder, how it's organized, and where to find the right dashboard without leaving Grafana.

Common uses include:

{{< admonition type="note" >}}

It may take a few minutes for your changes to reflect on your screen. If they don't, refresh the UI manually.

{{< /admonition >}}

Browse multiple documents with tabs

A folder can hold more than one Markdown document. When it does, Grafana shows a tab for each document above the rendered content, similar to how a Git provider presents documentation files.

Use this to keep related documentation next to the dashboards it describes, such as a runbook, an onboarding guide, or contribution and ownership details in separate documents.

Navigate to resources from Markdown links

When a link doesn't match a provisioned resource, Grafana opens the file in the host repository, so the same README still works when viewed on GitHub or GitLab.

Render Mermaid diagrams

Grafana renders Mermaid diagrams defined in your Markdown documents. Use them to embed architectural diagrams, incident runbook flows, or data pipeline overviews next to the dashboards that monitor them. Use a fenced code block with the mermaid language to include a diagram:

```mermaid
flowchart LR
  A[Ingest] --> B[Store]
  B --> C[Query]
  C --> D[Dashboard]
```

Grafana replaces the code block with the rendered diagram and matches it to your light or dark theme.

Best practices

Follow these recommendations when working with provisioned dashboards:

Refer to Work with provisioned repositories for general guidance about using Git Sync.

Usage and performance limitations

{{< admonition type="note" >}}

Git Sync functionalities are constantly evolving. Contact Grafana for support or to report any issues you encounter and help us improve this feature.

{{< /admonition >}}

Performance considerations

When Git Sync is enabled, the database load might increase, especially if your Grafana instance has many folders and nested folders. Evaluate the performance impact, if any, in a non-production environment.

Usage tiers and limits

The following Git Sync per-tier limits apply:

Tier Cloud - Free Cloud - Other On-prem OSS On-prem Enterprise
Amount of repositories 1 10 10 (default) 10 (default)
Amount of synced resources per repository 20 1,000 No limit No limit

{{< admonition type="note" >}}

On Grafana Cloud, tier limits can't be changed from configuration. Contact Support if you need a higher connection limit, or shard by capacity instead of using a single connection per team or microservice.

On self-managed Grafana (OSS or Enterprise), the repository count of 10 in the table is the default, not a hard ceiling. Raise or remove it with [provisioning] max_repositories in the Grafana configuration file. On-prem resources per repository stay unlimited by default (max_resources_per_repository = 0), and you can set a cap if you want one.

Refer to Modify your usage limits for more details.

{{< /admonition >}}

On Grafana Cloud, the repository-connection limit is a per-stack limit. On self-managed Grafana, the effective limit is whatever you set for max_repositories (default 10).

Do not sync more than 1,000 resources per repository connection as of today. This isn't an arbitrary cap: beyond roughly 1,000 resources per connection, the sync workflow puts noticeable load on Grafana itself, which may result in slower syncs and increased database load.

Both of these limits are early figures. As Git Sync matures, these limits will raise by orders of magnitude. The goal is for Git Sync to support around 100 repository connections per stack, and up to roughly 1,000 in the longer term, with similar increases to the number of resources you can sync per connection.

If these limits are affecting your use of Git Sync, get in touch and explain your situation. You may also share any ideas or suggestions.

For details on usage and storage limits, refer to Dashboard and folder limits.

Modify your usage limits

Before changing your usage limits, study your specific use case. Design the repository structure carefully, and determine how many repositories and how many resources you can support. For example, setting over 1,000 resources per repository may impact your system's performance.

In many cases, splitting a single repository across multiple connections, as described in Scale beyond 1,000 resources per repository, is a better option than raising the limits.

Grafana Cloud

On Grafana Cloud, limits are enforced per tier and can't be edited from configuration. The 10-connection limit can be increased slightly on request, but splitting a single repository across multiple connections is the recommended way to sync more resources without changing tier limits.

Limit increases aren't granted automatically. When you request one, Support will ask you to describe your use case to understand why the current limits aren't enough, and assess whether the increase is necessary and safe for your stack's performance.

Grafana OSS/Enterprise

In Grafana On-prem (OSS or Enterprise), the table value of 10 repositories is only the default, not a hard ceiling. Resources per repository remain unlimited by default. The 1,000-resources-per-connection figure is a performance recommendation, not an on-prem default cap.

If you're an on-prem user you can change the limits directly through configuration settings, as described below.

To customize the limits use one of these configuration settings:

Scale beyond 1,000 resources per repository

If a single repository holds more than 1,000 resources, you don't have to raise the per-repository limit. Instead, connect the same repository multiple times, with each connection pointing to a different folder (path) in the repository. Each connection syncs its own subset of resources and counts toward the 1,000-resource recommendation independently.

Because a stack allows up to 10 repository connections, this lets you sync up to roughly 10,000 resources from a single repository while keeping each connection within the recommended range.

When you split a single repository across several connections, use folderless sync for each connection. With folder sync (the default), every connection creates its own wrapper folder named after the repository, so splitting the repository would change your folder hierarchy and nest resources one level deeper. Folderless sync places each connection's resources at the top level, so the split has no effect on how your dashboards and folders are organized in Grafana.

If you need more than that, the 10-connection limit can be increased slightly for Cloud stacks on request. Contact Support to discuss your use case.

On Grafana Cloud, syncing this many resources may also exceed your stack's maximum number of dashboards. If it does, you'll need to increase that limit as well. Refer to Dashboard and folder limits to review and adjust your dashboard limits.

Shard by capacity, not by team

When you have many teams or tenants, it's tempting to create one connection per team so each team maps to its own connection. Avoid this: it consumes connections quickly, doesn't scale as teams grow, and on Grafana Cloud a single stack can't be granted the hundreds of connections this would require.

Instead, shard by capacity. Create one repository and group teams into a small number of shard folders, each holding up to about 1,000 resources, and connect each shard folder separately. For example, a customer with 190 teams and 900 resources fits comfortably in a single shard today:

your-org/grafana-manifests/
├── shard-1/        ← ~900 resources today, connected now
│   ├── team-a/
│   ├── team-b/
│   └── ...
├── shard-2/        ← empty for now, add a connection when shard-1 approaches 1,000
├── shard-3/        ← add later as you keep growing
└── shard-4/        ← ...up to 10 shards / connections per stack

As the number of resources grows, add shard-2, shard-3, and later shards, and connect each one. You can move teams between shards at any time to balance the load, so you only pay for the connections you actually need and can grow up to the 10-connection limit without restructuring your repository.

If sharding isn't practical for your setup, try raising the per-connection resource limit modestly, for example from 1,000 to a limit in the 1,200 - 1,500 range. This is a small adjustment for a bit of extra headroom, not an order-of-magnitude increase. Still, do not go beyond 1,500 resources per connection because of the performance impact on Grafana. For substantially larger scales, sharding remains the recommended approach.

Git Sync across multiple organizations

Git Sync works across multiple organizations for self-managed Grafana instances. You can set up Git Sync independently in each of your organizations, and what you sync in one organization does not affect another. Teams that share a single Grafana instance across separate organizations can each manage their own provisioning from Git.

This feature is available starting in Grafana 13.0.4, but not supported until Grafana 13.1.1.

Compatible Git providers

Git Sync is available for any Git provider through a Pure Git repository type, and has specific enhanced integrations for GitHub, GitLab and Bitbucket.

Provider Available in Authentication
Pure Git Cloud, OSS, Enterprise Personal Access Token or OAuth App
GitHub Cloud, OSS, Enterprise Personal Access Token, GitHub App, or OAuth App
GitHub Enterprise (Server and Cloud) Cloud, Enterprise Personal Access Token, GitHub App, or OAuth App
GitLab Cloud, Enterprise Personal Access Token or OAuth App
Bitbucket Cloud, Enterprise API token with scopes or OAuth App

Note that Pure Git, GitLab and Bitbucket are supported in Grafana v12.4.x or later only. Refer to Enable Git providers to set them up. To make OAuth Apps available in self-managed Grafana, refer to Enable OAuth connection types.

To learn more about Git, refer to Getting Started - About Version Control of the Pro Git book in the official Git documentation.

The Pure Git repository type

The Pure Git repository type uses the Smart HTTP protocol v2 (Git over HTTPS), with no provider-specific logic. Pure Git delivers the core Git Sync workflow: your repository is the source of truth, you may edit dashboards in the UI, and Grafana stays in sync.

{{< admonition type="note" >}}

Pure Git only supports Smart HTTP protocol v2. Earlier protocol versions (v1, v0) and SSH transport are not supported.

Make sure your Git server supports protocol v2 over HTTPS. Some providers, like Azure DevOps, only use v1 and are therefore not compatible with Git Sync.

{{< /admonition >}}

However, Pure Git doesn't include any features that require provider APIs, such as webhook-driven instant sync, automated PR comments, or deep links to source files.

Enhanced integrations: GitHub, GitHub Enterprise, GitLab, Bitbucket

If your Git provider is GitHub, GitHub Enterprise (Server and Cloud), GitLab, or Bitbucket, use the enhanced integration. Enhanced integrations understand the platform you're using, allowing workflows that feel native: automated pull request comments with dashboard previews, instant webhook-based sync, or direct navigation from Grafana to source files in your provider's UI.

The GitHub enhanced integration is the most feature-complete experience today. It's available in Grafana OSS, Enterprise, and Cloud, and enables richer pull request workflows, deeper linking between Grafana and GitHub, and tighter integration into review processes.

The GitLab and Bitbucket integrations are only available in Grafana Enterprise and Grafana Cloud. While they're less developed for the moment, expect continued improvements around pull request workflows, linking, and sync behavior.

Resource support and compatibility

Git Sync only supports dashboards and folders. Alerts, data sources, panels and other resources are not supported yet.

If you're a Grafana Cloud user, you can check the Grafana roadmap portal to learn about future improvements.

Resource compatibility

If you're using Git Sync in Grafana OSS or Grafana Enterprise, some supported resources might be in an incompatible data format. If this happens, syncing will be blocked. Compatibility issues will be fixed with an upcoming migration tool.

A resource can be:

Is the resource? Compatible Incompatible
Supported The resource can be managed with Git Sync. The resource is supported but has compatibility issues. It cannot be managed with Git Sync.
Unsupported The resource is not supported and cannot be managed with Git Sync. Not applicable.

Nested folders

Git Sync supports up to four nested folders within a repository.

Known limitations

Migration to Git Sync

Full-instance sync is experimental.

When migrating to resources to Git Sync, you can still create, edit or delete resources, but changes may not be exported. The duration of this process depends on the number of resources involved.

Use existing resources

To add existing resources as provisioned Git Sync resources, you can use the Import dashboards tool, save them from the UI, save them as JSON files and commit them to the synced repository, or use gcx. Refer to Export non-provisioned resources from Grafana for more details. When migrating existing dashboards, the folder structure will be replicated in the provisioned repository.

{{< admonition type="caution" >}}

Git Sync cannot take ownership of a resource while an unmanaged resource with the same UID exists in Grafana, so you may need to manually remove the original resources to ensure the transition to provisioned resources. Since deleting resources implies certain operational caveats, refer to How to delete existing resources in Grafana for more information.

{{< /admonition >}}

Restore resources

Restoring resources from the UI is currently not possible. As an alternative, you can restore dashboards directly in your GitHub repository by raising a PR, and they will be updated in Grafana.

Work with provisioned repositories in Git Sync

{{< admonition type="note" >}}

Git Sync functionalities are constantly evolving. Contact Grafana for support or to report any issues you encounter and help us improve this feature.

{{< /admonition >}}

After you sync your resources, Git Sync creates a dashboard that provides a summary of resources, health, pull status, webhook, sync jobs, resources, and files. To access it, follow these steps:

  1. Log in to your Grafana server with an account that has the Grafana Admin flag set.
  2. Select Administration > General > Provisioning in the left-side menu to access the Git Sync configuration screen.
  3. Go to the Repositories tab, and locate the repository you want to work with. You can view the current status of the sync, run pulls, or update your settings.

Refer to Work with provisioned dashboards for more information about working with provisioned files.

View the current status of your synchronized repository

Use the View section to see detailed information about the current status of your sync and troubleshoot possible issues:

Troubleshoot synchronization

{{< admonition type="note" >}}

Before you proceed to troubleshoot, understand the Usage and performance known limitations.

{{< /admonition >}}

Monitor the View status page for synchronization issues and status updates. The status page displays the following events:

Dashboard sync errors

Dashboard import errors

Synchronize changes

To sync resources between the provisioned repositories and your Grafana instance, click Pull under the repository you want to sync. The synchronization process runs and completes.

Grafana overwrites existing dashboards with the same uid.

Update or delete your settings

To update or delete your repository configuration after you complete setup:

  1. Log in to your Grafana server with an account that has the Grafana Admin flag set.
  2. Select Administration > General > Provisioning.
  3. Go to the Repositories tab, and locate the repository you want to modify.
  4. Select Settings to access the Configure repository screen:

{{< admonition type="note" >}}

It may take a few minutes for your changes to reflect on your screen. If they don't, refresh the UI manually.

{{< /admonition >}}

Manage folder permissions

By default, users keep their roles in folders provisioned with Git Sync.

Grafana Role Folder Permission
Admin Admin
Editor Editor
Viewer Viewer

Refer to Git Sync permissions to understand and set up permissions in Git Sync.

Modify folder permissions

{{< admonition type="note" >}} To modify permissions, each provisioned folder must include the _folder.json metadata file, which gives the folder a stable UID. Without it, the folder's permissions are lost if you move or rename that folder in the Git repository. Refer to The Git Sync folder metadata file for details about this file and why it exists. {{< /admonition >}}

Folder permissions attach to the folder's UID (stored in _folder.json), not to its repository path, so you can only set them after Git Sync has created the folder. To add or modify folder permissions:

The Git Sync folder metadata file

Each provisioned folder in a synced repository contains a _folder.json metadata file at its root. This file is the folder's manifest: it stores the folder's stable UID and its display name, so both survive changes to the repository layout.

The file exists because a directory path isn't a stable identity. Without _folder.json, Grafana derives the folder UID from a hash of the directory path in the repository. If you move or rename that directory in Git, the hash changes and Grafana treats it as a different folder: everything attached to the old UID, such as custom folder permissions, bookmarks, and API references, is lost. With _folder.json, the UID travels with the directory, so you can reorganize your repository without breaking permissions or references.

The file also separates the folder's display name from the directory name. The directory keeps a filesystem-friendly name, for example team-platform, while the spec.title field holds the name shown in the Grafana UI, for example Team Platform.

Grafana manages this file for you:

The file has the following format:

{
  "apiVersion": "<API_VERSION>",
  "kind": "Folder",
  "metadata": {
    "name": "<FOLDER_UID>"
  },
  "spec": {
    "title": "<FOLDER_UI_NAME>"
  }
}

Where:

Cherry-pick and copy individual dashboards

{{< admonition type="note" >}}

At the moment Git Sync only manages dashboards and folders. Alerts, data sources, and library panels are not supported yet.

To migrate your existing dashboards, refer to Migrate existing dashboards.

{{< /admonition >}}

Use these methods to add a copy of one or more dashboards to a folder provisioned with Git Sync. Each copy is created with a new UID, so the original dashboards stay exactly as they are and nothing needs to be deleted. Existing links continue to point to the original dashboards, not the copies.

Add a dashboard using Import dashboards

You can import dashboards directly into your Git Sync provisioned folders using the Grafana UI or the HTTP API.

Import dashboard

To access the Import dashboard tool from the Git Sync UI:

  1. Go to the Dashboards tab of you connection.
  2. On the top right corner, click New.
  3. Select Import dashboard and you'll be redirected to the wizard.
  4. Upload or paste the dashboard JSON.
  5. Fill in the relevant fields, including the branch and repository folder, and press Import.
  6. Open the pull request, follow your regular workflow, and merge. Note that it could take a few minutes until the imported dashboard appears.

Keep in mind the following:

For more information refer to Import dashboards in the Data Visualization documentation.

{{< admonition type="note" >}}

It may take a few minutes for your changes to reflect on your screen. If they don't, refresh the UI manually.

{{< /admonition >}}

Copy an existing dashboard from the Grafana UI

You can also save a copy of dashboard directly from the Grafana UI to your provisioned folder. This creates a new dashboard with a new UID and leaves the original in place.

To do so, follow these steps:

  1. Make sure the dashboard is in Editable mode.
  2. Select Save or Save as from the top-right corner.
  3. In the menu:
    • Target folder: Select the provisioned folder from your Grafana UI where you want to save the dashboard in.
    • Branch: Type in the name of the branch of the provisioned repository you want to work in, or create a new branch. Committing directly to main is not supported.
    • Folder: Type in the folder in your sync repository, if any.
    • Fill in the rest of the fields accordingly.
  4. Click Save.
  5. In your synced GitHub repository, merge the branch with the dashboard you want to sync.

Migrate existing dashboards to Git Sync

{{< admonition type="note" >}}

At the moment Git Sync only manages dashboards and folders. Alerts, data sources, and library panels are not supported yet. Refer to Before you begin for details.

To copy a specific dashboard, refer to Cherry-pick and copy individual dashboards.

{{< /admonition >}}

If you migrate your existing non-provisioned dashboards under Git Sync, dashboards keep their UIDs, so existing links, references, and bookmarks keep working. Because the UID is preserved, Git Sync adopts the resource in place, and you must delete the original resource so Git Sync can take over its UID.

The migration follows these steps:

  1. Before you begin: Back up your instance and understand what Git Sync manages.
  2. Step 1: Export the resources to your repository: Export with UIDs preserved and commit.
  3. Step 2: Delete the original dashboards: Required so Git Sync can adopt each dashboard by UID.
  4. Step 3: Validate the migration: Confirm the resources are synced before moving on.

Before you begin

Because migrating involves deleting resources, read this section and plan carefully before you start.

To migrate safely, keep in mind the following:

Manage and delete migrated resources

With a folder or folderless sync, Git Sync recreates dashboards and folders:

Refer to Step 2: Delete the original dashboards for more details.

{{< admonition type="note" >}}

Full-instance migrations have different cleanup behavior and can delete unmanaged folders, so follow the full-instance migration guidance instead.

{{< /admonition >}}

Keep your original resources and set them apart

After a migration you'll have your original folder (holding any alerts and library panels) alongside the new Git Sync folder of the same name. To avoid confusion, rename your original folders or move them under a single top-level Alerts & Library Panels folder. This keeps the unsupported resources intact and clearly separated from the provisioned dashboards.

Step 1: Export the resources to your repository

Export the dashboards you want to migrate so that each file keeps the dashboard's original UID (metadata.name), then commit the files to your Git repository. You can use either of the following:

Export with the Grafana CLI

You can export existing dashboards from the terminal or from agentic coding tools using the gcx CLI. With gcx you can download the resources you want to sync from Grafana, and then commit and push those files to your provisioned Git repository. Git Sync will then detect the commit, and synchronize with Grafana.

{{< admonition type="note" >}}

For more information refer to the gcx documentation.

{{< /admonition >}}

To export dashboards with gcx, follow these steps:

  1. Set up the gcx context to point to your instance as documented in Defining contexts.

  2. Pull the resources you want to sync from the instance to your local repository:

    gcx resources pull dashboards --path <REPO_PATH>

  3. Commit and push the resources to your Git repository:

    git add <DASHBOARDS_PATH>
git commit -m "Add dashboards from Grafana"
git push

    Where:

    • <GIT_REPO>: The path to the repository synced with Git Sync
    • <DASHBOARDS_PATH>: The path where the dashboards you want to export are located. The dashboards path must be under the repository

After you commit the resources, continue to Step 2.

Export as a JSON resource file

To export a dashboard as a JSON resource file, you need to:

  1. Export the dashboard as JSON.
  2. Convert it to the Custom Resource Definition (CRD) format required by the Grafana App Platform.
  3. Commit the converted file to your Git repository.

After you commit the file, continue to Step 2.

To export a dashboard as a JSON file it must follow this CRD structure:

{
  'apiVersion': 'dashboard.grafana.app/v1',
  'kind': 'Dashboard',
  'metadata': { 'name': 'dcf2lve9akj8xsd' },
  'spec': { /* Original dashboard JSON goes here */ },
}

The structure includes:

Step 2: Delete the original dashboards

Delete each original dashboard you're migrating so Git Sync can take over its UID. Since the exported files keep the original UID, Git Sync can't provision any dashboard if an unmanaged dashboard with the same UID (metadata.name) still exists in Grafana.

When you delete a dashboard, keep in mind the following:

Manage migrated folders

You don't need to delete your migrated folders, as Git Sync creates its own folders with new, path-derived UIDs. Don't delete or recreate folders that contain alert rules or library panels, since those resources are deleted permanently and Git Sync does not recreate them. See Before you begin for how to keep and set apart your original folders.

Step 3: Validate the migration

  1. Trigger a new pull to complete the sync. The dashboards are recreated as provisioned, with their original UIDs, so existing links keep working.
  2. Confirm each dashboard appears in the provisioned folder and opens correctly. It may take a few minutes for changes to appear; if they don't, refresh the UI manually.
  3. Confirm that the alert rules and library panels in your original folders are still present and working.

After you've validated one folder, repeat the process for the next one until the migration is complete.

By default, Git Sync gives each synced folder a new UID derived from its path in the repository. This means links and URLs that point to your original folders keep pointing to the original folders, not the new provisioned ones.

If you need existing folder links and URLs to resolve to the provisioned folders instead, you can pin a folder's UID with a folder metadata file so Git Sync reuses the original folder's UID.

Save unsupported resources

Because this option reuses the original folder's UID, the synced folder collides with your existing unmanaged folder, and Git Sync can't take over a UID that still belongs to an unmanaged folder, with the sync failing with a conflict.

To avoid losing unsupported resources, complete these steps for each folder before you sync:

  1. Create a new folder and move all alert rules, library panels, and other unsupported resources out of the original folder into it.
  1. Delete the original folder. Its dashboards should already be exported to the repository from Step 1.
  2. Reapply any custom permissions on the new provisioned folder, as folder permissions don't carry over. Refer to Git Sync permissions and access control.

Reuse the original folder's UID

{{< admonition type="note" >}}

Folder metadata requires the provisioningFolderMetadata feature, which is enabled by default. If your administrator has disabled it, metadata.name is ignored and folders always get a path-derived UID.

{{< /admonition >}}

To reuse an original folder's UID, add a _folder.json file to that folder's directory in the repository:

{
  "apiVersion": "folder.grafana.app/v1beta1",
  "kind": "Folder",
  "metadata": { "name": "<ORIGINAL_FOLDER_UID>" },
  "spec": { "title": "<FOLDER_TITLE>" }
}

Where <ORIGINAL_FOLDER_UID> is the UID of your existing folder. You can find it in the folder's URL.

Before you begin

Before you begin to set up Git Sync, ensure you have the following:

Enable required feature toggles

The provisioning feature toggle is enabled by default in Grafana Cloud and, starting in Grafana v13, for OSS and Enterprise as well. No manual configuration is required.

For more information about feature toggles, refer to Configure feature toggles.

Enable Git providers

If you're using Grafana Enterprise v12.4.0 and want to set up Git Sync with pure Git, GitLab or Bitbucket, or if you're using Grafana OSS v12.4.0 and want to set up Git Sync with pure Git, add them to your configuration file:

  1. Open your Grafana configuration file, either grafana.ini or custom.ini.

  2. Add the available providers:

    [provisioning]
repository_types = "git|github|bitbucket|gitlab|local"

  3. Save the changes to the file and restart Grafana.

Enable OAuth connection types

If you're using self-managed Grafana and the OAuth App connection type for your provider isn't available, add it to connection_types in your configuration file:

  1. Open your Grafana configuration file, either grafana.ini or custom.ini.

  2. Add the connection types:

    [provisioning]
connection_types = "github|githubOAuth|gitOAuth"

  3. Save the changes to the file and restart Grafana.

The available OAuth connection types are:

Connection type Provider Available in
githubOAuth GitHub OSS, Enterprise
gitOAuth Pure Git OSS, Enterprise
githubEnterpriseOAuth GitHub Enterprise Enterprise
gitlabOAuth GitLab Enterprise
bitbucketOAuth Bitbucket Enterprise

Network connectivity and IP allowlisting

Git Sync requires network connectivity between your Grafana instance and Git server. Understanding the traffic patterns helps you configure firewall rules and allowlists correctly.

Traffic types

Git Sync uses two types of network traffic:

AWS PrivateLink and Private Data Source Connect

Git Sync doesn't route over AWS PrivateLink or Private Data Source Connect (PDC). Instead, Git Sync uses the normal public path from the Hosted Grafana IPs, which is independent from PrivateLink and PDC. AWS PrivateLink and PDC provide a separate tunnel for data source query traffic, from Grafana to your private databases or data sources.

If you use AWS PrivateLink or PDC for data sources, you can still use Git Sync. The two features neither interfere with nor depend on each other.

Allow internal or private Git servers

{{< admonition type="note" >}} This setting is available for Grafana v13.0.4 and Grafana v13.1.1 and later. It only applies to self-managed Grafana (OSS and Enterprise), but it's not configurable in Grafana Cloud. {{< /admonition >}}

While public Git servers such as github.com, gitlab.com, and bitbucket.org resolve to public addresses and are always allowed, by default Git Sync rejects repository URLs with a host that resolves to a loopback, a private (RFC 1918), link-local, or an unspecified address. This protects your Grafana instance against server-side request forgery (SSRF).

If you connect Git Sync to a Git server on a private network such as a self-hosted GitHub Enterprise, GitLab, or Bitbucket instance reachable only through an internal address, add its host to the allowed_git_urls allowlist:

  1. Open your Grafana configuration file, either grafana.ini or custom.ini.

  2. Add each internal Git host to allowed_git_urls as a comma-separated list:

    [provisioning]
allowed_git_urls = git.internal.example.com, ghe.example.com:8443

  3. Save the changes to the file and restart Grafana.

Each entry can be a hostname, host:port, a full URL (only the host is used), a literal IP address, or a CIDR range. If possible, use specific hosts or narrow ranges, since a broad CIDR such as 10.0.0.0/8 re-exposes the entire private range that SSRF protection blocks.

Only add hosts you trust because an allowlisted host receives the configured Git token on every sync, fetch, and push.

Resource and role permissions

By default, folders provisioned with Git Sync have these roles:

Refer to Git Sync permissions for details on how to set up permissions in Git Sync. To modify them, refer to Manage folder permissions.

Refer to Roles and permissions for more information about Grafana roles.

Create a GitHub App

GitHub Apps are tools that extend GitHub functionality. They use fine-grained permissions and short-lived tokens, giving you more control over which repositories are being accessed. Find out more in the GitHub Apps official documentation.

If you chose to authenticate with a newly created GitHub App, you'll need the following parameters:

There are many ways to create a GitHub App. The following instructions are informative only, always refer to official GitHub documentation for more details.

To create the GitHub App, follow these steps:

  1. Go to https://github.com/settings/apps and click on New Github App, or navigate directly to https://github.com/settings/apps/new
  2. Fill in the following fields:
    • Name: Must be unique
    • Homepage URL: For example, your Grafana Cloud instance URL
  3. Scroll down to the Webhook section and uncheck the Active box
  4. In the Permissions section, go to Repository permissions and set these parameters:
    • Administration: Read-only permission (enables validation of branch protection rules against the configured branch when users can push directly to it; may be used in the future to check other repository settings and make the setup process smoother)
    • Contents: Read and write permission
    • Metadata: Read-only permission
    • Pull requests: Read and write permission
    • Webhooks: Read and write permission
  5. Finally, under Where can this GitHub App be installed?, select Only on this account
  6. Click on Create Github App to complete the process.

On the app page:

  1. Copy the AppID from the About section
  2. Select the Generate private key from the banner or scroll down to to the Private Keys section to generate a key
  3. A PEM file containing your private key will be downloaded to your computer

Finally, install the app:

  1. At the top left of the App page, click on Install App
  2. Choose for which user you need to install it, you’ll be redirected to the repository selection screen
  3. Choose for which repositories you want to install the app
  4. Click Install.
  5. On the installation page, copy installationID from the page URL https://github.com/settings/installations/installationID

You can now proceed to Set up Git Sync!

Create an OAuth App

An OAuth App lets Grafana act on your behalf in your Git provider. You create the app in your provider, then authorize it once from Grafana. Grafana stores the access token and refreshes it when the provider supports refresh tokens.

{{< admonition type="caution" >}} An OAuth App connection acts as the user who authorized it. Every repository that uses the connection and every user who can run its sync jobs will use the access token of the user that authorized the connection. The token can reach every repository that user can access, not only those connected to Grafana. Removing the user from Grafana doesn't revoke the token. Revoke it in your Git provider.

To limit access, authorize the app with a dedicated account that can only access the repositories you sync. For GitHub, consider a GitHub App, which is scoped to the repositories where it's installed. {{< /admonition >}}

If you chose to authenticate with an OAuth App, you need the following parameters:

Every OAuth App needs the Grafana callback URL. Grafana shows it in the setup wizard, and it has this format:

<GRAFANA_URL>/admin/provisioning/connections/oauth-callback

Replace <GRAFANA_URL> with the root URL of your Grafana instance, including any sub-path.

The following instructions are informative only. Always refer to the official documentation of your Git provider for more details.

GitHub and GitHub Enterprise

To create a GitHub OAuth App, follow these steps:

  1. In GitHub, go to Settings > Developer settings > OAuth apps and click New OAuth App. For GitHub Enterprise, use the same menu on your enterprise instance.
  2. Enter an Application name and a Homepage URL, for example your Grafana instance URL.
  3. Paste the Grafana callback URL in Authorization callback URL.
  4. Click Register application.
  5. Copy the Client ID, then click Generate a new client secret and copy the secret.

Grafana requests the repo scope when you authorize the app. For more details, refer to Creating an OAuth app.

GitLab

To create a GitLab OAuth application, follow these steps:

  1. In GitLab, create the application for your user or for a group:
    • For your user, select your avatar, then Edit profile > Access > Applications > Add new application.
    • For a group, go to the group and select Settings > Applications.
  2. Enter a Name and paste the Grafana callback URL in Redirect URI.
  3. Keep Confidential selected, and select the api scope.
  4. Click Save application.
  5. Copy the Application ID and the Secret.

For more details, refer to Configure GitLab as an OAuth 2.0 authentication identity provider.

Bitbucket

To create a Bitbucket OAuth consumer, follow these steps:

  1. In Bitbucket, go to Workspace settings > Apps and features > OAuth consumers and click Add consumer.
  2. Enter a Name and paste the Grafana callback URL in Callback URL.
  3. Set these permissions:
    • Repositories: Read and write permission
    • Pull requests: Read and write permission
    • Webhooks: Read and write permission
  4. Click Save.
  5. Select the consumer name to show the Key and the Secret, copy them, and note the name of the workspace.

For more details, refer to Use OAuth on Bitbucket Cloud.

Other Git providers

For any other Git provider that supports the OAuth 2.0 authorization code flow, create an OAuth application with the Grafana callback URL as the redirect URI. In addition to the client ID and secret, you need:

Both URLs must use https://, unless you set allow_insecure in the [provisioning] section of your configuration file.

Next steps

For further details on how Git Sync operates, refer to:

Set up Git Sync as code

You can also configure Git Sync using gcx, the Grafana CLI. Since Git Sync configuration is managed as code using Custom Resource Definitions (CRDs), you can create your required resources in YAML files and push them to Grafana using gcx. This approach enables automated, GitOps-style workflows for managing Git Sync configuration instead of using the Grafana UI.

For more information, refer to the following documents:

To provision resources from a local file system as code instead of a Git repository, refer to Set up file provisioning as code.

Set up Git Sync as code with the Grafana CLI

To set up Git Sync as code with gcx, follow these steps:

  1. Understand Usage and performance limitations
  2. Create the connection and repository CRDs
  3. Push the CRDs to Grafana
  4. Manage repository resources
  5. Verify setup

Create the resources CRDs

If you're connecting with any of the supported Git providers using a Personal Access Token, you need to create a repository resource to define the connection between your repositories and your Grafana instance.

If you're connecting to Git Sync with GitHub App or an OAuth App, in addition to the repository resource you need to create a connection resource as well.

Create the connection resource

If you're connecting Git Sync with GitHub App, create a connection.yaml file defining your Git Sync connection configuration:

apiVersion: provisioning.grafana.app/v0alpha1
kind: Connection
metadata:
  name: '<GITHUB_CONNECTION_NAME>'
  namespace: default
spec:
  title: '<REPOSITORY_TITLE>'
  type: github
  url: https://github.com
  github:
    appID: '<GITHUB_APP_ID>'
    installationID: '<GITHUB_INSTALL_ID>'
    serverUrl: '<GITHUB_ENTERPRISE_SERVER_URL>' # Only required for GitHub Enterprise
secure:
  privateKey:
    create: '<GITHUB_PRIVATE_KEY>'

Replace the placeholders with your values:

Create an OAuth App connection resource

If you're connecting Git Sync with an OAuth App, create a connection.yaml file with the client ID and secret of the app, and the fields for your provider:

apiVersion: provisioning.grafana.app/v0alpha1
kind: Connection
metadata:
  name: '<OAUTH_CONNECTION_NAME>'
  namespace: default
spec:
  title: '<CONNECTION_TITLE>'
  type: '<CONNECTION_TYPE>' # githubOAuth, githubEnterpriseOAuth, gitlabOAuth, bitbucketOAuth, or gitOAuth
  oauth:
    clientID: '<CLIENT_ID>'
  # GitHub Enterprise only:
  githubEnterpriseOAuth:
    serverUrl: '<GITHUB_ENTERPRISE_SERVER_URL>'
  # Bitbucket only:
  bitbucket:
    workspace: '<BITBUCKET_WORKSPACE>'
  # Pure Git only:
  gitOAuth:
    authURL: '<AUTHORIZATION_URL>'
    tokenURL: '<TOKEN_URL>'
    scopes:
      - '<SCOPE>'
secure:
  clientSecret:
    create: '<CLIENT_SECRET>'

Replace the placeholders with your values:

An OAuth App connection needs a user to approve access in the browser. After you push the connection, it shows as Disconnected until you authorize it:

  1. Select Administration > General > Provisioning and go to the Connections tab.
  2. Select the connection and click Reauthorize.
  3. Approve the access in the provider tab that opens.

Create the repository resource

Next, create a repository.yaml file defining your Git Sync configuration. Depending on your Git provider and authentication method, add your Personal Access Token information or the connection name.

apiVersion: provisioning.grafana.app/v0alpha1
kind: Repository
metadata:
  name: '<REPOSITORY_NAME>'
spec:
  sync:
    enabled: true
    intervalSeconds: 60
    target: folder
  workflows:
    - write
    - branch
  title: '<REPOSITORY_TITLE>'

# Git Sync for GitHub:
spec:
  type: github
  github:
    url: '<GIT_REPO_URL>'
    branch: '<BRANCH>'
    path: grafana/
  # GitHub App or OAuth App connection only:
  connection:
    name: '<GITHUB_CONNECTION_NAME>'
# GitHub Personal Access Token only:
secure:
  token: { create: 'GIT_PAT' }

# Git Sync for GitHub Enterprise:
spec:
  type: githubEnterprise
  githubEnterprise:
    url: '<GIT_REPO_URL>'
    branch: '<BRANCH>'
    path: grafana/
  # GitHub Enterprise App connection only:
  connection:
    name: '<GITHUB_ENTERPRISE_CONNECTION_NAME>'
# GitHub Personal Access Token only:
secure:
  token: { create: 'GIT_PAT' }

# GitLab Personal Access Token only:
spec:
  type: gitlab
  gitlab:
    url: '<GIT_REPO_URL>'
    branch: '<BRANCH>'
secure:
  token: { create: 'GIT_PAT' }

# Bitbucket Personal Access Token only:
spec:
  type: bitbucket
  bitbucket:
    url: '<GIT_REPO_URL>'
    branch: '<BRANCH>'
    tokenUser: tokenuser
secure:
  token: { create: 'GIT_PAT' }

# Pure Git only:
spec:
  type: git
  git:
    url: '<GIT_REPO_URL>'
    branch: '<BRANCH>'
    path: 'grafana/'
    tokenUser: tokenuser
secure:
  token: { create: 'GIT_PAT' }

To use an OAuth App connection, set spec.connection.name instead of secure.token. The repository type must match the connection type: githubOAuth for github, githubEnterpriseOAuth for githubEnterprise, gitlabOAuth for gitlab, bitbucketOAuth for bitbucket, and gitOAuth for git.

Replace the placeholders with your values:

{{< admonition type="note" >}}

Git Sync supports two sync targets: target: folder (the default) creates a folder named after the repository and places synced resources inside it, while target: folderless places synced resources at the top level without creating a wrapper folder. Refer to Sync targets for details.

{{< /admonition >}}

Configuration parameters

The following configuration parameters are available:

Field Description
metadata.name Unique identifier for this repository resource
spec.title Human-readable name displayed in Grafana UI
spec.type Repository type (github, githubEnterprise)
spec.github.url GitHub repository URL
spec.github.branch Branch to sync
spec.github.path Directory path containing dashboards
spec.github.generateDashboardPreviews Generate preview images (true/false) (Only available in GitHub)
spec.sync.enabled Enable synchronization (true/false)
spec.sync.intervalSeconds Sync interval in seconds
spec.sync.target Where to place synced dashboards (folder or folderless)
spec.workflows Enabled workflows: write (direct commits), branch (PRs)
secure.token.create GitHub Personal Access Token

Push the resources to Grafana

Before pushing any resources, configure gcx with your Grafana instance details. Refer to the Grafana CLI documentation for setup instructions.

Push the repository configuration. If you're using GitHub App to connect Git Sync, push the connection resource configuration file as well.

gcx resources push --path <DIRECTORY>

The --path parameter has to point to the directory containing your repository.yaml and connection.yaml files.

After pushing, Grafana will:

  1. Create the required resources (repository and, for GitHub App, connection)
  2. Connect to your GitHub repository
  3. Pull dashboards from the specified path
  4. Begin syncing at the configured interval

Manage repository resources

List repositories

To list all repositories:

gcx resources get repositories

Get repository details

To get details for a specific repository:

gcx resources get repository/<REPOSITORY_NAME>
gcx resources get repository/<REPOSITORY_NAME> -o json
gcx resources get repository/<REPOSITORY_NAME> -o yaml

Update the repository

To update a repository:

gcx resources edit repository/<REPOSITORY_NAME>

Delete the repository

To delete a repository:

gcx resources delete repository/<REPOSITORY_NAME>

Verify setup

Check that Git Sync is working:

# List repositories
gcx resources get repositories

# Check Grafana UI
# Navigate to: Administration → Provisioning → Git Sync

Set up instantaneous pulling and dashboard previews in Pull Requests

After setup, you can optionally extend Git Sync by enabling pull request notifications and image previews of dashboard changes.

Capability Benefit Requires
A table summarizing changes to your pull request A convenient way to save changes back to GitHub Webhooks configured
A dashboard preview image to a PR A snapshot of dashboard changes to a pull request outside Grafana Image renderer and webhooks configured

Set up webhooks for real-time notification and pull request integration

Real-time notifications (or automatic pulling) is enabled and configured by default in Grafana Cloud.

In Grafana OSS/Enterprise, Git Sync uses webhooks to enable real-time updates from GitHub public repositories, or to enable pull request integrations. Without webhooks the polling interval is set during configuration, and is 60 seconds by default. You can set up webhooks with whichever service or tooling you prefer: Cloudflare Tunnels with a Cloudflare-managed domain, port-forwarding and DNS options, or a tool such as ngrok.

To set up webhooks:

  1. Expose your Grafana instance to the public Internet.
    • Use port forwarding and DNS, ngrok, or any other method you prefer.
    • The permissions set in your GitHub access token provide the authorization for this communication.
  2. After you have the public URL, add it to your Grafana configuration file: 
    [server]
root_url = https://<PUBLIC_DOMAIN>
  3. Replace <PUBLIC_DOMAIN> with your public domain.

To check the configured webhooks, go to Administration > General > Provisioning and click the View link for your GitHub repository.

Webhook limits

Git Sync registers its own webhooks for each repository connection, so when several Grafana instances sync the same repository, each connection adds to this total. Since Git providers limit the amount of webhooks you can set for each repository, when the limit is exceeded, your Git provider rejects new webhooks.

For example, in GitHub you can get an error such as:

GitHub API error (HTTP 422: Validation Failed: The "pull_request" event cannot have more than 20 hooks; The "push" event cannot have more than 20 hooks)

If you hit this limit, you can either:

Webhook limitations per provider

The following limits apply:

Set a publicly-reachable URL

If you need to point [server] root_url to an internal address (for example, when Grafana runs behind a private ingress in a Kubernetes cluster), set the publicly-reachable URL with [provisioning] public_root_url instead. This URL is used both to register webhook callbacks with the Git provider and as the base for screenshot images embedded in pull-request comments, which the Git provider's servers fetch from the public internet.

[server]
root_url = http://internal.cluster.local

[provisioning]
public_root_url = https://<PUBLIC_DOMAIN>

The per-repository spec.webhook.baseUrl field still overrides public_root_url for webhook registration, and screenshot URLs always use public_root_url (or root_url when unset).

Expose necessary paths only

If your security setup doesn't permit publicly exposing the Grafana instance, you can either choose to allowlist the Git provider's IP addresses, or expose only the necessary paths.

For information about the traffic between Grafana and your Git server, refer to Network connectivity and IP allowlisting.

In RegExp, the necessary paths required to be exposed are /apis/provisioning\.grafana\.app/v0(alpha1)?/namespaces/[^/]+/repositories/[^/]+/(webhook|render/.*)$.

Set up image rendering for dashboard previews

{{< admonition type="caution" >}}

Available in Grafana OSS and Grafana Enterprise only.

{{< /admonition >}}

To set up image rendering to add visual previews of dashboard updates directly in pull requests, install the Grafana Image Renderer in your Grafana instance. For more information and installation instructions, refer to the Image Renderer service.

Image rendering requires webhooks.

Next steps

To learn more about using Git Sync refer to the following documents:

Set up Git Sync with Terraform

You can also configure Git Sync via the Grafana provisioning app platform using Terraform.

Before you begin

Before you begin, make sure to have the following:

{{< admonition type="note" >}} Save all of the following Terraform configuration files in the same directory. {{< /admonition >}}

Configure the Grafana provider

Use this Terraform configuration to configure the Grafana provider with the authentication required to set up Git Sync.

  1. Create a service account and token in Grafana. For more information refer to Service account tokens or Create and manage a Grafana Cloud stack using Terraform.

  2. Make sure that the token has the Admin or the Provisioning:Repositories writer permission.

  3. Create a file named main.tf and add the following:

       terraform {
     required_providers {
       grafana = {
         source  = "grafana/grafana"
         version = ">= 4.28.1"
       }
     }
   }

   provider "grafana" {
     url = "https://<your-stack>.grafana.net/"
     auth = var.grafana_service_account_token
     stack_id = var.grafana_stack_id
   }

Replace the following field values:

{{< admonition type="caution" >}}

The Git Sync Terraform resources are App Platform resources and talk directly to the Grafana stack's API.

Do not use Grafana Cloud provider settings in this configuration. Configure the provider with url and auth (not cloud_api_url and cloud_access_policy_token). If you're using Grafana Cloud, set stack_id to your stack ID.

{{< /admonition >}}

Create the resources to use Git Sync

You need two resources for configure and manage Git Sync:

For better understanding of the required resources, refer to Git Sync key concepts.

Git Sync for development and production environments

Use separate Grafana instances for development and production. Each syncs with different Git locations to test dashboards before production.

Use it for

Architecture

┌────────────────────────────────────────────────────────────┐
│              GitHub Repository                             │
│   Repository: your-org/grafana-manifests                 │
│   Branch: main                                             │
│                                                            │
│   grafana-manifests/                                     │
│   ├── dev/                                                │
│   │   ├── dashboard-new.json    ← Development dashboards │
│   │   └── dashboard-test.json                            │
│   │                                                       │
│   └── prod/                                               │
│       ├── dashboard-stable.json  ← Production dashboards │
│       └── dashboard-approved.json                        │
└────────────────────────────────────────────────────────────┘
           ↕                                 ↕
    Git Sync (dev/)              Git Sync (prod/)
           ↕                                 ↕
┌─────────────────────┐          ┌─────────────────────┐
│  Dev Grafana        │          │  Prod Grafana       │
│                     │          │                     │
│  Repository:        │          │  Repository:        │
│  - path: dev/       │          │  - path: prod/      │
│                     │          │                     │
│  Creates folder:    │          │  Creates folder:    │
│  "grafana-manifests"│         │  "grafana-manifests"│
└─────────────────────┘          └─────────────────────┘

Repository structure

In Git:

your-org/grafana-manifests
├── dev/
│   ├── dashboard-new.json
│   └── dashboard-test.json
└── prod/
    ├── dashboard-stable.json
    └── dashboard-approved.json

In Grafana Dashboards view:

Dev instance:

Dashboards
└── 📁 grafana-manifests/
    ├── New Dashboard
    └── Test Dashboard

Prod instance:

Dashboards
└── 📁 grafana-manifests/
    ├── Stable Dashboard
    └── Approved Dashboard

Configuration parameters

Development:

Production:

How it works

  1. Developers create and modify dashboards in development.
  2. Git Sync commits changes to dev/.
  3. You review changes in Git.
  4. You promote approved dashboards from dev/ to prod/.
  5. Production syncs from prod/.
  6. Production dashboards update.

Alternative: Use branches

Instead of using different paths, you can configure instances to use different branches:

Development instance:

Production instance:

With this approach:

Alternative: Use separate repositories for stricter isolation

For stricter isolation, use completely separate repositories:

Development instance:

Production instance:

Git Sync for high availability environments

Primary–replica scenario

Use a primary Grafana instance and one or more replicas synchronized with the same Git location to enable failover.

Use it for

Architecture

┌─────────────────────────────────────────────────────┐
│           GitHub Repository                         │
│   Repository: your-org/grafana-manifests          │
│   Branch: main                                      │
│                                                     │
│   grafana-manifests/                              │
│   └── shared/                                      │
│       ├── dashboard-metrics.json                   │
│       ├── dashboard-alerts.json                    │
│       └── dashboard-logs.json                      │
└─────────────────────────────────────────────────────┘
              ↕                           ↕
        Git Sync (shared/)        Git Sync (shared/)
              ↕                           ↕
┌────────────────────┐          ┌────────────────────┐
│  Master Grafana    │          │  Replica Grafana   │
│    (Active)        │          │    (Standby)       │
│                    │          │                    │
│  Repository:       │          │  Repository:       │
│  - path: shared/   │          │  - path: shared/   │
└────────────────────┘          └────────────────────┘
         │                               │
         └───────────┬───────────────────┘
                     ↓
          ┌──────────────────────┐
          │  Reverse Proxy       │
          │  (Failover)          │
          └──────────────────────┘

Repository structure

In Git:

your-org/grafana-manifests
└── shared/
    ├── dashboard-metrics.json
    ├── dashboard-alerts.json
    └── dashboard-logs.json

In Grafana Dashboards view (both instances):

Dashboards
└── 📁 grafana-manifests/
    ├── Metrics Dashboard
    ├── Alerts Dashboard
    └── Logs Dashboard

Configuration parameters

Both master and replica instances use identical parameters:

Master instance:

Replica instance:

How it works

  1. Both instances stay synchronized through Git.
  2. Reverse proxy routes traffic to primary.
  3. Users edit on primary. Git Sync commits changes.
  4. Both instances pull latest changes to keep replica in sync.
  5. On primary failure, proxy fails over to replica.

Failover considerations

Load balancer scenario

Run multiple active Grafana instances behind a load balancer. All instances sync from the same Git location.

Use it for

Architecture

┌─────────────────────────────────────────────────────┐
│           GitHub Repository                         │
│   Repository: your-org/grafana-manifests          │
│   Branch: main                                      │
│                                                     │
│   grafana-manifests/                              │
│   └── shared/                                      │
│       ├── dashboard-metrics.json                   │
│       ├── dashboard-alerts.json                    │
│       └── dashboard-logs.json                      │
└─────────────────────────────────────────────────────┘
              ↕                           ↕
        Git Sync (shared/)        Git Sync (shared/)
              ↕                           ↕
┌────────────────────┐          ┌────────────────────┐
│  Grafana Instance 1│          │  Grafana Instance 2│
│    (Active)        │          │    (Active)        │
│                    │          │                    │
│  Repository:       │          │  Repository:       │
│  - path: shared/   │          │  - path: shared/   │
└────────────────────┘          └────────────────────┘
         │                               │
         └───────────┬───────────────────┘
                     ↓
          ┌──────────────────────┐
          │  Load Balancer       │
          │  (Round Robin)       │
          └──────────────────────┘

Repository structure

In Git:

your-org/grafana-manifests
└── shared/
    ├── dashboard-metrics.json
    ├── dashboard-alerts.json
    └── dashboard-logs.json

In Grafana Dashboards view (all instances):

Dashboards
└── 📁 grafana-manifests/
    ├── Metrics Dashboard
    ├── Alerts Dashboard
    └── Logs Dashboard

Configuration parameters

All instances use identical parameters:

Instance 1:

Instance 2:

How it works

  1. All instances stay synchronized through Git.
  2. Load balancer distributes incoming traffic across all active instances.
  3. Users can view dashboards from any instance.
  4. When a user modifies a dashboard on any instance, Git Sync commits the change.
  5. All other instances pull the updated dashboard during their next sync cycle, or instantly if webhooks are configured.
  6. If one instance fails, load balancer stops routing traffic to it and remaining instances continue serving.

Important considerations

Git Sync with regional replication

Deploy multiple Grafana instances across regions. Synchronize them with the same Git location to ensure consistent dashboards everywhere.

Use it for

Architecture

┌─────────────────────────────────────────────────────┐
│           GitHub Repository                         │
│   Repository: your-org/grafana-manifests          │
│   Branch: main                                      │
│                                                     │
│   grafana-manifests/                              │
│   └── shared/                                      │
│       ├── dashboard-global.json                    │
│       ├── dashboard-metrics.json                   │
│       └── dashboard-logs.json                      │
└─────────────────────────────────────────────────────┘
              ↕                           ↕
       Git Sync (shared/)         Git Sync (shared/)
              ↕                           ↕
┌────────────────────┐          ┌────────────────────┐
│  US Region         │          │  EU Region         │
│  Grafana           │          │  Grafana           │
│                    │          │                    │
│  Repository:       │          │  Repository:       │
│  - path: shared/   │          │  - path: shared/   │
└────────────────────┘          └────────────────────┘

Repository structure

In Git:

your-org/grafana-manifests
└── shared/
    ├── dashboard-global.json
    ├── dashboard-metrics.json
    └── dashboard-logs.json

In Grafana Dashboards view (all regions):

Dashboards
└── 📁 grafana-manifests/
    ├── Global Dashboard
    ├── Metrics Dashboard
    └── Logs Dashboard

Configuration parameters

All regions:

How it works

  1. All regional instances pull dashboards from shared/.
  2. Any region’s change commits to Git.
  3. Other regions pull updates during the next sync (or via webhooks).
  4. Changes propagate across regions per sync interval.

Considerations

Git Sync in a Grafana instance shared by multiple teams

Use a single Grafana instance with multiple Repository resources, one per team. Each team manages its own dashboards while sharing Grafana.

Use it for

Architecture

┌─────────────────────────┐  ┌─────────────────────────┐
│  Platform Team Repo     │  │  Data Team Repo         │
│  platform-dashboards    │  │  data-dashboards        │
│                         │  │                         │
│  platform-dashboards/   │  │  data-dashboards/       │
│  └── grafana/           │  │  └── grafana/           │
│      ├── k8s.json       │  │      ├── pipeline.json  │
│      └── infra.json     │  │      └── analytics.json │
└─────────────────────────┘  └─────────────────────────┘
           ↕                            ↕
    Git Sync (grafana/)          Git Sync (grafana/)
           ↕                            ↕
        ┌──────────────────────────────────────┐
        │       Grafana Instance               │
        │                                      │
        │  Repository 1:                       │
        │  - repo: platform-dashboards         │
        │  → Creates "platform-dashboards"     │
        │                                      │
        │  Repository 2:                       │
        │  - repo: data-dashboards             │
        │  → Creates "data-dashboards"         │
        └──────────────────────────────────────┘

Repository structure

In Git (separate repositories):

Platform team repository:

your-org/platform-dashboards
└── grafana/
    ├── dashboard-k8s.json
    └── dashboard-infra.json

Data team repository:

your-org/data-dashboards
└── grafana/
    ├── dashboard-pipeline.json
    └── dashboard-analytics.json

In Grafana Dashboards view:

Dashboards
├── 📁 platform-dashboards/
│   ├── Kubernetes Dashboard
│   └── Infrastructure Dashboard
└── 📁 data-dashboards/
    ├── Pipeline Dashboard
    └── Analytics Dashboard

Configuration parameters

Platform team repository:

Data team repository:

How it works

  1. Each team has their own Git repository for complete autonomy.
  2. Each repository resource in Grafana creates a separate folder.
  3. Platform team dashboards sync from your-org/platform-dashboards repository.
  4. Data team dashboards sync from your-org/data-dashboards repository.
  5. Teams can independently manage their repository settings, access controls, and workflows.
  6. All teams can view each other's dashboards in Grafana but edit only their own.

Scale to more teams

Adding additional teams is straightforward. For a third team, create a new repository and configure:

This creates a new "security-dashboards" folder in the same Grafana instance.

Alternative: Shared repository with different paths

For teams that prefer sharing a single repository, use different paths to separate team dashboards:

In Git:

your-org/grafana-manifests
├── team-platform/
│   ├── dashboard-k8s.json
│   └── dashboard-infra.json
└── team-data/
    ├── dashboard-pipeline.json
    └── dashboard-analytics.json

Configuration:

Platform team:

Data team:

This approach provides simpler repository management but less isolation between teams.

Alternative: Different branches per team

For teams wanting their own branch in a shared repository:

Platform team:

Data team:

This allows teams to use Git branch workflows for collaboration while sharing the same repository.

Single instance Git Sync

Use a single Grafana instance synchronized with a Git repository. This is the foundation for Git Sync and helps you understand bidirectional synchronization.

Use it for

Architecture

┌─────────────────────────────────────────────────────┐
│           GitHub Repository                         │
│   Repository: your-org/grafana-manifests          │
│   Branch: main                                      │
│                                                     │
│   grafana-manifests/                              │
│   └── grafana/                                     │
│       ├── dashboard-1.json                         │
│       ├── dashboard-2.json                         │
│       └── dashboard-3.json                         │
└─────────────────────────────────────────────────────┘
                        ↕
              Git Sync (bidirectional)
                        ↕
        ┌─────────────────────────────┐
        │    Grafana Instance         │
        │                             │
        │  Repository Resource:       │
        │  - url: grafana-manifests  │
        │  - branch: main             │
        │  - path: grafana/           │
        │                             │
        │  Creates folder:            │
        │  "grafana-manifests"       │
        └─────────────────────────────┘

Repository structure

In Git:

your-org/grafana-manifests
└── grafana/
    ├── dashboard-1.json
    ├── dashboard-2.json
    └── dashboard-3.json

In Grafana Dashboards view:

Dashboards
└── 📁 grafana-manifests/
    ├── Dashboard 1
    ├── Dashboard 2
    └── Dashboard 3

Configuration parameters

Configure your Grafana instance to synchronize with:

How it works

  1. From Grafana to Git: When users create or modify dashboards in Grafana, Git Sync commits changes to the grafana/ directory on the main branch.
  2. From Git to Grafana: When dashboard JSON files are added or modified in the grafana/ directory, Git Sync pulls these changes into Grafana.

Testing sandboxes with Git Sync

Use a sandbox branch that users can push to directly, without opening a pull request for every change. A sandbox branch is a low-friction space for short-lived projects and experimentation, while keeping every dashboard version-controlled in Git. When work is ready, you promote it to the main branch.

A sandbox is a branch in the same repository, synced to Grafana with the write workflow enabled. With the write workflow, saving a dashboard commits directly to the sandbox branch, so the editing experience feels close to using Grafana without Git Sync. The difference is that every change is still committed to Git, so nothing is lost and history is preserved.

This scenario uses a single shared sandbox branch that everyone works in. A shared sandbox uses only one Repository connection regardless of how many users work in it, which keeps you well within the per-stack limit of 10 repository connections. If you need stricter isolation, you can instead give each user their own sandbox branch. Refer to Individual or shared sandboxes.

Use it for

Architecture

┌────────────────────────────────────────────────────────────┐
│              GitHub Repository                             │
│   Repository: your-org/grafana-manifests                 │
│                                                            │
│   Branch: main       ← Promoted, reviewed work            │
│   └── grafana/                                             │
│       ├── dashboard-stable.json                            │
│       └── dashboard-approved.json                          │
│                                                            │
│   Branch: sandbox    ← Direct pushes, shared experiments  │
│   └── grafana/                                             │
│       ├── alice/dashboard-wip.json                         │
│       └── bob/dashboard-test.json                          │
└────────────────────────────────────────────────────────────┘
                        ↕
              Git Sync (write workflow)
                        ↕
              ┌─────────────────────────┐
              │    Grafana Instance     │
              │                         │
              │  Repository Resource:   │
              │  - branch: sandbox      │
              │  - path: grafana/       │
              │  - workflows: [write]   │
              │                         │
              │  Save = direct commit   │
              │  to sandbox             │
              └─────────────────────────┘

Repository structure

In Git:

your-org/grafana-manifests
├── (branch: main)
│   └── grafana/
│       ├── dashboard-stable.json
│       └── dashboard-approved.json
└── (branch: sandbox)
    └── grafana/
        ├── alice/
        │   └── dashboard-wip.json
        └── bob/
            └── dashboard-test.json

In Grafana Dashboards view (synced to sandbox):

Dashboards
└── 📁 grafana-manifests/
    ├── 📁 alice/
    │   └── Work in Progress Dashboard
    └── 📁 bob/
        └── Test Dashboard

Configuration parameters

Configure the Repository resource to sync with the sandbox branch and enable the write workflow:

In the Repository resource, this looks like:

spec:
  github:
    url: 'https://github.com/your-org/grafana-manifests'
    branch: 'sandbox'
    path: grafana/
  workflows:
    - write

{{< admonition type="note" >}}

If you enable only the write workflow, saving a dashboard always commits directly to the sandbox branch. To also allow users to open pull requests from Grafana, add the branch workflow: workflows: [write, branch]. Refer to Configuration parameters for details.

{{< /admonition >}}

How it works

  1. Create a sandbox branch in your repository, for example sandbox.
  2. Configure a Repository resource that syncs that branch with the write workflow enabled.
  3. Users create and edit dashboards in Grafana, optionally in their own subfolder. Each save commits directly to the sandbox branch.
  4. The experience feels similar to using Grafana without Git Sync, but every change is committed to Git.
  5. When work is ready, you promote it to the main branch. Refer to Promote a sandbox to main.

Individual or shared sandboxes

A sandbox can be shared by everyone or scoped to a single user. Each sandbox is a branch synced through its own Repository connection, so the choice is shaped by the per-stack limit of 10 repository connections. Refer to Usage and performance limitations for details.

Both approaches promote to main the same way. Refer to Promote a sandbox to main.

Protect sandbox branches

Even though sandbox branches allow direct pushes, protect them against destructive operations. We recommend blocking force pushes on sandbox branches in your Git provider.

Force pushes rewrite branch history and can silently discard commits that Git Sync relies on. Blocking them keeps history append-only, so Grafana and Git stay consistent and you can always trace how a dashboard reached its current state.

You typically keep pull requests optional on sandbox branches so users can push directly, while still blocking force pushes.

Promote a sandbox to main

When sandbox work is ready, promote it to the main branch. Because everything already lives in Git, promotion is a standard Git operation. Choose the option that fits your review requirements:

Option 1: Open a pull request from the sandbox branch (recommended)

Use your Git provider to open a pull request from the sandbox branch into main.

  1. In your Git provider, open a pull request from sandbox into main.
  2. Review the changes as a normal Git diff and request approvals as needed.
  3. Merge the pull request. The instance syncing main picks up the promoted dashboards on its next sync.

This keeps a full review and audit trail for everything that reaches main, and is the recommended path when main feeds production.

If you also enable the branch workflow on the sandbox Repository resource, users can open these pull requests directly from Grafana when they save, instead of switching to the Git provider.

Option 2: Cherry-pick or merge selected changes

When only some dashboards are ready, promote them selectively instead of merging the whole branch.

  1. Cherry-pick the relevant commits from the sandbox branch onto a branch based on main, or copy the specific dashboard files.
  2. Open a pull request into main and review it.
  3. Merge to complete the promotion.

Use this when a shared sandbox contains a mix of finished and experimental work.

Option 3: Promote through a separate instance or path

When main is synced to a production instance, treat the sandbox as the development stage of a promotion flow.

  1. Develop in the sandbox instance (synced to the sandbox branch with the write workflow).
  2. Promote approved dashboards to main using a pull request (Option 1) or by copying files into the main path.
  3. The production instance syncing main updates automatically.

This mirrors the development and production environments scenario, with the sandbox branch acting as the development stage.

Option 4: Promote from the Grafana UI

If the main branch is synced to a folder in the same instance (through a second Repository connection) or to a separate instance, users can move a dashboard into it directly from Grafana, without touching Git:

This suits users who prefer working entirely in the UI. Note that these paths commit to main without a pull request, so use them only when main allows direct writes; when main requires review, prefer Option 1.

After promotion, you can delete short-lived sandbox branches, or keep a long-lived shared sandbox and continue iterating.

Learn more

Refer to the following documents to learn more:

Manage Grafana dashboards with GitOps using ArgoCD

This guide shows you how to set up a continuous deployment pipeline using ArgoCD to synchronize your Grafana dashboards with a Git repository. You'll use the Grafana Dashboard Custom Resource provided by the Grafana Operator to manage dashboard configurations declaratively.

Prerequisites

Before you begin, make sure you have the following:

Set up your Git repository

Create a directory structure in your repository to organize your Grafana and dashboard configurations. For this tutorial, create a folder named grafana.

Set up the Grafana Operator

The Grafana Operator allows you to authenticate with the Grafana instance using the Grafana Custom Resource (CR).

Create the Grafana API Token Secret

Store the Grafana API Token in a secret. Create a file named grafana-token.yml in the grafana folder in your Git repository:

apiVersion: v1
kind: Secret
metadata:
  name: grafana-cloud-credentials
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
stringData:
  GRAFANA_CLOUD_INSTANCE_TOKEN: '<GRAFANA_API_KEY>'
type: Opaque

Replace the placeholders with your values:

Configure the Grafana Custom Resource

Set up the connection to your Grafana Cloud instance. Create a file named grafana-cloud.yml in the grafana folder in your Git repository:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: '<GRAFANA_CLOUD_STACK_NAME>'
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
  labels:
    dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
spec:
  external:
    url: https://<GRAFANA_CLOUD_STACK_NAME>.grafana.net/
    apiKey:
      name: grafana-cloud-credentials
      key: GRAFANA_CLOUD_INSTANCE_TOKEN

Replace the placeholders with your values:

Add dashboards to your Git repository

In your grafana directory, create a sub-folder called dashboards.

This guide shows you how to create three separate dashboards. For all dashboard configurations, replace the placeholders with your values:

Create a simple dashboard

Under the dashboards folder, create a file named simple-dashboard.yaml:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: grafanadashboard-sample
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
spec:
  resyncPeriod: 30s
  instanceSelector:
    matchLabels:
      dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
  json: >
    {
    "id": null,
    "title": "Simple Dashboard",
    "tags": [],
    "style": "dark",
    "timezone": "browser",
    "editable": true,
    "hideControls": false,
    "graphTooltip": 1,
    "panels": [],
    "time": {
        "from": "now-6h",
        "to": "now"
    },
    "timepicker": {
        "time_options": [],
        "refresh_intervals": []
    },
    "templating": {
        "list": []
    },
    "annotations": {
        "list": []
    },
    "refresh": "5s",
    "schemaVersion": 17,
    "version": 0,
    "links": []
    }

Create a dashboard from ConfigMap

Under the dashboards folder, create a file named dashboard-from-cm.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: dashboard-definition
  namespace: <GRAFANA_OPERATOR_NAMESPACE>
data:
  json: >
    {
    "id": null,
    "title": "Simple Dashboard from ConfigMap",
    "tags": [],
    "style": "dark",
    "timezone": "browser",
    "editable": true,
    "hideControls": false,
    "graphTooltip": 1,
    "panels": [],
    "time": {
        "from": "now-6h",
        "to": "now"
    },
    "timepicker": {
        "time_options": [],
        "refresh_intervals": []
    },
    "templating": {
        "list": []
    },
    "annotations": {
        "list": []
    },
    "refresh": "5s",
    "schemaVersion": 17,
    "version": 0,
    "links": []
    }
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: grafanadashboard-from-configmap
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
spec:
  instanceSelector:
    matchLabels:
      dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
  configMapRef:
    name: dashboard-definition
    key: json

Create a dashboard from Grafana.com

Under the dashboards folder, create a file named dashboard-from-id.yaml:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: node-exporter-latest
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
spec:
  instanceSelector:
    matchLabels:
      dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
  grafanaCom:
    id: 1860

Configure ArgoCD to sync the Git repository

After you commit all changes to Git, log in to the ArgoCD user interface or use the CLI.

Create an ArgoCD application

Using the UI:

  1. Navigate to New App and complete the form with your Git repository details and the path to your grafana folder
  2. Enable Directory Recurse
  3. Set the sync policy to Automatic

Using the CLI:

Prepare an application manifest named argo-application.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: Grafana
  namespace: '<ARGOCD_NAMESPACE>'
spec:
  destination:
    name: ''
    namespace: ''
    server: 'https://kubernetes.default.svc'
  source:
    path: '<PATH_TO_GRAFANA_FOLDER>'
    repoURL: '<GIT_REPO_URL>'
    targetRevision: HEAD
    directory:
      recurse: true
  sources: []
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 2
    backoff:
      duration: 5s
      maxDuration: 3m0s
      factor: 2

Replace the placeholders with your values:

Create the application in ArgoCD:

kubectl apply -f argo-application.yaml

Verify sync status in ArgoCD

  1. Monitor the newly created ArgoCD application to ensure it successfully syncs your dashboard configuration
  2. Visit the ArgoCD dashboard and check the sync status. If it's successful, your Grafana dashboards should be up to date with the configuration from your Git repository

Update your dashboards

To update an existing dashboard:

  1. Make changes to the dashboard JSON configuration in your Git repository
  2. Commit and push the changes
  3. ArgoCD detects the update and synchronizes the changes to your Custom Resource
  4. Grafana Operator then syncs changes to the Grafana instance

Validate your dashboard updates

Log in to your Grafana dashboard and confirm that the changes are applied. You should see the dashboard updates reflected in the Grafana UI.

Next steps

You've successfully set up a GitOps workflow to manage Grafana dashboards using ArgoCD and the Grafana Operator. Your dashboards are now version-controlled and can be consistently deployed across environments. This approach provides a reliable and auditable way to manage observability dashboards and scale your operations.

To learn more about managing Grafana using Grafana Operator:

Additional considerations

Manage folders, data sources, and dashboards using the Grafana Operator

This guide shows you how to manage data sources, folders, and dashboards using the Grafana Operator. You'll create these resources declaratively using Kubernetes custom resources.

Prerequisites

Before you begin, make sure you have the following:

Set up the Grafana Operator

The Grafana Operator allows you to authenticate with your Grafana instance using the Grafana Custom Resource (CR).

Create the Grafana API Token Secret

Store the Grafana API Token in a secret with the following content in a file named grafana-token.yml:

apiVersion: v1
kind: Secret
metadata:
  name: grafana-cloud-credentials
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
stringData:
  GRAFANA_CLOUD_INSTANCE_TOKEN: '<GRAFANA_API_KEY>'
type: Opaque

Replace the placeholders with your values:

Configure the Grafana Custom Resource

Set up connection to your Grafana Cloud instance. Create a file named grafana-cloud.yml:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: '<GRAFANA_CLOUD_STACK_NAME>'
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
  labels:
    dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
spec:
  external:
    url: https://<GRAFANA_CLOUD_STACK_NAME>.grafana.net/
    apiKey:
      name: grafana-cloud-credentials
      key: GRAFANA_CLOUD_INSTANCE_TOKEN

Replace the placeholders with your values:

Add a data source

{{< admonition type="note" >}}

This example uses the Prometheus data source. Note that the required arguments vary depending on the data source you select.

{{< /admonition >}}

Create a data source configuration

Create and save a new YAML file datasource.yml with your data source's configuration:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: '<DATA_SOURCE_NAME>'
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
spec:
  instanceSelector:
    matchLabels:
      dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
  allowCrossNamespaceImport: true
  datasource:
    access: proxy
    database: prometheus
    jsonData:
      timeInterval: 5s
      tlsSkipVerify: true
    name: '<DATA_SOURCE_NAME>'
    type: prometheus
    url: '<DATA_SOURCE_URL>'

Replace the placeholders with your values:

Add a dashboard to a folder

Use the following YAML definition to create a simple dashboard in the Grafana instance under a custom folder. If the folder defined under the spec.folder field doesn't exist, the operator creates it before placing the dashboard inside the folder.

Prepare the dashboard configuration. In dashboard.yml, define the dashboard and assign it to a folder:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: '<FOLDER_NAME>'
  namespace: '<GRAFANA_OPERATOR_NAMESPACE>'
spec:
  instanceSelector:
    matchLabels:
      dashboards: '<GRAFANA_CLOUD_STACK_NAME>'
  folder: '<FOLDER_NAME>'
  json: >
    {
      "title": "as-code dashboard",
      "uid" : "ascode"
    }

Replace the placeholders with your values:

Apply the Kubernetes manifests

In a terminal, run the following commands from the directory where all of the above Kubernetes YAML definitions are located.

Create Kubernetes Custom resources for all of the configurations:

kubectl apply -f grafana-token.yml grafana-cloud.yml datasource.yml dashboard.yml

Validate your configuration

After you apply the configurations, verify that:

Next steps

You've successfully created a data source, folder, and dashboard using the Grafana Operator. Your Grafana resources are now managed declaratively through Kubernetes custom resources.

To learn more about managing Grafana:

Set up file provisioning

{{< admonition type="note" >}}

On-prem file provisioning is available in Grafana v12 and later for open source and Enterprise editions. It's not available in Grafana Cloud. This new file provisioning feature is only available for dashboards and doesn't replace classic provisioning for the time being.

For classic provisioning using configuration files refer to Provision Grafana.

{{< /admonition >}}

Use local file provisioning to include in your Grafana instance resources that you store in your local file system.

The local path mount is referred to as a repository. File provisioning uses the same Repository resource as Git Sync, with the repository type set to local and a path that points to a directory on the Grafana server's file system.

Using the local path lets you also use it with a tool like fuse, allowing you to mount S3 buckets as local paths. You can also use tools like restic to automatically back up your dashboards to your preferred backup storage solution.

You can set up file provisioning in two ways:

Both approaches require you to first configure the permitted paths in the Grafana configuration file.

{{< admonition type="note" >}}

The UI refers to this feature as file provisioning and to the storage location as local storage. In the Repository resource, the corresponding repository type value is local and the path is set in spec.local.path.

{{< /admonition >}}

New file provisioning capabilities

Local file provisioning using Administration > Provisioning will eventually replace the traditional methods Grafana has used for referencing local file systems for dashboard files.

{{< admonition type="note" >}} For production systems, use the folderFromFilesStructure capability instead of Administration > Provisioning to include dashboards from a local file system in your Grafana instance. Refer to Provision Grafana for more information. {{< /admonition >}}

Before you begin

To set up file provisioning, you need:

Configure permitted paths

The provisioning feature toggle is enabled by default in Grafana Cloud and, starting in Grafana v13, for OSS and Enterprise as well. No manual configuration of the feature toggle is required.

The local setting must be a relative path and its relative path must be configured in the permitted_provisioned_paths configuration option. The configuration option is relative to your working directory, i.e. where you are running Grafana from; this is usually /usr/share/grafana or similar.

Local file paths can point to any directory that is permitted by the configuration. The default paths is devenv/dev-dashboards and conf/provisioning in your grafana installation directory.

The path must behave as a standard file directory on the system of choice. Any subdirectories are automatically included.

The values that you enter for the permitted_provisioning_paths become the base paths for those entered when you enter a local path in the Connect to local storage wizard or in the spec.local.path field of a Repository resource.

  1. Open your Grafana configuration file, either grafana.ini or custom.ini. For file location based on operating system, refer to Configuration file location.

  2. Locate or add a [paths] section. To add more than one location, use the pipe character (|) to separate the paths. The list should not include empty paths or trailing pipes. Add these values:

    [paths]
; This is devenv/dev-dashboards and conf/provisioning by default.
permitted_provisioning_paths = grafana/ | /etc/grafana/provisioning/

  3. Save the changes to the file and start Grafana.

Set up file provisioning using the UI

To use file-based provisioning, you need the file path to the grafana folder where your dashboards are stored in the repository.

To start setting up file-based provisioning:

  1. Log in to your Grafana server with an account that has the Grafana Admin flag set.
  2. Select Administration in the left-side menu and then Provisioning.
  3. Select Configure file provisioning.

Connect to local storage

The local path can point to any directory that is permitted by the configuration. Refer to Configure permitted paths for information.

The starting path is always your working grafana directory. The prefix that must be entered is determined by the locations configured in permitted_provisioning_paths. The default paths are devenv/dev-dashboards and conf/provisioning in your grafana installation directory. The value you enter in the Grafana UI must begin with any of the configured values. For example, conf/provisioning/test is valid, but conf/test is not.

  1. Enter the Local path, for example grafana/. This must begin with any of the configured permitted_provisioned_paths.
  2. Select Choose what to synchronize.

The set up process verifies the path and provides an error message if a problem occurs.

Choose what to synchronize

Synchronization limitations

To have access to full instance sync you must explicitly enable the option.

The following applies:

Set up synchronization

You can sync external resources into a new folder without affecting the rest of your instance.

To set up synchronization:

  1. Select which resources you want to sync.

  2. Enter a Display name for the repository connection. Resources stored in this connection appear under the chosen display name in the Grafana UI.

  3. Click Synchronize to continue.

  4. You can repeat this process for up to 10 connections.

{{< admonition type="note" >}}

Optionally, you can export any unmanaged resources into the provisioned folder. See how in Synchronize with external storage.

{{< /admonition >}}

Synchronize with external storage

In this step you proceed to synchronize the resources selected in the previous step. Optionally, you can check the Migrate existing resources box to migrate your unmanaged dashboards to the provisioned folder.

Select Begin synchronization to start the process. After this one time step, all future updates are automatically saved to the local file path and provisioned back to the instance.

Note that during the initial synchronization, your dashboards will be temporarily unavailable. No data or configurations will be lost. How long the process takes depends upon the number of resources involved.

Choose additional settings

If you wish, you can make any files synchronized as as Read only so no changes can be made to the resources through Grafana. Any resources made outside of Grafana and saved to the local repository will be reflected in Grafana.

Select Finish.

Set up file provisioning as code

Because file provisioning is managed as code using the same Repository Custom Resource Definition (CRD) as Git Sync, you can define it in a YAML file and push it to Grafana with gcx. This approach enables automated, GitOps-style workflows for managing file provisioning instead of using the Grafana UI.

To set up file provisioning as code with gcx, follow these steps:

  1. Create the repository resource.
  2. Push the resource to Grafana.
  3. Manage the repository resource.
  4. Verify your dashboards in Grafana.

Create the repository resource

Create a repository.yaml file defining your file provisioning configuration. Set the repository type to local and use spec.local.path for the directory that holds your dashboards.

apiVersion: provisioning.grafana.app/v0alpha1
kind: Repository
metadata:
  name: '<REPOSITORY_NAME>'
spec:
  title: '<REPOSITORY_TITLE>'
  type: local
  local:
    path: '<LOCAL_PATH>'
  sync:
    enabled: true
    intervalSeconds: 60
    target: folder
  workflows:
    - write

Replace the placeholders with your values:

{{< admonition type="note" >}}

File provisioning supports two sync targets: target: folder (the default) creates a folder named after the repository and places synced resources inside it, while target: folderless places synced resources at the top level without creating a wrapper folder. Refer to Sync targets for details.

{{< /admonition >}}

Configuration parameters

The following configuration parameters are available:

Field Description
metadata.name Unique identifier for this repository resource
spec.title Human-readable name displayed in the Grafana UI
spec.type Repository type. Set to local for file provisioning
spec.local.path Path to the directory that holds your dashboards, within a permitted path
spec.sync.enabled Enable synchronization (true/false)
spec.sync.intervalSeconds Sync interval in seconds
spec.sync.target Where to place synced dashboards (folder or folderless)
spec.workflows Enabled workflows: write (direct changes). Leave empty for a read-only mount

Push the resource to Grafana

Before pushing any resources, configure gcx with your Grafana instance details. Refer to the Grafana CLI documentation for setup instructions.

Push the repository configuration:

gcx resources push --path <DIRECTORY>

The --path parameter has to point to the directory containing your repository.yaml file.

After pushing, Grafana will:

  1. Create the repository resource
  2. Read the dashboards from the configured local path
  3. Begin syncing at the configured interval

Manage the repository resource

Use the following commands to manage file provisioning repositories.

To list all repositories:

gcx resources get repositories

To get details for a specific repository:

gcx resources get repository/<REPOSITORY_NAME>
gcx resources get repository/<REPOSITORY_NAME> -o json
gcx resources get repository/<REPOSITORY_NAME> -o yaml

To update a repository:

gcx resources edit repository/<REPOSITORY_NAME>

To delete a repository:

gcx resources delete repository/<REPOSITORY_NAME>

Verify your dashboards in Grafana

To verify that your dashboards are available at the location that you specified, click Dashboards. The name of the dashboard is listed in the Name column.

Create and manage dashboards using Terraform and GitHub Actions

Learn how to create and manage multiple dashboards represented as JSON source code for Grafana using Terraform and GitHub Actions.

Prerequisites

Before you begin, ensure you have the following:

Add dashboards to a GitHub repository

This guide shows you how to add dashboards for ElasticSearch, InfluxDB, and AWS EC2. You can use different dashboards according to your configured data sources.

  1. In your GitHub repository, create a folder named dashboards in the root directory.

  2. In the dashboards folder create three sub-folders. For this guide, we will create three sub-folders named elasticsearch, influxdb, and aws.

  3. Add dashboard JSON source code to each of the three sub-folders.

Terraform configuration for Grafana provider

This Terraform configuration configures the Grafana provider to provide necessary authentication when creating folders and dashboards in the Grafana instance.

  1. Create a service account and token in the Grafana instance by following these steps:

    1. Create a service account in Grafana
    2. Add a token to a service account

  2. Create a file named main.tf in the Git root directory and add the following code block:

    terraform {
   required_providers {
      grafana = {
         source  = "grafana/grafana"
         version = ">= 2.9.0"
      }
   }
}

provider "grafana" {
   alias = "cloud"

   url   = "<Grafana-instance-url>"
   auth  = "<Grafana-Service-Account-token>"
}

  3. Replace the following field values:

    • <Grafana-instance-url> with the URL of your Grafana instance, for example "https://my-stack.grafana.net/".
    • <Grafana-Service-Account-token> with a Service Account token from the Grafana instance.

Terraform configuration for folders

This Terraform configuration creates three folders named ElasticSearch, InfluxDB and AWS in the Grafana instance using grafana_folder (Resource).

Create a file named folders.tf in the Git root directory and add the following code block:

resource "grafana_folder" "ElasticSearch" {
  provider = grafana.cloud

  title = "ElasticSearch"
}

resource "grafana_folder" "InfluxDB" {
  provider = grafana.cloud

  title = "InfluxDB"
}

resource "grafana_folder" "AWS" {
  provider = grafana.cloud

  title = "AWS"
}

Terraform configuration for dashboards

This Terraform configuration iterates through the JSON files in the three folders (elasticsearch, influxdb and aws) you created in the GitHub repository and adds them to the respective folders in the Grafana instance using a Grafana dashboard resource. Available resources include:

For example, the dashboard represented as JSON source code in the elasticsearch folder in the GitHub repository will be created in the ElasticSearch folder in the Grafana instance.

Create a file named dashboards.tf in the Git root directory and add the following code block:

resource "grafana_dashboard" "elasticsearch" {
  provider = grafana.cloud

  for_each    = fileset("${path.module}/dashboards/elasticsearch", "*.json")
  config_json = file("${path.module}/dashboards/elasticsearch/${each.key}")
  folder      = grafana_folder.ElasticSearch.id
}

resource "grafana_dashboard" "influxdb" {
  provider = grafana.cloud

  for_each    = fileset("${path.module}/dashboards/influxdb", "*.json")
  config_json = file("${path.module}/dashboards/influxdb/${each.key}")
  folder      = grafana_folder.InfluxDB.id
}

resource "grafana_dashboard" "aws" {
  provider = grafana.cloud

  for_each    = fileset("${path.module}/dashboards/aws", "*.json")
  config_json = file("${path.module}/dashboards/aws/${each.key}")
  folder      = grafana_folder.AWS.id
}

GitHub workflow for managing dashboards using Terraform

This GitHub workflow consists of the following steps:

  1. In your GitHub repository, create a folder named .github in the root directory .

  2. In the .github folder create a sub-folder named workflows.

  3. To add the GitHub workflow to your GitHub repository, create a file named terraform.yml in the workflows directory and add the following code block:

    name: Terraform

on:
  push:
    branches:
      - 'main'
  pull_request:

jobs:
  terraform:
    runs-on: ubuntu-latest

    steps:
      # Checkout the repository to the GitHub Actions runner
      - name: Checkout
        uses: actions/checkout@v3

      # Install the latest version of Terraform CLI
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v1

      # Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
      - name: Terraform Init
        run: terraform init

      # Checks that all Terraform configuration files adhere to a canonical format
      - name: Terraform Format
        run: terraform fmt -check

      # Previews the changes that Terraform will make
      - name: Plan Terraform
        id: plan
        continue-on-error: true
        run: terraform plan -input=false -no-color

      # Post the preview (terraform plan) from the previous step as a GitHub pull request comment
      - name: Post Plan to GitHub PR
        if: github.ref != 'refs/heads/main'
        uses: mshick/add-pr-comment@v1
        with:
          repo-token: ${{ secrets.GITHUB_TOKEN }}
          repo-token-user-login: 'github-actions[bot]'
          message: |
            Applying:

            ```
            ${{ steps.plan.outputs.stdout }}
            ```

      # Applies the terraform configuration files when the branch is `main`
      - name: Apply Terraform
        if: github.ref == 'refs/heads/main'
        id: apply
        continue-on-error: true
        run: |
          terraform apply -auto-approve

  4. Commit the changes made to the terraform.yml in the previous step to the main branch in your GitHub repository. Once the changes are committed, The GitHub workflow you created should start to run automatically as the workflow we defined in the previous step runs when a pull request is created or when changes are committed to main branch.

Managing the Terraform state

If you are not using a Terraform backend to store the .tfstate file, add the following code block to the end of the GitHub workflow file to make sure the Terraform state file is stored in Git.

- name: commit the terraform state
  if: github.ref == 'refs/heads/main'
  uses: stefanzweifel/git-auto-commit-action@v4
  with:
    commit_message: Updating Terraform state
    file_pattern: terraform.tfstate

When you run terraform apply, Terraform automatically manages and updates the terraform.tfstate file to store state about your infrastructure and configuration. This step uses the stefanzweifel/git-auto-commit-action@v4 action to auto-commit the terraform.tfstate file for changes made by running the terraform apply step.

{{< admonition type="note" >}} The Terraform state file (terraform.tfstate) shouldn't be stored in Git to avoid leakage of sensitive data. Instead, store Terraform state files using a remote backend like AWS S3 with proper RBAC. For more information, refer to Terraform state. {{< /admonition >}}

Validation

Once the GitHub workflow run is successful, you should be able to verify the following:

Summary

In this guide, you created a GitHub workflow using Terraform to manage dashboard source code. Using this workflow, the dashboards in the Grafana instance are synchronized with the JSON source code files for dashboards in GitHub.

To learn more about managing Grafana Cloud using Terraform, refer to the Grafana provider documentation.

Manage saved queries using Terraform

This guide shows you how to manage saved queries, also known as the query library, using the Grafana Terraform provider. Managing saved queries as code lets you version-control your query library and keep it consistent across Grafana instances.

{{< admonition type="note" >}} Saved queries are only available on Grafana Enterprise and Grafana Cloud. {{< /admonition >}}

Before you begin

Before you begin, ensure you have the following:

Configure the Grafana provider

Create a file named main.tf and add the following to set up the Grafana provider with the authentication required to manage saved queries:

terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 4.44.0"
    }
  }
}

provider "grafana" {
  url  = "<Grafana-URL>"
  auth = "<Service-account-token>"
}

Replace the following field values:

Create a saved query resource

Create a file named saved-queries.tf and add the following:

resource "grafana_apps_queries_query_v1" "example" {
  metadata {
    uid = "example-saved-query"
  }

  spec {
    title       = "Requests per second"
    description = "Prometheus rate of HTTP requests"
    is_visible  = true
    tags        = ["http", "prometheus"]

    targets {
      properties_json = jsonencode({
        refId = "A"
        expr  = "rate(http_requests_total[$__rate_interval])"
        datasource = {
          type = "prometheus"
          uid  = "my-prometheus-uid"
        }
      })
    }
  }
}

Because the shape of a data source query depends on the data source, the query stored in each target (properties_json), a target's variable replacements (variables_json), and a variable's value list (value_list_definition_json) are passed as raw JSON strings. Use jsonencode() to construct them.

The spec block supports the following fields:

Field Description
title The display name of the saved query. Required.
description A longer description of the saved query.
is_visible Whether the saved query is visible in the query library.
is_locked Whether the saved query is locked and can't be edited in the UI. This is for UI display only, not for security.
tags The tags used to filter the saved query.
targets The query targets that make up the saved query. At least one target is required.
vars The template variables that can be interpolated into the query targets.

For the complete schema, refer to the grafana_apps_queries_query_v1 resource documentation.

Import an existing saved query

To bring an existing saved query under Terraform management, import it using its UID:

terraform import grafana_apps_queries_query_v1.example example-saved-query

Summary

In this guide, you learned how to create and import saved queries using Terraform.

To learn more about saved queries, refer to Saved queries.

Manage Cloud Provider Observability in Grafana Cloud using Terraform

Manage Cloud Provider Observability, including Amazon CloudWatch and Microsoft Azure resources, in Grafana Cloud using Terraform. For more information on Cloud Provider Observability, refer to the Cloud Provider Observability documentation.

Before you begin

Before you begin, ensure you have the following:

{{< admonition type="note" >}} Save all of the following Terraform configuration files in the same directory. {{< /admonition >}}

Configure authentication for the Grafana Provider

The Grafana Provider is a logical abstraction of an upstream API that you can use to interact with Grafana Cloud resources. You must configure it with the following information:

To configure authentication for the Grafana Provider:

  1. Create a Grafana Cloud access policy and token.

    • To create an access policy for your organization, refer to the Create an access policy for a stack steps and use the following scopes listed for the supported Amazon CloudWatch or Microsoft Azure resources:
      • Amazon CloudWatch
        • Metrics scrape or resource metadata scrape
          • integration-management:read
          • integration-management:write
          • stacks:read
        • Metric streams
          • metrics:write
        • ALB access logs, logs with Lambda, or logs with Amazon Data Firehose
          • logs:write
    • Microsoft Azure
      • Serverless metrics
        • integration-management:read
        • integration-management:write
        • stacks:read
      • Logs with Azure functions
        • logs:write

  2. Obtain the regional Cloud Provider API endpoint.

    • To obtain the regional Cloud Provider API endpoint, use your access policy token and the following command to return a list of all of the Grafana stacks you own, along with their respective Cloud Provider API hostnames:
    curl -sH "Authorization: Bearer @@@GRAFANA_CLOUD_ACCESS_POLICY_TOKEN@@@" "https://grafana.com/api/instances" | \
jq '[.items[]|{stackName: .slug, clusterName:.clusterSlug, cloudProviderAPIURL: "https://cloud-provider-api-\(.clusterSlug).grafana.net"}]'

    {{< admonition type="note" >}} If this endpoint does not work, your Grafana Cloud Provider API endpoint may be using the new URL format. For more information on Grafana Cloud URLs, refer to Determine Grafana Cloud URLs based on region. {{< /admonition >}}

  3. Create a file named cloud-provider.tf and add the following code block:

    terraform {
    required_providers {
    grafana = {
        source  = "grafana/grafana"
    }
  }
}

provider "grafana" {
    cloud_provider_url      = "<@@@CLOUD_PROVIDER_URL@@@"
    cloud_access_policy_token     = "@@@CLOUD_ACCESS_POLICY_TOKEN@@@>"
}

  4. Create a variables.tf file and paste the CLOUD_ACCESS_POLICY_TOKEN and CLOUD_PROVIDER_URL variables with your values.

  5. Run the following Terraform command:

    terraform apply -var-file="variables.tf"

Configure your resources

To find instructions for configuring specific Amazon CloudWatch and Microsoft Azure resources in Cloud Provider Observability using Terraform, refer to the following documents:

Grafana cloud provider resources

You can define the following Cloud Provider Observability resources and data sources using Terraform:

Resource name Description
grafana_cloud_provider_aws_account Represents an AWS IAM role that authorizes Grafana Cloud to pull Amazon CloudWatch metrics for a set of regions. Usually, there's one of these resources per configured AWS account. For a full reference of this resource, refer to the Terraform Grafana Provider reference documentation.
grafana_cloud_provider_aws_cloudwatch_scrape_job Represents a Grafana AWS scrape job. This configures Grafana to fetch a list of metrics and statistics for one or many AWS services, and for a given grafana_cloud_provider_aws_account. For a full reference of this resource, refer to the Terraform Grafana Provider reference documentation.
grafana_cloud_provider_aws_resource_metadata_scrape_job Represents a Grafana AWS Resource Metadata scrape job. This resource configures Grafana to fetch resource metadata for one or multiple AWS services, for a given grafana_cloud_provider_aws_account. For a full reference of this resource, refer to the Terraform Grafana Provider reference documentation.
grafana_cloud_provider_azure_credential A resource representing an Azure Service Principal credential used by Grafana Cloud to pull Azure Monitor metrics from one or more subscriptions. For a full reference of this resource, refer to the Terraform Grafana Provider resource documentation.

Create and manage a Grafana Cloud stack using Terraform

Learn how to add a data source, a dashboard, and a folder to a Grafana Cloud stack using Terraform.

Prerequisites

Before you begin, ensure you have the following:

{{< admonition type="note" >}} All of the following Terraform configuration files should be saved in the same directory. {{< /admonition >}}

Create a Cloud stack

  1. Create a Terraform configuration file.

    This Terraform configuration will create a Grafana Cloud stack and a second token needed for your Grafana instance.

    Create a file named cloud-stack.tf and add the following:

    terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 2.9.0"
    }
  }
}
// Step 1: Create a stack
provider "grafana" {
  alias = "cloud"
  cloud_access_policy_token = "<cloud-access-token>"
}


resource "grafana_cloud_stack" "my_stack" {
  provider = grafana.cloud

  name               = "<stack-name>"
  slug               = "<stack-name>"
  region_slug        = "<region>" # Example "us","eu" etc
  delete_protection  = true
}

// Step 2: Create a service account and key for the stack
resource "grafana_cloud_stack_service_account" "cloud_sa" {
  provider   = grafana.cloud
  stack_slug = grafana_cloud_stack.my_stack.slug

  name        = "<service-account-name>"
  role        = "Admin"
  is_disabled = false
}

resource "grafana_cloud_stack_service_account_token" "cloud_sa" {
  provider   = grafana.cloud
  stack_slug = grafana_cloud_stack.my_stack.slug

  name               = "terraform serviceaccount key"
  service_account_id = grafana_cloud_stack_service_account.cloud_sa.id
}

// Step 3: Create resources within the stack
provider "grafana" {
  alias = "my_stack"

  url  = grafana_cloud_stack.my_stack.url
  auth = grafana_cloud_stack_service_account_token.cloud_sa.key
}
resource "grafana_folder" "my_folder" {
  provider = grafana.my_stack

  title = "Test Folder"
}

  2. Replace the following field values:

    • <cloud-access-token> with your Grafana Cloud Access Policy Token. To create a new one, refer to Grafana Cloud Access Policies. Add all stacks to the realms list. The scopes needed for the example are:
      • dashboards:read
      • orgs:read
      • stack-dashboards:read
      • stacks:read
      • dashboards:write
      • orgs:write
      • stack-dashboards:write
      • stacks:write
      • stack-service-accounts:write
      • dashboards:delete
      • stack-dashboards:delete
      • stacks:delete
    • <stack-name> with the name of your stack.
    • <region> with the region in which you want to create the stack. For example us, eu.
    • <service-account-name> with a name for the serviceaccount that will be created to use for operations within the stack/instance.

The first provider block, grafana.cloud, uses the Cloud Access Policy Token from the Cloud Portal and is referenced as a parameter when creating the Cloud stack and the token in the Grafana instance to provide the necessary authentication.

The second provider block, grafana.my_stack, is referenced as a parameter when creating resources inside the Grafana instance.

Add a data source

This guide uses the InfluxDB data source. The required arguments for grafana_data_source (Resource) vary depending on the type of data source you select.

  1. Create a file named datasource.tf and add the following:

    resource "grafana_data_source" "<data-source-name>" {
  provider = grafana.my_stack

  type          = "influxdb"
  name          = "<data-source-name>"
  url           = "<data-source-url>"
  username      = "<username>"
  password      = "<password>"
  database_name = "<db-name>"
}

  2. Replace the following field values:

    • <data-source-name> with the name of the data source to be added in Grafana.
    • <data-source-url> with URL of your data source.
    • <username> with the username for authenticating with your data source.
    • <password> with password for authenticating with your data source.
    • <db-name> with name of your database.

Add a folder

This Terraform configuration creates a folder in your Grafana instance using grafana_folder (Resource).

  1. Create a file named folder.tf and add the following:

    resource "grafana_folder" "<folder-name>" {
  provider = grafana.my_stack

  title = "<folder-name>"
}

  2. Replace the following field value:

    • <folder-name> with a name for the folder.

Add a dashboard to the folder

This Terraform configuration creates a dashboard inside the folder created above in your Grafana instance using grafana_dashboard (Resource).

  1. Create a file named dashboard.tf and add the following:

    # Using a JSON file
resource "grafana_dashboard" "dashboard" {
  provider = grafana.my_stack

  config_json = file("<file-name>.json")
  folder = grafana_folder.<folder-name>.id
}

  2. Replace the following field value:

    • <file-name> with the name of the JSON file that has the source code for the dashboard.

    The dashboard is represented by its JSON source code and referenced in the config_json parameter.

Apply the Terraform configuration

In a terminal, run the following commands from the directory where all of the configuration files are located.

  1. Initialize a working directory containing Terraform configuration files.

    terraform init

  2. Preview the changes that Terraform will make.

    terraform plan

  3. Apply the configuration files.

    terraform apply

Validation

Once you apply the changes in the Terraform configurations, you should be able to verify the following:

Summary

In this guide, you created a Grafana Cloud stack along with a data source, folder, and dashboard imported from a JSON file using Terraform.

To learn more about managing Grafana Cloud using Terraform, refer to the Grafana provider documentation.

Manage Fleet Management in Grafana Cloud using Terraform

Learn how to create Grafana Fleet Management collectors and pipelines in Grafana Cloud using Terraform. This guide shows you how to create an access policy and a token for Fleet Management, preregister a collector with remote attributes, and create a configuration pipeline.

Fleet Management supports Grafana Alloy and the OpenTelemetry Collector. The two collector types connect to Fleet Management differently and use different configuration syntax, so this guide shows both paths side by side. Follow the Alloy steps, the OpenTelemetry Collector steps, or both if you're managing a mixed fleet.

Before you begin

Before you begin, ensure you have the following:

{{< admonition type="note" >}} All of the following Terraform configuration files should be saved in the same directory. {{< /admonition >}}

Configure a provider for Grafana Cloud

This Terraform configuration configures the Grafana provider to provide necessary authentication when interacting with the Cloud API. The grafana_cloud_stack (Data Source) is used to retrieve the user ID and URL details of your instance.

  1. Create a Grafana Cloud access policy and token. To create a new one, refer to Grafana Cloud Access Policies. Add your stack to the realms list. The scopes needed for the examples in this guide are:

    • accesspolicies:read
    • accesspolicies:write
    • accesspolicies:delete
    • stacks:read

  2. Create a file named cloud-provider.tf and add the following code block:

    terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 3.19.0"
    }
  }
}

provider "grafana" {
  alias = "cloud"

  cloud_access_policy_token = "<CLOUD_ACCESS_POLICY_TOKEN>"
}

data "grafana_cloud_stack" "stack" {
  provider = grafana.cloud

  slug = "<STACK_SLUG>"
}

  3. Replace the following field values:

    • <CLOUD_ACCESS_POLICY_TOKEN> with the access policy token you created in the first step
    • <STACK_SLUG> with your stack slug, which is the subdomain where your Grafana Cloud instance is available: https://<STACK_SLUG>.grafana.net

Create an access policy and token for Fleet Management

This Terraform configuration creates the following:

  1. Create a file named fm-access-policy.tf and add the following code block:

    resource "grafana_cloud_access_policy" "fm_policy" {
  provider = grafana.cloud

  name   = "fleet-management-policy"
  region = data.grafana_cloud_stack.stack.region_slug

  scopes = [
    "fleet-management:read",
    "fleet-management:write"
  ]

  realm {
    type       = "stack"
    identifier = data.grafana_cloud_stack.stack.id
  }
}

resource "grafana_cloud_access_policy_token" "fm_token" {
  provider = grafana.cloud

  name             = "fleet-management-token"
  region           = grafana_cloud_access_policy.fm_policy.region
  access_policy_id = grafana_cloud_access_policy.fm_policy.policy_id
}

Configure a provider for Fleet Management

This Terraform configuration configures the Grafana provider to provide necessary authentication when interacting with the Fleet Management API.

  1. Create a file named fm-provider.tf and add the following code block:

    locals {
  fm_id    = data.grafana_cloud_stack.stack.fleet_management_user_id
  fm_token = grafana_cloud_access_policy_token.fm_token.token
  fm_url   = data.grafana_cloud_stack.stack.fleet_management_url
}

provider "grafana" {
  alias = "fm"

  fleet_management_auth = "${local.fm_id}:${local.fm_token}"
  fleet_management_url  = local.fm_url
}

Create an access policy and token for your collector

Alloy and the OpenTelemetry Collector need properly scoped access to read remote configuration from Fleet Management and write telemetry to Grafana Cloud.

The Terraform provider creates the alloy_token for you when you run terraform apply later in this guide.

OpenTelemetry Collectors require an access token before you can successfully apply the Terraform configurations. You create the otel_token manually in this step, but you won't need it until later in the guide.

Alloy

This Terraform configuration creates the following:

  1. Create a file named alloy-access-policy.tf and add the following code block:

    resource "grafana_cloud_access_policy" "alloy_policy" {
  provider = grafana.cloud

  name   = "alloy-policy"
  region = data.grafana_cloud_stack.stack.region_slug

  scopes = [
    "set:alloy-data-write"
  ]

  realm {
    type       = "stack"
    identifier = data.grafana_cloud_stack.stack.id
  }
}

resource "grafana_cloud_access_policy_token" "alloy_token" {
  provider = grafana.cloud

  name             = "alloy-token"
  region           = grafana_cloud_access_policy.alloy_policy.region
  access_policy_id = grafana_cloud_access_policy.alloy_policy.policy_id
}

output "alloy_token" {
  value     = grafana_cloud_access_policy_token.alloy_token.token
  sensitive = true
}

OpenTelemetry Collector

This Terraform configuration creates the following:

  1. Create a file named otel-access-policy.tf and add the following code block:

    resource "grafana_cloud_access_policy" "otel_policy" {
  provider = grafana.cloud

  name   = "otel-policy"
  region = data.grafana_cloud_stack.stack.region_slug

  scopes = [
    "set:otel-data-write"
  ]

  realm {
    type       = "stack"
    identifier = data.grafana_cloud_stack.stack.id
  }
}

resource "grafana_cloud_access_policy_token" "otel_token" {
  provider = grafana.cloud

  name             = "otel-token"
  region           = grafana_cloud_access_policy.otel_policy.region
  access_policy_id = grafana_cloud_access_policy.otel_policy.policy_id
}

Create a configuration file for your collector

In this step, you create a local configuration file for Alloy or the OpAMP Supervisor.

Alloy

This Terraform configuration creates an Alloy configuration file with the remotecfg block for Fleet Management, using local_file (Resource).

  1. Create a file named config.alloy.tftpl and add the following content:

    remotecfg {
    id             = "${collector_id}"
    url            = "${fm_url}"
    poll_frequency = "60s"

    basic_auth {
        username = "${fm_id}"
        password = sys.env("GCLOUD_RW_API_KEY")
    }
}

  2. Create a file named alloy-config.tf and add the following code block:

    resource "local_file" "alloy_config" {
  filename = "<ALLOY_CONFIG_PATH>"
  content = templatefile(
    "config.alloy.tftpl",
    {
      collector_id = "prod_alloy_collector",
      fm_id        = local.fm_id,
      fm_url       = local.fm_url,
    },
  )
  directory_permission = "0644"
  file_permission      = "0644"
}

  3. Replace the following field values:

    • <ALLOY_CONFIG_PATH> with the path the Alloy configuration file should be written to, for example config.alloy

OpenTelemetry Collector

This Terraform configuration creates a Supervisor configuration file that connects the OpenTelemetry Collector to Fleet Management over OpAMP, using local_file (Resource).

  1. Create a file named supervisor.yaml.tftpl and add the following content:

    server:
  endpoint: '${fm_url}/v1/opamp'
  headers:
    Authorization: 'Basic ${fm_auth_base64}'

capabilities:
  reports_effective_config: true
  accepts_remote_config: true

agent:
  executable: '<OTELCOL_EXECUTABLE_PATH>'
  instance_id: '${collector_id}'

storage:
  directory: '<STORAGE_DIRECTORY>'

  2. Create a file named otel-supervisor-config.tf and add the following code block:

    resource "local_file" "otel_supervisor_config" {
  filename = "${path.module}/supervisor.yaml"
  content = templatefile(
    "supervisor.yaml.tftpl",
    {
      collector_id   = "<COLLECTOR_ID>"
      fm_auth_base64 = base64encode("${local.fm_id}:${grafana_cloud_access_policy_token.otel_token.token}"),
      fm_url         = local.fm_url,
    },
  )
  directory_permission = "0644"
  file_permission      = "0644"
}

  3. Replace the following field values:

    • <OTELCOL_EXECUTABLE_PATH> with the path to your OpenTelemetry Collector executable
    • <STORAGE_DIRECTORY> with the path to a writable directory to use for persistent data storage
    • <COLLECTOR_ID> with the universally unique identifier (UUID) of the Collector.

Create a Fleet Management collector

This step registers a collector with a remote attribute, using grafana_fleet_management_collector (Resource). The collector_type argument tells Fleet Management which configuration syntax and validation to expect for the collector: ALLOY (the default) or OTEL.

The following configurations only preregister the collector. You must complete the Run the collector step for the collector to connect to Fleet Management and be assigned remote attributes.

Alloy

  1. Create a file named fm-collector-alloy.tf and add the following code block:

    resource "grafana_fleet_management_collector" "fm_collector_alloy" {
  provider = grafana.fm

  id = "prod_alloy_collector"
  collector_type = "ALLOY"
  remote_attributes = {
    "env" = "PROD"
  }
  enabled = true
}

OpenTelemetry Collector

  1. Create a file named fm-collector-otel.tf and add the following code block:

    resource "grafana_fleet_management_collector" "fm_collector_otel" {
  provider = grafana.fm

  id = "<COLLECTOR_ID>"
  collector_type = "OTEL"
  remote_attributes = {
    "env" = "PROD"
  }
  enabled = true
}

  2. Replace <COLLECTOR_ID> with the same unique identifier used in the Supervisor configuration.

Create a Fleet Management pipeline

The following Terraform configurations create pipelines with matchers for the collectors declared in the previous step, using grafana_fleet_management_pipeline (Resource). The config_type argument tells Fleet Management how to parse and validate the contents argument: ALLOY (the default) expects Alloy configuration syntax, and OTEL expects OpenTelemetry Collector YAML.

Fleet Management delivers a pipeline only to collectors of the matching collector_type, so an ALLOY pipeline and an OTEL pipeline can safely share the same matchers.

Alloy profiling pipeline

This pipeline profiles Alloy itself and writes the profiles to Grafana Cloud Profiles.

  1. Create a file named profiling.alloy.tftpl and add the following content:

    // This pipeline scrapes pprof Go profiles from Alloy and sends them to Pyroscope.
//
// It requires the following environment variables to be set where Alloy is running:
//   Required:
//     * GCLOUD_RW_API_KEY: The Grafana Cloud API key with write access to Pyroscope.
//   Optional:
//     * ALLOY_ADDRESS: The address Alloy listens on. Defaults to 127.0.0.1:12345.
pyroscope.scrape "alloy" {
    targets = [
        {
            "__address__" = coalesce(
                sys.env("ALLOY_ADDRESS"),
                "127.0.0.1:12345",
            ),
            "service_name" = "alloy",
        },
    ]
    forward_to = [pyroscope.write.grafana_cloud.receiver]

    profiling_config {
        profile.process_cpu {
            enabled = true
        }

        profile.memory {
            enabled = true
        }

        profile.mutex {
            enabled = true
        }

        profile.block {
            enabled = true
        }

        profile.goroutine {
            enabled = true
        }
    }
}

pyroscope.write "grafana_cloud" {
    endpoint {
        url = "${profiles_url}"

        basic_auth {
            username = "${profiles_id}"
            password = sys.env("GCLOUD_RW_API_KEY")
        }
    }
}

  2. Create a file named fm-pipeline-alloy-profiles.tf and add the following code block:

    locals {
  profiles_id  = data.grafana_cloud_stack.stack.profiles_user_id
  profiles_url = data.grafana_cloud_stack.stack.profiles_url
}

resource "grafana_fleet_management_pipeline" "profiles" {
  provider = grafana.fm

  name        = "alloy_profiling"
  config_type = "ALLOY"
  contents = templatefile(
    "profiling.alloy.tftpl",
    {
      profiles_id  = local.profiles_id,
      profiles_url = local.profiles_url,
    },
  )
  matchers = [
    "env=\"PROD\""
  ]
  enabled = true
}

OpenTelemetry Collector host metrics pipeline

This pipeline collects host metrics with the hostmetrics receiver and writes them to Grafana Cloud over OTLP.

  1. Create a file named hostmetrics.otel.yaml.tftpl and add the following content:

    # This pipeline collects host metrics and sends them to Grafana Cloud over OTLP.
receivers:
  hostmetrics:
    scrapers:
      load:
      memory:

processors:
  resourcedetection:
    detectors: ['env', 'system']
    override: false
  transform/add_host_metric_attributes:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["host.name"], resource.attributes["host.name"])
          - set(attributes["os.type"], resource.attributes["os.type"])

exporters:
  otlp_http/grafana_cloud:
    endpoint: '<OTLP_ENDPOINT>'
    headers:
      Authorization: 'Basic $${env:GCLOUD_BASIC_AUTH_BASE64}'
    sending_queue:
      enabled: true
      batch: {}

service:
  pipelines:
    metrics/hostmetrics:
      receivers: [hostmetrics]
      processors: [resourcedetection, transform/add_host_metric_attributes]
      exporters: [otlp_http/grafana_cloud]

  2. Replace <OTLP_ENDPOINT> with the OTLP endpoint for your stack, which you can find in your Grafana Cloud portal.

  3. Create a file named fm-pipeline-otel-hostmetrics.tf and add the following code block:

    resource "grafana_fleet_management_pipeline" "host_metrics" {
  provider = grafana.fm

  name        = "otel_host_metrics"
  config_type = "OTEL"
  contents = templatefile("hostmetrics.otel.yaml.tftpl", {})
  matchers = [
    "env=\"PROD\""
  ]
  enabled = true
}

Apply the Terraform configuration

In a terminal, run the following commands from the directory where all of the configuration files are located.

  1. Initialize a working directory containing Terraform configuration files:

    terraform init

  2. Preview the Terraform changes:

    terraform plan

  3. Apply the configuration:

    terraform apply

Run the collector

The final step is to start your collector. Make sure to set an environment variable containing the access token. The pipelines used in this guide won't work without it.

Alloy

  1. Run the following command to view the Alloy token:

    terraform output -raw alloy_token

  2. Set the environment variable GCLOUD_RW_API_KEY to the value from the first step.

  3. Run Alloy. To learn how to start or restart Alloy, refer to Run Grafana Alloy.

OpenTelemetry Collector

  1. Run the following command to export the Base64-encoded credentials the Collector uses to send telemetry to Grafana Cloud. Replace <INSTANCE_ID> with your actual instance ID, which you can find on the API tab in the Fleet Management application.

    export GCLOUD_BASIC_AUTH_BASE64=$(printf '%s' "<INSTANCE_ID>:$(terraform output -raw otel_token)" | base64 | tr -d '\n')

  2. Start the Supervisor:

    ./opampsupervisor --config=supervisor.yaml

Validation

After you apply the changes in the Terraform configurations and run your collector, you should be able to verify the following:

Summary

In this guide, you created an access policy and a token for Fleet Management, registered an Alloy instance or OpenTelemetry Collector with remote attributes, and created a configuration pipeline based on collector type, all using Terraform.

To learn more about managing Grafana Cloud using Terraform, refer to the Grafana provider documentation.

Manage Frontend Observability in Grafana Cloud with Terraform

Learn how to use Terraform to manage Grafana Frontend Observability resources, such as your applications. This guide shows you how to create an access policy and a token for Frontend Observability so that you can connect to the Frontend Observability API.

Before you begin

Before you begin, ensure you have the following:

{{< admonition type="note" >}} All of the following Terraform configuration files should be saved in the same directory. {{< /admonition >}}

Configure a provider for Grafana Cloud

This Terraform configuration configures the Grafana provider to provide necessary authentication when interacting with the Cloud API. The grafana_cloud_stack (Data Source) is used to retrieve the details of your instance.

  1. Create a Grafana Cloud access policy and token. To create a new one, refer to Grafana Cloud Access Policies. Add your stack to the realms list. The scopes needed for the examples in this guide are:

    • accesspolicies:read
    • accesspolicies:write
    • accesspolicies:delete
    • dashboards:read
    • dashboards:write
    • dashboards:delete
    • orgs:read
    • orgs:write
    • stacks:read
    • stacks:write
    • stacks:delete
    • stack-dashboards:read
    • stack-dashboards:write
    • stack-dashboards:delete
    • stack-service-accounts:write

  2. Create a file named cloud-provider.tf and add the following code block:

    terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
    }
  }
}

provider "grafana" {
  alias = "cloud"

  cloud_access_policy_token = "<CLOUD_ACCESS_POLICY_TOKEN>"
}

data "grafana_cloud_stack" "stack" {
  provider = grafana.cloud

  slug = "<STACK_SLUG>"
}

  3. Replace the following field values:

    • <CLOUD_ACCESS_POLICY_TOKEN> with the access policy token you created in the first step
    • <STACK_SLUG> with your stack slug, which is the subdomain where your Grafana Cloud instance is available: https://<STACK_SLUG>.grafana.net

Create an access policy and token for Frontend Observability

You must create a Terraform configuration with the following:

Configure the provider to use the Frontend Observability API

After you have created the token, you can configure the provider as follows:

provider "grafana" {
  frontend_o11y_api_access_token = "<access token from previous step>"
}

Summary

In this guide, you created an access policy and a token for Frontend Observability using Terraform.

To learn more about managing Grafana Cloud using Terraform, refer to the Grafana provider documentation.

Manage Grafana IRM in Grafana Cloud using Terraform

Learn how to use Terraform to manage Grafana IRM resources. This guide shows you how to connect an integration, configure escalation policies, and add on-call schedules using Terraform.

To illustrate the use of IRM across multiple teams, this guide features examples with two teams: Devs and SREs. Additionally, it includes the necessary steps to configure Slack for IRM.

{{< admonition type="note" >}} Grafana IRM supports Terraform-based configuration for a limited set of resources, primarily those related to OnCall functionality. These resources use the grafana_oncall_ naming convention in Terraform. Additional IRM components are not yet configurable via Terraform. {{< /admonition >}}

Before you begin

Before you begin, ensure you have the following:

{{< admonition type="note" >}} All of the following Terraform configuration files should be saved in the same directory. {{< /admonition >}}

Connect Slack to Grafana IRM

Before including Slack settings in your Terraform setup, you must first configure the Slack integration with Grafana IRM.

To connect your Slack workspace to Grafana IRM, refer to the Slack integration for Grafana IRM documentation.

Configure the Grafana provider

This Terraform configuration sets up the Grafana provider to provide necessary authentication when managing resources for Grafana IRM.

You can reuse a similar setup to the one described in Creating and managing a Grafana Cloud stack using Terraform to set up a service account and a token.

  1. Create a Service account and token in Grafana. To create a new one, refer to Service account tokens.

  2. Create a file named main.tf and add the following:

    terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 3.15.3"
    }
  }
}

provider "grafana" {
  alias = "oncall"

  url  = "<Stack-URL>"
  auth = "<Service-account-token>"
  oncall_url = "<OnCall-URL>"
}

  3. Replace the following field values:

    • <Stack-URL> with the URL of your Grafana stack
    • <Service-account-token> with the service account token that you created
    • <OnCall-URL> with the API URL found on the Admin & API tab of the IRM Settings page

{{< admonition type="note" >}} If the service account has the right permissions, this provider setup also allows you to manage other Grafana resources. {{< /admonition >}}

Authentication via OnCall API tokens (deprecated)

OnCall API tokens are being deprecated. While existing tokens will continue to work, we recommend using Grafana Cloud service account tokens for all new API authentication.

{{< collapse title="Authentication via OnCall API tokens" >}} To use an existing OnCall API token:

  1. Log into your Grafana Cloud instance

  2. Select Alerts & IRM > IRM

  3. Click Settings, and then select Admin & API

  4. Locate the Grafana IRM API section

  5. View, copy or revoke existing OnCall API tokens

  6. Create a file named main.tf and add the following:

    terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 2.9.0"
    }
  }
}

provider "grafana" {
  alias = "oncall"

  oncall_access_token = "<OnCall-API-Token>"
  oncall_url = "<OnCall-URL>"
}

  7. Replace the following field values:

    • <OnCall-API-Token> with your existing OnCall API Token
    • <OnCall-URL> with the API URL found on the Admin & API tab of the IRM Settings page {{< /collapse >}}

Add on-call schedules

This Terraform configuration sets up two on-call schedules, SREs and Devs, using the grafana_oncall_schedule resource to define the schedules within Grafana IRM. Additionally, this configuration includes Slack channels to receive notifications for the on-call schedules of each team.

To learn more about managing on-call schedules, refer to the On-call schedules documentation.

  1. Create two new calendars in your calendar service, one for Devs and one for SREs

  2. Locate and save the secret iCal URLs. For example, in a Google calendar, these URLs can be found in Settings > Settings for my calendars > Integrate calendar

  3. Create a file named schedule.tf and add the following:

    # Name of the Slack channel to notify about on-call schedules for Devs
data "grafana_oncall_slack_channel" "Devs" {
  provider = grafana.oncall

  name = "<Devs-channel-name>"
}

# Name of the Slack channel to notify about on-call schedules for SREs
data "grafana_oncall_slack_channel" "SREs" {
  provider = grafana.oncall

  name = "<SREs-channel-name>"
}

resource "grafana_oncall_schedule" "schedule_Devs" {
  provider = grafana.oncall

  name             = "Devs"
  type             = "ical"
  ical_url_primary = "<secret-iCal-URL-for-devs-calendar>"
  slack {
    channel_id = data.grafana_oncall_slack_channel.Devs.slack_id
  }
}

resource "grafana_oncall_schedule" "schedule_SREs" {
  provider = grafana.oncall

  name             = "SREs"
  type             = "ical"
  ical_url_primary = "<secret-iCal-URL-for-SREs-calendar>"
  slack {
    channel_id = data.grafana_oncall_slack_channel.SREs.slack_id
  }
}

  4. Replace the following field values:

    • <Devs-channel-name> with name of the Slack channel to notify about on-call schedules for Devs
    • <SREs-channel-name> with name of the Slack channel to notify about on-call schedules for SREs
    • <secret-iCal-URL-for-devs-calendar> with the secret iCal URL created in the first step for Devs Calendar
    • <secret-iCal-URL-for-SREs-calendar> with the secret iCal URL created in the first step for SREs Calendar

Add escalation chains

This Terraform configuration creates two escalation chains named SREs and Devs in Grafana IRM using the grafana_oncall_escalation_chain (Resource). The configuration also adds the following three steps to each escalation chain using the grafana_oncall_escalation (Resource):

  1. Create a file named escalation-devs.tf and add the following:

    resource "grafana_oncall_escalation_chain" "Devs" {
  provider = grafana.oncall

  name = "Devs"
}

// Notify users from on-call schedule
resource "grafana_oncall_escalation" "notify_schedule_step_Devs" {
  provider = grafana.oncall

  escalation_chain_id          = grafana_oncall_escalation_chain.Devs.id
  type                         = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.schedule_Devs.id
  position                     = 0
}

// Wait step for 5 Minutes
resource "grafana_oncall_escalation" "wait_step_Devs" {
  provider = grafana.oncall

  escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
  type                = "wait"
  duration            = 300
  position            = 1
}

// Notify default Slack channel step
resource "grafana_oncall_escalation" "notify_step_Devs" {
  provider = grafana.oncall

  escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
  type                = "notify_whole_channel"
  important           = true
  position            = 2
}

  2. Create a file named escalation-sre.tf and add the following:

    resource "grafana_oncall_escalation_chain" "SREs" {
  provider = grafana.oncall

  name = "SREs"
}

// Notify users from on-call schedule
resource "grafana_oncall_escalation" "notify_schedule_step_SREs" {
  provider = grafana.oncall

  escalation_chain_id          = grafana_oncall_escalation_chain.SREs.id
  type                         = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.schedule_SREs.id
  position                     = 0
}

// Wait step for 5 Minutes
resource "grafana_oncall_escalation" "wait_step_SREs" {
  provider = grafana.oncall

  escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
  type                = "wait"
  duration            = 300
  position            = 1
}

// Notify default Slack channel step
resource "grafana_oncall_escalation" "notify_step_SREs" {
  provider = grafana.oncall

  escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
  type                = "notify_whole_channel"
  important           = true
  position            = 2
}

Connect an integration to Grafana IRM

This Terraform configuration connects Alertmanager to Grafana IRM using the grafana_oncall_integration (Resource). It also adds the Devs escalation chain as the default route for alerts.

  1. Create a file named integrations.tf and add the following:

    resource "grafana_oncall_integration" "AlertManager" {
  provider = grafana.oncall

  name = "AlertManager"
  type = "alertmanager"
  default_route {
    escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
  }
}

  2. To configure Alertmanager, refer to the Alertmanager integration for Grafana OnCall documentation.

Set up a route to configure escalation behavior for alert group notifications

This Terraform configuration sets up a route to the Alertmanager integration using the grafana_oncall_route (Resource). This route ensures that notifications for alerts with \"namespace\" *: *\"ops-.*\" in the payload are escalated to the SREs escalation chain.

Create a file named routes.tf and add the following:

resource "grafana_oncall_route" "route_SREs" {
  provider = grafana.oncall

  integration_id      = grafana_oncall_integration.AlertManager.id
  escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
  routing_regex       = "\"namespace\" *: *\"ops-.*\""
  position            = 0
}

Apply the Terraform configuration

In a terminal, run the following commands from the directory where all of the configuration files are located.

  1. Initialize a working directory containing Terraform configuration files.

    terraform init

  2. Preview the changes that Terraform will make.

    terraform plan

  3. Apply the configuration files.

    terraform apply

Validation

After you apply the changes in the Terraform configurations, you can verify the following:

Summary

In this guide, you learned how to use Terraform to manage Grafana IRM by connecting an integration, configuring escalation policies, and setting up on-call schedules.

To learn more about managing Grafana Cloud using Terraform, refer to the Grafana provider documentation.

Install plugins in Grafana Cloud using Terraform

This guide shows you how to install plugins in Grafana Cloud using Terraform. For more information about Grafana plugins, refer to the Find and use Grafana plugins documentation.

Before you begin

Before you begin, ensure you have the following:

{{< admonition type="note" >}} Save all of the following Terraform configuration files in the same directory. {{< /admonition >}}

Configure the Grafana provider

Use this Terraform configuration to set up the Grafana provider to provide the authentication required to manage plugin resources.

  1. Create a service account and token in Grafana. For more information on creating a service account and token, refer to Service account tokens. You can also refer to Creating and managing a Grafana Cloud stack using Terraform to set up a service account and a token.

  2. Make sure that the token has the following permissions:

Next, create a file named main.tf and add the following:

   terraform {
     required_providers {
       grafana = {
         source  = "grafana/grafana"
         version = ">= 4.5.3"
       }
     }
   }

   provider "grafana" {
     cloud_api_url      = "<Stack-URL>"
     cloud_access_policy_token     = "<Service-account-token>"
   }

Replace the following field values:

Create new plugin resource

Create a file named plugins.tf and add the following:

resource "grafana_cloud_plugin_installation" "grafana-clock-panel" {
  stack_slug   = "<Your-Stack-Slug>"
  slug         = "grafana-clock-panel"
  version      = "latest"
}

Summary

In this guide, you learned how to install a plugin in Grafana Cloud using Terraform.

To learn more about plugin installation, refer to the Grafana provider documentation.

Manage secrets in Grafana Cloud using Terraform

Use the Terraform Grafana provider to create and manage secure values in Grafana Secrets Management as code. With Terraform, you keep secrets such as API keys, tokens, passwords, and certificates in version-controlled, declarative infrastructure alongside the rest of your Grafana stack.

This guide covers secure values, which you manage with the grafana_apps_secret_securevalue_v1beta1 resource. By default, Grafana stores secure values in the built-in system keeper, which requires no setup. If you configure an additional keeper and set it as the active keeper for your namespace, Grafana stores new secure values there automatically. For more information, refer to Activate and use the keeper. To store secure values in an external secret manager instead, provision a keeper.

To create and manage secure values with Terraform, you complete the following tasks:

  1. Create a service account token to authenticate the Terraform provider.
  2. Configure the Grafana provider.
  3. Define one or more secure values.
  4. Run terraform apply to provision your secure values.

Before you begin

To provision secure values with Terraform, you need the following:

You also need the following tools at the minimum supported versions:

Tool Minimum version
Terraform CLI 1.11, for write-only arguments
grafana/grafana Terraform provider 4.26.0

If you don't have the Terraform CLI, refer to the Terraform install documentation. The grafana/grafana provider downloads automatically when you run terraform init in Provision the secure value with Terraform.

{{< admonition type="note" >}} All of the following Terraform configuration files should be saved in the same directory. {{< /admonition >}}

Create a service account token

Terraform authenticates against Grafana with a service account token.

To create a service account and token, follow these steps:

  1. Create a service account and token in Grafana. To create them, refer to Service account tokens. You can also refer to Create and manage a Grafana Cloud stack using Terraform to set up a service account and token.

  2. Make sure the service account has the role-based access control (RBAC) actions needed to manage secure values:

    • secret.securevalues:create
    • secret.securevalues:read
    • secret.securevalues:write
    • secret.securevalues:delete

    If you also provision a keeper (refer to Provision a keeper), the service account also needs the keeper actions, which cover activating a keeper:

    • secret.keepers:create
    • secret.keepers:read
    • secret.keepers:write
    • secret.keepers:delete

  3. Copy the token and store it securely, because you can't view it again after you leave the page.

Configure the Grafana provider

In your Terraform working directory, create a main.tf file:

terraform {
  required_version = ">= 1.11.0"

  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 4.26.0"
    }
  }
}

variable "grafana_auth" {
  type      = string
  sensitive = true
}

provider "grafana" {
  url  = "<GRAFANA_URL>"
  auth = var.grafana_auth

  # Required for app-platform resources, including secure values.
  stack_id = <STACK_ID>
}

Replace the placeholders as follows:

Provide each input variable through a matching environment variable that follows the Terraform TF_VAR_<variable_name> naming convention. The part after TF_VAR_ is the name of the variable in your configuration: Terraform maps TF_VAR_grafana_auth to the grafana_auth variable, TF_VAR_external_api_key to external_api_key, and so on. You can also use a secrets manager or a terraform.tfvars file. Don't set variables in a file that you commit.

For other authentication options, refer to the Grafana provider documentation. For end-to-end stack management, refer to Create and manage a Grafana Cloud stack using Terraform.

Define a secure value

The following example stores an API key as a secure value and authorizes Synthetic Monitoring to decrypt it. Add it to main.tf, or to any other .tf file in the same working directory. Terraform loads all .tf files in the directory together.

variable "external_api_key" {
  type      = string
  sensitive = true
  ephemeral = true # Keeps the value out of state and plan files.
}

resource "grafana_apps_secret_securevalue_v1beta1" "external_api_key" {
  metadata {
    uid = "external-api-key" # The secure value's name.
  }

  spec {
    description = "External API key"
    value       = var.external_api_key
    decrypters  = ["synthetic-monitoring"]
  }
}

Keep the following points in mind for this example:

Schema reference

The full schema is on the Terraform Registry. The following tables describe the fields you use for secure values.

The metadata block contains the following fields:

Field Required Description
uid Yes The secure value's name. Up to 253 characters, DNS-compatible, and immutable.
folder_uid No Not used for secure values. Leave it unset.
uuid, version, url, annotations No Read-only. Grafana populates these fields.

The spec block contains the following fields:

Field Required Description
description Yes, enforced server-side 1 to 25 characters.
value Conditional Plaintext secret value, 1 to 24,576 bytes. Sensitive and write-only in Terraform 1.11 or later. Sent to Grafana on apply but never written to state or plan files. Set exactly one of value or ref.
ref Conditional Reference to a secret already stored in a third-party keeper, 1 to 1,024 characters. Valid only when the active keeper for the namespace isn't the system keeper. Set exactly one of value or ref.
decrypters No List of services permitted to decrypt this secure value. Up to 64 unique entries. Each entry must be a valid Kubernetes label value: letters, numbers, and the characters ., -, and _, with no slashes or spaces. For more information, refer to Decrypters.
value_hash Computed Read-only SHA-256 hash of the stored value. The provider uses it to detect rotation. Don't set it yourself.

The options block is optional and contains the following fields:

Field Required Description
manager_identity No Overrides the manager identity stamped on this resource. The default is grafana-terraform-provider. Useful when multiple Terraform workspaces manage resources in the same Grafana instance.
overwrite No Set to true to overwrite an existing resource with the same UID when applicable.

At the top level, the resource exposes a read-only id field, which Grafana derives from the secure value's UUID.

Decrypters

The decrypters list controls which Grafana services can decrypt a secret. If a service isn't in the list, it can't read the plaintext value, even if it has access to the secure value's metadata. An empty or unset list means no service can decrypt the value.

Grafana supports the following decrypters:

Decrypter Consumer
k6-cloud Grafana Cloud k6
synthetic-monitoring Synthetic Monitoring checks

Grant the minimum set of decrypters required for your use case.

Provision the secure value with Terraform

To apply the configuration, follow these steps:

  1. Initialize the working directory. This downloads the Grafana provider.

    terraform init

  2. Set the environment variables, then preview and apply the changes:

    export TF_VAR_grafana_auth='<TOKEN>'
export TF_VAR_external_api_key='<SECRET>'
terraform plan
terraform apply

    Replace the placeholders as follows:

    • <TOKEN> is the service account token you created in Create a service account token.
    • <SECRET> is the plaintext secret value that Terraform stores as the secure value.

    This keeps secrets out of Terraform files. If your shell writes history, prefer a secure secret-injection flow, such as CI/CD secrets, a secret manager, or a nested shell with history disabled.

    Terraform shows the execution plan and asks for confirmation:

    Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

    After you confirm, Terraform creates the secure value:

    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

If a secure value with the same metadata.uid already exists in the namespace, for example from a previous apply or one created outside Terraform, the apply fails with HTTP 409 - AlreadyExists. To bring the existing secure value under Terraform management instead of recreating it, import it. To replace it on apply, set overwrite = true in the resource's options block.

Verify that the plaintext stays out of state

Because spec.value is write-only, Terraform never writes the plaintext to state. It writes only the value_hash. To confirm this, search your local state file for the plaintext:

grep -c '<SECRET_VALUE>' terraform.tfstate

Replace <SECRET_VALUE> with the plaintext secret you provisioned. The search returns no matches:

0

The stored spec shows value as null and retains only the hash:

"spec": {
  "decrypters": ["synthetic-monitoring"],
  "description": "External API key",
  "ref": null,
  "value": null,
  "value_hash": "65f087ba121c13dec3fa0cb7697d4309ed5e624ba528a758bb9071bee220825a"
}

To verify the result, open Administration > Secrets Management in Grafana, or call the Secrets Management HTTP API directly. The namespace is stacks-<STACK_ID>:

curl -H "Authorization: Bearer <TOKEN>" \
  "<GRAFANA_URL>/apis/secret.grafana.app/v1beta1/namespaces/stacks-<STACK_ID>/securevalues/external-api-key"

The response includes the secure value's metadata, its spec with description and decrypters, and the keeper that stores it. It never includes the plaintext value:

{
  "kind": "SecureValue",
  "apiVersion": "secret.grafana.app/v1beta1",
  "metadata": {
    "name": "external-api-key",
    "namespace": "stacks-<STACK_ID>",
    "annotations": {
      "grafana.app/managedBy": "terraform",
      "grafana.app/managerId": "grafana-terraform-provider"
    }
  },
  "spec": {
    "description": "External API key",
    "decrypters": ["synthetic-monitoring"]
  },
  "status": {
    "version": 1,
    "keeper": "system"
  }
}

There is no value field in the response. The API doesn't return decrypted secret contents.

Rotate a secure value

To rotate a secret, change the value, or the variable that feeds it, and run terraform apply again.

The provider computes a SHA-256 hash of your configured spec.value at plan time and stores it in value_hash. When the hash changes, Terraform plans an update. When the hash doesn't change, Terraform plans no change. This lets you rotate secrets without storing plaintext in state.

resource "grafana_apps_secret_securevalue_v1beta1" "external_api_key" {
  metadata {
    uid = "external-api-key"
  }

  spec {
    description = "External API key"
    value       = var.external_api_key # Supply the new value here.
    decrypters  = ["synthetic-monitoring"]
  }
}

When the value changes, the recomputed value_hash drives an in-place update:

  ~ spec {
      ~ value_hash = (sensitive value)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

If you apply again without changing the value, the hash matches and Terraform plans no change:

No changes. Your infrastructure matches the configuration.

Rotation considerations

Keep the following points in mind when you rotate secure values:

Handle the secret value safely

{{< admonition type="warning" >}} Never check plaintext secret values into Terraform configuration. {{< /admonition >}}

Instead, use one of the following approaches:

Import an existing secure value

The grafana_apps_secret_securevalue_v1beta1 resource supports import. The import ID is the secure value name, which is the value of metadata.uid:

terraform import grafana_apps_secret_securevalue_v1beta1.external_api_key external-api-key

After you import a secure value, keep the following points in mind:

Provision a keeper

A keeper defines where Grafana stores the encrypted material for your secure values. By default, Grafana uses the built-in system keeper, which encrypts values at rest in the Grafana database and requires no setup.

To store secrets in an external secret manager instead, provision a keeper with the grafana_apps_secret_keeper_v1beta1 resource. Grafana currently supports AWS Secrets Manager as a keeper.

Grafana connects to AWS Secrets Manager through cross-account role assumption with AWS Security Token Service, and never stores your AWS credentials. You create an AWS Identity and Access Management (IAM) role in your AWS account that Grafana can assume, and the keeper references that role.

Keeper requirements

In addition to the requirements for secure values, you need the following:

Define a keeper

The following example provisions an AWS Secrets Manager keeper. Add it to main.tf, or to any other .tf file in the same working directory.

resource "grafana_apps_secret_keeper_v1beta1" "aws_secrets_manager" {
  metadata {
    uid = "aws-secrets-manager" # The keeper's name.
  }

  spec {
    description = "AWS Secrets Manager keeper"

    aws {
      region = "<REGION>"

      assume_role {
        assume_role_arn = "arn:aws:iam::<ACCOUNT_ID>:role/grafana-secrets-manager"
        external_id     = "<EXTERNAL_ID>"
      }
    }
  }
}

Replace the placeholders as follows:

The role name in assume_role_arn must be grafana-secrets-manager. Grafana can't assume a role with any other name.

Apply the keeper with the same terraform apply workflow you use for a secure value, described in Provision the secure value with Terraform. Unlike a secure value, a keeper has no secret value, so it needs no ephemeral variable of its own: TF_VAR_grafana_auth for the provider is the only variable you export. Creating a keeper doesn't make it active, so don't create secure values in the same apply until you add the activation resource in Activate and use the keeper and make new secrets depend on it.

Keeper schema reference

The full schema is on the Terraform Registry. The following tables describe the fields you use for a keeper.

The metadata block contains the following fields:

Field Required Description
uid Yes The keeper's name. Up to 253 characters, DNS-compatible, and immutable.

The spec block contains the following fields:

Field Required Description
description Yes Short description for the keeper, 1 to 253 characters.
aws Yes AWS Secrets Manager configuration. Grafana currently supports only AWS, so this block is required. For the fields, refer to the following aws table.

The aws block contains the following fields:

Field Required Description
region Yes AWS region that hosts your secrets, for example us-east-1.
assume_role Yes Role that Grafana assumes to access AWS Secrets Manager. Required for AWS. assume_role_arn and external_id are both required.

The assume_role block contains the following fields:

Field Required Description
assume_role_arn Yes ARN of the IAM role that Grafana assumes.
external_id Yes External ID that the IAM role's trust policy requires.

At the top level, the resource exposes a read-only active status that's true when the keeper is the active keeper for the namespace.

Activate and use the keeper

Creating a keeper with Terraform doesn't make it active. Each namespace has at most one active keeper, which determines where Grafana stores new secure values.

To set a keeper as active, use the grafana_apps_secret_keeper_activation_v1beta1 resource and reference the keeper's uid:

resource "grafana_apps_secret_keeper_activation_v1beta1" "aws_secrets_manager" {
  metadata {
    uid = grafana_apps_secret_keeper_v1beta1.aws_secrets_manager.metadata.uid
  }
}

The activation resource already depends on the keeper because it references the keeper's uid. Any secure value that should use this keeper must depend on the activation, otherwise Terraform can create the secret while the system keeper is still active:

resource "grafana_apps_secret_securevalue_v1beta1" "external_api_key" {
  metadata {
    uid = "external-api-key"
  }

  spec {
    description = "External API key"
    value       = var.external_api_key
    decrypters  = ["synthetic-monitoring"]
  }

  depends_on = [grafana_apps_secret_keeper_activation_v1beta1.aws_secrets_manager]
}

Keep the following points in mind for this resource:

You can also set the active keeper outside Terraform, in the Grafana UI under Administration > Secrets Management.

After the keeper is active, Grafana routes new secure values to it automatically. A value-backed secure value like the one in Define a secure value needs no field changes to use the active keeper, but it still needs depends_on if you create it in the same configuration as the activation. Destroying a value-backed secure value that lives in AWS Secrets Manager deletes the AWS secret immediately, with no recovery window.

To reference a secret that already exists in AWS Secrets Manager instead of sending a new value, use ref in place of value. Set ref to the AWS secret name or the full ARN, not a path inside Grafana:

resource "grafana_apps_secret_securevalue_v1beta1" "db_password" {
  metadata {
    uid = "db-password"
  }

  spec {
    description = "Production DB password"
    ref         = "grafana-secrets-manager/stacks-<STACK_ID>/db-password/1"
    decrypters  = ["synthetic-monitoring"]
  }

  depends_on = [grafana_apps_secret_keeper_activation_v1beta1.aws_secrets_manager]
}

Replace <STACK_ID> with the same numeric stack ID you set on the provider. You can also set ref to the full secret ARN, for example arn:aws:secretsmanager:<REGION>:<ACCOUNT_ID>:secret:grafana-secrets-manager/stacks-<STACK_ID>/db-password/1-AbCdEf.

If you use ref while the system keeper is active, the API returns the following error:

tried to create secure value using reference with system keeper, references can only be used with 3rd party keepers

Summary

In this guide, you created a service account token, configured the Grafana provider, and provisioned a secure value with Terraform. You also learned how to rotate and import secure values, and how to provision and activate a keeper to store secrets in an external secret manager.

To learn more about managing Grafana Cloud using Terraform, refer to the Grafana provider documentation.

Next steps

Create custom model rules using Terraform

Custom model rules in Knowledge Graph allow you to define how entities are discovered and modeled based on Prometheus queries. These rules enable you to create custom entity types, define their relationships, and specify how they should be enriched with additional data.

For information about managing entities and relations in the Knowledge Graph UI, refer to Manage entities and relations.

Basic custom model rules

Create a file named custom-model-rules.tf and add the following:

# Basic custom model rule for services
resource "grafana_asserts_custom_model_rules" "basic_service" {
  provider = grafana.asserts

  name = "basic-service-model"

  rules {
    entity {
      type = "Service"
      name = "service"

      defined_by {
        query = "up{job!=''}"
        label_values = {
          service = "job"
        }
        literals = {
          _source = "up_query"
        }
      }
    }
  }
}

Advanced service model with scope and lookup

Define service entities with environment scoping and relationship mappings:

# Advanced service model with environment scoping
resource "grafana_asserts_custom_model_rules" "advanced_service" {
  provider = grafana.asserts

  name = "advanced-service-model"

  rules {
    entity {
      type = "Service"
      name = "workload | service | job"

      scope = {
        namespace = "namespace"
        env       = "asserts_env"
        site      = "asserts_site"
      }

      lookup = {
        workload  = "workload | deployment | statefulset | daemonset | replicaset"
        service   = "service"
        job       = "job"
        proxy_job = "job"
      }

      defined_by {
        query = "up{job!='', asserts_env!=''}"
        label_values = {
          service     = "service"
          job         = "job"
          workload    = "workload"
          namespace   = "namespace"
        }
        literals = {
          _source = "up_with_workload"
        }
      }

      defined_by {
        query    = "up{job='maintenance'}"
        disabled = true
      }
    }
  }
}

Multi-entity model configuration

Define multiple entity types in a single configuration:

# Multiple entity types in a single model
resource "grafana_asserts_custom_model_rules" "multi_entity" {
  provider = grafana.asserts

  name = "kubernetes-entities"

  rules {
    # Service entity
    entity {
      type = "Service"
      name = "service"

      scope = {
        namespace = "namespace"
        cluster   = "cluster"
      }

      defined_by {
        query = "up{service!=''}"
        label_values = {
          service   = "service"
          namespace = "namespace"
          cluster   = "cluster"
        }
      }
    }

    # Pod entity
    entity {
      type = "Pod"
      name = "Pod"

      scope = {
        namespace = "namespace"
        cluster   = "cluster"
      }

      lookup = {
        service   = "service"
        workload  = "workload"
      }

      defined_by {
        query = "kube_pod_info{pod!=''}"
        label_values = {
          Pod       = "pod"
          namespace = "namespace"
          cluster   = "cluster"
          service   = "service"
        }
        literals = {
          _entity_type = "Pod"
        }
      }
    }

    # Namespace entity
    entity {
      type = "Namespace"
      name = "namespace"

      scope = {
        cluster = "cluster"
      }

      defined_by {
        query = "kube_namespace_status_phase{namespace!=''}"
        label_values = {
          namespace = "namespace"
          cluster   = "cluster"
        }
      }
    }
  }
}

Complex entity with enrichment

Create service entities with multiple data sources and enrichment:

# Service entity with enrichment from multiple sources
resource "grafana_asserts_custom_model_rules" "enriched_service" {
  provider = grafana.asserts

  name = "enriched-service-model"

  rules {
    entity {
      type = "Service"
      name = "service"

      enriched_by = [
        "prometheus_metrics",
        "kubernetes_metadata",
        "application_logs"
      ]

      scope = {
        environment = "asserts_env"
        region      = "asserts_site"
        team        = "team"
      }

      lookup = {
        deployment = "workload"
        Pod        = "pod"
        container  = "container"
      }

      # Primary definition from service up metrics
      defined_by {
        query = "up{service!='', asserts_env!=''}"
        label_values = {
          service     = "service"
          environment = "asserts_env"
          region      = "asserts_site"
          team        = "team"
        }
        literals = {
          _primary_source = "service_up"
        }
      }

      # Secondary definition from application metrics
      defined_by {
        query = "http_requests_total{service!=''}"
        label_values = {
          service     = "service"
          environment = "environment"
          version     = "version"
        }
        literals = {
          _secondary_source = "http_metrics"
        }
      }

      # Disabled definition for testing
      defined_by {
        query    = "test_metric{service!=''}"
        disabled = true
      }
    }
  }
}

Database and infrastructure entities

Define database and infrastructure entity models:

# Database and infrastructure entity models
resource "grafana_asserts_custom_model_rules" "infrastructure" {
  provider = grafana.asserts

  name = "infrastructure-entities"

  rules {
    # Database entity
    entity {
      type = "Database"
      name = "database_instance"

      scope = {
        environment = "env"
        region      = "region"
      }

      lookup = {
        host     = "instance"
        port     = "port"
        db_name  = "database"
      }

      defined_by {
        query = "mysql_up{instance!=''}"
        label_values = {
          database_instance = "instance"
          database         = "database"
          env             = "environment"
          region          = "region"
        }
        literals = {
          _db_type = "mysql"
        }
        metric_value = "1"
      }

      defined_by {
        query = "postgres_up{instance!=''}"
        label_values = {
          database_instance = "instance"
          database         = "datname"
          env             = "environment"
        }
        literals = {
          _db_type = "postgresql"
        }
      }
    }

    # Load balancer entity
    entity {
      type = "LoadBalancer"
      name = "lb_instance"

      scope = {
        environment = "env"
      }

      defined_by {
        query = "haproxy_up{proxy!=''}"
        label_values = {
          lb_instance = "instance"
          proxy      = "proxy"
          env        = "environment"
        }
        literals = {
          _lb_type = "haproxy"
        }
      }
    }
  }
}

Resource reference

grafana_asserts_custom_model_rules

Manage Knowledge Graph custom model rules through the Grafana API. This resource allows you to define custom entity models based on Prometheus queries with advanced mapping and enrichment capabilities.

Arguments

Name Type Required Description
name string Yes The name of the custom model rules. This field is immutable and forces recreation if changed.
rules list(object) Yes The rules configuration containing entity definitions. Refer to rules block for details.

Rules block

Each rules block supports the following:

Name Type Required Description
entity list(object) Yes List of entity definitions. Refer to entity block for details.

Entity block

Each entity block supports the following:

Name Type Required Description
type string Yes The type of the entity (for example, Service, Pod, Namespace).
name string Yes The name pattern for the entity. Can include pipe-separated alternatives.
defined_by list(object) Yes List of queries that define this entity. Refer to defined_by block for details.
disabled bool No Whether this entity is disabled. Defaults to false.
enriched_by list(string) No List of enrichment sources for the entity.
lookup map(string) No Lookup mappings for the entity to relate different label names.
scope map(string) No Scope labels that define the boundaries of this entity type.

defined_by block

Each defined_by block supports the following:

Name Type Required Description
query string Yes The Prometheus query that defines this entity.
disabled bool No Whether this query is disabled. Defaults to false.
label_values map(string) No Label value mappings for extracting entity attributes from query results.
literals map(string) No Literal value mappings for adding static attributes to entities.
metric_value string No Metric value to use from the query result.

{{< admonition type="note" >}} When disabled = true is set for a defined_by query, only the query field is used for matching. All other fields in the block are ignored. {{< /admonition >}}

Best practices

Consider the following best practices when creating custom model rules with Terraform.

Entity models

Query design and performance

Relationships and enrichment

Label and attribute management

Validation

After applying the Terraform configuration, verify that:

Get started with Terraform for Knowledge Graph

Learn how to configure Terraform to manage Grafana Cloud Knowledge Graph resources. This guide walks you through setting up the Grafana Terraform provider and preparing your environment.

Before you begin

Before you begin, ensure you have the following:

{{< admonition type="note" >}} You don't need to enable the Knowledge Graph before using Terraform. The grafana_asserts_stack resource handles the full onboarding flow, including enabling the Knowledge Graph on your stack. {{< /admonition >}}

{{< admonition type="note" >}} All Terraform configuration files should be saved in the same directory. {{< /admonition >}}

Configure the Grafana provider

This Terraform configuration sets up the Grafana provider to provide necessary authentication when managing knowledge graph resources.

You can reuse a similar setup to the one described in Creating and managing a Grafana Cloud stack using Terraform to set up a service account and a token.

Steps

  1. Create a Service account and token in Grafana.

    To create a new one, refer to Service account tokens.

  2. Create a file named main.tf and add the following:

    terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 2.9.0"
    }
  }
}

provider "grafana" {
  alias = "asserts"

  url      = "<Stack-URL>"
  auth     = "<Service-account-token>"
  stack_id = "<Stack-ID>"
}

  3. Replace the following field values:

    • <Stack-URL> with the URL of your Grafana stack (for example, https://my-stack.grafana.net/)
    • <Service-account-token> with the service account token that you created
    • <Stack-ID> with your Grafana Cloud stack ID

{{< admonition type="note" >}} The stack_id parameter is required for Knowledge Graph resources to identify the stack where the resources belong. {{< /admonition >}}

Apply Terraform configurations

After creating your Terraform configuration files, apply them using the following commands:

  1. Initialize a working directory containing Terraform configuration files:

    terraform init

  2. Preview the changes that Terraform makes:

    terraform plan

  3. Apply the configuration files:

    terraform apply

Verify your setup

After applying the configuration, verify your setup by checking that:

Best practices

When managing Knowledge Graph resources with Terraform, consider the following best practices.

Name conventions

Version control

State management

Security

Next steps

Now that you have configured the Terraform provider, you can start managing knowledge graph resources:

Configure the Knowledge Graph stack using Terraform

The Knowledge Graph stack resource manages the full onboarding flow for your Grafana Cloud stack. It provisions API tokens, configures datasets based on available metrics, and enables the stack.

By default, datasets are auto-configured based on detected metrics. To manually configure datasets, for example when your metrics use non-standard label names, use the dataset block.

Before you begin

Before you begin, ensure you have the following:

Dataset types

The following dataset types are available:

Create the required tokens

Before you configure the Knowledge Graph stack, create the Cloud Access Policy and Grafana Service Account tokens that the resource requires.

Create a file named knowledge-graph-stack.tf and add the following:

# Create a Cloud Access Policy with required scopes
resource "grafana_cloud_access_policy" "knowledge_graph" {
  name         = "knowledge-graph-stack-policy"
  display_name = "Knowledge Graph Stack Policy"

  scopes = [
    "stacks:read",
    "metrics:read",
    "metrics:write",
  ]

  realm {
    type       = "stack"
    identifier = var.stack_id
  }
}

# Create a token from the Cloud Access Policy
resource "grafana_cloud_access_policy_token" "knowledge_graph" {
  name             = "knowledge-graph-stack-token"
  access_policy_id = grafana_cloud_access_policy.knowledge_graph.policy_id
}

# Create a Grafana Service Account for dashboards and Grafana Managed Alerts
resource "grafana_cloud_stack_service_account" "knowledge_graph" {
  stack_slug  = var.stack_slug
  name        = "knowledge-graph-managed-alerts-sa"
  role        = "Admin"
  is_disabled = false
}

resource "grafana_cloud_stack_service_account_token" "knowledge_graph" {
  stack_slug         = var.stack_slug
  service_account_id = grafana_cloud_stack_service_account.knowledge_graph.id
  name               = "knowledge-graph-managed-alerts-token"
}

variable "stack_id" {
  description = "The Grafana Cloud stack ID"
  type        = string
}

variable "stack_slug" {
  description = "The Grafana Cloud stack slug"
  type        = string
}

Replace the following values:

Basic stack configuration with auto-detected datasets

The simplest configuration provisions tokens and auto-detects datasets based on available metrics. This is the recommended approach when your metrics use standard label names.

Add the following to knowledge-graph-stack.tf:

resource "grafana_asserts_stack" "main" {
  cloud_access_policy_token = grafana_cloud_access_policy_token.knowledge_graph.token
  grafana_token             = grafana_cloud_stack_service_account_token.knowledge_graph.key
}

When you apply this configuration, the resource:

  1. Provisions API tokens for Grafana Cloud, Mimir, and the assertion detector
  2. Auto-detects available datasets based on your stack's metrics
  3. Enables the stack with the detected datasets

Configure the dataset manually

Use manual configuration when your metrics use non-standard label names, for example a custom environment label. Each dataset block configures one dataset type.

resource "grafana_asserts_stack" "manual" {
  cloud_access_policy_token = grafana_cloud_access_policy_token.knowledge_graph.token
  grafana_token             = grafana_cloud_stack_service_account_token.knowledge_graph.key

  dataset {
    type = "kubernetes"

    filter_group {
      env_label  = "deployment_environment"
      site_label = "cluster"

      env_label_values  = ["production", "staging"]
      site_label_values = ["us-east-1", "eu-west-1"]
    }
  }
}

{{< admonition type="note" >}} When you specify one or more dataset blocks, auto-detection is skipped. You must define all datasets you want to configure. {{< /admonition >}}

Configure multiple datasets with filters

Configure multiple dataset types with custom label mappings and metric filters:

resource "grafana_asserts_stack" "multi_dataset" {
  cloud_access_policy_token = grafana_cloud_access_policy_token.knowledge_graph.token
  grafana_token             = grafana_cloud_stack_service_account_token.knowledge_graph.key

  dataset {
    type = "kubernetes"

    filter_group {
      env_label  = "deployment_environment"
      site_label = "cluster"

      env_label_values  = ["production", "staging"]
      site_label_values = ["us-east-1", "eu-west-1"]
    }
  }

  dataset {
    type = "prometheus"

    filter_group {
      env_label = "environment"
      env_name  = "prod"

      filter {
        name     = "region"
        operator = "=~"
        values   = ["us-.*", "eu-.*"]
      }
    }
  }
}

Resource reference

grafana_asserts_stack

Manage the Knowledge Graph stack configuration through the Grafana API. This resource handles the full onboarding flow including token provisioning, dataset configuration, and stack enablement.

Arguments

Name Type Required Description
cloud_access_policy_token string Yes A Grafana Cloud Access Policy token with the following scopes: stacks:read, metrics:read, metrics:write. Used for Grafana Cloud API access, Mimir authentication, and assertion detector webhook authentication. Sensitive.
grafana_token string No A Grafana Service Account token for installing dashboards and Grafana Managed Alerts. Create using grafana_cloud_stack_service_account_token. Sensitive.
dataset list(object) No Manual dataset configuration. When specified, auto-detection is skipped. Refer to dataset block for details.

Attributes

Name Type Description
enabled bool Whether the stack is currently enabled.
status string Current onboarding status of the stack.
version int Configuration version number.

Dataset block

Each dataset block supports the following:

Name Type Required Description
type string Yes The dataset type: kubernetes, otel, prometheus, or aws.
disabled_vendors list(string) No List of vendors to disable for this dataset.
filter_group list(object) No Filter groups for custom label mappings. Refer to filter group block for details.

Filter group block

Each filter_group block supports the following:

Name Type Required Description
env_label string No The metric label name used for environment (for example, env, environment, deployment_environment).
env_name string No A friendly name for the environment.
site_label string No The metric label name used for site or cluster.
env_label_values list(string) No Specific values of the environment label to match.
site_label_values list(string) No Specific values of the site label to match.
filter list(object) No Additional metric filters. Refer to filter block for details.

Filter block

Each filter block supports the following:

Name Type Required Description
name string Yes The label name to filter on.
operator string Yes The filter operator: =, !=, =~, or !~.
values list(string) Yes The values to match against.

Example

resource "grafana_asserts_stack" "example" {
  cloud_access_policy_token = grafana_cloud_access_policy_token.knowledge_graph.token
  grafana_token             = grafana_cloud_stack_service_account_token.knowledge_graph.key

  dataset {
    type = "kubernetes"

    filter_group {
      env_label         = "deployment_environment"
      site_label        = "cluster"
      env_label_values  = ["production"]
      site_label_values = ["us-east-1"]
    }
  }
}

Best practices

Consider the following best practices when configuring the Knowledge Graph stack with Terraform.

Token management

Dataset configuration

Filter groups

Validation

After applying the Terraform configuration, verify that:

You can inspect the stack status using Terraform outputs:

output "stack_enabled" {
  value       = grafana_asserts_stack.main.enabled
  description = "Whether the Knowledge Graph stack is enabled"
}

output "stack_status" {
  value       = grafana_asserts_stack.main.status
  description = "Current onboarding status of the stack"
}

output "stack_version" {
  value       = grafana_asserts_stack.main.version
  description = "Configuration version number"
}

Configure log correlation using Terraform

Log configurations in Knowledge Graph allow you to define how log data is queried and correlated with entities. You can specify data sources, entity matching rules, label mappings, and filtering options for spans and traces.

For information about configuring log correlation in the Knowledge Graph UI, refer to Configure logs correlation.

Basic log configuration

Create a file named log-configs.tf and add the following:

# Basic log configuration for services
resource "grafana_asserts_log_config" "production" {
  provider = grafana.asserts

  name            = "production"
  priority        = 1000
  default_config  = false
  data_source_uid = "grafanacloud-logs"
  error_label     = "error"

  match {
    property = "asserts_entity_type"
    op       = "EQUALS"
    values   = ["Service"]
  }

  match {
    property = "environment"
    op       = "EQUALS"
    values   = ["production", "staging"]
  }

  entity_property_to_log_label_mapping = {
    "otel_namespace" = "service_namespace"
    "otel_service"   = "service_name"
    "environment"    = "env"
    "site"           = "region"
  }

  filter_by_span_id  = true
  filter_by_trace_id = true
}

Log configuration with multiple match rules

Configure log correlation with multiple entity matching criteria:

# Development environment log configuration
resource "grafana_asserts_log_config" "development" {
  provider = grafana.asserts

  name            = "development"
  priority        = 2000
  default_config  = true
  data_source_uid = "elasticsearch-dev"
  error_label     = "error"

  match {
    property = "asserts_entity_type"
    op       = "EQUALS"
    values   = ["Service"]
  }

  match {
    property = "environment"
    op       = "EQUALS"
    values   = ["development", "testing"]
  }

  match {
    property = "site"
    op       = "EQUALS"
    values   = ["us-east-1"]
  }

  match {
    property = "service"
    op       = "EQUALS"
    values   = ["api"]
  }

  entity_property_to_log_label_mapping = {
    "otel_namespace" = "service_namespace"
    "otel_service"   = "service_name"
    "environment"    = "env"
    "site"           = "region"
    "service"        = "app"
  }

  filter_by_span_id  = true
  filter_by_trace_id = true
}

Minimal log configuration

Create a minimal configuration for all entities:

# Minimal configuration for all entities
resource "grafana_asserts_log_config" "minimal" {
  provider = grafana.asserts

  name            = "minimal"
  priority        = 3000
  default_config  = false
  data_source_uid = "loki-minimal"

  match {
    property = "asserts_entity_type"
    op       = "IS_NOT_NULL"
    values   = []
  }
}

Advanced log configuration with complex match rules

Configure logs with multiple operations and advanced match rules:

# Advanced configuration with multiple operations
resource "grafana_asserts_log_config" "advanced" {
  provider = grafana.asserts

  name            = "advanced"
  priority        = 1500
  default_config  = false
  data_source_uid = "loki-advanced"
  error_label     = "level"

  match {
    property = "service_type"
    op       = "CONTAINS"
    values   = ["web", "api"]
  }

  match {
    property = "environment"
    op       = "NOT_EQUALS"
    values   = ["test"]
  }

  match {
    property = "team"
    op       = "IS_NOT_NULL"
    values   = []
  }

  entity_property_to_log_label_mapping = {
    "service_type"   = "type"
    "team"           = "owner"
    "environment"    = "env"
    "version"        = "app_version"
  }

  filter_by_span_id  = true
  filter_by_trace_id = false
}

Resource reference

grafana_asserts_log_config

Manage Knowledge Graph log configurations through the Grafana API.

Arguments

Name Type Required Description
name string Yes The name of the log configuration. This field is immutable and forces recreation if changed.
priority number Yes Priority of the log configuration. Higher priority configurations are evaluated first.
default_config bool Yes Whether this is the default configuration. Default configurations cannot be deleted.
data_source_uid string Yes DataSource UID to be queried (for example, a Loki instance).
match list(object) No List of match rules for entity properties. Refer to match block for details.
error_label string No Label name used to identify error logs.
entity_property_to_log_label_mapping map(string) No Mapping of entity properties to log labels for correlation.
filter_by_span_id bool No Whether to filter logs by span ID for distributed tracing correlation.
filter_by_trace_id bool No Whether to filter logs by trace ID for distributed tracing correlation.

Match block

Each match block supports the following:

Name Type Required Description
property string Yes Entity property to match against.
op string Yes Operation to use for matching. One of: EQUALS, NOT_EQUALS, CONTAINS, DOES_NOT_CONTAIN, IS_NULL, IS_NOT_NULL.
values list(string) Yes Values to match against. Can be empty for IS_NULL and IS_NOT_NULL operations.

Example

resource "grafana_asserts_log_config" "example" {
  provider = grafana.asserts

  name            = "example-logs"
  priority        = 1000
  default_config  = false
  data_source_uid = "loki-prod"
  error_label     = "level"

  match {
    property = "asserts_entity_type"
    op       = "EQUALS"
    values   = ["Service", "Pod"]
  }

  entity_property_to_log_label_mapping = {
    "service"     = "app"
    "namespace"   = "k8s_namespace"
    "environment" = "env"
  }

  filter_by_span_id  = true
  filter_by_trace_id = true
}

Best practices

Consider the following best practices when configuring log correlation with Terraform.

Priority management

Data source configuration

Label map strategy

Match rules design

Distributed trace integration

Validation

After applying the Terraform configuration, verify that:

Configure notification alerts using Terraform

Notification alerts configurations in Knowledge Graph allow you to manage how alerts are processed and routed. You can specify match labels to filter alerts, add custom labels, set duration requirements, and control silencing.

For information about configuring notification alerts in the Knowledge Graph UI, refer to Configure notifications.

Basic notification alerts configuration

Create a file named alert-configs.tf and add the following:

# Basic alert configuration with silencing
resource "grafana_asserts_notification_alerts_config" "prometheus_remote_storage_failures" {
  provider = grafana.asserts

  name = "PrometheusRemoteStorageFailures"

  match_labels = {
    alertname   = "PrometheusRemoteStorageFailures"
    alertgroup  = "prometheus.alerts"
    asserts_env = "prod"
  }

  silenced = true
}

# High severity alert with specific job and context matching
resource "grafana_asserts_notification_alerts_config" "error_buildup_notify" {
  provider = grafana.asserts

  name = "ErrorBuildupNotify"

  match_labels = {
    alertname               = "ErrorBuildup"
    job                     = "acai"
    asserts_request_type    = "inbound"
    asserts_request_context = "/auth"
  }

  silenced = false
}

Notification alerts with additional labels and duration

Configure alerts with custom labels and timing requirements:

# Alert with additional labels and custom duration
resource "grafana_asserts_notification_alerts_config" "payment_test_alert" {
  provider = grafana.asserts

  name = "PaymentTestAlert"

  match_labels = {
    alertname         = "PaymentTestAlert"
    additional_labels = "asserts_severity=~\"critical\""
    alertgroup        = "alex-k8s-integration-test.alerts"
  }

  alert_labels = {
    testing = "onetwothree"
  }

  duration = "5m"
  silenced = false
}

Latency and performance notification alerts

Monitor and alert on latency and performance issues:

# Latency alert for shipping service
resource "grafana_asserts_notification_alerts_config" "high_shipping_latency" {
  provider = grafana.asserts

  name = "high shipping latency"

  match_labels = {
    alertname            = "LatencyP99ErrorBuildup"
    job                  = "shipping"
    asserts_request_type = "inbound"
  }

  silenced = false
}

# CPU throttling alert with warning severity
resource "grafana_asserts_notification_alerts_config" "cpu_throttling_sustained" {
  provider = grafana.asserts

  name = "CPUThrottlingSustained"

  match_labels = {
    alertname         = "CPUThrottlingSustained"
    additional_labels = "asserts_severity=~\"warning\""
  }

  silenced = true
}

Infrastructure and service notification alerts

Configure alerts for infrastructure components and services:

# Ingress error rate alert
resource "grafana_asserts_notification_alerts_config" "ingress_error" {
  provider = grafana.asserts

  name = "ingress error"

  match_labels = {
    alertname            = "ErrorRatioBreach"
    job                  = "ingress-nginx-controller-metrics"
    asserts_request_type = "inbound"
  }

  silenced = false
}

# MySQL Galera cluster alert
resource "grafana_asserts_notification_alerts_config" "mysql_galera_not_ready" {
  provider = grafana.asserts

  name = "MySQLGaleraNotReady"

  match_labels = {
    alertname = "MySQLGaleraNotReady"
  }

  silenced = false
}

Resource reference

grafana_asserts_notification_alerts_config

Manage Knowledge Graph notification alerts configurations through the Grafana API.

Arguments

Name Type Required Description
name string Yes The name of the notification alerts configuration. This field is immutable and forces recreation if changed.
match_labels map(string) No Labels to match for this notification alerts configuration. Used to filter which alerts this configuration applies to.
alert_labels map(string) No Labels to add to alerts generated by this notification alerts configuration.
duration string No Duration for which the condition must be true before firing (for example, '5m', '30s'). Maps to 'for' in Knowledge Graph API.
silenced bool No Whether this notification alerts configuration is silenced. Defaults to false.

Example

resource "grafana_asserts_notification_alerts_config" "example" {
  provider = grafana.asserts

  name = "ExampleAlert"

  match_labels = {
    alertname = "HighCPUUsage"
    job       = "monitoring"
  }

  alert_labels = {
    severity = "warning"
    team     = "platform"
  }

  duration = "5m"
  silenced = false
}

Best practices

Consider the following best practices when managing notification alerts with Terraform.

Label management

Silence strategy

Duration configuration

Validation

After applying the Terraform configuration, verify that:

Configure profile correlation using Terraform

Profile configurations in Knowledge Graph allow you to define how continuous profiling data is queried and correlated with entities. You can specify data sources, entity matching rules, and label mappings for performance profiling.

For information about configuring profile correlation in the Knowledge Graph UI, refer to Configure profiles correlation.

Basic profile configuration

Create a file named profile-configs.tf and add the following:

# Basic profile configuration for services
resource "grafana_asserts_profile_config" "production" {
  provider = grafana.asserts

  name            = "production"
  priority        = 1000
  default_config  = false
  data_source_uid = "grafanacloud-profiles"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service"]
  }

  match {
    property = "deployment_environment"
    op       = "="
    values   = ["production", "staging"]
  }

  entity_property_to_profile_label_mapping = {
    "cluster"        = "k8s_cluster_name"
    "namespace"      = "k8s_namespace_name"
    "container"      = "k8s_container_name"
    "otel_service"   = "service_name"
    "otel_namespace" = "service_namespace"
  }
}

Profile configuration with multiple match rules

Configure profile correlation with multiple entity matching criteria:

# Development environment profile configuration
resource "grafana_asserts_profile_config" "development" {
  provider = grafana.asserts

  name            = "development"
  priority        = 2000
  default_config  = false
  data_source_uid = "pyroscope-dev"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service"]
  }

  match {
    property = "deployment_environment"
    op       = "="
    values   = ["development", "testing"]
  }

  match {
    property = "asserts_site"
    op       = "="
    values   = ["us-east-1"]
  }

  match {
    property = "service"
    op       = "="
    values   = ["api"]
  }

  entity_property_to_profile_label_mapping = {
    "cluster"        = "k8s_cluster_name"
    "namespace"      = "k8s_namespace_name"
    "container"      = "k8s_container_name"
    "otel_service"   = "service_name"
    "otel_namespace" = "service_namespace"
    "pod"            = "k8s_pod_name"
  }
}

Minimal profile configuration

Create a minimal configuration for all entities:

# Minimal configuration for all entities
resource "grafana_asserts_profile_config" "minimal" {
  provider = grafana.asserts

  name            = "minimal"
  priority        = 3000
  default_config  = false
  data_source_uid = "pyroscope-minimal"

  match {
    property = "asserts_entity_type"
    op       = "IS NOT NULL"
    values   = []
  }

  entity_property_to_profile_label_mapping = {
    "cluster"        = "k8s_cluster_name"
    "otel_service"   = "service_name"
    "otel_namespace" = "service_namespace"
  }
}

Advanced profile configuration with complex match rules

Configure profiles with multiple operations and advanced match rules:

# Advanced configuration with multiple operations
resource "grafana_asserts_profile_config" "advanced" {
  provider = grafana.asserts

  name            = "advanced"
  priority        = 1500
  default_config  = false
  data_source_uid = "pyroscope-advanced"

  match {
    property = "service_type"
    op       = "CONTAINS"
    values   = ["web", "api"]
  }

  match {
    property = "deployment_environment"
    op       = "<>"
    values   = ["test"]
  }

  match {
    property = "team"
    op       = "IS NOT NULL"
    values   = []
  }

  match {
    property = "cpu_threshold"
    op       = ">="
    values   = ["80"]
  }

  entity_property_to_profile_label_mapping = {
    "service_type"   = "service_type"
    "team"           = "team_owner"
    "environment"    = "deployment_env"
    "version"        = "app_version"
    "region"         = "cloud_region"
    "node"           = "k8s_node_name"
  }
}

Resource reference

grafana_asserts_profile_config

Manage Knowledge Graph profile configurations through the Grafana API.

Arguments

Name Type Required Description
name string Yes The name of the profile configuration. This field is immutable and forces recreation if changed.
priority number Yes Priority of the profile configuration. A lower number means a higher priority.
default_config bool Yes Whether this is the default configuration. Default configurations cannot be deleted.
data_source_uid string Yes DataSource UID to be queried (for example, a Pyroscope instance).
match list(object) No List of match rules for entity properties. Refer to match block for details.
entity_property_to_profile_label_mapping map(string) No Mapping of entity properties to profile labels for correlation.

Match block

Each match block supports the following:

Name Type Required Description
property string Yes Entity property to match against.
op string Yes Operation to use for matching. One of: =, <>, <, >, <=, >=, IS NULL, IS NOT NULL, STARTS WITH, CONTAINS.
values list(string) Yes Values to match against. Can be empty for IS NULL and IS NOT NULL operations.

Example

resource "grafana_asserts_profile_config" "example" {
  provider = grafana.asserts

  name            = "example-profiles"
  priority        = 1000
  default_config  = false
  data_source_uid = "pyroscope-prod"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service", "Pod"]
  }

  match {
    property = "deployment_environment"
    op       = "STARTS WITH"
    values   = ["prod"]
  }

  entity_property_to_profile_label_mapping = {
    "service"     = "service_name"
    "namespace"   = "k8s_namespace_name"
    "environment" = "deployment_env"
    "cluster"     = "k8s_cluster_name"
    "container"   = "k8s_container_name"
  }
}

Best practices

Consider the following best practices when configuring profile correlation with Terraform.

Priority management

Data source configuration

Label map strategy

Match rules design

Continuous profiling integration

Validation

After applying the Terraform configuration, verify that:

Manage Prometheus rules using Terraform

Prometheus rules in the Knowledge Graph allow you to define custom recording and alerting rules that are evaluated against your metrics data. Recording rules pre-compute frequently used or computationally expensive expressions and save the results as new time series. Alerting rules define conditions that, when met, trigger alerts.

Using the grafana_asserts_prom_rule_file resource, you can manage these rules as code, enabling version control, review processes, and consistent deployments across environments.

Before you begin

To manage Prometheus rules using Terraform, you need:

Create a basic recording rule

Recording rules allow you to pre-compute PromQL expressions and store the results as new metrics. This is useful for expensive queries that you run frequently.

Create a file named prom-rules.tf and add the following:

# Basic recording rule for request rate
resource "grafana_asserts_prom_rule_file" "request_rates" {
  provider = grafana.asserts

  name   = "request-rates"
  active = true

  group {
    name     = "request_rate_rules"
    interval = "30s"

    rule {
      record = "job:http_requests_total:rate5m"
      expr   = "sum(rate(http_requests_total[5m])) by (job)"

      labels = {
        aggregation = "job"
        source      = "custom"
      }
    }
  }
}

Create alerting rules

Alerting rules define conditions that trigger alerts when met. Use the alert field instead of record to define an alerting rule.

# Alerting rules for service health
resource "grafana_asserts_prom_rule_file" "service_alerts" {
  provider = grafana.asserts

  name   = "service-health-alerts"
  active = true

  group {
    name     = "service_health"
    interval = "1m"

    rule {
      alert    = "HighErrorRate"
      expr     = "sum(rate(http_requests_total{code=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) > 0.05"
      duration = "5m"

      labels = {
        severity = "critical"
        team     = "platform"
      }

      annotations = {
        summary     = "High error rate detected"
        description = "Error rate is above 5% for the last 5 minutes"
        runbook_url = "https://docs.example.com/runbooks/high-error-rate"
      }
    }

    rule {
      alert    = "ServiceDown"
      expr     = "up == 0"
      duration = "2m"

      labels = {
        severity = "critical"
      }

      annotations = {
        summary     = "Service is down"
        description = "{{ $labels.job }} has been down for more than 2 minutes"
      }
    }
  }
}

Create multiple rule groups

# Multiple rule groups for comprehensive monitoring
resource "grafana_asserts_prom_rule_file" "comprehensive_rules" {
  provider = grafana.asserts

  name   = "comprehensive-monitoring"
  active = true

  # Latency recording rules
  group {
    name     = "latency_recording"
    interval = "30s"

    rule {
      record = "job:http_request_duration_seconds:p99"
      expr   = "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))"

      labels = {
        quantile = "0.99"
      }
    }

    rule {
      record = "job:http_request_duration_seconds:p95"
      expr   = "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))"

      labels = {
        quantile = "0.95"
      }
    }

    rule {
      record = "job:http_request_duration_seconds:p50"
      expr   = "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))"

      labels = {
        quantile = "0.50"
      }
    }
  }

  # Latency alerting rules
  group {
    name     = "latency_alerts"
    interval = "1m"

    rule {
      alert    = "HighP99Latency"
      expr     = "job:http_request_duration_seconds:p99 > 1"
      duration = "5m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "High P99 latency detected"
        description = "P99 latency for {{ $labels.job }} is above 1 second"
      }
    }

    rule {
      alert    = "CriticalLatency"
      expr     = "job:http_request_duration_seconds:p99 > 5"
      duration = "2m"

      labels = {
        severity = "critical"
      }

      annotations = {
        summary     = "Critical latency detected"
        description = "P99 latency for {{ $labels.job }} is above 5 seconds"
      }
    }
  }

  # Throughput rules
  group {
    name     = "throughput_rules"
    interval = "1m"

    rule {
      record = "job:http_requests:rate1m"
      expr   = "sum(rate(http_requests_total[1m])) by (job)"
    }

    rule {
      alert    = "LowThroughput"
      expr     = "job:http_requests:rate1m < 10"
      duration = "10m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "Low throughput detected"
        description = "Request rate for {{ $labels.job }} is below 10 requests per second"
      }
    }
  }
}

Create resource utilization rules

Define rules to monitor resource utilization across your infrastructure:

# Resource utilization monitoring rules
resource "grafana_asserts_prom_rule_file" "resource_utilization" {
  provider = grafana.asserts

  name   = "resource-utilization"
  active = true

  group {
    name     = "cpu_rules"
    interval = "30s"

    rule {
      record = "instance:cpu_utilization:avg5m"
      expr   = "1 - avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) by (instance)"
    }

    rule {
      alert    = "HighCPUUtilization"
      expr     = "instance:cpu_utilization:avg5m > 0.85"
      duration = "10m"

      labels = {
        severity = "warning"
        resource = "cpu"
      }

      annotations = {
        summary     = "High CPU utilization"
        description = "CPU utilization on {{ $labels.instance }} is above 85%"
      }
    }

    rule {
      alert    = "CriticalCPUUtilization"
      expr     = "instance:cpu_utilization:avg5m > 0.95"
      duration = "5m"

      labels = {
        severity = "critical"
        resource = "cpu"
      }

      annotations = {
        summary     = "Critical CPU utilization"
        description = "CPU utilization on {{ $labels.instance }} is above 95%"
      }
    }
  }

  group {
    name     = "memory_rules"
    interval = "30s"

    rule {
      record = "instance:memory_utilization:ratio"
      expr   = "1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)"
    }

    rule {
      alert    = "HighMemoryUtilization"
      expr     = "instance:memory_utilization:ratio > 0.85"
      duration = "10m"

      labels = {
        severity = "warning"
        resource = "memory"
      }

      annotations = {
        summary     = "High memory utilization"
        description = "Memory utilization on {{ $labels.instance }} is above 85%"
      }
    }
  }

  group {
    name     = "disk_rules"
    interval = "1m"

    rule {
      record = "instance:disk_utilization:ratio"
      expr   = "1 - (node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"})"
    }

    rule {
      alert    = "DiskSpaceLow"
      expr     = "instance:disk_utilization:ratio > 0.80"
      duration = "15m"

      labels = {
        severity = "warning"
        resource = "disk"
      }

      annotations = {
        summary     = "Disk space running low"
        description = "Disk utilization on {{ $labels.instance }} is above 80%"
      }
    }
  }
}

Create Kubernetes-specific rules

Define rules for monitoring Kubernetes workloads:

# Kubernetes workload monitoring rules
resource "grafana_asserts_prom_rule_file" "kubernetes_rules" {
  provider = grafana.asserts

  name   = "kubernetes-workloads"
  active = true

  group {
    name     = "kubernetes_pod_rules"
    interval = "30s"

    rule {
      record = "namespace:pod_restarts:rate1h"
      expr   = "sum(increase(kube_pod_container_status_restarts_total[1h])) by (namespace)"
    }

    rule {
      alert    = "PodCrashLooping"
      expr     = "rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 3"
      duration = "5m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "Pod is crash looping"
        description = "Pod {{ $labels.namespace }}/{{ $labels.pod }} is restarting frequently"
      }
    }

    rule {
      alert    = "PodNotReady"
      expr     = "kube_pod_status_ready{condition=\"true\"} == 0"
      duration = "10m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "Pod not ready"
        description = "Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for more than 10 minutes"
      }
    }
  }

  group {
    name     = "kubernetes_deployment_rules"
    interval = "1m"

    rule {
      record = "deployment:replicas_unavailable:count"
      expr   = "kube_deployment_status_replicas_unavailable"
    }

    rule {
      alert    = "DeploymentReplicasMismatch"
      expr     = "kube_deployment_spec_replicas != kube_deployment_status_replicas_available"
      duration = "10m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "Deployment replicas mismatch"
        description = "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for more than 10 minutes"
      }
    }
  }

  group {
    name     = "kubernetes_node_rules"
    interval = "1m"

    rule {
      alert    = "NodeNotReady"
      expr     = "kube_node_status_condition{condition=\"Ready\",status=\"true\"} == 0"
      duration = "5m"

      labels = {
        severity = "critical"
      }

      annotations = {
        summary     = "Kubernetes node not ready"
        description = "Node {{ $labels.node }} has been unready for more than 5 minutes"
      }
    }

    rule {
      alert    = "NodeMemoryPressure"
      expr     = "kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"} == 1"
      duration = "5m"

      labels = {
        severity = "warning"
      }

      annotations = {
        summary     = "Node under memory pressure"
        description = "Node {{ $labels.node }} is under memory pressure"
      }
    }
  }
}

Use conditional rule disabling

Disable specific rules in certain groups using the disable_in_groups field:

# Rules with conditional disabling
resource "grafana_asserts_prom_rule_file" "conditional_rules" {
  provider = grafana.asserts

  name   = "conditional-alerting"
  active = true

  group {
    name     = "production_alerts"
    interval = "1m"

    rule {
      alert    = "HighErrorRate"
      expr     = "sum(rate(http_requests_total{code=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) > 0.01"
      duration = "5m"

      labels = {
        severity    = "critical"
        environment = "production"
      }

      annotations = {
        summary     = "High error rate in production"
        description = "Error rate is above 1% for the last 5 minutes"
      }

      # Disable this rule in staging group
      disable_in_groups = ["staging_alerts"]
    }
  }

  group {
    name     = "staging_alerts"
    interval = "1m"

    rule {
      alert    = "HighErrorRate"
      expr     = "sum(rate(http_requests_total{code=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) > 0.10"
      duration = "10m"

      labels = {
        severity    = "warning"
        environment = "staging"
      }

      annotations = {
        summary     = "High error rate in staging"
        description = "Error rate is above 10% for the last 10 minutes"
      }
    }
  }
}

Manage inactive rule files

Temporarily disable an entire rules file without deleting it:

# Inactive rules file (not evaluated)
resource "grafana_asserts_prom_rule_file" "experimental_rules" {
  provider = grafana.asserts

  name   = "experimental-rules"
  active = false  # Rules are not evaluated

  group {
    name     = "experimental_alerts"
    interval = "1m"

    rule {
      alert    = "ExperimentalAlert"
      expr     = "some_experimental_metric > 100"
      duration = "5m"

      labels = {
        severity = "info"
      }

      annotations = {
        summary = "Experimental alert triggered"
      }
    }
  }
}

Resource reference

grafana_asserts_prom_rule_file

Manage Prometheus recording and alerting rules through the Knowledge Graph API. This resource allows you to create and manage custom Prometheus rules that are evaluated against your metrics data.

Arguments

Name Type Required Description
name string Yes The name of the Prometheus rules file. This field is immutable and forces recreation if changed.
active bool No Whether the rules file is active. Inactive rules are not evaluated. Defaults to true.
group list(object) Yes List of Prometheus rule groups. Refer to group block for details.

Group block

Each group block contains a set of related rules with a shared evaluation interval:

Name Type Required Description
name string Yes The name of the rule group (for example, latency_monitoring).
interval string No Evaluation interval for this group (for example, 30s, 1m). If not specified, uses the global evaluation interval.
rule list(object) Yes List of Prometheus rules in this group. Refer to rule block for details.

Rule block

Each rule block defines a recording or alerting rule. Either record or alert must be specified, but not both:

Name Type Required Description
record string Conditional The name of the time series to output for recording rules. Required if alert is not specified.
alert string Conditional The name of the alert for alerting rules. Required if record is not specified.
expr string Yes The PromQL expression to evaluate.
duration string No How long the condition must be true before firing the alert (for example, 5m). Only for alerting rules. Maps to for in Prometheus.
labels map(string) No Labels to attach to the resulting time series or alert.
annotations map(string) No Annotations to add to alerts (for example, summary, description). Only applicable for alerting rules.
disable_in_groups set(string) No List of group names where this rule should be disabled. Useful for conditional rule enablement.

Example

resource "grafana_asserts_prom_rule_file" "example" {
  provider = grafana.asserts

  name   = "example-rules"
  active = true

  group {
    name     = "example_group"
    interval = "1m"

    # Recording rule
    rule {
      record = "job:http_requests:rate5m"
      expr   = "sum(rate(http_requests_total[5m])) by (job)"
    }

    # Alerting rule
    rule {
      alert    = "HighErrorRate"
      expr     = "job:http_errors:rate5m > 0.05"
      duration = "5m"

      labels = {
        severity = "critical"
      }

      annotations = {
        summary     = "High error rate detected"
        description = "Error rate for {{ $labels.job }} is above 5%"
      }
    }
  }
}

Best practices

Consider the following best practices when managing Prometheus rules with Terraform.

Rule organization

Recording rules

Alerting rules

Performance considerations

Testing and validation

Validation

After applying the Terraform configuration, verify that:

Troubleshooting

Rules not being evaluated

If your rules are not being evaluated:

  1. Verify that active = true is set on the rule file
  2. Check that the rule filename follows naming validation rules (alphanumeric, hyphens, underscores)
  3. Ensure the PromQL expression is valid by testing it in Grafana

Recording rule not producing data

If a recording rule is not producing data:

  1. Verify the source metrics exist and have data
  2. Check the PromQL expression returns results
  3. Ensure the evaluation interval is appropriate
  4. Review the rule evaluation logs for errors

Alerts not firing

If alerting rules are not firing:

  1. Verify the condition is actually met by querying the expression manually
  2. Check that the duration period has elapsed
  3. Ensure the alerting system is properly configured to receive alerts
  4. Review annotation templates for syntax errors

Import error when recreating rules

If you receive errors when recreating rules:

Configure suppressed assertions using Terraform

Suppressed assertions configurations allow you to disable specific alerts or assertions based on label matching in Knowledge Graph. This is useful for maintenance windows, test environments, or when you want to temporarily suppress certain types of alerts.

For information about suppressing insights in the Knowledge Graph UI, refer to Suppress insights.

Basic suppressed assertions configuration

Create a file named suppressed-assertions.tf and add the following:

# Basic suppressed alert configuration for maintenance
resource "grafana_asserts_suppressed_assertions_config" "maintenance_window" {
  provider = grafana.asserts

  name = "MaintenanceWindow"

  match_labels = {
    service     = "api-service"
    maintenance = "true"
  }
}

# Suppress specific alertname during deployment
resource "grafana_asserts_suppressed_assertions_config" "deployment_suppression" {
  provider = grafana.asserts

  name = "DeploymentSuppression"

  match_labels = {
    alertname = "HighLatency"
    job       = "web-service"
    env       = "staging"
  }
}

# Suppress alerts for specific test environment
resource "grafana_asserts_suppressed_assertions_config" "test_environment_suppression" {
  provider = grafana.asserts

  name = "TestEnvironmentSuppression"

  match_labels = {
    alertgroup  = "test.alerts"
    environment = "test"
  }
}

Service-specific suppression configurations

Suppress alerts for specific services during maintenance or operational activities:

# Suppress alerts for specific services during maintenance
resource "grafana_asserts_suppressed_assertions_config" "api_service_maintenance" {
  provider = grafana.asserts

  name = "APIServiceMaintenance"

  match_labels = {
    service     = "api-gateway"
    job         = "api-gateway"
    maintenance = "scheduled"
  }
}

# Suppress database alerts during backup operations
resource "grafana_asserts_suppressed_assertions_config" "database_backup" {
  provider = grafana.asserts

  name = "DatabaseBackupSuppression"

  match_labels = {
    service     = "postgresql"
    job         = "postgres-exporter"
    backup_mode = "active"
  }
}

# Suppress monitoring system alerts during updates
resource "grafana_asserts_suppressed_assertions_config" "monitoring_update" {
  provider = grafana.asserts

  name = "MonitoringSystemUpdate"

  match_labels = {
    service = "prometheus"
    job     = "prometheus"
    update  = "in_progress"
  }
}

Environment and team-based suppression

Create suppression rules based on environment or team:

# Suppress all alerts for development environment
resource "grafana_asserts_suppressed_assertions_config" "dev_environment" {
  provider = grafana.asserts

  name = "DevelopmentEnvironmentSuppression"

  match_labels = {
    environment = "development"
    team        = "platform"
  }
}

# Suppress alerts for specific team during their maintenance window
resource "grafana_asserts_suppressed_assertions_config" "team_maintenance" {
  provider = grafana.asserts

  name = "TeamMaintenanceWindow"

  match_labels = {
    team        = "backend"
    maintenance = "team_scheduled"
    timezone    = "UTC"
  }
}

# Suppress alerts for staging environment during testing
resource "grafana_asserts_suppressed_assertions_config" "staging_testing" {
  provider = grafana.asserts

  name = "StagingTestingSuppression"

  match_labels = {
    environment = "staging"
    testing     = "automated"
    job         = "integration-tests"
  }
}

Alert type and severity-based suppression

Suppress alerts based on their type or severity:

# Suppress low severity alerts during business hours
resource "grafana_asserts_suppressed_assertions_config" "low_severity_business_hours" {
  provider = grafana.asserts

  name = "LowSeverityBusinessHours"

  match_labels = {
    severity = "warning"
    timezone = "business_hours"
  }
}

# Suppress specific alert types during known issues
resource "grafana_asserts_suppressed_assertions_config" "known_issue_suppression" {
  provider = grafana.asserts

  name = "KnownIssueSuppression"

  match_labels = {
    alertname = "HighMemoryUsage"
    service   = "legacy-service"
    issue_id  = "LEG-123"
  }
}

# Suppress infrastructure alerts during planned maintenance
resource "grafana_asserts_suppressed_assertions_config" "infrastructure_maintenance" {
  provider = grafana.asserts

  name = "InfrastructureMaintenance"

  match_labels = {
    alertgroup = "infrastructure.alerts"
    maintenance_type = "planned"
    affected_services = "all"
  }
}

Complex multi-label suppression

Define complex suppression rules with multiple labels:

# Complex suppression for multi-service deployments
resource "grafana_asserts_suppressed_assertions_config" "multi_service_deployment" {
  provider = grafana.asserts

  name = "MultiServiceDeploymentSuppression"

  match_labels = {
    deployment_id = "deploy-2024-01-15"
    services      = "api,worker,frontend"
    environment   = "production"
    deployment_type = "blue_green"
  }
}

# Suppress alerts for specific cluster during maintenance
resource "grafana_asserts_suppressed_assertions_config" "cluster_maintenance" {
  provider = grafana.asserts

  name = "ClusterMaintenanceSuppression"

  match_labels = {
    cluster     = "production-cluster-1"
    maintenance = "cluster_upgrade"
    affected_nodes = "all"
    estimated_duration = "2h"
  }
}

# Suppress alerts for specific region during network issues
resource "grafana_asserts_suppressed_assertions_config" "regional_network_issue" {
  provider = grafana.asserts

  name = "RegionalNetworkIssueSuppression"

  match_labels = {
    region      = "us-west-2"
    issue_type  = "network"
    affected_services = "external_dependencies"
    incident_id = "NET-456"
  }
}

Resource reference

grafana_asserts_suppressed_assertions_config

Manage Knowledge Graph suppressed assertions configurations through the Grafana API.

Arguments

Name Type Required Description
name string Yes The name of the suppressed assertions configuration. This field is immutable and forces recreation if changed.
match_labels map(string) No Labels to match for this suppressed assertions configuration. Used to determine which alerts should be suppressed.

Example

resource "grafana_asserts_suppressed_assertions_config" "example" {
  provider = grafana.asserts

  name = "ExampleSuppression"

  match_labels = {
    alertname = "TestAlert"
    env       = "development"
  }
}

Best practices

Consider the following best practices when managing suppressed assertions with Terraform.

Suppression strategy

Label match rules

Lifecycle management

Validation

After applying the Terraform configuration, verify that:

Configure thresholds using Terraform

Threshold configurations in Knowledge Graph allow you to define custom thresholds for request, resource, and health assertions. These configurations help you set specific limits and conditions for monitoring your services and infrastructure.

For information about managing thresholds in the Knowledge Graph UI, refer to Manage thresholds.

Basic threshold configuration

Create a file named thresholds.tf and add the following:

# Basic threshold configuration with all three types
resource "grafana_asserts_thresholds" "basic" {
  provider = grafana.asserts

  request_thresholds = [{
    entity_name     = "payment-service"
    assertion_name  = "ErrorRatioBreach"
    request_type    = "inbound"
    request_context = "/charge"
    value           = 0.01
  }]

  resource_thresholds = [{
    assertion_name = "Saturation"
    resource_type  = "container"
    container_name = "worker"
    source         = "metrics"
    severity       = "warning"
    value          = 75
  }]

  health_thresholds = [{
    assertion_name = "ServiceDown"
    expression     = "up < 1"
    entity_type    = "Service"
  }]
}

Request threshold configurations

Configure thresholds for different service request types and contexts:

# Multiple request thresholds for different services
resource "grafana_asserts_thresholds" "request_thresholds" {
  provider = grafana.asserts

  request_thresholds = [
    {
      entity_name     = "api-service"
      assertion_name  = "ErrorRatioBreach"
      request_type    = "inbound"
      request_context = "/api/v1/users"
      value           = 0.02
    },
    {
      entity_name     = "api-service"
      assertion_name  = "LatencyP99ErrorBuildup"
      request_type    = "inbound"
      request_context = "/api/v1/orders"
      value           = 500
    },
    {
      entity_name     = "payment-gateway"
      assertion_name  = "RequestRateAnomaly"
      request_type    = "outbound"
      request_context = "/payment/process"
      value           = 1000
    }
  ]
}

Resource threshold configurations

Define resource thresholds for different severity levels:

# Resource thresholds for different severity levels
resource "grafana_asserts_thresholds" "resource_thresholds" {
  provider = grafana.asserts

  resource_thresholds = [
    {
      assertion_name = "Saturation"
      resource_type  = "container"
      container_name = "web-server"
      source         = "metrics"
      severity       = "warning"
      value          = 75
    },
    {
      assertion_name = "Saturation"
      resource_type  = "container"
      container_name = "web-server"
      source         = "metrics"
      severity       = "critical"
      value          = 90
    },
    {
      assertion_name = "ResourceRateBreach"
      resource_type  = "Pod"
      container_name = "database"
      source         = "logs"
      severity       = "warning"
      value          = 80
    }
  ]
}

Health threshold configurations

Configure health checks with Prometheus expressions:

# Health thresholds with Prometheus expressions
resource "grafana_asserts_thresholds" "health_thresholds" {
  provider = grafana.asserts

  health_thresholds = [
    {
      assertion_name = "ServiceDown"
      expression     = "up{job=\"api-service\"} < 1"
      entity_type    = "Service"
    },
    {
      assertion_name = "HighMemoryUsage"
      expression     = "memory_usage_percent > 85"
      entity_type    = "Service"
    },
    {
      assertion_name = "DatabaseConnectivity"
      expression     = "db_connection_pool_active / db_connection_pool_max > 0.9"
      entity_type    = "Service"
    }
  ]
}

Comprehensive threshold configuration

Define comprehensive thresholds for production environments:

# Production environment with comprehensive thresholds
resource "grafana_asserts_thresholds" "production" {
  provider = grafana.asserts

  request_thresholds = [
    {
      entity_name     = "frontend"
      assertion_name  = "ErrorRatioBreach"
      request_type    = "inbound"
      request_context = "/"
      value           = 0.005
    },
    {
      entity_name     = "backend-api"
      assertion_name  = "LatencyP99ErrorBuildup"
      request_type    = "inbound"
      request_context = "/api"
      value           = 200
    }
  ]

  resource_thresholds = [
    {
      assertion_name = "Saturation"
      resource_type  = "container"
      container_name = "frontend"
      source         = "metrics"
      severity       = "warning"
      value          = 70
    },
    {
      assertion_name = "Saturation"
      resource_type  = "container"
      container_name = "backend-api"
      source         = "metrics"
      severity       = "critical"
      value          = 85
    }
  ]

  health_thresholds = [
    {
      assertion_name = "ServiceDown"
      expression     = "up < 1"
      entity_type    = "Service"
    },
    {
      assertion_name = "NodeDown"
      expression     = "up{job=\"node-exporter\"} < 1"
      entity_type    = "Service"
    }
  ]
}

Resource reference

grafana_asserts_thresholds

Manage Knowledge Graph threshold configurations through the Grafana API. This resource allows you to define custom thresholds for request, resource, and health assertions.

Arguments

Name Type Required Description
request_thresholds list(object) No List of request threshold configurations. Refer to request thresholds block for details.
resource_thresholds list(object) No List of resource threshold configurations. Refer to resource thresholds block for details.
health_thresholds list(object) No List of health threshold configurations. Refer to health thresholds block for details.

Request thresholds block

Each request_thresholds block supports the following:

Name Type Required Description
entity_name string Yes The name of the entity to apply the threshold to.
assertion_name string Yes The name of the assertion to configure.
request_type string Yes The type of request (inbound, outbound).
request_context string Yes The request context or path to apply the threshold to.
value number Yes The threshold value.

Resource thresholds block

Each resource_thresholds block supports the following:

Name Type Required Description
assertion_name string Yes The name of the assertion to configure.
resource_type string Yes The type of resource (container, Pod, node).
container_name string Yes The name of the container to apply the threshold to.
source string Yes The source of the metrics (metrics, logs).
severity string Yes The severity level (warning, critical).
value number Yes The threshold value.

Health thresholds block

Each health_thresholds block supports the following:

Name Type Required Description
assertion_name string Yes The name of the assertion to configure.
expression string Yes The Prometheus expression for the health check.
entity_type string Yes Entity type for the health threshold (for example, Service, Pod, Namespace, Volume).
alert_category string No Optional alert category label for the health threshold.

Example

resource "grafana_asserts_thresholds" "example" {
  provider = grafana.asserts

  request_thresholds = [{
    entity_name     = "api-service"
    assertion_name  = "ErrorRatioBreach"
    request_type    = "inbound"
    request_context = "/api/v1/users"
    value           = 0.02
  }]

  resource_thresholds = [{
    assertion_name = "Saturation"
    resource_type  = "container"
    container_name = "web-server"
    source         = "metrics"
    severity       = "warning"
    value          = 75
  }]

  health_thresholds = [{
    assertion_name = "ServiceDown"
    expression     = "up{job=\"api-service\"} < 1"
    entity_type    = "Service"
  }]
}

Best practices

Consider the following best practices when configuring thresholds with Terraform.

Threshold configuration management

Request threshold best practices

Resource threshold best practices

Health threshold best practices

Value selection guidelines

Validation

After applying the Terraform configuration, verify that:

Configure trace correlation using Terraform

Trace configurations in Knowledge Graph allow you to define how trace data is queried and correlated with entities. You can specify data sources, entity matching rules, and label mappings for distributed tracing.

For information about configuring trace correlation in the Knowledge Graph UI, refer to Configure traces correlation.

Basic trace configuration

Create a file named trace-configs.tf and add the following:

# Basic trace configuration for services
resource "grafana_asserts_trace_config" "production" {
  provider = grafana.asserts

  name            = "production"
  priority        = 1000
  default_config  = false
  data_source_uid = "grafanacloud-traces"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service"]
  }

  match {
    property = "deployment_environment"
    op       = "="
    values   = ["production", "staging"]
  }

  entity_property_to_trace_label_mapping = {
    "cluster"        = "resource.k8s.cluster.name"
    "namespace"      = "resource.k8s.namespace"
    "container"      = "resource.container.name"
    "otel_service"   = "resource.service.name"
    "otel_namespace" = "resource.service.namespace"
  }
}

Trace configuration with multiple match rules

Configure trace correlation with multiple entity matching criteria:

# Development environment trace configuration
resource "grafana_asserts_trace_config" "development" {
  provider = grafana.asserts

  name            = "development"
  priority        = 2000
  default_config  = false
  data_source_uid = "tempo-dev"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service"]
  }

  match {
    property = "deployment_environment"
    op       = "="
    values   = ["development", "testing"]
  }

  match {
    property = "asserts_site"
    op       = "="
    values   = ["us-east-1"]
  }

  match {
    property = "service"
    op       = "="
    values   = ["api"]
  }

  entity_property_to_trace_label_mapping = {
    "cluster"        = "resource.k8s.cluster.name"
    "namespace"      = "resource.k8s.namespace"
    "container"      = "resource.container.name"
    "otel_service"   = "resource.service.name"
    "otel_namespace" = "resource.service.namespace"
    "pod"            = "span.k8s.pod.name"
  }
}

Minimal trace configuration

Create a minimal configuration for all entities:

# Minimal configuration for all entities
resource "grafana_asserts_trace_config" "minimal" {
  provider = grafana.asserts

  name            = "minimal"
  priority        = 3000
  default_config  = false
  data_source_uid = "tempo-minimal"

  match {
    property = "asserts_entity_type"
    op       = "IS NOT NULL"
    values   = []
  }

  entity_property_to_trace_label_mapping = {
    "cluster"        = "resource.k8s.cluster.name"
    "otel_service"   = "resource.service.name"
    "otel_namespace" = "resource.service.namespace"
  }
}

Advanced trace configuration with complex match rules

Configure traces with multiple operations and advanced match rules:

# Advanced configuration with multiple operations
resource "grafana_asserts_trace_config" "advanced" {
  provider = grafana.asserts

  name            = "advanced"
  priority        = 1500
  default_config  = false
  data_source_uid = "tempo-advanced"

  match {
    property = "service_type"
    op       = "CONTAINS"
    values   = ["web", "api"]
  }

  match {
    property = "deployment_environment"
    op       = "<>"
    values   = ["test"]
  }

  match {
    property = "team"
    op       = "IS NOT NULL"
    values   = []
  }

  match {
    property = "priority_level"
    op       = ">="
    values   = ["5"]
  }

  entity_property_to_trace_label_mapping = {
    "service_type"   = "resource.service.type"
    "team"           = "resource.team.owner"
    "environment"    = "resource.deployment.environment"
    "version"        = "resource.service.version"
    "region"         = "resource.cloud.region"
  }
}

Resource reference

grafana_asserts_trace_config

Manage Knowledge Graph trace configurations through the Grafana API.

Arguments

Name Type Required Description
name string Yes The name of the trace configuration. This field is immutable and forces recreation if changed.
priority number Yes Priority of the trace configuration. A lower number means a higher priority.
default_config bool Yes Whether this is the default configuration. Default configurations cannot be deleted.
data_source_uid string Yes DataSource UID to be queried (for example, a Tempo instance).
match list(object) No List of match rules for entity properties. Refer to match block for details.
entity_property_to_trace_label_mapping map(string) No Mapping of entity properties to trace labels for correlation.

Match block

Each match block supports the following:

Name Type Required Description
property string Yes Entity property to match against.
op string Yes Operation to use for matching. One of: =, <>, <, >, <=, >=, IS NULL, IS NOT NULL, STARTS WITH, CONTAINS.
values list(string) Yes Values to match against. Can be empty for IS NULL and IS NOT NULL operations.

Example

resource "grafana_asserts_trace_config" "example" {
  provider = grafana.asserts

  name            = "example-traces"
  priority        = 1000
  default_config  = false
  data_source_uid = "tempo-prod"

  match {
    property = "asserts_entity_type"
    op       = "="
    values   = ["Service", "Pod"]
  }

  match {
    property = "deployment_environment"
    op       = "STARTS WITH"
    values   = ["prod"]
  }

  entity_property_to_trace_label_mapping = {
    "service"     = "resource.service.name"
    "namespace"   = "resource.k8s.namespace"
    "environment" = "resource.deployment.environment"
    "cluster"     = "resource.k8s.cluster.name"
  }
}

Best practices

Consider the following best practices when configuring trace correlation with Terraform.

Priority management

Data source configuration

Label map strategy

Match rules design

Distributed trace integration

Validation

After applying the Terraform configuration, verify that: