Fundamentals Grafana dashboards overview Have you ever wondered what a dashboard is? In the observability world, this term is frequently used, but what exactly does it mean? The concept is borrowed from automobiles, where a dashboard gives drivers access to the controls necessary to operate a vehicle. Similarly, digital dashboards help us comprehend and manage systems. This topic explains how Grafana dashboards function, enabling you to create your own with greater ease. The following image illustrates a sample Grafana dashboard: {{< figure src="/media/docs/grafana/dashboards-overview/complex-dashboard-example.png" max-width="750px" caption="Example Grafana dashboard" >}} A Grafana dashboard consists of panels displaying data in beautiful graphs, charts, and other visualizations. These panels are created using components that transform raw data from a data source into visualizations. The process involves passing data through three gates: a plugin, a query, and an optional transformation. The image below displays all the gates, followed by detailed explanations of their purpose, usage, and significance. {{< figure src="/media/docs/grafana/dashboards-overview/dashboard-component-architecture.png" max-width="750px" caption="Dashboard component architecture" >}} Data sources A data source refers to any entity that consists of data. It can be an SQL database, Grafana Loki, Grafana Mimir, or a JSON-based API. It can even be a basic CSV file. The first step in creating a dashboard visualization is selecting the data source that contains the data you need. It can be difficult to understand the distinctions between different data sources as each possesses its own structure and requires different query methods. However, in dashboards, you can see different data sources visualized in one single view, making it easier to understand your data overall. Plugins A Grafana plugin is software that adds new capabilities to Grafana. They come in many types, but right now we'll address data source plugins. The job of a Grafana data source plugin is to take a query you want answered, retrieve the data from the data source, and reconcile the differences between the data model of the data source and the data model of Grafana dashboards. It does this using a unified data structure called a data frame. The data coming into the plugin from the data source might be in many different formats (such as JSON, rows and columns, or CSV), but when it leaves the plugin and moves through the rest of the gates toward a visualization, it's always in data frames. Currently, Grafana offers a diverse range of 155 data sources that you can use. The most commonly used options are already pre-installed and accessible. Before exploring other options, look for an existing data source that matches your requirements. Grafana constantly updates the list, but if you don't find a suitable data source, you can browse through the plugin catalog or create a plugin. Queries Queries allow you to reduce the entirety of your data to a specific dataset, providing a more manageable visualization. They help answer questions you have about system and operational processes. For instance, a company with an online store might want to determine the number of customers who add products to their shopping carts. This can be achieved through a query that aggregates access metrics for the shopping cart service, revealing the number of users accessing the service per second. When working with data sources, it's crucial to recognize that each one has its own distinct query language. For example, Prometheus data sources utilize PromQL, while LogQL is used for logs, and particular databases employ SQL. A query is the foundation of every visualization in Grafana, and a dashboard might use a range of query languages. The following image shows the Query Editor associated with the Prometheus data source. The node_cpu_seconds_total query is written in PromQL and requests just one metric: {{< figure src="/media/docs/grafana/dashboards-overview/example-query.png" max-width="750px" caption="Example query" >}} Transformations When the data format in a visualization doesn’t meet your requirements, you can apply a transformation that manipulates the data returned by a query. You might not need to transform data when you're first starting out, but they're powerful and worth mentioning. Transforming data is useful in the following kinds of situations: You want to combine two fields together, for example, concatenating Given Name and Family Name into a Full Name field. You have CSV data (all text), and you want to convert a field type (such as parsing a date or a number out of a string). You want to filter, join, merge, or perform other SQL-like operations that might not be supported by the underlying data source or query language. Transformations are located in the Transform tab in the edit dialog for a panel. Select the transformation you want, and define the transformation. The following image shows that you can have as many transformations as you want, just like queries. For example, you can chain together a series of transformations that make a change to a data type, filter results, organize columns, and sort the result into one data pipeline. Every time the dashboard is refreshed, the transformation applies to the latest data from the data source. The following image shows the transformation dialog: {{< figure src="/media/docs/grafana/dashboards-overview/example-transform-chain.png" max-width="750px" caption="Example chain of transformations" >}} Panels After the data is sourced, queried, and transformed, it passes to a panel, which is the final gate in the journey to a Grafana visualization. A panel is a container that displays the visualization and provides you with various controls to manipulate it. The panel configuration is where you specify how you want to see the data. For example, you use a drop-down menu in the top-right of the panel to specify the type of visualization you want to see, such as a bar chart, pie chart, or histogram. The panel options let you customize many aspects of the visualization and the options differ based on which visualization you select. Panels also contain queries that specify the data the panel is visualizing. The following image shows a table panel being edited, the panel settings showing the query at the bottom, and the panel options on the right. In this image, you can see how the data source, plugin, query, and panel all come together. {{< figure src="/media/docs/grafana/dashboards-overview/example-table-panel.png" max-width="750px" caption="Example table panel" >}} Selecting the best visualization depends on the data and how you want it presented. To see examples of dashboards in one place that you can browse and inspect, refer to Grafana Play, which has feature showcases and a variety of examples. Conclusion With the data source, plugin, query, transformation, and panel model in mind, you can now see right through any Grafana dashboard you encounter, and imagine how to build your own. Building a Grafana dashboard is a process that starts with determining your dashboard requirements and identifying which data sources support those requirements. If you want to integrate a specialized database with a Grafana dashboard, you must ensure the correct plugin is installed so that you can add a data source to use with that plugin. And with the data source identified and the plugin installed, you can write your query, transform the data, and format the visualization to meet your needs. This component architecture is part of what makes Grafana so powerful and general. Given the data source plugin and data frame abstraction, any data source you can access can work with the same general approach. Introduction to exemplars An exemplar is a specific trace representative of measurement taken in a given time interval. While metrics excel at giving you an aggregated view of your system, traces give you a fine grained view of a single request; exemplars are a way to link the two. Suppose your company website is experiencing a surge in traffic volumes. While more than eighty percent of the users are able to access the website in under two seconds, some users are experiencing a higher than normal response time resulting in bad user experience. To identify the factors that are contributing to the latency, you must compare a trace for a fast response against a trace for a slow response. Given the vast amount of data in a typical production environment, it's an extremely laborious and time-consuming effort. Use exemplars to help isolate problems within your data distribution by pinpointing query traces exhibiting high latency within a time interval. After you localize the latency problem to a few exemplar traces, you can combine it with additional system based information or location properties to perform a root cause analysis faster, leading to quick resolutions to performance issues. Support for exemplars is available for the Prometheus data source only. After you enable the functionality, exemplar data is available by default. For more information on exemplar configuration and how to enable exemplars, refer to the Exemplars section in Prometheus configuration options. Grafana shows exemplars alongside a metric in the Explore view and in dashboards. Each exemplar displays as a highlighted star. You can hover your cursor over an exemplar to view the unique trace ID, which is a combination of a key value pair. To investigate further, click the blue button next to the traceID property. {{< figure src="/media/docs/grafana/exemplars/screenshot-exemplars.png" class="docs-image--no-shadow" max-width= "750px" caption="Screenshot showing the detail window of an exemplar" >}} Refer to View exemplar data for instructions on how to drill down and view exemplar trace details from metrics and logs. To know more about exemplars, refer to the blog post Intro to exemplars, which enable Grafana Tempo’s distributed tracing at massive scale. View exemplar data When support for exemplar support is enabled for a Prometheus data source, you can view exemplar data either in the Explore view or from the Loki log details. In Explore Explore visualizes exemplar traces as highlighted stars alongside metrics data. For more information on how Explore visualizes trace data, refer to Tracing in Explore. To examine the details of an exemplar trace: Place your cursor over an exemplar (highlighted star). Depending on the trace data source you are using, you may see a blue button with the label Query with . In the following example, the tracing data source is Tempo. {{< figure src="/media/docs/grafana/exemplars/screenshot-exemplar-details.png" class="docs-image--no-shadow" max-width= "350px" caption="Screenshot showing exemplar details" >}} Click the Query with Tempo option next to the traceID property. The trace details, including the spans within the trace are listed in a separate panel on the right. {{< figure src="/media/docs/grafana/exemplars/screenshot-exemplar-explore-view.png" class="docs-image--no-shadow" max-width= "900px" caption="Explorer view with panel showing trace details" >}} For more information on how to drill down and analyze the trace and span details, refer to the Analyze trace and span details section. In logs You can also view exemplar trace details from the Loki logs in Explore. Use regular expressions within the Derived fields links for Loki to extract the traceID information. Now when you expand Loki logs, you can see a traceID property under the Detected fields section. To learn more about how to extract a part of a log message into an internal or external link, refer to using derived fields in Loki. To view the details of an exemplar trace: Expand a log line and scroll down to the Fields section. Depending on your backend trace data source, you may see a blue button with the label . Click the blue button next to the traceID property. Typically, it has the name of the backend data source. In the following example, the tracing data source is Tempo. The trace details, including the spans within the trace are listed in a separate panel on the right. {{< figure src="/media/docs/grafana/exemplars/screenshot-exemplar-loki-logs.png" class="docs-image--no-shadow" max-width= "750px" caption="Explorer view with panel showing trace details" >}} For more information on how to drill down and analyze the trace and span details, refer to the Analyze trace and span details section. Analyze trace and spans This panel shows the details of the trace in different segments. The top segment displays the trace ID to indicate that the query results correspond to the specific trace. You can add more traces to the results using the Add query button. The next segment shows the entire span for the specific trace as a narrow strip. All levels of the trace from the client all the way down to database query is displayed, which provides a bird's eye view of the time distribution across all layers over which the HTTP request was processed. You can click within this strip view to display a magnified view of a smaller time segment within the span. This magnified view shows up in the bottom segment of the panel. In the magnified view, you can expand or collapse the various levels of the trace to drill down to the specific span of interest. For example, if the strip view shows that most of the latency was within the app layer, you can expand the trace down the app layer to investigate the problem further. To expand a particular layer of span, click the left icon. The same button can collapse an expanded span. To see the details of the span at any level, click the span itself. This displays additional metadata associated with the span. The metadata itself is initially shown in a narrow strip but you can see more details by clicking the metadata strip. {{< figure src="/media/docs/grafana/exemplars/screenshot-exemplar-span-details.png" class="docs-image--no-shadow" max-width= "600px" caption="Span details" >}} Get started with Grafana and InfluxDB {{< docs/shared lookup="influxdb/intro.md" source="grafana" version="" >}} Get InfluxDB You can download InfluxDB and install it locally or you can sign up for InfluxDB Cloud. Windows installers are not available for some versions of InfluxDB. Install other InfluxDB software Install Telegraf. This tool is an agent that helps you get metrics into InfluxDB. For more information, refer to Telegraf documentation. If you chose to use InfluxDB Cloud, then you should download and install the InfluxDB Cloud CLI. This tool allows you to send command line instructions to your cloud account. For more information, refer to Influx CLI documentation. Get data into InfluxDB If you downloaded and installed InfluxDB on your local machine, then use the Quick Start feature to visualize InfluxDB metrics. If you are using the cloud account, then the wizards will guide you through the initial process. For more information, refer to Configure Telegraf. Note for Windows users: Windows users might need to make additional adjustments. Look for special instructions in the InfluxData documentation and Using Telegraf on Windows blog post. The regular system monitoring template in InfluxDB Cloud is not compatible with Windows. Windows users who use InfluxDB Cloud to monitor their system will need to use the Windows System Monitoring Template. Add your InfluxDB data source to Grafana You can have more than one InfluxDB data source defined in Grafana. Follow the general instructions to add a data source. Decide if you will use InfluxQL or Flux as your query language. Configure the data source for your chosen query language. Each query language has its own unique data source settings. For querying features specific to each language, see the data source's query editor documentation. InfluxDB guides InfluxDB publishes guidance for connecting different versions of their product to Grafana. InfluxDB OSS or Enterprise 1.8+. To turn on Flux, refer to Configure InfluxDB. Select your InfluxDB version in the upper right corner. InfluxDB OSS or Enterprise 2.x. Refer to Use Grafana with InfluxDB. Select your InfluxDB version in the upper right corner. InfluxDB Cloud. Refer to Use Grafana with InfluxDB Cloud. Important tips Make sure your Grafana token has read access. If it doesn't, then you'll get an authentication error and be unable to connect Grafana to InfluxDB. Avoid apostrophes and other non-standard characters in bucket and token names. If the text name of the organization or bucket doesn't work, then try the ID number. If you change your bucket name in InfluxDB, then you must also change it in Grafana and your Telegraf .conf file as well. Add a query This step varies depending on the query language that you selected when you set up your data source in Grafana. InfluxQL query language In the query editor, click select measurement. Grafana displays a list of possible series. Click one to select it, and Grafana graphs any available data. If there is no data to display, then try another selection or check your data source. Flux query language Create a simple Flux query. Add a panel. In the query editor, select your InfluxDB-Flux data source. For more information, refer to Queries. Select the Table visualization. In the query editor text field, enter buckets() and then click outside of the query editor. This generic query returns a list of buckets. You can also create Flux queries in the InfluxDB Explore view. In your browser, log in to the InfluxDB native UI (OSS is typically something like http://localhost:8086 or for InfluxDB Cloud use: https://cloud2.influxdata.com). Click Explore to open the Data Explorer. The InfluxDB Data Explorer provides two mechanisms for creating Flux queries: a graphical query editor and a script editor. Using the graphical query editor, create a query. It will look something like this: Click Script Editor to view the text of the query, and then copy all the lines of your Flux code, which will look something like this: In Grafana, add a panel and then paste your Flux code into the query editor. Click Apply. Your new panel should be visible with data from your Flux query. Check InfluxDB metrics in Grafana Explore In your Grafana instance, go to the Explore view and build queries to experiment with the metrics you want to monitor. Here you can also debug issues related to collecting metrics. Start building dashboards There you go! Use Explore and Data Explorer to experiment with your data, and add the queries that you like to your dashboard as panels. Have fun! Here are some resources to learn more: Grafana documentation: InfluxDB data source InfluxDB documentation: Comparison of Flux vs InfluxQL Get started with Grafana and MS SQL Server Microsoft SQL Server is a popular relational database management system that is widely used in development and production environments. This topic walks you through the steps to create a series of dashboards in Grafana to display metrics from a MS SQL Server database. Download MS SQL Server MS SQL Server can be installed on Windows or Linux operating systems and also on Docker containers. Refer to the MS SQL Server downloads page, for a complete list of all available options. Install MS SQL Server You can install MS SQL Server on the host running Grafana or on a remote server. To install the software from the downloads page, follow their setup prompts. If you are on a Windows host but want to use Grafana and MS SQL data source on a Linux environment, refer to the WSL to set up your Grafana development environment blog post. This will allow you to leverage the resources available in grafana/grafana GitHub repository. Here you will find a collection of supported data sources, including MS SQL Server, along with test data and pre-configured dashboards for use. Add the MS SQL data source There are several ways to authenticate in MSSQL. Start by: Click Connections in the left-side menu and filter by mssql. Select the Microsoft SQL Server option. Click Create a Microsoft SQL Server data source in the top right corner to open the configuration page. Select the desired authentication method and fill in the right information as detailed below. Click Save & test. General configuration Name Description Name The data source name. This is how you refer to the data source in panels and queries. Host The IP address/hostname and optional port of your MS SQL instance. If port is omitted, the default 1433 will be used. Database Name of your MS SQL database. SQL Server Authentication Name Description User Database user's login/username. Password Database user's password. Windows Active Directory (Kerberos) Below are the four possible ways to authenticate via Windows Active Directory/Kerberos. {{< admonition type="note" >}} Windows Active Directory (Kerberos) authentication is not supported in Grafana Cloud at the moment. {{< /admonition >}} Method Description Username + password Enter the domain user and password Keytab file Specify the path to a valid keytab file to use that for authentication. Credential cache Log in on the host via kinit and pass the path to the credential cache. The cache path can be found by running klist on the host in question. Credential cache file This option allows multiple valid configurations to be present and matching is performed on host, database, and user. See the example JSON below this table. [ { "user": "grot@GF.LAB", "database": "dbone", "address": "mysql1.mydomain.com:3306", "credentialCache": "/tmp/krb5cc_1000" }, { "user": "grot@GF.LAB", "database": "dbtwo", "address": "mysql2.gf.lab", "credentialCache": "/tmp/krb5cc_1000" } ] For installations from the grafana/grafana repository, gdev-mssql data source is available. Once you add this data source, you can use the Datasource tests - MSSQL dashboard with three panels showing metrics generated from a test database. Optionally, play around this dashboard and customize it to: Create different panels. Change titles for panels. Change frequency of data polling. Change the period for which the data is displayed. Rearrange and resize panels. Start building dashboards Now that you have gained some idea of using the pre-packaged MS SQL data source and some test data, the next step is to setup your own instance of MS SQL Server database and data your development or sandbox area. To fetch data from your own instance of MS SQL Server, add the data source using instructions in Step 4 of this topic. In Grafana Explore build queries to experiment with the metrics you want to monitor. Once you have a curated list of queries, create dashboards to render metrics from the SQL Server database. For troubleshooting, user permissions, known issues, and query examples, refer to Using Microsoft SQL Server in Grafana. Get started with Grafana and Prometheus Prometheus is an open source monitoring system for which Grafana provides out-of-the-box support. This topic walks you through the steps to create a series of dashboards in Grafana to display system metrics for a server monitored by Prometheus. {{< admonition type="tip" >}} Check out our Prometheus Learning Journeys. Connect to a Prometheus data source in Grafana Cloud to visualize your metrics directly from where they are stored. Send metrics to Grafana Cloud using Prometheus remote write to explore Grafana Cloud without making significant changes to your existing configuration. {{< /admonition >}} Grafana and Prometheus: Download Prometheus and Node exporter Install Prometheus Node exporter Install and configure Prometheus Configure Prometheus for Grafana Check Prometheus metrics in Grafana Explore view Start building dashboards Download Prometheus and Node exporter Download the following components: Prometheus Node exporter Like Grafana, you can install Prometheus on many different operating systems. Refer to the Prometheus download page to see a list of stable versions of Prometheus components. Install Prometheus Node exporter Install Node exporter on all hosts you want to monitor. This guide shows you how to install it locally. Prometheus Node exporter is a widely used tool that exposes system metrics. For instructions on installing Node exporter, refer to the Installing and running the Node exporter section in the Prometheus documentation. When you run Node exporter locally, navigate to http://localhost:9100/metrics to check that it is exporting metrics. {{< admonition type="note" >}} The instructions in the referenced topic are intended for Linux users. You may have to alter the instructions slightly depending on your operating system. For example, if you are on Windows, use the windows_exporter instead. {{< /admonition >}} Install and configure Prometheus After downloading Prometheus, extract it and navigate to the directory. tar xvfz prometheus-*.tar.gz cd prometheus-* Locate the prometheus.yml file in the directory. Modify the Prometheus configuration file to monitor the hosts where you installed Node exporter. By default, Prometheus looks for the file prometheus.yml in the current working directory. This behavior can be changed via the --config.file command line flag. For example, some Prometheus installers use it to set the configuration file to /etc/prometheus/prometheus.yml. The following example shows you the code you should add. Notice that static configs targets are set to ['localhost:9100'] to target node-explorer when running it locally. # A scrape configuration containing exactly one endpoint to scrape from Node exporter running on a host: scrape_configs: # The job name is added as a label `job=` to any timeseries scraped from this config. - job_name: 'node' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['localhost:9100'] Start the Prometheus service: ./prometheus --config.file=./prometheus.yml Confirm that Prometheus is running by navigating to http://localhost:9090. You can see that the Node exporter metrics have been delivered to Prometheus. Next, the metrics will be sent to Grafana. Configure Prometheus for Grafana When running Prometheus locally, there are two ways to configure Prometheus for Grafana. You can use a hosted Grafana instance at Grafana Cloud or run Grafana locally. This guide describes configuring Prometheus in a hosted Grafana instance on Grafana Cloud. Sign up for https://grafana.com/. Grafana gives you a Prometheus instance out of the box. Because you are running your own Prometheus instance locally, you must remote_write your metrics to the Grafana.com Prometheus instance. Grafana provides code to add to your prometheus.yml config file. This includes a remote write endpoint, your user name and password. Add the following code to your prometheus.yml file to begin sending metrics to your hosted Grafana instance. remote_write: - url: basic_auth: username: password: {{< admonition type="note" >}} To configure your Prometheus instance to work with Grafana locally instead of Grafana Cloud, install Grafana here and follow the configuration steps listed here. {{< /admonition >}} Troubleshooting These are some of the troubleshooting steps you can try if Prometheus isn’t running or functioning as expected. The steps provided have been selected based on the Learning Journeys we offer for Prometheus. If you’d like to explore further, check out the Prometheus Learning Journey if you want to visualize data in Grafana Cloud without sending or storing data in Grafana Cloud, such as for local retention needs. Alternatively, if you already have a Prometheus setup and want to explore Grafana Cloud without making significant changes, visit the Prometheus remote write learning journey. 1. Checking if Prometheus is running If the Prometheus web UI is inaccessible (e.g., "Connection refused" error in the browser) or Prometheus queries fail (e.g., errors in Grafana like "Data source unavailable" or "No data points"), a good place to start is confirming that the Prometheus process and service are running. You can do this by checking the system process or verifying the service status: Linux sudo systemctl status prometheus Shows whether the process is running and if the service is active. MacOS pgrep prometheus Returns the process ID (PID) if Prometheus is running. Windows ( PowerShell) Get-Process -Name prometheus -ErrorAction SilentlyContinue Checks if the Prometheus process is running by name. 2. If Prometheus is not running Start by checking for common causes: Check for port conflicts. Prometheus runs on port 9090 by default. If another process is using this port, Prometheus may fail to start. You can check for port conflicts with: Linux & MacOS lsof -i :9090 Windows ( PowerShell) netstat -ano | findstr :9090 Shows if another process is using port 9090. Verify the Prometheus binary placement: ensure Prometheus binaries ( prometheus and promtool) are correctly installed. Linux & MacOS ls /usr/local/bin/prometheus /usr/local/bin/promtool If missing, move them to /usr/local/bin or a directory in your system’s PATH. Check if Prometheus is in the path. which prometheus which promtool If there’s no output, the binaries are not in the system PATH. Ensure configuration & data files are in place. Linux & MacOs ls /etc/prometheus /var/lib/prometheus ls /etc/prometheus/prometheus.yml Makes sure that Prometheus has its necessary configuration and data storage directories. Check permissions: If Prometheus is running as a dedicated user, ensure the correct ownership: sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus (Optional) Secure Prometheus by creating a dedicated user sudo useradd --no-create-home --shell /bin/false prometheus Recommended for security: runs Prometheus as a non-login user. 3. Checking if Prometheus is running as a service If Prometheus is running as a process, check whether it is properly set up and managed as a service to ensure it restarts automatically after reboots or failures. Check Prometheus service status Linux systemctl status prometheus.service Windows sc query prometheus MacOs pgrep prometheus If the service is inactive (dead) or stopped, proceed to the next steps. 4. If Prometheus is not running as a service If Prometheus is not running as a managed service, ensure it is correctly configured and can restart automatically. Verify service configuration (Linux & MacOs) Check the service unit file to ensure correct paths: sudo nano /etc/systemd/system/prometheus.service Look for the ExecStart line: ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/ Ensure: The binary path ( /usr/local/bin/prometheus) is correct. The configuration file ( /etc/prometheus/prometheus.yml) is in place. The storage path ( /var/lib/prometheus/) exists. Restart and enable Prometheus service (Linux & MacOs) sudo systemctl daemon-reload sudo systemctl enable prometheus sudo systemctl start prometheus sudo systemctl status prometheus Check Prometheus health status After restarting, verify if Prometheus is responsive: curl -s http://localhost:9090/-/ready If successful, this confirms Prometheus is ready to serve requests. Restart Prometheus service (Windows) If running as a Windows service, restart it: net stop prometheus net start prometheus 5. Checking if Prometheus is capturing metrics If you installed Node exporter to expose your system metrics, you can check if Prometheus is capturing metrics by sending a request to the /metrics endpoint. curl http://localhost:9090/metrics It should return a number of metrics and metadata about the metrics being exposed. Check Prometheus metrics in Grafana Metrics Drilldown In your Grafana instance, go to the Drilldown view and experience query-less browsing of Prometheus-compatible metrics. Start building dashboards Now that you have a curated list of queries, create dashboards to render system metrics monitored by Prometheus. When you install Prometheus and Node exporter or windows_exporter, you will find recommended dashboards for use. The following image shows a dashboard with three panels showing some system metrics. To learn more: Grafana documentation: Prometheus data source Prometheus documentation: What is Prometheus? Glossary This topic lists words and abbreviations that are commonly used in the Grafana documentation and community. app plugin An extension of Grafana that allows users to provide additional functionality to enhance their experience by including a set of panel and data source plugins, as well as custom pages. See also data source plugin, panel plugin, and plugin. dashboard A set of one or more panels, organized and arranged into one or more rows, that provide an at-a-glance view of related information. data source A file, database, or service providing the data. Grafana supports several data sources by default, and can be extended to support additional data sources through plugins. data source plugin Extends Grafana with support for additional data sources. See also data source, app plugin, panel plugin, and plugin. exemplar An exemplar is any data that serves as a detailed example of one of the observations aggregated into a metric. An exemplar contains the observed value together with an optional timestamp and arbitrary labels, which are typically used to reference a trace. Explore Explore allows a user to focus on building a query. Users can refine the query to return the expected metrics before building a dashboard. For more information, refer to the Explore topic. export or import dashboard Grafana includes the ability to export your dashboards to a file containing JSON. Community members sometimes share their created dashboards on the Grafana Dashboards page. Dashboards previously exported or found on this site may be imported by other users. exporter An exporter translates data that comes out of a data source into a format that Prometheus can digest. gcx A command-line tool that enables users to authenticate, manage multiple environments, and perform administrative tasks through Grafana's REST API. It's compatible with AI agents. Integration (Grafana Cloud) Each Integration in Grafana Cloud uses the cloud agent to connect your data source to Grafana Cloud for visualizing. Note: Prometheus uses the word “integrations” to refer to software that exposes Prometheus metrics without needing an exporter, which is a different use of the same word we use here. graph A commonly-used visualization that displays data as points, lines, or bars. grafanactl A command-line tool that enables users to authenticate, manage multiple environments, and perform administrative tasks through Grafana's REST API. mixin A mixin is a set of Grafana dashboards and Prometheus rules and alerts, written in Jsonnet and packaged together in a bundle. panel Basic building block in Grafana, composed by a query and a visualization. Can be moved and resized within a dashboard. panel plugin Extends Grafana with additional visualization options. See also panel, plugin, app plugin, and data source plugin. plugin An extension of Grafana that allows users to provide additional functionality to enhance their experience. See also app plugin, data source plugin, and panel plugin. query Used to request data from a data source. The structure and format of the query depend on the specific data source. time series A series of measurements, ordered by time. Time series are stored in data sources and returned as the result of a query. trace An observed execution path of a request through a distributed system. For more information, refer to What is Distributed Tracing? transformation Transformations process the result set of a query before it’s passed on for visualization. For more information, refer to the Transformations overview topic. visualization A graphical representation of query results. Introduction to histograms and heatmaps A histogram is a graphical representation of the distribution of numerical data. It groups values into buckets (sometimes also called bins) and then counts how many values fall into each bucket. Instead of graphing the actual values, histograms graph the buckets. Each bar represents a bucket, and the bar height represents the frequency (such as count) of values that fell into that bucket's interval. Histogram example This histogram shows the value distribution of a couple of time series. You can easily see that most values land between 240-300 with a peak between 260-280. Here is an example showing height distribution of people. {{< figure src="/static/img/docs/histogram-panel/histogram-example-v8-0.png" max-width="625px" caption="Bar chart example" >}} For more information about histogram visualization options, refer to Histogram. Histograms only look at value distributions over a specific time range. The problem with histograms is that you cannot see any trends or changes in the distribution over time. This is where heatmaps become useful. Heatmaps A heatmap is like a histogram, but over time, where each time slice represents its own histogram. Instead of using bar height as a representation of frequency, it uses cells, and colors the cell proportional to the number of values in the bucket. In this example, you can clearly see what values are more common and how they trend over time. For more information about heatmap visualization options, refer to Heatmap. Pre-bucketed data There are a number of data sources supporting histogram over time, like Elasticsearch (by using a Histogram bucket aggregation) or Prometheus (with histogram metric type and Format as option set to Heatmap). But generally, any data source could be used as long as it meets the requirement that it either returns series with names representing bucket bounds, or that it returns series sorted by the bounds in ascending order. Raw data vs aggregated If you use the heatmap with regular time series data (not pre-bucketed), then it's important to keep in mind that your data is often already aggregated by your time series backend. Most time series queries do not return raw sample data, but instead include a group by time interval or maxDataPoints limit coupled with an aggregation function (usually average). This all depends on the time range of your query of course. But the important point is to know that the histogram bucketing that Grafana performs might be done on already aggregated and averaged data. To get more accurate heatmaps, it is better to do the bucketing during metric collection, or to store the data in Elasticsearch or any other data source which supports doing histogram bucketing on the raw data. If you remove or lower the group by time (or raise maxDataPoints) in your query to return more data points, your heatmap will be more accurate, but this can also be very CPU and memory taxing for your browser, possibly causing hangs or crashes if the number of data points becomes unreasonably large. What is Prometheus? Observability focuses on understanding the internal state of your systems based on the data they produce, which helps determine if your infrastructure is healthy. Prometheus is a core technology for monitoring and observability of systems, but the term "Prometheus" can be confusing because it is used in different contexts. Understanding Prometheus basics, why it’s valuable for system observability, and how users use it in practice will both help you better understand it and help you use Grafana. Prometheus began in 2012 at SoundCloud because existing technologies were insufficient for their observability needs. Prometheus offers both a robust data model and a query language. Prometheus is also simple and scalable. In 2018, Prometheus graduated from Cloud Native Computing Foundation (CNCF) incubation, and today has a thriving community. Prometheus as data The following panel in a Grafana dashboard shows how much disk bandwidth on a Mac laptop is being used. The green line represents disk reads, and the yellow line represents writes. {{< figure src="/media/docs/grafana/intro-prometheus/disk-io.png" max-width="750px" caption="Disk I/O dashboard" >}} Data like these form time series. The X-axis is a moment in time and the Y-axis is a number or measurement; for example, 5 megabytes per second. This type of time series data appears everywhere in systems monitoring, as well as in places such as seasonal temperature charts and stock prices. This data is simply some measurement (such as a company stock price or Disk I/O) through a series of time instants. Prometheus is a technology that collects and stores time series data. Time series are fundamental to Prometheus; its data model is arranged into: metrics that consist of a timestamp and a sample, which is the numeric value, such as how many disk bytes have been read or a stock price a set of labels called dimensions, for example, job and device You can store time series data in any relational database, however, these systems are not developed to store and query large volumes of time series data. Prometheus and similar software provide tools to compact and optimize time series data. Simple dashboard using PromQL The following Grafana dashboard image shows a Disk I/O graph of raw data from Prometheus derived from a laptop. The Metrics browser field contains the following query: node_disk_written_bytes_total{job="integrations/macos-node", device!=""} In this example, the Y-axis shows the total number of bytes written, and the X-axis shows dates and times. As the laptop runs, the number of bytes written increases over time. Below Metrics browser is a counter that counts the number of bytes written over time. {{< figure src="/media/docs/grafana/intro-prometheus/dashboard-example.png" max-width="750px" caption="Metrics browser and counter" >}} The query is a simple example of PromQL, the Prometheus Query Language. The query identifies the metric of interest ( node_disk_written_bytes_total) and provides two labels ( job and device). The label selector job="integrations/macos-node" filters metrics. It both reduces the scope of the metrics to those coming from the MacOS integration job and specifies that the “device” label cannot be empty. The result of this query is the raw stream of numbers that the graph displays. Although this view provides some insight into the performance of the system, it doesn’t provide the full story. A clearer picture of system performance requires understanding the rate of change that displays how fast the data being written is changing. To properly monitor disk performance, you need to also see spikes in activity that illustrate if and when the system is under load, and whether disk performance is at risk. PromQL includes a rate() function that shows the per-second average rate of increase over 5m (5-minute) intervals. This view provides a much clearer picture of what’s happening with the system. {{< figure src="/media/docs/grafana/intro-prometheus/rate-function.png" max-width="750px" caption="Prometheus rate function" >}} A counter metric is just one type of metric; it is a number (such as total bytes written) that only increases. Prometheus supports several others, such as the metric type gauge, which can increase or decrease. The following gauge visualization displays the total RAM usage on a computer. {{< figure src="/media/docs/grafana/intro-prometheus/gauge-example.png" max-width="750px" caption="Gauge visualization" >}} The third metric type is called a histogram, which counts observations and organizes them into configurable groups. The following example displays floating-point numbers grouped into ranges that display how frequently each occurred. {{< figure src="/media/docs/grafana/intro-prometheus/histogram-example.png" max-width="750px" caption="Histogram visualization" >}} These core concepts of time series, metrics, labels, and aggregation functions are foundational to Grafana and observability. Why this is valuable Software and systems are a difficult business. Sometimes things go wrong. Observability helps you understand a system’s state so that issues can be quickly identified and proactively addressed. And when problems do occur, you can be alerted to them to diagnose and solve them within your Service Level Objectives (SLOs). The three pillars of observability are metrics, logs, and traces. Prometheus supports the metrics pillar. When software on a computer runs slowly, observability can help you identify whether CPU is saturated, the system is out of memory, or if the disk is writing at maximum speed so you can proactively respond. Prometheus as software Prometheus isn’t just a data format; it is also considered an open source systems monitoring and alerting toolkit. That’s because Prometheus is software, not just data. Prometheus can scrape metric data from software and infrastructure and store it. Scraping means that Prometheus software periodically revisits the same endpoint to check for new data. Prometheus scrapes data from a piece of software instrumented with a client library. For example, a NodeJS application can configure the prom-client to expose metrics easily at an endpoint, and Prometheus can regularly scrape that endpoint. Prometheus includes a number of other tools within the toolkit to instrument your applications. Prometheus as deployment The first section of this document introduced the Prometheus as Data concept and how the Prometheus data model and metrics are organized. The second section introduced the concept of Prometheus as Software that is used to collect, process, and store metrics. This section describes how Prometheus as Data and Prometheus as Software come together. Consider the following example. Suppose a 'MyApp' application uses a Prometheus client to expose metrics. One approach to collecting metrics data is to use a URL in the application that points to an endpoint http://localhost:3000/metrics that produces Prometheus metrics data. The following image shows the two metrics associated with the endpoint. The HELP text explains what the metric means, and the TYPE text indicates what kind of metric it is (in this case, a gauge). MyAppnodejs_active_request_total indicates the number of requests (in this case, 1). MyAppnodejs_heap_size_total_bytes indicates the heap size reported in bytes. There are only two numbers because this data shows the value at the moment the data was fetched. {{< figure src="/media/docs/grafana/intro-prometheus/endpoint-data.png" max-width="750px" caption="Endpoint example" >}} The 'MyApp' metrics are available in an HTTP endpoint, but how do they get to Grafana, and subsequently, into a dashboard? The process of recording and transmitting the readings of an application or piece of infrastructure is known as telemetry. Telemetry is critical to observability because it helps you understand exactly what's going on in your infrastructure. The metrics introduced previously, for example, MyAppnodejs_active_requests_total, are telemetry data. To get metrics into Grafana, you can use either the Prometheus software or Grafana Alloy to scrape metrics. Grafana Alloy collects and forwards the telemetry data to open-source deployments of the Grafana Stack, Grafana Cloud, or Grafana Enterprise, where your data can be analyzed. For example, you can configure Grafana Alloy to pull the data from 'MyApp' every five seconds and send the results to Grafana Cloud. Metrics data is only one type of telemetry data; the other kinds are logs and traces. Using Grafana Alloy can be a great option to send telemetry data because as you scale your observability practices to include logs and traces, which Grafana Alloy also supports, you've got a solution already in place. The following image illustrates how Grafana Alloy works as an intermediary between 'MyApp' and Grafana Cloud. {{< figure src="/media/docs/alloy/flow-diagram-small-alloy.png" alt="Grafana Alloy" caption="Grafana Alloy" >}} Bringing it together The combination of Prometheus and Grafana Alloy gives you control over the metrics you want to report, where they come from, and where they’re going. Once the data is in Grafana, it can be stored in a Grafana Mimir database. Grafana dashboards consist of visualizations populated by data queried from the Prometheus data source. The PromQL query filters and aggregates the data to provide you the insight you need. With those steps, we’ve gone from raw numbers, generated by software, into Prometheus, delivered to Grafana, queried by PromQL, and visualized by Grafana. What’s next? Now that you understand how Prometheus metrics work, what will you build? One great next step is to build a dashboard in Grafana and start turning that raw Prometheus telemetry data into insights about what’s going with your services and infrastructure. Another great step is to learn about Grafana Mimir, which is essentially a database for Prometheus data. If you’re wondering how to make this work for a large volumes of metrics with a lot of data and fast querying, check out Grafana Mimir. If you’re interested in working with Prometheus data in Grafana directly, check out the Prometheus data source documentation, or check out PromQL basics. Introduction to time series Imagine you wanted to know how the temperature outside changes throughout the day. Once every hour, you'd check the thermometer and write down the time along with the current temperature. After a while, you'd have something like this: Time Value 09:00 24°C 10:00 26°C 11:00 27°C Temperature data like this is one example of what we call a time series — a sequence of measurements, ordered in time. Every row in the table represents one individual measurement at a specific time. Tables are useful when you want to identify individual measurements, but they make it difficult to see the big picture. A more common visualization for time series is the graph, which instead places each measurement along a time axis. Visual representations like the graph make it easier to discover patterns and features of the data that otherwise would be difficult to see. {{< figure src="/static/img/docs/example_graph.png" class="docs-image--no-shadow" max-width="850px" alt="Temperature data displayed on dashboard" >}} Temperature data like the one in the example, is far from the only example of a time series. Other examples of time series are: CPU and memory usage Sensor data Stock market index While each of these examples are sequences of chronologically ordered measurements, they also share other attributes: New data is appended at the end, at regular intervals — for example, hourly at 09:00, 10:00, 11:00, and so on. Measurements are seldom updated after they were added — for example, yesterday's temperature doesn't change. Time series are powerful. They help you understand the past by letting you analyze the state of the system at any point in time. Time series could tell you that the server crashed moments after the free disk space went down to zero. Time series can also help you predict the future, by uncovering trends in your data. If the number of registered users has been increasing monthly by 4% for the past few months, you can predict how big your user base is going to be at the end of the year. Some time series have patterns that repeat themselves over a known period. For example, the temperature is typically higher during the day, before it dips down at night. By identifying these periodic, or seasonal, time series, you can make confident predictions about the next period. If you know that the system load peaks every day around 18:00, you can add more machines right before. Aggregating time series Depending on what you're measuring, the data can vary greatly. What if you wanted to compare periods longer than the interval between measurements? If you'd measure the temperature once every hour, you'd end up with 24 data points per day. To compare the temperature in August over the years, you'd have to combine the 31 times 24 data points into one. Combining a collection of measurements is called aggregation. There are several ways to aggregate time series data. Here are some common ones: Average returns the sum of all values divided by the total number of values. Min and Max return the smallest and largest value in the collection. Sum returns the sum of all values in the collection. Count returns the number of values in the collection. For example, by aggregating the data in a month, you can determine that August 2017 was, on average, warmer than the year before. Instead, to see which month had the highest temperature, you'd compare the maximum temperature for each month. How you choose to aggregate your time series data is an important decision and depends on the story you want to tell with your data. It's common to use different aggregations to visualize the same time series data in different ways. Time series and monitoring In the IT industry, time series data is often collected to monitor things like infrastructure, hardware, or application events. Machine-generated time series data is typically collected with short intervals, which allows you to react to any unexpected changes, moments after they occur. As a consequence, data accumulates at a rapid pace, making it vital to have a way to store and query data efficiently. As a result, databases optimized for time series data have seen a rise in popularity in recent years. Time series databases A time series database (TSDB) is a database explicitly designed for time series data. While it's possible to use any regular database to store measurements, a TSDB comes with some useful optimizations. Modern time series databases take advantage of the fact that measurements are only ever appended, and rarely updated or removed. For example, the timestamps for each measurement change very little over time, which results in redundant data being stored. Look at this sequence of Unix timestamps: 1572524345, 1572524375, 1572524404, 1572524434, 1572524464 Looking at these timestamps, they all start with 1572524, leading to poor use of disk space. Instead, we could store each subsequent timestamp as the difference, or delta, from the first one: 1572524345, +30, +29, +30, +30 We could even take it a step further, by calculating the deltas of these deltas: 1572524345, +30, -1, +1, +0 If measurements are taken at regular intervals, most of these delta-of-deltas will be 0. Because of optimizations like these, TSDBs use drastically less space than other databases. Another feature of a TSDB is the ability to filter measurements using tags. Each data point is labeled with a tag that adds context information, such as where the measurement was taken. Here's an example of the InfluxDB data format that demonstrates how each measurement is stored. weather,location=us-midwest temperature=82 1465839830100400200 | -------------------- -------------- | | | | | | | | | +-----------+--------+-+---------+-+---------+ |measurement|,tag_set| |field_set| |timestamp| +-----------+--------+-+---------+-+---------+ Here are some of the TSDBs supported by Grafana: Graphite InfluxDB Prometheus Collecting time series data Now that we have a place to store our time series, how do we actually gather the measurements? To collect time series data, you'd typically install a collector on the device, machine, or instance you want to monitor. Some collectors are made with a specific database in mind, and some support different output destinations. Here are some examples of collectors: collectd statsd Prometheus exporters Telegraf A collector either pushes data to a database or lets the database pull the data from it. Both methods come with their own set of pros and cons: Pros Cons Push Easier to replicate data to multiple destinations. The TSDB has no control over how much data gets sent. Pull Better control of how much data that gets ingested, and its authenticity. Firewalls, VPNs or load balancers can make it hard to access the agents. Since it would be inefficient to write every measurement to the database, collectors pre-aggregate the data and write to the time series database at regular intervals. Time series dimensions In Introduction to time series, the concept of labels, also called tags, is introduced: Another feature of a TSDB is the ability to filter measurements using tags. Each data point is labeled with a tag that adds context information, such as where the measurement was taken. With time series data, the data often contain more than a single series, and is a set of multiple time series. Many Grafana data sources support this type of data. {{< figure src="/static/img/docs/example_graph_multi_dim.png" class="docs-image--no-shadow" max-width="850px" alt="Temperature by location" >}} The common case is issuing a single query for a measurement with one or more additional properties as dimensions. For example, querying a temperature measurement along with a location property. In this case, multiple series are returned back from that single query and each series has unique location as a dimension. To identify unique series within a set of time series, Grafana stores dimensions in labels. Labels Each time series in Grafana optionally has labels. Labels are a set of key/value pairs for identifying dimensions. Example labels could be {location=us} or {country=us,state=ma,city=boston}. Within a set of time series, the combination of its name and labels identifies each series. For example, temperature {country=us,state=ma,city=boston} could identify the series of temperature values for the city of Boston in the US. Different sources of time series data have dimensions stored natively, or common storage patterns that allow the data to be extracted into dimensions. Time series databases (TSDBs) usually natively support dimensionality. Prometheus also stores dimensions in labels. In TSDBs such as Graphite or OpenTSDB the term tags is used instead. In table databases such SQL, these dimensions are generally the GROUP BY parameters of a query. Multiple dimensions in table format In SQL or SQL-like databases that return table responses, additional dimensions are usually represented as columns in the query response table. Single dimension For example, consider a query like: SELECT BUCKET(StartTime, 1h), AVG(Temperature) AS Temp, Location FROM T GROUP BY BUCKET(StartTime, 1h), Location ORDER BY time asc This query would return a table with three columns with data types time, number, and string respectively: StartTime Temp Location 09:00 24 LGA 09:00 20 BOS 10:00 26 LGA 10:00 22 BOS The table format is a long formatted time series, also called tall. It has repeated time stamps, and repeated values in Location. In this case, we have two time series in the set that would be identified as Temp {Location=LGA} and Temp {Location=BOS}. Individual time series from the set are extracted by using the time typed column StartTime as the time index of the time series, the numeric typed column Temp as the series name, and the name and values of the string typed Location column to build the labels, such as Location=LGA. Multiple dimensions If the query is updated to select and group by more than just one string column, for example, GROUP BY BUCKET(StartTime, 1h), Location, Sensor, then an additional dimension is added: StartTime Temp Location Sensor 09:00 24 LGA A 09:00 24.1 LGA B 09:00 20 BOS A 09:00 20.2 BOS B 10:00 26 LGA A 10:00 26.1 LGA B 10:00 22 BOS A 10:00 22.2 BOS B In this case the labels that represent the dimensions will have two keys based on the two string typed columns Location and Sensor. This data results four series: Temp {Location=LGA,Sensor=A}, Temp {Location=LGA,Sensor=B}, Temp {Location=BOS,Sensor=A}, and Temp {Location=BOS,Sensor=B}. {{< admonition type="note" >}} More than one dimension is currently only supported in the Logs queries within the Azure Monitor service as of version 7.1. {{< /admonition >}} {{< admonition type="note" >}} Multiple dimensions are not supported in a way that maps to multiple alerts in Grafana, but rather they are treated as multiple conditions to a single alert. For more information, see the documentation on creating alerts with multiple series. {{< /admonition >}} Multiple values In the case of SQL-like data sources, more than one numeric column can be selected, with or without additional string columns to be used as dimensions. For example, AVG(Temperature) AS AvgTemp, MAX(Temperature) AS MaxTemp. This, if combined with multiple dimensions, can result in a lot of series. Selecting multiple values is currently only designed to be used with visualization. Additional technical information on tabular time series formats and how dimensions are extracted can be found in the developer documentation on data frames as time series.