Troubleshooting

• Generate a data source diagnostic bundle
• Troubleshoot with HAR captures
• Send a panel to Grafana Labs support

Generate a data source diagnostic bundle

{{< admonition type="note" >}} On-demand data source diagnostics is an experimental feature that helps you share troubleshooting evidence with Grafana Labs Technical Support. The feature is disabled by default. To switch it on, refer to Enable the feature toggle. {{< /admonition >}}

When a panel returns an error, no data, or data that looks wrong, Grafana Labs Technical Support needs to know what your data source actually returned. In Grafana Cloud, you can grant support access to your stack so an engineer can investigate live. A self-managed instance isn't reachable by support, so you collect the evidence yourself in a diagnostic bundle and send it in.

When you generate a diagnostic bundle, Grafana re-runs the panel's queries with traffic capture active and packages the result as a single .tar.gz file containing the requests and responses exchanged with your data source, the data the data source plugin returned to Grafana, the panel and dashboard configuration, and metadata about the capture.

{{< admonition type="warning" >}} A diagnostic bundle records upstream traffic and error messages verbatim, without redaction. Request headers, query parameters, request and response bodies, and error text are all stored as they occurred, so a bundle can contain authentication tokens, passwords, and the contents of the data your queries returned.

Treat a bundle as you would treat a credential. Review it before you share it, and send it only to Grafana Labs Technical Support, through your Grafana Labs support ticket. {{< /admonition >}}

Before you begin

To generate a diagnostic bundle, you need the following:

• A self-managed Grafana instance running version 13.2 or later. Diagnostic bundles aren't available in Grafana Cloud, and earlier self-managed versions don't include this feature.
• The Grafana server administrator role. The menu items described on this page don't appear for other users.
• The grafana.onDemandDiagnostics feature toggle enabled.
• A data source whose plugin supports diagnostic bundles. Refer to Data source support for capturing upstream traffic.

Enable the feature toggle

The grafana.onDemandDiagnostics feature toggle is disabled by default. While it's disabled, Grafana hides the diagnostics UI elements and the underlying endpoints return 404.

To enable it, add the following to your Grafana configuration file and restart Grafana:

[feature_toggles]
enable = grafana.onDemandDiagnostics

To set a feature toggle in other ways, such as environment variables, refer to the feature_toggles section of Configure Grafana and Manage feature toggles.

Generate a bundle for a single panel

1. Open the dashboard that contains the panel you want to diagnose.

2. Hover over any part of the panel to display the panel menu in the top right corner.

3. Click the menu and select More > Download diagnostics.

   Grafana opens a drawer describing what the bundle contains, including a May contain sensitive data warning.

4. Click Download diagnostics.

   Grafana re-runs the panel's queries with capture active and downloads the bundle. The button reads Generating… while this happens, which can take a moment.

5. Review the bundle, then attach it to your support ticket.

Generate a bundle for an entire dashboard

Use a dashboard bundle when you don't know which panel is at fault, or when the problem involves more than one panel.

1. Open the dashboard you want to diagnose.

2. In the dashboard toolbar, click Export > Download diagnostics.

3. Click Download diagnostics.

   Grafana captures panels one at a time and reports its progress, for example Capturing panel 3 of 12. Large dashboards can take a while to capture.

4. Review the bundle, then attach it to your support ticket.

Grafana captures each panel independently, so one failing panel doesn't stop the rest of the bundle from being produced. Skipped and failed panels are recorded in manifest.json.

What the bundle contains

Extract the bundle to review it:

tar -xzf <bundle>.tar.gz

A single-panel bundle contains the following files:

File	Contents
traffic.har	The HTTP requests and responses exchanged with the data source, in HAR 1.2 format, including bodies and timings. Connection failures are recorded as an entry with a comment. Not available for every data source; refer to Data source support for capturing upstream traffic.
querydata.json	The queries Grafana submitted and the data the data source plugin returned, including per-query status, errors, and the schema and values of each returned frame. Large responses are reduced to a summary that records what was truncated.
query-error.txt	The query error text, when the queries failed.
querydata-error.txt	Present only when the query data couldn't be written. The rest of the bundle is still produced.
panel.json	The panel configuration: queries, transformations, field options, and visualization settings.
dashboard.json	The dashboard configuration, for context.

A dashboard bundle contains a panels/<id>-<title>/ directory holding those files for each panel, plus a manifest.json recording when the bundle was generated, how many panels ran, and the data sources, sizes, and any errors for each panel.

Data source support for capturing upstream traffic

traffic.har is usually the most valuable file in the bundle, but it isn't available for every data source. When upstream traffic can't be captured, Grafana still generates the rest of the bundle, and querydata.json still shows what the data source plugin returned.

Data sources that support capture today

• Amazon Managed Service for Prometheus
• Azure Data Explorer
• Azure Monitor
• Azure Prometheus
• CSV
• Dynatrace
• Elasticsearch
• Falcon LogScale
• Google Sheets
• Graphite
• InfluxDB
• Jaeger
• Loki
• OpenTSDB
• Prometheus
• Sentry
• Tempo, for its standard HTTP query path—TraceQL streaming is not supported.
• Zabbix
• Zipkin

Limitations

As an experimental feature, on-demand data source diagnostics currently has the following limitations:

• Grafana re-runs your queries. A bundle reflects a fresh execution at the moment you generate it, not the original failure, and it bypasses the query cache. An intermittent failure might not reproduce, so a healthy-looking bundle doesn't prove the problem is resolved.

• Queries run as you. Because you generate the bundle as a server administrator, failures that depend on the affected user's identity might not reproduce. Examples include per-user OAuth forwarding and row-level database permissions. If a specific user is affected, note that in your ticket.

• A bundle captures panel and dashboard queries only. Failures in the variable picker, in a data source's Save & test button, in annotation queries, or during alert rule evaluation don't appear.

• Server-side expressions and panel transformations are recorded in panel.json as configuration, not as the data that flows between them. So if querydata.json looks correct but the panel doesn't, check that configuration.

• Large responses might be trimmed, and less commonly, generation fails outright. The following limits apply:

  • querydata.json is capped at 1 GiB for a single panel and 1.5 GiB across a whole-dashboard bundle. A response over that limit is replaced by a summary that records what was left out, and the rest of the bundle still generates. For a dashboard bundle, once the shared 1.5 GiB budget is used up, the remaining panels' query data is skipped, and the reason is recorded in manifest.json.
  • traffic.har is limited to 64 MiB per request or response body and 256 MiB of retained traffic per panel. A body over the per-body limit is kept as a truncated prefix, and its entry reports bodySize: -1 with the actual size in content.size. When a panel's traffic exceeds the per-panel limit, further entries keep their headers, sizes, and timings but omit the body text.

• Generating a bundle can fail outright, instead of trimming, when the entire request to Grafana exceeds 100 MiB. That request includes the queries, the panel and dashboard JSON, and, for a single panel, the frames your browser is holding. This is uncommon, because it requires an unusually large payload from the browser itself—which a large data source response alone doesn't produce.

  If you hit one of these limits, or a bundle takes an unreasonably long time to generate, reproduce the problem on a minimal setup instead of the original dashboard:

  • Use a temporary panel or dashboard holding only the misbehaving panel.
  • Narrow the time range to the shortest window that still reproduces the problem. Fewer returned data points means a smaller bundle and a faster capture.

  If you generate the bundle from a reduced dashboard, note that in your ticket. A smaller, reproducible setup is also easier for Grafana Labs Technical Support to work with.

Related resources

• Send a panel to Grafana Labs support
• Generate a HAR capture to send to Grafana Labs support
• Troubleshoot data sources

Troubleshoot with HAR captures

Support may request information about the network requests your browser generated in the form of a HAR file.

Important: HAR files can contain sensitive information, including cookies, session data, personal information, and sometimes credentials or tokens entered during troubleshooting. Before sharing a HAR file, make sure you review or sanitize it. You can then attach the redacted HAR file to your Grafana Support ticket.

To generate a HAR capture, follow the steps for your browser:

Chrome

1. In your Chrome browser, go to the page where the issue is occurring.
2. Click the menu icon and go to More Tools > Developer Tools.
3. In the drawer that opens, click the Network tab.
4. In the upper left hand corner of the drawer, make sure the round Record button is red. If it's grey, click the button to begin recording.
5. Select the Preserve log option box if it isn't already selected.
6. Click the grey crossed circle button to clear any existing requests from the panel.
7. Reproduce the issue while recording the network requests.
8. Click the download button to download the HAR file.
9. Upload the HAR file to your Grafana Support ticket for investigation.

Firefox

1. In your Firefox browser, go to the page where the issue is occurring.
2. Click the menu icon and go to More Tools > Web Developer Tools.
3. In the drawer that opens, click the Network tab.
4. In the upper left corner of the drawer, click the trash can icon to clear any existing requests from the recording.
5. Reproduce the issue while recording the network requests.
6. To save the HAR file, click the gear icon in the upper right hand corner of the panel and select Save All As HAR.
7. Upload the HAR file to your Grafana Support ticket for investigation.

Safari

1. In your Safari browser, go to the page where the issue is occurring.
2. Open the Develop menu option and select Show Web Inspector. {{< admonition type="note" >}} The Show features for web developers option needs to be enabled for the Develop option to appear. For more information, refer to Use the developer tools in the Develop menu in Safari on Mac. {{< /admonition >}}
3. In the panel that appears, click the Network tab.
4. In the upper right corner of the panel, click the trash can icon to clear any existing requests from the recording.
5. Reproduce the issue while recording the network requests.
6. Click the Export button to save the HAR file:
7. Upload the HAR file to your Grafana Support ticket for investigation.

Edge

1. In your Edge browser, go to the page where the issue is occurring.
2. Click the ... button and go to More Tools > Developer Tools.
3. In the panel that appears, click on the Network tab.
4. In the upper left hand corner of the panel, make sure the round Record button is red. If it's grey, click the button to begin recording
5. Verify that the Preserve log option box is checked.
6. Click the grey crossed circle button to clear any existing requests from the panel.
7. Reproduce the issue while the network requests are recorded.
8. Click the download button to save the HAR file:
9. Upload the HAR file to your Grafana Support ticket for investigation.

Send a panel to Grafana Labs support

When you encounter problems with any of your visualizations, you can send the panel JSON model to Grafana Labs Technical Support and request help with troubleshooting your issue.

The panel that you send includes all query response data and all visualizations settings. Upon receiving your panel, Grafana Labs Technical Support imports your data into a local version of Grafana and begins researching your problem.

1. Open the dashboard that contains the panel you want to send to Grafana Labs.

2. Hover over any part of the panel to display the actions menu on the top right corner.

3. Click the menu and select More > Get help.

   Grafana opens a standalone support dashboard that contains the data you are sending to Grafana Labs Technical Support.

   If you enable iframes, the support dashboard includes the visualization that looks similar to the following image.

   

4. To send the panel data to Grafana Labs via Github:

   a. Click Copy to clipboard.

   b. In the Grafana/Grafana repository, create an issue, and paste the contents of the support dashboard.

5. To send the panel data to Grafana Labs via a support ticket:

   a. Click Dashboard.

   Grafana downloads the support dashboard to a TXT file.

   b. Attach the TXT file to a support ticket that you send to Grafana Labs Technical Support.