Pyroscope data source on Grafana Labs

Configure the Pyroscope data source

Mon, 24 Aug 2026 19:23:44 +0000

Configure the Pyroscope data source

The Pyroscope data source sets how Grafana connects to your Pyroscope backend.

You can configure the data source using either the data source interface in Grafana or using a configuration file. This page explains how to set up and enable the data source capabilities using Grafana.

If you make any changes, select Save & test to preserve those changes.

If you’re using your own installation of Grafana, you can provision the Pyroscope data source using a YAML configuration file. For more information about provisioning and available configuration options, refer to Provisioning Grafana.

Before you begin

To configure a Pyroscope data source, you need administrator rights to your Grafana instance and a running Pyroscope backend that’s receiving profiling data.

If you’re provisioning a Pyroscope data source, then you also need administrative rights on the server hosting your Grafana instance.

Add or modify a data source

You can use these procedures to configure a new Pyroscope data source or to edit an existing one.

Create a new data source

To configure basic settings for the data source, complete the following steps:

Select Connections in the main menu.
Enter Grafana Pyroscope in the search bar.
Select Grafana Pyroscope.
Select Add new data source in the top-right corner of the page.
On the Settings tab, complete the Name, Connection, and Authentication sections.
- Use the Name field to specify the name used for the data source in panels, queries, and Explore. Toggle the Default switch for the data source to be pre-selected for new panels.
- Under Connection, enter the URL of the Pyroscope instance. For example, https://example.com:4040. Refer to Connection URL for more information.
- Complete the Authentication section.
Optional: Use Additional settings to configure other options.
Select Save & test.

Update an existing data source

To modify an existing Pyroscope data source:

Select Connections in the main menu.
Select Data sources to view a list of configured data sources.
Select the Pyroscope data source you want to modify.
Optional: Use Additional settings to configure or modify other options.
After completing your updates, select Save & test.

Connection URL

The data source connection URL should point to a location of a running Pyroscope backend.

Grafana Cloud Profiles

Your Grafana Cloud instance automatically includes a fully provisioned data source.

If you are running a self-managed Grafana instance or need to configure an additional Pyroscope data source pointing to Grafana Cloud Profiles, you can find the Pyroscope URL under the Manage your stack section for your organization.

To authenticate against Grafana Cloud Profiles, enable Basic auth and enter your numeric instance ID as the username and an access policy token as the password. The token must belong to an access policy that includes the profiles:read scope and targets the correct stack. For more information, refer to Grafana Cloud access policies.

Self-managed Pyroscope backend

The connection URL for a self-managed Pyroscope backend depends on how Pyroscope is deployed. Refer to the steps under Query profiles in Grafana for more information on how to configure the data source.

If you plan to use the Profiles Drilldown application and you are running a self-managed Pyroscope backend in microservices mode, the data source connection URL should point to a gateway or proxy that routes requests to the corresponding Pyroscope service. Refer to the Helm ingress configuration for specific routing requirements.

Authentication

Use this section to select an authentication method to access the data source.

Note
Use Transport Layer Security (TLS) for an additional layer of security when working with Pyroscope. For additional information on setting up TLS encryption with Pyroscope, refer to Pyroscope configuration.

To set up authentication:

Select an authentication method from the drop-down list:
- Basic authentication: Authenticates your data source using a username and password
- Forward OAuth identity: Forwards the OAuth access token and the OIDC ID token, if available, of the user querying to the data source
- No authentication: No authentication is required to access the data source
For Basic authentication only: Enter the User and Password.
Optional: Complete the TLS settings for additional security methods.

TLS Client Authentication

Toggle on to use client authentication. When enabled, it adds the Server name, Client cert, and Client key fields. The client provides a certificate that is validated by the server to establish the client’s trusted identity. The client key encrypts the data between client and server. These details are encrypted and stored in the Grafana database.

Add self-signed certificate

Activate this option to use a self-signed TLS certificate. You can add your own Certificate Authority (CA) certificate on top of one generated by the certificate authorities for additional security measure.

Skip TLS certification validation

When activated, it bypasses TLS certificate verification. Not recommended, unless absolutely necessary for testing.
Optional: Add HTTP Headers. You can pass along additional context and metadata data about the request and response. Select Add header to add Header and Value fields.
Select Save & test to preserve your changes.

Additional settings

Additional settings are optional settings that give you more control over your data source. Select the Additional settings section to expand it and configure these options.

Advanced HTTP settings

The Grafana Proxy deletes forwarded cookies. Use the Allowed cookies field to specify cookies that should be forwarded to the data source by name.

The Timeout field sets the HTTP request timeout in seconds.

Private data source connect

Note
This feature is only available in Grafana Cloud.

Use private data source connect (PDC) to connect to and query data within a secure network without opening that network to inbound traffic from Grafana Cloud.

Refer to Private data source connect for more information on how PDC works and Configure Grafana private data source connect (PDC) for steps on setting up a PDC connection.

Use the drop-down list to select a configured private data source. If you make changes, select Test & save to preserve your changes.

Use Manage private data source connect to configure and manage any private data sources you have configured.

Querying

Use the Minimal step field to set the minimum step used for metric queries that return time-series data. The default value is 15 seconds. Enter a value in Grafana duration format using a number and a time unit specifier, such as 15s, 1m, or 1h. Set it to the same value as, or higher than, the scrape interval configured in your Pyroscope database.

Adjusting this option can help prevent gaps when you zoom in to profiling data.

Verify the connection

After you configure the data source, select Save & test to verify the connection. When the connection is successful, Grafana displays a Data source is working message. If the test fails, Grafana displays the error returned by the Pyroscope backend. For help resolving connection errors, refer to Troubleshoot the Pyroscope data source.

Provision the data source

You can provision the Pyroscope data source using YAML configuration files or Terraform, as part of the Grafana provisioning system. For more information about provisioning and available configuration options, refer to Provisioning Grafana.

YAML provisioning example

The following example provisions a Pyroscope data source:

apiVersion: 1

datasources:
  - name: Grafana Pyroscope
    type: grafana-pyroscope-datasource
    url: http://localhost:4040
    jsonData:
      minStep: '15s'

To provision with basic authentication, for example when connecting to Grafana Cloud Profiles:

apiVersion: 1

datasources:
  - name: Grafana Pyroscope
    type: grafana-pyroscope-datasource
    url: https://example.grafana.net
    basicAuth: true
    basicAuthUser: '<INSTANCE_ID>'
    jsonData:
      minStep: '15s'
    secureJsonData:
      basicAuthPassword: <ACCESS_POLICY_TOKEN>

Replace the following:

<INSTANCE_ID>: Your numeric instance ID, found under Manage your stack.
<ACCESS_POLICY_TOKEN>: An access policy token with the profiles:read scope.

Terraform example

To provision the data source with Terraform, use the grafana_data_source resource:

resource "grafana_data_source" "pyroscope" {
  type = "grafana-pyroscope-datasource"
  name = "Grafana Pyroscope"
  url  = "http://localhost:4040"

  json_data_encoded = jsonencode({
    minStep = "15s"
  })
}

To provision with basic authentication:

resource "grafana_data_source" "pyroscope" {
  type                = "grafana-pyroscope-datasource"
  name                = "Grafana Pyroscope"
  url                 = "http://localhost:4040"
  basic_auth_enabled  = true
  basic_auth_username = "<USERNAME>"

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = "<PASSWORD>"
  })
}

Replace the following:

<USERNAME>: Your Pyroscope username.
<PASSWORD>: Your Pyroscope password.

Configure Trace to profiles

Mon, 24 Aug 2026 19:23:44 +0000

Configure Trace to profiles

Trace to profiles lets you navigate from a trace span directly to the profiling data for that span. You configure it in the Tempo data source, which connects your tracing data in Tempo with your profiling data in Pyroscope.

Note
Your application must be instrumented for profiles and traces. For more information, refer to Link traces to profiles.

Using Trace to profiles, you can use the ability to correlate different signals in Grafana by adding the functionality to link between traces and profiles.

Trace to profiles lets you link your Grafana Pyroscope data source to tracing data. When configured, this connection lets you run queries from a trace span into the profile data using Explore. Each span links to your queries. Clicking a link runs the query in a split panel. If tags are configured, Grafana dynamically inserts the span attribute values into the query. The query runs over the time range of the (span start time - 60) to (span end time + 60 seconds).

Embedded flame graphs are also inserted into each span details section that has a linked profile. This lets you see resource consumption in a flame graph visualization for each span without having to navigate away from the current view.

There are two ways to configure the trace to profiles feature:

Use a basic configuration with default query, or
Configure a custom query where you can use a template language to interpolate variables from the trace or span.

Before you begin

To use Trace to profiles, you need:

A Grafana Pyroscope data source connected to your Pyroscope instance.
A Tempo data source configured with the Trace to profiles settings described on this page.
Your application instrumented with all three of the following:
1. Profiling: A Pyroscope SDK or Grafana Alloy sending profiling data to Pyroscope. Refer to Configure the client to send profiles for setup instructions.
2. Tracing: An OpenTelemetry SDK sending trace data to Tempo. Refer to the OpenTelemetry getting started documentation for setup instructions.
3. Span profiling bridge: A language-specific OpenTelemetry integration package that links profiling data with trace spans. Without this package, traces and profiles are independent signals with no connection between them. Refer to Link tracing and profiling with span profiles for per-language setup instructions.

Use a basic configuration

To use a basic configuration, follow these steps:

In the left menu, select Connections > Data sources.
Select your configured Tempo data source from the Data source list.
Scroll down to the Traces to profiles section.
Select a Pyroscope data source in the Data source drop-down.
Optional: Add one or more tags to use in the query. If left blank, the default values of service.name and service.namespace are used.

The tags you configure must be present in the spans attributes or resources for a trace-to-profiles span link to appear.

You can optionally configure a new name for the tag. This is useful if the tag has dots in the name and the target data source doesn’t allow dots in labels. In that case, you can remap service.name to service_name.
Select one or more profile types to use in the query. Select the drop-down and choose options from the menu.

The profile type or app must be selected for the query to be valid. Grafana doesn’t show any data if the profile type or app isn’t selected when a query runs.
Select Save and Test.

If you have configured a Pyroscope data source and no profile data is available or the Profiles for this span button and the embedded flame graph isn’t visible, verify that the pyroscope.profile.id key-value pair exists in your span tags.

Configure a custom query

To use a custom query with the configuration, follow these steps:

In the left menu, select Connections > Data sources.
Select a configured Tempo data source from the Data source list.
Scroll down to the Traces to profiles section.
Select a Pyroscope data source in the Data source drop-down.
Optional: Choose any tags to use in the query. If left blank, the default values of service.name and service.namespace are used.

These tags can be used in the custom query with ${__tags} variable. This variable interpolates the mapped tags as list in an appropriate syntax for the data source. Only tags present in the span are included. Tags that aren’t present are omitted.

You can also configure a name for the tag. Tag names are useful where the tag has dots in the name and the target data source doesn’t allow using dots in labels. For example, you can remap service.name to service_name. If you don’t map any tags here, you can still use any tag in the query, for example: method="${__span.tags.method}". Learn more about custom query variables.
Select one or more profile types to use in the query. Select the drop-down and choose options from the menu.
Switch on Use custom query to enter a custom query.
Specify a custom query to be used to query profile data. You can use various variables to make that query relevant for current span. The link shows only if all the variables are interpolated with non-empty values to prevent creating an invalid query. You can interpolate the configured tags using the $__tags keyword.
Select Save and Test.

Configuration options

The following table describes options for configuring your Trace to profiles settings:

Setting name	Description
Data source	Defines the target data source. You can select a Pyroscope [profiling] data source.
Tags	Defines the tags to use in the profile query. Default: `cluster`, `hostname`, `namespace`, `pod`, `service.name`, `service.namespace`. You can change the tag name for example to remove dots from the name if they’re not allowed in the target data source. For example, map `http.status` to `http_status`.
Profile type	Defines the profile type to use in the query.
Use custom query	Toggles use of custom query with interpolation.
Query	Input to write custom query. Use variable interpolation to customize it with variables from span.

Verify the connection

After you configure the Tempo data source and instrument your application, verify that Trace to profiles works.

Open a trace in Explore using your Tempo data source.
Expand a span to view its details.
Confirm that the Profiles for this span button appears in the span details.
Confirm that an embedded flame graph appears in the span details section.

If the span attribute pyroscope.profile.id isn’t present on the span, the OpenTelemetry bridge package isn’t configured correctly. Refer to Link tracing and profiling with span profiles for per-language setup instructions.

Troubleshoot trace to profiles

Use the following checklist to diagnose common issues with Trace to profiles.

No “Profiles for this span” button appears

Verify that you installed and configured the OTel bridge package for your language. This is a separate package from the Pyroscope SDK and OpenTelemetry SDK. Refer to Link tracing and profiling with span profiles for per-language instructions.
Check that the pyroscope.profile.id attribute exists on the span in Tempo. If it’s missing, the bridge package isn’t tagging spans correctly.
Check that the span_name label exists on the profiling data in Pyroscope. If it’s missing, the bridge package isn’t labeling profiling samples correctly.
Verify that the tags you configured in the Tempo data source are present in the span’s attributes or resources. If the tags don’t match, the span link doesn’t appear.

Button appears but no profile data is shown

The span might be shorter than the sampling interval of the profiler. Span profiling is most effective on spans longer than 20ms.
CPU profiling only captures time spent actively executing on the CPU. If the span is mostly waiting on I/O, network calls, or database queries, the CPU profile for that span may be sparse or empty.
Verify that the profile type selected in the Tempo data source configuration matches the profile type your application sends.

Tags don’t match

Tags you configure in the Tempo data source must be present in the span’s attributes or resources. If a tag has dots in its name, for example, service.name, and the Pyroscope data source doesn’t allow dots in labels, remap it to a name without dots, for example, service_name.

Query profile data

Mon, 24 Aug 2026 19:23:44 +0000

Query profile data

You can query your profile data using the open source Grafana Profiles Drilldown app or the built-in Pyroscope data source query editor.

Profiles Drilldown provides a queryless experience for inspecting your profiling data with opinionated workflows to assist your investigation.
Data source query editor provides complete control over your data exploration and is recommended for experienced users.

In addition, you can embed flame graph panels in Grafana dashboards. Refer to the Flame graph panel documentation for details.

Profiles Drilldown

Grafana Profiles Drilldown is a native Grafana application designed to integrate seamlessly with Pyroscope, the open source continuous profiling platform, providing a smooth, queryless experience for browsing and analyzing profiling data.

You can use Profiles Drilldown in Grafana Cloud or in your own Grafana instance. For more information, refer to Access or install Profiles Drilldown.

Use cases

There are several different modes for viewing, analyzing, and comparing profiling data.

The main use cases are the following:

Proactive: Cutting costs, addressing latency issues, or optimizing memory usage for applications
Reactive: Resolving incidents with line-level accuracy or debugging active latency/memory issues

Profiles Drilldown provides an intuitive interface to specifically support proactive and reactive use cases. You get a holistic view of all of your services and how they’re functioning, but also the ability to drill down for more targeted root cause analysis.

Profiles Drilldown offers a convenient platform to analyze profiles and get insights that are impossible to get from using other traditional signals like logs, metrics, or tracing.

Give it a try using Grafana Play

With Grafana Play, you can explore and see how it works, learning from practical examples to accelerate your development. This feature can be seen on the Grafana Play site.

Try it

Pyroscope query editor

The Pyroscope data source query editor gives you access to a profile type selector, a label selector, and collapsible options.

Like Profiles Drilldown, the query editor also provides a flame graph to visualize data.

To access the query editor:

Sign into Grafana or Grafana Cloud.
Select your Pyroscope data source.
From the menu, choose Explore.
Select a profile type from the drop-down menu.

Profile selector
Use the labels selector input to filter by labels. Pyroscope uses similar syntax to Prometheus to filter labels. Refer to Label selector syntax for available operators and syntax.

While the label selector can be left empty to query all profiles without filtering by labels, you must select a profile type or app for the query to be valid.

Grafana doesn’t show any data if the profile type or app isn’t selected when a query runs.
Expand the Options section to view the available query options.

Label selector syntax

The label selector uses a syntax similar to Prometheus. Enclose one or more label matchers in curly braces, separated by commas. Each matcher combines a label name, an operator, and a value in double quotes.

The label selector supports the following operators.

Operator	Description
`=`	Selects labels that exactly match the value.
`!=`	Selects labels that don’t match the value.
`=~`	Selects labels that match the regular expression.
`!~`	Selects labels that don’t match the regular expression.

The following examples show common label selectors:

{service_name="my-service"}
{service_name="my-service", namespace="production"}
{service_name=~"cart.*"}
{service_name="checkout", namespace!="staging"}

Leave the label selector empty, or set it to {}, to query all profiles for the selected profile type without filtering by labels.

Query options

The Options section provides the following settings.

Option	Description
Query Type	Sets the type of data the query returns: Profile for flame graph data, Metric for time-series data, or Both. You can only select Both in Explore. Panels on dashboards allow only one visualization.
Group by	Groups metric data by one or more labels. Without a Group by label, metric data aggregates over all labels into a single time series. Group by applies only to metric data and doesn’t change the profile results.
Limit	Sets the maximum number of time series returned when Group by is set. The series returned are always ordered by descending value for the total aggregated data over the time period. Doesn’t apply to profile queries.
Span ID	Sets the span ID from which to search for profiles, so you can view the profile associated with a specific trace span. This option supports Trace to profiles, for example, `64f170a95f537095`.
Max Nodes	Sets the maximum number of nodes rendered in the flame graph. Lower values improve performance for large profiles by aggregating smaller nodes. The field placeholder shows the default value, `16384`.
Annotations	Includes profiling annotations, such as sampling and throttling events, on time-series results. Use these annotations to identify when Pyroscope reduced or limited ingestion for a service.

Select a query type to return the profile data. Data is shown in the Flame Graph, metric data visualized in a graph, or both.

Using Group by, you can group metric data by a specified label. You can use multiple labels to group by.

In conjunction with Group by, you can set a positive number in the Limit input to limit the maximum number of time series returned by the data source.

Exemplars

When the profilesExemplars feature is enabled, an Exemplars toggle appears in the Options section. Exemplars link individual profile samples to points on a metrics graph, so you can jump from a spike in a time series to the specific profile that produced it.

The profilesExemplars feature is generally available and enabled by default.

Heatmap

When the profilesHeatmap feature is enabled, a Heatmap toggle appears in the Options section. The heatmap visualizes the distribution of profile samples over time, which helps you spot patterns and outliers across many profiles.

When you enable Heatmap, use the Heatmap Type option to choose how samples are aggregated:

Individual: Shows individual profile samples.
Span: Aggregates samples by span duration, for use with Trace to profiles.

Note
The profilesHeatmap feature is experimental. Experimental features are subject to change and may be removed in a future release. Enable it using the profilesHeatmap feature toggle.

UTF-8 label names

When the pyroscopeUTF8LabelNames feature is enabled, the label selector supports UTF-8 label names. Enclose label names that contain characters such as dots in double quotes, for example, {"k8s.namespace"="production"}.

The pyroscopeUTF8LabelNames feature is in public preview and enabled by default.

Query examples

The following examples show common ways to build queries in the query editor. Each example lists the profile type to select and the label selector to enter.

Investigate high CPU usage for a service:

Select the process_cpu:cpu:nanoseconds:cpu:nanoseconds profile type and filter to the affected service:

{service_name="checkout"}

Set Query Type to Both in Explore to view the CPU flame graph alongside a time series of CPU usage.

Compare memory allocations across environments:

Select the memory:alloc_space:bytes:space:bytes profile type, set Query Type to Metric, and set Group by to namespace:

{service_name="cart"}

The metrics graph shows a separate time series for each namespace, which lets you compare allocation behavior between environments such as production and staging.

Analyze goroutine growth:

Select the goroutine:goroutine:count:goroutine:count profile type to inspect the number of goroutines and where they’re created:

{service_name=~"api.*"}

View the profile for a specific trace span:

Select a profile type, set Query Type to Profile, and enter the span ID in the Span ID field, for example, 64f170a95f537095. The flame graph then shows only the profile data associated with that span. This example supports Trace to profiles.

Profiles query results

Profiles can be visualized in a flame graph. Refer to the Flame Graph documentation to learn about the visualization and its features.

Pyroscope returns profiles aggregated over a selected time range. The absolute values in the flame graph grow as the time range gets bigger while keeping the relative values meaningful. You can zoom in on the time range to get a higher granularity profile up to the point of a single scrape interval.

Metrics query results

Metrics results represent the aggregated sum value over time of the selected profile type.

This allows you to quickly see any spikes in the value of the scraped profiles and zoom in to a particular time range.

Next steps

Pyroscope template variables

Mon, 24 Aug 2026 19:23:44 +0000

Pyroscope template variables

Template variables let you create dynamic, reusable dashboards by replacing hard-coded values, such as service names or profile types, with selectable variables. Grafana displays these variables as drop-down menus at the top of the dashboard, letting viewers change the displayed data without editing queries.

For an introduction to templating and template variables, refer to Templating and Add and manage variables.

Query variable types

Query variables query the Pyroscope data source to populate drop-down values. When you create a query variable, select a Pyroscope data source and choose a query type.

Query type	Required inputs	Description
Profile type	None	Returns the profile types available from the data source. The drop-down displays a readable label, while the variable value is the full profile type ID, such as `process_cpu:cpu:nanoseconds:cpu:nanoseconds`.
Label	Profile type	Returns the label names available for the selected profile type, such as `service_name`, `namespace`, and `region`.
Label value	Profile type, label	Returns the values for the selected label, scoped to the selected profile type.

Because the Label and Label value query types require a profile type, and Label value also requires a label, you can chain these variables so that each one depends on the previous selection.

Create a query variable

To create a query variable:

Navigate to Dashboard settings > Variables.
Select Add variable.
Set Select variable type to Query.
Select the Pyroscope data source.
Select a Query type of Profile type, Label, or Label value.
For a Label or Label value query, select a Profile type.
For a Label value query, select a Label.
Select Apply to save the variable.

Query variable examples

The following examples show common ways to use Pyroscope query variables.

Select a profile type from a drop-down:

Create a variable named profile_type to let viewers switch the profile type without editing the query:

Set Query type to Profile type.
Save the variable. The drop-down displays readable labels, while each value is the full profile type ID, such as process_cpu:cpu:nanoseconds:cpu:nanoseconds.

You can then reference $profile_type in the query editor’s profile type field.

Populate a drop-down with services:

Create a variable named service that lists the services sending profiles:

Set Query type to Label value.
Set Profile type to a profile type, such as process_cpu:cpu:nanoseconds:cpu:nanoseconds.
Set Label to service_name.
Save the variable. The drop-down now lists the values of the service_name label.

Chain dependent variables:

Combine variables so each one narrows the next. This example builds a profile type selector, a label selector, and a label value selector that all update together:

Create a profile_type variable with Query type set to Profile type.
Create a label variable with Query type set to Label and Profile type set to $profile_type.
Create a label_value variable with Query type set to Label value, Profile type set to $profile_type, and Label set to $label.

When a viewer changes profile_type, the label drop-down refreshes with the labels for that profile type, and label_value refreshes with the values for the selected label.

Use variables in queries

After you create a variable, reference it in the query editor using the $variable or ${variable} syntax. You can reference variables in both the profile type field and the label selector.

For example, if you have a variable named service created with the Label value query type on the service_name label, reference it in the label selector:

{service_name="$service"}

To let viewers switch the profile type, enter $profile_type in the query editor’s profile type field.

When Multi-value or Include All is enabled for the variable, the value becomes a regular expression pattern, such as value1|value2. Use the =~ operator instead of = in your label selector:

{service_name=~"$service"}

Where you can use template variables

The Pyroscope data source interpolates template variables in only some parts of the query editor. The following table lists which fields support variables.

Query editor field	Supports template variables
Profile type	Yes
Label selector	Yes
Group by	No
Span ID	No
Max Nodes	No
Limit	No

Query variable definitions also support variables. You can reference one variable in another to chain Label and Label value queries, as shown in Chain dependent variables.

Note
Fields other than the profile type and label selector don’t interpolate template variables. If you enter a variable such as $group in the Group by, Span ID, Max Nodes, or Limit fields, Grafana passes the literal text to the backend instead of the variable’s value.

Filters variable

The Pyroscope data source supports the Filters variable type, which lets dashboard viewers add label filters without editing queries.

To set up a Filters variable:

Navigate to Dashboard settings > Variables.
Select Add variable.
Set Select variable type to Filters.
Select the Pyroscope data source.
Select Apply to save the variable.

After you add the variable, a filter bar appears at the top of the dashboard. Viewers can add filters by selecting a label, an operator (=, !=, =~, !~), and a value. Grafana applies these filters to all Pyroscope queries on the dashboard.

Query profile data: Use variables in the label selector.
Troubleshoot the Pyroscope data source: Resolve variable-related query issues.

Troubleshoot Pyroscope data source issues

Mon, 24 Aug 2026 19:23:44 +0000

Troubleshoot Pyroscope data source issues

This document provides solutions to common issues you may encounter when configuring or using the Pyroscope data source. For configuration instructions, refer to Configure the Pyroscope data source.

Connection and configuration errors

These errors occur when setting up the data source or when connecting to the Pyroscope backend.

“Data source is not working” or connection fails

Symptoms:

Save & test fails with a connection error
Error message indicates the Pyroscope backend is unreachable
Timeout errors when testing the data source

Possible causes and solutions:

Cause	Solution
Incorrect URL	Verify the URL points to your Pyroscope instance. For self-managed deployments, the default port is `4040` (for example, `http://localhost:4040`).
Pyroscope service not running	Check that the Pyroscope backend is running and accessible. Use `curl` or a browser to test connectivity to the Pyroscope URL.
Network connectivity issues	Verify network connectivity from the Grafana server to the Pyroscope endpoint. Check firewall rules allow outbound connections on the required port.
Microservices mode routing	If running Pyroscope in microservices mode, ensure the URL points to a gateway or proxy that routes requests correctly. Refer to the Helm ingress configuration for routing requirements.
TLS/SSL certificate issues	If using HTTPS, verify the certificate is valid and trusted by the Grafana server. Configure TLS settings in the data source if using self-signed certificates.

URL format issues

Symptoms:

Data source test fails immediately
Error indicates invalid URL

Solutions:

Ensure the URL includes the protocol (http:// or https://).
Remove any trailing slashes from the URL.
For Grafana Cloud Profiles, find the correct URL under Manage your stack in your organization settings.
Verify the port number is correct for your deployment.

“Connection refused” or timeout errors

Symptoms:

Data source test fails with network errors
Queries fail with connection errors
Intermittent connectivity issues

Solutions:

Verify network connectivity from the Grafana server to the Pyroscope endpoint.
Check firewall rules allow outbound connections on the required port (default: 4040).
For Kubernetes deployments, verify the service is exposed correctly.
Check if a proxy is required and configure it in the data source settings.

SSL/TLS certificate errors

Symptoms:

Certificate validation failures
SSL handshake errors
“Certificate not trusted” messages

Solutions:

Ensure the system time is correct on the Grafana server.
Verify the certificate is valid and not expired.
For self-signed certificates, enable Skip TLS Verify in the data source settings (not recommended for production).
Add the CA certificate to the Grafana server’s trusted certificates.
Check that intermediate certificates are included in the certificate chain.

Private connectivity or reverse proxy issues

Symptoms:

The data source works from some networks but not others.
Connecting through a reverse proxy, load balancer, or private link fails.

Solutions:

Include the scheme in the Connection URL. When connecting through a private endpoint or reverse proxy, use https:// explicitly rather than a bare host name.
Ensure the reverse proxy or gateway forwards the Pyroscope API paths unchanged. Rewriting or stripping the request path prevents the data source from reaching the backend.
For Grafana Cloud connecting to a Pyroscope backend on a private network, configure Private data source connect.
Verify that any firewall or private link allows outbound HTTPS from Grafana to the Pyroscope endpoint.

Authentication errors

These errors occur when the data source cannot authenticate with the Pyroscope backend.

“Authentication failed” or “Unauthorized”

Symptoms:

Save & test fails with authentication errors
Queries return 401 or 403 status codes
Profile types don’t load

Possible causes and solutions:

Cause	Solution
Invalid credentials	Verify the username and password or API key are correct.
Expired credentials	Generate new credentials and update the data source configuration.
Incorrect authentication type	Ensure you’ve selected the correct authentication method for your Pyroscope deployment (Basic auth, API key, or no authentication for local setups).
Missing authentication headers	For custom authentication, verify the required headers are configured correctly in the data source settings.

Basic authentication not working

Symptoms:

Basic auth credentials are rejected
Works in browser but fails in Grafana

Solutions:

Verify Basic authentication is enabled in the data source settings.
Ensure the username and password are entered correctly (no extra spaces).
Check that the Pyroscope backend is configured to accept Basic authentication.
For Grafana Cloud Profiles, use Basic authentication with your access policy token as the password. Refer to Grafana Cloud access policy and token issues.

API key authentication issues

Symptoms:

API key is rejected
Error indicates invalid token

Solutions:

Verify you’re using the correct API key format for your Pyroscope deployment.
For Grafana Cloud, generate a new access policy token with the profiles:read scope.
Ensure the credentials have the required permissions to read profiling data.
Check if the token has expired and generate a new one if necessary.

Grafana Cloud access policy and token issues

Symptoms:

Authentication fails when connecting a data source to Grafana Cloud Profiles.
Queries return 401 or 403 after a backend migration or stack change.
The data source reads from a different stack or tenant than expected.

Cause:

Grafana Cloud Profiles authenticates with an access policy token rather than a user password. The token must belong to an access policy that includes the profiles:read scope and targets the correct stack. A token with the wrong scope, or one created for a different stack, fails authentication or returns data from the wrong tenant. Tokens and connection URLs are stack-specific and don’t carry over during a migration.

Solutions:

Create or select an access policy that includes the profiles:read scope and whose realm targets the correct stack, then generate a token for that policy.
In the data source, enable Basic auth. Enter the numeric instance ID as the username and the access policy token as the password. Find the instance ID and endpoint under Manage your stack.
Confirm the Connection URL points to the Pyroscope endpoint for the same stack as the token.
After a backend migration or stack change, regenerate the token, then update both the URL and credentials.

Query errors

These errors occur when executing queries against the Pyroscope data source.

“No data” or empty results

Symptoms:

Query executes without error but returns no data
Flame graph shows “No data” message
Metrics graph is empty

Possible causes and solutions:

Cause	Solution
Time range doesn’t contain data	Expand the dashboard time range. Profiling data may not exist for the selected period.
Application not sending profiles	Verify your application is configured to send profiles to Pyroscope. Check the Pyroscope agent or SDK configuration.
Incorrect profile type selected	Select a different profile type from the drop-down menu. Not all profile types are available for all applications.
Label selector too restrictive	Remove or modify label filters to broaden the query. Start without filters to verify data exists.
Wrong service or application selected	Verify the `service_name` label matches your application’s configuration.
Profile type or app not selected	Ensure you’ve selected a profile type or app in the query editor. The query returns no data if neither is selected.

Note
If the data source connection tests successfully but no profiles exist for any query, the problem is usually on the ingestion side rather than in the data source. The data source only reads profiles from Pyroscope; it doesn’t collect or send them. Verify that your application, SDK, or collector, such as Grafana Alloy or an OpenTelemetry Collector, is sending profiles to your Pyroscope backend. For ingestion setup, refer to Configure the client to send profiles.

“Profile type not found” or empty profile type selector

Symptoms:

Profile type drop-down is empty
Error indicates profile type doesn’t exist

Solutions:

Verify the Pyroscope backend is receiving profiles from your applications.
Check that the time range includes periods when profiles were collected.
Test the data source connection to ensure it’s working correctly.
For new deployments, wait a few minutes for initial profile data to be ingested.

Label names or values not loading

Symptoms:

Label selector doesn’t populate
No labels appear in the drop-down menus
Autocomplete doesn’t work

Solutions:

Verify the data source connection is working by testing it in the settings.
Select a profile type first, as labels are loaded based on the selected profile type.
Expand the time range to ensure label data is available.
Check the browser console for JavaScript errors that might indicate issues.

Query syntax errors

Symptoms:

Error message indicates invalid query syntax
Label selector is rejected

Solutions:

Use the correct label selector syntax. Pyroscope uses a syntax similar to Prometheus.
Enclose label values in double quotes: {service_name="my-service"}.
Use valid operators: =, !=, =~, !~.
Escape special characters in regular expression patterns when using =~ or !~.

Example valid queries:

{service_name="my-app"}
{service_name="my-app", env="production"}
{service_name=~"my-app.*"}

Queries return only recent data even though older data is retained

If a query over a long time range returns data only for the most recent days, the query time range likely exceeds a server-side query limit. This is most commonly reported in Grafana Cloud Profiles. Refer to Query time range and retention limits.

Flame graph issues

These errors are specific to the flame graph visualization.

Flame graph not rendering

Symptoms:

Query returns data but flame graph doesn’t display
Flame graph area is blank
Loading indicator never completes

Solutions:

Check that the Query Type is set to include profile data (not metrics only).
Verify the response contains valid profile data using the Query Inspector.
Try refreshing the page or re-running the query.
For large profiles, increase the browser memory limits or reduce the time range.

Flame graph times out or fails to load on large time ranges

Symptoms:

A flame graph renders for a short time range, such as five minutes, but times out for a longer range, such as 30 minutes or more.
Queries over large time ranges are slow or fail intermittently.

Cause:

Rendering a flame graph over a large time range requires the Pyroscope backend to merge many profiles, which increases query time and memory use. Large or high-cardinality ranges can exceed request timeouts before a result returns.

Solutions:

Narrow the time range. Start with a short window, such as five minutes, then widen it gradually to find the largest range that renders reliably.
Add label filters to reduce the volume of profiles the backend must merge.
Confirm the range is within the server-side query limits described in Query time range and retention limits.
For self-managed deployments, review backend resources and query timeout settings if large ranges consistently fail.

Flame graph shows aggregated data without detail

Symptoms:

Flame graph lacks expected granularity
Function names are missing or truncated
Stack traces appear incomplete

Solutions:

Zoom in to a smaller time range to get more detailed profiles.
Verify your application is instrumented to capture full stack traces.
Check if the profiling agent is configured with sufficient sampling rate.
For compiled languages, ensure debug symbols are available.

Unable to interact with flame graph

Symptoms:

Clicking on flame graph elements doesn’t work
Tooltip doesn’t appear
Zoom and filter controls are unresponsive

Solutions:

Verify JavaScript is enabled in your browser.
Check the browser console for JavaScript errors.
Try using a different browser or clearing the browser cache.
Update Grafana to the latest version.

Template variable errors

These errors occur when using template variables with the Pyroscope data source.

Variables return no values

Symptoms:

The variable drop-down is empty
Dependent variables don’t populate

Solutions:

Verify the data source connection is working by testing it in the settings.
For Label and Label value variables, verify that a profile type is selected. Labels and label values are scoped to the selected profile type.
Expand the dashboard time range to ensure the variable query covers a period when profiles were collected.
Confirm the Pyroscope backend is receiving profiles from your applications.

Variables don’t filter queries as expected

Symptoms:

Query results don’t change when the variable value changes
Queries return no data after selecting a variable value

Solutions:

Verify the variable is referenced correctly in the label selector, for example, {service_name="$service"}.
When Multi-value or Include All is enabled, the variable value becomes a regular expression pattern. Use the =~ operator instead of =, for example, {service_name=~"$service"}.
Confirm the label used in the variable exists on the profiling data for the selected profile type.

For more information, refer to Pyroscope template variables.

Profiles Drilldown issues

These errors are specific to the Profiles Drilldown application.

Profiles Drilldown not available

Symptoms:

Profiles Drilldown option doesn’t appear in the menu
Error when trying to access Profiles Drilldown

Solutions:

Verify the Profiles Drilldown plugin is installed and enabled.
For self-managed Grafana, install the plugin from the Grafana plugin catalog.
Restart Grafana after installing the plugin.
Check that you have the required permissions to access the application.

Services or profiles not appearing in Profiles Drilldown

Symptoms:

Profiles Drilldown shows no services
Expected applications are missing

Solutions:

Verify the Pyroscope data source is configured and working.
Check that profiles are being ingested with the correct service_name label.
Ensure the time range includes periods when profiles were collected.
For microservices mode, verify the data source URL points to the correct gateway.

Trace to profiles issues

These errors occur when linking tracing and profiling data.

Trace to profiles is the most configuration-sensitive feature of the data source. Most issues come from an unmet prerequisite rather than a data source misconfiguration.

Prerequisites checklist:

Confirm all of the following before troubleshooting further:

Your application is instrumented with a Pyroscope language SDK for profiling and with OpenTelemetry for tracing.
You installed the OpenTelemetry span-profiling bridge package for your language. This is a separate package from the Pyroscope SDK and the OpenTelemetry SDK.
Profiling uses SDK-based instrumentation. Span profiles aren’t available with eBPF-based collection.
The Tempo data source has Trace to profiles configured and points to your Pyroscope data source.
The tags configured in the Tempo data source are present on the span’s attributes or resources.

For per-language setup instructions, refer to Link traces to profiles.

Span profiles not appearing

Symptoms:

Trace spans don’t show profile links
“Profiles for this span” option is missing

Solutions:

Verify the Tempo data source is configured with Trace to profiles enabled and that it points to your Pyroscope data source.
Confirm the pyroscope.profile.id attribute exists on the span in Tempo. If it’s missing, the bridge package isn’t tagging spans correctly.
Confirm the span_name label exists on the profiling data in Pyroscope. If it’s missing, the bridge package isn’t labeling samples correctly.
Verify that the tags you configured in the Tempo data source match the span’s attributes or resources. If they don’t match, the link doesn’t appear.
Verify profile data exists for the time range of the trace.

For more information, refer to Configure Trace to profiles.

Profile doesn’t match the trace span

Symptoms:

Profile data doesn’t correlate with the trace
Wrong time range in profile

Solutions:

Verify the trace and profile timestamps are synchronized.
Check that the application’s clock is accurate.
Ensure the span_id and trace_id labels are correctly propagated to profiles.

Span profiles aren’t available with eBPF-based collection

Symptoms:

Traces and profiles both exist, but no span links appear.
Profiling is collected with eBPF, for example through Grafana Alloy or Beyla.

Cause:

Span profiling correlates profiling samples with the active trace span, which requires a language SDK and the OpenTelemetry span-profiling bridge. eBPF-based collection can’t associate samples with individual spans, so it doesn’t produce span profiles.

Solutions:

Instrument the application with a supported Pyroscope language SDK and the matching bridge package to enable span profiles.
If you must use eBPF profiling, view profiles by service instead. Per-span correlation isn’t currently supported for eBPF-collected data.

Filter span profiles by span name or trace ID, not span ID

Symptoms:

Queries filtered by span_id return no data, while span_name works.
Correlating profiles to a specific span by span_id is unreliable.

Cause:

The span-profiling bridge labels profiling samples with span_name and trace_id. Depending on the SDK, profiling context may be attached only to the root span of a trace, with child spans inheriting it. span_id is high cardinality and isn’t a dependable correlation label.

Solutions:

Filter profiling data by span_name or trace_id rather than span_id.
Use the Profiles for this span link, which correlates using the pyroscope.profile.id span attribute rather than a label query.

.NET profiler conflict with OpenTelemetry auto-instrumentation

Symptoms:

On .NET, span profiles don’t work when the Pyroscope .NET profiler and OpenTelemetry auto-instrumentation are both enabled.
One profiler fails to attach.

Cause:

The .NET CLR allows only one profiler to attach to a process at a time. The Pyroscope .NET profiler and OpenTelemetry auto-instrumentation each rely on a separate CLR profiler, so they conflict.

Solutions:

Use OpenTelemetry manual instrumentation instead of auto-instrumentation.
Add the Pyroscope.OpenTelemetry package and register the PyroscopeSpanProcessor in your tracing pipeline.
For setup instructions, refer to Span profiles with Traces to profiles for .NET.

Correlate logs with profiles

Symptoms:

You want to navigate from a log line directly to profiling data, but no link exists.

Cause:

Grafana supports Trace to profiles, but there’s no direct logs-to-profiles correlation.

Solutions:

Correlate through shared labels, such as service_name, to move between logs and profiles for the same service and time range.
If your logs contain a trace_id, use a Loki derived field to open the trace, then use Profiles for this span from the trace to reach the profiling data.

Performance issues

These issues relate to slow queries or high resource usage.

Slow queries or timeouts

Symptoms:

Queries take a long time to complete
Requests time out before returning results
Dashboard panels fail to load

Solutions:

Reduce the time range to decrease the amount of data processed.
Add label filters to narrow the query scope.
Increase the Timeout setting in the data source’s Additional settings.
Adjust the Minimal step setting to reduce the number of data points.
For large deployments, ensure the Pyroscope backend has sufficient resources.

High memory usage in browser

Symptoms:

Browser becomes unresponsive when viewing profiles
Memory warnings appear
Page crashes when loading flame graphs

Solutions:

Reduce the time range to load smaller profiles.
Close unnecessary browser tabs to free up memory.
Use a smaller aggregation window for very large profiles.
Consider using Profiles Drilldown for large-scale analysis.

Grafana Cloud-specific issues

These issues are most commonly reported in Grafana Cloud Profiles. Some limits also apply to self-managed Pyroscope deployments, where you can adjust them directly in the Pyroscope configuration.

Query time range and retention limits

Symptoms:

A query over a long time range, such as 30 days, returns data only for the most recent days.
Older profiles appear to be missing even though the retention period hasn’t elapsed.
Users interpret the missing data as data loss or a bug.

Cause:

Pyroscope enforces query limits on the server side that are separate from, and often shorter than, the data retention period. These limits are enforced by the Pyroscope query frontend, not by the Grafana data source, so the data source can’t change or override them.

Limit	Default	Description
`querier.max-query-lookback`	`7d`	How far back from the current time a query can read. Requests for data older than this window don’t fail. Instead, the range is silently truncated to the allowed window, which looks like missing data.
`querier.max-query-length`	`24h`	The maximum time span of a single query. Requests that exceed this span are rejected.

Because retention is typically longer than the query window, profiles can be retained but not queryable. For example, Grafana Cloud Profiles retains data for 14 days on free plans and 30 days on paid plans, while the default query window is 7 days.

Note
The default values shown are the Pyroscope OSS defaults. Grafana Cloud and self-managed deployments may configure different values.

Solutions:

Confirm this is a limit rather than a gap in ingestion by narrowing the time range to the most recent period and verifying that data appears.
To query further back than the query window, raise the limit rather than splitting the query into smaller ranges. Because the window is measured from the current time, splitting a long range into 7-day increments doesn’t return older data.
For self-managed deployments, increase querier.max-query-lookback and, if needed, querier.max-query-length in the Pyroscope limits configuration. Set a value to 0 to disable that limit.
For Grafana Cloud Profiles, contact Grafana Support to adjust these limits for your tenant.
If a single query exceeds querier.max-query-length, reduce the query’s time span so it stays within the configured limit.

Enable debug logging

To capture detailed error information for troubleshooting:

Set the Grafana log level to debug in the configuration file:
ini
```
[log]
level = debug
```
Review logs in /var/log/grafana/grafana.log (or your configured log location).
Look for Pyroscope-specific entries that include request and response details.
Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you’ve tried the solutions in this guide and still encounter issues:

Check the Grafana community forums for similar issues.
Review the Pyroscope GitHub issues for known bugs.
Refer to Pyroscope documentation for additional guidance.
Contact Grafana Support if you’re an Enterprise, Cloud Pro, or Cloud Contracted user.
When reporting issues, include:
- Grafana version
- Pyroscope version (if self-managed)
- Error messages (redact sensitive information)
- Steps to reproduce
- Data source configuration (redact credentials)

Pyroscope data source on Grafana Labs

Configure the Pyroscope data source

Configure the Pyroscope data source

Before you begin

Add or modify a data source

Create a new data source

Update an existing data source

Connection URL

Authentication

Additional settings

Advanced HTTP settings

Private data source connect

Querying

Verify the connection

Provision the data source

YAML provisioning example

Terraform example

Configure Trace to profiles

Configure Trace to profiles

Before you begin

Use a basic configuration

Configure a custom query

Configuration options

Verify the connection

Troubleshoot trace to profiles

No “Profiles for this span” button appears

Button appears but no profile data is shown

Tags don’t match

Query profile data

Query profile data

Profiles Drilldown

Use cases

Give it a try using Grafana Play

Pyroscope query editor

Label selector syntax

Query options

Exemplars

Heatmap

UTF-8 label names

Query examples

Profiles query results

Metrics query results

Next steps

Pyroscope template variables

Pyroscope template variables

Query variable types

Create a query variable

Query variable examples

Use variables in queries

Where you can use template variables

Filters variable

Related resources

Troubleshoot Pyroscope data source issues

Troubleshoot Pyroscope data source issues

Connection and configuration errors

“Data source is not working” or connection fails

URL format issues

“Connection refused” or timeout errors

SSL/TLS certificate errors

Private connectivity or reverse proxy issues

Authentication errors

“Authentication failed” or “Unauthorized”

Basic authentication not working

API key authentication issues

Grafana Cloud access policy and token issues

Query errors

“No data” or empty results

“Profile type not found” or empty profile type selector

Label names or values not loading

Query syntax errors

Queries return only recent data even though older data is retained

Flame graph issues

Flame graph not rendering

Flame graph times out or fails to load on large time ranges

Flame graph shows aggregated data without detail

Unable to interact with flame graph

Template variable errors

Variables return no values

Variables don’t filter queries as expected

Profiles Drilldown issues