Graphite data source on Grafana Labs

Configure the Graphite data source

Tue, 29 Sep 2026 22:18:37 +0000

Configure the Graphite data source

This document provides instructions for configuring the Graphite data source and explains available configuration options. For general information on managing data sources, refer to Data source management.

Before you begin

You must have the Organization administrator role to configure the Graphite data source. Organization administrators can also configure the data source via YAML with the Grafana provisioning system.
Grafana comes with a built-in Graphite data source plugin, eliminating the need to install a plugin.
Familiarize yourself with your Graphite security configuration and gather any necessary security certificates and client keys.

Add the Graphite data source

To configure basic settings for the data source, complete the following steps:

Click Connections in the left-side menu.
Click Add new connection
Type Graphite in the search bar.
Select the Graphite data source.
Click Add new data source in the upper right.

Grafana takes you to the Settings tab, where you set up your Graphite configuration.

Configuration options in the UI

Following is a list of configuration options for Graphite.

Setting	Description
Name	The display name for the data source. This is how you reference it in panels and queries. For example, `graphite-1` or `graphite-metrics`.
Default	When enabled, sets this data source as the default for dashboard panels. Grafana automatically selects it when you create new panels.

HTTP:

Setting	Description
URL	Sets the HTTP protocol, IP, and port of your `graphite-web` or `graphite-api` installation. Because the access method is set to Server, the URL must be accessible from the Grafana backend.
Allowed cookies	By default, Grafana removes forwarded cookies. Specify cookie names here to allow them to be forwarded to the data source.
Timeout	Sets the HTTP request timeout in seconds.

Auth:

Setting	Description
Basic Auth	Toggle on to enable basic authentication to the data source.
User	Sets the username used for basic authentication.
Password	Enter the password used for basic authentication.
With Credentials	Toggle on to include cookies and authentication headers in cross-origin requests.
TLS Client Auth	Toggle on to enable TLS client authentication (both server and client are verified).
ServerName	The server name used to verify the hostname on the certificate returned by the server.
Client Cert	Client certificate generated by a Certificate Authority (CA) or self-signed.
Client Key	Private key used to encrypt communication between the client and server. Also generated by a CA or self-signed.
With CA Cert	Toggle on to authenticate with a CA certificate.
CA Cert	CA certificate used to validate the server certificate.
Skip TLS Verify	Toggle on to bypass TLS certificate validation. Not recommended unless necessary or for testing purposes.
Forward OAuth Identity	Toggle on to forward the user’s upstream OAuth identity to the data source. Grafana includes the access token in the request.

Custom HTTP Headers:

Pass along additional information and metadata about the request or response.

Setting	Description
Header	Add a custom header. This allows custom headers to be passed based on the needs of your Graphite instance.
Value	The value of the header.

Graphite details:

Setting	Description
Version	Select your Graphite version from the drop-down. This controls which functions are available in the Graphite query editor. Use `1.1.x` for Grafana Cloud Graphite.
Graphite backend type	Select the Graphite backend type. Choosing `Metrictank` enables additional features like query processing metadata. (`Metrictank` is a multi-tenant time series engine compatible with Graphite.) Use `Default` for Grafana Cloud Graphite.
Rollup indicator	Toggle on to display an info icon in panel headers when data aggregation occurs. Only available when `Metrictank` is selected.

Label mappings:

Label mappings are the rules you define to tell Grafana how to pull pieces of the Graphite metric path into Loki labels when switching data sources. They are currently only supported between Graphite and Loki queries.

When you change your data source from Graphite to Loki, your queries are automatically mapped based on the rules you define. To create a mapping, specify the full path of the metric and replace the nodes you want to map with label names, using parentheses. The corresponding label values are extracted from your Graphite query during the data source switch.

Grafana automatically maps all Graphite tags to labels, even if you haven’t defined explicit mappings. When using matching patterns with {} (for example, metric.{a,b}.value), Grafana converts them to the Loki regular expression matching syntax. If your queries include functions, Graphite extracts the relevant metrics and tags, then matches them against your mappings.

For example, given the mapping servers.(cluster).(server).*:

Graphite Query	Mapped to Loki Query
`alias(servers.west.001.cpu,1,2)`	`{cluster="west", server="001"}`
`alias(servers..{001,002}.,1,2)`	`{server=~"(001\|002)"}`
`interpolate(seriesByTag('foo=bar', 'server=002'), inf)`	`{foo="bar", server="002"}`

Setting	Description
Private data source connect	Only for Grafana Cloud users. Establishes a private, secured connection between a Grafana Cloud stack and data sources within a private network. Use the drop-down to locate the PDC URL. For setup instructions, refer to Private data source connect (PDC) and Configure PDC. Click Manage private data source connect to open your PDC connection page and view your configuration details.

After configuring your Graphite data source options, click Save & test at the bottom to test the connection. You should see a confirmation dialog box that says:

Data source is working

Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for lists of common configuration options and JSON data options, refer to Provisioning data sources.

YAML example

The following example provisions a Graphite data source:

apiVersion: 1

datasources:
  - name: Graphite
    type: graphite
    access: proxy
    url: http://localhost:8080
    jsonData:
      graphiteVersion: '1.1'
      graphiteType: default

The following jsonData options are supported:

Option	Type	Description
`graphiteVersion`	string	The Graphite version. Supported values are `0.9`, `1.0`, and `1.1`. Defaults to `1.1`.
`graphiteType`	string	The Graphite backend type. Supported values are `default` and `metrictank`.
`rollupIndicatorEnabled`	boolean	Displays an info icon in panel headers when data is aggregated. Applies only to `metrictank`.

Provision with Terraform

You can provision the Graphite data source using Terraform with the Grafana Terraform provider.

For more information about provisioning resources with Terraform, refer to the Grafana as code using Terraform documentation.

Terraform example

The following example provisions a Graphite data source:

terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 2.0.0"
    }
  }
}

provider "grafana" {
  url  = "<YOUR_GRAFANA_URL>"
  auth = "<YOUR_SERVICE_ACCOUNT_TOKEN>"
}

resource "grafana_data_source" "graphite" {
  type = "graphite"
  name = "Graphite"
  url  = "http://localhost:8080"

  json_data_encoded = jsonencode({
    # Graphite version: 0.9, 1.0, or 1.1
    graphiteVersion = "1.1"
    # Backend type: default or metrictank
    graphiteType = "default"
  })
}

Terraform example with basic authentication

The following example provisions a Graphite data source with basic authentication:

resource "grafana_data_source" "graphite_auth" {
  type                = "graphite"
  name                = "Graphite"
  url                 = "http://localhost:8080"
  basic_auth_enabled  = true
  basic_auth_username = "<USERNAME>"

  json_data_encoded = jsonencode({
    graphiteVersion = "1.1"
    graphiteType    = "default"
  })

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = "<PASSWORD>"
  })
}

Replace the following placeholders:

<YOUR_GRAFANA_URL>: Your Grafana instance URL (for example, https://your-org.grafana.net for Grafana Cloud)
<YOUR_SERVICE_ACCOUNT_TOKEN>: A service account token with data source permissions
<USERNAME>: The username for basic authentication
<PASSWORD>: The password for basic authentication

Graphite query editor

Tue, 29 Sep 2026 22:18:37 +0000

Graphite query editor

Grafana includes a Graphite-specific query editor to help you build queries. The query editor helps you quickly navigate the metric space, add functions, and change function parameters. It supports a variety of Graphite queries, including complex nested queries, through the use of query references.

For general documentation on querying data sources in Grafana, refer to Query and transform data.

Query editor elements

The query editor consists of the following elements:

Series: A series in Graphite is a unique time-series dataset, represented by a specific metric name and timestamped values. Click select metric to choose a metric from the drop-down.
Functions: Graphite uses functions to manipulate data. Click the + sign to view a list of functions in the drop-down. You can add multiple functions to a query.

To view the raw query, click the Pencil icon in the upper right. Click the Pencil icon again to continue adding series and functions.

Choose metrics to query

Click select metric to browse the available metrics. You can navigate using your mouse or arrow keys. You can also select a wildcard.

Functions

Click the + sign next to Function to add a function from the drop-down. You can also search by typing the first few letters of the function name.

After selecting a function, Grafana adds it to your query and automatically places your cursor in the first parameter field.

To edit a parameter, click it to open an editable text box.

To remove a function, click it, then click the X icon that appears above it.

Some functions like aliasByNode support an optional second argument. To add this argument, hover your mouse over the argument and a dialog box opens. To remove the second optional parameter, click it to delete it.

Refer to Functions in the Graphite documentation for more information.

Warning
Some functions accept a second argument, which can itself be another function that returns a series. If you need to add a second argument that is a function, Grafana recommends using a series reference from a second query instead of embedding the function directly.

Currently, the query editor does not support parsing a second function argument when switching between the query builder and the code editor.

Sort labels

If the same labels appear on multiple graphs, they may be sorted differently and assigned different colors.

To ensure consistent sorting and coloring, use the sortByName() function to order labels alphabetically.

Modify metric names in tables or charts

Use alias functions, such as aliasByNode() or aliasSub(), to change metric names on Grafana tables or graphs.

Consolidate data points

Grafana consolidates all Graphite metrics so that Graphite doesn’t return more data points than there are pixels in the graph. By default, Grafana consolidates data points using the avg function. To control how Graphite consolidates metrics, use the Graphite consolidateBy() function.

Note
Grafana calculates legend summary values like max, min, and total on the client side, after data has been calculated. Depending on the consolidation function used, only one or two of these values may be accurate at the same time.

Combine time series

To combine time series, click Combine in the Functions list.

Select and explore data with tags

Graphite supports tagged series, so you can filter data by tag key-value pairs instead of only by metric path.

When exploring data, previously selected tags filter the remaining result set. To select data, use the seriesByTag function, which takes tag expressions (=, !=, =~, !=~) to filter the time series.

The Grafana query builder does this for you automatically when you select a tag.

Note
Regular expression searches can be slow on high-cardinality tags, so try to use other tags to reduce the scope first. To help reduce the results, start by filtering on a particular name or namespace.

Nested queries

Grafana lets you reference one query from another using its query letter, similar to how cell references work in a spreadsheet.

For example, if you add a second query and want to build on the results of query A, you can reference it using #A.

This approach lets you build compound or nested queries so your panels are more flexible and easier to manage.

Use wildcards to make fewer queries

To display multiple time series on the same graph, use wildcards in your query to return all matching series at once.

For example, to monitor CPU utilization across a variety of metrics, you can use a single query like cpu.percent.*.g to retrieve all matching time series. This approach is more efficient than writing separate queries for each series, such as cpu.percent.user.g, cpu.percent.system.g, and others, which would result in multiple queries to the data source.

Example queries

The following examples show common query patterns. Each one combines a metric path with one or more Graphite functions.

Rename series for readable legends

Wildcard queries return series whose names are full metric paths. Use aliasByNode() to display a specific path segment instead. The following query labels each series with the second path node, for example the host name:

aliasByNode(servers.*.cpu.percent.user, 1)

Aggregate across many series

Combine matching series into a single line with an aggregation function. The following query sums request counts across all hosts:

sumSeries(servers.*.requests.count)

Use averageSeries(), maxSeries(), or minSeries() when you want the average, maximum, or minimum instead of the sum.

Convert a counter to a per-second rate

For ever-increasing counters, use perSecond() to plot the rate of change:

perSecond(servers.web01.nginx.requests.count)

Summarize into time buckets

Use summarize() to roll data into fixed intervals, which is useful for bar charts and reducing noise. The following query totals errors in five-minute buckets:

summarize(sumSeries(servers.*.errors.count), '5min', 'sum', false)

Control consolidation

When a panel shows a long time range, Grafana consolidates points. Use consolidateBy() to choose how points combine so peaks aren’t averaged away:

consolidateBy(servers.web01.response.max, 'max')

Show the top series

Use a filter function to reduce a large wildcard result to the most significant series. The following query keeps the five hosts with the highest peak CPU:

highestMax(servers.*.cpu.percent.user, 5)

Select series by tag

Use seriesByTag() to select data by tag expressions instead of a metric path:

seriesByTag('name=requests.count', 'env=production')

Build on another query

Reference a query by its letter to reuse its result. If query A returns per-host request counts, query B can sum them:

sumSeries(#A)

Apply annotations

Annotations overlay rich event information on top of graphs. Graphite supports two ways to query annotations: a regular metric query and a Graphite events query.

For details on both modes, example queries, and how to create events in Graphite, refer to Graphite annotations.

Integration with Loki

When you change the data source to Loki in Explore, your Graphite queries are automatically converted to Loki queries. Loki label names and values are extracted based on the mapping information defined in your Graphite data source configuration. Grafana automatically transforms queries that use tags with seriesByTag() without requiring additional setup.

Graphite template variables

Tue, 29 Sep 2026 22:18:37 +0000

Graphite template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana lists these variables in drop-down selection boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables.

For an introduction to templating and template variables, refer to the Templating and Add and manage variables documentation.

To view an example templated dashboard, refer to Graphite Templated Nested dashboard.

Use query variables

Graphite populates Query variables. You can still use other variable types, such as Custom, Interval, and Text box, on a dashboard that queries Graphite. Those types don’t send a query to Graphite.

To create a Graphite query variable, choose the query type from the Select query type drop-down and enter your query in the Query field. Grafana supports three query types for Graphite variables:

Query type	Description	Example
Default Query	Runs a metric-find query and returns metric nodes, or tag keys and values when you use `tags()` or `tag_values()`.	`tag_values(apps.*.requests.count, app)`
Value Query	Runs the query through the render endpoint and returns the numeric values of the resulting series.	`groupByNode(apps.*.requests.count, 1, 'sum')`
Metric Name Query	Runs the query through the render endpoint and returns the series names of the result.	`highestAverage(apps.*.requests.count, 5)`

Choose a variable syntax

The Graphite data source supports two variable syntaxes for use in the Query field.

Grafana allows two ways to reference variables in a query:

Syntax	Example
`$varname`	`apps.frontend.$server.requests.count`
`${varname}`	`apps.frontend.${server}.requests.count`

Shorthand syntax ($varname) is convenient for standalone path segments but doesn’t work when the variable is adjacent to other characters (for example, cpu$coreLoad).
Full syntax (${varname}) is more flexible and works in any part of the string, including embedded within words.

Choose the format that best fits the structure of your Graphite metric path.

Use tag variables

Grafana supports tag-based variables for Graphite, allowing you to dynamically populate drop-downs based on tag keys and values in your metric series. To do this, use the Graphite functions tags() and tag_values() in your variable queries.

Query	Description
`tags()`	Returns a list of all tag keys in the Graphite database.
`tags(server=~backend*)`	Returns tag keys only from series that match the provided filter expression.
`tag_values(server)`	Returns all values for the specified tag key.
`tag_values(server, server=~backend*)`	Returns tag values for a given key, filtered to only those that appear in matching series.

You can use multiple filter expressions, and those expressions can include other Grafana variables. For example:

tag_values(server, server=~backend*, app=~${apps:regex})

This query returns all server tag values from series where the server tag matches backend* and the app tag matches the regular expression-filtered values from another variable, ${apps}.

For details, refer to the Graphite docs on the autocomplete API for tags.

Using regular expression formatting and the equal tilde operator =~:

server=~${servers:regex}

This query tells Grafana to format the selected values in the servers variable as a regular expression (for example, (server1|server2) if two servers are selected).

For more information, refer to Advanced variable format options.

Filter with multiple expressions

When using multi-value variables in tag queries, append ${var:regex} to the variable name to apply regular expression formatting.

tag_values(server, app=~${apps:regex})

This query returns only series where the app tag matches the selected values in ${apps}, formatted as a regular expression. =~ is the regular expression operator.

Non-tag queries use the default glob formatting for multi-value variables.

Use other query variables

When writing queries, use the Default Query type to retrieve dynamic metric values.

For example, the query prod.servers.* populates the variable with all values that exist at the wildcard position (*).

Note that the results include only the values found at the last level of the query path.

To return full metric paths that match your query, use the expand() function:

expand(*.servers.*)

Compare expanded and non-expanded metric search results

When querying Graphite metrics in Grafana, you can choose between using an expanded or non-expanded query:

Expanded queries (using the expand() function) return the full metric paths that match your query.
Non-expanded queries return only the last segment of each matching metric path, which limits your ability to extract or filter based on deeper parts of the metric name.

Expanded queries are especially useful when working with regular expressions to match or extract specific parts of the metric path.

Suppose your Graphite database contains the following metrics:

prod.servers.001.cpu
prod.servers.002.cpu
test.servers.001.cpu

The following table illustrates the difference between expanded and non-expanded queries:

Non-expanded query	Results	Expanded query	Expanded results
`*`	`prod`, `test`	`expand(*)`	`prod`, `test`
`*.servers`	`servers`	`expand(*.servers)`	`prod.servers`, `test.servers`
`test.servers`	`servers`	`expand(test.servers)`	`test.servers`
`.servers.`	`001`, `002`	`expand(.servers.)`	`prod.servers.001`, `prod.servers.002`, `test.servers.001`
`test.servers.*`	`001`	`expand(test.servers.*)`	`test.servers.001`
`.servers..cpu`	`cpu`	`expand(.servers..cpu)`	`prod.servers.001.cpu`, `prod.servers.002.cpu`, `test.servers.001.cpu`

Note
A non-expanded query works like an expanded query but returns only the final segment of each matched metric.

Grafana also supports nested variables, which allow you to reference other variables in a query.

For example:

apps.$app.servers.*

This query uses the selected value of the $app variable to dynamically filter the metric path. The variable $app contains one or more application names and servers.* matches all servers for the given application.

Filter query variable results with `__searchFilter`

Grafana provides the variable __searchFilter, which you can use to dynamically filter query results based on what the user types into the variable drop-down. When the drop-down is empty or blank, __searchFilter defaults to *, which means it returns all possible values. If you type a string, Grafana replaces __searchFilter with that input.

To use __searchFilter as part of the query field to enable searching for server while the user types in the drop-down select box:

Query:

apps.$app.servers.$__searchFilter

TagValues:

tag_values(server, server=~${__searchFilter:regex})

Graphite annotations

Tue, 29 Sep 2026 22:18:37 +0000

Graphite annotations

Annotations overlay event data on your dashboard graphs, helping you correlate events with metrics. You can use Graphite as a data source for annotations to display events such as deployments, incidents, or maintenance windows on your visualizations.

For general information about annotations in Grafana, refer to Annotate visualizations.

Common use cases

Graphite annotations help you explain changes in your metrics by marking when related events happened. Common use cases include:

Deployments: Mark each release so you can see whether a deployment changed latency, error rate, or throughput.
Incidents and outages: Mark when incidents start and resolve to correlate them with metric anomalies.
Maintenance windows: Mark planned maintenance so expected dips aren’t mistaken for problems.
Configuration changes: Mark when you change a feature flag or setting to track its effect over time.
Scaling events: Mark when you add or remove capacity to explain shifts in load or performance.

Each use case maps to one of the two query modes described in this page: a metric query when an existing metric already marks the event, or an events query when you record discrete events in Graphite.

Before you begin

Before creating Graphite annotations, ensure you have:

A configured Graphite data source in Grafana.
Metrics or events in Graphite that represent the events you want to annotate.
Read access to the Graphite instance.

Annotation query modes

Graphite supports two ways to query annotations. You configure both in the annotation query editor:

Mode	Field	Description
Metric query	Graphite Query	Runs a standard Graphite metric query. Each returned data point with a non-zero value becomes an annotation, and the series name becomes the title.
Events query	Graphite events tags	Queries the Graphite events API and filters by one or more event tags. Each matching event becomes an annotation.

Use the metric query mode when a metric is non-zero only at the moments you want to mark. A counter that stays above zero after the first event marks every interval, because Grafana creates an annotation for every non-zero point. Use the events query mode when you record discrete events in Graphite and tag them.

Create an annotation query

To add a Graphite annotation to your dashboard:

Navigate to the dashboard you want to update and click Edit.
Click the Add new element icon (blue plus sign).
Click Annotation query.
Enter a name for the annotation query.
If you don’t want to use the annotation query right away, clear the Enabled checkbox.
Select a color for the annotation event markers.
Select an option in the Show annotation controls in drop-down list to control where on the dashboard the annotation is displayed.
Select an option in the Show in drop-down list to control the panels in which the annotation is displayed.
Click Open query editor to open the Annotation Query dialog box.
Select the Graphite data source from the Data source drop-down list.
Configure one of the annotation query modes:
- In the Graphite Query field, enter a metric query, for example statsd.application.counters.*.count.
- In the Graphite events tags field, enter one or more event tags, for example deploy.
(Optional) Click Test annotation query to ensure that the query is working properly.
Click Close when you’ve completed the query setup.
Click Save.
(Optional) Enter a description of the changes you’ve made.
Click Save.
Click Exit edit.

Note
Enter a value in either the Graphite Query field or the Graphite events tags field, not both. If you provide a metric query, Grafana uses the metric query mode and ignores the tags.

Metric query annotations

In metric query mode, Grafana runs the value in the Graphite Query field as a standard Graphite query over the dashboard time range. Grafana converts the results into annotations using these rules:

Every data point with a non-zero value creates an annotation. Data points with a value of zero or null are skipped.
The annotation timestamp is the timestamp of the data point.
The annotation title is the series name returned by Graphite.

Because every non-zero data point creates an annotation, use a metric that only produces values at the moments you want annotated. Metrics that report continuous values produce an annotation at every step interval and flood your dashboard.

Example metric queries

The following examples show common ways to turn a metric into annotations.

Deployment markers

Record deployments as Graphite events and query them by tag. Refer to Events query annotations.

If you only have a counter, query the increase between points. nonNegativeDerivative() is zero while the counter is flat and non-zero only when it goes up, so annotations appear at each deploy:

nonNegativeDerivative(deploys.myservice.count)

Service restarts

Use the same pattern for restarts. Query a metric that is non-zero only when a restart happens, or record restarts as events:

nonNegativeDerivative(stats.myservice.restart.count)

Error spikes above a threshold

Because Grafana skips zero and null data points, use removeBelowValue() to annotate only the moments a metric crosses a threshold. The following query annotates every interval where the error count is 100 or greater:

removeBelowValue(stats.myservice.errors.count, 100)

Values below 100 become null and are skipped, so annotations appear only during error spikes.

Host down

Annotate while a host is down by inverting an up metric so that a down state produces a non-zero value:

offset(scale(hosts.web01.up, -1), 1)

This query returns 1 when up is 0. Grafana skips the zero points from when the host is up, and creates an annotation for each interval the host stays down.

Events query annotations

In events query mode, Grafana queries the Graphite events API and filters events by the tags you provide in the Graphite events tags field. Grafana converts each matching event into an annotation using these rules:

The annotation timestamp is the event’s when value.
The annotation title is the event’s what value.
The tags you query with are attached to the annotation.

Enter multiple tags separated by a space. Grafana sends those tags to the Graphite events API without set=union, so an event must include every tag.

You can filter events in the following ways:

Value	Result
Empty	Returns all events in the dashboard time range.
A single tag	Returns events that include that tag, for example `deploy`.
Multiple tags	Returns events that include every tag. `deploy myservice` matches events tagged with both.
A wildcard	Returns events whose tags match the pattern, for example `web*`.

Create events in Graphite

Before you can query events, record them in Graphite. Graphite exposes an HTTP endpoint for creating events:

curl -X POST http://<GRAPHITE_HOST>/events/ \
  -H "Content-Type: application/json" \
  -d '{
    "what": "Deployed myservice v2.5.0",
    "tags": ["deploy", "myservice"],
    "data": "Deployed new feature flags and performance improvements"
  }'

Replace <GRAPHITE_HOST> with your Graphite server address. The what field becomes the annotation title, and the tags field is what you filter on in the Graphite events tags field.

Example events queries

The following examples show what to enter in the Graphite events tags field for common scenarios.

Goal	Value	Result
Show all events	Empty	Displays every event in the dashboard time range.
Show all deployments	`deploy`	Displays events tagged `deploy`.
Deployments for one service	`deploy myservice`	Displays events tagged with both `deploy` and `myservice`.
All production events	`prod*`	Displays events whose tags match the `prod*` pattern.
Incidents	`incident`	Displays events tagged `incident`.

Correlate multiple event types

To compare different kinds of events on the same dashboard, add more than one annotation query and give each a distinct color. For example:

Add a Deployments annotation query with the tag deploy.
Add an Incidents annotation query with the tag incident.
Add a Maintenance annotation query with the tag maintenance.

Each query renders in its own color, so you can see how deployments, incidents, and maintenance windows line up with your metrics.

Use template variables in annotations

You can use template variables in both annotation query modes to filter annotations based on dashboard variable selections. Grafana resolves the variables at query time using the current dashboard variable values.

Metric query with a variable:

nonNegativeDerivative(deploys.$service.count)

Events query with a variable:

deploy $environment

For more information about template variables, refer to Graphite template variables.

End-to-end example: correlate deployments with latency

This walkthrough shows how to combine a Graphite events annotation query with a metric panel so you can see whether deployments affect request latency. The result is a time series panel that plots latency with a vertical marker at each deployment.

Before you start

Ensure you have the following:

A Graphite data source configured in Grafana.
A latency metric in Graphite, for example stats.timers.myservice.request.latency.p95.
Permission to create events in Graphite.

Step 1: Record a deployment event

When your pipeline deploys the service, post an event to Graphite tagged deploy and myservice:

curl -X POST http://<GRAPHITE_HOST>/events/ \
  -H "Content-Type: application/json" \
  -d '{
    "what": "Deployed myservice v2.5.0",
    "tags": ["deploy", "myservice"],
    "data": "Rolled out connection pool changes"
  }'

Replace <GRAPHITE_HOST> with your Graphite server address.

Step 2: Build the latency panel

Create a dashboard and add a Time series panel.
Select the Graphite data source.
In the query editor, build the latency query, for example stats.timers.myservice.request.latency.p95.
Confirm the panel plots your latency data over the selected time range.

Step 3: Add the deployment annotation query

In the dashboard toolbar, click the Add new element icon (blue plus sign), then click Annotation query.
Enter Deployments as the name and select a color that stands out against the latency line.
Click Open query editor and select the Graphite data source.
In the Graphite events tags field, enter deploy myservice.
(Optional) Click Test annotation query to confirm the query returns events.
Click Close, then Save.

Step 4: Interpret the result

The panel now shows a vertical marker at each deployment. Hover over a marker to see the deployment title from the event’s what value. If latency rises immediately after a marker, the deployment is a likely cause. If latency is steady across markers, deployments aren’t degrading performance.

To extend the dashboard, add a second annotation query tagged incident in a different color, so you can tell whether latency changes follow deployments or unrelated incidents.

Troubleshoot annotation issues

The following section addresses common issues you might encounter when using Graphite annotations.

Annotations don’t appear

Possible causes and solutions:

Cause	Solution
Time range doesn’t include the events	Expand the dashboard time range to include the event timestamps.
Metric query returns only zero values	Annotations are created only for non-zero data points. Verify the metric produces non-zero values at event times.
No matching events	Confirm events exist in Graphite for the supplied tags and time range.
Both fields are populated	Clear one of the fields. Grafana uses the metric query mode whenever the Graphite Query field has a value.

Annotation title is empty

In events query mode, Grafana uses the event’s what value as the title. Ensure the what field is populated when you create events in Graphite.

For more troubleshooting guidance, refer to Troubleshoot Graphite data source issues.

Graphite alerting

Tue, 29 Sep 2026 22:18:37 +0000

Graphite alerting

You can use Grafana Alerting with Graphite to create alerts based on your time series data. This lets you monitor metrics, detect anomalies, and receive notifications when specific conditions are met.

For general information about Grafana Alerting, refer to Grafana Alerting.

Before you begin

Before creating alerts with Graphite, ensure you have:

A Graphite data source configured in Grafana. Refer to Configure the Graphite data source.
Permission to create alert rules.
An understanding of the metrics you want to monitor.

Supported queries

Grafana Alerting evaluates any Graphite query that returns time series data. Grafana runs the query on the backend, reduces each series to a single value, and compares that value against the condition you define.

Query pattern	Alerting support	Notes
Single metric path	Yes	The most predictable option for alerting.
Aggregated series	Yes	Use `sumSeries()`, `averageSeries()`, or similar to combine matching series.
Wildcard metric path	Yes	Returns one series per match. Reduce or aggregate before the condition.
Tag query with `seriesByTag()`	Yes	Requires Graphite 1.1 or later.

Create an alert rule

To create an alert rule using Graphite:

Navigate to Alerting > Alert rules.
Click New alert rule.
Enter a name for the alert rule.
In the query section, select your Graphite data source.
Build your query:
- Click Select metric to choose the metric path, or click the pencil icon to write the query in code.
- Add Graphite functions to aggregate or shape the series, for example sumSeries() or summarize().
Add a Reduce expression to convert the series to a single value, for example Last or Mean.
Add a Threshold expression that defines when the alert fires, for example when the value is above a limit.
Set the evaluation interval and pending period.
Configure notifications and labels.
Click Save rule.

For detailed instructions, refer to Create a Grafana-managed alert rule.

Example alert queries

The following examples show common alerting scenarios with Graphite. Each query returns a single series that the alert condition reduces to one value.

Alert on high CPU usage

Average CPU usage across every core on a host and alert when it exceeds 90%:

averageSeries(servers.web01.cpu.*.percent.user)

Condition: Reduce with Mean, then alert when the value is above 90.

Alert on low free disk space

Monitor available disk space and alert when it drops below 10 GB:

servers.web01.disk.root.free_bytes

Condition: Reduce with Last, then alert when the value is below 10737418240.

Alert on high error rate

Combine error counts across a service and alert on spikes:

summarize(sumSeries(stats.myservice.errors.count), '5min', 'sum', false)

Condition: Reduce with Last, then alert when the value is above 100.

Alert on request latency

Monitor 95th percentile latency and alert when it exceeds 500 ms:

stats.timers.myservice.request.latency.p95

Condition: Reduce with Mean, then alert when the value is above 500.

Alert across many hosts

Use a wildcard to evaluate every matching host and alert when any host crosses the threshold:

aliasByNode(servers.*.cpu.percent.user, 1)

Because the wildcard returns one series per host, configure the alert rule to evaluate each series separately so you get a distinct alert instance for each host that breaches the threshold.

Limitations

When using Graphite with Grafana Alerting, be aware of the following limitations.

Template variables not supported

Alert queries can’t contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like $server or $environment aren’t resolved.

If your dashboard query uses template variables, create a separate query for alerting with hard-coded values.

Nested queries not supported

Alert queries can’t reference another query by letter, such as #A. Build a self-contained query for each alert rule instead of chaining query references.

Consolidation affects results

Grafana consolidates data points so that a query doesn’t return more points than needed. Consolidation can change the value the alert evaluates. Use consolidateBy() or summarize() to control how Graphite aggregates points, and match the interval to your evaluation frequency.

Metrics that stop reporting

When a metric stops reporting, the query returns no data rather than a zero value, so a threshold alone can’t detect the gap. To alert when a service stops sending metrics, set the alert rule’s No Data state to Alerting. Configure this behavior in the alert rule under the evaluation options. For more information, refer to Configure no data and error handling.

Best practices

Follow these best practices when creating Graphite alerts:

Return a single series: Aggregate wildcard queries with sumSeries() or averageSeries(), or evaluate each series separately when you want per-target alerts.
Control consolidation: Use consolidateBy() or summarize() so the evaluated value matches what you expect.
Test queries first: Verify your query returns the expected values in Explore before creating an alert.
Set realistic thresholds: Base thresholds on historical data patterns to avoid false positives.
Use meaningful names: Give alert rules descriptive names that indicate what they monitor.

Next steps

Build queries to explore your metrics before creating alerts.
Troubleshoot Graphite data source issues if alerts aren’t firing as expected.

Troubleshoot Graphite data source issues

Tue, 29 Sep 2026 22:18:37 +0000

Troubleshoot Graphite data source issues

This document provides solutions for common issues you might encounter when using the Graphite data source.

Connection issues

Use the following troubleshooting steps to resolve connection problems between Grafana and your Graphite server.

Data source test fails with “Unable to connect”:

If the data source test fails, verify the following:

The URL in your data source configuration is correct and accessible from the Grafana server.
The Graphite server is running and accepting connections.
Any firewall rules or network policies allow traffic between Grafana and the Graphite server.
If using TLS, ensure your certificates are valid and properly configured.

To test connectivity, run the following command from the Grafana server:

curl -v <GRAPHITE_URL>/render

Replace <GRAPHITE_URL> with your Graphite server URL. A successful connection returns a response from the Graphite server.

Authentication errors:

If you receive 401 or 403 errors:

Verify your Basic Auth username and password are correct.
Ensure the With Credentials toggle is enabled if your Graphite server requires cookies for authentication.
Check that your TLS client certificates are valid and match what the server expects.

For detailed authentication configuration, refer to Configure the Graphite data source.

Deprecation notice about browser access mode:

If the data source configuration page shows a deprecation notice about browser access mode:

The data source is configured to use browser (direct) access, which is deprecated and slated for removal.
Edit the data source so it uses Server (proxy) access instead. Server access routes requests through the Grafana backend, which avoids browser cross-origin restrictions.
Click Save & test to re-test the connection after switching.

Query issues

Use the following troubleshooting steps to resolve problems with Graphite queries.

No data returned:

If your query returns no data:

Verify the metric path exists in your Graphite server by testing directly in the Graphite web interface.
Check that the time range in Grafana matches when data was collected.
Ensure wildcards in your query match existing metrics.
Confirm your query syntax is correct for your Graphite version.

HTTP 500 errors with HTML content:

Graphite-web versions before 1.6 return HTTP 500 errors with full HTML stack traces when a query fails. If you see error messages containing HTML tags:

Check the Graphite server logs for the full error details.
Verify your query syntax is valid.
Ensure the requested time range doesn’t exceed your Graphite server’s capabilities.
Check that all functions used in your query are supported by your Graphite version.

Parser errors in the query editor:

If the query editor displays parser errors:

Check for unbalanced parentheses in function calls.
Verify that function arguments are in the correct format.
Ensure metric paths don’t contain unsupported characters.

Queries don’t map correctly when switching from Graphite to Loki:

Label mappings apply only when you switch between Graphite and Loki queries. If mapped queries produce unexpected labels or no results:

Verify your label mappings are defined in the Graphite data source configuration.
Remember that Grafana automatically maps all Graphite tags to labels, so you only need explicit mappings for non-tag metric nodes.
Check that multi-value patterns such as {001,002} map to the expected regular expression, for example {server=~"(001|002)"}.

For details, refer to the label mappings section in Configure the Graphite data source.

For query syntax help, refer to Graphite query editor.

Version and feature issues

Use the following troubleshooting steps to resolve problems related to Graphite versions and features.

Functions missing from the query editor:

If expected functions don’t appear in the query editor:

Verify the correct Graphite version is selected in the data source configuration.
The available functions depend on the configured version. For example, tag-based functions require Graphite 1.1 or later.
If using a custom Graphite installation with additional functions, ensure the version setting matches your server.

Tag-based queries not working:

If seriesByTag() or other tag functions fail:

Confirm your Graphite server is version 1.1 or later.
Verify the Graphite version setting in your data source configuration matches your actual server version.
Check that tags are properly configured in your Graphite server.

Template variable issues

Use the following troubleshooting steps when dashboard variables don’t behave as expected.

Variable not interpolated in a query:

When a variable is adjacent to other characters, use the full syntax ${varname} instead of $varname. For example, use cpu.${core}.load rather than cpu.$coreLoad.
Confirm the variable name matches exactly, including case.

Multi-value variable returns no data in tag queries:

Tag queries expect regular expression formatting for multi-value variables. Append :regex to the variable, for example tag_values(server, app=~${apps:regex}).
Non-tag queries use the default glob formatting and don’t require :regex.

Dependent variable is empty:

Define the parent variable above the child variable in the dashboard variable list so it resolves first.
Verify the parent variable has a selected value. An empty parent produces an empty child query.

For details, refer to Graphite template variables.

Performance issues

Use the following troubleshooting steps to address slow queries or timeouts.

Queries timing out:

If queries consistently time out:

Increase the Timeout setting in the data source configuration.
Reduce the time range of your query.
Use more specific metric paths instead of broad wildcards.
Consider using summarize() or consolidateBy() functions to reduce the amount of data returned.
Check your Graphite server’s performance and resource utilization.

Slow autocomplete in the query editor:

If metric path autocomplete is slow:

This often indicates a large number of metrics in your Graphite server.
Use more specific path prefixes to narrow the search scope.
Check your Graphite server’s index performance.

MetricTank backend issues

If you’re using MetricTank as your Graphite backend, use the following troubleshooting steps.

Rollup indicator not appearing:

If the Rollup indicator doesn’t display when expected:

Verify Metrictank is selected as the Graphite backend type in the data source configuration.
Ensure the Rollup indicator toggle is enabled.
The indicator only appears when data aggregation actually occurs.

Unexpected data aggregation:

If you see unexpected aggregation in your data:

Check the data aggregation (rollup) configuration in your MetricTank instance.
Adjust the time range or use consolidateBy() to control aggregation behavior.
Review the query processing metadata in the panel inspector for details on how data was processed.

Enable debug logging

To capture detailed request and response information when troubleshooting Graphite issues:

Set the Grafana log level to debug in the configuration file:
ini
```
[log]
level = debug
```
Restart Grafana to apply the change.
Reproduce the issue, then review the logs in /var/log/grafana/grafana.log (or your configured log location) for Graphite-related entries that include the request URL and response details.
Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you continue to experience issues:

Check the Grafana community forums for similar issues and solutions.
Review the Graphite documentation for additional configuration options.
Contact Grafana Support if you’re an Enterprise, Cloud Pro, or Cloud Advanced customer.

When reporting issues, include the following information:

Grafana version
Graphite version (for example, 1.1.x) and backend type (Default or MetricTank)
Authentication method (Basic Auth, TLS, or none)
Error messages (redact sensitive information)
Steps to reproduce the issue
Relevant configuration such as data source settings, timeout values, and Graphite version setting (redact passwords and other credentials)
Sample query (if applicable, with sensitive data redacted)

Graphite data source on Grafana Labs

Configure the Graphite data source

Configure the Graphite data source

Before you begin

Add the Graphite data source

Configuration options in the UI

Provision the data source

YAML example

Provision with Terraform

Terraform example

Terraform example with basic authentication

Graphite query editor

Graphite query editor

Query editor elements

Choose metrics to query

Functions

Sort labels

Modify metric names in tables or charts

Consolidate data points

Combine time series

Select and explore data with tags

Nested queries

Use wildcards to make fewer queries

Example queries

Rename series for readable legends

Aggregate across many series

Convert a counter to a per-second rate

Summarize into time buckets

Control consolidation

Show the top series

Select series by tag

Build on another query

Apply annotations

Integration with Loki

Graphite template variables

Graphite template variables

Use query variables

Choose a variable syntax

Use tag variables

Filter with multiple expressions

Use other query variables

Compare expanded and non-expanded metric search results

Filter query variable results with __searchFilter

Graphite annotations

Graphite annotations

Common use cases

Before you begin

Annotation query modes

Create an annotation query

Metric query annotations

Example metric queries

Deployment markers

Service restarts

Error spikes above a threshold

Host down

Events query annotations

Create events in Graphite

Example events queries

Correlate multiple event types

Use template variables in annotations

End-to-end example: correlate deployments with latency

Before you start

Step 1: Record a deployment event

Step 2: Build the latency panel

Step 3: Add the deployment annotation query

Step 4: Interpret the result

Troubleshoot annotation issues

Annotations don’t appear

Annotation title is empty

Graphite alerting

Graphite alerting

Before you begin

Supported queries

Create an alert rule

Example alert queries

Alert on high CPU usage

Alert on low free disk space

Alert on high error rate

Alert on request latency

Alert across many hosts

Filter query variable results with `__searchFilter`