Grafana Labs blog on Grafana Labs

Get observability in your pocket with the Grafana mobile app

Ksenia Yadav — Thu, 15 Oct 2026 22:39:33 +0000

When you’re on the go, your phone is often the only observability tool you have. And what you can do with it in the first few minutes matters.

With the incredible innovation that has gone into mobile CPUs, GPUs and displays, a phone is a powerful device where you can immediately see what’s happening and start acting on it. And the Grafana mobile app now allows you to both see and act.

The app (available for Android and iOS) has always been good at notifying you of what's happening—alerts, incidents, and on-call schedules, previously under the Grafana IRM name. Now you can also ask questions in natural language, view your dashboards, and take actions across your stack. It is not a smaller, more cramped version of your computer, but a powerful starting point for the first few minutes of an incident.

Grafana Assistant on your phone

Grafana Assistant is a context-aware agent that can answer questions and take action across your stack. It already lives where you work: as a sidebar chat in the Grafana web UI, in Workspace if you prefer a conversation-focused experience for Grafana, in Slack or Microsoft Teams, or through gcx on the command line. It now runs on your phone, too, with the same conversations and the same abilities.

What makes Assistant useful is that it never starts from zero. It knows which stack you're in, which dashboard or alert you're looking at, and how your data is shaped.

With Assistant on your phone, you can:

Understand alerts and incidents: Catch up on what’s happened, explore what changed, and check whether a problem has occurred before.
Find relevant dashboards: Ask which dashboards can help you understand an issue and open them right from the conversation.
Check your on-call schedule: Find out when you’re on rotation without hunting through schedules.
Launch an investigation: Get Assistant working on a complex issue before you reach your laptop.
Pick up where you left off: Continue the same conversation on your computer, with the context intact.

Getting started takes just a question. For example, when an alert fires, you can open Assistant directly from it. The alert context is automatically included in the conversation, so you can immediately ask:

Incidents work the same way, with the conversation pre-seeded with the incident context:

Follow-up questions stay in the same context, so you can narrow down without repeating yourself:

Assistant can recommend an existing dashboard based on the conversation, allowing you to see it right away. Or you can easily ask:

And when you get back to your computer, you can continue the same conversation without losing any context:

Incident response on the go

Incident response is core to how effective engineering teams work. The app allows you to see alert groups, incidents, and on-call schedules:

You can also see the timeline of an incident, who was paged, and any other details:

If you use several devices, you can choose which of them gets paged in the notifications settings:

The app can answer less pressing questions, too. If you’re thinking about a weekend away at the end of the month and want to know whether you’re on the rotation before you commit, that’s a question Assistant can easily answer:

And when the issue seems complex, you can ask Assistant to kick off an investigation right from your phone. By the time you get to your laptop, there's a report waiting: the hypotheses it pursued, how each one was validated or ruled out, and where that trail led.

Your dashboards, wherever you are

Dashboards are the workhorses of Grafana. They're what most people open first in the morning, what's on the screen mounted on the office wall, and what a team builds over years as it learns what actually matters about its systems. Those same dashboards now open on your phone. They reflow to fit a small screen, and the ones you've starred come first:

Most dashboard use is a glance. You may have shipped a deploy, and on the train ride home you need to check that the error rate is flat. That's a perfect use case for a phone:

Have a key metric that you want to check regularly? You can add any panel to the app’s home tab showing your preferred data and time frame right after opening the app:

A different way of working, not a smaller one

None of this makes the phone a replacement for a real workstation. If you're trying to correlate five signals across three services to track down a root cause, you want a big screen, a keyboard, and four tabs open. That's not changing, and it shouldn't.

But the Grafana mobile app gives us a different way of working. Your first question gets asked in a queue for coffee. An investigation is launched from the back of a taxi. More context to understand an issue is gathered while you wait for a train. And the resulting conversations, investigations, dashboards, or acknowledgements are waiting in the Grafana web UI when you finally reach your computer, so that you could continue rather than opening it cold.

The rename from Grafana IRM to the Grafana app is a statement of direction. In addition to integrating Assistant and dashboards, more of Grafana is on the way as the app grows beyond its IRM roots. What we're building toward is a real window into your stack that happens to live in your pocket.

Coverage-first or cost-first: two ways to automatically apply Adaptive Metrics' recommendations

Arun Ulagappan — Thu, 15 Oct 2026 13:50:51 +0000

Adaptive Metrics , which is part of the Adaptive Telemetry suite, looks at how your metrics are actually used and recommends how much detail you should keep for each metric. This helps you cut costs while still retaining the data you actually need.

Last year we added auto-apply capabilities that let you implement those recommendations automatically, but only with coverage-first behavior, maximizing coverage of whatever you're querying. If your team wanted anything different, the only option was to review and apply recommendations by hand.

But now, with the General Availability release of auto-apply, we've added a second mode: cost-first, a built-in safeguard that only applies recommendations that won't increase your total active series—ensuring optimization never turns into a cost spike.

Auto-Apply continuously applies recommendations in either mode with no human intervention, while your existing dashboards, alerts, and queries keep working unchanged.

How cost-first mode decides what to apply

Every recommendation Adaptive Metrics generates comes with a projected change to your series count—some recommendations decrease it, others increase it. Cost-first mode can now take those projected changes into account, much like a careful manual review would, in three steps:

Series-reducing recommendations always apply. Anything that lowers your projected series count goes through immediately, including brand-new recommendations.
Series-increasing recommendations get ranked. Anything that would raise your projected series count is sorted from least impactful to most impactful.
Adaptive Metrics stops at your current total. It works down that ranked list, applying recommendations until the next one would push your projected series count above where it started the cycle.

How coverage-first mode decides what to apply

Coverage-first works more simply than cost-first does: every recommendation gets applied as soon as it's generated, whether it lowers your series count or raises it.

Every recommendation applies, no ranking required. Coverage-first doesn't weigh one recommendation against another or check a budget. It just applies what Adaptive Metrics finds.
Increases go through immediately. If a query comes in for data that's already been aggregated away, the recommendation to restore it at full resolution applies right away, because someone clearly needs that data.
Your metrics stay in lockstep with usage. As what you query shifts week to week, coverage-first adjusts automatically, so nothing you actually use ever falls out of resolution.

The tradeoff is the one cost-first exists to solve: because every increasing recommendation applies too, a spike in queries against previously-aggregated data shows up as a spike in series count.

A simplified example makes this concrete. Say a cycle generates three recommendations:

Recommendation	Effect on series count	Cost-first applied?	Coverage-first applied?	Why
1	-10	Yes	Yes	Decreases series count
2	+5	Yes	Yes	Running total still at or below where it started
3	+10	No	Yes	Would push the running total above where it started

The net effect is that your series count stays flat or drops. It never rises above where you started the cycle, which is the guarantee cost-first mode is built around.

Cost-first vs. coverage-first: when to apply each one

Adaptive Metrics' original auto-apply behavior, coverage-first, maximizes coverage of whatever you're actually querying. It's still there, and it's still the better fit for some of your data. If a dataset is business critical and your team knows exactly what it's querying and why, you want nothing standing between a real query and the raw data behind it. That's the mode we at Grafana Labs use internally for our own critical telemetry.

Cost-first mode is the better fit where cost matters more than guaranteed full coverage. For example, it would be well suited for a PromptQL query that asks for more than the person writing it realized, or an agent issuing queries on your behalf with no visibility into the cost impact downstream. That's increasingly common as less experienced users get access to larger datasets without realizing the potential impact of their queries. In those cases, you'd rather have Adaptive Metrics hold the line on cost and let you review and restore specific data later than have it de-aggregate automatically every time something gets queried unintentionally.

You also don't have to pick one mode for all of your data. Adaptive Metrics' segments feature lets you partition your telemetry along a dimension you choose (e.g., by team, environment, or cluster) and set a different mode on each segment independently. Keep coverage-first on the segments your experts own, and turn on cost-first mode for the segments where cost predictability matters more. It's the same underlying system; you're just telling it which guarantee you want and where.

Try auto-apply in Adaptive Metrics

If you're already using Adaptive Metrics in Grafana Cloud, both versions of auto-apply are available today, no extra setup required. Head to your Adaptive Metrics configuration, pick a segment, and turn it on to see your chosen mode applied on your next recommendation cycle.

For the full mechanics and configuration options, see the auto-apply documentation. It’s worth creating exemptions first for anything you can't afford to lose, and you'll need the Admin role to make the change.

You can also check your Adaptive Telemetry savings to see the impact on your Grafana Cloud bill.

New to auto-apply? Walk through the full learning journey in our docs: Enable auto-apply in Grafana Adaptive Metrics.

Go from alert to post-incident review with Grafana Cloud IRM and AI

Ashley Somerville — Wed, 14 Oct 2026 17:39:29 +0000

You’ve got 10 minutes before your next meeting when your phone starts ringing. You’re being paged because the frontend is erroring with failed API requests.

You open one tool to see what fired, then another to figure out what’s going on. You message a teammate in Slack to see if they’re noticing the same thing.

That’s a lot of context switching, and it’s still how a lot of teams' incident response works today. Opening another tool and catching someone up takes time. Details can get lost when you manually copy context from one place to another.

It doesn’t have to be this way. With Grafana Cloud IRM, you get on-call support, incident management, chatops in Slack or Microsoft Teams, investigations, and the Grafana mobile app, all connected through the incident record. Let’s follow the frontend incident from the first page through the response and post-incident review to illustrate how your team can start improving their processes today.

The page already has context

When you get paged, you need to understand how serious the problem is and what to check first. With IRM, you’re starting with more than an alert name.

When you open the alert group, it shows the affected service, the dashboard panel tied to the alert, its history, and how many individual alerts were folded into the page.

Imagine the difference between “the frontend is exposing API errors" and “the frontend is exposing HTTP 500 errors from the adService, the issue has already flapped twice in 24 hours, and three related alerts are firing in the same alert group.”

Figuring out what else is affected

Next, you need to figure out whether the frontend errors are isolated or part of something bigger, and whether the platform team is getting paged, too.

A lot of teams will do this by bouncing between dashboards, service maps, observability data, and Slack or Teams.

From the Slack thread, you ask Grafana Assistant, “What else is firing, and how is it related?”

It uses the same alert context you’re looking at and posts the answer back in the same thread.

Click Investigate this alert in the Slack message to get a quick summary of what’s firing and suggestions for what to check next.

Everyone in the thread can see what Assistant found, including teammates who join the response later. If someone has a follow-up, they can ask @Grafana in the thread.

You can ask it to help write a query to check the frontend’s error rate or find the dashboard you need. You can also give Assistant a service-specific playbook through Skills.

Declaring the incident

Once it’s clear this needs a coordinated response, you declare an incident in IRM.

As soon as you do so, IRM handles a number of automated actions for you, removing the need for manual creation every time. You immediately have a dedicated Slack channel, a Zoom call, and an AI investigation that starts looking for the root cause.

You don’t need to worry about the team remembering to create these components. The system does it for you.

Keeping the messy stuff

A lot of the useful information shows up in Slack before anyone knows if it matters.

A teammate remembers a deployment from earlier and suggests it might explain the frontend errors. Someone else has a hypothesis about a dependency. Another person spots something weird on a dashboard.

Usually some of that makes it into the incident record and the rest dies in Slack history.

In IRM, a teammate can react to a Slack message with the robot emoji and drop it onto the incident timeline. The original message, author, and timestamp come with it.

It doesn’t have to be a proven theory. The point is to retain potentially useful context while everyone is still figuring things out.

Managing the incident from Slack

Once the response is underway, the incident commander is juggling roles, tasks, comms, other teams, and the actual investigation.

That can happen in the incident channel without leaving Slack. You can assign roles, create tasks with quick actions, and sync those tasks to Jira or GitHub so they still exist after the incident is closed.

If another team needs to get involved, you can page them from the same place instead of hunting down their on-call schedule. Even better, enter "@Grafana assistant - who is on call?"

The incident owners post status updates that are shared in Slack and pushed to the mobile app so stakeholders can follow along.

An investigation you can actually inspect

The AI investigation starts automatically when you declare the incident.

It takes the context that’s already there (alert data, linked dashboard panels, the timeline, and anything people added from Slack), combines it with your actual data, and starts forming hypotheses about what caused the problem.

The Assistant Workspace feature provides a collaborative environment for you to work alongside the agent and track what it's doing, finding, and thinking.

You can add a hypothesis, reopen one that got dismissed, mark something disproven, or ask the investigation a question. You can also see the sources behind its conclusions.

A root-cause suggestion is a lot more useful when you can see the evidence, rather than having to decide whether to simply trust a black box. The investigation points to errors from the ad service reaching the frontend as 500 responses. The investigation can also correlate information from outside Grafana through MCP integrations. For example, it checks the earlier deployment hypothesis by checking GitHub directly and finds no evidence linking it to the errors.

It doesn’t matter that it started as a random Slack message. Once it’s in the incident context, it gets evaluated with everything else.

Turning the root cause into the incident record

Once you’ve reviewed the evidence for the leading hypothesis, the findings are added to the incident timeline and shared in Slack as well as pushed to the mobile app. That skips the usual cleanup job of taking what engineering figured out and rewriting it somewhere else so people outside the incident can follow it.

Resolving the incident: mobile app FTW!

You’ve found the root cause with help from the AI investigation and applied a fix to mitigate the issue. You know you still need to write up a post-incident review (PIR), but it’s easy to forget once everyone is back to their normal work.

Thankfully, with the Grafana mobile app, you can keep working from your phone, whether you’re responding to an incident or sorting out your next on-call shift.

So even if you can't get to the PIR immediately and some of the details become fuzzy in your memory, the incident record, timeline, investigation, hypotheses, and sources are all available in the mobile app.

Simply open the investigation on your phone and ask Assistant to create the post-incident review and send it to Confluence. The review is generated from the same history and evidence the team already used during the response. You’re not reconstructing the timeline from memory.

While you’re there, you even check when your next on-call shift starts.

And if you want to see the entire mobile workflow, check out the video below for all the details.

Why this matters

Your team still works in Slack, joins calls, and writes reviews in the tools it already uses. The incident record keeps that work connected. Check out Grafana Cloud IRM today to see how it looks against your own alerts

Take Grafana Labs' 5th annual Observability Survey

Trevor Jones — Thu, 08 Oct 2026 17:00:28 +0000

Over the past year, "agentic" has gone from an emerging buzzword to an integral part of worklife for many engineers—and that's just scratching the surface on all that's changed. With so much upheaval, there's never been a better time to capture the pulse of the industry, which is why we want you to take Grafana Labs' fifth annual Observability Survey.

We're asking the questions you're probably already arguing about in Slack: Do you trust an agent to ship without review? Are your dashboards still the front door, or has that become a chat window? How many observability tools are too many? And of course, the more things change, the more they stay the same, so we'll continue to track year-over-year trends around cost, tooling, open source, and maturity.

Available in Brazilian Portuguese, English, Japanese, and Spanish, this is the largest community-driven survey on the state of observability (last year more than 1,350 industry leaders and practitioners participated), and your answers directly shape the free report we'll publish early next year. It also helps you see how you compare to your peers, so please take five to 10 minutes to complete it today.

Enter to win some swag

Of course, it wouldn't be an Observability Survey without the chance to win some cool swag. Every month, we'll randomly select two participants and send them a special Grafana hoodie. And if you see the QR code for the survey at ObservabilityCON, ObservabilityCON on the Road, or a Grafana Labs booth during one of the many events we'll be at between now and Jan. 1, you might even get some special giveaways just for participating.

And if you don't want to enter to win, that's OK, too. All responses will be kept anonymous; we just want to hear what the community has to say about the state of observability today and where you want it to go in the future.

Manage your OpenTelemetry Collectors with Fleet Management in Grafana Cloud

Marko Bachvarovski — Thu, 08 Oct 2026 14:04:56 +0000

If you’ve built your telemetry pipelines around OpenTelemetry Collectors, you’ve already invested in a collector distribution, YAML configuration, and a deployment model that fits your infrastructure. As that deployment grows, managing it means keeping shared configuration consistent, accommodating different workloads, and understanding whether your collectors are healthy.

Fleet Management in Grafana Cloud brings those tasks together in one place. With support for the Open Agent Management Protocol (OpAMP) now generally available, you can bring supported OpenTelemetry Collector distributions into Fleet Management while continuing to use the OTel configuration language and components you already know.

We started with Grafana Alloy, Grafana’s distribution of the OpenTelemetry Collector. OpAMP support extends that management workflow to organizations running upstream OTel Collector distributions, so you can adopt centralized management without first migrating to Alloy.

Keep your OTel configuration, manage it across your fleet

For a platform team supporting multiple services or environments, collector configuration often includes both shared policies and workload-specific requirements. You might apply common processing across your fleet while enabling additional collection pipelines only for particular teams, services, or environments.

Fleet Management lets you define those pipelines in OTel YAML and match them to the collectors that should run them. You can update a shared pipeline centrally and apply it across the matching collectors, reducing the work of maintaining configuration separately for each deployment.

For teams using infrastructure as code, Terraform support brings this workflow into your existing processes. You can create pipelines with YAML configuration and register active collectors with Fleet Management, making configuration and enrollment part of a repeatable, version-controlled workflow.

You also get visibility into the collectors running those pipelines. When you enroll an OTel Collector, Fleet Management automatically configures it to send self-monitoring metrics, logs, and traces to Grafana Cloud, regardless of its operating system or deployment mode. For collectors running outside Kubernetes, we also attach a host monitoring pipeline so you can investigate the underlying infrastructure alongside the collector.

These monitoring pipelines remain under your control. Like any other pipeline, you can modify them, turn them off, or send their data to a different backend.

Making OTel configuration work across multiple pipelines

Managing OTel Collectors centrally requires more than accepting YAML. A collector may receive several Fleet Management pipelines alongside its local configuration, and those pieces need to work together as one effective configuration.

This differs from how Alloy handles configuration. Alloy components are defined in blocks and connected by references to their exports. OTel Collectors define components at the top level of the configuration and reference them from one or more telemetry pipelines.

Alloy also keeps local and remote configuration separate. OTel Collectors combine local and remote configuration into one effective configuration. When multiple sources define a component with the same name, one definition can override another, changing how the resulting pipelines behave.

That creates a practical challenge for reusable configuration. Two independently written pipelines might use the same component name, even though each expects that component to behave differently. Teams shouldn’t have to coordinate every component name across all their Fleet Management pipelines to use them together.

To address this, Fleet Management post-processes the remote configuration with a merge algorithm. It adds the Fleet Management pipeline name to OTel pipeline and component identifiers and updates their references accordingly. When multiple sources define components with the same type and configuration, it deduplicates them.

This lets you manage remote pipelines as separate, reusable pieces while Fleet Management handles combining them for the collector. You can organize configuration around what each pipeline does and where it belongs, without manually assembling a separate configuration file for every combination of pipelines.

Use the deployment pattern that fits your infrastructure

Configuration is only one part of managing an OTel deployment. Where you run your collectors, and how telemetry flows, also affects how you apply shared processing and maintain your fleet.

Fleet Management works across collector deployment patterns. For organizations managing a large or mixed fleet, one option worth considering is the gateway pattern.

Separate local collection from shared processing

The gateway pattern separates collection into two layers. Edge collectors gather telemetry from services, pods, and nodes, then send it to gateway collectors. The gateways apply shared processing before forwarding telemetry to a backend.

For example, you might add team labels to metrics, tail-sample traces, or filter logs at the gateway. Keeping these policies in the gateway layer means you can manage shared processing without putting the same logic on every edge collector.

Fleet Management lets you target configuration to each collector’s role: collection pipelines for the edge and shared processing pipelines for the gateways. This gives you a way to manage both layers centrally while keeping their responsibilities distinct.

Introduce OTel gateways into a mixed fleet

Your edge agents and gateways don’t have to use the same software. If you already run software such as Fluent Bit or Telegraf, you can keep those agents at the edge and send telemetry to OTel gateways using compatible protocols.

You can then use Fleet Management to manage the supported OTel gateways and their shared processing configuration. This provides an adoption path for teams that want to introduce OpenTelemetry without replacing every existing agent at once.

The gateway pattern isn’t required for Fleet Management. Whether your collectors send directly to a backend or process telemetry through a gateway layer, OpAMP support lets you bring supported OTel Collectors into a common management workflow.

Bring your OTel fleet into Fleet Management

You can adopt centralized collector management while retaining your OTel distributions, YAML pipelines, and deployment model. Fleet Management brings configuration targeting and collector health into Grafana Cloud, with Terraform support for teams that want to manage the workflow as code.

To get started, enroll an OTel Collector, assign it a YAML pipeline, and view its health. From there, you can extend the same workflow across the collectors and environments you already operate.

Manage synthetic checks at scale: Introducing folders in Grafana Cloud Synthetic Monitoring

Virginia Cepeda — Wed, 07 Oct 2026 15:00:41 +0000

As your use of Grafana Cloud Synthetic Monitoring grows, so does the number of checks you need to manage across services, environments, and teams. Eventually, a single flat list of checks becomes difficult to navigate, and even simple questions get harder to answer: Which checks belong to the payments team? Can I disable everything in staging during a maintenance window? Who should be able to edit the checks for this service?

Synthetic Monitoring folders, now generally available, help solve this challenge. You can organize checks into the same Grafana Cloud folders you already use for dashboards and alert rules, making it easier to group checks by team, service, or environment, manage multiple checks at once, and control who can edit them.

The benefits of folders in Synthetic Monitoring

Grafana Cloud Synthetic Monitoring is a black box monitoring solution that lets you proactively assess system reliability and performance by running checks from probe locations around the world. You can set up different types of checks, including ping checks, browser checks, scripted checks, DNS checks, and HTTP checks, to verify system performance at various network levels.

But as your Synthetic Monitoring setup grows, the process of managing checks one at a time doesn’t scale. You could have dozens of checks for the same service, separate sets for production and staging, or hundreds of checks owned by different teams. Without a way to group them, finding the right checks, making changes across them, and keeping ownership clear all require more manual work.

Folders give you a way to manage checks in a way that aligns with how your organization already works. Group checks by team, service, environment, or another structure that makes sense for you, then use that grouping to navigate your checks, take action across them, and control access. Because Synthetic Monitoring uses standard Grafana Cloud folders, the same organizational structure you use for dashboards and alert rules can now extend to your synthetic checks.

Folders bring three key benefits to your Synthetic Monitoring setup:

Find the checks you need faster: Browse checks grouped by folder instead of scanning a flat list, and expand only the folders you care about.
Manage related checks as a group: Enable, disable, move, or delete every check in a folder in a single operation. This is ideal for managing checks during maintenance windows or environment cleanups.
Give teams clearer ownership: Combine Grafana Cloud folder permissions with Synthetic Monitoring roles so each team manages the checks in their own folder.

How folders work

A Synthetic Monitoring check can live in any Grafana Cloud folder you can edit, such as your team's folder, a service folder, or anywhere else in your folder tree. When you create a check in the Synthetic Monitoring app, the default Synthetic Monitoring folder is preselected, and you can change it to any other folder by searching or browsing your folder tree. Checks created without a folder through the API or Terraform are stored in the default folder.

If a folder containing checks is deleted from Grafana Cloud, its checks are automatically moved to the default Synthetic Monitoring folder and keep running without interruption.

Assign checks to folders

When you create or edit a check, use the Folder field to select where it should live. You can also create a new folder right from the check editor.

To organize existing checks, select multiple checks in the check list and use Move to folder to move them all in one operation.

You can also assign checks to folders using the gcx CLI.

Browse and act on groups of checks

The check list offers a folder view in addition to the card and list views. Checks are grouped under their folders, so you can expand only what you need.

From a folder's actions, you can operate on all its checks at once:

Enable or disable all checks in the folder (for example, to silence an entire environment during a planned maintenance window).
Move all checks to another folder.
Delete all checks in the folder.
Move the folder itself to another location in your folder tree.

These actions apply to the checks you've selected in the folder or, when nothing is selected, to every check in it, including checks in nested subfolders. Deleting checks never deletes the folder itself.

Manage team access with folder permissions

This is where folders go beyond organization. Folder permissions work together with Synthetic Monitoring roles, so you can control which folders each user or team can manage. For example, the payments team could have edit permissions on the payments folder, while everyone else in the org only gets view access.

For more information on how to set up access permissions, refer to our documentation.

Provision folders as code

If you manage your synthetic checks through automation, folders are supported there, too. Options include:

Synthetic Monitoring API: checks accept a folderUid field on create and update.
Terraform: the grafana_synthetic_monitoring_check resource accepts a folder_uid attribute (Grafana Terraform provider v4.37.0 or later). Terraform remains the source of truth for the checks it manages: if someone moves a Terraform-managed check to a different folder in the UI, the next terraform plan reports the change as drift, and applying moves the check back to the folder defined in your configuration (or to the default folder, if none is set).

Wrapping up

Folders make it easier to scale your Synthetic Monitoring setup, giving you a consistent way to organize and manage checks. And because folders work across the UI, API, and Terraform, you can bring that same structure to both manual and automated workflows.

To learn more and get started, refer to our documentation for organizing checks in folders.

Grafana Cloud is the easiest way to get started with synthetic monitoring. We have a generous forever-free tier that includes 100k test executions per month and more. Sign up for free now!

Tempo 3.1 release: new features for Kafka, TraceQL metrics updates, trace redaction, and more

Tiffany Jernigan — Wed, 30 Sep 2026 19:03:30 +0000

Building on the major release of Tempo 3.0, Tempo 3.1 is here, delivering community-contributed Kafka client improvements, query-based trace redaction, sampling-aware TraceQL metrics, and more.

Together, the updates in 3.1 make it easier to operate Tempo, get accurate insights from your trace data, and investigate issues more efficiently. You can continue reading and check out the video below to learn more about the latest features. The Tempo 3.1 release notes and changelog provide more in-depth details and include all changes in this release.

Kafka client improvements from the community

Tempo 3.1 adds several community-contributed improvements to Kafka-based ingestion, with new options to secure connections, reduce data transfer costs, and support more Kafka-compatible backends.

TLS and more SASL mechanisms

Tempo 3.0 moved microservices-mode ingestion onto Kafka, and the client it shipped with could authenticate, but only with SASL PLAIN and only over an unencrypted connection. It had no TLS settings, so a broker that requires TLS would refuse the connection. Tempo 3.1 adds TLS and four more mechanisms: SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, and AWS_MSK_IAM.

ingest:
  kafka:
    address: kafka.example.com:9093
    topic: tempo-traces
    sasl_mechanism: SCRAM-SHA-512
    sasl_username: ${KAFKA_USERNAME}
    sasl_password: ${KAFKA_PASSWORD}
    tls_enabled: true
    tls_ca_path: /etc/tempo/kafka-ca.pem
    tls_cert_path: /etc/tempo/kafka-client.crt   # optional, for mTLS
    tls_key_path: /etc/tempo/kafka-client.key    # optional, for mTLS

Pass -config.expand-env=true to expand the environment variables.

Thanks to @heytrav for this contribution. To learn more, see the PR and the Configure authentication and TLS docs.

Rack-aware fetching

Tempo always fetched from the partition leader, wherever it happened to live. When the leader sits in another availability zone, every trace you read crosses a zone boundary, and your cloud provider bills you for the transfer.

The new client_rack option helps reduce those costs by enabling rack-aware fetching (KIP-392), so consumers can read from a replica in their own zone:

ingest:
  kafka:
    client_rack: us-east-1a

This is a read-side setting, so it applies to block-builders, live-stores, and metrics-generators, not to the distributor writing records into Kafka. Your brokers need rack IDs configured for it to take effect.

Thanks to @KyriosGN0 for this contribution. To learn more, see the PR and the ingest configuration docs.

Configurable producer compression

Tempo's distributor compresses every batch it writes to Kafka, and until 3.1, the codec wasn't configurable. That's fine unless your backend accepts only one codec: for example, Azure Event Hubs supports the Kafka protocol but accepts only gzip, which ruled it out as a Tempo backend.

The new producer_compression option lets you pick from none, gzip, snappy, lz4, or zstd:

ingest:
  kafka:
    producer_compression: gzip

Together with the TLS support above, this option makes Event Hubs a usable Kafka backend for Tempo. If you leave producer_compression unset, the Kafka client uses snappy by default. If your backend doesn’t support snappy, set producer_compression to a supported algorithm, or set it to none to disable compression.

Thanks to @fleighton for this contribution. To learn more, see the PR and the ingest configuration docs.

Protect sensitive data: redact traces with a TraceQL query

In Tempo 3.0, we added trace redaction, allowing you to permanently remove sensitive data, such as an email address, an auth token, or an account number that ended up in a span attribute, from your trace data without waiting for retention to expire. But tempo-cli redact only accepted trace IDs, so you had to enumerate every affected trace. Sensitive data usually lands in whatever traffic hits a given code path, which can be more traces than you can practically list. Any trace you miss is data still sitting in object storage, and potentially queryable.

Now, with Tempo 3.1, you can redact using a TraceQL query instead:

tempo-cli redact \
  --tenant= \
  --query '{span.attribute = ""}' \
  --dry-run \
  :

Start with --dry-run, as shown above. Tempo evaluates the query and counts what it would remove without modifying any blocks. The command prints the batch ID and the number of jobs created. Job counts and how many traces were matched or removed are per-tenant metrics that can be seen on the Redaction row of the Backend Work dashboard, which ships with Tempo's monitoring mixin. Once the count matches what you expect, run the same command without --dry-run to rewrite the blocks. This cannot be undone.

Because a wrong query permanently deletes data you didn't mean to remove, we're deliberately keeping the accepted syntax small for now: a single spanset filter with equality comparisons against resource.* and span.* attributes, combined using && and ||. Anything outside that is rejected when you submit the job. We are continuing to improve this functionality.

If you know the time range where data needs to be redacted, you can use it to run the job faster and more efficiently. This is important in high-volume installs because Tempo holds compaction off for a tenant while a redaction is applying. Use --start and --end, which accept now, a relative offset like now-7d, or an RFC3339 timestamp. Compaction catches up between runs.

Note: Only use --start and --end once every scheduler and worker in your cell is running at least Tempo 3.1. An older worker ignores the window and removes every query match in each block it is given, regardless of timestamp, with no error and no way to recover the data.

To learn more, see the query selector PR, the time range PR, and the Redact traces docs for the full query syntax and constraints.

Query trace metrics with greater accuracy, flexibility, and speed: updates to TraceQL metrics

In Tempo 3.0, TraceQL metrics became generally available, letting you query ad-hoc metrics directly from trace data. This makes it easier to answer questions about performance, error rates, and service behavior across distributed systems.

With the 3.1 release, we’re rolling out several updates that make TraceQL metrics more flexible and efficient, and more accurate when working with sampled trace data.

Sampling-aware metrics queries

Let’s say you sample 50% of your traces before they reach Tempo. If you ran a rate() query against a service, you would get half the traffic it actually served, because Tempo counts only the spans that made it through sampling.

This is especially useful with Adaptive Traces and other tail sampling setups, where sampling rates can vary across services. With sampling-aware queries, TraceQL metrics can account for those differences and more accurately reflect the underlying traffic.

With Tempo 3.1, you can correct for sampling at query time with the new experimental with(extrapolate=true) hint:

{ } | rate() with(extrapolate=true)

Samplers that implement OpenTelemetry's probability sampling specification, which is still in development and not supported in every language yet, stamp the rate they sampled at onto the span, in a field called tracestate. The hint reads that rate back and scales the span accordingly. At 50% sampling, each stored span counts as two, so a query that matches 1,000 spans reports 2,000. Spans that arrive with no sampling rate recorded count as one, which means partial adoption is safe: if only two of your services sample, the other services' numbers don't change.

Nothing new is written to your blocks. tracestate is already stored with every span, and queries that don't use the hint don't read it. If you already correct for sampling in the metrics-generator by setting enable_tracestate_span_multiplier, which we added in Tempo 3.0, the query path reads the rate the same way, so an ad-hoc query and your pre-aggregated tempo_spanmetrics_* series agree.

Extrapolation applies to rate, count_over_time, sum_over_time, avg_over_time, histogram_over_time, quantile_over_time, and compare. It doesn't apply to min_over_time or max_over_time, because sampling doesn't change the smallest or largest value you actually observed.

Note: This hint is experimental and requires vParquet4 blocks or later.

To learn more, see the PR and the TraceQL metrics functions docs.

Combine metrics queries with arithmetic

Many of the metrics you want to derive from traces require combining two metrics queries rather than running just one. A common example is error rate. Traditionally, TraceQL could measure errors and total traffic separately but couldn't divide one by the other, so you’d have to run two queries and divide them with a math expression in Grafana.

With Tempo 3.1, you can write the division in TraceQL itself. The operators +, -, *, and / work between two metrics queries, so an error rate is one query:

({ status = error } | rate() by (resource.service.name)) / ({ } | rate() by (resource.service.name))

Each sub-query needs its own parentheses. A scalar can go on either side of the operator:

# Calculate error percent in range of 0 .. 100
100 * ({status=error} | rate()) / ({ } | rate())

# Calculate p95 in milliseconds
({} | quantile_over_time(duration, 0.95)) * 1000

If you group your queries, two series combine only when their label sets match exactly, and a side with no labels is broadcast across every series on the other side.

You can combine with(extrapolate=true) and arithmetic. One hint at the end of the expression applies to both sub-queries:

({ status = error } | rate()) / ({ } | rate()) with(extrapolate=true)

The hint has to go at the end. Putting with(...) inside a sub-query is a syntax error.

To learn more, see the arithmetic PR, the scalars PR, and the arithmetic expressions docs, and the July Community Call.

Faster metrics queries by default

Metrics queries are also faster: we've seen simple ones run close to twice as fast. The span-only fetch path we introduced as experimental in Tempo 3.0 is now on by default for vParquet5 blocks, which is what Tempo 3.1 writes.

Queries that don't need the full trace structure now process individual spans instead of whole traces, which reduces latency and memory use. In one warmed-up test, { } | rate() with span-only fetch enabled completed in slightly over half the time it took with span-only fetch disabled. You don't have to change your queries to use it, and you can opt out per tenant or per query if needed.

To learn more, see the PR and the faster read path docs.

Enhancements to the metrics-generator

Metrics-generator is an optional Tempo component that derives metrics from ingested spans, giving you both RED (Rate/Error/Duration) metrics and service graphs, which show the relationships between your services. Here we cover a few of the improvements to metrics-generator included in Tempo 3.1, but please refer to the changelog and release notes for the complete list.

See why your service map is missing edges

Tempo builds a service graph edge by pairing the client and server spans for a connection. When one side never arrives, the edge expires and leaves a hole in your map, and you can't tell a connection that doesn't exist from one Tempo couldn't match.

In Tempo 3.1, we added an unmatched_span_kind label to the expired edges counter, so you can see which side went missing. A lot of expired edges labeled SPAN_KIND_SERVER, for example, point to server-to-server instrumentation that will never resolve into a node.

To learn more, see the PR, the service graphs docs, and the expired edges troubleshooting docs.

Recognize `db.system.name` from updated OTel semantic conventions

OpenTelemetry semantic conventions v1.30.0 renamed the db.system attribute to db.system.name. Instrumentation that emits only the new name stopped being recognized as a database call, so those nodes disappeared from your service map after an SDK upgrade, with no error to tell you.

Tempo 3.1 accepts both attributes for identifying database requests and for naming virtual nodes. When a span carries both, db.system takes precedence, so upgrading your SDKs won't rename the nodes your dashboards and alerts already point at.

Thanks to @iamrajiv for this contribution. To learn more, see the PR and the database name attributes docs.

Cut per-span metrics-generator costs

The generator processes every span you ingest, so per-span work sets the floor on what it costs to run. Building label sets accounted for a significant share of that work.

The span-metrics and service-graphs processors now borrow pooled label buffers from the registry instead of allocating new ones each time, and our steady-state service-graphs benchmarks report zero allocations per operation. Metric names, labels, and values are unchanged, so you won't see a difference in your dashboards, just generator CPU and memory you no longer have to provision.

To learn more, see the span metrics PR and the service graph PR.

Make large traces easier to read with span pruning

A single user request can produce many spans. If your application runs 100 nearly identical database queries behind that request, each one becomes its own span, and you end up with 1 root span and 100 database spans that all look the same. A trace like that is harder to read, and the more queries the request makes, the worse it gets.

With Tempo 3.1, the trace-by-ID v2 endpoint collapses each group of similar leaf spans into a single summary span, which carries aggregation.span_count and the minimum, maximum, average, and total duration of the spans it replaced. In the example above, you get back 2 spans instead of 101, in a correspondingly smaller response. Spans only group when their name, kind, status, and parent name match, so an error is never folded in with successes. This happens at read time, so nothing leaves storage.

Ask for it per query:

GET /api/v2/traces/?span_pruning=true

Set span_pruning_enabled_by_default to true in your Tempo configuration, and you won’t need to pass span_pruning on each request:

query_frontend:
  trace_by_id:
    span_pruning_enabled_by_default: true

That also covers the Grafana UI and the Tempo MCP server's get-trace tool, neither of which sends the parameter yet. An explicit span_pruning in a request still takes precedence, so a caller that needs the complete trace can always ask for it. You can also set that default per tenant.

We also added TraceQL filter support to the trace-by-ID v2 endpoint. It takes a single spanset filter on the q parameter, so you can fetch one trace and get back only the spans that match instead of all of them. Those spans come back on their own by default; add keep_hierarchy=true, and you get each match's ancestor path to the root too.

GET /api/v2/traces/?q={ span.http.status_code = 500 }&keep_hierarchy=true

The gcx CLI is being updated to expose both on gcx traces get: --prune for pruning and --filter for the query.

Note: Span pruning is experimental. The span_pruning parameters, their behavior, and the response format may change in future releases.

To learn more, see the trace pruning PR, the filter PR, the Query V2 API docs for the request parameters, the query-frontend configuration docs, and the August community call.

Make trace comparisons faster and more efficient for AI

Sometimes you want to compare two traces: an endpoint got slower after a deploy, and you have one trace from before and one from after, or the same request succeeds for one caller and fails for another. Either way, comparing them by hand means scrolling through two traces looking for the span that changed, appeared, or disappeared.

With Tempo 3.1, you can use traces diff to send two trace IDs to the query-frontend and get back what changed, instead of pulling both traces and comparing them yourself. This creates a much smaller context window for an LLM, too, which makes AI-driven investigations faster and cheaper:

POST /api/v2/traces/diff

{
  "base": { "traceId": "" },
  "compare": { "traceId": "" }
}

By default, you get a span-level patch listing every span that was added, removed, or modified. Partial traces are rejected, since spans missing from an incomplete trace would look like spans that were removed.

You can also ask for a summary instead of the patch. It reports whether trace latency, errors, and span count went up or down, whether the structure changed, and which services were involved, with a millisecond delta for the ones that moved. It tells you which direction each number moved, not whether that's a regression. A trace that got faster might just be failing earlier.

The summary also catches drift that the patch filters out. When every span in a service is slightly slower, no single span clears the per-span tolerance, but the service's summed span duration does, so the service still shows up as changed.

There are multiple tools you can use to get the traces diff. The gcx CLI wraps the API in an experimental gcx traces diff, so you can compare two IDs from the terminal without fetching either trace yourself:

gcx traces diff --context prod -d

tempo-cli experimental traces-diff diffs two local trace JSON files instead, which is handy when you've already exported them. It emits the patch by default, and you can get the summary with --format. The Tempo MCP server we added in 2.9 now includes a traces-diff tool that returns the summary first, so an AI assistant connected to Tempo gets a usable answer without always pulling back a large patch.

Note: This endpoint is experimental. The request and response formats may change in future releases.

To learn more, see the API PR, the CLI PR, the MCP PR, the traces diff API docs, the experimental traces diff Tempo CLI docs, the gcx traces diff reference, and the Compare traces Tempo MCP docs.

Improved query performance and lower memory usage: vParquet5 is now the default block format

Tempo stores traces in the Apache Parquet columnar block format, and continues to iterate on the layout for better performance and efficiency. We introduced vParquet5 as an opt-in format in Tempo 2.10, and in 3.1, it becomes the default.

The faster metrics read path mentioned earlier in this post is one payoff of the new default. It's implemented for vParquet5 only, and Tempo falls back to the standard path for blocks in older formats, so your metrics queries get faster as new blocks accumulate. Another payoff is that newly written blocks support the span:childCount intrinsic. We added it in 2.10, but only vParquet5 supports it, so previously you had to opt into vParquet5 to use it.

vParquet5 also offers more customization for the way the Parquet columns are configured. You can now have twice as many dedicated columns for strings, support for integers, and very long or high-cardinality strings (known as "blobs"), at each of resource, span, and event levels. The best way to take advantage of this is to use the new tempo-cli command suggest columns. It will analyze your data and tell you the optimal layout, and make it easy to copy and paste the output into your Tempo configuration.

To learn more about the features vParquet5 brings, see the docs for dedicated columns and the vParquet5 section of the Tempo 2.10 release blog post. You can also see the PR and the Apache Parquet block format docs.

How to learn more

To see the full list of improvements, bug fixes, and breaking changes in Tempo 3.1, please refer to the release notes and changelog. Before you upgrade, please review Upgrade your Tempo installation for the changes that need action.

If you are interested in hearing more about Tempo news, please join us on the Grafana Labs Community Slack channel #tempo, post a question in our community forums, or join our monthly Tempo community call. See you there!

What if your agent's hallucinations had a budget? How to start using SLOs for agent behavior

Ivana Huckova — Thu, 24 Sep 2026 17:14:13 +0000

At Grafana Labs, observability is what we do. So as we started building AI agents, we naturally reached for the same instincts we bring to every system: measure it, set targets, and make reliability something you can reason about instead of hope for.

That instinct led us somewhere unexpectedly useful. It turns out one of the oldest ideas in reliability engineering, the error budget, maps beautifully onto one of the newest problems in software: how do you know if an AI agent is actually any good?

This is what we discovered when we pointed our observability toolkit at agent behavior. We think it's a genuinely clean way to think about agent quality, and we want to share it with you.

The part of the agent you can't see

You can have every usual signal in place: latency, token counts, error rates, traces of every tool call. And none of them answer the question that matters most for an agent: Is it good?

A response can come back in 800 milliseconds, cost almost nothing, throw zero errors, and still be confidently, fluently wrong.

That gap matters because every change to your agent is a small gamble when you can't measure quality. Tweak a prompt and hope; swap a model and cross your fingers; ship it and wait for someone to complain. These are the exact blind spot observability exists to remove.

Turn behavior into a number with evaluations

So we measure behavior directly using evaluations; the same way we once turned "Is the site up?" into a simple percentage.

That's really all an evaluation is, after all. You ask a judge—often another language model—to read a conversation, or a single message, or one tool call, and score it.

Was it grounded in the context it was given, or did it make things up? Did it actually do what the user asked? Was it toxic? Did it leak personal information? Did it fall for a prompt injection? The judge hands back a verdict, pass or fail, or a rating on a scale, and suddenly the thing you couldn't see has a value you can look at.

Some of these you don't even need a model for. A humble regex knows perfectly well whether a response leaked an API key or name-dropped a competitor. In Grafana Cloud's Agent Observability, the cheap deterministic checks and the fancier model-based judges sit happily side by side.

One thing worth saying early: quality isn't a single number. "Good" is really a handful of separate behaviors, and they fail independently. An agent can be perfectly grounded and still be slow. It can be helpful and still occasionally toxic. It can nail the answer and quietly cost three times as much as it should to get there.

So instead of chasing one blended accuracy score that averages all of that away, it helps to pick the few behaviors that actually matter for your agent and give each its own measurement. Groundedness. Fulfillment. Toxicity. Latency. Cost. Each becomes a number in its own right, and each can get its own budget.

You can calibrate the judge to score based on your desired behavior. Sometimes a plain pass or fail is enough. Other times a rating is more informative, and that's perfectly fine. For example, if you score groundedness on a one-to-five scale, just draw a line to turn it into a target: say anything four and above counts as a pass. Either way, what you end up with is a rate you can hold a target against—the fraction of conversations that cleared the bar.

The point is that behavior, the thing that felt too fuzzy to pin down, is now a metric. You've made agent quality measurable, and the moment you do, it's finally in view and ready to act on.

And getting there in Grafana Cloud can start with a simple ask: have Grafana Assistant create an evaluation for a behavior you care about.

As it runs on real conversations, you'll start to see how often your agent clears the bar.

From there, Assistant creates and enables evaluation rules that are specific to each agent.

A number alone isn't enough

Here's the part where observability experience pays off, because we've seen this trap before.

A quality score sitting alone on a dashboard leads to a predictable problem. Maybe you start reacting to every little dip, or you cry wolf a few too many times. Before you know it, everyone quietly learns to ignore the graph. We've all got that Slack channel full of warnings nobody reads. A shiny new agent metric can easily become another one.

What the number is missing is a decision. That's exactly what a service level objective, or SLO, adds.

The idea is simpler than the acronym. You take your measurement and you commit to a target for it. In this case, let's say you want to measure the fraction of conversations a judge marks as fulfilled, so you set it at 95% fulfilled over 30 days. The interesting part is what's left over. If you're aiming for 95%, the other 5% isn't a failure. It's a budget.

That small shift changes how the whole thing feels. Unreliability, up to a point, stops being something to stamp out and becomes something to spend.

The budget is permission, not pressure

The best part of this idea isn't math; it's permission.

When you have the remaining budget, the message is clear: go for it. Try the ambitious prompt, test the new model, or take the swing, because you planned for a little failure. When the budget runs low, it flips just as clearly: ease off the new features, quality needs some love. Red light, green light, except the light is your own data instead of a gut feeling in a meeting.

There's a lovely bit from an episode of "Grafana's Big Tent" podcast where an SRE calls the error budget a fail-safe. Nobody thinks about it all day, he says, but it brings you back to reality when you mess up. That's the vibe we want on an agent team: not fear, not slamming the brakes the second a judge scores wobbles, just a quiet "it's OK, go build things," with a number that taps you on the shoulder when it's time to slow down.

And the budget cuts both ways. Before you ship a new prompt or model, you can replay it against conversations you already know and confirm it still lands inside your targets, so a change that quietly breaks one behavior never reaches users. Agents need that safety net more than most software, because the pressure to move fast is intense, and it's so tempting to just ship on vibes.

The quiet drift is the one to worry about

Once you start thinking about budgets, your attention shifts. You stop fixating on the shape of any single graph and start watching how fast you're spending. That shift matters a lot for agents specifically.

A sharp spike from a rough afternoon of bad conversations is loud and tends to get noticed. And against a 30-day budget it might barely leave a mark, which is genuinely useful to know, because it saves you from panicking over something you'll survive.

The one to actually keep an eye on is the opposite: the slow, quiet burn. A tiny, steady decline where no single conversation looks alarming, but a little quality drains away each day. Maybe a prompt has slowly gone stale. Maybe a model upgrade was subtly worse for your use case. Maybe your users are gradually asking for different things. Agents are almost designed to hide this, because their answers stay fluent and plausible even when they're drifting, and no simple threshold will catch it. But a budget will, because it measures the accumulation rather than any one moment.

That's also why budget-based alerts are worth trusting. When they fire, something is usually really wrong, and it's worth looking at. In a world already tired of noisy alerts, that peace of mind counts for a lot.

But what target do you even pick?

This is usually the next honest question: what number do you aim for? Is 85% fulfillment good? Is 95% unrealistic? It's genuinely harder than picking an uptime target because there's less gut feel for what's achievable with an agent.

You don't have to pluck it from the air. Once your evaluations have been running long enough to build up a useful sample, ask Assistant to recommend SLO targets based on their scores. In your own Grafana Cloud environment, that gives you a starting point grounded in how your agent actually behaves. You can then ask Assistant to create the SLOs, turning those measurements into targets and budgets.

From there it's your call. Nudge the target up if you can afford to be stricter, or down if you need more room. A young agent racing into a new space might happily accept a coin-flip chance of missing its target, while an agent in front of paying customers might want only a 5% chance of a slip. The point is choosing with your eyes open.

We don't have to reinvent this part

The striking thing, when you zoom out, is how little of this is actually new. Agent teams are facing a question reliability folks chewed on for years: how do you move fast, take real risks, and still keep people's trust? The answer was always to make what you care about measurable, pick a target, and treat the gap as a budget to spend. And that same answer travels surprisingly well to today. The only new piece is the measurement, and evaluations give us that.

So the pieces link up cleanly. Evaluations turn behavior into a number, the number gets a target, and the gap becomes a budget: permission to experiment, an early warning for regressions, and a shared language a PM and an on-call engineer can both speak. None of it asks you to invent a new philosophy of software, just to point out the one observability already given us at your agent.

So give your agent's behavior a number, give that number a budget, and then, inside that budget, build a little more freely. We'd genuinely love to hear how it goes.

From behavior to budgets, with a little assistance

Everything we’ve described sounds useful, but it also sounds like work. Decide what behaviors matter. Create evaluations for them. Watch the results. Pick sensible targets. Turn those targets into SLOs.

As we’ve seen along the way, Assistant can help you put those pieces together. You can simply ask it to create evaluations for one of your agents, or all of them. Let those evaluations run for a while and build up a useful sample of real data. Then ask Assistant to recommend targets and create SLOs based on what it’s seeing.

You choose what matters for your agents, and Assistant helps with the setup. From your first evaluation to targets and budgets grounded in real conversations, you can get started just by asking.

Turns out, even your agents can use a little help from an agent.

Grafana Alerting: Scale alert routing without scaling complexity using multiple notification policies

Matt Jacobson — Mon, 21 Sep 2026 18:45:23 +0000

Alert routing often starts simple. A team creates a few contact points, adds some label matchers, and builds a notification policy tree that sends each alert to the right destination.

But alerting configurations rarely stay simple.

As an organization grows, its notification policy tree must accommodate more teams, services, and routing requirements. Changes for one team still require editing a global configuration, making ownership less clear and independent provisioning harder. Over time, even small updates can require navigating an increasingly complex routing tree.

With Grafana 13.2, multiple notification policies are now generally available in Grafana-managed alerting. You can create named notification policy trees, assign alert rules to a specific policy, and manage each policy independently.

Existing alert rules continue to use the default notification policy unless you assign them to a named policy, so you can adopt the feature incrementally without migrating every rule at once.

What multiple notification policies enable

Multiple notification policies provide clearer boundaries for organizing, managing, and securing alert routing. With them, you can:

Split routing logic into smaller, named trees organized around a team, service, domain, or other ownership boundary
Assign alert rules directly to the policy that should handle their alerts
Provision and manage each policy independently through the Grafana UI, API, or Terraform
Control access using policy-level role-based access control
Update one policy without affecting the routing configuration or notification state of alerts assigned to another policy

The limits of one global notification policy tree

Grafana Alerting’s notification routing model builds on Prometheus Alertmanager, where routing is represented as a single global notification policy tree with a top-level route and nested child routes.

Within that tree, alert labels are matched against routes that determine how alerts are grouped, when notifications are sent, and which contact points receive them.

This model works well when a routing configuration is small or centrally managed. But it introduces challenges as more teams begin sharing the same Grafana stack:

Every team’s routing logic must fit within the same hierarchy
Provisioning or updating policies means operating on the entire tree
Teams need broad access to a shared configuration, even when they own only a small part of it
Large policy trees become harder to understand, review, and safely change

A common strategy as organizations scale is to create a top-level branch for each team, service, or domain. For example, a global tree might have separate branches for payments, platform, and security alerts.

This provides some logical organization, but the branches are still part of one shared configuration. They have the same lifecycle, the same provisioning boundary, and the same permissions boundary. Updating one team’s branch still means modifying the global tree that contains every other team’s branch.

With multiple notification policies, each team or service can have its own named policy instead of existing only as a branch within the global tree. Each policy can then be provisioned, secured, and managed independently.

The same familiar routing model, with better boundaries

Multiple notification policies let you split that global routing configuration into smaller, named policy trees.

Each named policy has its own root policy and child routes. Within the selected tree, routing continues to work as it always has: alert labels are matched against routes, and those routes control grouping, notification timings, and contact point selection.

The main difference is that an alert rule can now select which policy tree should handle its alerts.

For example, an alert rule owned by the payments team can select a policy named “payments.” Alerts produced by that rule are then routed only through the payments tree.

The policy selector is part of the alert rule configuration. Rules that do not explicitly select a named policy continue to use the default notification policy, preserving the behavior of existing alerting configurations.

This means you can adopt multiple notification policies incrementally. You can keep the existing tree as your default and introduce additional policies as teams or services are ready to manage their routing independently.

Independently manage and provision each policy

With a single global tree, provisioning was effectively all or nothing. Updating one team’s routes meant submitting or replacing a configuration that also contained every other team’s routing logic.

Named notification policies are individual resources. You can create, retrieve, update, delete, export, and provision one policy without replacing the others.

That makes it possible to align the lifecycle of a policy with the team or service that owns it. A platform team can deploy changes to its “platform” policy without also deploying the “payments” or “security” policies.

Policies are also isolated from one another. Changes to one policy’s routing tree do not alter the routing configuration or notification state of alerts assigned to another policy. Updating the “platform” policy, for example, doesn’t modify or disrupt the alerts being routed through the “payments” policy.

For teams managing alerting as code, this provides a much safer deployment boundary. A Terraform workspace or automation pipeline can own a single named policy instead of requiring ownership of the organization’s complete routing configuration.

Organize routing around how your organization works

There is no single correct way to divide notification policies. The right boundary depends on how your organization owns and operates its systems.

Policies can be organized by:

Team, such as “payments,” “security,” or “platform”
Service, such as “checkout-api” or “identity”
Domain, such as “infrastructure,” “customer-facing,” or “data-platform”
Environment, when different environments require substantially different routing and ownership

The goal is not necessarily to create a policy for every alert rule. Instead, policies should represent meaningful ownership or lifecycle boundaries.

Smaller trees are easier to review because each one contains only the routing decisions relevant to its purpose. A team can understand its default contact point, escalation paths, grouping configuration, and timing behavior without navigating unrelated routes owned by the rest of the organization.

Delegate ownership with more granular access control

Named policies also provide a natural boundary for more granular role-based access control.

Instead of granting users permission to modify the entire global notification policy tree, administrators can control who is allowed to view, create, update, or delete individual policies. This makes it easier to delegate alert routing to the teams closest to a service while retaining central control over shared or sensitive policies.

Policy-level permissions can be managed directly through the Grafana UI. Administrators can grant teams access to the policies they own without granting broad access to the rest of the organization’s notification routing configuration.

For example, the payments team can manage the “payments” policy without ever receiving permission to change, or even view, the “security” policy. A central observability team can retain control of the default policy and organization-wide fallbacks.

This helps organizations distribute alerting ownership without distributing unrestricted access to the complete notification configuration.

Manage policies through the UI, API, or Terraform

You can create and manage named notification policies from the Grafana UI. Navigate to:

Alerts & IRM → Alerting → Notification configuration → Notification policies

From there, create a new policy or select an existing policy to manage its routing tree. When configuring an alert rule, select the notification policy that should receive alerts from that rule.

Named policies are also available through the Grafana Alerting API, allowing automation systems to manage each routing tree as an individual resource.

For Terraform users, the grafana_apps_notifications_routingtree_v1beta1 resource manages named routing trees. The name is supplied through the resource metadata, so separate Terraform resources can own separate policies.

Example:

resource "grafana_apps_notifications_routingtree_v1beta1" "payments" {
  metadata {
    uid = "payments"
  }

  spec {
    defaults {
      receiver        = "payments-slack"
      group_by        = ["alertname", "cluster"]
      group_wait      = "30s"
      group_interval  = "5m"
      repeat_interval = "4h"
    }

    routes {
      receiver = "payments-oncall"

      matchers = [
        {
          label = "severity"
          type  = "="
          value = "critical"
        }
      ]
    }

    routes {
      receiver = "payments-slack"

      matchers = [
        {
          label = "severity"
          type  = "="
          value = "warn"
        }
      ]
    }
  }
}

Each resource can be managed by the workspace or repository responsible for that policy. The Terraform resource supports nested routes, matchers, grouping, notification timing options, and provenance controls that determine whether the policy remains editable outside Terraform.

Adopt multiple policies incrementally

Multiple notification policies were introduced in Grafana 13.1 behind the alertingMultiplePolicies feature toggle, and they became generally available in Grafana 13.2.

Existing configurations continue to work through the default notification policy, and there is no requirement to immediately divide an existing tree or update every alert rule.

A practical adoption path is:

Create a named policy for a team, service, or domain.
Reproduce the relevant routing behavior in the new policy.
Assign that team’s alert rules to the policy.
Validate the resulting notification behavior.
Move management of the policy to the owning team or automation workflow.

Other rules can remain on the default policy until there is a reason to move them.

Alert routing that scales with your organization

A single notification policy tree is simple when an alerting setup is small. At scale, however, it can become a shared configuration bottleneck that is difficult to own, provision, and safely change.

Multiple notification policies preserve Grafana Alerting’s familiar label-based routing model while introducing clearer lifecycle and ownership boundaries. Teams can work with smaller trees, provision policies independently, limit the scope of changes, and apply more granular access controls.

Whether you organize policies by team, service, or domain, the result is alert routing that more closely reflects how your organization actually operates.

Digital Experience Monitoring with Grafana Cloud: Session Replay, synthetic checks, and faster investigations

Bukola Ayodele — Mon, 14 Sep 2026 18:02:58 +0000

When something breaks in production, the questions that matter most are also the toughest to answer from metrics alone: who was affected, what did they actually see, and is this worth waking someone up for? Answering those questions requires a fuller picture of the issue and its impact on your users.

That’s where Digital Experience Monitoring (DEM) in Grafana Cloud comes in. By combining Frontend Observability and Synthetic Monitoring, DEM connects real user experiences with proactive testing, helping engineering teams understand the scope of an issue, investigate its cause, and resolve it faster, all within Grafana Cloud.

In this blog post, we'll walk through some of the latest DEM updates in Grafana Cloud, and how to get started. You can also learn more by watching the video below.

First, what is Digital Experience Monitoring?

Digital Experience Monitoring in Grafana Cloud gives you a complete picture of how users experience your web applications, from real user data to proactive synthetic checks.

DEM helps your team achieve:

Real user visibility: know how users truly experience your web application, not just what your backend metrics suggest.
Proactive detection: catch problems before your users do, using automated checks against your critical user journeys.
End-to-end correlation: connect a frontend signal to the backend trace behind it.
Faster resolution: cut your mean time to recovery from hours to minutes.

Session Replay: see exactly what your users saw

Session Replay in Grafana Cloud Frontend Observability lets you visually replay what a user saw and did inside your web application. Your team can watch exactly what users experienced and correlate it with real user monitoring signals like Core Web Vitals, user actions, and traces, which makes it a powerful tool for investigating bugs and running root cause analysis.

Session Replay is powered by Faro, Grafana's open source JavaScript instrumentation library for collecting real user monitoring data. Let's walk through how to set it up.

Step 1: Add the Replay instrumentation to Faro

We start by adding the Faro web SDK to our web application. When we initialize Faro, we add the Replay instrumentation alongside the standard Faro web instrumentations:

import { getWebInstrumentations, initializeFaro } from '@grafana/faro-web-sdk';
import { ReplayInstrumentation } from '@grafana/faro-instrumentation-replay';

initializeFaro({
  url: 'https://your-faro-endpoint.com',
  instrumentations: [
    // Standard Faro instrumentations: errors, web vitals, user actions, and more
    ...getWebInstrumentations(),
    // Enables session recording
    new ReplayInstrumentation(),
  ],
});

During initialization, the default configuration is “privacy first,” but you can further tweak the masking, privacy, and sampling options that fit your needs. For example, you can mask specific input types or record replays for only a percentage of sessions.

Step 2: Generate some session data

Once the web application is instrumented, we can see it in action. For this example, we'll use QuickPizza, our demo web app that lets you create pizza combinations. On the website, we'll click a few buttons and experiment with the app, the same way a real user would. All of that activity is recorded and sent to Frontend Observability.

Step 3: Find your session recordings

Next, we'll head into the Frontend Observability app, open Sessions, and scroll down to find our session recordings.

Step 4: Explore the replay

Our sessions are populated and we can select one to replay. Private information is masked client-side, which means it is never sent to Grafana Cloud, but you can still clearly follow mouse movements and DOM actions. To view session recordings, you have two options: you can open a session directly from the session details page, or click the green button to open the player in a new tab. We recommend the second option, as it lets you watch the recording side-by-side with the full user journey that always stays synchronized with the player's state.

The replay player gives you full control over playback:

Playback controls: play, pause, and skip ten seconds forward or backward
Adjustable speed: from 0.25x all the way up to 16x, which makes it easy to scan longer sessions
Skip inactivity: jump past the quiet parts of the recording, so you stay focused on what matters
Share button: copies a link to a specific moment in the replay, so you can send a teammate the exact second an issue happens

On the side, you can see the user journey with timestamps, showing what both the user and the browser were experiencing. You can also see errors from this view, skip straight to them, and filter and sort by errors.

This is where Session Replay gets really powerful. Because replays are correlated with your existing Faro telemetry, you can jump from a frontend error in your dashboard directly into the replay for that session, watch exactly what the user was doing when the error occurred, and then dig even further into the correlated traces.

To learn more about Session Replay in Frontend Observability, please check out this blog post and our technical docs.

Synthetic Monitoring and Frontend Observability: better together

Grafana Cloud Synthetic Monitoring runs automated checks against your critical user journeys, so you catch issues before your real users ever see them, while Grafana Cloud Frontend Observability captures what your real users are actually experiencing, turning your frontend performance into something you can measure and act on. Together, they form the foundation of Digital Experience Monitoring in Grafana Cloud, giving you both a proactive and real-world view of your users’ digital experience.

We recently made it easier to move between Synthetic Monitoring and Frontend Observability when investigating an issue, enabling end-to-end correlation from a synthetic check through the frontend experience and into the traces behind it.

Every time a Synthetic Monitoring browser check runs, it passively creates a matching Frontend Observability session, and that session is a fully controlled, repeatable run of a real user journey. From inside a check, you can now pull up Frontend Observability data right there as context and jump straight into the exact session that run created. Once you're in that session, you have access to the session replay, the user journey, and traces.

Your synthetic browser checks are no longer just a pass or a fail. You get feedback from every run, so when a check fails, you can see exactly what happened. That means you can finally answer the question every on-call engineer asks: is this failing check a real user problem, and how many users were impacted?

Walking through a failing check

Let's jump into Synthetic Monitoring to see the integration in action. We've already set up a browser check that mimics a typical QuickPizza user flow, and we just got an alert that this check is failing.

Open the check dashboard. Right away, we can see the failing check.
Find the failed execution. Down in the timepoint explorer, we can see each individual execution, including the failed ones. We'll select one.
View the frontend session. When we scroll down, we see a button that says View Frontend Session. With one click, it opens the Frontend Observability session for this exact run.

This view has everything we need to investigate: the session replay, which is a step-by-step visual replay of every action the check performed; the user experience metrics from the run; the full user journey; and the traces behind it. In just a few clicks, we went from an alert, to a failed execution, to watching exactly what happened during the check.

Learn more

To learn more, visit our Digital Experience Monitoring solutions page, or explore each tool in greater depth by checking out our technical docs for Frontend Observability and Synthetic Monitoring.

Grafana Cloud is the easiest way to get started with Digital Experience Monitoring. We have a generous free tier that includes 100k test executions per month and more. Sign up for free now!