Grafana Labs blog on Grafana Labs — tagged 'security'

How to scale access control in Grafana Cloud

Jake Batty — Tue, 07 Jul 2026 17:02:52 +0000

One of the primary reasons organizations adopt Grafana Cloud is to create a single pane of glass across the data they collect from self-hosted systems, cloud providers, and third-party platforms. Bringing those signals together enables richer correlations, reduces tool sprawl, and makes it easier for teams to understand what's happening across their environment.

But as observability grows and becomes more centralized, access management becomes more important. When infrastructure metrics, application logs, business KPIs, and customer-specific data all live in the same platform, organizations need a scalable way to ensure the right people have access to the right resources without creating additional administrative overhead.

The good news is this can all be done directly in Grafana Cloud. To illustrate how this can work in practice, let's look at a fictional company modeled after a common scenario.

Meet AcmeCloud (and the challenge of manual provisioning access)

AcmeCloud, a platform-as-a-service provider, has set up a new Grafana Cloud instance to visualize infrastructure metrics, business metrics, sensitive application logs pertaining to each of their customers, and more—all in one place. Now they need to provision access across their organization.

They have hundreds of users that require access, and different roles have different needs:

Admins need full control of data and users
SREs need to configure data sources, explore telemetry, and build dashboards
Contractors, working on behalf of specific AcmeCloud customers, need to log in but only to see the dashboards and data relevant to their tenant

At this scale, manual provisioning isn't an option. Neither is trying to keep users in sync between AcmeCloud's identity provider and Grafana Cloud by hand. It's simply too time-consuming and too prone to errors.

For organizations facing similar growth, this is often the point where access management starts to become a scalability challenge rather than an administrative task.

Establishing a single source of truth with SSO and SCIM

Instead of handling everything manually, AcmeCloud relies on SSO (Single Sign-On) for authentication and SCIM (System for Cross-domain Identity Management) for provisioning.

SSO handles how users logged in, while SCIM ensures the right users and groups are automatically available in Grafana Cloud. There are many options for authenticating users in Grafana Cloud, including native integrations with identity providers as well as support for generic authentication methods. To see the full list of supported methods and integrations, read more here.

As users are added, removed, or updated in the identity provider, those changes are reflected in Grafana Cloud automatically. The same concept applies to groups, which are mapped directly to Grafana teams. Groups within the identity provider typically organize users by job function, which correlate to a Grafana team in most cases, since users working in the same role typically have the same use cases and permission levels.

The result is a single source of truth for identity, with no need to manually reconcile users or worry about configuration drift.

For organizations implementing this approach, Grafana's SSO and SCIM integrations make it possible to manage users and groups from the identity provider rather than treating Grafana as a separate identity system.

If you are new to the concept of SCIM provisioning, check out this blog to learn more.

Tip: Establish clear group naming conventions in your identity provider before enabling SCIM. It makes permission mapping significantly easier as your environment grows.

Building a layered access model with RBAC

With users now syncing into Grafana Cloud, the next step is defining what those users can actually do.

AcmeCloud opts to manage access primarily through role-based access control (RBAC).This helps to maintain scalability while providing greater flexibility and security through a combination of basic roles, team memberships, and resource-level permissions. Dive deeper on creating a RBAC strategy and configuration through the guidance in our documentation.

Basic roles in Grafana set the foundation, with assignments mapped from the identity provider:

Admins are assigned the Admin role
Internal AcmeCloud users from the SRE team receive the Editor role
Contractors receive no basic role at all

This allows the internal SRE team to access dashboards and data broadly. They can build dashboards and alerts, as well as query metrics, logs, and traces across all tenants. All that is missing now is the ability to configure or alter data sources.

Contractors can log in, but they're presented with an empty Grafana experience by default. For now, there are no available actions to take in Grafana, but they need the ability to view dashboards based on data solely containing their own tenant id.

To build on the permissions provided through basic roles, teams that were provisioned through SCIM are used to layer on additional access.

Users inherit permissions from both their basic role and any teams they belong to. This is an important distinction: permissions in Grafana are additive, not restrictive. A user's effective access is the combination of all assigned roles and team memberships.

For internal users from the SRE team, a shared team is granted additional capabilities, including the ability to configure and manage data sources. This extends their access beyond what the Editor role provides by default.

For organizations designing their own access model, a useful starting point is to assign the minimum role required and then use teams to grant additional capabilities as needed.

Tip: Avoid using highly permissive basic roles as a shortcut. Teams are often a more scalable way to manage access as responsibilities evolve.

Restricting dashboard visibility with teams and folders

Contractors are grouped into tenant-specific teams, with no additional capabilities added at that level. Instead, access is controlled through folder permissions.

Each team is given view access to a specific dashboard folder, so contractors can only see only the dashboards relevant to their tenant and nothing else.

Folder permissions provided a simple and scalable way to partition dashboard visibility without creating separate Grafana instances for every customer.

For AcmeCloud, this approach balanced operational simplicity with tenant isolation. Organizations managing multiple customers, business units, or environments often find folder permissions to be one of the simplest ways to segment dashboard access.

Tip: Design your folder structure with future growth in mind. Reorganizing hundreds of dashboards later can become a significant effort.

Enforcing data-level isolation with LBAC

Up to this point, AcmeCloud has been using RBAC to determine what users could access within Grafana Cloud. But dashboard visibility and permissions were only part of the problem.

AcmeCloud also needs to ensure that access to dashboards doesn't automatically mean access to all underlying data. This is where label-based access control (LBAC) came into play.

By using LBAC, data is scoped by the tenant. That way, even if a user has access to a dashboard, queries are restricted so they only return data associated with their assigned tenant. Read more here for guidance on configuration and limitations.

This adds a final layer of protection, ensuring that customer data remains isolated-even within a shared Grafana instance.

For organizations operating multi-tenant environments, this distinction is important. Dashboard permissions control what users can see, but data-level controls determine what data they can access. Both are required to build a complete access strategy.

Delivering the right experience for every user

From the contractor's perspective, the experience is intentionally limited.

After logging in via SSO, they're placed into their tenant-specific team and land on a predefined home dashboard. They can only view the dashboards within their assigned folder and they can't explore beyond that scope—no visibility into other tenants, and no risk of accessing unrelated data.

What made this setup work isn't any single feature. Instead, it's how all these features are combined to address scalability, flexibility, and security demands.

SCIM handles scale and lifecycle management. RBAC—implemented through basic roles, teams, and folder permissions—defines who can access resources. LBAC then ensures data-level isolation within those resources.

Individually, each of these capabilities is straightforward. Together, they form a layered access model that allows organizations to onboard large numbers of users, both internal and external, without sacrificing control.

Bringing it all together

As Grafana usage grows, access management becomes just as important as observability itself. Designing around user types, scope, and data boundaries from the start makes it possible to scale confidently without losing track of who can see and do what.

The AcmeCloud example demonstrates a pattern that many organizations can adopt: centralize identity with SSO and SCIM, use roles and teams to define capabilities, control visibility through folders, and enforce data isolation with LBAC.

A single pane of glass doesn't have to mean broad access to everything. With a layered approach to access control, organizations can centralize observability while still maintaining the security and governance required as their environments grow.

Post-incident review for TanStack npm supply chain ransom incident: No unauthorized access to customer production systems

Joe McManus — Tue, 23 Jun 2026 20:24:27 +0000

On May 27, we completed our internal investigation of the recent TanStack supply chain ransom incident and confirmed our initial findings: The incident was strictly limited to Grafana Labs' GitHub environment. There was no unauthorized access to customer production systems, and the Grafana Cloud platform was not affected.

For an additional, independent audit, we engaged Mandiant, a leader in cybersecurity and incident response. We provided them with API access to Grafana Labs' log environment to conduct queries across our systems for their investigation, which started on June 1. Mandiant confirmed that there was “no evidence of code tampering or repository poisoning within public organizations or production repositories delivered to end users.”

Since we discovered the incident, the Grafana Labs security teams have been running two parallel workstreams: completing the investigation and hardening our security operations. We are publishing this blog in the spirit of transparency to share more details about our incident response and remediation efforts.

Summary and impact

If you’re looking for the short version instead of reading our previous updates, here is the TL;DR: The TanStack supply chain attack hit us on May 11 via the Mini Shai-Hulud campaign. At the time, we believed we had successfully rotated every credential involved in this incident. We missed one. I won’t blame this oversight on hubris; the data we had at the time simply led us to believe our rotation was exhaustive. We were mistaken.

A bad actor utilized that overlooked credential to clone our entire repository collection. They then reached out on May 16, demanding a ransom to prevent a code leak.

Since Grafana Labs is an open source company, you might wonder why this is a concern. While most of our source code is public, we do maintain private repos for things like internal tools and specific Grafana Cloud features. It was a heavy decision, but we stuck to our principles and the FBI’s documented guidance: We did not pay.

We launched our mitigation efforts immediately, and we confirmed that there was no unauthorized access to customer production systems, and the Grafana Cloud platform was not affected. We also confirmed that while our codebase was downloaded, it was not altered. Our customers and open source users do not need to take any action.

Grafana Labs’ response

We were alerted to the incident on a Saturday, and teams across the entire company took action quickly and decisively. (Or to borrow a phrase from one my favorite rappers Big Daddy Kane, ain't no half-stepping at Grafana Labs.)

In response, Grafana Labs suspended all GitHub applications on May 17, initiated a global code freeze on May 18, and conducted a cross-platform audit of Vault, GitHub, Okta, Kubernetes, AWS, GCP, and host logs to verify that no production customer data was compromised.

In the weeks following, our engineering teams contributed to a comprehensive audit that included but was not limited to:

Completing 1,500 security-focused PR reviews
Auditing 280 GitHub applications, stripping permissions and removing several
Scanning 1,200 repositories for any signs of tampering
Executing 2,300 PR reviews looking for unauthorized changes in a single critical repo
Finishing infrastructure audits and retiring legacy systems
Performing wide-ranging new access audits

It was a massive undertaking, but each team stepped up in an extraordinary way to do their part. Engineering, security, and cross-functional partners worked tirelessly to respond, demonstrating the collaboration and the shared commitment we have to our community and our customers that I have always valued here at Grafana Labs.

After the initial assessment, we found that in addition to source code, the downloaded content included GitHub repositories that some Grafana Labs teams use to collaborate on and store internal operational information and other details about our business. This includes, for example, business contact names and email addresses that would be exchanged in a professional setting and email addresses that were used in some past marketing campaigns. This was not information pulled from or processed through the use of production systems or the Grafana Cloud platform.

If you wish to know if email addresses with your domain were identified, please reach out to Grafana Labs support.

Incident timeline

All times are in UTC

19:21 11 May - First malicious code executed on self-hosted runners by Shai Hulud threat actors, leaking credentials. Rotated credentials.
07:21 14 May - First malicious commit made by the threat actor using grafana-delivery-bot, leaked from Shai Hulud attackers.
13:28 14 May - Data exfiltration of repos begins.
20:57 15 May - Data extortion threat actor publishes their extortion demand.
08:30 16 May - Grafana Labs security team becomes aware of the claimed ransom and begins seeking confirmation.
17:39 16 May - Compromise confirmed; incident declared.
19:33 16 May - All known affected credentials and GitHub applications suspended/rotated. Suspension and rotation of all other GitHub applications and accessible credentials begins.
21:10 16 May - Suspension of all GitHub applications completed.
16:40 17 May - All code changes made by GitHub application accounts associated with the threat actor identified and reverted.
16:52 17 May - Root cause, attack chain of compromise identified.
17:21 17 May - DockerHub credentials determined not compromised.
17:51 17 May - All malicious workflow runs identified. Final list of affected secrets compiled and rotated. Rotation of all other ci/common secrets from affected repos continues.
23:23 17 May - Last of the potentially accessible credentials confirmed rotated or suspended.
03:08 18 May - Begin freeze of all non-critical code and deployment changes.
08:00 25 May - All-engineering security hardening week commences.
10:58 26 May - Commit review completed, service thawing begins. A repository needs to have been fully reviewed and transitioned to use a GitHub application token broker for short-term, finely-scoped credentials before being thawed.
10:54 27 May - Transition from repos directly pushing images to DockerHub to pushing to Google Cloud Artifact Registry occurs.
27 May - Internal investigation complete. No additional attack activity or compromised credentials were discovered.
08:00 2 June - All-engineering security hardening week concludes.
20:43 3 June - Review of repositories for data loss completed.
18 June - Mandiant investigation completed, corroborating internal investigation.

What’s next

The investigation is now closed, but our work to improve security operations at Grafana Labs will continue. Dostoyevsky once noted that "when reason fails, the devil helps!" I’m quoting “Crime and Punishment” to underscore our philosophy: We only wanted to implement changes that actually moved the needle on security.

We’ve spent the past month executing high-impact controls, including a token broker, fine-grained access controls, additional alerting, and static analysis. In addition, we have moved off of certain GitHub Actions and now use more tightly scoped actions with short-lived tokens.

We have also started the process of compartmentalizing our GitHub organizations and isolating all archived repos into a dedicated organization with actions disabled.

We will share an overview of our response efforts and the technical details of how we improved our security posture from our post-incident review in the coming weeks.

Grafana Labs security update: Latest on TanStack npm supply chain ransomware incident

Joe McManus — Tue, 19 May 2026 18:11:35 +0000

Updated on June 23, 2026: For more information about our incident response and remediation, please read our post-incident review blog post .
Updated on June 1, 2026: Our internal investigation is complete, and it confirms our initial findings that there was no unauthorized access to customer production systems or the Grafana Cloud platform. We have engaged Mandiant, a leader in cybersecurity and incident response, to perform an additional audit, and expect its post-incident report to be completed in June. As part of our standard security practices, we will share in the coming weeks additional information from our post-incident review and ways in which we have improved our security posture.
Should we ever determine that our customers' systems or operations have been impacted, we will notify them directly.

On May 16, 2026, Grafana Labs confirmed a targeted attack by a cybercrime group that gained unauthorized access to our GitHub repositories and downloaded our codebase. They then issued a ransom demand under threat of data disclosure.

Since we posted our initial findings that day, our investigation has continued, and we are publishing this blog to share more details about our incident response and mitigation. A post-incident report will be published when our investigation is complete.

To date, the investigation has found no evidence that customer production systems or operations have been compromised. This incident was strictly limited to the Grafana Labs GitHub environment and did not affect our production systems or the Grafana Cloud platform.

After the initial assessment, we found that in addition to source code, the downloaded content included GitHub repositories that some Grafana Labs teams use to collaborate on and store internal operational information and other details about our business. This includes business contact names and email addresses that would be exchanged in a professional relationship context, not information pulled from or processed through the use of production systems or the Grafana Cloud platform.

To be clear to the users of Grafana Labs' open source projects and the Grafana Cloud platform: our codebase was downloaded, but it was not altered. No action is needed from our customers or open source users at this time.

Our investigation is ongoing as we continue to review logs, telemetry, and all available data within our company-wide GitHub repos. Should we ever determine that any customer's systems or operations are impacted, we will notify them directly.

At Grafana Labs, earning and maintaining our community’s trust is foundational to everything we do. We recognize that customers rely on us as a trusted partner, and we do not take that responsibility lightly. We are sharing this update in the spirit of transparency because we understand you may have questions and because we take this matter seriously.

Summary and background

The incident originated from a TanStack npm supply chain attack via the Mini Shai-Hulud campaign. We detected the malicious activity on May 11 and immediately initiated our incident response plan.

We performed analysis and quickly rotated a significant number of GitHub workflow tokens, but a missed token led to the attackers gaining access to our GitHub repositories. A subsequent review confirmed that a specific GitHub workflow we originally deemed not impacted had, in fact, been compromised.

On May 16, we received a demand from a bad actor for a ransom payment to prevent the release of our codebase. Grafana Labs determined the appropriate path forward is not to pay the ransom. This decision aligns with the FBI’s formal position that paying a ransom does not guarantee security and only serves to incentivize further criminal enterprise.

As soon as we were contacted by the ransom gang, we launched mitigation efforts, which have included rotating automation tokens, implementing enhanced monitoring, auditing all commits since the May 11 incident, and significantly hardening our GitHub security posture.

We have also notified federal law enforcement and will maintain an ongoing dialogue with them about the situation.

Impact and response

Current findings indicate the scope of this incident is limited to the Grafana Labs GitHub repositories, which include public and private source code along with internal GitHub repos.

There is no evidence that customer production systems or operations have been compromised.

As part of our standard security practices, we will share additional information from our post-incident review when our investigation is complete.

Grafana Labs is also taking steps to increase security measures to protect our systems. We are currently implementing significant measures to further secure our CI/CD (continuous integration and continuous deployment) pipelines and prevent a recurrence of this type of issue.

Our teams remain focused on the continued investigation and the deployment of increased security controls.

Grafana security release: Critical and high severity security fixes for CVE-2026-27876 and CVE-2026-27880

Mariell Hoversholm — Thu, 26 Mar 2026 04:43:23 +0000

Today we are releasing Grafana 12.4.2 along with patches for Grafana 12.3, 12.2, 12.1, and 11.6, which include critical and high severity security fixes. We recommend that you install the newly released versions as soon as possible.

Grafana 12.4.2 with security fixes:

Download Grafana 12.4.2

Grafana 12.3.6 with security fixes:

Download Grafana 12.3.6

Grafana 12.2.8 with security fixes:

Download Grafana 12.2.8

Grafana 12.1.10 with security fixes:

Download Grafana 12.1.10

Grafana 11.6.14 with security fixes:

Download Grafana 11.6.14

As per our security policy, Grafana Labs customers have received security patched versions two weeks in advance under embargo, and Grafana Cloud has been patched.

We have also coordinated closely with all cloud providers licensed to offer Grafana Cloud. They received early notification under embargo and confirmed that their offerings are secure at the time of this announcement. This is applicable to Amazon Managed Grafana and Azure Managed Grafana.

CVE-2026-27876: SQL expressions arbitrary file write enabling remote code execution

Grafana's SQL expressions feature enables transforming query data with familiar SQL syntax. This syntax, however, also permitted writing arbitrary files to the file system in such a way that one could chain several attack vectors to achieve remote code execution.

The CVSS score for this vulnerability is 9.1 CRITICAL (CVSS link).

The following prerequisites are required for this vulnerability:

Access to execute data source queries (Viewer permissions or higher)
The sqlExpressions feature toggle must be enabled on the Grafana instance.

Impact

An attacker with access to execute data source queries could overwrite a Sqlyze driver or write an AWS data source configuration file in order to achieve full remote code execution. We have confirmed this vulnerability could be exploited to acquire an SSH connection to the Grafana host.

Impacted versions

Grafana versions v11.6.0 and later are impacted by this vulnerability.

Solutions and mitigations

We recommend upgrading to one of the patched versions listed above as soon as possible.

If an upgrade is not immediately possible, the following workarounds reduce risk. Note: these may cause disruption to Grafana users and do not fully remediate the vulnerability.

Option 1: Disable the sqlExpressions feature toggle.

Option 2: Perform ALL of the following:

If you have Sqlyze installed: update to at least v1.5.0 or disable it.
Disable all AWS data sources you have installed.

CVE-2026-27880: Unauthenticated denial-of-service via OpenFeature endpoint

Grafana's OpenFeature feature flag validation endpoints do not require authentication and accept unbounded user input. This input is read into memory.

The CVSS score for this vulnerability is 7.5 HIGH (CVSS link).

Impact

An attacker could crash the Grafana server by sending requests that exhaust available memory.

Impacted versions

Grafana versions v12.1.0 and later are impacted by this vulnerability.

Solutions and mitigations

We recommend upgrading to one of the patched versions listed above as soon as possible.

If an upgrade is not immediately possible, any of the following workarounds reduces risk:

Deploy Grafana in a highly available environment with automatic restarts.
Implement a reverse proxy in front of Grafana that limits input payload size. Cloudflare does this by default. Nginx supports this via explicit configuration.

Timeline and post-incident review

Here is a detailed incident timeline. All times are in UTC.

CVE-2026-27876

Date/Time (UTC)	Event
2025-02-06	sqlExpressions feature reimplemented with MySQL syntax and released in v11.6.0
2026-02-23 13:33	Internal incident declared
2026-02-23 15:08	Grafana Cloud patched
2026-03-09	Private release issued to customers under embargo
2026-03-25	Public release
2026-03-26 04:00	Blog published

CVE-2026-27880

Date/Time (UTC)	Event
2025-06-27	New OpenFeature evaluation endpoint introduced and released in v12.1.0
2026-02-24 13:12	Internal incident declared
2026-02-24 17:49	Grafana Cloud stacks not behind Cloudflare were patched; Cloudflare-backed stacks were not affected
2026-03-09	Private release issued to customers under embargo
2026-03-25	Public release
2026-03-26 04:00	Blog published

Acknowledgements

We would like to thank Liad Eliyahu, Head of Research at Miggo Security, for responsibly disclosing CVE-2026-27876 through our bug bounty program.

CVE-2026-27880 was discovered internally by the Grafana Labs security team.

Reporting security issues

If you think you have found a security vulnerability, please go to our Report a security issue page to learn how to send a security report.

Grafana Labs will send you a response indicating the next steps in handling your report. After the initial reply to your report, the security team will keep you informed of the progress towards a fix and full announcement, and may ask for additional information or guidance.

Important: We ask you to not disclose the vulnerability before it has been fixed and announced, unless you received a response from the Grafana Labs security team that you can do so.

You can also read more about our bug bounty program and have a look at our Security Hall of Fame.

Security announcements

We maintain a security advisories page, where we always post a summary, remediation, and mitigation details for any patch containing security fixes. You can also subscribe to our RSS feed.

Bridge the DevSec divide: Using Grafana Cloud and Miggo for runtime protection

Jonathan Price — Tue, 17 Mar 2026 15:17:47 +0000

Note: This blog post is co-authored by Daniel Shechter, CEO and co-founder of Miggo Security.

Modern runtime security is critical to understand complex systems and detect and protect against attacks, especially in rapidly evolving cloud native architectures. For many security teams, however, achieving deep visibility into runtime risks remains a moving target.

To address this challenge, Grafana Labs and Miggo Security, a leader in AI-driven runtime security and application detection and response (ADR), have partnered on a joint solution to deliver evidence-based runtime security without the typical runtime costs. The solution builds directly on existing production telemetry in Grafana Cloud to help organizations determine what is truly exposed, what actually executes in production, and which vulnerabilities represent real, exploitable risk. Most importantly, it does this without duplicating instrumentation or introducing operational overhead.

In this blog, we’ll explore why securing modern cloud native workloads has traditionally been a challenge, the limitations of traditional runtime approaches, and how building security directly on top of Grafana Cloud represents a major step forward for CISOs, application owners, and platform engineering teams.

The problem: Securing cloud native applications

The shift toward microservices has fundamentally changed the attack surface. As development velocity increases, it’s more difficult than ever to understand how your systems really function, how they might be attacked, or how they’re already under attack today.

Traditional security methods like static application security testing (SAST) and software composition analysis (SCA) rely on scanning our code and dependencies at build time. While this can surface potential issues in our own code and every published vulnerability possibly present in our dependencies, the true positive rate is painfully low: only 2% of discovered dependency vulnerabilities are actually exploitable. The other 98% exists in code the running system never calls. Tools like dynamic or interactive application security testing (DAST/IAST) try to mitigate this problem, but trade false positive rates for intensive setup and maintenance.

While widely utilized, such approaches are no longer sufficient. Teams are currently overloaded with vulnerabilities that turn out to be irrelevant, while actual critical vulnerabilities go unnoticed. The common practice of pushing this noise onto developers or ops teams to manage erodes trust: when 98% of the issues we ship to them have no real risk, developers are right not to trust us.

The promise and pain of runtime security

The solution the industry is gravitating towards is runtime security. If we deploy standard sensors on our running workloads, we can know what services are actually running in production, what they connect to, and what functions they actually call. Not only can we see how connected and sensitive a service is to prioritize it, we can see if a vulnerability is reachable and if the vulnerable section of code is actually called. This information about reachability allows us to immediately focus on the 2% of vulnerabilities that actually matter. We can show owning software teams exactly why a vulnerability matters, building trust rather than hurting it.

This has great promise, but also comes with challenges. There are a few key reasons why achieving visibility into runtime risk is particularly difficult.

Challenge 1: The deployment tax

Runtime security has a cost.

Runtime application self-protection (RASP) of the 2010s proved that deep runtime instrumentation could unlock real application visibility, but at a steep cost: performance drag, vendor lock-in, and fragile upgrades.

eBPF emerged over the 2020s as an answer, sidestepping runtime dependencies by hooking directly into the kernel. This dramatically improves ease of deployment over RASP, and has led to hundreds of vendors leveraging eBPF sensors to collect runtime signals to power a range of security products. This innovation, again, comes with a cost: each eBPF sensor can add 1-3% of compute overhead. There is no standardization for this collection in the security industry, so security teams can end up paying different vendors to collect and store the same data to power their different products, and further pay compute overhead over 10% on their running workloads.

This further strains relationships and erodes trust: platform teams do not want heavy security sensors slowing down their production environments—or worse, breaking them.

Challenge 2: Runtime context

Most runtime security solutions operate at the level of a single process or container, blind to broader context. Cutting through the noise requires understanding the actual threat model: whether an attacker can realistically exploit a vulnerability, whether the vulnerable service is even reachable, and the potential blast radius.

The result is an overwhelmed security team, fielding both static and runtime findings with no principled way of connecting and prioritizing. Fixing everything is impossible, and prioritizing incorrectly means expending energy without reducing risk.

Challenge 3: The affected `programRoutine` problem

In order to understand if you’re running a vulnerable piece of code from a dependency, we need to know the actual pieces of code we’re running. The way this is intended to work is that dependency maintainers include a programRoutine (a specific method or function) in the list of affected systems in a CVE record. Then, we can look for invocations of that function on our running services, and know if the vulnerability is reachable.

This would be great, if dependency maintainers consistently included these definitions in their CVE records. Unfortunately, they don’t; less than 10% of CVE records contain an affected programRoutine. And actually, Grafana Labs is one of those maintainers not consistently including these in our CVEs records. To be clear, this is partially intentional: calling out the affected programRoutine would make it easier to write an exploit. However, this still limits the full promise of runtime visibility.

Enter Miggo and Grafana Cloud: Execution-backed security with zero added complexity

All of these challenges are solvable. While every runtime security vendor has their own bespoke sensor, they do not need to. And this doesn’t require defining new standards; the standards already exist. Runtime security depends on the same core system signals, such as metrics, logs, traces, and profiles, that already exist within most organizations’ observability stacks.

Grafana Labs and Miggo are excited to announce our partnership leveraging these synergies. Grafana Cloud customers already collect the traces and profiles that are at the core of runtime security. Instead of adding new sensors and re-collecting the same signals, Miggo builds directly on Grafana Cloud Traces, the fully managed distributed tracing system powered by Tempo, and Grafana Cloud Profiles, the hosted continuous profiling tool powered by Pyroscope. This solves the deployment tax challenge, as this model requires deploying and maintaining zero additional sensors for security teams.

With Miggo’s advanced vulnerability and system analysis, we also solve the runtime context and affected programRoutine challenges. Miggo’s vulnerability research team finds the affected functions not disclosed in CVEs, giving users true visibility into what vulnerabilities are actively exploitable in their systems. Miggo further analyzes the telemetry data to give security teams a complete picture of their system state, and integrates with security systems to drive mitigation.

In a nutshell, the process works like this:

Step 1: Collect (Grafana Cloud)

Developers collect telemetry data on their systems leveraging the same tools they already use, including direct instrumentation, Alloy, Beyla, or any other OTel collector
Telemetry data is ingested into Grafana Cloud Traces (request flows, service interactions, endpoints) and Grafana Cloud Profiles (what code paths actually ran)

Step 2: Analyze (Miggo)

Ingest via Grafana Cloud APIs, correlate with vulnerability intel
Determine reachability, exposure, blast radius, and exploitability indicators
Build dynamic runtime application graph/inventory

Step 3: Prioritize

Execution-backed prioritization
Focus remediation on what materially impacts risk

Step 4: Mitigate

Miggo WAF Copilot generates vulnerability-specific protections (virtual patches) based on observed traffic patterns.

Use cases

Below, we walk through three examples of how this powerful combination solves critical security challenges.

1. Establishing runtime truth

The first challenge is visibility: security starts with knowing what’s running. Miggo constructs a dynamic runtime application graph and continuously updated inventory of services, APIs, and third parties derived directly from Grafana Cloud telemetry.

By building an inventory and behavioral baseline of the application, MCP server, or agent, customers get an exposure map and Runtime BOM without interrupting developers' processes. Runtime data eliminates outdated architecture maps and confusion regarding what was planned vs. implemented. You get an up-to-date view of what’s actually running.

2. Evidence-based prioritization

Once you establish what the application looks like, you can move beyond guessing to providing an opinion based on runtime truth. Miggo correlates vulnerabilities with Grafana Cloud Traces and Cloud Profiles to determine reachability and execution.

Traces bring environment context into vulnerabilities. Consider two services: an internet-facing API with access to PII in Postgres and Amazon S3 vs. an internal reporting job that runs every 24 hours and never touches user input. To your team, a high vulnerability on the former is more pressing than on the latter.

Profiles flesh out execution. Code can sit on disk and never run, yet still trigger scanner alerts. By analyzing profiles, Miggo determines whether a dependency is actually loaded—and whether a vulnerable function is ever called.

By validating static findings against runtime truth, organizations see a 60–90% reduction in critical vulnerability backlog noise. Because the evidence comes from traces and profiles that engineering and ops already work with, findings land with credibility and help bridge the divide between dev and security teams.

3. Immediate mitigation via WAF Copilot

Runtime mitigation tools like pod termination are blunt instruments: they aren’t tailored to a specific vulnerability or a service’s runtime context, and carry the risk of disrupting production.

To reduce this friction, Miggo guides remediation with virtual patching via WAF Copilot. This provides a faster, less risky alternative to traditional patching, reducing the exposure window immediately while development teams implement permanent code fixes. WAF Copilot understands what a vulnerability does and how it can be exploited, generating precise rules that are robust against multiple exploitation patterns without blocking legitimate traffic.

What’s next? See the solution live at RSAC 2026

We are just scratching the surface of what is possible when you embed security directly into observability workflows. This Miggo and Grafana Cloud integration delivers immediate ROI, including improved audit defensibility with evidence-based reporting and zero added complexity to the observability stack.

If you are attending RSAC 2026, we invite you to see the solution live:

Joint workshop: Join us for our session "Security Observability: Turning Production Telemetry into Real Security Decisions" on March 25 at 3:00 PM.
Live demos: Visit the Miggo Security booth #6573 in the North Hall, or check out one of our two private sessions.

For more details or to schedule a meeting on-site, click here.

To learn more about how Grafana Labs and Miggo Security are turning production telemetry into real security decisions, visit here.

The Grafana Cloud identity blueprint: balancing security and scale

Sarah Constant — Tue, 24 Feb 2026 20:38:39 +0000

If you've ever rolled out Grafana Cloud to a growing engineering organization, this pattern may sound familiar:

Everything feels simple at first. You invite a few teammates, give them access, and dashboards start appearing. Then the team grows. Then the number of stacks grows. Over time, a model that once felt fast and empowering starts to feel risky, difficult to understand, and even harder to undo.

This post is about avoiding that moment.

In a previous post, I described the single-stack model, where Grafana teams and folders handle multi-tenancy inside a single Grafana stack. That model scales well operationally, but only if your identity and access approach scales alongside it.

As organizations grow, a new question tends to surface: How do I give engineers fast and automated access to Grafana without turning my entire Grafana Cloud account into a shared administrative surface?

The answer is not more manual role assignments or more process. Instead, it is a layered identity model, with SCIM acting as the mechanism that allows that model to work reliably at scale.

The starting point: standard Grafana Cloud access

Most teams begin with the default Grafana Cloud access model.

Users are added directly in the Grafana Cloud portal and assigned an organization role such as admin, editor, or viewer. That role is automatically inherited as a basic role by every Grafana stack in the account.

Authentication is intentionally simple. Users can sign in with a username and password or with a social login provider such as Google or GitHub. For small teams, this simplicity is a major advantage. You can be productive almost immediately, without investing time in identity integration.

At the same time, this model relies on an assumption that does not hold indefinitely. It assumes that anyone who needs Grafana access should also have account-wide visibility and permissions.

As usage grows, that assumption becomes difficult to defend. Universal access is harder to audit, harder to govern, and riskier to revoke cleanly. The solution is not to abandon simplicity, but to separate where Grafana is administered from where it is actually used.

Why SCIM matters for operating at scale

For organizations using Okta or Entra ID, SCIM provisioning is what makes a layered identity model practical.

Rather than manually creating and maintaining teams, SCIM allows your identity provider to push users and teams into Grafana automatically, with membership already defined. Your IdP becomes the system of record for who exists and where they belong.

SCIM also enables immediate deprovisioning. When a user is disabled in your identity provider, their access to the Grafana stack is removed in near real-time. This significantly reduces exposure from stale accounts and offboarding gaps.

Just as importantly, SCIM enables day-zero readiness when combined with RBAC, folders, and data source permissions. New engineers arrive to find their dashboards, folders, and team memberships already in place, without manual setup or access requests.

As organizations grow, manual access models do not merely slow things down. They eventually become a source of operational and security risk. With SCIM now generally available for Grafana Cloud stacks, this approach is accessible well before teams reach that breaking point.

Note: SCIM provisioning uses credentials (a token or service account) to automate provisioning. Treat it like an admin credential: store it in a vault/secret manager, rotate it regularly, restrict who can create/manage it, and monitor provisioning activity for anomalies.

Layer one: the Grafana Cloud portal as an administrative vault

The Grafana Cloud portal acts as the control plane for your account. It governs billing, stack creation, support access, and account-level configuration. Because it represents your entire observability footprint, it benefits from being treated as a tightly controlled administrative space.

In practice, this means that access to the portal should be limited to a small group of platform or observability administrators. Most engineers don’t need to sign in to grafana.com at all. Keeping this group small reduces the potential blast radius of mistakes and simplifies compliance and auditing.

Access to the portal is best managed declaratively or through your identity provider. Terraform-based management and private SSO both ensure that authentication strength and lifecycle policies live in your IdP rather than being managed manually in Grafana.

It is also a good practice to maintain a single "break glass" account. This account should not rely on SSO and should be stored securely in a password vault with access reviewed and tested periodically. Its purpose is to ensure you retain access if your identity provider is unavailable.

Bottom line: This layer is intentionally restrictive. The Grafana Cloud portal is designed for operators, not for day-to-day engineering work.

Layer two: the Grafana stack as an automated workspace

The Grafana stack is where engineering teams spend their time. This is where dashboards are explored, alerts are investigated, and incidents are understood.

At this layer, the priority shifts from strict control to predictable automation.

In more security-conscious environments, engineers should authenticate directly against their stack URL, such as your-org.grafana.net. Stack-level SSO reduces the risk that the portal becomes an unintended access path to production data.

User and team creation at the stack level should be driven by your identity provider rather than by manual invitation. Engineers should not have to wait for access to be granted. Ideally, access already exists before they ever sign in.

Even outside of Okta or Entra ID environments, stack-level private SSO using SAML, OIDC, or OAuth allows Grafana to defer authentication decisions to the system you already trust.

This separation is what allows tight governance at the account level while preserving a smooth experience for engineers.

As organizations grow, manual access isn't just slow, it can become a security risk. Using SCIM for Grafana stacks (which is now generally available for Okta and Entra ID) lets you automate the boring parts of access control before things get messy.

Once it’s running, you should set up alerts for your provisioning logs to flag anything unusual, such as:

Unexpected spikes in user creation.
Bulk deletes or disables that weren't planned.
Changes to team memberships in sensitive groups.

You can find the technical details in our SCIM documentation, or speak to your account team if you want to try the Private Preview for Grafana Cloud portal SSO.

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!

Grafana Enterprise security update: critical severity security fix for CVE-2025-41115

Vardan Torosyan — Wed, 19 Nov 2025 18:28:02 +0000

Updated Nov. 28, 2025: In the timeline, the time reported for 2025-11-04 was corrected from 19:14 UTC to 15:59 UTC.

Along with the release of Grafana Enterprise 12.3, we are releasing updated versions of Grafana Enterprise 12.2.1, 12.1.3 and 12.0.6, all of which contain a fix for a critical severity vulnerability (CVE-2025-41115) discovered in the SCIM (System for Cross-domain Identity Management). This issue could allow privilege escalation or user impersonation under certain configurations.

Grafana Enterprise 12.3.0 latest release with security patch:

Download Grafana Enterprise 12.3

Grafana Enterprise 12.2.1 with security patch:

Download Grafana Enterprise 12.2.1

Grafana Enterprise 12.1.3 with security patch:

Download Grafana Enterprise 12.1.3

Grafana Enterprise 12.0.6 with security patch:

Download Grafana Enterprise 12.0.6

Grafana Labs customers received patch versions in advance and appropriate patches have been applied to Grafana Cloud. As always, we closely coordinated with all cloud providers licensed to offer Grafana Cloud Pro. They have received early notification under embargo and confirmed that their offerings are secure at the time of this announcement. This is applicable to Amazon Managed Grafana and Azure Managed Grafana.

Grafana OSS users are not affected by this issue.

Incorrect privilege assignment (CVE-2025-41115)

Summary

SCIM provisioning was introduced in Grafana Enterprise and Grafana Cloud in April to improve how organizations manage users and teams in Grafana by introducing automated user lifecycle management.

In Grafana versions 12.x where SCIM provisioning is enabled and configured, a vulnerability in user identity handling allows a malicious or compromised SCIM client to provision a user with a numeric externalId, which in turn could allow to override internal user IDs and lead to impersonation or privilege escalation.

This vulnerability applies only if all of the following conditions are met:

enableSCIM feature flag set to true (docs on the SCIM provisioning feature)
user_sync_enabled config option in the [auth.scim] block set to true

The CVSS score for this vulnerability is 10.0 Critical.

Impact

Grafana maps the SCIM externalId directly to the internal user.uid; therefore, numeric values (e.g. "1") may be interpreted as internal numeric user IDs.

In specific cases this could allow the newly provisioned user to be treated as an existing internal account, such as the Admin, leading to potential impersonation or privilege escalation.

Impacted versions

The vulnerability impacts Grafana Enterprise running on the following versions:

Grafana Enterprise 12.0.0 to 12.2.1

Appropriate patches have been applied to Grafana Cloud.

Solutions and mitigations

If your instance is vulnerable, we strongly recommend upgrading to one of the patched versions as soon as possible.

Timeline and post-incident review

Here is a detailed incident timeline starting from when we originally introduced the issue. All times are in UTC.

2025-11-04 15:59 As part of internal audit and testing, we discovered that there is a scenario where user IDs can be overwritten when using SCIM with specific configuration.
2025-11-04 16:30 Internal incident declared. CVE-2025-41115 reserved.
2025-11-04 16:45 We concluded that the vulnerability has not been exploited in Grafana Cloud. Introduced immediate patch.
2025-11-05 17:52 Private release.
2025-11-19 10:33 Public release.
2025-11-19 19:30 Blog post published.

Reporting security issues

If you think you have found a security vulnerability, please go to our Report a security issue page to learn how to send a security report.

Important: We ask you to not disclose the vulnerability before it has been fixed and announced, unless you received a response from the Grafana Labs security team that you can do so.

You can also read more about our bug bounty program and find out who has made our Security Hall of Fame.

Security announcements

We maintain a security category on our blog, where we will always post a summary, remediation, and mitigation details for any patch containing security fixes. You can also subscribe to our RSS feed.

Single-tenant vs. multi-tenant architecture with Grafana Cloud: A guide to choosing the right approach

Naima Alexander — Thu, 18 Sep 2025 15:29:37 +0000

Grafana Cloud’s flexibility is one of its greatest strengths, but the breadth of choices can sometimes be overwhelming. We see this a lot when it comes to selecting the right architectural approach, with organizations unsure of how many stacks they need to host their environment.

Grafana Cloud provides robust features for managing tenancy, enabling organizations to effectively handle diverse teams and projects. In this blog, we’ll explore the advantages and disadvantages of managing Grafana Cloud tenants through multiple stacks for data isolation and permission control. We'll also look at the pros and cons of maintaining a single stack and managing permissions via role-based access control (RBAC) and labeling.

And while we recommend using a single-stack approach in most cases, this should help you and your organization come to an informed decision about the architectural approach that best fits your specific needs.

The pros and cons of maintaining a single-stack architecture with Grafana Cloud

Role-based access control, or RBAC, paired with a single-stack approach provides a standardized way of granting, changing, and revoking access when it comes to viewing and modifying Grafana Cloud resources such as dashboards, reports, and administrative settings. RBAC also provides the ability to create custom roles that go beyond “Admin,” “Editor,” and “Viewer” to control who can access which resources d on a more granular level.

Taking that a step further, you can use label-based access control, or LBAC, to create access policies so users can only query the metrics or logs data that meet specific label requirements. LBAC is useful if you want to separate access to the telemetry backends within a single stack. Instead of giving someone access to all of the metrics or logs within a stack, you only give them access to a specific label set of the metrics or logs in that stack.

The LBAC feature allows you to associate multiple sets of Prometheus label selectors with a policy. As a result, queries only return data that match at least one of the provided selectors.

Advantages

Organized configuration: LBAC rules can easily be attached to Cloud Access Policies
Centralized management: Managing users, roles, and permissions in one place is easier than configuring and maintaining multiple instances. This reduces administrative overhead.
Simplified maintenance: Updates, backups, and maintenance tasks are simplified with a single instance. You can apply changes across all dashboards without having to replicate efforts across multiple stacks.
Easier integration: Integrating with other tools and systems (like authentication providers) is more straightforward in a single stack, reducing complexity in the overall architecture.

Disadvantages

Risks of misconfiguration: If LBAC is misconfigured, it could lead to potential data leaks.
Limited granularity: Different teams and departments may have unique requirements for their dashboards, and labels and RBAC don’t provide the level of granularity around managing permissions that having multiple stacks does.

The advantages and disadvantages of a multi-tenant architecture with Grafana Cloud

With a multi-tenant architecture approach, you create a stack with a distinct UI frontend for every team that needs one. Each team logs into the same Grafana Cloud instance, accessing their distinct stack backends as data sources.

Multiple production stacks are recommended when the goal is complete isolation among departments or teams and you don't expect any need to combine data or resources from different stacks. This is typically used by resellers and managed service providers that have customers who shouldn’t have access to one another’s data.

Advantages

Performance: This approach ensures better query performance and limits the impact of teams, departments, or clients on one another.
Cost attribution: When you’re using multiple stacks and each stack belongs to a team, it’s easy to attribute costs to each team, client, or department.
Centralized billing: Each Grafana Cloud instance of each stack includes a grafanacloud-usage data source that powers the included billing dashboard. On the central billing dashboard it’s easy to see the usage of each telemetry signal per stack.
Data segregation: Each team, department, or client can have its own stack, ensuring sensitive data isn't accessible to unauthorized users from other teams.
Compliance: Isolating data can help meet regulatory requirements. It’s easier to control data access and ensure compliance within distinct environments.
Customizability: Teams can customize their own stack settings, plugins, and dashboards without impacting others, allowing for tailored solutions.
Easier dashboard sharing: You have several options to share dashboards. If all stacks are in the same region, you can leverage multi-stack querying, so you can centralize visibility across environments without duplicating data.

Disadvantages

Cross-tenant querying: Querying all tenants together by the admin team requires additional configurations, and the admin team may need to log into each Grafana Cloud instance or consolidate metrics tenants into one stack.
Correlation challenges: Grafana Labs recommends customers use a single stack as their production observability environment because cross-stack correlation won't function properly.
Consistency and best practices: Ensuring consistent practices and standards across multiple stacks can be challenging, potentially leading to varying levels of quality and performance.

Make the choice that's right for your organization

Choosing the right architectural approach in Grafana Cloud ultimately depends on your organization’s priorities around isolation, collaboration, compliance, and administrative overhead.

Multiple stacks provide strong boundaries, cost attribution, and customization, making them well-suited for scenarios where complete isolation is non-negotiable, such as with managed service providers or highly regulated environments.

On the other hand, a single-stack architecture with RBAC and LBAC offers simplicity, centralized management, and easier integrations, which often aligns better with organizations seeking operational efficiency and cross-team visibility. This approach is recommended for most Grafana Cloud customers, since they are managing their own stacks and aren't under strict regulatory requirements.

There’s no one-size-fits-all answer, but by weighing the trade-offs outlined here, you can select the model that best balances your technical requirements with your business goals.

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!

Managing access in Grafana: a single stack journey with teams, roles, and real-world patterns

Sarah Constant — Mon, 08 Sep 2025 15:56:59 +0000

When multiple teams use Grafana, it can start to feel a bit messy. Dashboards pile up, permissions become unclear, and teams accidentally overwrite each other’s work.

To help you and your organization stay clear, collaborative, and secure, we recommend putting all users in a single Grafana Cloud stack and managing access with teams, roles, and folders. To illustrate this, I'll share a hypothetical example of how you can put this into practice across three teams.

Let's dive in!

Three teams, one stack

Let’s imagine that it’s Monday morning, and I’m sitting down with my coffee ready to start the work day. In this hypothetical, I'm part of the observability team serving an engineering organization of 50 people, and my goal today is to set up Grafana for our various engineering teams.

As part of this process, I want to reduce setup and maintenance work. I also want to make sure teams can access the right dashboards and data, avoid stepping on each other’s work, and still collaborate when needed. I do all of this in a single Grafana Cloud stack—no multi-stack complexity required.

In our setup, we have three teams:

Team A builds product features and needs autonomy with their own dashboards and data sources.
Team B handles data engineering and needs autonomy with their own dashboards.
Team C is our observability team (that’s us), and we act as admins of the Grafana stack.

Note: For the purpose of this blog post, we'll focus on the steps you'd need to follow in Grafana Cloud. However, the same basic concepts can be followed in Grafana Enterprise.

1. Create the stack

With a single stack, I get unified SSO, centralized configuration, and plugin management. Equally important: Having everything under one stack means teams are not isolated in silos. They can discover each other’s dashboards, share work, and learn from one another.

When everyone works in a single stack, collaboration becomes much easier. A dashboard built by one team can be shared directly with others, without duplicating it across stacks. Teams can choose to open specific folders or dashboards, encouraging re-use and cross-learning while still maintaining ownership of their own work.

Shared spaces, like an “Everyone” folder, make it simple to publish executive dashboards or cross-team metrics that all groups can benefit from. And because everything lives in one environment, users are easily set up to adapt when teams reorganize, collaborate across groups, or swarm together on new projects.

2. Create teams and their folders

I start by adding new users with the Viewer role by default. It’s a safe baseline: they can see everything, but they can't change anything.

Each team needs to be added to the same stack and to a Grafana Team. To do this, I create a folder for each team that they can manage and grant them the fixed:teams:read fixed role. This means they can share items in their team folder with other teams, to encourage collaboration and learning from each other.

I upgrade Team C to the Admin role so that they can manage all resources in the instance.

To encourage creativity while keeping things tidy, I create a personal folder for each team member that they can manage for drafting content and then moving it into their team folder when ready.

3. Design top-level folder structure

I will limit the number of top-level folders to keep things organized and secure. We will have an "Everyone" folder for shared items that all teams can manage, an "Admins" folder for sensitive content only we can access (Team C’s folder), and one folder per team.

This setup helps users quickly understand where to go, what they can access, and what they can manage.

4. Configure data access

Next, I focus on how teams interact with data. All teams forward metrics to a shared database tenant, but not everyone should see all of the data. I create one shared metrics data source and grant each team query access only to their slice of the data, based on label based access controls (LBAC) per team. This way, we maintain a central observability pipeline but still preserve data separation.

I grant the datasources:explorer fixed role to all teams so they can use the Drilldown app for easily exploring data sources.

Team A, however, needs to build and manage their own data sources for product-specific use cases. For them, I grant the datasources:creator fixed role so they can create and manage their own data sources independently.

5. Consider non-folder resources and stack-wide permissions

Not all Grafana resources live in folders. Service accounts, alert contact points, Fleet Management collectors, and other feature resources are managed at the stack level. For these, I check the role options in the team management UI and assign fixed roles to teams carefully.

For example, I grant teams the Frontend Observability writer fixed role so that they can create and manage services.

6. Scale it with Terraform and SSO

Once the model is working, I codify it.

All new users get added from an Identity Provider through SCIM (I use Okta in this example). As they sync they are automatically assigned to the right team based on their Okta group membership. I use role sync logic to assign the Admin role for our team instead of the default, which I’ve set to the Viewer role.

I use Terraform to provision teams their folders, fixed roles, and shared datasource LBAC rules.

Let's say I need to add a new team called Team D. Now I’ll only need to push new Okta groups to Grafana and run my Terraform script, and then they’ll automatically be set up and ready to start using Grafana.

Where to go from here

Getting access and permissions right in Grafana is one of the best ways to unlock its full potential. With the right structure in place, teams can collaborate more easily, stay secure without extra overhead, and build on each other’s work instead of starting from scratch. To really get a deeper understanding of these ideas, here are a few resources that can help.

Our least privilege custom role explainer blog walks through how to design roles that keep things simple and safe, so people have just the access they need. My LBAC for metrics data sources demo shows how you can give every team a clear view of their own data while still benefiting from a shared pipeline. And our introducing SCIM blog covers how to connect Grafana to your identity provider, making it easy to bring new users on board and keep permissions in sync as your organization grows.

Taken together, these resources will help you put the single-stack model into practice and set your teams up to get the very most from Grafana.

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!

Canary tokens: Learn all about the unsung heroes of security at Grafana Labs

Mostafa Moradian — Mon, 25 Aug 2025 15:44:51 +0000

We've written at length about a recent security incident tied to a GitHub action, but what we haven't publicized yet is how we found the trail left by the attacker: canary tokens.

To quickly recap the security incident in question, on April 26 a vulnerable GitHub Action workflow allowed an unauthorized user access to a limited number of tokens.

Our investigation into the incident concluded May 12, and we confirmed that there were no code modifications, unauthorized access to production systems, exposure of customer data, or access to personal information.

After the initial incident was triggered (but before our investigation concluded), the attacker deleted their fork, PRs, and workflow runs in an attempt to cover their tracks. This led to unauthorized access to secrets across five public repositories, though this did not impact production systems or customer data either. We disabled affected workflows, rotated all exposed credentials, and audited our repositories with Trufflehog, Zizmor, and Gato-X. We also conducted a full access log review in Grafana Loki and other data sources. You can read the initial announcement here and the PIR here.

Without canary tokens and our watchful team, it would have taken longer to detect the attack. In this blog, we'll explain what canary tokens are and show you how they've helped us. We'll also tell you how they can help you detect intrusions before they become serious security incidents.

What are canary tokens? And how do we use them at Grafana Labs?

Canary tokens are digital tripwires or decoys that look valuable to an intruder but have no legitimate use. If someone finds and uses one you've deployed, you'll receive an immediate alert.

Named after coal-mining canaries (early warning for toxic gas), these tokens are far lighter than honeypots. They can take many simple forms, such as API keys, files, URLs, or DNS entries, making them fast and easy to deploy across your environment.

In our case, canary tokens weren’t just an experiment, they were the primary signal that told us an attacker was inside: An AWS API key was validated by the attacker; we got a real-time alert; teams swarmed; and the intrusion was contained within minutes.

Here are the actionable lessons from that detection and why placement and deployment matter more than the token itself:

Speed wins: Canary tokens turn hours of triage into minutes of containment.
Placement matters: Choose the right token types and place them where attackers actually look.
Precision in pinpointing: Utilize organization-level tokens for broad compromise detection and repository-level tokens to identify specific entry points.
Metadata is gold: Comprehensive reminder text, token names, and locations are crucial for efficient triage.
Integration is key: Canary tokens only shine when paired with clear naming, automated alerting, and integrated response workflows.
Iterative scaling: Begin with a small-scale placement in high-risk areas. Only then, by refining your response strategy, can you scale use of canary tokens.

How we built our canary token infrastructure

There are two options for deploying canary tokens:

DIY setup: Generate tokens (API keys, fake files, custom URLs) yourself and build or plug into your existing monitoring system to catch any token use. This gives you full control over token behavior and alert handling, making it ideal for custom platforms.
Automated platforms: Solutions like Thinkst or TraceBit offer out-of-the-box token creation, monitoring, and notifications across major cloud providers and apps. For any bespoke tokens or corner-case scenarios, you’ll still need to handle monitoring yourself to ensure complete coverage.

While a DIY approach could be sufficient for a relatively small environment, we needed something that could scale without manual intervention. So, after experimenting with Thinkst’s OSS and free editions, we switched to their cloud version for features missing in OSS, such as undetectable tokens, a robust API, audit-trail logging, and a full authentication/authorization system.

To make the most out of our setup, we connected the notifications of the Thinkst platform to Grafana Cloud IRM webhooks, as shown below:

The webhook is then being listened to by the Grafana Cloud IRM integration:

This setup allows us to route canary tokens’ notifications to Slack (as you can see in the Routes section above) and automate escalation chains (as you can see in the screenshot below). If you'd like to do something similar, further details on Grafana Cloud IRM integrations are available here.

Canary token lifecycle

We use a simple, four-step lifecycle to operationalize canary tokens and turn them into an actionable detection and response capability:

Create: Generate a unique token, like an AWS API key, PDF link, or DNS subdomain that “phones home” when accessed.
Place: Embed tokens where attackers would look, like code repos, config directories, vaults, cloud consoles, databases, or shared drives.
Monitor: A background service watches for any token use and sends an alert with metadata (timestamp, IP, hostname). Keep in mind: If you’re not using a platform, you must build custom monitoring.
Respond: Investigate alerts, isolate affected assets, and remediate before any real data is touched. Keep the triggered token for forensics and replace it immediately.

Canary tokens and TruffleHog to the rescue!

We planted AWS API key canary tokens (alongside a few others) in organization- and repository-level secrets. When the attacker exfiltrated the secrets and ran TruffleHog to validate them, TruffleHog’s sts:GetCallerIdentity call against our tokens triggered the canary, unlike free canary tokens that are automatically detected by TruffleHog without calling the AWS API.

Because Thinkst was watching those AWS accounts, we got an immediate Slack alert. That one notification, thanks to our strategic token placement, revealed the exact entry point. The Detection & Response team reacted to the notification and raised an incident. Teams across R&D and Security sprang into action to contain and remediate the incident, which, again, resulted in no code modifications, unauthorized access to production systems, exposure of customer data, or access to personal information.

Our canary token placement strategy

Given GitHub's central role as our primary source of truth, safeguarding it with canary tokens is crucial. Specifically, we strategically deploy AWS API key canary tokens within GitHub Secrets by placing the tokens at the organization level (to flag any repo breach) and per repository (to precisely pinpoint intrusions at the repository level). Other platforms get their own flavors. For example, Google Docs tokens live in shared Google Drive files, email-based tokens in dummy inboxes, and DNS tokens in unused subdomains.

The key is to map out every asset (repos, CI/CD pipelines, cloud consoles, vaults, pods, containers, etc.) and sprinkle a diverse mix of tokens wherever an attacker might snoop. We’ve deployed tens of thousands (or an infinite amount, for that matter), so watch your steps, since there’s a tripwire behind every click.

How you can utilize canary tokens in your security setup

Now that we've walked through how we deploy canary tokens, we want to share some tips for how you can utilize them as well. For the purposes of this blog, we'll focus our advice on the same tools we use internally (because we believe in them as tried and tested technologies), but most of the general guidance still applies for other similar tools.

Shaping your placement strategy with team discussion

Some key questions you should consider before deciding on your canary placement and alerting strategy include:

Which token types catch attackers best?

Pick from Thinkst’s catalog (over two dozen types) to match your environment and threats. Mix cloud/API keys (AWS, Azure, Slack) with file- or link-based tokens (PDFs, DNS, URLs) so you cover code repos, docs, and networks. Use a variety to broaden detection.

How do you make alerts impossible to miss?

Tune your alert thresholds and channels to strike the right balance between urgency and noise. Consider:

Acknowledge alerts early: Acknowledge alerts promptly on the Thinkst platform to avoid grouping subsequent alerts and missing notifications of repeated canary token triggers from different IP addresses.
METADATA IS IMPORTANT: For quicker investigation, ensure every notification includes canary token metadata (name, location, reminder text). The Thinkst platform automates this if the reminder text is correctly configured.

Where do alerts need to land?

Choose channels your team actually watches:

Immediate channels: Slack (with webhooks) or the Grafana Cloud IRM mobile app for 24/7 paging
Secondary channels: Security mailing lists, SIEM, or ticketing (e.g., Splunk) for structured incident tracking

Map each severity level to the right channel so nothing slips through.

Where should tokens live?

Sprinkle tokens in high-risk and high-value spots:

CI/CD pipelines: As dummy environment variables, test jobs, or secondary workflows
Private GitHub/GitLab repositories: In config files, comments or branch-specific .env files
Developer workstations & build servers: In ~/.aws/credentials, hidden directories or registry entries
Cloud accounts & consoles: Nested within IAM policies or metadata documents
Shared network drives & documentation: In ordinary looking directories or PDFs
Public GitHub/GitLab repositories: Be cautious here! Alert fatigue is a real thing. Limit use of canary tokens in public repositories to avoid false positives from automated secret scanners triggered by inadvertently exposed sensitive information like AWS API keys in Git branches (i.e., obvious places).

How do you prevent accidental triggers?

Educate teams: Explain the “what” and “where” of canary tokens.
Demo alerts: Show how tokens work and what notifications look like.
Announce deployments: Share rollout plans in team channels.
Document “off-limits” zones: Post common token names and locations in your wiki.
Use runbooks: Guide responders through investigations to avoid confusion.

You have two communication strategies for canary tokens:

Silent: No internal notice, which is better for catching insiders but higher false positives.
Communicative: Teams know about tokens, with fewer false positives but risk tipping off attackers.

Who are you trying to catch?

Identify target adversaries for canary tokens. Understand that advanced attackers use unique methods, so tokens alone are insufficient for full access prevention.

External attackers probing repos or cloud assets
Automated scanners hunting for exposed secrets
Insider threats snooping on token-laden areas
Supply-chain attackers targeting CI/CD
Phishing or social-engineering via document or email tokens

By answering these, you’ll align token types, alerting, and placement with your organization’s risks and tools, thus turning every token into a reliable tripwire.

Best practices, limitations, and gotchas

Canary tokens provide valuable, high-confidence alerts of compromise, but triggering one means an attacker is already inside your system. While effective, they are not a foolproof solution and have limitations.

Provider support varies: Not every token type is available on every platform. If you build your own bespoke API or secret-generation service, you’ll need to roll your own monitoring. One way to do that is to log token IDs (never the secret!) into Grafana Loki, set up Grafana Alerting alerts on those IDs, and feed incidents into Grafana Cloud IRM.
Monitoring availability: If your token-watching infrastructure goes down, or worse, becomes compromised, you lose visibility entirely. So, secure your infrastructure!
False positives from insiders: Before we formally announced our token rollout, a handful of teammates accidentally tripped tokens simply by poking around. By asking them why they accessed those credentials, we quickly labeled those alerts as benign and moved on.
Obfuscated attacker source: VPNs hide real IPs, so location data can be misleading. Some browser-based tokens can add fingerprinting, but that only works for web-triggered tokens.
API rate limits: Even when vendors advertise “unlimited” API calls, their infrastructure typically enforces limitations. Blast too many token creation or update requests at once and you’ll hit throttling or even break the API.

Here are the key best practices we’ve learnt over time:

Use descriptive reminders: Leverage Thinkst’s “reminder” text field to tag each token with owner, environment, and location. You can also store JSON in that field and retrieve it later via the UI or API. That metadata lets you pinpoint precisely what was touched.
One token per location: When a token fires, archive it for forensics, since an attacker might reuse or sell it, then replace it ideally with a fresh name that blends in (e.g., AWS_API_CREDS instead of CANARY_TOKEN_AWS).
Group into flocks: Organize tokens by environment, team, or platform into flocks to simplify bulk management.
Integrate alerts into incident response: Route notifications into your messaging system, your SIEM, and/or ticketing system so that nothing slips through the cracks.
Mix token types: Don’t rely solely on AWS API keys. Deploy file, URL/DNS, email, and command tokens to cover every attacker tactic.
Automate placement: Use Thinkst’s scripts or IaC examples (Terraform, Ansible, Puppet, and AWS CloudFormation) to ensure consistency. We also worked closely with Thinkst support to master their UI, API, and tooling and learn some best practices.
Test your tokens and document actions: Trigger a few tokens in a safe environment after deployment to verify alert delivery and downstream automation. This will also enhance your incident response plan. This practice allows you to simulate real-world scenarios and refine your response procedures. Document these simulated incidents and the corresponding steps taken to create comprehensive runbooks. These runbooks will serve as valuable guides when canary tokens are triggered in the future, ensuring a more efficient and effective response and less confusion.
Communicate widely and wisely: Make your teams (or even your broader community) aware of canary tokens. That transparency reduces accidental triggers and builds security awareness.

By understanding these limitations and following these practices, you’ll keep your canary token deployment lean, reliable, and ready to catch real attacks the moment they unfold.

Conclusion

Canary tokens have been a force multiplier for us, turning every corner of our infrastructure into a potential tripwire and giving us the precious minutes needed to stop a breach before it escalates.

Seed your environment with right token types, integrate alerts for automation, and never underestimate the advantage of knowing an adversary is inside before they ever touch your real assets.

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!

Grafana Labs blog on Grafana Labs — tagged 'security'

How to scale access control in Grafana Cloud

Meet AcmeCloud (and the challenge of manual provisioning access)

Establishing a single source of truth with SSO and SCIM

Building a layered access model with RBAC

Restricting dashboard visibility with teams and folders

Enforcing data-level isolation with LBAC

Delivering the right experience for every user

Bringing it all together

Post-incident review for TanStack npm supply chain ransom incident: No unauthorized access to customer production systems

Summary and impact

Grafana Labs’ response

Incident timeline

What’s next

Grafana Labs security update: Latest on TanStack npm supply chain ransomware incident

Summary and background

Impact and response

Grafana security release: Critical and high severity security fixes for CVE-2026-27876 and CVE-2026-27880

CVE-2026-27876: SQL expressions arbitrary file write enabling remote code execution

Impact

Impacted versions

Solutions and mitigations

CVE-2026-27880: Unauthenticated denial-of-service via OpenFeature endpoint

Impact

Impacted versions

Solutions and mitigations

Timeline and post-incident review

Acknowledgements

Reporting security issues

Security announcements

Bridge the DevSec divide: Using Grafana Cloud and Miggo for runtime protection

The problem: Securing cloud native applications

The promise and pain of runtime security

Challenge 1: The deployment tax

Challenge 2: Runtime context

Challenge 3: The affected programRoutine problem

Enter Miggo and Grafana Cloud: Execution-backed security with zero added complexity

Step 1: Collect (Grafana Cloud)

Step 2: Analyze (Miggo)

Step 3: Prioritize

Step 4: Mitigate

Use cases

1. Establishing runtime truth

2. Evidence-based prioritization

3. Immediate mitigation via WAF Copilot

What’s next? See the solution live at RSAC 2026

The Grafana Cloud identity blueprint: balancing security and scale

The starting point: standard Grafana Cloud access

Why SCIM matters for operating at scale

Layer one: the Grafana Cloud portal as an administrative vault

Layer two: the Grafana stack as an automated workspace

Grafana Enterprise security update: critical severity security fix for CVE-2025-41115

Incorrect privilege assignment (CVE-2025-41115)

Summary

Impact

Impacted versions

Solutions and mitigations

Timeline and post-incident review

Reporting security issues

Security announcements

Single-tenant vs. multi-tenant architecture with Grafana Cloud: A guide to choosing the right approach

The pros and cons of maintaining a single-stack architecture with Grafana Cloud

Advantages

Disadvantages

The advantages and disadvantages of a multi-tenant architecture with Grafana Cloud

Advantages

Disadvantages

Make the choice that's right for your organization

Managing access in Grafana: a single stack journey with teams, roles, and real-world patterns

Three teams, one stack

1. Create the stack

2. Create teams and their folders

3. Design top-level folder structure

4. Configure data access

5. Consider non-folder resources and stack-wide permissions

6. Scale it with Terraform and SSO

Where to go from here

Canary tokens: Learn all about the unsung heroes of security at Grafana Labs

What are canary tokens? And how do we use them at Grafana Labs?

How we built our canary token infrastructure

Challenge 3: The affected `programRoutine` problem