This is documentation for the next version of Grafana documentation. For the latest stable release, go to the latest version.

Documentation Grafana documentation Administration Data source management LBAC for data sources Configure LBAC for Tempo or Cloud Traces

Configure LBAC for Tempo or Cloud Traces

This document explains how to configure Label-Based Access Control (LBAC) for data sources for a Tempo or Cloud Traces data source. LBAC for data sources lets you restrict which spans a team can access by defining rules based on trace attributes, providing fine-grained, team-based access control within a single tenant. It mirrors the experience used for logs and metrics LBAC in Grafana Cloud.

Grafana uses LBAC as an umbrella term for label-based access control across all data sources. Traces use attributes, not labels, for access control, but the Grafana UI surfaces this functionality as LBAC for consistency.

How LBAC works

When a user queries a tracing data source, Grafana evaluates the user’s team memberships and the LBAC rules assigned to those teams. Grafana adds these rules to the request, so the Cloud Traces data source returns only permitted spans or attributes.

LBAC rules use attribute selectors, such as:

traceql Copy

{ resource.service.name="checkout", resource.env="prod" }

Multiple conditions in the same rule are combined with AND (,), while multiple rules across teams are combined with OR.

If none of a user’s teams have LBAC rules for the data source, that user can query all traces, subject to their data source permissions.

Before you begin

Before you configure LBAC for data sources, ensure you have the following:

• Permission to create a Tempo or Cloud Traces tenant in Grafana Cloud.
• Administrator permissions for Grafana.
• A team set up in Grafana.

Known limitations

Be aware of the following limitations for traces LBAC:

• LBAC for traces is available only at the team level. Data source-level LBAC rules, configured through cloud access policies, aren’t currently supported for traces. This differs from logs and metrics LBAC, which supports both team-level and data source-level rules.
• Autocomplete in search is still under development.
• Rules are restricted to resource scope attributes.
• Users with multiple rules may experience a slight performance degradation.

Configure a new Tempo or Cloud Traces data source

Follow this workflow when adding a new data source. The data source must be hosted by Grafana and not self-managed.

1. Start your Grafana Cloud instance.

2. Access Tempo or Cloud Traces data source details for your stack.

3. Copy the details of your Tempo or Cloud Traces setup.

4. In Grafana Cloud, navigate to Administration > Users and access > Cloud Access Policies.

   • Create an access policy for the Tempo or Cloud Traces data source.
   • Ensure the access policy includes traces:read permissions.
   • Ensure the access policy doesn’t include labels rules. Data source-level LBAC isn’t supported for traces, so any labels rules in the access policy are ignored.

5. In Grafana, select the Tempo or Cloud Traces data source, or create a new one.

6. Navigate back to the Tempo or Cloud Traces data source.

   • Set up the data source using basic authentication. Use the userID or tenantID as the username. Use the token from your access policy as the password.
   • Select Save & test.

7. Go to the Permissions tab of the newly created data source. Here, you find the LBAC for data sources rules section.

8. Choose a team from the Team drop-down.

9. Enter attribute selectors, such as:

   traceql Copy

   { resource.service.name="checkout", resource.env="prod" }

10. Select Save.

Examples of LBAC rules

An LBAC rule for traces is an attribute selector that filters spans based on trace attributes. Each rule operates independently as its own filter, separate from other rules within a team.

Follow these guidelines when you write rules:

• Use only resource scope attributes, for example { resource.env="prod" }.
• Use double quotes for string values. Only string values are supported.
• Use the =~ operator for regular expression matching, for example { resource.team =~ "team-a|team-b" }.
• If you use negation (!= or !~), refer to Troubleshoot missing service traces. TraceQL negation behaves differently from PromQL when an attribute is missing.
• Use up to two conditions in a single rule, separated by a comma (,), which acts as an AND operator.

For more detail, refer to Create LBAC for data sources rules for a supported data source.

Single rule

Multiple conditions in a single rule are combined with AND. The following rule grants access to spans that match both resource.service.name="payments-api" and resource.env="prod":

traceql Copy

{ resource.service.name="payments-api", resource.env="prod" }

Match multiple values with a regular expression

Use the =~ operator to match an attribute against a regular expression. The following rule grants access to spans from team-a or team-b:

traceql Copy

{ resource.team =~ "team-a|team-b" }

Multiple rules

Multiple rules assigned to the same team are combined with OR. The following two rules grant access to spans in the prod environment or spans from the billing team:

traceql Copy

{ resource.env="prod" }
{ resource.team="billing" }

User on multiple teams

Users on multiple teams receive access based on all the combined rules assigned to each team, combined with OR. For example, if the teams have the following rules:

• Team A: { resource.cluster="us-east-1" }
• Team B: { resource.service.name="frontend" }

A user on both teams can access spans that match resource.cluster="us-east-1" or resource.service.name="frontend".

How LBAC affects returned data

How LBAC filters data depends on the endpoint:

• Trace-by-ID lookups use one of three redaction modes, described in the following sections, that control how non-matching spans are handled.
• Search, metrics, and autocomplete endpoints always return only the spans that match the LBAC rules, regardless of the configured redaction mode.

Cloud Traces supports three redaction modes for trace-by-ID lookups. The active mode is configured per tenant. To change the mode for your organization, contact Grafana Support.

Attributes mode (default)

Non-matching spans are included in the response, but their attributes and intrinsics are redacted. Only the following minimal fields remain visible on redacted spans:

• traceId, spanId, parentId
• name, kind
• timestamps
• status

You can extend the set of always-visible fields by configuring allowed_attributes, which sets a list of scoped attributes and intrinsics that are never redacted. For example, you can set span:name, resource.service.name, event:name, or scope.version so they’re never redacted, even on non-matching spans. Contact Grafana Support to configure allowed_attributes for your organization.

Spans mode

Non-matching spans are removed from the response entirely. This can result in broken traces where a returned span has no matching parent, creating gaps in the span tree.

Error mode

If any span in a requested trace doesn’t match the LBAC policy, the entire request returns a 404 error. This is the strictest mode and suits environments where partial trace visibility isn’t acceptable.

Troubleshoot missing service traces

If a team can’t see traces from a service you expect, a negation rule combined with a missing attribute may be excluding those spans.

If you’re familiar with PromQL label matching, TraceQL attribute selectors behave differently for negation.

In PromQL, labelFoo != "abc" matches series that don’t have labelFoo or have a value other than "abc". In TraceQL, resource.attr != "abc" and resource.attr !~ "abc" match only spans that have resource.attr and whose value matches the condition. Spans that don’t include that attribute aren’t returned.

This difference matters when you combine attributes in one rule with , (AND). For example, this rule matches only spans from checkout-api that also have the resource.k8s.namespace.name attribute. Services such as checkout-api that don’t set resource.k8s.namespace.name won’t match, even if you want to include them.

Copy

{ resource.service.name="checkout-api", resource.k8s.namespace.name !~ "prod-main" }

To include services that don’t set the attribute, add a separate rule for each service. Multiple rules for a team combine with OR:

Copy

{ resource.service.name="checkout-api", resource.k8s.namespace.name !~ "prod-main" }
{ resource.service.name="checkout-api" }

Add similar OR rules for other services that don’t set the namespace attribute.

To find services that don’t set an attribute, run this query in Grafana Explore:

Copy

{ resource.k8s.namespace.name = nil } | rate() by (resource.service.name)

This query returns services that don’t have resource.k8s.namespace.name set.

Manage LBAC rules

You can edit or delete existing LBAC rules from the data source’s Permissions tab.

Edit an LBAC rule

1. Open your stack and select your Tempo or Cloud Traces data source.
2. Select Permissions.
3. Scroll to Data access.
4. Select the rule you want to edit and select the pencil (edit) icon.
5. Modify the Attribute filters.
6. Select Save.

Delete an LBAC rule

1. Open your stack and select your Tempo or Cloud Traces data source.
2. Select Permissions.
3. Scroll to Data access.
4. Select the rule you want to delete.
5. Select the X (delete) icon.
6. Confirm the deletion.

Related resources

• LBAC for data sources overview explains how LBAC works, which data sources are supported, and current limitations.
• Create LBAC for data sources rules for a supported data source explains how to define and manage rules.
• Construct a TraceQL query explains attribute selector syntax, including !=, !~, and = nil.
• Search traces using the query builder explains how to run the diagnostic query in Explore.