Federation-frontend on Grafana Labs

Cross-tenant query federation

Thu, 24 Jul 2025 15:08:34 +0000

Cross-tenant query federation

GET supports creating access policies that can span multiple tenants. Viewers in Grafana Enterprise can see data coming from more than one tenant simultaneously.

By default, the GET gateway service is used as the target for querying GET (for example, using trace ID or TraceQL queries) and sending trace span data (using OTLP gRPC). Cross-tenant query federation, however, requires an extra component called the federation-frontend service.

Architecture

The federation-frontend service acts as the target for queries instead of the GET gateway service. The diagram below shows this as a visualization:

The data from multiple tenants is aggregated in a single trace lookup or search query as multiple queries to the GET gateway. The gateway then carries out these queries and returns the data to the federation service which combines them as a final query result.

The federation-frontend runs with tenant federation enabled by default. The federation-frontend service can be deployed independently from a GET installation, however, there are certain circumstances in which some queries are not valid in this case.

Cross-tenant queries are made by sending queries via the federation-frontend service that include basic authentication with a username signifying that multiple tenants should be inspected. In this case, the username can be either tenant names or a wildcard to search all tenants when configuring the Authentication in the GET plugin. These tenants must be accessible using the same access policy token and separated by the pipe (|) symbol, or using the wildcard (*) symbol. Using the wildcard symbol searches all tenants that can be queried by the supplied access token.

This page covers the ability to query data from multiple tenants at once.

Before you begin

The procedures on this page assume there are two GET tenants: team-engineering and team-finance. To create a tenant, refer to Set up a GET tenant.

The federation-frontend target itself requires a configured GET installation installed using the tempo-distributed Helm chart. The examples provided use a GET installation that does not use TLS.

For cross-tenant querying, the federation-frontend component is usually deployed as part of a larger GET installation, and therefore has access to all of the configuration for that installation. Specify cross-tenant requests in the form <tenant1>|<tenant2>|...<tenantN>, in which case the federation-frontend separate the request into individual tenant requests.

To allow requests from any valid tenant associated with the supplied token, use a wildcard tenant name (*) as the username in cross-tenant requests. To use a wildcard tenant name, your federation-frontend must access the admin storage bucket as the GET clusters so the service can retrieve the specific tenants that pertain to the access policy that the token represents. Verify that the federation-frontend configuration includes an admin_client section. For example:

auth_enabled: true
admin_client:
  storage:
    backend: s3
    s3:
      bucket_name: <bucketName>
      endpoint: <s3Endpoint>
      access_key_id: <accessKey>
      secret_access_key: <secretKey>

In situations where a wildcard tenant name passes to the federation-frontend, you must supply an admin_client configuration section if the component is not part of a GET installation.

Configure cross-tenant queries

To enable the federation-frontend for an installed GET cluster, you need to add an extra configuration in your Helm chart’s values.yaml file and upgrade the GET cluster using the values.yaml file. This configuration assumes that TLS is not enabled as part of the GET installation.

This example configuration enables the federation-frontend service, and then details the targets that it should use to send onward queries to, via the proxy_targets URL list (this being the GET gateway). The default access policy management in GET allows only for tokens created in that GET cluster to access its own tenants, and therefore there is only one entry in the proxy_targets list.

For more information, refer to the federation-frontend configuration documentation.

Locate your values.yaml file used with your tempo-distributed Helm chart installation.

Open the file and add the following section:

enterpriseFederationFrontend:
  enabled: true
  proxy_targets:
	- url: http://myinstall-get-cluster-tempo-enterprise-gateway.get.svc:3100
  	tls_insecure_skip_verify: true

Save the changes.

Use the command below to upgrade the GET cluster with the changes in your values.yaml file.
- Change the GET installation name (MYINSTALL-GET-CLUSTER) to your GET cluster
- Add the appropriate namespace by replacing GET-NAMESPACE
- Add the version number of the Helm chart (HELM-CHART-VERSION). Run the command to upgrade the GET cluster.
Bash
```
helm upgrade -f values.yaml MYINSTALL-GET-CLUSTER grafana/tempo-distributed --namespace GET-NAMESPACE --version HELM-CHART-VERSION
```

Helm updates the GET installation to include the federation-frontend service, which allows any queries routed to it to be cross-tenant compatible.

Queries against single tenants can be sent via the GET gateway, if required, although it may be more convenient to use the service for all queries regardless of cross-tenant status.

Set up an access policy with tenant federation and a token

For demonstration purposes, we’ll create two new tenants, create a shared access policy for them, and then query trace data using a single data source and query.

Create two new tenants

First, create two new tenants, team-engineering and team-finance, using the Grafana Enterprise Traces plugin for Grafana.

Within Grafana, select Apps > Grafana Enterprise Traces.
Select Tenants and then choose Create tenant.
Name the first tenant team-engineering.
Select Create tenant again and name the second tenant team-finance.

Create an access policy for the tenants

To allow queries to span both GET tenants, create a new access policy called leadership. For demonstration purposes, name these tenants team-engineering and team-finance. To create a raw access policy:

Create a new access policy leadership.
Enable the traces:read scope.
Add the tenants team-engineering and team-finance. Alternatively, you can add the special tenant name * to create an access policy that has access to all tenants in the cluster.
Select Create to add the new access policy.
Select Add token to create a new token for the access-policy.
Select Copy to clipboard to store the token in your clipboard.

Set up a Grafana data source using the access policy

In the standard configuration, the Grafana Enterprise Traces plugin creates a data source for each tenant, which uses the GET gateway and configures it specifically for the single tenant specified. To use the cross-tenant capabilities, create a new Tempo data source that uses the federation-frontend service and specifies both tenants.

To set up a data source using the access policy:

Create a new Tempo data source from the Grafana Connections menu and select Data sources. Select Add new data source.
Search for Tempo and select Tempo data source.
Under Connection field, select URL from the drop-down. Enter the URL of the frontend-federation Kubernetes service in your GET cluster. For example, where mycluster-get-install-tempo-enterprise-federation-frontend represents your GET namespace: http://mycluster-get-install-tempo-enterprise-federation-frontend.get.svc.cluster.local:3100.
From the Authentication section, enable Basic authentication.
1. In the User field, enter: team-engineering|team-finance where all the names of the tenants that you want to query across are separated by the | pipe character.
2. In the Password field, paste the token created in the token creation process for the tenant policy.
At the bottom of the page, select Save to preserve your changes and Test to validate the configuration.

The test should respond with ‘Data source successfully connected’.

Use OpenTelemetry `telemetrygen` to generate tracing data

Next, use OpenTelemetry telemetrygen to generate tracing data to test your configuration.

In the following instructions, the GET gateway is port-forwarded to a local development machine on port 3100 using the kubectl port-forward command.

To use this example, replace the content in upper case with the information for your cluster.

kubectl port-forward --address 0.0.0.0 service/MYCLUSTER-GET-INSTALL-TEMPO-ENTERPRISE-GATEWAY 3100:3100 --namespace GET-NAMESPACE

Install telemetrygen using the installation procedure. NOTE: You don’t need to configure an OpenTelemetry Collector because you’re using Grafana Alloy.
Create a new base64 token for both the team-engineering and team-finance tenants. For example:
Bash
```
echo 'team-engineering:<generatedToken>' | base64
echo 'team-finance:<generatedToken>' | base64
```
Run this command twice to generate traces using telemetrygen for each separate tenant and token for each token. Replace the base64Token with the one generated for each tenant.
Bash
```
telemetrygen traces --traces 1 --otlp-insecure --otlp-header 'Authorization="Basic <base64Token>"' --otlp-endpoint localhost:3100
```

This configuration sends a single trace to each of the tenants. When a cross-tenant query runs, you should see two traces instead of a single trace for one tenant.

Use cross-tenant query federation

Queries are performed using the cross-tenant configured data source in either Explore or inside of dashboards are performed across all the tenants that you specified in the User field. These queries are processed as if all the data were in a single tenant.

To submit a query across all tenants that your access policy has access rights to, you can either:

Explicitly set the name of all the tenants separated by a pipe character “|” in the username. For example, to query across tenant1, tenant2, and tenant3 you would enter tenant1|tenant2|tenant3.
Set the username to a wildcard character *. This will query all tenants that the access policy grants you access to, without requiring you to explicitly specify their names.

Conversely, if you use a wildcard username in your data source configuration with an access policy with specific tenants, that data source has access to only those tenants.

Go to your Grafana instance and select the Explorer.
Select your cross-tenant data source.
Selection Query type > Search and select Run query. You should see two traces returned, one from each tenant.

Cross-cluster query federation

Wed, 22 Apr 2026 04:21:40 +0000

Cross-cluster query federation

Cross-cluster query federation lets you gather data from multiple Grafana Enterprise Traces (GET) clusters and provide access controls for the tracing data. Each GET cluster has to have its own license. A team that is globally responsible for all observability can make a single query across the GET clusters for all regions to retrieve trace results.

Using cross-cluster query federation, you control who can access tracing data at a cluster-level. Consider an organization that separates observability data via region. Each region may have its own GET cluster. Engineering teams in that region may (or may not, refer to cross-tenant query federation) have access to the data for that region. Depending upon the configuration, developers probably won’t have access to other regions’ GET data.

Architecture

The diagram below shows the cross-cluster federation architecture.

Before you begin

The federation-frontend target itself requires a configured GET installation installed using the tempo-distributed Helm chart.

No other GET components are required. You can run the federation-frontend service on its own. The service configuration examples used in these procedures target the GET cluster it is configured alongside and a Grafana Cloud Traces tenant.

You also need:

The list of GET clusters and their gateway URLs. The clusters can be GET, Tempo open source, and Cloud Traces clusters.
Authentication credentials for each cluster, including a tenant user and token for a GET cluster and Grafana Cloud Traces; refer to the Grafana Cloud Access Policies documentation

Configure authentication for cross-cluster query federation

The federation-frontend target itself requires a configured GET installation. No other GET components are required.

Set up the federation-frontend

A minimal configuration of the federation-frontend target has authentication disabled, because it forwards the HTTP authentication header (Authorization: Basic ...) that is supplied by its clients to the target clusters.

You need to configure a list of target clusters within the enterpriseFederationFrontend.proxy_targets block. Currently, there are no equivalent CLI flags available. Each entry requires a URL that points to a Tempo-compatible API. For GET clusters, use this URL pattern with your GET gateway, port, and HTTP API prefix: http://<get-gateway>:<get-port>/<http_api_prefix>.

Optionally, you can configure each proxy_target to have Basic auth credentials, which override the user-supplied ones.

When you configure Basic auth with the proxy_target configuration, these credentials take precedence over the ones supplied by the client. Without enabling authentication on the federation-frontend, any client that can reach the federation-frontend can perform queries on the target clusters by using those credentials.

To set the target option, follow these steps:

Locate your values.yaml file used with your tempo-distributed Helm chart installation.
Open the file and add the following section:
YAML
```
enterpriseFederationFrontend:
 enabled: true
 proxy_targets:
   - url: http://myinstall-get-cluster-tempo-enterprise-gateway.get.svc:3100
     tls_insecure_skip_verify: true
```
This configures the federation-frontend to use your GET cluster as the first target. This example uses a single GET cluster. Because of this, authentication details can pass from the Grafana data source straight through the federation-frontend to the GET cluster gateway, for it to authenticate the tenant and token.
Go to Grafana Cloud, and generate a read token for your Grafana Cloud Traces instance. Refer to the Cloud Access Policies documentation for instructions.
Retrieve the Grafana Cloud Traces user ID, the generated access token, and Grafana Cloud Traces endpoint. The user ID will be a number, the token a base64 encoded string, and the endpoint will look something like this: https://tempo-us-central1.grafana.net/tempo.
Add the following to the proxy_targets section to your values.yaml file:
YAML
```
   - url: https://<gctEndpoint>/tempo
     basic_auth:
       username: <userID>
       password: <token>
```
This target includes basic authentication. The authentication details passed by the request from Grafana are for the GET instance and not the Grafana Cloud instance. The federation-frontend lets you specify authentication details that will override any that are passed to it in a query. In this case, for the appropriate Grafana Cloud user.

Warning
This gives any client that can reach the federation-frontend access to your traces data in Grafana Cloud Traces without further authentication.
Upgrade the GET cluster by running the following command. Before you execute the command, change the GET installation name (MYINSTALL-GET-CLUSTER) and add the appropriate namespace (GET-NAMESPACE) and version number of the Helm chart (HELM-CHART-VERSION).
Bash
```
helm upgrade -f values.yaml MYINSTALL-GET-CLUSTER grafana/tempo-distributed --namespace GET-NAMESPACE --version HELM-CHART-VERSION
```

Helm updates the GET installation to include the federation-frontend service, which allows any queries routed to it to be cross-cluster compatible.

Queries can be sent directly to the appropriate GET gateways, if required.

Set up an access policy with tenant federation and a token

For demonstration purposes, we’ll create a new tenant using the Grafana Enterprise Traces plugin for Grafana, create a shared access policy for them, and then query trace data using a single data source and query.

Use the GET application plugin to create a new tenant called team-engineering.
Select ‘Create tenant’ and create the new tenants.
Create a new access policy called engineering.

To create a raw access policy:

Create a new access policy called engineering.
Enable the traces:read scope (or all scopes for admin and writing access as well).
Under the Tenants section, add the team-engineering tenant.
Select Create to create the new access policy.
Select Add token to create a new token for the access-policy. Click Copy to clipboard to store the token in your clipboard.

Set up a Grafana data source using the access policy

In the standard configuration, the GET plugin creates a data source for each tenant, which uses the GET gateway and configures it specifically for the single tenant specified. To use the cross-cluster capabilities, create a new Tempo data source that uses the federation-frontend service (not the default, which is the GET gateway) and specifies the tenant and generated token for basic auth.

To set up a data source using the access policy:

Create a new Tempo data source. From the Grafana Connections menu, select Data sources and then click Add new data source.
Search for Tempo and select Tempo data source.
Under Connection field, select URL from the drop-down and enter the URL of the frontend-federation Kubernetes service in your GET cluster. For example (after replacing the appropriate ): http://mycluster-get-install-tempo-enterprise-federation-frontend.get.svc.cluster.local:3100.
From the Authentication section, enable Basic authentication.
1. In the User field, enter: team-engineering.
2. In the Password field, paste the token created in the token creation process for the tenant policy.
At the bottom of the page, select Save to preserve your changes and Test to validate the configuration. The test should respond with ‘Data source successfully connected’.

Generate tracing data with OpenTelemetry `telemetrygen`

Next, you can use OpenTelemetry telemetrygen to generate tracing data to test your configuration.

In the following instructions, the GET gateway is port-forwarded to a local development machine on port 3100 using the kubectl port-forward command. To use this example, replace the content in upper case with the information for your cluster.

kubectl port-forward --address 0.0.0.0 service/MYCLUSTER-GET-INSTALL-TEMPO-ENTERPRISE-GATEWAY 3100:3100 --namespace GET-NAMESPACE

To generate data and send it to your GET cluster:

Install telemetrygen using the installation procedure. NOTE: You do not need to configure an OpenTelemetry Collector because you’re using Grafana Alloy.
Create two new base64 token, one for team-engineering and the other for the Grafana Cloud Traces user tenants. For example:
```
echo 'team-engineering:<generatedToken>' | base64
echo '<gctUser>:<gctToken>' | base64
```
Run this command to generate a trace using telemtrygen for the team-engineering tenant. Replace the base64Token with the one generated for the tenant.
```
telemetrygen traces --traces 1 --otlp-insecure --otlp-header 'Authorization="Basic <base64Token>"' --otlp-endpoint localhost:3100
```
Run this command to generate a trace using telemtrygen for the Grafana Cloud Traces user. Replace the base64Token with the one generated for the Grafana Cloud Traces user. Replace <GCTendpoint> with the Grafana Cloud Traces endpoint.
```
telemetrygen traces --traces 1 --otlp-header 'Authorization="Basic <token>"' --otlp-endpoint <GCTendpoint>:443
```

This configuration sends a single trace to the tenant. This means when GET performs a cross-cluster query, you should see two traces instead of a single trace for one cluster.

Use cross-cluster query federation

Queries run with the configured data source in either Explore or inside of dashboards are performed across all the clusters specified in federation-frontend proxy targets. These queries are processed as if all the data were from a single source.

Go to your Grafana instance and select Explore.
Select your cross-cluster data source.
Selection Query type > Search and select Run query. You should see two traces returned, one from the GET cluster and the other from Grafana Cloud.

Aggregate traces from a local GET cluster and Grafana Cloud Traces instance

The federation-frontend provides an aggregated view of traces stored in a local GET cluster and a hosted Grafana Cloud Traces instance. With the following configuration, you can query both of the clusters as though they were one:

federation:
  proxy_targets:
    - name: own-data-center
      url: http://get/tempo
    - name: grafana-cloud
      url: https://tempo-us-central1.grafana.net/tempo
      basic_auth:
        username: <instance-id>
        password: <token>

This gives any client that can reach the federation-frontend access to your traces data in Grafana Cloud Traces without further authentication.

By using the authentication credentials of the local GET cluster in your query request, you can execute a query against both clusters. To do so, set the access policy’s token as a variable for subsequent commands:

export API_TOKEN="the long token string you copied"
curl -s -u "<tenant-id>:$API_TOKEN" http://federation-frontend/tempo/api/traces/4d65822107fcfd52 | jq

The federation-frontend passes the credentials from your query command to cluster="own-data-center". Because the basic_auth block is configured to include credentials for the grafana-cloud cluster, the credentials supplied in the query command are overridden with those from the configuration file. The credentials in the configuration file are used to authenticate with the grafana-cloud cluster.

Federation-frontend on Grafana Labs

Cross-tenant query federation

Cross-tenant query federation

Architecture

Before you begin

Configure cross-tenant queries

Set up an access policy with tenant federation and a token

Create two new tenants

Create an access policy for the tenants

Set up a Grafana data source using the access policy

Use OpenTelemetry telemetrygen to generate tracing data

Use cross-tenant query federation

Cross-cluster query federation

Cross-cluster query federation

Architecture

Before you begin

Configure authentication for cross-cluster query federation

Set up the federation-frontend

Set up an access policy with tenant federation and a token

Set up a Grafana data source using the access policy

Generate tracing data with OpenTelemetry telemetrygen

Use cross-cluster query federation

Aggregate traces from a local GET cluster and Grafana Cloud Traces instance

Use OpenTelemetry `telemetrygen` to generate tracing data

Generate tracing data with OpenTelemetry `telemetrygen`