---
title: "Configure Application Observability | Grafana Cloud documentation"
description: "Configure Application Observability."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Configure Application Observability

Admin users can configure the Application Observability plugin.

## Data source

You can configure the default data source for each signal type, when you select an option the changes are automatically saved.

On the **Data source** tab, you can configure the default data source for each signal type: metrics, logs, traces, and profiles.

By default, Application Observability uses the automatically provisioned data sources for the stack. This gives users a simple getting-started workflow that only needs them to send traces, and Application Observability generates the necessary metrics.

If you send metrics to a different Grafana Cloud hosted Prometheus or Mimir data source, you don’t need the automatic metrics generation provided by Application Observability and can [disable](#metrics-generation) it to reduce your Grafana Cloud usage and bill.

### Supported data sources

The **metrics** data source must be a Grafana Cloud hosted Prometheus or Mimir data source. You can’t select a self-hosted Prometheus or Mimir data source, or a Grafana Cloud data source that you reach through a proxy or CNAME that you own.

Application Observability and the Knowledge Graph need direct access to your metrics to run the relabeling behind service discovery and the entity and dependency graph. Because Grafana Cloud can’t run that relabeling on metrics it can’t read directly, Application Observability doesn’t offer those data sources.

The **logs**, **traces**, and **profiles** data sources do support custom data sources.

## Logs query

On the **Logs query** tab, you can configure the default log queries and formatting Application Observability uses to query and format logs.

### Default log query format

Select a log query format that matches your logs ingestion path, and customize your default log query:

- Use the “Loki exporter” query if you are exporting logs directly to Loki using the [OpenTelemetry Collector Loki Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md) or the Alloy [`otelcol.exporter.loki`](/docs/grafana-cloud/send-data/alloy/reference/components/otelcol/otelcol.exporter.loki/)
- Use the “OTLP gateway / native Loki OTLP query” mode if you are exporting logs via the [Grafana OTLP endpoint](/docs/loki/next/send-data/otel/)

The log query format needs to match the log storage and query format used by the ingestion method. If your log queries use a previous log format version, Application Observability prompts you to update the query.

> Warning
> 
> Updating the default log query format overwrites changes you have made to your default log queries. Copy any queries you would like to keep before updating.

You can switch between log query formats on the **Logs** tab, changes won’t persist beyond the current session. If you would like to permanently change the default log query format, update the setting from the **Configuration** page.

### Logs query, without namespace

You can configure the logs query to execute in the logs tab if `service.namespace` resource attribute isn’t provided. Available variables: `$job`, and `$serviceName`.

### Logs query, with namespace

You can configure the logs query to execute in the logs tab when you supply the `service.namespace` resource attribute. Available variables: `$job`, `$serviceName`, and `$serviceNamespace`.

### Logs query formatting

You can specify the formatting section of a log query to automatically appended to the end of logs queries to improve readability.

For example, the default formatting appended to log queries is:

LogQL ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
line_format "\u001b[1m{{if .severity}}{{alignRight 5 .severity}}{{end}}\u001b[0m \u001b[90m[{{alignRight 10 .resources_service_instance_id}}{{if .attributes_thread_name}}/{{alignRight 20 .attributes_thread_name}}{{end}}]\u001b[0m \u001b[36m{{if .instrumentation_scope_name }}{{alignRight 40 .instrumentation_scope_name}}{{end}}\u001b[0m{{if .traceid}} \u001b[37m\u001b[3m[traceid={{.traceid}}]{{end}}: {{.body}}"
```

Important: the formatting is automatically applied at the end of the actual query. Messing this up can lead to invalid queries.

### Filtering logs by trace and span ID

Application Observability provides options to automatically filter logs by trace ID and span ID when navigating from a trace view to logs. These options are available in the Configuration page as:

- **Filter by trace id**: When enabled, navigating to logs from a trace view automatically add a filter for the trace ID. This allows you to see only the logs associated with the selected trace, making it easier to correlate trace and log data.
- **Filter by span id**: When enabled, navigating to logs from a trace view automatically add a filter for the span ID. This further narrows the log results to those associated with a specific span within the trace.

Both options are enabled by default. You can enable or disable them in the Configuration page under the logs query sections.

Requirements:

- Your logs must include the trace ID and/or span ID fields, typically as `traceid` and `spanid` attributes, for these filters to work. Ensure that your log ingestion pipeline is configured to include these fields if you want to use this feature.
- Most logging auto-instrumentation libraries automatically add `traceid` and `spanid` attributes to your logs, so these filters work out of the box for most users.

These filters help you quickly correlate logs and traces, improving troubleshooting and root cause analysis workflows.

## Settings

You can use the settings tab to configure the behavior of different Application Observability features.

### UI

By default, Application Observability does not allow you to sort services on the service inventory page. This prevents performance issues for very large lists of services.

If the amount of services is not too large, you can select the toggle to enable service inventory sorting and gain the ability to sort by any of the `Rate`, `Errors` or `Duration` columns, as well as the always available `Name` and `Namespace` columns.

### Deployment environment attribute and value

By default Application Observability uses the [OpenTelemetry recommended attribute](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/resource#environment) `deployment.environment` to distinguish environments. You can select a custom attribute from a list of possible attributes from the data.

> Note
> 
> When you select a custom attribute to represent your environment, ensure that all your traces include the attribute.

### Group and filter attributes

To enable data grouping and filtering in Application Observability, first add the necessary attributes to traces that you would like to group and filter by, and then activate them as dimensions on the generated metrics.

Dimensions are managed in the Knowledge Graph configuration: navigate to **Observability > Configuration > Traces metrics generation > Dimensions**. For more information, refer to the [Configure Traces metrics generation](/docs/grafana-cloud/platform/knowledge-graph/configure/traces-metrics-generation/) documentation.

By default you can group and filter by the deployment environment.

> Warning
> 
> When you enable additional dimensions it contributes to your Grafana Cloud data usage and bill.
> 
> Think carefully about the cardinality of your data and select attributes with minimal variation in value, for example geographical region and cloud provider have less cardinality than instance id.

### Include web applications and mobile devices

Application Observability can display web applications, including services like load generators, and mobile devices when metrics are generated for spans with a `CLIENT` and `PRODUCER` [spanKind](https://opentelemetry.io/docs/specs/otel/trace/api/#spankind).

If you use Grafana Cloud metrics generation, a filter rule controls which span kinds generate metrics: navigate to **Observability > Configuration > Traces metrics generation > Filter rules** and add **Client** and **Producer** to the span kind rule. Keep **Server** and **Consumer** selected, otherwise metrics are no longer generated for your backend services. For more information, refer to the [Configure Traces metrics generation](/docs/grafana-cloud/platform/knowledge-graph/configure/traces-metrics-generation/) documentation.

If you generate metrics with [span metrics connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector), you need to update your configuration to not ignore `CLIENT` and `PRODUCER` spans:

To view the Grafana Alloy configuration, select the **river** tab below. To view the OpenTelemetry Collector configuration, select the **yaml** tab below.

Alloy YAML

![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
otelcol.processor.filter "drop_unneeded_span_metrics" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.filter/
	error_mode = "ignore"

	metrics {
		datapoint = [
			"IsMatch(metric.name, \"calls|duration\") and IsMatch(attributes[\"span.kind\"], \"SPAN_KIND_INTERNAL\")",
		]
	}

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}
```

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
filter/drop_unneeded_span_metrics:
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor
  error_mode: ignore
  metrics:
    datapoint:
      - 'IsMatch(metric.name, "calls|duration") and IsMatch(attributes["span.kind"], "SPAN_KIND_INTERNAL")'
```

## System

Use the system configuration tab to configure system-wide settings.

### Span metrics source

Application Observability uses metrics generated from traces for most of the visualizations in the application. This data can come from different places depending on your setup. You can change which source you use from the **System** tab, under the **Span metrics source** section:

- **Tempo (Grafana Cloud)**: the default span metrics source, for metrics generation through the [Tempo metrics generator](https://github.com/grafana/tempo/tree/main/modules/generator)
- **OpenTelemetry Collector, Grafana Alloy, Grafana Agent**: for advanced setups using the [OpenTelemetry collector](https://opentelemetry.io/docs/collector/), [Grafana Alloy](/oss/alloy-opentelemetry-collector/) or [Grafana Agent](/oss/agent/), with the [span metrics connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector) and the [service graph metrics connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/servicegraphconnector)
- **Legacy**: for customers who use the OpenTelemetry Collector with metric names that match those used by the Tempo metrics generator.

The following metric names are used by each of the options:

Expand table

| Tempo (Grafana Cloud)                                | OpenTelemetry Collector, Grafana Alloy, Grafana Agent | Beyla                                                | Legacy                                               |
|------------------------------------------------------|-------------------------------------------------------|------------------------------------------------------|------------------------------------------------------|
| `traces_target_info`                                 | `target_info`                                         | `traces_target_info`                                 | `target_info`                                        |
| `traces_service_graph_dropped_spans_total`           | `n/a`                                                 | `traces_service_graph_dropped_spans_total`           | `traces_service_graph_dropped_spans_total`           |
| `traces_service_graph_unpaired_spans_total`          | `n/a`                                                 | `traces_service_graph_unpaired_spans_total`          | `traces_service_graph_unpaired_spans_total`          |
| `traces_service_graph_request_total`                 | `traces_service_graph_request_total`                  | `traces_service_graph_request_total`                 | `traces_service_graph_request_total`                 |
| `traces_service_graph_request_failed_total`          | `traces_service_graph_request_failed_total`           | `traces_service_graph_request_failed_total`          | `traces_service_graph_request_failed_total`          |
| `traces_service_graph_request_client_seconds_bucket` | `traces_service_graph_request_client_seconds_bucket`  | `traces_service_graph_request_client_seconds_bucket` | `traces_service_graph_request_client_seconds_bucket` |
| `traces_service_graph_request_client_seconds_count`  | `traces_service_graph_request_client_seconds_count`   | `traces_service_graph_request_client_seconds_count`  | `traces_service_graph_request_client_seconds_count`  |
| `traces_service_graph_request_client_seconds_sum`    | `traces_service_graph_request_client_seconds_sum`     | `traces_service_graph_request_client_seconds_sum`    | `traces_service_graph_request_client_seconds_sum`    |
| `traces_service_graph_request_server_seconds_bucket` | `traces_service_graph_request_server_seconds_bucket`  | `traces_service_graph_request_server_seconds_bucket` | `traces_service_graph_request_server_seconds_bucket` |
| `traces_service_graph_request_server_seconds_count`  | `traces_service_graph_request_server_seconds_count`   | `traces_service_graph_request_server_seconds_count`  | `traces_service_graph_request_server_seconds_count`  |
| `traces_service_graph_request_server_seconds_sum`    | `traces_service_graph_request_server_seconds_sum`     | `traces_service_graph_request_server_seconds_sum`    | `traces_service_graph_request_server_seconds_sum`    |
| `traces_spanmetrics_calls_total`                     | `traces_span_metrics_calls_total`                     | `traces_spanmetrics_calls_total`                     | `traces_spanmetrics_calls_total`                     |
| `traces_spanmetrics_latency_bucket`                  | `traces_span_metrics_duration_seconds_bucket`         | `traces_spanmetrics_latency_bucket`                  | `traces_spanmetrics_latency_bucket`                  |
| `traces_spanmetrics_latency_count`                   | `traces_span_metrics_duration_seconds_count`          | `traces_spanmetrics_latency_count`                   | `traces_spanmetrics_latency_count`                   |
| `traces_spanmetrics_latency_sum`                     | `traces_span_metrics_duration_seconds_sum`            | `traces_spanmetrics_latency_sum`                     | `traces_spanmetrics_latency_sum`                     |

### Metrics generation

Enable or disable metrics generation for the default provisioned Traces data source for the stack.

> Note
> 
> You can safely disable metrics generation if you are using a custom data source.

The Tempo metrics generation settings, such as dimensions, filter rules, histogram buckets, service graph metrics, instance label, and span multiplier key, are managed in the Knowledge Graph configuration under **Observability > Configuration > Traces metrics generation**. For more information, refer to the [Configure Traces metrics generation](/docs/grafana-cloud/platform/knowledge-graph/configure/traces-metrics-generation/) documentation.

### Service graph metrics

Service graph metrics generation is managed in the Knowledge Graph configuration: navigate to **Observability > Configuration > Traces metrics generation > General** and toggle **Service graph metrics**. For more information, refer to the [Configure Traces metrics generation](/docs/grafana-cloud/platform/knowledge-graph/configure/traces-metrics-generation/) documentation.

### Histogram type

Histogram type and bucket settings are managed in the Knowledge Graph configuration: navigate to **Observability > Configuration > Traces metrics generation > Histogram buckets**. For more information, refer to the [Configure Traces metrics generation](/docs/grafana-cloud/platform/knowledge-graph/configure/traces-metrics-generation/) documentation.

> Note
> 
> Application Observability panels automatically display data from native histograms when both histogram types are present.

### Beyla

Beyla can generate its own span metrics and service graph metrics. When used together with Grafana Cloud metrics generation, this can lead to duplicate series, which can display erroneous data in Application Observability.

To exclude Beyla traces from Grafana Cloud metrics generation, navigate to **Observability > Configuration > Traces metrics generation > Filter rules** and add the recommended rule from the catalog that excludes Beyla traces. For more information, refer to the [Configure Traces metrics generation](/docs/grafana-cloud/platform/knowledge-graph/configure/traces-metrics-generation/) documentation.

## Sampling

To configure head or tail sampling to reduce the cost of processing and storing large volumes of data, consult the [sampling](/docs/grafana-cloud/monitor-applications/application-observability/sampling/) documentation.

## Troubleshooting

### A custom Prometheus data source doesn’t appear in the metrics picker

If your custom Tempo and Loki data sources appear on the **Data source** tab but your custom Prometheus data source doesn’t, check the following:

- The data source isn’t Grafana Cloud hosted. Application Observability only offers Grafana Cloud hosted Prometheus and Mimir data sources for metrics. For more information, refer to [Supported data sources](#supported-data-sources).
- You don’t have permission to read data sources. When Grafana can’t list your data sources, the metrics picker is empty while the logs, traces, and profiles pickers still show options. Ask an administrator to grant you the **Data sources > Reader** role, described in the [role-based access control](/docs/grafana-cloud/observe-and-act/monitor-applications/application-observability/manual/rbac) documentation.
- The data source is an Amazon Managed Service for Prometheus data source. Application Observability supports Prometheus and Mimir data sources for metrics, so Amazon Managed Service for Prometheus data sources don’t appear in the picker.