Get started on Grafana Labs

Best practices for traces

Tue, 10 Jun 2025 20:46:23 +0000

Best practices for traces

This page provides some general best practices for tracing.

Span and resource attributes

Traces are built from spans, which denote units of work such as a call to, or from, an upstream service. Spans are constructed primarily of span and resource attributes. Spans also have a hierarchy, where parent spans can have children or siblings.

In the screenshot below, the left side of the screen (1) shows the list of results for the query. The right side (2) lists each span that makes up the selected trace.

A span attribute is a key/value pair that provides context for its span. For example, if the span deals with calling another service via HTTP, an attribute could include the HTTP URL (maybe as the span attribute key http.url) and the HTTP status code returned (as the span attribute http.status_code). Span attributes can consist of varying, non-null types.

Unlike a span attribute, a resource attribute is a key/value pair that describes the context of how the span was collected. Generally, these attributes describe the process that created the span. This could be a set of resource attributes concerning a Kubernetes cluster, in which case you may see resource attributes, for example: k8s.namespace, k8s.container_name, and k8s.cluster. These can also include information on the libraries that were used to instrument the spans for a trace, or any other infrastructure information.

For more information, read the Attribute and Resource sections in the OpenTelemetry specification.

Naming conventions for span and resource attributes

The OpenTelemetry project defines a number of semantic conventions for attributes, which can help you to determine which attributes are most important to include in your spans. These conventions provide a common vocabulary for describing different types of entities, which can help to ensure that your data is consistent and meaningful.

When naming attributes, use consistent, nested namespaces to ensures that attribute keys are obvious to anyone observing the spans of a trace and that common attributes can be shared by spans. Using our example from above, the http prefix of the attribute is the namespace, and url and status_code are keys within that namespace. Attributes can also be nested, for example http.url.protocol might be HTTP or HTTPS, whereas http.url.path might be /api/v1/query.

For more details around semantic naming conventions, refer to the Recommendations for OpenTelemetry Authors and OpenTelemetry Semantic Conventions documentation.

Some third-party libraries provide auto-instrumentation that generate span and span attributes when included in a source base.

For more information about instrumenting your app for tracing, refer to the Instrument for distributed tracing documentation.

How many attributes should spans have?

The decision of how many attributes to include in your spans is up to you. There is no hard and fast rule. Keep the number of attributes to a minimum, as each attribute adds overhead to the tracing system. In Grafana Cloud, this results in higher tracing costs.

Only include attributes that are relevant to the operation that the span represents. For example, if you are tracing an HTTP request, you might include attributes such as the request method, the URL, and the response status code.

If you are unsure whether or not to include an attribute, it is always better to err on the side of caution and leave it out. You can always add additional attributes later if you need them.

In general, consider the following guidelines:

Don’t include metrics or logs as attributes in your spans.
Don’t use redundant attributes.
When determining which attributes to add, consider an application’s service flow, and execution in the context of only the current span.

The OpenTelemetry project doesn’t specify a maximum number of attributes that a span can have. However, the default limits for the number of attributes per span is 128 entries, so you will have to adjust that. There are also default limits on attribute value and name character lengths.

Determining where to add spans

When instrumenting, determine the smallest piece of work that you need to observe in a trace to be of value to ensure that you don’t over (or under) instrument.

Creating a new span for any work that has a relatively significant duration allows the observation of a trace to immediately show where significant amounts of time are spent during the processing of a request into your application or system.

For example, adding a span for a call to another services (either instrumented or not) may take an unknown amount of time to complete, and therefore being able to separate this work shows when services are taking longer than expected.

Adding a span for a piece of work that might call many other functions in a loop is a good signal of how long that loop is taking (you might add a span attribute that counts how many time the loop runs to determine if the duration is acceptable). However, adding a span for each method or function call in that loop might not, as it might produce hundreds or thousands of worthless spans.

Span length

While there are some (high) default limits to the length that a span (and by definition, the traces they belong to) can be, these can be adjusted by these configurations. Traces that include a large number of spans and/or long-running spans can have an impact on the time taken to query them once stored. Cloud Traces users should contact Grafana Support to modify overrides.

For long-running spans and traces, the best way to see this impact on requests is to send a few test cases and see what the performance looks like and to evaluate the trace size.

From there, you can modify the configuration for Tempo or determine ways to re-architect how the trace is produced.

You can consider breaking up the spans in several ways:

Decompose the query
- For example, if a complex SQL query involves multiple operations (for example, uses joins, subqueries, or unions), consider creating separate spans for each significant operation.
Improve granulation of long-running spans
- For long-running operations, you could create a new span for every predetermined interval of execution time.
  Note
  This requires time-based tracking in your application’s code and is more complex to implement.
Use span linking
- Should data flow hit bottlenecks where further operations on that data might be batched at a later time, the use of span links can help keep traces constrained to an acceptable time range, while sharing context with other traces that work on the same data. This can also improve the readability of traces.

Instrument for distributed tracing

Tue, 10 Jun 2025 20:46:23 +0000

Instrument for distributed tracing

Client instrumentation is the first building block to a functioning distributed tracing visualization pipeline. Client instrumentation is the process of adding instrumentation points in the application that create and offload spans.

Check out these resources for help instrumenting tracing with your favorite languages. Most of these guides include complete end-to-end examples with Grafana, Loki, Mimir, and Tempo.

Instrumentation frameworks

Most of the popular client instrumentation frameworks have SDKs in the most commonly used programming languages. You should pick one according to your application needs.

OpenTelemetry has the most active development in the community and may be a better long-term choice.

OpenTelemetry

A collection of tools, APIs, and SDKs, OpenTelemetry helps engineers instrument, generate, collect, and export telemetry data such as metrics, logs, and traces, to analyze software performance and behavior. For more information refer to OpenTelemetry overview.

Auto-instrumentation frameworks

OpenTelemetry provides auto-instrumentation agents and libraries of Java, .NET, Python, Go, and JavaScript applications, among others. For more information, refer for the OpenTelemetry Instrumentation documentation.

These libraries capture telemetry information from a client application with minimal manual instrumentation of the codebase.

Note
Jaeger client libraries have been deprecated. For more information, refer to the Deprecating Jaeger clients article. Jaeger recommends using OpenTelemetry SDKs.

Additional OTel resources

Other instrumentation resources

Zipkin

Zipkin Language Specific Instrumentation

Grafana Blog

The Grafana blog periodically features instrumentation posts.

Community resources

Tracing in Grafana

Tue, 10 Jun 2025 20:46:23 +0000

Tracing in Grafana

Grafana has a built-in Tempo data source that can be used to query Tempo and visualize traces. This page describes the high-level features and their availability. Use the latest versions for best compatibility and stability.

Query your data

Using tracing data in Grafana and Grafana Cloud Traces, you can search for traces, generate metrics from spans, and link your tracing data with logs, metrics, and profiles.

Use Traces Drilldown to investigate tracing data

Note
Traces Drilldown is currently in public preview. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.

Grafana Traces Drilldown helps you visualize insights from your Tempo traces data. Using the app, you can:

Use Rate, Errors, and Duration (RED) metrics derived from traces to investigate issues
Uncover related issues and monitor changes over time
Browse automatic visualizations of your data based on its characteristics
Do all of this without writing TraceQL queries

Expand your observability journey and learn about the Drilldown apps suite.

Search for traces

Search for traces using common dimensions such as time range, duration, span tags, service names, etc. Use the Explore trace view to quickly diagnose errors and high latency events in your system.

Search is non-deterministic

Most search functions are deterministic. When given the same criteria, a deterministic algorithm returns consistent results. For example, let’s say that you query a search engine for the definition of “traces.” The results list the same top matches for each query for “traces” in that search engine.

However, Tempo search is non-deterministic. If you perform the same search twice, you’ll get different lists, assuming the possible number of results for your search is greater than the number of results you have your search set to return.

When performing a search, Tempo does a massively parallel search over the given time range, and takes the first N results. Even identical searches differ due to things like machine load and network latency. This approach values speed over predictability and is quite simple; enforcing that the search results are consistent would introduce additional complexity (and increase the time the user spends waiting for results). TraceQL follows the same behavior.

By adding most_recent=true to your TraceQL queries, the search results become deterministic. For more information, refer to Retrieve most recent results

Use trace search results as panels in dashboards

You can embed tracing panels and visualizations in dashboards. You can also save queries as panels. For more information, refer to the Traces Visualization documentation.

For example dashboards, visit play.grafana.org.

Use TraceQL to query data and generate metrics

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces.

Using Grafana Explore, you can search traces. The default traces search reviews the whole trace. TraceQL provides a method for formulating precise queries so you can zoom in to the data you need. Query results return faster because the queries limit what is searched.

You can construct queries using the TraceQL query editor or use the Search query type.

For details about constructing queries, refer to the TraceQL documentation.

TraceQL metrics queries

Note
TraceQL metrics is an experimental feature. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Enable the feature toggle in Grafana to use this feature. Do not enable this feature in production environments.

TraceQL language provides metrics queries as an experimental feature. Metric queries extend trace queries by applying an aggregate function to trace query results. For example: { span:name = "foo" } | rate() by (span:status) This powerful feature creates metrics from traces, much in the same way that LogQL metric queries create metrics from logs.

Grafana Traces Drilldown is powered by metrics queries.

For more information about available queries, refer to TraceQL metrics queries.

Generate metrics from spans

RED metrics can drive service graphs and other ready-to-go visualizations of your span data. RED metrics represent:

Rate, the number of requests per second
Errors, the number of those requests that are failing
Duration, the amount of time those requests take

For more information about RED method, refer to The RED Method: how to instrument your services.

To enable metrics-generator in Grafana Cloud, refer to Enable metrics-generator.

To enable metrics-generator for Tempo, refer to Configure metrics-generator.

These metrics exist in your Hosted Metrics instance and can also be easily used to generate powerful custom dashboards.

Metrics automatically generate exemplars as well which allows easy metrics to trace linking.

Service graph view

Service graph view displays a table of request rate, error rate, and duration metrics (RED) calculated from your incoming spans. It also includes a node graph view built from your spans.

To use the service graph view, you need to enable service graphs and span metrics. After it’s enabled, this pre-configured view is immediately available in Explore > Service Graphs.

Refer to the service graph view documentation for further information.

Integrate other telemetry signals

Link traces and logs

If you’re already doing request/response logging with trace IDs, they can be easily extracted from logs to jump directly to your traces.

In the other direction, you can configure Grafana and Grafana Cloud to create a link from an individual span to your Loki logs. If you see a long-running span or a span with errors, you can immediately jump to the logs of the process causing the error.

Note
Cloud Traces only supports custom tags added by Grafana Support. Cloud Traces supports these default tags: cluster, hostname, namespace, and pod. Contact Support to add a custom tag.

Link traces and metrics

Grafana can correlate different signals by adding the functionality to link between traces and metrics. Trace to metrics lets you navigate from a trace span to a selected data source. Using trace to metrics, you can quickly see trends or aggregated data related to each span.

For example, you can use span attributes to metric labels by using the $__tags keyword to convert span attributes to metrics labels.

To set up Trace to metrics for your data source, refer to Trace to metric configuration.

Link traces and profiles

Using Trace to profiles, you can correlate different signals by adding the functionality to link between traces and profiles.

Trace to profiles lets you link your Grafana Pyroscope data source to tracing data in Grafana or Grafana Cloud. When configured, this connection lets you run queries from a trace span into the profile data.

For more information, refer to the Traces to profiles documentation and the Grafana Pyroscope data source documentation.

For Cloud Traces, Refer to the Traces to profiles documentation for configuration instructions.

Metrics from traces

Tue, 10 Jun 2025 20:46:23 +0000

Metrics from traces

Metrics provide a powerful insight into the systems you are monitoring with your observability strategy. Instead of running an additional service to generate metrics, you can use Grafana Tempo to generate metrics from traces.

Grafana Tempo can generate metrics from tracing data using the metrics-generator, TraceQL metrics (experimental), and the metrics summary API (deprecated). Refer to the table for a summary of these metrics and their capabilities. Metrics summary is not included because it is deprecated.

	Metrics-generator	TraceQL metrics
Functionality	An optional component within Tempo that processes incoming spans to produce predefined metrics, specifically focusing on RED (Rate, Error, Duration) metrics and service graphs.	An experimental feature in Tempo that allows for on-the-fly computation of metrics directly from trace data using the TraceQL query language, without the need for a separate metrics storage backend.
Capabilities	Span metrics: Calculates the total count and duration of spans based on dimensions like service name, operation, span kind, status code, and other span attributes. Service graphs: Analyzes traces to map relationships between services, identifying transactions and recording metrics related to request counts and durations.	Ad-hoc aggregation and analysis of trace data by applying functions to trace query results, similar to how LogQL operates with logs.
Output	The generated metrics are written to a Prometheus-compatible database, enabling integration with time-series databases for storage and analysis.	Generates metrics dynamically at query time, facilitating flexible and detailed investigations into specific behaviors or patterns within the trace data.
Use case	Ideal for continuous monitoring and alerting, leveraging predefined metrics that will be stored in a time-series database. Less expressive for trace-specific analysis as it focuses on standard telemetry dimensions and RED metrics.	More expressive and flexible for analyzing trace data directly, enabling complex trace-based queries and fine-grained exploration. Suited for exploratory analysis and debugging, allowing users to derive insights from trace data without prior metric definitions or storage considerations.
Setup	Configure the metrics-generator in the Tempo configuration file, enable processors like span metrics or service graphs, and send metrics to a Prometheus-compatible database.	Configure the local-blocks processor in overrides and in the metrics-generator configurations.
Query range	Supports querying over long time ranges, limited only by retention of the metrics backend.	Limited to a maximum query range of 3 hours by default (as of now), as metrics are computed from stored traces in real time.
Query language	Metrics are consumed using PromQL via Prometheus/Grafana.	Uses TraceQL which has a PromQL-inspired syntax, but not all PromQL features are supported; it’s a similar but distinct subset with different semantics.

Metrics-generator

Tempo can generate metrics from ingested traces using the metrics-generator, an optional Tempo component. The metrics-generator runs two different processors: service graphs and span metrics.

The metrics-generator looks at incoming spans, and calculates rate, error, and duration (RED) metrics from them, which it then writes to a time series database like Prometheus. By querying Prometheus, you can see the overall request rate, erroring request rate, and distribution of request latency in your system. By using the labels on those metrics, you can get a more granular view of request rate, error rate, and latency at a per-service, per-namespace, or per-operation level.

Useful for investigating	Metric	Meaning
Unusual spikes in activity	Rate	Number of requests per second
Overall issues in your tracing ecosystem	Error	Number of those requests that are failing
Response times and latency issues	Duration	Amount of time those requests take, represented as a histogram

The metrics-generator generates metrics from tracing data using the services_graphs, span_metrics, and local_blocks processors. The service_graphs and span_metrics processors generate metrics that are written to a Prometheus-compatible backend. The local_blocks processor adds support for TraceQL metrics and provides the capability of answering TraceQL metric queries to the generators without writing any data to the Prometheus backend. The metrics-generator processes spans and write metrics using the Prometheus remote write protocol.

For more information, refer to Metrics generator.

Use-cases for span metrics

Span metrics are of particular interest if your system isn’t monitored with metrics but it has distributed tracing implemented. You get out-of-the-box metrics from your tracing pipeline.

Note
In Grafana Cloud, the metrics-generator is disabled by default. Contact Grafana Support to enable metrics generation in your organization.

After the metrics-generator is enabled in your organization, refer to Metrics-generator configuration for information about metrics-generator options.

These metrics exist in your Hosted Metrics instance and can also be easily used to generate powerful custom dashboards.

The metrics-generator automatically generates exemplars as well which allows easy metrics to trace linking. Exemplars are available in Grafana Cloud.

TraceQL metrics (experimental)

Traces are a unique observability signal that contain causal relationships between the components in your system.

Do you want to know how many database calls across all systems are downstream of your application?
What services beneath a given endpoint are currently failing?
What services beneath an endpoint are currently slow?

The experimental TraceQL metrics can answer all these questions by parsing your traces in aggregate.

You can query data generated by TraceQL metrics in a similar way that you would query results stored in Prometheus, Grafana Mimir, or other Prometheus-compatible Time-Series-Database (TSDB). TraceQL metrics queries allows you to calculate metrics on trace span data on-the-fly with Tempo (your tracing database), without requiring a time-series-database like Prometheus.

TraceQL metrics, powered by the API of the same name, return Prometheus-like time series for a given metrics query. Metrics queries apply a function to trace query results. TraceQL metrics uses the local_blocks processor in metrics-generator.

TraceQL metrics power the Grafana Traces Drilldown app. You can explore the power of visualizing your metrics in the Grafana Traces Drilldown app using Grafana Play.

Give it a try using Grafana Play

With Grafana Play, you can explore and see how it works, learning from practical examples to accelerate your development. This feature can be seen on the Grafana Play site.

Try it

Refer to these resources for additional information:

Metrics summary API (deprecated)

Warning
The metrics summary API is deprecated as of Tempo 2.7. Features powered by the metrics summary API, like the Aggregate by table, are also deprecated in Grafana Cloud and Grafana 11.3 and later. It will be removed in a future release.

The metrics summary API was an early capability in Tempo for generating ad hoc RED metrics at query time. This data was displayed in the Aggregate by table in Grafana Explore, where you could see request rate, error rate, and latency values of your system over the last hour, computed from your trace data. Those values were broken down by any and all attributes attached to your traces.

When you used the “Aggregate by” option, Grafana made a call to Tempo’s metrics summary API, which returned these RED metrics based on spans of kind=server seen in the last hour.

The metrics summary API returns RED metrics for kind=server spans sent to Tempo in the last hour. The metrics summary feature creates metrics from trace data without using metrics-generator.

Refer to Traces to metrics: Ad hoc RED metrics in Grafana Tempo with ‘Aggregate by’ for more information about metrics summary API.

The metrics summary API and the Aggregate by table have been deprecated in favor of TraceQL metrics. For more information, refer to Deprecation in favor of TraceQL metrics.