TraceQL on Grafana Labs

Trace structure and TraceQL

Wed, 19 Feb 2025 20:52:00 +0000

Trace structure and TraceQL

Inspired by PromQL and LogQL, TraceQL uses similar syntax and semantics. The differences being inherent to the very nature of searching spans and traces.

Trace structure

Traces are telemetry data structured as trees. Traces are made of spans (for example, a span tree); there is a root span that can have zero to multiple branches that are called child spans. Each child span can itself be a parent span of one or multiple child spans, and so on so forth.

In the specific context of Tempo and TraceQL query language, a span has the following associated fields:

name: the span name
duration: difference between the end time and start time of the span
status: enum: {ok, error, unset}. For details, refer to OTel span status documentation.
kind: enum: {server, client, producer, consumer, internal, unspecified}. For more details, refer to OTel span kind documentation.
Attributes

The first four properties are intrinsics. They are fundamental to the span.

Attributes are custom span metadata in the form of key-value pairs. There are four types of attributes: span attributes, resource attributes, event attributes, and link attributes.

Span attributes are key-value pairs that contain metadata that you can use to annotate a span to carry information about the operation it’s tracking. For example, in an eCommerce application, if a span tracks an operation that adds an item to a user’s shopping cart, the user ID, added item ID and cart ID can be captured and attached to the span as span attributes.

A resource attribute represents information about an entity producing the span. For example, a span created by a process running in a container deployed by Kubernetes could link a resource that specifies the cluster name, namespace, pod, and container name. Resource attributes are resource-related metadata (key-value pairs) that are describing the Resource.

An event attribute represents a unique point in the time during the span’s duration. For more information, refer to All about span events and Span events in the OTEL documentation.

A link attribute lets you query link data in a span. A span link associates one span with one or more other spans that are a casual relationship. For more information on span links, refer to the Span Links documentation in the Open Telemetry project.

The OpenTelemetry specification defines Semantic Attributes for Spans and for Resources. Semantic Span Attributes are a set of naming schemes for attributes shared across languages, frameworks, and runtimes. For more details, refer to Trace Semantic Conventions and Resource Semantic Conventions.

TraceQL queries

The purpose of TraceQL is to search or query for spans. The query returns a set of spans, also called a spanset.

A TraceQL query can select traces based on:

span attributes, timing and duration
structural relationships between spans
aggregated data from the spans in a trace

As with PromQL and LogQL, the query is structured as a pipeline of operations (filters and aggregators). The query expression is evaluated on one trace at a time, selecting or discarding spans from the result. At each stage of the query pipeline, the selected spans for a trace are grouped in a spanset (set of spans). The associated trace is also returned. The result of the query is the spansets (and their associated traces) for all the traces evaluated.

The simplest query is this one:

{ }

The curly braces encompass the select/filter conditions. In theory, each span (and the trace it belongs to) matching those conditions is returned by the query. In the previous example, since there are no filter conditions, all spans are matching and thus returned with their associated traces.

In practice, the query is performed against a defined time interval, relative (for example, the last 3 hours) or absolute (for example, from X date-time to Y date-time). The query response is also limited by the number of traces (Limit) and spans per spanset (Span Limit).

TraceQL query editor
Query options: Limit, Span Limit and Table Format (Traces or Spans).
Trace (by Trace ID). The Name and Service columns are displaying the trace root span name and associated service.
Spans associated to the Trace

How TraceQL works

Tue, 11 Mar 2025 21:31:02 +0000

How TraceQL works

The TraceQL engine connects the Tempo API handler with the storage layer. The TraceQL engine:

Parses incoming requests and extract flattened conditions the storage layer can work with
Pulls spansets from the storage layer and revalidates that the query matches each span
Returns the search response

The default Tempo search reviews the whole trace. TraceQL provides a method for formulating precise queries so you can zoom in to the data you need. Query results are returned faster because the queries limit what’s searched.

For a deeper look at TraceQL, read the TraceQL: A first-of-its-kind query language to accelerate trace analysis in Tempo 2.0 blog post.

For examples of query syntax, refer to Construct a TraceQL query.

Active development and limitations

TraceQL will be implemented in phases. The initial iteration of the TraceQL engine includes spanset selection and pipelines.

For more information about TraceQL’s design, refer to the TraceQL extensions abd TraceQL Concepts design proposals.

Write TraceQL queries in Grafana

Mon, 13 Jan 2025 17:58:36 +0000

Write TraceQL queries in Grafana

The Tempo data source’s query editor helps you query and display traces from Tempo in Explore. The queries use TraceQL, the query language designed specifically for tracing.

For general documentation on querying data sources in Grafana, refer to Query and transform data.

Before you begin

You can compose TraceQL queries in Grafana and Grafana Cloud using Explore and a Tempo data source.

Choose a query editing mode

The query editor has three modes, or Query types, that you can use to explore your tracing data. You can use these modes by themselves or in combination to create building blocks to generate custom queries.

The three Query types are:

The Search query builder provides a user interface for building a TraceQL query.
The TraceQL query editor lets you write your own TraceQL query with assistance from autocomplete.
The Service Graph view displays a visual relationship between services. Refer to the Service graph view documentation for more information.

Search query builder

The Search query builder provides drop-down lists and text fields to help you write a query. The query builder is ideal for people who aren’t familiar with or want to learn TraceQL.

Refer to the Search using the TraceQL query builder documentation to learn more about creating queries using convenient drop-down menus.

TraceQL query editor

The TraceQL query editor lets you search by trace ID and write TraceQL queries using autocomplete.

Refer to the TraceQL query editor documentation to learn more about constructing queries using a code-editor-like experience.

You can also search for a Trace ID by entering a trace ID into the query field.

Service graph view

Grafana’s service graph view uses metrics to display span request rates, error rates, and durations, as well as service graphs. Once the requirements are set up, this pre-configured view is immediately available.

Using the service graph view, you can:

Discover spans which are consistently erroring and the rates at which they occur
Get an overview of the overall rate of span calls throughout your services
Determine how long the slowest queries in your service take to complete
Examine all traces that contain spans of particular interest based on rate, error, and duration values (RED signals)

For more information about the service graph, refer to Service graph view.

Use TraceQL panels in dashboards

To add TraceQL panels to your dashboard, refer to the Traces panel documentation.

To learn more about Grafana dashboards, refer to the Use dashboards documentation.

TraceQL metrics queries

Thu, 24 Apr 2025 21:41:19 +0000

TraceQL metrics queries

Note
TraceQL metrics is an experimental feature. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Enable the feature toggle in Grafana to use this feature. Do not enable this feature in production environments.

TraceQL metrics is an experimental feature in Grafana Tempo that creates metrics from traces.

Metric queries extend trace queries by applying a function to trace query results. This powerful feature allows for ad hoc aggregation of any existing TraceQL query by any dimension available in your traces, much in the same way that LogQL metric queries create metrics from logs.

Traces are a unique observability signal that contain causal relationships between the components in your system.

TraceQL metrics can help answer questions like this:

How many database calls across all systems are downstream of your application?
What services beneath a given endpoint are failing?
What services beneath an endpoint are slow?

TraceQL metrics can help you answer these questions by parsing your traces in aggregate.

TraceQL metrics are powered by the TraceQL metrics API.

RED metrics, TraceQL, and PromQL

RED is an acronym for three types of metrics:

Rate, the number of requests per second
Errors, the number of those requests that are failing
Duration, the amount of time those requests take

For more information about the RED method, refer to The RED Method: how to instrument your services.

You can write TraceQL metrics queries to compute rate, errors, and durations over different groups of spans.

For more information on how to use TraceQL metrics to investigate issues, refer to Solve problems with metrics queries.

Enable and use TraceQL metrics

To use TraceQL metrics, you need to enable them on your Tempo database. Refer to Configure TraceQL metrics for more information.

From there, you can either query the TraceQL metrics API directly (for example, with curl) or using Grafana (recommended). To run TraceQL metrics queries in Grafana, you need Grafana Cloud or Grafana 10.4 or later. No extra configuration is needed. Use a Tempo data source that points to a Tempo database with TraceQL metrics enabled.

Refer to Solve problems using metrics queries for some real-world examples.

Functions

TraceQL metrics queries currently include the following functions for aggregating over groups of spans: rate, count_over_time, max_over_time, min_over_time, avg_over_time, quantile_over_time, histogram_over_time, and compare. These functions can be added as an operator at the end of any TraceQL query.

For detailed information and example queries for each function, refer to TraceQL metrics functions.

Exemplars

Exemplars are a powerful feature of TraceQL metrics. They allow you to see an exact trace that contributed to a given metric value. This is particularly useful when you want to understand why a given metric is high or low.

Exemplars are available in TraceQL metrics for all range queries. To get exemplars, you need to configure it in the query-frontend with the parameter query_frontend.metrics.max_exemplars, or pass a query hint in your query.

Example:

{ span:name = "GET /:endpoint" } | quantile_over_time(duration, .99) by (span.http.target) with (exemplars=true)

Note
TraceQL metric queries with exemplars aren’t fully supported in Grafana Explore. They will be supported in a future Grafana release.