--- title: "Tempo CLI | Grafana Tempo documentation" description: "Guide to using tempo-cli" --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Tempo CLI Tempo CLI is a separate executable that contains utility functions related to the Tempo software. Although it’s not required for a working installation, Tempo CLI can be helpful for deeper analysis or for troubleshooting. ## Tempo CLI command syntax The general syntax for commands in Tempo CLI is: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli command [subcommand] [options] [arguments...] ``` `--help` or `-h` displays the help for a command or subcommand. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli -h tempo-cli command [subcommand] -h ``` ## Run Tempo CLI Tempo CLI is available as source code and as a Docker image. ### Run with Docker (published image) Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash docker run --rm grafana/tempo-cli [arguments...] ``` ### Run directly with `go run` Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash go run ./cmd/tempo-cli [arguments...] ``` ### Build and run a local binary To build a local binary, you need a working Go installation and build environment. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash make tempo-cli ./bin/$(go env GOOS)/tempo-cli-$(go env GOARCH) [arguments...] ``` ### Build and run a local Docker image Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash make docker-tempo-cli docker run --rm tempo-cli [arguments...] ``` ## Backend options Tempo CLI connects directly to the storage backend for some commands, meaning that it requires the ability to read from S3, GCS, Azure or file-system storage. You can configure the backend using the following options: - Load an existing tempo configuration file using the `--config-file` (`-c`) option. This is the recommended option for frequent usage. Refer to [Configuration](../../configuration/) documentation for more information. - Specify individual settings: - `--backend ` The storage backend type, one of `s3`, `gcs`, `azure`, and `local`. - `--bucket ` The bucket name. The meaning of this value is backend-specific. Refer to [Configuration](../../configuration/) documentation for more information. - `--s3-endpoint ` The S3 API endpoint (i.e. s3.dualstack.us-east-2.amazonaws.com). - `--s3-user `, `--s3-pass ` The S3 user name and password (or access key and secret key). Optional, as Tempo CLI supports the same authentication mechanisms as Tempo. Refer to [S3 permissions documentation](../../configuration/hosted-storage/s3/) for more information. - `--insecure-skip-verify` skip TLS verification, only applies to S3 and GCS. Each option applies only to the command in which it’s used. For example, `--backend ` doesn’t permanently change where Tempo stores data. It only changes it for command in which you apply the option. ## TLS options All `query api` commands support these options for HTTPS and gRPC TLS connections. Use an `https://` URL for `trace-id`, or `--secure` for the other commands. - `--tls-cert `, `--tls-key ` PEM client certificate and matching unencrypted private key for mutual TLS (mTLS). Both must be provided together. - `--tls-ca ` PEM CA bundle used instead of the system trust roots for server certificate verification. - `--tls-server-name ` Override the server name used for certificate verification and SNI. The CA and server-name options can also be used without a client certificate. ## Query API command ### Trace ID Call the Tempo API and retrieve a trace by ID. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api trace-id ``` Arguments: - `api-endpoint` URL for the Tempo API. - `trace-id` Trace ID as a hexadecimal string. Options: - [TLS options](#tls-options) - `--org-id ` Organization ID (for use in multi-tenant setup). - `--header ` Extra HTTP header to send with the request. Can be specified multiple times. - `--v1` use v1 API (use /api/traces endpoint to fetch traces, default: /api/v2/traces). Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api trace-id http://tempo:3200 f1cfe82a8eef933b ``` Example with a custom authentication header: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api trace-id http://tempo:3200 f1cfe82a8eef933b --header "X-TOKEN=" ``` Replace *``* with your authentication token. ### Search Call the Tempo API and search using TraceQL. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search [ ] ``` Arguments: - `host-port` A host/port combination for Tempo. The scheme is inferred from the options. - `trace-ql` TraceQL query. - `start` Start of the time range to search in RFC3339 format (e.g. `2024-01-01T00:00:00Z`) or relative (e.g. `now-1h`) - `end` End of the time range to search in RFC3339 format (e.g. `2024-01-01T01:00:00Z`) or relative (e.g. `now`) Options: - [TLS options](#tls-options) - `--org-id ` Organization ID (for use in multi-tenant setup). - `--header ` Extra header to send with the request (as gRPC metadata when using `--use-grpc`). Can be specified multiple times. - `--use-grpc` Use GRPC streaming - `--spss ` Number of spans to return for each spanset - `--limit ` Number of results to return - `--path-prefix ` String to prefix search paths with - `--secure` Use HTTPS or gRPC with TLS > Note > > Set the `stream_over_http_enabled` flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to [Tempo GRPC API documentation](../../api_docs/). Example searching for error spans using relative time: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search localhost:3200 '{status = error}' now-1h now ``` Example searching for error spans using absolute time: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search localhost:3200 '{status = error}' 2024-01-01T00:00:00Z 2024-01-01T01:00:00Z ``` Example using GRPC streaming with organization ID: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search --use-grpc --org-id my-org localhost:3200 '{span.http.status_code >= 400}' now-1h now ``` Example with a custom authentication header over GRPC: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search --use-grpc --header "X-TOKEN=" localhost:9095 '{status = error}' now-1h now ``` ### Search tags Call the Tempo API and search attribute names. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search-tags [ ] ``` Arguments: - `host-port` A host/port combination for Tempo. The scheme will be inferred based on the options provided. - `start` Start of the time range to search in RFC3339 format (e.g. `2024-01-01T00:00:00Z`) or relative (e.g. `now-1h`) - `end` End of the time range to search in RFC3339 format (e.g. `2024-01-01T01:00:00Z`) or relative (e.g. `now`) Options: - [TLS options](#tls-options) - `--org-id ` Organization ID (for use in multi-tenant setup). - `--header ` Extra header to send with the request (as gRPC metadata when using `--use-grpc`). Can be specified multiple times. - `--use-grpc` Use GRPC streaming - `--path-prefix ` String to prefix search paths with - `--secure` Use HTTPS or gRPC with TLS > Note > > Set the `stream_over_http_enabled` flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to [Tempo GRPC API documentation](../../api_docs/). Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search-tags localhost:3200 ``` Example with relative time range: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search-tags localhost:3200 now-1h now ``` Example with absolute time range: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search-tags localhost:3200 2024-01-01T00:00:00Z 2024-01-02T00:00:00Z ``` ### Search tag values Call the Tempo API and search attribute values. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search-tag-values [ ] ``` Arguments: - `host-port` A host/port combination for Tempo. The scheme is inferred from the options. - `tag` The fully qualified TraceQL tag to search for. For example, `resource.service.name`. - `start` Start of the time range to search in RFC3339 format (e.g. `2024-01-01T00:00:00Z`) or relative (e.g. `now-1h`) - `end` End of the time range to search in RFC3339 format (e.g. `2024-01-01T01:00:00Z`) or relative (e.g. `now`) Options: - [TLS options](#tls-options) - `--org-id ` Organization ID (for use in multi-tenant setup). - `--header ` Extra header to send with the request (as gRPC metadata when using `--use-grpc`). Can be specified multiple times. - `--query ` TraceQL query to filter attribute results by. - `--use-grpc` Use GRPC streaming - `--path-prefix ` String to prefix search paths with - `--secure` Use HTTP or gRPC with TLS > Note > > Set the `stream_over_http_enabled` flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to [Tempo GRPC API documentation](../../api_docs/). Example to find all service names: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search-tag-values localhost:3200 resource.service.name ``` Example with query filter to find service names that have errors: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api search-tag-values --query '{status = error}' localhost:3200 resource.service.name ``` ### Metrics Call the Tempo API and generate metrics from traces using TraceQL. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api metrics [ ] ``` Arguments: - `host-port` A host/port combination for Tempo. The scheme will be inferred based on the options provided. - `trace-ql metrics query` TraceQL metrics query. - `start` Start of the time range to search in RFC3339 format (e.g. `2024-01-01T00:00:00Z`) or relative (e.g. `now-1h`) - `end` End of the time range to search in RFC3339 format (e.g. `2024-01-01T01:00:00Z`) or relative (e.g. `now`) Options: - [TLS options](#tls-options) - `--org-id ` Organization ID (for use in multi-tenant setup). - `--header ` Extra header to send with the request (as gRPC metadata when using `--use-grpc`). Can be specified multiple times. - `--use-grpc` Use GRPC streaming - `--instant` Perform an instant query instead of a range query. - `--path-prefix ` String to prefix search paths with - `--secure` Use HTTPS or gRPC with TLS > Note > > Set the `stream_over_http_enabled` flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to [Tempo GRPC API documentation](../../api_docs/). Example range query for request rates by service: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api metrics localhost:3200 '{} | rate() by (resource.service.name)' now-1h now ``` Example instant query for current error rates: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query api metrics --instant localhost:3200 '{status = error} | rate()' now-1h now ``` ## Query trace-id command Iterate over all backend blocks and dump all data found for a given trace ID. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query trace-id ``` > Note > > This can be intense as it downloads every bloom filter and some percentage of indexes/trace data. Arguments: - `trace-id` Trace ID as a hexadecimal string. - `tenant-id` Tenant to search. Options: - [Backend options](#backend-options) - `--percentage ` Percentage of blocks to scan (for example, 0.1 for 10%). Useful for sampling large datasets. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query trace-id f1cfe82a8eef933b single-tenant ``` Example scanning only 10% of blocks: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query trace-id --percentage 0.1 f1cfe82a8eef933b single-tenant ``` ## Query trace summary command Iterate over all backend blocks and dump a summary for a given trace ID. The summary includes: - number of blocks the trace is found in - span count - trace size - trace duration - root service name - root span info - top frequent service names Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query trace-summary ``` > Note > > This can be intense as it downloads every bloom filter and some percentage of indexes/trace data. Arguments: - `trace-id` Trace ID as a hexadecimal string. - `tenant-id` Tenant to search. Options: - [Backend options](#backend-options) - `--percentage ` Percentage of blocks to scan (for example, 0.1 for 10%). Useful for sampling large datasets. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query trace-summary f1cfe82a8eef933b single-tenant ``` ## List blocks Lists information about all blocks for the given tenant, and optionally perform integrity checks on indexes for duplicate records. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli list blocks ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. Options: - `--include-compacted` Include blocks that have been compacted. Default behavior is to display only active blocks. **Output:** Explanation of output: - `ID` Block ID. - `Lvl` Compaction level of the block. - `Objects` Number of objects stored in the block. - `Size` Data size of the block after any compression. - `Vers` Block version. - `Window` The window of time that was considered for compaction purposes. - `Start` The earliest timestamp stored in the block. - `End` The latest timestamp stored in the block. - `Duration` Duration between the start and end time. - `Age` The age of the block. - `Cmp` Whether the block has been compacted (present when –include-compacted is specified). Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli list blocks -c ./tempo.yaml single-tenant ``` ## List compaction summary Summarizes information about all blocks for the given tenant based on compaction level. This command is useful to analyze or troubleshoot compaction behavior. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli list compaction-summary ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli list compaction-summary -c ./tempo.yaml single-tenant ``` ## List cache summary Prints information about the number of bloom filter shards per day per compaction level. This command is useful to estimate and fine-tune cache storage. Read the [caching topic](../caching/) for more information. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli list cache-summary ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli list cache-summary -c ./tempo.yaml single-tenant ``` ## List column Lists values in a given column of a block. Useful for inspecting parquet block data directly. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli list column [column-name] ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. - `block-id` The block ID as UUID string. - `column-name` Column name to list values of (default: `TraceID`). Options: - [Backend options](#backend-options) Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli list column -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1 TraceID ``` ## View schema View block metadata, parquet schema structure, and column sizes for a given block. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli view schema ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. - `block-id` The block ID as UUID string. Options: - [Backend options](#backend-options) The output includes: - Block metadata - Parquet schema structure - Column sizes in KB Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli view schema -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1 ``` ## Benchmark profile Profile a local block for read-path benchmarking. Writes a JSON file recording what had to be measured from the block — its metadata, its row-group count, present and absent trace IDs to look up, and its attributes ranked by size — so that a benchmark run does not have to inspect the block, and every variant of an experiment works from the same measurements. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli benchmark profile ``` Arguments: - `block-path` Path to the block directory on local disk, laid out as `//`. Options: - `--trace-ids` Number of present trace IDs to sample, or `all` to enumerate every ID at run time rather than embedding them. Defaults to `10000`. One absent ID is derived per present ID. Pass `0` to skip trace IDs, and with them the full scan they require. - `--attributes` Number of attributes to profile in each scope (resource, span) and kind of value (string, numeric, boolean), ranked by total bytes as `analyse block` counts them: a resource attribute once per resource, a span attribute once per span. The `name`, `status`, `kind` and `duration` intrinsics are always included. Each records its cardinality, the share of spans carrying it, and the share carrying each of its 100 most frequent values, or, for a numeric attribute with more than 100 distinct values, quantiles of its values. Defaults to `100`. Pass `0` to skip the full scan it requires. Supported for vParquet5 blocks. - `-o`, `--out` File to write the profile to. Defaults to stdout. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli benchmark profile /data/traces/single-tenant/ca314fba-efec-4852-ba3f-8d2b0bbf69f1 --trace-ids=10000 -o profile.json ``` ## Benchmark run Run read-path benchmark queries against a local block and write the measurements as JSON. Takes a profile from `benchmark profile`, so the run does not inspect the block and every variant of an experiment measures the same queries. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli benchmark run -p ``` Arguments: - `block-path` Path to the block directory on local disk, laid out as `//`. Options: - `-p`, `--profile` Profile of the block, from `benchmark profile`. Required. - `-o`, `--out` File to write the result to. Defaults to stdout. - `-k`, `--case` Cases to run, as glob patterns over case IDs in the syntax `benchmark compare` takes, such as `'metrics/ranked_*/*/high/*'`. Repeat the flag or separate patterns with commas. Defaults to every case. Patterns matching no case are an error. - `--repeat` Passes over the query set. Defaults to `1`. - `--target-bytes-per-request` Bytes per search shard, mirroring the query frontend option of the same name. Defaults to `100MiB`. - `--search-limit` Traces a search returns per shard. Defaults to `20`. - `--max-series` Series a metrics query returns. Defaults to `1000`. - `--exemplars` Exemplars a metrics query collects. Defaults to `0`. - `--max-tag-values`, `--stale-value-threshold` A tag-value request’s `limit` and `maxStaleValues`. Each defaults to `0`, meaning no limit, as for a request without them. - `--read-buffer-size` Storage read buffer size, such as `8MiB`. Defaults to `0`, meaning Tempo’s default. This is the knob an experiment varies. - `--backend-latency`, `--backend-bandwidth` Simulate an object store: every backend request waits the latency plus its size over the bandwidth, a size per second such as `100MiB`. Each defaults to `0`, which reads the local block as is. Without them, a local read costs almost nothing, so an option that trades request count for bytes read, like `--read-buffer-size`, only shows up in `backend.*` and not in latency. Each case records how many results it matched, and one `metrics` map. Two runs are only comparable if the match counts agree, so a difference there means the comparison is invalid rather than interesting. Every measurement has the same shape whatever its source, so nothing reading a result needs a rule per source: - `total` is the sum over the case for a `counter`, or the value left behind for a `gauge`. - `summary` describes the per-execution values, with quantiles so a box plot needs nothing else. A total on its own hides the tail, which on the read path is usually the interesting part. Keys are `source.name`: - `harness.*` is what the benchmark timed itself: `wallNs`, `cpuNs`, `allocBytes`, `allocCount`. - `backend.*` is object-store traffic: `reads`, `bytes`, `timeNs`. - `response.*` is what a query API reported, under Tempo’s own metric names. A metric Tempo did not report is absent rather than zero, because the two are different claims, so a summary’s `count` says how many executions reported it. - `process.*` is what Tempo emitted to its Prometheus registry, including the Go runtime and process collectors. Metrics are collected rather than listed, so a metric added to Tempo appears without a change to the benchmark. Only metrics that moved are kept. The query set has fixed cases, plus cases generated from the profile’s ranked attributes. Fixed cases: - `traceid/present`, `traceid/absent`: trace lookups by ID, for IDs the block holds and IDs it does not. - `search/nopredicate`: an unfiltered search. - `metrics/rate`, `metrics/rate-by-service`: `rate()` and `rate() by (resource.service.name)`. - `metrics/drilldown/*`: the queries Traces Drilldown runs when it opens, such as `{nestedSetParent<0 && true} | rate() with(sample=true)`. - `metadata/tagnames/`: a tag-name lookup in each attribute scope. Generated cases, from the top 10 attributes with a dedicated column and the top 10 without: - `` is the group an attribute was picked from: `ranked_dedicated` for an attribute with a column of its own, which every intrinsic and `resource.service.name` have, and `ranked_generic` for one in the generic key/value columns. - Each attribute is queried with up to three of its values, one per selectivity band: `high` is its most common value, `medium` the middle of its profiled values, and `low` the least common value the profile kept. Every value is one the block holds, so every query matches. - `search///`: `{ attribute = value }`. - `metrics////`: `{ attribute = value } | rate()` and `{ attribute = value } | quantile_over_time(duration, 0.99, 0.95)`. - `metadata//tagvalues/`: a tag-value lookup with no TraceQL, for the top 20 string attributes in each group. - Attributes with only quantile values are skipped, and a profile built with `--attributes 0` generates no cases. To run only some cases, pass `--case` a glob, such as `--case 'metrics/ranked_generic/*/low/*'`. Metrics queries run over the block’s whole time range, stepping at `max(60s, window/30)` to land about 30 points. Searches and metrics queries are split into shards of row groups, mirroring how the query frontend splits a job. A case that fails is recorded with its error and the rest of the run continues. Benchmarking trace lookups reads the block’s bloom filters, so a partial block copy without them can still be profiled but only its search cases will run. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli benchmark profile /data/traces/single-tenant/ca314fba-efec-4852-ba3f-8d2b0bbf69f1 -o profile.json tempo-cli benchmark run /data/traces/single-tenant/ca314fba-efec-4852-ba3f-8d2b0bbf69f1 -p profile.json -o result.json ``` ## Benchmark compare Compare two or more results from `benchmark run`, and write the comparison as markdown to stdout, to read or paste into a pull request, or serve it as web pages to explore. For each metric it lays every case out as benchstat does: the baseline’s value, then each other run’s value and its change from the baseline. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli benchmark compare ... ``` A change of 10% or more is in bold. A run that cannot be compared with the baseline on a case reads `not comparable`, with the reason under the table: its match or execution count per pass differs, or it is missing the case or failed it. When a run’s name is too long to head a column, runs are numbered, and listed with their numbers. With `--http`, it serves the comparison as web pages instead, rendered on each request as pprof’s web view is. The summary page has a table per metric, with links to switch the metric and the percentile. Each case has a page with the box plots and table of every metric at once, and hovering a box plot’s row shows its numbers. A box spans the 25th to 75th percentile, with a mark at the median. Its whisker runs from the minimum to the 99th percentile, with a tick at the 90th. The axis stops near the highest 99th percentile, so a maximum far past it is marked at the edge and written out, rather than squashing every box to make room for it. Clicking a run on either page makes it the baseline. As with pprof, an address without a host, like `:8080`, is served on localhost only. Arguments: - `results` Results to compare, as `name=path` or a path. The first is the baseline. Options: - `-m`, `--metric` Metrics to show, as glob patterns over the metric keys. Defaults to `harness.wallNs`, `harness.cpuNs`, `harness.allocBytes`, `backend.bytes`, and `backend.reads`. - `-k`, `--case` Cases to show, as glob patterns over the case IDs, for example `traceid/*`. Defaults to every case. - `--percentile` Percentile the summaries show: `min`, `p25`, `p50`, `p75`, `p90`, `p99`, or `max`. Defaults to `p99`, since tail latency is what hurts most. - `--http` Serve the comparison as web pages on this address, like `:8080`, instead of writing markdown. A result given as a path is named after the settings that set it apart from the others: the run options and git SHA that differ between the runs, or, when those are all the same, the Go version, `GOMAXPROCS`, or host. When one setting differs, the name is its value, like `4MiB`. When several do, the name lists each, like `targetBytesPerRequest=2MiB readBufferSize=4MiB`. Runs set up alike, such as repeats of one setup, are named after their files instead. When the one setting that differs is a number, the runs are put in its order, with those left at Tempo’s default first, so reading across the columns follows the setting as it grows. The baseline stays the run given first wherever it lands. Above the tables, each run has a line saying how it differs from the baseline, like `readBufferSize default → 4MiB`. The baseline’s line shows what the others are measured from: its value of every setting that differs, then its git SHA, Go version, `GOMAXPROCS`, and host. A difference in where a run happened, its Go version, `GOMAXPROCS`, or host, is flagged with ⚠, since latencies from two environments are hard to compare. The summaries are per execution: one trace lookup, or one shard of a search, metrics, or tag-name query. The spread of a box is across those executions, not across repeated runs, so it describes how the inputs differ, not how noisy the measurement is. Example, where the runs are named `default`, `4MiB`, and `16MiB` from their read buffer sizes: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli benchmark compare main.json read-buffer-4mib.json read-buffer-16mib.json -k 'traceid/*' > comparison.md tempo-cli benchmark compare main.json read-buffer-4mib.json read-buffer-16mib.json --http=:8080 ``` ## Query search command Search blocks in a given time range for a specific key/value pair. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query search ``` > Note > > This can be intense as it downloads all relevant blocks and iterates through them. Arguments: - `name` Name of the attribute to search for, for example, `http.method`. - `value` Value of the attribute to search for, for example, `GET`. - `start` Start of the time range to search in RFC3339 format (e.g. `2024-01-01T00:00:00Z`) or relative (e.g. `now-1h`) - `end` End of the time range to search in RFC3339 format (e.g. `2024-01-01T01:00:00Z`) or relative (e.g. `now`) - `tenant-id` Tenant to search. Options: - [Backend options](#backend-options) Example using relative time: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query search http.method GET now-1h now single-tenant --backend=gcs --bucket=tempo-trace-data ``` Example using absolute time: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli query search http.method GET 2024-01-01T00:00:00Z 2024-01-01T00:05:00Z single-tenant --backend=gcs --bucket=tempo-trace-data ``` ## Parquet convert A to B command Converts a vParquet file (actual data.parquet) of format A to a block of newer format B with an optional list of dedicated attribute columns. Actual supported versions for A and B vary by Tempo release. This utility command is useful when testing the impact of different combinations of dedicated columns. ### Convert vParquet4 to vParquet5 Converts a vParquet4 block to vParquet5 format with an optional list of dedicated attribute columns. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli parquet convert-4to5 [] [] ``` Arguments: - `in file` Path to an existing vParquet4 block directory. - `out path` Path to write the vParquet5 block to. The default is `./out`. - `list of dedicated columns` Optional list of columns to make dedicated. Columns use TraceQL syntax with scope. For example, `span.http.method`, `resource.namespace`, `event.exception.message`. Column prefixes: - `int/` marks the column as an integer type. For example, `int/span.http.status_code`. - `blob/` marks the column for blob encoding. For example, `blob/span.db.statement`. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli parquet convert-4to5 ./block-in ./block-out "span.http.method" "int/span.http.status_code" "blob/span.db.statement" ``` ## Migrate tenant command Copy blocks from one backend and tenant to another. Blocks can be copied within the same backend or between two different backends. The data format isn’t converted but the tenant ID in `meta.json` is rewritten. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli migrate tenant ``` Arguments: - `source tenant` Tenant to copy blocks from - `dest tenant` Tenant to copy blocks into Options: - `--source-config-file ` (required) Configuration file for the source backend. - `--config-file ` Configuration file for the destination backend. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli migrate tenant --source-config-file source.yaml --config-file dest.yaml my-tenant my-other-tenant ``` ## Migrate overrides config command Migrate the overrides section of a full Tempo config file from the legacy flat format to the new scoped format. The command reads the full config, converts any legacy overrides, and outputs only user-set values. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli migrate overrides-config ``` Arguments: - `config-file` Path to the full Tempo config file. Options: - `-d, --config-dest ` Path to write the migrated overrides section. If not specified, output is printed to `stdout`. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli migrate overrides-config config.yaml -d migrated-overrides.yaml ``` > Warning > > - Fields set to Go zero values (`false`, `0`, `""`) may be silently dropped due to `omitempty` tags. Compare against your original config to ensure nothing is lost. > - Secret values (for example, `remote_write_headers`) are masked as `` in the output. You must manually restore the original values. > - Some struct fields without `omitempty` may appear with zero values (for example, `exclude: null`) that were not in your original config. ## Migrate overrides per-tenant command Migrate a per-tenant overrides file from the legacy flat format to the new scoped format. The command handles both legacy and new format entries, and outputs only tenant-specific values. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli migrate overrides-per-tenant ``` Arguments: - `overrides-file` Path to the per-tenant overrides file. Options: - `-d, --output-dest ` Path to write the migrated per-tenant overrides. If not specified, output is printed to `stdout`. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli migrate overrides-per-tenant overrides.yaml -d migrated-overrides.yaml ``` > Warning > > - Fields set to Go zero values (`false`, `0`, `""`) may be silently dropped due to `omitempty` tags. Compare against your original config to ensure nothing is lost. > - Secret values (for example, `remote_write_headers`) are masked as `` in the output. You must manually restore the original values. > - Some struct fields without `omitempty` may appear with zero values (for example, `exclude: null`) that were not in your original config. ## Migrate config command Migrate a Tempo 2.x configuration file to a valid 3.0 configuration. The command removes obsolete configuration sections (such as `ingester`, `ingester_client`, and `compactor`), adds Kafka ingest configuration for microservices mode, disables compaction in overrides for parallel operation during migration, and strips the removed `local-blocks` metrics-generator processor. The tool works at the YAML map level rather than rewriting the file from fully decoded Tempo structs, so environment variable references like `${VAR}` are preserved. Top-level sections that are not recognized by the Tempo 3.0 configuration are dropped from the output. Unknown nested keys normally cause validation to fail, but validation is best-effort when the configuration contains `${VAR}` placeholders where non-string types are expected, which may let unknown nested keys through. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli migrate config [options] ``` Arguments: - `config-file` Path to the 2.x Tempo configuration file. Options: - `--kafka-address
` Kafka broker address. Required when running in microservices mode. - `--kafka-topic ` Kafka topic name. Defaults to `tempo`. - `--mode ` Override automatic deployment mode detection. By default, the mode is detected from the `target` field (`all` or absent means monolithic, any other value means microservices). The migrated configuration is printed to `stdout`. Warnings are printed to `stderr`. ### Examples Monolithic mode (no Kafka flags needed): Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli migrate config old-config.yaml > new-config.yaml ``` Microservices mode: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli migrate config --kafka-address=kafka:9092 --kafka-topic=tempo-traces old-config.yaml > new-config.yaml ``` > Warning > > - The output is a starting point for your 3.0 configuration. Always review it before deploying. > - If your configuration uses legacy (flat) overrides, you must run `tempo-cli migrate overrides-config` first. > - If your configuration references an external per-tenant overrides file (`per_tenant_override_config`), you must manually add `compaction_disabled: true` for each tenant in that file. > - YAML comments and key ordering from the original file are not preserved. > - Remove `compaction_disabled: true` from overrides after fully decommissioning your 2.x deployment. ## Analyse block Analyses a block and outputs a summary of the block’s generic attributes. It’s of particular use when trying to determine candidates for dedicated attribute columns. The output includes span, resource, and event attributes with cardinality and size information. Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. - `block-id` The block ID as UUID string. Options: - [Backend options](#backend-options) - `--num-attr ` Number of attributes to output (default: 20) - `--num-int-attr ` Number of integer attributes to display. If set to 0, uses the `--num-attr` value (default: 5) - `--blob-threshold ` Mark attributes as blob candidates when their dictionary size per row group exceeds this value. Set to 0 to disable. (default: 4MiB) - `--include-well-known` Include well-known attributes in the analysis. Enable when generating dedicated columns for vParquet5 or higher. (default: false) - `--generate-jsonnet` Generate Jsonnet overrides for dedicated columns - `--generate-cli-args` Generate command-line arguments for the parquet conversion command - `--int-percent-threshold ` Threshold for integer attributes in dedicated columns (default: 0.05) - `--str-percent-threshold ` Threshold for string attributes in dedicated columns (default: 0.03) - `--simple-summary` Print only a single line of top attributes (default: false) - `--print-full-summary` Print full summary of the analysed block (default: true) Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli analyse block --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0 ``` Example with blob detection: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli analyse block --blob-threshold=4MiB --generate-jsonnet --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0 ``` ## Analyse blocks Analyses all blocks in a given time range and outputs a summary of the blocks’ generic attributes. It’s of particular use when trying to determine candidates for dedicated attribute columns. The output includes span, resource, and event attributes with cardinality and size information. Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single-tenant setups. Options: - [Backend options](#backend-options) - `--num-attr ` Number of attributes to output (default: 20) - `--num-int-attr ` Number of integer attributes to display. If set to 0, uses the `--num-attr` value (default: 5) - `--min-compaction-level ` Minimum compaction level to include in the analysis (default: 3) - `--max-blocks ` Maximum number of blocks to analyze (default: 10) - `--max-start-time ` Oldest start time for a block to be processed. RFC3339 format (default: disabled) - `--min-start-time ` Newest start time for a block to be processed. RFC3339 format (default: disabled) - `--blob-threshold ` Mark attributes as blob candidates when their dictionary size per row group exceeds this value. Set to 0 to disable. (default: 4MiB) - `--include-well-known` Include well-known attributes in the analysis. (default: false) - `--jsonnet` Generate Jsonnet overrides for dedicated columns - `--cli` Generate command-line arguments for the parquet conversion command - `--int-percent-threshold ` Threshold for integer attributes in dedicated columns (default: 0.05) - `--str-percent-threshold ` Threshold for string attributes in dedicated columns (default: 0.03) - `--simple-summary` Print only a single line of top attributes (default: false) - `--print-full-summary` Print full summary of the analysed block (default: true) Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli analyse blocks --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant ``` Example with blob detection and Jsonnet output: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli analyse blocks --blob-threshold=4MiB --jsonnet --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant ``` ## Suggest columns Suggests dedicated columns for a tenant based on analysis of blocks. This command analyzes block data and outputs configuration recommendations in YAML or Jsonnet format that can be used to configure dedicated attribute columns. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli suggest columns ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. Options: - [Backend options](#backend-options) - `--block-id ` Specific block ID to analyse. If not provided, analyzes multiple blocks. - `--min-compaction-level ` Minimum compaction level to analyse (default: 3) - `--max-blocks ` Maximum number of blocks to analyse (default: 10) - `--num-attr ` Number of attributes to display (default: 20) - `--num-int-attr ` Number of integer attributes to display. If set to 0, uses the `--num-attr` value (default: 5) - `--int-percent-threshold ` Threshold for integer attributes put in dedicated columns (default: 0.05) - `--str-percent-threshold ` Threshold for string attributes in dedicated columns (default: 0.03) - `--include-well-known` Include well-known attributes in the analysis (default: true) - `--blob-threshold ` Convert column to blob when dictionary size reaches this value (default: 4MiB) - `--max-start-time ` Oldest start time for a block to be processed. RFC3339 format. - `--min-start-time ` Newest start time for a block to be processed. RFC3339 format. - `-o, --out ` File to write output to. If not specified, output is printed to stdout. - `-f, --format ` Output format (default: yaml) Example outputting YAML to a file: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli suggest columns -c ./tempo.yaml --format yaml --out suggestions.yaml single-tenant ``` Example outputting Jsonnet for use in overrides: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli suggest columns -c ./tempo.yaml --format jsonnet single-tenant ``` ## Generate attribute index > Warning > > This command is EXPERIMENTAL and meant to facilitate experimentation with different kinds of indexes. Generate an attribute index for a parquet block. This creates an index file that can be used for faster attribute lookups. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli gen attr-index ``` Arguments: - `input-path` Path to the input parquet block directory. Options: - `--add-intrinsics` Add intrinsic attributes to the index such as name, kind, status, and others. - `--index-types ` Type of index to generate (default: rows,codes) Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli gen attr-index ./path/to/block ``` Example with intrinsic attributes: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli gen attr-index --add-intrinsics ./path/to/block ``` ## Experimental traces diff > Warning > > This command is experimental. The output format and behavior may change in future releases. Compare two local trace JSON files. The default `trace-patch-v0` format returns the complete mechanical change list. You can instead request a compact native summary or a composed summary with a size-bounded patch. Use this command to compare traces captured at different times or from different environments, for example, to understand how a deployment changed trace structure. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli experimental traces-diff --trace-a --trace-b ``` Arguments: - `--trace-a ` (required) Path to the baseline trace JSON file. - `--trace-b ` (required) Path to the comparison trace JSON file. Options: - `--format ` Output format: `trace-patch-v0` (default), `trace-summary-v0-native`, or `trace-summary-v0-composed`. - `-o, --out ` File to write output to. If not specified, output is printed to `stdout`. - `--pretty` Pretty-print JSON output. The input files can be either raw OpenTelemetry JSON traces or Tempo `TraceByIDResponse` JSON responses. The `trace-summary-v0-composed` format always includes a `trace-summary-v0-native` document. If the serialized `trace-patch-v0` document is no larger than 64 KiB, it is included in `patch`. Otherwise, `patchOmitted` reports its size and the reason `over_budget`; rerun with `--format trace-patch-v0` to retrieve the full patch. Example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli experimental traces-diff --trace-a baseline.json --trace-b compare.json --pretty ``` Example writing output to a file: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli experimental traces-diff --trace-a baseline.json --trace-b compare.json -o diff-output.json ``` Example producing the native summary: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli experimental traces-diff \ --trace-a baseline.json \ --trace-b compare.json \ --format trace-summary-v0-native \ --pretty ``` The native summary calculates latency, the sum of inclusive span durations, topology, and significant per-service duration drift from the normalized traces. It combines those values with matcher-derived change and error counts. `changedServices` and service rollups cover all matcher changes and significant duration drift; `changedServices` also includes structure-only changes, which do not have service rollups. Warnings report partial inputs, high-cardinality span names, duplicate span IDs, ambiguous duplicate-span matching, empty traces, and invalid durations. Invalid spans contribute zero to summary duration aggregates. ## Drop traces by ID Rewrites all blocks for a tenant that contain specific trace IDs. The traces are dropped from the new blocks and the rewritten blocks are marked compacted so they will be cleaned up. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli rewrite-blocks drop-traces ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. - `trace-ids` The comma-separated trace IDs to drop (also supports single trace ID). Options: - [Backend options](#backend-options) - `--drop-trace` By default, this command runs in dry run mode. Supplying this argument causes it to actually rewrite blocks with the traces dropped. - `--background` Run in background mode (default: false). Suppresses progress dots for use in automated scripts. ### Examples Dry run (default) to see which blocks would be affected: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli rewrite-blocks drop-traces --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2 ``` Drop one trace (actually perform the operation): Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli rewrite-blocks drop-traces --drop-trace --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2 ``` Drop multiple traces: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli rewrite-blocks drop-traces --drop-trace --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2,111fa1850042aea83c17cd7e674210b8 ``` ## Redact traces Remove traces containing personally identifiable information or other sensitive data from object storage without waiting for retention to expire. > Warning > > Redaction rewrites blocks in object storage and can’t be undone. You’re responsible for verifying the selection. If you redact by query, confirm in Grafana Explore that the query selects the right traces, then run it with `--dry-run` and check the match count. Explore shows only a sample; redaction removes every matching trace across the tenant. The `redact` command submits a redaction request to the [backend scheduler](/docs/tempo/next/reference-tempo-architecture/components/compaction/#backend-scheduler). The scheduler creates jobs that rewrite affected blocks in object storage to remove the specified traces. Unlike [`drop-traces`](#drop-traces-by-id), which operates directly on object storage from the CLI, `redact` delegates the work to the backend scheduler over gRPC. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli redact --tenant= --trace-id= [--trace-id= ...] ``` Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli redact --tenant= --query= [--start= --end=] ``` Arguments: - `scheduler-address` The backend scheduler gRPC address (`host:port`). Options: - `--tenant ` **(required)** Tenant ID. - `--trace-id ` Trace ID to redact, in hex format. Repeat the flag for several traces in one request (`--trace-id= --trace-id=`, not comma-separated), up to 1000. Every job the redaction creates carries the whole list, so a longer list costs one copy per block; use `--query` instead. Mutually exclusive with `--query`. - `--query ` TraceQL query selecting the traces to redact, for example `{ span.http.status_code = 500 }`. Mutually exclusive with `--trace-id`. The query is restricted to a single spanset filter over the matched span’s own `resource.*` or `span.*` attributes, joined by `&&` or `||`: `=` comparisons, and the existence check `attr != nil`. Regular expressions, `!=` against a value, ordered comparisons, `parent.`-scoped attributes, and pipelines or aggregates aren’t supported. - `--dry-run` Evaluate the selector without rewriting any blocks. After the dry-run jobs complete, match counts are added to `tempo_backend_scheduler_redaction_traces_found_total` (`mode="dry_run"`). The command doesn’t print the count (default: `false`). - `--start ` Start of the time window. Accepts `now`, a relative offset such as `now-7d`, or an RFC3339 timestamp. Must be given with `--end`, must be before `--end`, and cannot be combined with `--trace-id`. Omit both bounds to redact the whole tenant. - `--end ` End of the time window. Same forms as `--start`. Must be given with `--start`. - `--tls` Use TLS for the gRPC connection (default: `false`). - `--tls-server-name ` Override the TLS server name (SNI). - `--tls-ca ` Path to a PEM-encoded CA certificate file. You must provide exactly one of `--trace-id` or `--query`. Providing both, or neither, returns an error before the request is submitted. `attr != nil` is rewritten to an existence check, so it selects spans that **have** the attribute, not the ones missing it. Other `!=` forms are refused. This can be broader than it looks on a large tenant (`span.user_id != nil` matches every span carrying a user ID), so preview with `--dry-run` first. A tenant can have only one redaction in progress at a time, dry runs included. A submission made while an earlier one is still running, or still in its quiescence period, is rejected. A trace-ID list larger than the cap therefore has to be redacted as successive batches rather than several at once, which is another reason to prefer `--query`. On success, the command prints the batch ID and the number of jobs created: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none batch_id: jobs_created: ``` When `--dry-run` is set, the command also prints a line indicating that no blocks were rewritten: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none batch_id: jobs_created: mode: dry-run (jobs will report match counts; no blocks will be rewritten) ``` ### Before you submit a redaction Redaction only rewrites blocks that already exist in object storage when you submit. Traces still held by block-builders aren’t covered, so recently ingested traces can survive a run. 1. Stop ingesting the sensitive data at its source. 2. Wait for the current blocks to flush to object storage. Blocks flush on an interval of a few minutes; allow around 10 minutes to be safe, so the traces you want to remove land in blocks the redaction can reach. If you use `--query`: 1. [Check your redaction query in Grafana Explore](#check-your-redaction-query-in-grafana-explore) to confirm it selects the right traces. 2. Run the same query with `--dry-run`. The command returns as soon as jobs are created; it doesn’t print the match count. Wait until those jobs complete. 3. Monitor progress on the [`/status/backendscheduler`](/docs/tempo/next/api_docs/#backend-scheduler-job-status) endpoint. 4. Read `tempo_backend_scheduler_redaction_traces_found_total` for your tenant with `mode="dry_run"`. The metric is a counter that increments when each job finishes, so use an increase over the run or the **Dry-run Blast Radius / h** panel on the **Tempo - Backend Work** dashboard. A value of zero can mean no matches or that jobs haven’t finished yet. For the metric and dashboard, refer to [Key metrics](/docs/tempo/next/reference-tempo-architecture/components/compaction/#key-metrics). If the count is far larger than the Explore sample, narrow the query first. Then submit the redaction. For commands, refer to [Examples](#examples). ### Check your redaction query in Grafana Explore Use this procedure when you redact with `--query`. Enter the same query string you pass to `--query`. 1. In Grafana, go to **Explore** and select your Tempo data source. 2. For **Query type**, select **TraceQL**, then enter your redaction query, for example `{ span.http.status_code = 500 }`. 3. Select **Run query** and inspect the matching traces in the results. For help building and running TraceQL queries, refer to [TraceQL queries in Grafana](/docs/tempo/next/traceql/). The `--query` option accepts only a restricted subset of TraceQL: a single spanset filter with positive equality matchers. That subset is deliberately narrow so a redaction query can’t widen its own match set the way negation or regular expressions could. A broader query that works in Explore may be rejected at submission. Explore shows a sample; use `--dry-run` for the count before you apply the redaction. ### Redact a time window A redaction with no window covers every block the tenant has, which keeps the tenant’s compaction paused for the whole run and lets its block list grow. `--start` and `--end` scope a redaction to a time range, so a large tenant can be redacted in slices with compaction recovering in between. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli redact --tenant=my-tenant --query='{resource.namespace = "checkout"}' --start=now-7d --end=now-6d localhost:9095 ``` Both bounds are inclusive and must both be supplied. Blocks whose data range overlaps the window are read, as are blocks whose recorded range is unusable — those are included rather than skipped, so that a block whose timestamps cannot be judged is never silently left behind. Inside each block the window bounds the scan, and a trace is redacted if any part of it overlaps: a trace that starts before the window and ends inside it is removed in full, including its earlier spans. A window cannot be combined with `--trace-id`. The window scopes which blocks are read and is not applied per trace, so the pair would remove each listed trace only from the blocks that happen to overlap and leave the rest of it in place while reporting success. Redact by trace ID without a window. The window is resolved to absolute timestamps when the command runs, so a long redaction does not drift forward into data that arrived after it started. Traces outside every window you run are left in place. Repeat the command for each slice, but expect to wait between slices. A finished redaction is held briefly before it is cleared, and a second submission for the same tenant is rejected with `AlreadyExists` until that completes — a couple of minutes at the default maintenance interval. > Note > > A single run doesn’t remove everything. Two things bound what it covers: > > - Redaction snapshots the tenant’s block list at submission and only rewrites those blocks (plus output from compactions that were already running). Traces still held by block-builders, or ingested after you submit, are untouched, so a single pass over a live tenant is never complete. > - Search results are cached, so re-running the same search can still list redacted traces until that cache entry expires. To confirm a redaction, query by trace ID or vary the time range so the cache key differs. > Warning > > A backend-worker that predates `--start`/`--end` does not respect the window and scans each block it is given in full. The job still reports success, so there is no signal afterwards and the removed traces cannot be recovered. Wait until every worker for the cell supports the window before submitting a windowed redaction; an unwindowed redaction is unaffected. Monitor job progress through the [`/status/backendscheduler`](/docs/tempo/next/api_docs/#backend-scheduler-job-status) endpoint. ### Examples Redact a single trace: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d localhost:9095 ``` Redact multiple traces in one request: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d --trace-id=00000000000000000000000000000001 localhost:9095 ``` Preview the traces a query would match, without rewriting any blocks: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli redact --tenant=my-tenant --query='{ span.http.status_code = 500 }' --dry-run localhost:9095 ``` After you preview, redact all traces matching that query: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli redact --tenant=my-tenant --query='{ span.http.status_code = 500 }' localhost:9095 ``` With TLS and a custom CA: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d --tls --tls-ca=/path/to/ca.pem scheduler.example.com:9095 ```