--- title: "Push spans | Grafana Tempo documentation" description: "Learn how to push spans to Tempo using HTTP or gRPC with the OpenTelemetry Protocol (OTLP) receiver." --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Push spans with HTTP or gRPC Sometimes using a tracing system is intimidating because it seems like you need complex application instrumentation or a span ingestion pipeline to push spans. This guide shows a basic technique for pushing spans with HTTP/JSON or gRPC using the [OpenTelemetry](https://opentelemetry.io/docs/specs/otlp/) receiver. ## Before you begin This procedure uses an example Docker Compose setup to run Tempo, so you don’t need an existing installation. The Docker image also includes a Grafana container which lets you visualize traces. To use this procedure, you need: - Docker and `docker compose` installed - (Optional) [OpenTelemetry `telemetrygen`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen) installed for the gRPC examples ## Start Tempo using the quickstart Use the instructions in the [Quick start for Tempo documentation](/docs/tempo/next/getting-started/docker-example/) to start a local instance of Tempo and Grafana. ## Push spans with OTLP/HTTP Now that Tempo is running and listening on port 4318 for [OTLP/HTTP spans](https://opentelemetry.io/docs/specs/otlp/#otlphttp), you can push a span to Tempo using `curl`. Before you can use this example, you need to update the start and end time. The `startTimeUnixNano` and `endTimeUnixNano` fields are in nanoseconds, where 100000000 nanoseconds is 100 milliseconds. The nanosecond value can be obtained by any tool that provides the epoch date in nanoseconds. For example, under Linux, you can use `date +%s%N`. You can also use an online tool such as [Epoch Converter](https://www.epochconverter.com/) to get the current time in nanoseconds. 1. Copy and paste the `curl` command into a text editor. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -X POST -H 'Content-Type: application/json' http://localhost:4318/v1/traces -d ' { "resourceSpans": [{ "resource": { "attributes": [{ "key": "service.name", "value": { "stringValue": "my.service" } }] }, "scopeSpans": [{ "scope": { "name": "my.library", "version": "1.0.0", "attributes": [{ "key": "my.scope.attribute", "value": { "stringValue": "some scope attribute" } }] }, "spans": [ { "traceId": "5B8EFFF798038103D269B633813FC700", "spanId": "EEE19B7EC3C1B100", "name": "I am a span!", "startTimeUnixNano": 1689969302000000000, "endTimeUnixNano": 1689970000000000000, "kind": 2, "attributes": [ { "key": "my.span.attr", "value": { "stringValue": "some value" } }] }] }] }] }' ``` 2. Replace `startTimeUnixNano` and `endTimeUnixNano` with current values for the last 24 hours to allow you to search for them using a 24 hour relative time range. You can get this in seconds and milliseconds from the [Unix Epoch Clock](https://www.epochconverter.com/). For `startTimeUnixNano`, use the current time minus 24 hours. Use the link above to figure out the value. For `endTimeUnixNano`, use the current time. Multiply the milliseconds value by 1,000,000 to turn it into nanoseconds. You can do this from a bash terminal with the following command. Replace `` with the milliseconds value. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash echo $(( * 1000000)) ``` 3. Copy the updated curl command to a terminal window and run it. 1. View the trace in Grafana: 1. Open a browser window to http://localhost:3000. 2. Open the **Explore** page and select the Tempo data source. 3. Select the **Search** query type. 4. Select **Run query** to list available traces. 5. Select the trace ID (yellow box) to view details about the trace and its spans. ## Push spans with OTLP/gRPC Tempo also listens on port 4317 for [OTLP/gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc) spans. You can use [OpenTelemetry `telemetrygen`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen) to push test traces over gRPC. 1. Install `telemetrygen` following the [installation instructions](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen). 2. Push traces to Tempo over gRPC: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash telemetrygen traces --otlp-insecure --rate 20 --duration 5s --otlp-endpoint localhost:4317 ``` This sends 100 traces (20 per second for 5 seconds) to the Tempo distributor’s OTLP/gRPC receiver. 3. Verify the traces arrived by searching for them. The `telemetrygen` tool uses the service name `telemetrygen` by default: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -G -s http://localhost:3200/api/search --data-urlencode 'q={ resource.service.name = "telemetrygen" }' | jq . ``` ### Multi-tenancy If you’ve enabled multi-tenancy (`multitenancy_enabled: true`), pass the tenant ID using the `X-Scope-OrgID` header in gRPC metadata: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash telemetrygen traces --otlp-insecure --rate 20 --duration 5s \ --otlp-endpoint localhost:4317 \ --otlp-header "X-Scope-OrgID=my-tenant" ``` For more information about configuring OpenTelemetry SDKs to export traces over gRPC, refer to the [OpenTelemetry documentation](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc). ## Retrieve traces You can retrieve the trace by running a `curl` command against Tempo. The returned format is [OTLP](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto). 1. Replace the trace ID in the `curl` command with the trace ID that was generated from the push. This information is in the data that’s sent with the `curl`. You can use the Grafana **Explore** page to find this, as shown in the previous section. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl http://localhost:3200/api/v2/traces/5b8efff798038103d269b633813fc700 {"trace": {"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]},"scopeSpans":[{"scope":{"name":"my.library","version":"1.0.0"},"spans":[{"traceId":"W47/95gDgQPSabYzgT/HAA==","spanId":"7uGbfsPBsQA=","name":"I am a span!","kind":"SPAN_KIND_SERVER","startTimeUnixNano":"1689969302000000000","endTimeUnixNano":"1689970000000000000","attributes":[{"key":"my.span.attr","value":{"stringValue":"some value"}}],"status":{}}]}]}]}} ``` 2. Copy and paste the updated `curl` command into a terminal window. ### Use TraceQL to search for a trace Alternatively, you can also use [TraceQL](/docs/tempo/next/traceql/) to search for the trace that was pushed. You can search by using the unique trace attributes that were set: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -G -s http://localhost:3200/api/search --data-urlencode 'q={ resource.service.name = "my.service" }' {"traces":[{"traceID":"5b8efff798038103d269b633813fc700","rootServiceName":"my.service","rootTraceName":"I am a span!","startTimeUnixNano":"1694718625557000000","durationMs":10000,"spanSet":{"spans":[{"spanID":"eee19b7ec3c1b100","startTimeUnixNano":"1694718625557000000","durationNanos":"10000000000","attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]}],"matched":1},"spanSets":[{"spans":[{"spanID":"eee19b7ec3c1b100","startTimeUnixNano":"1694718625557000000","durationNanos":"10000000000","attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]}],"matched":1}]}],"metrics":{"inspectedBytes":"292781","completedJobs":1,"totalJobs":1}} ``` To format this in a more human-readable output, consider using a [tool such as `jq`](https://jqlang.github.io/jq/), which lets you to run the same `curl` command and pipe it to `jq` to format the block. For example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -G -s http://localhost:3200/api/search --data-urlencode 'q={ resource.service.name = "my.service" }' | jq ```