Developer on Grafana Labs

Go SDK (programmatic use)

Wed, 29 Jul 2026 17:47:59 +0200

Go SDK (programmatic use)

You can use the Grafana MCP server as a Go library to build custom MCP server contexts, for example with custom TLS or debug settings. The package is available on pkg.go.dev.

What you’ll achieve

You integrate the server into your own Go program, using ComposedStdioContextFunc and GrafanaConfig (and optionally TLSConfig) so the server runs with your chosen configuration.

Before you begin

A Go toolchain and the module in your project: go get github.com/grafana/mcp-grafana.

Compose a stdio context with config

Create a GrafanaConfig with your Grafana URL, credentials (or token), and options such as Debug. For TLS to Grafana, set TLSConfig with CertFile, KeyFile, and CAFile. Then build the context function with mcpgrafana.ComposedStdioContextFunc(grafanaConfig) and pass it to your MCP server setup so the server uses this context when handling requests.

Example (structure only):

grafanaConfig := mcpgrafana.GrafanaConfig{
    Debug: true,
    TLSConfig: &mcpgrafana.TLSConfig{
        CertFile: "/path/to/client.crt",
        KeyFile:  "/path/to/client.key",
        CAFile:   "/path/to/ca.crt",
    },
}
contextFunc := mcpgrafana.ComposedStdioContextFunc(grafanaConfig)

Refer to the package documentation for the exact types and constructors.

Next steps

Observability (metrics and tracing) for Prometheus and OTLP.
Client TLS (Grafana connection) for TLS flag reference.

Observability (metrics, tracing, and logs)

Thu, 01 Oct 2026 13:22:11 +0000

Observability (metrics, tracing, and logs)

The MCP server can expose Prometheus metrics and supports OpenTelemetry distributed tracing and log export, following the OTel MCP semantic conventions.

Metrics require the SSE or streamable-http transport. Tracing and log export use standard OTEL_* environment variables and work with any transport, independently of --metrics.

Note: mcp-grafana currently only supports the OTLP/gRPC transport for both traces and logs. OTEL_EXPORTER_OTLP_PROTOCOL (and its _TRACES_PROTOCOL / _LOGS_PROTOCOL variants) are not honored — gRPC is used regardless.

What you’ll achieve

You can scrape MCP operation metrics (HTTP transports only) and export traces and logs to Tempo, Loki, or Grafana Cloud under any transport, including stdio.

Before you begin

The server running with SSE or streamable-http (metrics are not available with stdio).

Enable Prometheus metrics

When using SSE or streamable HTTP transports, enable Prometheus metrics with --metrics:

# Metrics on the main server at /metrics
./mcp-grafana -t streamable-http --metrics

# Metrics on a separate listen address
./mcp-grafana -t streamable-http --metrics --metrics-address :9090

Available metrics:

Metric	Type	Description
`mcp_server_operation_duration_seconds`	Histogram	MCP operation duration (labels: `mcp_method_name`, `gen_ai_tool_name`, `error_type`, `network_transport`, `mcp_protocol_version`, and — for `tools/call` on selected tools — `mcp_tool_operation`, `mcp_tool_resource_type`, `mcp_tool_phase`)
`mcp_server_session_duration_seconds`	Histogram	MCP client session duration (labels: `network_transport`, `mcp_protocol_version`)
`http_server_request_duration_seconds`	Histogram	HTTP server request duration (from otelhttp)

Note: Metrics are only available when using SSE or streamable HTTP transports. They are not available with stdio transport.

Tool-call dimension labels

For tools/call, mcp_server_operation_duration_seconds can carry up to three extra low-cardinality labels so durations are sliceable by what the call was doing:

Label	Source	Notes
`mcp_tool_operation`	the tool’s `operation` argument	Multiplexer tools only (e.g. `alerting_rules_write`); one of the tool’s declared operations, else `other`.
`mcp_tool_resource_type`	the tool’s `type` argument	e.g. the datasource plugin type on `create_datasource`; a plugin type the server ships a schema for, else `other`.
`mcp_tool_phase`	the tool’s result `_meta`	Phase of a multi-call flow (e.g. `create_datasource` schema guidance vs. actual creation).

Arguments are raw client input, so the two argument-derived labels are allowlisted by tool and by value: unlisted tools emit neither, and unlisted values collapse into one other bucket, giving each label a fixed maximum series count. Tool validation alone would not bound them — a rejected operation is still instrumented, and create_datasource’s type is free text that succeeds for unknown values. mcp_tool_phase is exempt: the tool sets it on its own result, and tools must keep that value set small.

The high-cardinality target of a call (the datasource uid, else name) is never a metric label — it is span-only as mcp.tool.target, and empty for calls that name no entity (see below).

Metric-name discovery

Metric	Type	Description
`mcp_metric_names_response_size_bytes`	Native histogram	Bytes consumed from complete successful discovery HTTP responses, after HTTP decompression
`mcp_metric_names_count`	Native histogram	Number of metric names decoded successfully before local filtering and pagination

These metrics help us understand the responses we get from prometheus-type backends when discovering available metric names. The metrics only measure the sizes of successful responses. The bounded backend label distinguishes prometheus (including compatible backends) from cloud_monitoring. Unlike the prometheus backend, Cloud Monitoring measures the full descriptor response, which contain more than just names, so comparing response byte sizes between the two backends is not meaningful.

For example, the p95 response size over the last hour is:

histogram_quantile(0.95, sum by (backend) (rate(mcp_metric_names_response_size_bytes[1h])))

These metrics use Native Histograms, so the scraper must also support native histograms.

Loki cost guardrail metrics

When the Loki query cost guardrail is enabled (--loki-guardrail-mode is shadow or enforce), every query_loki_logs call it evaluates increments exactly one of four counters:

Metric	Type	Incremented when
`mcp_loki_guardrail_admitted_total`	Counter	The query passed every enabled check
`mcp_loki_guardrail_would_block_total`	Counter	The query failed a check in `shadow` mode and ran anyway
`mcp_loki_guardrail_blocked_total`	Counter	The query failed a check in `enforce` mode and was rejected
`mcp_loki_guardrail_fail_open_total`	Counter	The guardrail could not reach a verdict and admitted the query

The four partition the guarded population, so their sum is the number of calls the guardrail evaluated, and each is a query count rather than a check count.

Labels:

Label	On	Values
`reason`	`would_block`, `blocked`	`selector` (no selective label matcher), `range` (effective time range over the cap), `bytes` (index/stats estimate over the budget)
`cause`	`fail_open`	`unparseable` (no stream selector the scanner recognises), `estimate_failed` (index/stats unavailable)
`backend`	all four	`loki`, `victorialogs`, `unknown`

A query can trip more than one check. It is counted once, labelled with the check that ran first — selector, then range, then bytes. That ordering is deliberate: the selectivity check is unconditional, so a query attributed to selector would not be admitted by raising --loki-guardrail-max-range or --loki-guardrail-max-bytes. When promoting from shadow to enforce, read sum(rate(mcp_loki_guardrail_would_block_total[5m])) for the size of the affected population and sum by (reason) (...) for whether tuning the bounds would shrink it.

Watch mcp_loki_guardrail_fail_open_total alongside them: a quiet would_block rate means “nothing would be blocked” only if the fail-open rate is also low. A high cause="unparseable" rate on backend="loki" means the guardrail’s LogQL scanner does not recognise the query shapes in use, which calls for improving the scanner rather than tuning the bounds. On backend="victorialogs" a high unparseable rate is expected — brace-less LogsQL is the normal shape there, and the byte-budget check never runs on that backend at all, which is why backend is a label rather than being folded together.

Neither the stream selector nor the LogQL is exported as a label: both are unbounded, and line filters can carry sensitive literals. The extracted selectors are logged at WARN and the full query at DEBUG instead.

Enable OpenTelemetry tracing

When OTEL_EXPORTER_OTLP_ENDPOINT (or the signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) is set, the server exports traces via OTLP/gRPC.

Local example:

# Send traces to a local Tempo instance
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_INSECURE=true \
./mcp-grafana -t streamable-http

Grafana Cloud example:

# Send traces to Grafana Cloud with authentication
OTEL_EXPORTER_OTLP_ENDPOINT=https://tempo-us-central1.grafana.net:443 \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic ..." \
./mcp-grafana -t streamable-http

Tool call spans follow naming like tools/call <tool_name> and include attributes such as gen_ai.tool.name, mcp.method.name, and mcp.session.id. The server supports W3C trace context propagation from the _meta field of tool call requests.

Trace context propagation

Under an HTTP transport (SSE or streamable-http) the server takes part in W3C trace context propagation on both sides, so a caller, mcp-grafana, and Grafana appear as one connected trace:

Inbound: a traceparent/tracestate header on the incoming request (from an MCP client or an upstream proxy) parents the server span, continuing the caller’s trace instead of starting a new one.
Outbound: requests to the Grafana API carry a traceparent naming mcp-grafana’s own span, so Grafana’s spans hang off ours.

Propagation is always active — it does not require OTEL_EXPORTER_OTLP_ENDPOINT. With trace export off, no spans are recorded but an inbound trace context is still passed through to Grafana rather than dropped.

The propagators used are configured with the standard OTEL_PROPAGATORS environment variable, defaulting to tracecontext,baggage. Set it to interoperate with a non-W3C system (b3, b3multi, jaeger, xray, ottrace, in any comma-separated combination) or to none to disable propagation entirely.

Tool call requests may also carry traceparent/tracestate in their MCP _meta field; when present, that context parents the tool span. This is the only propagation channel available under stdio, where there is no HTTP request to read headers from.

Listing traceparent or tracestate in GRAFANA_FORWARD_HEADERS is not needed and no longer has an effect: forwarding the caller’s traceparent verbatim would parent Grafana’s spans onto the caller and cut mcp-grafana out of the middle of the trace, so a forwarded value never overrides the propagated one. Forwarded trace headers still apply when propagation is disabled with OTEL_PROPAGATORS=none.

Tool-call spans also carry the tool-call dimensions mcp.tool.operation, mcp.tool.resource_type, and mcp.tool.phase, plus the span-only mcp.tool.target (the datasource uid, else name) for grouping the spans that touch one entity. mcp.tool.target is empty when a call names no entity — notably create_datasource’s phase=schema call, where the datasource does not exist yet — so stitching that call to the later phase=created one relies on the shared trace (when the client propagates context via _meta) or on mcp.session.id plus mcp_tool_resource_type, rather than on the target. Unlike the metric labels, these are attached for all tools with their raw values (no allowlist, no other bucket), since traces are high-cardinality by design. They require an HTTP transport (SSE or streamable-http, so a server span exists to enrich) and tracing enabled, but not --metrics; with stdio or tracing off, enrichment is a no-op.

Enable OpenTelemetry logs

When OTEL_EXPORTER_OTLP_ENDPOINT (or the signal-specific OTEL_EXPORTER_OTLP_LOGS_ENDPOINT) is set, the server also exports structured logs via OTLP/gRPC in addition to the existing plain-text stderr output. Logs carry trace_id and span_id from the active span so they correlate with exported traces.

Traces and logs resolve their endpoints independently, so the two signals can be enabled separately:

Setting only OTEL_EXPORTER_OTLP_TRACES_ENDPOINT enables tracing without log export.
Setting only OTEL_EXPORTER_OTLP_LOGS_ENDPOINT enables log export without tracing.
Setting the generic OTEL_EXPORTER_OTLP_ENDPOINT enables both.

# Send logs and traces to a local OTel collector
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_INSECURE=true \
./mcp-grafana -t streamable-http

Stderr logging continues unchanged; operators can pipe stderr to /dev/null if they only want logs going to the OTel collector.

Logs can be sent directly to any managed backend that accepts OTLP/gRPC — for example, Grafana Cloud — by pointing OTEL_EXPORTER_OTLP_LOGS_ENDPOINT (or the generic OTEL_EXPORTER_OTLP_ENDPOINT) at the remote gRPC endpoint and supplying auth via OTEL_EXPORTER_OTLP_LOGS_HEADERS (or OTEL_EXPORTER_OTLP_HEADERS), mirroring the tracing example above. A local OTel collector is optional — useful for fan-out, batching, or multi-backend routing, but not required.

The signal-specific variants OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, OTEL_EXPORTER_OTLP_LOGS_HEADERS, OTEL_EXPORTER_OTLP_LOGS_INSECURE, OTEL_EXPORTER_OTLP_LOGS_CERTIFICATE, OTEL_EXPORTER_OTLP_LOGS_TIMEOUT, and OTEL_EXPORTER_OTLP_LOGS_COMPRESSION are honored and override their generic OTEL_EXPORTER_OTLP_* counterparts — see the OTel exporter spec for the full list and precedence rules.

Note: If the configured collector is unreachable, log records are buffered in memory (default queue: 2048) and the oldest records are dropped once the queue fills. The process continues without blocking the service. Configure a local OTel collector if you need lossless buffering during outages.

Logs are also exported under the stdio transport, which makes it easy to centralize logs from local mcp-grafana instances invoked by IDE clients.

Run with Docker (metrics, tracing, and logs)

docker run --rm -p 8000:8000 \
  -e GRAFANA_URL=http://localhost:3000 \
  -e GRAFANA_SERVICE_ACCOUNT_TOKEN=<your token> \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4317 \
  -e OTEL_EXPORTER_OTLP_INSECURE=true \
  grafana/mcp-grafana \
  -t streamable-http --metrics

Next steps

Build, test, and lint

Wed, 29 Jul 2026 17:47:59 +0200

Build, test, and lint

Contributions are welcome. Open an issue or pull request on GitHub. This project is written in Go.

What you’ll achieve

You can run the server from source, execute the test suites, and run linters locally.

Before you begin

Go installed for your platform.
Optional: Docker for integration tests.

Run the server locally

STDIO (default for local development):

make run

SSE:

go run ./cmd/mcp-grafana --transport sse

Build and run the Docker image

Build the image:

make build-image

Run in SSE mode (image default):

docker run -it --rm -p 8000:8000 mcp-grafana:latest

Run in stdio mode:

docker run -it --rm mcp-grafana:latest -t stdio

Run tests

There are three types of tests available: unit tests, integration tests, and cloud tests.

Unit tests (no external dependencies required):

make test-unit

You can also run:

make test

Integration tests (requires Docker containers):

make test-integration

Cloud tests (requires Grafana Cloud instance and credentials):

make test-cloud

Note: Cloud tests are automatically configured in CI. For local development, you’ll need to set up your own Grafana Cloud instance and credentials.

For the full integration suite, start the supporting services first (Grafana and its dependencies on port 3000):

make run-test-services
make test-integration

If you add tools, add integration tests; existing tests are a good template.

Run linters

make lint

The repository includes a custom linter for unescaped commas in jsonschema struct tags (description fields must be escaped with \\, to avoid truncation). Run only that linter:

make lint-jsonschema

Refer to the JSONSchema linter README in the repository.