Loki data source on Grafana Labs

Configure the Loki data source

Thu, 27 Aug 2026 18:41:57 +0000

Configure the Loki data source

This document explains how to configure the Loki data source and describes the available configuration options. For general information on adding and managing data sources, refer to Data source management.

Grafana includes built-in support for Loki, so you don’t need to install a plugin.

Before you begin

Before you configure the data source, ensure you have:

Grafana permissions: The Organization administrator role to add and configure data sources. Administrators can also configure the data source with YAML provisioning.
Loki server URL: The address of your Loki server, including the port. Loki listens on port 3100 by default.
Authentication details: Any credentials or certificates your Loki server requires, such as a basic auth user and password or TLS certificates.

Note
Use TLS (Transport Layer Security) for an additional layer of security when working with Loki. For information on setting up TLS encryption with Loki, refer to Grafana Loki configuration parameters.

Add the Loki data source

To add the Loki data source:

Click Connections in the left-side menu.
Click Add new connection.
Type Loki in the search bar.
Select the Loki data source.
Click Add new data source in the upper right.

Grafana takes you to the Settings tab, where you set up your Loki configuration.

Configure the data source in the UI

The following sections describe the configuration options available for the Loki data source.

The first options set the name of your connection:

Setting	Description
Name	The data source name. This is how you refer to the data source in panels and queries. Examples: `loki-1`, `loki_logs`.
Default	Toggle to make this the default data source. Grafana selects it by default in new panels and queries.

Connection

Configure how Grafana connects to your Loki server.

Setting	Description
URL	The base URL of your Loki server. If Loki runs locally, use `http://localhost:3100`. If Loki runs on a networked server, use its URL and port, for example `http://loki.example.com:3100`.

Note
Enter only the base URL. Don’t append API paths such as /loki/api/v1/push. That endpoint sends logs to Loki with an agent like Grafana Alloy; it isn’t used to query Loki from the data source.

If your Grafana instance runs on Grafana Cloud, a localhost or private network address refers to Grafana servers rather than your network, so it can’t reach a self-hosted Loki. Use Private data source connect (PDC) to query Loki on a private network.

Authentication

Configure how Grafana authenticates with your Loki server. Select an authentication method and provide any required credentials.

Setting	Description
Basic authentication	The most common authentication method. Enter the user name and password for your Loki server.
Forward OAuth Identity	Toggle on to forward the OAuth access token, and the OIDC ID token if available, of the user querying the data source.
With credentials	Toggle on to send credentials such as cookies or authentication headers with cross-site requests.
TLS Client Auth	Toggle on to authenticate with a client certificate. Provide the Server name, Client cert, and Client key.
With CA cert	Toggle on to verify a self-signed TLS certificate. Follow the instructions from your certificate authority (CA) to obtain the certificate file.
Skip TLS verify	Toggle on to bypass TLS certificate verification. Skipping verification isn’t recommended unless absolutely necessary or for testing.

Note
For Grafana Cloud-hosted Loki, use Basic authentication with your Grafana Cloud user ID as the user name and a Cloud Access Policy token as the password. The token’s access policy must include the logs:read scope. Create tokens in the Grafana Cloud Portal. A token’s value is shown only once, so copy it when you create it.

Custom HTTP headers

Add custom HTTP headers to pass values that your Loki instance requires.

Setting	Description
Header	The name of the custom header.
Value	The value of the header.

For a multi-tenant Loki, one configured with auth_enabled: true, add the X-Scope-OrgID header with your tenant ID so Loki knows which tenant to query. Without it, queries against a multi-tenant Loki fail with an authentication error or return no data.

Additional settings

The Additional settings section is collapsible and contains optional settings that give you more control over the data source. It’s open by default and includes advanced HTTP settings, alerting, query, and derived field options.

Advanced HTTP settings

Configure additional HTTP behavior for requests to Loki.

Setting	Description
Allowed cookies	Specify cookies by name that should be forwarded to the data source. By default, the Grafana proxy deletes forwarded cookies.
Timeout	The HTTP request timeout, in seconds. There’s no default, so set a value that suits your queries.

Secure Socks Proxy

The Secure Socks Proxy settings appear only when the secure SOCKS data source proxy is enabled in the Grafana configuration file. When enabled, you can route data source requests through a secure SOCKS proxy. For more information, refer to Configure a data source connection proxy.

Alerting

Configure how the data source works with Grafana Alerting.

Setting	Description
Manage alert rules in Alerting UI	Toggle on to manage data source-managed alert rules for the Loki data source in the Grafana Alerting UI. These rules are stored and evaluated by the Loki ruler. The default follows your Grafana configuration.

To manage other alerting resources, such as the alerts these rules generate, add an Alertmanager data source. For more information on alerting with Loki, refer to Loki alerting.

Queries

Configure query behavior for the data source.

Setting	Description
Maximum lines	Sets the maximum number of log lines Loki returns. Increase the limit for a larger result set for ad hoc analysis. Decrease the limit if your browser is slow when displaying log results. The default is `1000`.

Derived fields

Use derived fields to extract new fields from your logs and create a link from the value of the field.

For example, you can link to your tracing backend directly from your logs, or link to a user profile page when a log line contains a corresponding userId. These links appear in the log details.

You can add multiple derived fields.

Note
If you use Grafana Cloud, you can request modifications to this feature by opening a support ticket from the Grafana Cloud Portal.

Each derived field has the following options:

Setting	Description
Name	The field name. Displayed as a label in the log details.
Type	The type of the derived field. Choose Regex in log line to parse a value from the log content, or Label to match a label key.
Regex	When the type is Regex in log line, a regular expression that parses part of the log message. Include a capture group; Grafana uses the first captured group as the field value.
Label	When the type is Label, the input matches as a regular expression against label keys, so a pattern like `trace[_]?id` matches variations such as `traceid` and `trace_id`. Matches any label: indexed, parsed, or structured metadata.
URL/query	The full link URL for external links, or a query for the target data source for internal links. Interpolate the field value with the `${__value.raw}` macro.
URL Label	Optional. A custom display label for the link. Overrides the default label, which is the full external URL or the name of the linked internal data source.
Internal link	Toggle on to define an internal link to a tracing data source. Select the target data source from the selector.
Open in new tab	Toggle on to open the link in a new tab or window.
Show example log message	Paste an example log line to test the regular expression of your derived fields.

Caution
Using complex regular expressions in either type can affect browser performance when processing large volumes of logs. Use simpler patterns when possible.

Note
A derived field produces a single value per log line, so you can’t combine multiple labels or capture groups into one field or link.

For an internal link, the derived field only builds a link to the target data source using the extracted value. The trace must already be ingested into the target data source, such as Tempo, for the link to resolve. If a matching trace doesn’t exist, the link opens the target data source but returns no trace.

Troubleshoot interpolation

Use the debug section to see what your fields extract and how the URL interpolates. Click Show example log message to display a text area where you can enter a log message.

The new field with the link appears in the log details:

Verify the connection

Click Save & test at the bottom of the settings. Grafana attempts to connect to your Loki server and query its labels. When the test succeeds, Grafana displays a success message confirming the data source is working. If the test fails, review the error message and refer to Troubleshoot Loki issues.

Note
To troubleshoot configuration and other issues, check the log file located at /var/log/grafana/grafana.log on Unix systems, or in <grafana_install_dir>/data/log on other platforms and manual installations.

Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to Provisioning Grafana.

Provisioning examples

The following example provisions a basic Loki data source:

apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://localhost:3100
    jsonData:
      timeout: 60
      maxLines: 1000

Using basic authorization and a derived field:

You must escape the dollar ($) character in YAML values because it can be used to interpolate environment variables:

apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://localhost:3100
    basicAuth: true
    basicAuthUser: my_user
    jsonData:
      maxLines: 1000
      derivedFields:
        # Field with internal link pointing to data source in Grafana.
        # datasourceUid value can be anything, but it should be unique across all defined data source uids.
        - datasourceUid: my_jaeger_uid
          matcherRegex: "traceID=(\\w+)"
          name: TraceID
          # url will be interpreted as query for the datasource
          url: '$${__value.raw}'
          # optional for URL Label to set a custom display label for the link.
          urlDisplayLabel: 'View Trace'

        # Field with external link.
        - matcherRegex: "traceID=(\\w+)"
          name: TraceID
          url: 'http://localhost:16686/trace/$${__value.raw}'
    secureJsonData:
      basicAuthPassword: test_password

Using a Jaeger data source:

In this example, the Jaeger data source’s uid value should match the Loki data source’s datasourceUid value.

datasources:
  - name: Jaeger
    type: jaeger
    url: http://jaeger-tracing-query:16686/
    access: proxy
    # UID should match the datasourceUid in derivedFields.
    uid: my_jaeger_uid

Configure with Terraform

You can configure the Loki data source using Terraform with the Grafana Terraform provider.

For more information about provisioning resources with Terraform, refer to Grafana as code using Terraform.

The following example provisions a Loki data source with a maximum line limit:

resource "grafana_data_source" "loki" {
  name = "Loki"
  type = "loki"
  url  = "http://localhost:3100"

  json_data_encoded = jsonencode({
    maxLines = 1000
  })
}

The following example uses basic authentication and defines a derived field that links to a Jaeger data source:

resource "grafana_data_source" "loki" {
  name           = "Loki"
  type           = "loki"
  url            = "http://localhost:3100"
  basic_auth_enabled  = true
  basic_auth_username = "my_user"

  json_data_encoded = jsonencode({
    maxLines = 1000
    derivedFields = [
      {
        datasourceUid   = "my_jaeger_uid"
        matcherRegex    = "traceID=(\\w+)"
        name            = "TraceID"
        url             = "$${__value.raw}"
        urlDisplayLabel = "View Trace"
      }
    ]
  })

  secure_json_data_encoded = jsonencode({
    basicAuthPassword = "<LOKI_PASSWORD>"
  })
}

Replace the placeholders with your own values:

<LOKI_PASSWORD>: The password for the basic authentication user.

For all available configuration options, refer to the Grafana provider data source resource documentation.

Loki query editor

Thu, 27 Aug 2026 18:41:57 +0000

Loki query editor

The Loki data source’s query editor helps you create log and metric queries that use the Loki query language, LogQL.

For general documentation on querying data sources in Grafana, refer to Query and transform data.

Choose a query editing mode

The Loki query editor has two modes:

Builder mode, which provides a visual query designer.
Code mode, which provides a feature-rich editor for writing queries.

To switch between the editor modes, select the corresponding Builder and Code tabs.

To run a query, select Run queries located at the top of the editor.

Note
To run Loki queries in Explore, select Run query.

Each mode is synchronized, so you can switch between them without losing your work, although there are some limitations. Builder mode doesn’t support some complex queries. When you switch from Code mode to Builder mode with such a query, the editor displays a warning message that explains how you might lose parts of the query if you continue. You can then decide whether you still want to switch to Builder mode.

You can also augment queries by using template variables.

The query editor toolbar contains the following elements:

Kick start your query - Click to see a list of sample queries that help you quickly start creating LogQL queries. These include log query starters and metric query starters. Click the arrow next to each to see the available query options, then continue to complete your query.
Label browser - Use the Loki label browser to navigate through your labels and values, and build queries.

To navigate Loki and build a query:

Choose labels to locate.
Search for the values of your selected labels.

The search field supports fuzzy search, and the label browser also supports faceting to list only possible label combinations.
Select the Show logs button to display log lines based on the selected labels, or select the Show logs rate button to show the rate based on metrics such as requests per second. Additionally, you can validate the selector by clicking the Validate selector button. Click Clear to start from the beginning.

Explain query - Toggle to display a step-by-step explanation of all query components and operations.

Builder/Code - Click the corresponding Builder or Code tab on the toolbar to select an editor mode.

Builder mode

Builder mode helps you build queries using a visual interface without needing to manually enter LogQL. This option is best for users who have limited or no previous experience working with Loki and LogQL.

Label filters

Select labels and their values from the dropdown list. When you select a label, Grafana retrieves available values from the server.

Use the + button to add a label and the x button to remove a label. You can add multiple labels.

Select comparison operators from the following options:

= - equal to
!= - is not equal
=~ - matches a regular expression
!~ - doesn’t match a regular expression

Select values by using the dropdown, which displays all possible values based on the label selected.

Operations

Select the + Operations button to add operations to your query. The query editor groups operations into related sections, and you can type while the operations dropdown is open to search and filter the list.

The query editor displays a query’s operations as boxes in the operations section. Each operation’s header displays its name, and additional action buttons appear when you hover your cursor over the header:

Button	Action
	Replaces the operation with different operation of the same type.
	Opens the operation’s description tooltip.
	Removes the operation.

The query editor groups operations into the following sections:

Aggregations - see Built-in aggregation operators
Range functions - see Range Vector aggregation
Formats - see Log queries
Binary operations - see Binary operators
Label filters - see Label filter expression
Line filters - see Line filter expression

Some operations make sense only when used in a specific order. If adding an operation would result in a nonsensical query, the query editor adds the operation to the correct place. To reorder operations manually, drag the operation box by its name and drop it into the desired place. For more information, refer to Order of operations.

Hints

In some cases, the query editor can detect which operations are most appropriate for a selected log stream. When it does, it shows a hint next to the + Operations button. Click the hint to add the operations to your query.

Builder mode example

The following steps build a metric query that counts the rate of error lines for the payments app, without writing any LogQL:

Under Label filters, select the app label and the value payments.
Select + Operations, then from Line filters add a Line contains operation and enter error.
Select + Operations again, then from Range functions add Rate with a range of [5m].
From Aggregations, add Sum to combine the results into a single series.

The builder produces the equivalent LogQL query:

sum(rate({app="payments"} |= `error` [5m]))

Switch to Code mode at any time to view or refine the generated query.

Code mode

In Code mode, you can write complex queries using a text editor with autocompletion, syntax highlighting, and query validation. It also provides the label browser to further help you write queries.

For more information about the Loki query language, refer to the Loki documentation.

Use autocompletion

Code mode’s autocompletion feature works automatically while typing.

The query editor can autocomplete static functions, aggregations, and keywords, and also dynamic items like labels. The autocompletion dropdown includes documentation for the suggested items where available.

Options

The following options are the same for both Builder and Code mode:

Legend - Controls the time series name, using a name or pattern. For example, {{hostname}} is replaced with the label value for the label hostname.
Type - Selects the query type to run. The instant type queries against a single point in time, using the To time from the time range. The range type queries over the selected range of time.
Line limit - Defines the upper limit for the number of log lines returned by a query. The default is 1000.
Direction - Determines the search order. Backward is a backward search starting at the end of the time range. Forward is a forward search starting at the beginning of the time range. The default is Backward.
Step - Sets the step parameter of Loki metric queries. The default value equals the value of the $__auto variable, which Grafana calculates using the time range and the width of the graph in pixels.

Create a log query

Loki log queries return the contents of the log lines. You can query and display log data from Loki via Explore, and with the Logs panel in dashboards.

To display the results of a log query, select the Loki data source, then enter a LogQL query.

For more information about log queries and LogQL, refer to the Loki log queries documentation.

Log query examples

A log query has two parts: a stream selector that chooses log streams by label, and an optional pipeline that filters and parses the matching lines. Replace the label matchers with values from your own logs.

Return every log line from the payments app:

{app="payments"}

Keep only lines that contain the word error, and drop health-check noise:

{app="payments"} |= `error` != `healthcheck`

Match lines with a regular expression, for example any 4xx or 5xx status:

{app="nginx"} |~ `status=(4|5)..`

Parse structured logs, then filter on an extracted field. This example parses logfmt logs and keeps requests slower than 500 ms:

{app="payments"} | logfmt | duration > 500ms

Parse JSON logs and keep only error-level entries:

{namespace="prod", app="checkout"} | json | level=`error`

Use case: debug a failing service. Combine a namespace and app selector with a parser and label filter to isolate the errors for one service during an incident:

{namespace="prod", app="checkout"} |= `error` | logfmt | status >= 500

Show log context

In Explore, you can retrieve the context surrounding your log results by clicking the Show context button. You can then investigate the logs from the same log stream that came before and after the log message you’re interested in.

The initial log context query is created from all labels defining the stream for the selected log line. You can use the log context query editor to widen the search by removing one or more of the label filters from the log stream. If you used a parser in your original query, you can refine your search by using extracted label filters.

You can also adjust the size of the time window that the log context loads, so you can expand or narrow the range of surrounding logs without leaving the context view.

To reduce the repetition of selecting and removing the same labels when examining multiple log context windows, Grafana stores your selected labels and applies them to each open context window. This lets you navigate through various log context windows without having to reapply your filters.

To reset filters and use the initial log context query, click the Revert to initial query button next to the query preview.

Tail live logs

Loki supports live tailing of logs in real-time in Explore.

Live tailing relies on two WebSocket connections: one between the browser and Grafana server, and another between the Grafana server and Loki server.

To start tailing logs click the Live button in the top right corner of the Explore view.

Proxying examples

If you use reverse proxies, configure them accordingly to use live tailing:

Using Apache2 for proxying between the browser and the Grafana server:

ProxyPassMatch "^/(api/datasources/proxy/\d+/loki/api/v1/tail)" "ws://127.0.0.1:3000/$1"

Using NGINX:

This example provides a basic NGINX proxy configuration. It assumes that the Grafana server is available at http://localhost:3000/, the Loki server is running locally without proxy, and your external site uses HTTPS. If you also host Loki behind an NGINX proxy, repeat the following configuration for Loki.

In the http section of NGINX configuration, add the following map definition:

  map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
  }

In your server section, add the following configuration:

  location ~ /(api/datasources/proxy/\d+/loki/api/v1/tail) {
      proxy_pass          http://localhost:3000$request_uri;
      proxy_set_header    Host              $host;
      proxy_set_header    X-Real-IP         $remote_addr;
      proxy_set_header    X-Forwarded-for   $proxy_add_x_forwarded_for;
      proxy_set_header    X-Forwarded-Proto "https";
      proxy_set_header    Connection        $connection_upgrade;
      proxy_set_header    Upgrade           $http_upgrade;
  }

  location / {
      proxy_pass          http://localhost:3000/;
      proxy_set_header    Host              $host;
      proxy_set_header    X-Real-IP         $remote_addr;
      proxy_set_header    X-Forwarded-for   $proxy_add_x_forwarded_for;
      proxy_set_header    X-Forwarded-Proto "https";
  }

Create a metric query

You can use LogQL to wrap a log query with functions that create metrics from your logs.

For more information about metric queries, refer to the Loki metric queries documentation.

Metric query examples

Metric queries apply a range aggregation, such as rate or count_over_time, over a log query and an optional range like [5m]. You can then wrap the result in an aggregation operator, such as sum or topk, to group or rank the series.

Count the rate of error lines per second across the payments app:

sum(rate({app="payments"} |= `error` [5m]))

Break the request rate down by status code:

sum by (status) (rate({app="nginx"} | logfmt [5m]))

Count how many timeouts occurred in the last hour:

count_over_time({app="payments"} |= `timeout` [1h])

Calculate the 95th percentile request duration per route. This example parses logfmt logs and unwraps the numeric duration field:

quantile_over_time(0.95, {app="payments"} | logfmt | unwrap duration [5m]) by (route)

Use case: find the noisiest endpoints. Rank the top 10 request paths by log volume over the last hour:

topk(10, sum by (path) (count_over_time({app="nginx"} | logfmt [1h])))

Query splitting

To keep large queries responsive, Grafana automatically splits Loki queries that cover a long time range into a series of smaller sub-queries. Grafana runs the sub-queries and merges the results, and the query progresses incrementally so you see partial results as they arrive. Query splitting applies to both log and metric queries and requires no configuration.

Structured metadata

Loki stores three types of labels: indexed labels, parsed fields, and structured metadata. The query editor and log details treat all three types as labels you can filter on and use in derived fields. For more information, refer to the Loki structured metadata documentation.

Apply annotations

You can use Loki log queries as a source for dashboard annotations. For details, refer to Loki annotations.

Loki template variables

Thu, 27 Aug 2026 18:41:57 +0000

Loki template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables.

For an introduction to templating and template variables, refer to the Templating and Add and manage variables documentation.

Use query variables

Variables of the type Query help you query Loki for lists of labels or label values. The Loki data source provides a form to select the type of values expected for a given variable.

The form has these options:

Query type	Example label	Example stream selector	List returned
Label names	Not required	Not required	Label names.
Label values	`label`		Label values for `label`.
Label values	`label`	`log stream selector`	Label values for `label` in the specified `log stream selector`.

Query variable example

To create a variable that lists the values of the app label:

Add a new variable of type Query and select the Loki data source.
Set Query type to Label values and Label to app.
Optionally, enter a Stream selector such as {namespace="$namespace"} to return only the values that appear within another variable’s selection. This creates a chained variable that updates when the parent variable changes.

Reference the variable in a query with its name, prefixed by $. For a single-value variable, use an exact match:

{app="$app"}

When a variable has Multi-value or Include All enabled, Grafana joins the selected values into a regular expression, such as payments|checkout. Use the regular expression match operator =~ so the query matches any of the selected values:

{app=~"$app"}

Format variable values

By default, the Loki data source formats variable values based on the variable’s settings:

A single-value variable is inserted as-is, without escaping.
A Multi-value or Include All variable has each value escaped for use in a regular expression and joined with |, for example payments|checkout.

This default suits the regular expression match operator =~, but it can produce unexpected results when a custom or static value contains characters that Grafana escapes, or when you insert a value into a context that isn’t a regular expression. To control the formatting, use a format option in the variable syntax:

Syntax	Result
`${app:raw}`	Inserts the value with no escaping. Use this for custom or static values that must pass through unchanged.
`${app:pipe}`	Joins multiple values with `\|` without regular expression escaping.
`${app:regex}`	Escapes the values for a regular expression and joins them with `\|`.

For example, to use a raw custom value in an exact-match selector:

{app="${app:raw}"}

For the complete list of format options, refer to Advanced variable format options.

Use ad hoc filters

Loki supports ad hoc filters. Use them to specify any number of key/value filters that Grafana applies automatically to all of your Loki queries, without editing each query.

For example, if you add an ad hoc filter for level = error, Grafana appends the matcher to the stream selector of every Loki query on the dashboard, so a query like {app="payments"} runs as {app="payments", level="error"}.

For more information, refer to Add ad hoc filters.

Use the `$__auto` variable for Loki metric queries

Consider using the $__auto variable in your Loki metric queries, which will automatically be substituted with the step value for range queries, and with the selected time range’s value (computed from the starting and ending times) for instant queries.

Use it as the range in a range aggregation:

sum(rate({app="payments"} |= `error` [$__auto]))

Note
Prefer $__auto over $__range for the range in a metric query. The $__range variable resolves to the entire selected time range, so using it as a range vector, such as [$__range], makes each data point aggregate over the whole window. This can scan far more data and look back further than you intend.

For more information about variables, refer to Global built-in variables.

Known limitations

Keep the following limitations in mind when you use template variables with the Loki data source:

Only indexed labels are available for label variable queries. A Query variable of type Label values returns the values of indexed stream labels. Values that exist only as parsed fields or structured metadata aren’t available, because the variable query doesn’t run a log pipeline to extract them.
Long time ranges can return incomplete values. Label and label-value variable queries run over the dashboard’s selected time range. Over a long range with high log volume, the query can time out or hit the limits configured in Loki and return an incomplete list of values. Narrow the time range, or add a stream selector to the variable query, so it scans less data.

Label extraction and indexing in Loki

Labels are key to how Loki aggregates and queries logs. Labels are key-value pairs that add contextual information to log entries, and Loki uses them to organize, filter, and search log data efficiently.

Label extraction

When it ingests logs, Loki extracts labels from log lines using regular expressions, so you can define custom patterns that match your log formats. For example, given a log line like the following:

2023-07-25 12:34:56 INFO: Request from IP A.B.C.D to endpoint /api/data

You can define a regular expression that extracts the log level (INFO), IP address (A.B.C.D), and endpoint (/api/data) as labels. You can then use those labels to filter and aggregate log entries.

Index labels

Loki indexes the extracted labels. The index maps labels to their log entries, so Loki can retrieve logs by label without scanning the entire dataset. For example, if a job label represents the services in your application, Loki indexes each job’s logs separately, so you can query a single job quickly.

Combine Loki indexed labels with Grafana template variables to build dynamic queries. Use template variables to select and filter logs by labels such as job names, instance IDs, or severity levels, so you can explore and visualize your log data interactively.

Loki annotations

Thu, 27 Aug 2026 18:41:57 +0000

Loki annotations

Annotations overlay rich event information on top of graphs. With the Loki data source, annotations are built from log queries: each log line that a LogQL query returns within the dashboard time range becomes an annotation, so you can correlate log events with the rest of your dashboard data.

Before you begin

Before you create Loki annotations, ensure you have:

A configured Loki data source.
A LogQL log query that returns the events you want to annotate.

How Loki annotations work

Unlike data sources that build annotations from a separate query language, Loki annotations use the same LogQL log queries you use elsewhere. Grafana runs your log query for the dashboard time range and turns each returned log line into an annotation. By default, Grafana uses the log content as the annotation text and the log stream labels as tags, so you don’t need to create any additional mapping.

You can only use log queries as a source for annotations. Metric queries aren’t supported.

Create an annotation query

To add a Loki annotation to a dashboard:

Open the dashboard where you want to add annotations.
Click Edit, then click Settings in the top navigation.
Select the Annotations tab.
Click Add annotation query.
Enter a Name for the annotation, for example, Deploy events.
Select your Loki data source.
Enter a LogQL log query and configure the optional fields described in the following table.
Click Save dashboard.

The annotation query editor provides the following fields:

Field	Description
Query	The LogQL log query that selects the log lines to annotate.
Title	Optional. A literal name or a template for the annotation title. Reference labels with `{{label}}`, for example `{{level}}`.
Tags	Optional. A comma-separated list of label keys to use as annotation tags, for example `app,level`. When empty, Grafana uses the log stream labels as tags.
Text	Optional. A literal string or a template for the annotation text. Reference labels with `{{label}}`. When empty, Grafana uses the log line content as the text.

Template fields resolve against the labels on each returned log line, including parsed fields. To reference a value that isn’t an indexed label, add a parser such as logfmt or json to your query so the field is available.

Example

To annotate deployment events, configure the annotation query with the following values:

Field	Value
Query	`{app="deploy-bot"} \|= "deployed"`
Title	`{{app}} deployed`
Tags	`app,env`
Text	`{{message}}`

Each matching log line becomes an annotation titled with the deploying app, tagged with the app and env labels, and described with the parsed message field. Because the Text template references a parsed field, the query needs a parser, for example:

{app="deploy-bot"} |= `deployed` | logfmt

Troubleshoot annotations

If annotations don’t appear as expected, try the following solutions.

Annotations don’t appear

Verify the log query returns results in Explore for the same time range. Annotations come from returned log lines, so a query that returns no logs produces no annotations.
Confirm the query is a log query, not a metric query.
Widen the dashboard time range to include the log events you expect.

Too many annotations appear

Add label filters or a line filter to your LogQL query to narrow the returned log lines.
Reduce the maximum number of lines the query returns.

Loki alerting

Thu, 27 Aug 2026 18:41:57 +0000

Loki alerting

The Loki data source supports Grafana Alerting. You can alert on your logs so Grafana notifies you when a LogQL metric query crosses a threshold, for example when the rate of error logs spikes.

Loki works with two kinds of alert rules:

Grafana-managed alert rules: Grafana evaluates the rule against a LogQL metric query. This is the most common approach and works with any Loki data source.
Data source-managed alert rules: The Loki ruler stores and evaluates the rule. Use these when you want alert rules to live alongside your Loki deployment. Enable Manage alert rules in Alerting UI on the data source to create and edit them from Grafana.

Before you begin

Before you create alert rules with Loki data, ensure you have:

A configured Loki data source.
Permissions to create alert rules in Grafana.
Familiarity with Grafana Alerting concepts.
For data source-managed rules, a Loki deployment with the ruler enabled, and the Manage alert rules in Alerting UI setting turned on in the data source configuration. To route the resulting alerts, add a separate Alertmanager data source.

Supported queries for alerting

Alert rules require queries that return numeric data that Grafana can evaluate against a threshold. Wrap a Loki log query in a LogQL metric function, such as rate or count_over_time, to produce numeric time series.

Note
Use a metric query for alerting, not a plain log query. A log query returns log lines, which Grafana can’t evaluate against a threshold.

The following queries are common starting points for alerting. Replace the label matchers with values from your own logs.

Rate of error lines per second:

sum(rate({app="my-app"} |= `error` [5m]))

Error rate broken down by status code, to alert per endpoint or service:

sum by (status) (rate({app="nginx"} | logfmt | status=~`5..` [5m]))

Count of slow requests, for a latency alert:

sum(count_over_time({app="my-app"} | logfmt | duration > 1s [5m]))

Detect a service that stopped logging by alerting when the count drops to zero:

sum(count_over_time({app="my-app"} [10m]))

Create a Grafana-managed alert rule

To create a Grafana-managed alert rule using Loki data:

Go to Alerting > Alert rules.
Click New alert rule.
Enter a name for your alert rule.
In the Define query and alert condition section:
- Select your Loki data source.
- Enter a LogQL metric query.
- Add a Reduce expression if your query returns multiple series.
- Add a Threshold expression to define the alert condition.
Configure the Set evaluation behavior section:
- Select or create a folder and evaluation group.
- Set the evaluation interval and pending period.
Add labels and annotations to provide context for notifications.
Click Save rule.

For detailed instructions, refer to Create a Grafana-managed alert rule.

Example: alert on error log rate

This example fires when the rate of log lines that contain error exceeds a threshold:

Create a new alert rule.
Configure the query, replacing the label matcher with your own:
logql
```
sum(rate({app="my-app"} |= `error` [5m]))
```
Add expressions:
- Reduce: Last, to get the most recent value.
- Threshold: Is above 10.
Set evaluation to run every 1 minute with a 5-minute pending period.
Save the rule.

Manage data source-managed alert rules

When you turn on Manage alert rules in Alerting UI for the Loki data source, you can create and edit alert and recording rules that the Loki ruler stores and evaluates. These rules appear in the Grafana Alerting UI alongside your Grafana-managed rules, grouped under the Loki data source.

The Loki data source configuration doesn’t include Alertmanager routing. To route the resulting alerts, add a separate Alertmanager data source. For more information, refer to Configure the Loki data source and Data source-managed alert rules.

Best practices

Follow these recommendations to create reliable alerts with Loki data.

Test queries before alerting

Verify your query returns numeric data before you create an alert:

Go to Explore.
Select your Loki data source.
Run the LogQL metric query you plan to use for alerting.
Confirm the result is numeric and suitable for threshold evaluation.

Reduce a time series to a single value

A Grafana-managed rule evaluates a single number against the threshold. A range query returns a time series with many points, which the rule can’t evaluate directly. Either set the query Type to instant, or add a Reduce expression, such as Last or Mean, between the query and the Threshold expression. Without one of these, the rule can evaluate incorrectly.

Set a pending period to avoid false positives

A transient condition, such as a brief network blip or a momentary spike in errors, can cross the threshold for a single evaluation. Set a pending period so the rule fires only after the condition holds for a sustained duration. A pending period of a few evaluation intervals reduces false positives from short-lived conditions.

Handle no data versus zero results

A LogQL metric query returns no series when no log lines match, rather than returning a value of 0. Grafana evaluates a rule that receives no series as No Data, not Normal, which can cause an alert to enter the No Data state instead of resolving.

To avoid unexpected No Data states:

Apply the threshold in the alert condition rather than writing a query so restrictive that it returns nothing. For example, query the error rate and compare it in a Threshold expression, instead of writing a query that only returns lines above the threshold.
Set the No Data behavior under Configure no data and error handling to the state you want. The options are No Data, Alerting, Normal, and Keep Last State. Choose Normal if the absence of matching logs isn’t a problem.

Note
Editing and saving an alert rule resets its state to Normal. The rule re-enters Pending or Alerting on the next evaluation if the condition is still met. This is standard Grafana Alerting behavior and isn’t specific to the Loki data source.

Troubleshoot alerting

If your Loki alerts don’t work as expected, use the following sections to diagnose common issues.

Alerts don’t fire

Confirm the query is a LogQL metric query that returns numeric data in Explore.
Ensure the evaluation interval allows enough time for data to be available.
Review the alert rule’s health and any error messages in the Alerting UI.

Alert enters No Data unexpectedly

The query returned no series because no log lines matched. A LogQL metric query returns no series rather than 0 when nothing matches. Set the No Data behavior under Configure no data and error handling, or restructure the query so it returns a value that you compare with a threshold.
Confirm logs exist for the query and time range in Explore.

Alert fires on brief spikes

Add or increase the pending period so the rule fires only when the condition holds over several evaluations.
Widen the range in the metric query, for example from [1m] to [5m], so short-lived spikes have less effect.

Data source-managed rules don’t appear

Confirm Manage alert rules in Alerting UI is enabled in the Loki data source configuration.
Verify the Loki ruler is enabled and reachable from Grafana.

For more help, refer to Troubleshoot Loki issues.

Troubleshoot Loki data source issues

Thu, 27 Aug 2026 18:41:57 +0000

Troubleshoot Loki data source issues

This document provides solutions to common issues you might encounter when configuring or using the Loki data source. For configuration instructions, refer to Configure the Loki data source.

Note
This guide covers the Loki data source in Grafana, such as connecting to Loki, running queries, and configuring features. It doesn’t cover troubleshooting the Loki service itself. For help with a Loki deployment, including ingestion and query performance, refer to Manage and debug errors in the Loki documentation.

Connection errors

These errors occur when Grafana can’t reach your Loki server.

Save & test fails

Symptoms:

The data source test fails or times out.
Queries fail with network errors.

Possible causes and solutions:

Cause	Solution
Incorrect URL or port	Verify the URL setting. Loki listens on port `3100` by default, for example `http://localhost:3100`.
API path in the URL	Enter only the base Loki URL. Don’t append API paths such as `/loki/api/v1/push`, which sends logs to Loki rather than querying it.
Localhost URL on Grafana Cloud	On Grafana Cloud, a `localhost` or private address points to Grafana servers, not your network. Use PDC to reach a self-hosted Loki.
Missing tenant header	For a multi-tenant Loki, one with `auth_enabled: true`, add the `X-Scope-OrgID` custom HTTP header with your tenant ID.
Network or firewall block	Confirm the Grafana server can reach the Loki server, and that firewall rules allow the connection.
Authentication failure	Verify the credentials, tokens, or TLS certificates in the data source configuration. For Grafana Cloud Loki, confirm the access policy token includes the `logs:read` scope.
TLS verification failure	If Loki uses a self-signed certificate, add the CA certificate with With CA cert, or enable Skip TLS verify for testing.

Query errors

These errors occur when running queries against Loki.

No data or empty results

Symptoms:

A query runs without error but returns no data.
Panels show a “No data” message.

Possible causes and solutions:

Cause	Solution
Time range doesn’t contain logs	Expand the dashboard time range or confirm logs exist in Loki for that period.
Label selector matches no streams	Verify your stream selector labels and values with the label browser in the query editor.
Logs exceed retention	Confirm the logs you expect are within the retention period configured in Loki.

Query timeout or “too many outstanding requests”

Symptoms:

A query runs for a long time and then fails.
Errors mention timeouts, query limits, or too many outstanding requests.

Solutions:

Narrow the time range to reduce the volume of data scanned.
Add label filters or a line filter to reduce the result set.
Lower the Maximum lines setting on the data source or the line limit on the query.
Review the query and rate limits configured on your Loki deployment.

“maximum of series reached for a single query”

Symptoms:

A metric query fails with an error about the maximum number of series.

Solutions:

Add label matchers to reduce the number of streams the query touches.
Aggregate with sum by (...) to reduce cardinality.
Avoid high-cardinality labels in aggregations.

Performance issues

These issues relate to slow queries or rate limits.

Queries are slow or get rate limited

Solutions:

Narrow the time range and add label filters so queries scan less data.
Enable query caching, available in Grafana Enterprise and Grafana Cloud.
Review the query, ingestion, and rate limits on your Loki deployment.

Template variable errors

These errors occur when using template variables with the data source.

Variables return no values

Solutions:

Confirm the data source connection works by running Save & test.
Verify the label or stream selector in the variable query returns values in the query editor.
For cascading variables, confirm parent variables have valid selections.

Derived field and log-to-trace link issues

These issues affect derived fields, including the links from logs to traces.

The derived field or trace link doesn’t appear

Confirm the regular expression matches your log line format. A pattern that doesn’t match produces no value and no link. Use Show example log message in the data source configuration to test the pattern against a real log line.
For a Label type derived field, confirm a label key matches the pattern. The match runs against label keys, not values.
Check that the field value is captured. For a Regex in log line type, Grafana uses the first capture group, so the expression must include one.

The trace link opens but shows no trace

The derived field only builds a link to the target data source using the extracted value; it doesn’t create the trace. Confirm the trace is ingested into the target data source, such as Tempo, and that the extracted ID matches the trace ID stored there.
Verify the extracted value is the correct ID. An overly permissive regular expression can capture the wrong part of the log line. Anchor the pattern and make it specific, for example match traceID=(\w+) rather than a broad (\w+).

Can’t edit derived fields on a provisioned data source

Data sources created through provisioning are read-only in the UI, so you can’t change their derived fields from the data source configuration page.

For self-managed Grafana, edit the derived field configuration in the provisioning YAML file and restart Grafana, or reload provisioning.
For Grafana Cloud, open a support ticket from the Grafana Cloud Portal to request changes to a provisioned data source.

A single link needs values from more than one field

A derived field produces a single value per log line, so you can’t combine multiple labels or capture groups into one field or link. Create a separate derived field for each value you need to surface.

Live tailing issues

These issues affect real-time log tailing in Explore.

Live tailing doesn’t stream logs

Live tailing relies on WebSocket connections between the browser and Grafana, and between Grafana and Loki. Reverse proxies often block these connections.

Solutions:

Confirm your reverse proxy forwards WebSocket connections for the tail endpoint. Refer to the proxy examples in the Loki query editor.
Verify that intermediate load balancers and firewall rules allow WebSocket upgrades.
Confirm Loki itself is reachable and that the tail endpoint isn’t blocked.

Label-based access control (LBAC) issues

Label-based access control (LBAC) filters the logs a team can query based on labels. It’s available on Grafana Cloud and Grafana Enterprise, where it requires Grafana Enterprise Logs (GEL). If you can’t enable LBAC, or teams see more logs than expected, check the following.

Can’t enable LBAC on a provisioned data source

Note
You can’t enable LBAC on a provisioned or auto-provisioned Loki data source. Provisioning of LBAC data sources isn’t supported.

To use LBAC, create a new Loki data source manually and configure it with basic authentication, using your Cloud Access Policy (CAP) token as the password. Point it at the same Loki endpoint as the provisioned data source, then define LBAC rules on the new data source’s Permissions tab. For the full procedure, refer to Configure LBAC for data sources for Loki.

Teams can still query all logs

LBAC applies only when a team has at least one rule. If a team has no LBAC rules, its members can query all logs. Confirm each team has the rules you intend.
Grafana permissions are additive. A user’s basic role, such as Viewer, can grant access to the data source independently of team-scoped rules. Remove role and team permissions that aren’t required so that only the intended teams have Query permission on the LBAC data source.
Remove any label selectors from the Cloud Access Policy used by the data source. CAP label selectors override LBAC rules.

For details and limitations, refer to LBAC for data sources.

Enable debug logging

To capture detailed error information for troubleshooting:

Set the Grafana log level to debug in the configuration file:
ini
```
[log]
level = debug
```
Review logs in /var/log/grafana/grafana.log, or in your configured log location.
Look for Loki-specific entries that include request and response details.
Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you’ve tried these solutions and still encounter issues:

Check the Grafana community forums for similar issues.
Review the Loki data source GitHub issues for known bugs.
Consult the Loki documentation for service-specific guidance.
When reporting issues, include:
- Grafana and Loki versions
- Error messages, with sensitive information redacted
- Steps to reproduce
- Relevant configuration, with credentials redacted

Loki data source on Grafana Labs

Configure the Loki data source

Configure the Loki data source

Before you begin

Add the Loki data source

Configure the data source in the UI

Connection

Authentication

Custom HTTP headers

Additional settings

Advanced HTTP settings

Secure Socks Proxy

Alerting

Queries

Derived fields

Troubleshoot interpolation

Verify the connection

Provision the data source

Provisioning examples

Configure with Terraform

Loki query editor

Loki query editor

Choose a query editing mode

Toolbar elements

Builder mode

Label filters

Operations

Hints

Builder mode example

Code mode

Use autocompletion

Options

Create a log query

Log query examples

Show log context

Tail live logs

Proxying examples

Create a metric query

Metric query examples

Query splitting

Structured metadata

Apply annotations

Loki template variables

Loki template variables

Use query variables

Query variable example

Format variable values

Use ad hoc filters

Use the $__auto variable for Loki metric queries

Known limitations

Label extraction and indexing in Loki

Label extraction

Index labels

Loki annotations

Loki annotations

Before you begin

How Loki annotations work

Create an annotation query

Example

Troubleshoot annotations

Annotations don’t appear

Too many annotations appear

Related resources

Loki alerting

Loki alerting

Before you begin

Supported queries for alerting

Create a Grafana-managed alert rule

Example: alert on error log rate

Manage data source-managed alert rules

Best practices

Test queries before alerting

Reduce a time series to a single value

Set a pending period to avoid false positives

Handle no data versus zero results

Troubleshoot alerting

Alerts don’t fire

Alert enters No Data unexpectedly

Alert fires on brief spikes

Data source-managed rules don’t appear

Use the `$__auto` variable for Loki metric queries